Not-For-All-Audiences

nsfw

Instructions to use NeverSleep/Lumimaid-v0.2-12B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use NeverSleep/Lumimaid-v0.2-12B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="NeverSleep/Lumimaid-v0.2-12B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("NeverSleep/Lumimaid-v0.2-12B")
model = AutoModelForCausalLM.from_pretrained("NeverSleep/Lumimaid-v0.2-12B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Local Apps Settings

vLLM

How to use NeverSleep/Lumimaid-v0.2-12B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "NeverSleep/Lumimaid-v0.2-12B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NeverSleep/Lumimaid-v0.2-12B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/NeverSleep/Lumimaid-v0.2-12B

SGLang

How to use NeverSleep/Lumimaid-v0.2-12B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "NeverSleep/Lumimaid-v0.2-12B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NeverSleep/Lumimaid-v0.2-12B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "NeverSleep/Lumimaid-v0.2-12B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NeverSleep/Lumimaid-v0.2-12B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use NeverSleep/Lumimaid-v0.2-12B with Docker Model Runner:
```
docker model run hf.co/NeverSleep/Lumimaid-v0.2-12B
```

Feedback

by SerialKicked - opened Jul 27, 2024

Discussion

SerialKicked

Jul 27, 2024

•

edited Jul 27, 2024

Tried both the L3.1 (Q8) and this version (Q5KM). L3 is about what I'd expect from a L3 fine-tune. It's fine, better than 0.1 for sure.

This one, however... It's kinda weird. Vanilla Nemo is pretty much unaligned/uncensored as it is. While it's very bad at using asterisks, it's not the end of the world. This version, using the exact same scene(s) for comparison's sake, does feel like a downgrade. I can recognize the usual sentences I'd expect from the datasets, and the model itself is a lot worse at keeping track of locations, moods, and such. I'm sure it has more varied ERP vocab, but it's at the cost of a lot of coherence and context understanding. On the good side, it's less prone to repetitions without having to toy with the inference settings, and a tad better at using asterisks.

Also, it's no longer following (OOC: something) prompts like the base model used to do.

It really feels like it was over-fitted (and the fact it needs an even lower temp than the base model to be coherent seems to point in this direction).

Undi95

NeverSleep org Jul 27, 2024

Hello, thanks for your feedback.

It really feels like it was over-fitted (and the fact it needs an even lower temp than the base model to be coherent seems to point in this direction).

We released the epoch 2.5 from the model, here is the epoch 2 : https://huggingface.co/NeverSleepHistorical/lumi-nemo-e2.0
I made it public for the people who have somewhat the same issue than you.

Maybe a little less could be enough?
I will check that anyway, thank you!