Not-For-All-Audiences

Instructions to use Hastagaras/Halu-8B-Llama3-Blackroot with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Hastagaras/Halu-8B-Llama3-Blackroot with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Hastagaras/Halu-8B-Llama3-Blackroot")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Hastagaras/Halu-8B-Llama3-Blackroot")
model = AutoModelForCausalLM.from_pretrained("Hastagaras/Halu-8B-Llama3-Blackroot")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Local Apps Settings

vLLM

How to use Hastagaras/Halu-8B-Llama3-Blackroot with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Hastagaras/Halu-8B-Llama3-Blackroot"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Hastagaras/Halu-8B-Llama3-Blackroot",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Hastagaras/Halu-8B-Llama3-Blackroot

SGLang

How to use Hastagaras/Halu-8B-Llama3-Blackroot with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Hastagaras/Halu-8B-Llama3-Blackroot" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Hastagaras/Halu-8B-Llama3-Blackroot",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Hastagaras/Halu-8B-Llama3-Blackroot" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Hastagaras/Halu-8B-Llama3-Blackroot",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Hastagaras/Halu-8B-Llama3-Blackroot with Docker Model Runner:
```
docker model run hf.co/Hastagaras/Halu-8B-Llama3-Blackroot
```

Wow...

by Altotas - opened Jun 2, 2024

Discussion

Altotas

Jun 2, 2024

I tried many Llama3 finetunes for RP this week, and this one is something else. Unhinged, but in a good way. Characters behave as if they have their own opinions; they argue with me sometimes, talk with each other more, and I see small notes of sarcasm in some of their spoken lines. Narration is very good and has so far never repeated itself during my 50-ish message-long RP sequence (what I mean is that many other L3 models latch on to patterns in their own descriptive text and then repeat it with minimal modification in subsequent replies). This model seems to be very horny by default, and I had to specifically instruct it to avoid sexual themes.

Hastagaras

Owner Jun 2, 2024

Thank you for your feedback! The base model itself is pretty good at following instructions, so when it merged with the story LORA from Blackroot, it seems that the model became a storytelling instruction model. However, based on the other feedback, it might not be the best at 1-on-1 RP due to its storytelling tendencies. I'm glad if you like my model!

Also, damn that is a very long message! What context size did you use? I never test the model longer than 4k context. How does the model behave for longer context?

Altotas

Jun 2, 2024

Ah, I'm opposite - always set it to 16k for all Llama 3 based models.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment