Instructions to use abacusai/Smaug-34B-v0.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use abacusai/Smaug-34B-v0.1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="abacusai/Smaug-34B-v0.1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("abacusai/Smaug-34B-v0.1")
model = AutoModelForCausalLM.from_pretrained("abacusai/Smaug-34B-v0.1", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use abacusai/Smaug-34B-v0.1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "abacusai/Smaug-34B-v0.1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "abacusai/Smaug-34B-v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/abacusai/Smaug-34B-v0.1

SGLang

How to use abacusai/Smaug-34B-v0.1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "abacusai/Smaug-34B-v0.1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "abacusai/Smaug-34B-v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "abacusai/Smaug-34B-v0.1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "abacusai/Smaug-34B-v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use abacusai/Smaug-34B-v0.1 with Docker Model Runner:
```
docker model run hf.co/abacusai/Smaug-34B-v0.1
```

Some benchs

by Nexesenex - opened Jan 29, 2024

Discussion

Nexesenex

Jan 29, 2024

•

edited Jan 29, 2024

These benchs are made with Llama CPP :

abacusai_Smaug-Yi-34B-v0.1-b1989-iMat-c32_ch3250-Q4_K_M.gguf,-,Hellaswag,86,,400,2024-01-28 01:40:00,,34b,Yi,200000,,,GGUF,AbacusAI,Nexesenex,
abacusai_Smaug-Yi-34B-v0.1-b1989-iMat-c32_ch3250-Q4_K_M.gguf,-,Hellaswag_Bin,82,,400,2024-01-28 01:40:00,,34b,Yi,200000,,,GGUF,AbacusAI,Nexesenex,
abacusai_Smaug-Yi-34B-v0.1-b1989-iMat-c32_ch3250-Q4_K_M.gguf,-,Arc-Challenge,61.53846154,,299,2024-01-28 05:40:00,,34b,Yi,200000,,,GGUF,AbacusAI,Nexesenex,
abacusai_Smaug-Yi-34B-v0.1-b1989-iMat-c32_ch3250-Q4_K_M.gguf,-,Arc-Easy,82.98245614,,570,2024-01-28 05:40:00,,34b,Yi,200000,,,GGUF,AbacusAI,Nexesenex,
abacusai_Smaug-Yi-34B-v0.1-b1989-iMat-c32_ch3250-Q4_K_M.gguf,-,MMLU,42.49201278,,313,2024-01-28 05:40:00,,34b,Yi,200000,,,GGUF,AbacusAI,Nexesenex,
abacusai_Smaug-Yi-34B-v0.1-b1989-iMat-c32_ch3250-Q4_K_M.gguf,-,Thruthful-QA,44.06364749,,817,2024-01-28 05:40:00,,34b,Yi,200000,,,GGUF,AbacusAI,Nexesenex,
abacusai_Smaug-Yi-34B-v0.1-b1989-iMat-c32_ch3250-Q4_K_M.gguf,-,Winogrande,79.9526,,1267,2024-01-28 05:40:00,,34b,Yi,200000,,,GGUF,AbacusAI,Nexesenex,
abacusai_Smaug-Yi-34B-v0.1-b1989-iMat-c32_ch3250-Q4_K_M.gguf,-,wikitext,5.2535,512,512,2024-01-28 01:40:00,,34b,Yi,200000,,,GGUF,AbacusAI,Nexesenex,
abacusai_Smaug-Yi-34B-v0.1-b1989-iMat-c32_ch3250-Q4_K_M.gguf,-,wikitext,6.6019,4096,4096,2024-01-28 01:40:00,,34b,Yi,200000,,,GGUF,AbacusAI,Nexesenex,
abacusai_Smaug-Yi-34B-v0.1-b1989-iMat-c32_ch3250-Q4_K_M.gguf,-,wikitext,7.7781,8192,8192,2024-01-28 01:40:00,,34b,Yi,200000,,,GGUF,AbacusAI,Nexesenex,
abacusai_Smaug-Yi-34B-v0.1-b1989-iMat-c32_ch3250-Q4_K_M.gguf,-,wikitext,6.8504,12288,12288,2024-01-28 01:40:00,,34b,Yi,200000,,,GGUF,AbacusAI,Nexesenex,

Smaug's benchs are very solid, but il is on the fence with its perplexity (due to Bagel's own troubles, I presume?), which raises quite high (instead of aiming at 4.5 as Yi-34b models usually do at 4k and beyond), even if it stabilizes and even decreases between 8k and 12k. It's worth more tests for sure.
In any case, thanks for sharing this!

brucethemoose

Jan 29, 2024

due to Bagel's own troubles, I presume?

For me, Bagel struggles at high context, which might be showing up in the perplexity evals? And not just the DPO either, even the SFT version seems to drop off.

Perplexity aside, the evals aren't going to show this.

Nexesenex

Jan 29, 2024

•

edited Jan 29, 2024

I noticed also the trouble with Bagel (both DPO 34b & non DPO 34b, i didn't try the nontoxic) in use, and it comfirmed totally with the perplexity rocketing to 10+ at 4096 ctx or even before actually.

Bagel-34b-v0.2-4.65bpw-h6-exl2,-,wikitext,6.386557579040527,512,512,2024-01-02 20:27:51,,34b,Yi,200000,07:55,1.40,Exl2-2,JonDurbin,LoneStriker,
Bagel-34b-v0.2-4.65bpw-h6-exl2,-,wikitext,5.4337968826293945,512,1024,2024-01-02 20:43:57,,34b,Yi,200000,13:19,0.83,Exl2-2,JonDurbin,LoneStriker,
Bagel-34b-v0.2-4.65bpw-h6-exl2,-,wikitext,9.19597339630127,512,2048,2024-01-02 21:09:28,,34b,Yi,200000,24:36,0.45,Exl2-2,JonDurbin,LoneStriker,

bagel-dpo-34b-v0.2-Q3_K_M.gguf,-,Arc-Challenge,57.19063545,,299,2024-01-26 05:40:00,,34b,Yi,200000,,,GGUF,JonDurbin,TheBloke,
bagel-dpo-34b-v0.2-Q3_K_M.gguf,-,Arc-Easy,71.05263158,,570,2024-01-26 05:40:00,,34b,Yi,200000,,,GGUF,JonDurbin,TheBloke,
bagel-dpo-34b-v0.2-Q3_K_M.gguf,-,Hellaswag,76.25,,400,2024-01-26 01:40:00,,34b,Yi,200000,,,GGUF,JonDurbin,TheBloke,
bagel-dpo-34b-v0.2-Q3_K_M.gguf,-,Hellaswag_Bin,66.5,,400,2024-01-26 01:40:00,,34b,Yi,200000,,,GGUF,JonDurbin,TheBloke,
bagel-dpo-34b-v0.2-Q3_K_M.gguf,-,MMLU,38.97763578,,313,2024-01-26 05:40:00,,34b,Yi,200000,,,GGUF,JonDurbin,TheBloke,
bagel-dpo-34b-v0.2-Q3_K_M.gguf,-,Thruthful-QA,43.94124847,,817,2024-01-26 05:40:00,,34b,Yi,200000,,,GGUF,JonDurbin,TheBloke,
bagel-dpo-34b-v0.2-Q3_K_M.gguf,-,Winogrande,77.3481,,1267,2024-01-26 05:40:00,,34b,Yi,200000,,,GGUF,JonDurbin,TheBloke,
bagel-dpo-34b-v0.2-Q3_K_M.gguf,-,wikitext,9.8356,512,512,2024-01-26 01:40:00,,34b,Yi,200000,,,GGUF,JonDurbin,TheBloke,
bagel-dpo-34b-v0.2-Q3_K_M.gguf,-,wikitext,16.3509,4096,4096,2024-01-26 01:40:00,,34b,Yi,200000,,,GGUF,JonDurbin,TheBloke,

That problem didn't show on Bagel 7b (even if Jon warned about context beyond 7168, and beyond 8192 we enter in the context sliding window).

Smaug tamed a bit Bagel's behavior (and for a finetune made over it, even on the non DPO version, that's great!), as well as many merges using Bagel due to its partial "drowning" into the resulting merge I guess.

brucethemoose

Jan 29, 2024

Also just wanna say I love all these perplexity/eval tests you are posting. I need to do more high context perplexity testing myself, figure out which 200K 34Bs "lose" their context and which don't, but it's tricky on a 24GB GPU.

Nexesenex

Jan 29, 2024

Thanks!

I'll leave you some interesting models (bench-wise) I tested on your last megamerge repo.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment