Instructions to use sambanovasystems/QwQ-0.5B-SFT-Draft with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use sambanovasystems/QwQ-0.5B-SFT-Draft with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="sambanovasystems/QwQ-0.5B-SFT-Draft")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("sambanovasystems/QwQ-0.5B-SFT-Draft")
model = AutoModelForMultimodalLM.from_pretrained("sambanovasystems/QwQ-0.5B-SFT-Draft")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use sambanovasystems/QwQ-0.5B-SFT-Draft with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "sambanovasystems/QwQ-0.5B-SFT-Draft"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sambanovasystems/QwQ-0.5B-SFT-Draft",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/sambanovasystems/QwQ-0.5B-SFT-Draft

SGLang

How to use sambanovasystems/QwQ-0.5B-SFT-Draft with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "sambanovasystems/QwQ-0.5B-SFT-Draft" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sambanovasystems/QwQ-0.5B-SFT-Draft",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "sambanovasystems/QwQ-0.5B-SFT-Draft" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sambanovasystems/QwQ-0.5B-SFT-Draft",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use sambanovasystems/QwQ-0.5B-SFT-Draft with Docker Model Runner:
```
docker model run hf.co/sambanovasystems/QwQ-0.5B-SFT-Draft
```

You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Model Details:

Base Model: Qwen/Qwen2-0.5B-Instruct
Teacher Model: Qwen/QwQ-32B-Preview
Distillation Framework: Instruction Tuning
Task Type: Conversational AI / Causal Language Modeling
Parameters: 0.5B
Special Features:
- Integrated gradient checkpointing for efficient training
- Step-by-step reasoning capabilities for better problem-solving

Training:

QwQ-0.5B-Distilled was trained using the amphora/QwQ-LongCoT-130K-2, PowerInfer/QWQ-LONGCOT-500K, and PowerInfer/LONGCOT-Refine-500K with supervised finetuning. This model can be used as a competitive reasoning model on edge devices as well as a draft model for Qwen/QwQ-32B-Preview.

Training Progress:

[▓▓▓▓▓▓▓▓▓▓] 100%

Example Usage:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
# Model name
model_name = "kz919/QwQ-0.5B-Distilled-SFT"
# Load the model
print(f"Starting to load the model {model_name} into memory")
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map={"": 0}
)
# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name)
# Define the prompt
prompt = "How many r in strawberry."
messages = [
    {"role": "system", "content": "You are a helpful and harmless assistant. You are Qwen developed by Alibaba. You should think step-by-step."},
    {"role": "user", "content": prompt}
]
# Tokenize the input
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
# Generate a response
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=4096
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
# Decode the response
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

Applications:

Conversational Assistants:
Suitable for AI chatbots that require reasoning and long-context understanding.
Educational Tools:
Provides step-by-step explanations, making it ideal for learning environments.
Creative Writing:
Assists in generating coherent, contextually aware long-form content.
Technical Support:
Handles complex customer queries with precision and clarity.

Draft model for Qwen/QwQ-32B-Preview:

This model can be used as a draft model for Qwen/QwQ-32B-Preview in sepculative decoding. We observe out of 5 tokens it generates, on average 3 tokens are accepted for math queries and 2.3 tokens are accepted for general reasoning queries.

Limitations:

While distilled for efficiency, performance on highly complex reasoning tasks may slightly trail the teacher model.
This model could still be under trained, merely a proof of concept. Don't yell at me if it's outputing nonesense.

Citation:

If you use this model in your research or applications, please cite it as:

@model{qwq_0.5B_distilled,
  author = {Kaizhao Liang},
  title = {Mini-QwQ: A Reasoning Model for Edge Devices},
  year = {2024},
  publisher = {Hugging Face},
  version = {1.0}
}

This model is an example of how efficient fine-tuning and distillation methods can deliver robust conversational AI capabilities in a smaller, more manageable footprint.

Downloads last month: 2

Safetensors

Model size

0.5B params

Tensor type

BF16

Model tree for sambanovasystems/QwQ-0.5B-SFT-Draft

Base model

Qwen/Qwen2.5-0.5B

Finetuned

Qwen/Qwen2.5-0.5B-Instruct