Instructions to use simaai/Qwen2.5-0.5B-Instruct-Autoround-Safetensors with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use simaai/Qwen2.5-0.5B-Instruct-Autoround-Safetensors with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="simaai/Qwen2.5-0.5B-Instruct-Autoround-Safetensors")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("simaai/Qwen2.5-0.5B-Instruct-Autoround-Safetensors")
model = AutoModelForCausalLM.from_pretrained("simaai/Qwen2.5-0.5B-Instruct-Autoround-Safetensors", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use simaai/Qwen2.5-0.5B-Instruct-Autoround-Safetensors with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "simaai/Qwen2.5-0.5B-Instruct-Autoround-Safetensors"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "simaai/Qwen2.5-0.5B-Instruct-Autoround-Safetensors",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/simaai/Qwen2.5-0.5B-Instruct-Autoround-Safetensors

SGLang

How to use simaai/Qwen2.5-0.5B-Instruct-Autoround-Safetensors with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "simaai/Qwen2.5-0.5B-Instruct-Autoround-Safetensors" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "simaai/Qwen2.5-0.5B-Instruct-Autoround-Safetensors",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "simaai/Qwen2.5-0.5B-Instruct-Autoround-Safetensors" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "simaai/Qwen2.5-0.5B-Instruct-Autoround-Safetensors",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use simaai/Qwen2.5-0.5B-Instruct-Autoround-Safetensors with Docker Model Runner:
```
docker model run hf.co/simaai/Qwen2.5-0.5B-Instruct-Autoround-Safetensors
```

Qwen2.5-0.5B-Instruct Autoround

Pre-LLiMa Hugging Face checkpoint based on Qwen/Qwen2.5-0.5B-Instruct at source revision 7ae557604adf67be50417f59c2c2f167def9a775. It remains subject to the base model's license, intended use, and limitations.

Quantization

Component	Method	Weight format	Details
Decoder Linear layers	AutoRound	symmetric INT4, G128	168 targets; `lm_head` excluded
`lm_head`	GPTQ	symmetric INT4, G128	static act-order, block size 128, dampening 0.01
Mixed-precision exceptions	—	source dtype	non-Linear parameters remain at source precision

Calibration used HuggingFaceH4/ultrachat_200k (dataset revision not captured), train_sft[:512], rendered with the source chat template. Token IDs were deterministically concatenated and packed into 512 full 1024-token spans (no shuffle); AutoRound used 200 iterations and batch size 1.

Evaluation

Full wikitext-2-raw-v1 evaluation used the wikitext lm-eval task, no example limit, batch size 1, CUDA, and the same evaluator for source and quantized checkpoints on 2026-07-19.

Checkpoint	Word perplexity	Status
`Qwen/Qwen2.5-0.5B-Instruct` source	20.260257	Full run
This UltraChat checkpoint	22.966478	Full run
Absolute degradation	2.706221	Lower is better
Relative degradation	13.357%	`100 * (quantized / source - 1)`

Finite-scale validation and a Transformers chat-generation smoke test passed.

Reproduction

This directory includes the exact quantize.py, recipe.yaml, and versions.txt.

python quantize.py --model-path /path/to/models--Qwen--Qwen2.5-0.5B-Instruct --output-dir /path/to/output

Environment

Exact Python, CUDA, Torch, Transformers, llmcompressor, AutoRound, and compressed-tensors versions are recorded in versions.txt.

Deployment

This is the pre-LLiMa quantized Hugging Face artifact. Compile it separately for the target Sima.ai platform and keep compiler output separate.

Limitations

Quantization quality can vary by language, domain, prompt format, context length, and deployment runtime; validate the intended workload independently.