---
base_model: Qwen/Qwen2.5-0.5B-Instruct
language:
- multilingual
library_name: transformers
license: apache-2.0
pipeline_tag: text-generation
tags:
- quantized
- autoround
- gptq
- compressed-tensors
- sima-ai
---

# Qwen2.5-0.5B-Instruct Autoround

Pre-LLiMa Hugging Face checkpoint based on [`Qwen/Qwen2.5-0.5B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) at source revision `7ae557604adf67be50417f59c2c2f167def9a775`. It remains subject to the base model's license, intended use, and limitations.

## Quantization

| Component | Method | Weight format | Details |
| --- | --- | --- | --- |
| Decoder Linear layers | AutoRound | symmetric INT4, G128 | 168 targets; `lm_head` excluded |
| `lm_head` | GPTQ | symmetric INT4, G128 | static act-order, block size 128, dampening 0.01 |
| Mixed-precision exceptions | — | source dtype | non-Linear parameters remain at source precision |

Calibration used `HuggingFaceH4/ultrachat_200k` (dataset revision not captured), `train_sft[:512]`, rendered with the source chat template. Token IDs were deterministically concatenated and packed into 512 full 1024-token spans (no shuffle); AutoRound used 200 iterations and batch size 1.

## Evaluation

Full `wikitext-2-raw-v1` evaluation used the `wikitext` lm-eval task, no example limit, batch size 1, CUDA, and the same evaluator for source and quantized checkpoints on 2026-07-19.

| Checkpoint | Word perplexity | Status |
| --- | ---: | --- |
| `Qwen/Qwen2.5-0.5B-Instruct` source | 20.260257 | Full run |
| This UltraChat checkpoint | 22.966478 | Full run |
| Absolute degradation | 2.706221 | Lower is better |
| Relative degradation | 13.357% | `100 * (quantized / source - 1)` |

Finite-scale validation and a Transformers chat-generation smoke test passed.

## Reproduction

This directory includes the exact `quantize.py`, `recipe.yaml`, and `versions.txt`.

```bash
python quantize.py --model-path /path/to/models--Qwen--Qwen2.5-0.5B-Instruct --output-dir /path/to/output
```

## Environment

Exact Python, CUDA, Torch, Transformers, llmcompressor, AutoRound, and compressed-tensors versions are recorded in `versions.txt`.

## Deployment

This is the pre-LLiMa quantized Hugging Face artifact. Compile it separately for the target Sima.ai platform and keep compiler output separate.

## Limitations

Quantization quality can vary by language, domain, prompt format, context length, and deployment runtime; validate the intended workload independently.