---
license: apache-2.0
base_model:
- Qwen/Qwen3.5-2B
base_model_relation: quantized
library_name: llama.cpp
pipeline_tag: text-generation
thumbnail: https://huggingface.co/TheStageAI/Qwen3.5-2B-GGUF/resolve/main/assets/thestage-edge-models-header.png
tags:
- gguf
- llama.cpp
- quantized
- mixed-precision
- local-inference
- qwen3.5
---
Qwen3.5 2B
Four GGUF checkpoints for llama.cpp, from 738 MB to 2.01 GB.
Start with M: 1.07 GB and ≈100% of the BF16 instruction-strict IFEval score.
## Choose a checkpoint
| Tier | Size | Best for | File |
| --- | ---: | --- | --- |
| XS | 738 MB | Minimum footprint | [Download](./Qwen3.5-2B-XS-TS-Q3_K_S.gguf) |
| S | 967 MB | Compact | [Download](./Qwen3.5-2B-S-TS-Q4_K_S.gguf) |
| **M** | **1.07 GB** | **Recommended** | [Download](./Qwen3.5-2B-M-TS-Q4_K_M.gguf) |
| L | 2.01 GB | High-precision Q8 | [Download](./Qwen3.5-2B-L-TS-Q8_0.gguf) |
**Other Qwen 3.5 sizes:** 0.8B · 4B · 9B
## Quickstart
```bash
llama-cli \
--hf-repo TheStageAI/Qwen3.5-2B-GGUF \
--hf-file Qwen3.5-2B-M-TS-Q4_K_M.gguf
```
## Why we recommend M
At 1.07 GB, M matches the BF16 instruction-strict IFEval score within evaluation variation. It uses 47% less disk than L.
| Tier | IFEval P / I (%) | MMLU-Pro (%) |
| --- | ---: | ---: |
| BF16 reference | 65.43 / 74.70 | — |
| L | 65.80 / 74.94 | — |
| **M** | 66.54 / 75.54 | — |
| S | 63.22 / 73.38 | — |
| XS | 52.68 / 64.15 | — |
P / I means prompt-strict / instruction-strict. IFEval uses deterministic non-thinking decoding; MMLU-Pro uses sampled long-form reasoning. Only complete model-level scores are reported. A dash means not reported.
> **Reasoning:** XS is intended for non-thinking use. Run it with `--reasoning off`. Choose S, M, or L for long-form reasoning.
Evaluation protocol
- **IFEval:** 541 prompts, native chat template, `enable_thinking=false`, temperature 0.
- **MMLU-Pro:** 12,032 questions, vLLM, 0-shot, native chat template `qwen_mc_json_v1`, `enable_thinking=true`; `temperature=1`, `top_p=0.95`, `top_k=20`, `min_p=0`, `presence_penalty=1.5`, `frequency_penalty=0`, `repetition_penalty=1`, `seed=42`; `max_model_len=40960`, `max_new_tokens=32768`; dataset revision `b189ec765aa7ed75c8acfea42df31fdae71f97be`.
- The headline BF16 comparison uses instruction-strict IFEval; the raw scores are shown in the table.
In matched long-form reasoning diagnostics, XS produced longer trajectories and reached the 32,768-token output limit more often than S. For Qwen 0.8B and 2B, a complete comparable MMLU-Pro matrix was not available at release time. The table omits incomplete subject runs.
## How the checkpoints are built
All four checkpoints use the same production PTQ pipeline. The precision map is the only tier-specific part.
### 1. Fit native GGUF codes
Calibration activations produce a curvature objective weighted by true Fisher information for each quantized projection. We adapt the scale and minimum initialization from [NeUQI](https://arxiv.org/abs/2505.17595) to that objective, then solve integer codes on the target GGUF grid with a guarded cyclic coordinate-descent solver inspired by [QuantEase](https://arxiv.org/abs/2309.01885). A final K-quant pass tunes stored scales and minima while keeping packed codes fixed.
### 2. Reconstruct the deployed trajectory
Layers are calibrated in execution order against activations from the already-quantized prefix. A dense reference path measures accumulated drift. [Quantization Error Propagation (QEP)](https://arxiv.org/abs/2504.09629) adds that drift to the next reconstruction target, so later layers optimize for the inputs they receive at inference time.
### 3. Allocate the byte budget
XS and S can choose Q2_K through Q8_0 for each quantizable group. The optimizer trades changes in the teacher distribution against the actual encoded byte cost, including scale and minimum metadata. [ANNA](https://docs.thestage.ai/qlip/docs/source/anna_api.html) provides constrained configuration search. [RCO](https://arxiv.org/abs/2605.00649) provides an exact-budget route ([code](https://github.com/IST-DASLab/RCO)). For this model, ANNA selected both the XS and S precision maps. M and L keep the decoder qtypes at Q4_K_M and Q8_0, respectively, and use the same reconstruction and scale-tuning stages.
After schedule selection, PTQ runs again from the source weights. Each layer is then calibrated with the final upstream precision choices.
### 4. Align the full model
A short affine distillation pass tunes native FP16 scales and minima while qtypes, packed codes, dense weights, and tensor layouts stay fixed. The loss matches the teacher's next-token distribution without changing file size or runtime layout.
We load-test the shipping GGUF and evaluate it on a held-out set of 3,072 sequences with next-token KL. [`release-manifest.json`](./release-manifest.json) records its SHA-256, downstream evaluation IDs, and tensor metadata. Final recommendations use complete-model benchmarks.
File details
| Tier | Hub selector | GGUF file type | Whole-file BPW |
| --- | --- | --- | ---: |
| XS | `Q3_K_S` | `MOSTLY_Q2_K` | 3.140 |
| S | `Q4_K_S` | `MOSTLY_Q2_K` | 4.109 |
| M | `Q4_K_M` | `MOSTLY_Q4_K_M` | 4.562 |
| L | `Q8_0` | `MOSTLY_Q8_0` | 8.558 |
The Hub selector controls sidebar grouping and download discovery. For XS and S it approximates the whole-file size class; [`release-manifest.json`](./release-manifest.json) contains the exact tensor mix. M and L keep their decoder qtypes at Q4_K_M and Q8_0 within the same production PTQ pipeline.
Runtime memory also includes KV cache and buffers, which grow with context length.
## TheStageAI deployment stack
These GGUF files target llama.cpp-compatible runtimes. [edge-lm](https://github.com/TheStageAI/edge-lm) runs compressed MLX models on Macs and iPhones. [ANNA](https://docs.thestage.ai/qlip/docs/source/anna_api.html) searches compression configurations under size or compute constraints. The [TheStageAI Platform](https://app.thestage.ai/) and [documentation](https://docs.thestage.ai/) cover compression, compilation, and serving workflows.
For a specific device, latency target, or memory budget, [talk to our team](https://app.thestage.ai/contact).
## Reproducibility
- **Release:** July 21, 2026.
- **Base model:** [`Qwen/Qwen3.5-2B`](https://huggingface.co/Qwen/Qwen3.5-2B) at revision [`15852e8c`](https://huggingface.co/Qwen/Qwen3.5-2B/tree/15852e8c16360a2fea060d615a32b45270f8a8fc).
- **Manifest:** [`release-manifest.json`](./release-manifest.json) records the exact base revision, byte sizes, GGUF file types, whole-file BPW, tensor inventories, SHA-256 digests, held-out KL values, and evaluation IDs.
- **Runtime gate:** export and load checks used [llama.cpp revision `bec4772f`](https://github.com/ggml-org/llama.cpp/commit/bec4772f6a2527d371557b5d2032641e5ff7619c).
## Citation
If you use this checkpoint, cite this release and follow the upstream model's citation guidance:
```bibtex
@misc{thestageai2026qwen3p52bgguf,
author = {{TheStageAI}},
title = {Qwen3.5 2B: TheStageAI GGUF Release},
year = {2026},
month = {jul},
howpublished = {Hugging Face model release},
url = {https://huggingface.co/TheStageAI/Qwen3.5-2B-GGUF},
note = {XS, S, M, and L deployment tiers},
}
```
References
- [ANNA](https://docs.thestage.ai/qlip/docs/source/anna_api.html), TheStageAI's constrained compression configuration search.
- [RCO: Model Compression with Exact Budget Constraints via Riemannian Manifolds](https://arxiv.org/abs/2605.00649) ([code](https://github.com/IST-DASLab/RCO)).
- [NeUQI: Near-Optimal Uniform Quantization Parameter Initialization for Low-Bit LLMs](https://arxiv.org/abs/2505.17595).
- [QuantEase: Optimization-based Quantization for Language Models](https://arxiv.org/abs/2309.01885).
- [Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization](https://arxiv.org/abs/2504.09629).
## License
The checkpoint weights use the upstream model's **Apache-2.0** license. llama.cpp and other runtime software keep their own licenses.