---
license: apache-2.0
base_model:
- Qwen/Qwen3.5-4B
base_model_relation: quantized
library_name: llama.cpp
pipeline_tag: text-generation
thumbnail: https://huggingface.co/TheStageAI/Qwen3.5-4B-GGUF/resolve/main/assets/thestage-edge-models-header.png
tags:
- gguf
- llama.cpp
- quantized
- mixed-precision
- local-inference
- qwen3.5
---
Qwen3.5 4B — TheStageAI GGUF
Four deployment tiers for local inference with llama.cpp · 1.52 GB–4.49 GB
Start with M: 2.39 GB and 98% of BF16 instruction-strict IFEval.
Qwen 3.5 family: 0.8B · 2B · 4B · 9B
## Start here
| Tier | Size | Best for | File |
| --- | ---: | --- | --- |
| XS | 1.52 GB | Minimum footprint | [Download](./Qwen3.5-4B-XS-TS-Q3_K_S.gguf) |
| S | 1.90 GB | Compact | [Download](./Qwen3.5-4B-S-TS-Q4_K_S.gguf) |
| **M** | **2.39 GB** | **Recommended · Balanced** | [Download](./Qwen3.5-4B-M-TS-Q4_K_M.gguf) |
| L | 4.49 GB | High-precision Q8 | [Download](./Qwen3.5-4B-L-TS-Q8_0.gguf) |
Exact byte counts, SHA-256 hashes, and tensor metadata: [`release-manifest.json`](./release-manifest.json).
## Run with llama.cpp
```bash
llama-cli \
--hf-repo TheStageAI/Qwen3.5-4B-GGUF \
--hf-file Qwen3.5-4B-M-TS-Q4_K_M.gguf
```
## Why M is the default
At 2.39 GB, M retains 98.4% of the BF16 instruction-strict IFEval score; on MMLU-Pro, it retains 99.1% of the BF16 score. It uses 47% less disk than L, making it the default for this release.
| Tier | IFEval strict — prompt / instruction (%) | MMLU-Pro (%) |
| --- | ---: | ---: |
| BF16 reference | 82.44 / 87.53 | 79.55 |
| XS | 70.43 / 78.30 | — |
| S | 77.82 / 83.93 | 74.39 |
| **M** | 80.22 / 86.09 | 78.86 |
| L | 81.70 / 87.05 | 79.59 |
IFEval measures deterministic non-thinking instruction following; MMLU-Pro measures sampled long-form reasoning. Only complete scores are shown; `—` means not reported.
> **XS and reasoning:** use S, M, or L for long-form reasoning. Run Qwen XS with `--reasoning off`.
Evaluation protocol
- **IFEval:** 541 prompts, native chat template, `enable_thinking=false`, temperature 0.
- **MMLU-Pro:** 12,032 questions, native chat template, `enable_thinking=true`, temperature 1, top-p 0.95, 32,768-token output limit.
- The headline BF16 comparison uses instruction-strict IFEval; the raw scores are shown in the table.
In matched long-thinking diagnostics, XS produced longer trajectories and reached the 32,768-token output limit more often than S. The XS MMLU-Pro cell is marked `—` for that reason.
## From source weights to deployment tiers
All four tiers are produced by the same production PTQ pipeline; only the precision map changes. The process moves from **native code fitting**, through **sequential reconstruction** and **budget-aware scheduling**, to a final **model-wide alignment** pass.
### 1. Fit native discrete codes
Calibration activations define a curvature objective weighted by true Fisher information for each quantized projection. [NeUQI](https://arxiv.org/abs/2505.17595) initializes every affine group's scale and minimum on that objective, so sensitive weight directions influence the grid more strongly.
With the grid fixed, a guarded cyclic coordinate-descent solver inspired by [QuantEase](https://arxiv.org/abs/2309.01885) searches the integer codes. Continuous sweeps can escape a poor initial projection; projected sweeps return to a valid discrete solution. Round-to-nearest remains a non-regression baseline, and a final K-quant refinement optimizes the stored scales and minima while keeping packed codes fixed.
### 2. Reconstruct the trajectory the model will run
Layers are processed in execution order. Every projection is calibrated against activations from the already-quantized prefix, while a dense reference path measures accumulated drift. [Quantization Error Propagation (QEP)](https://arxiv.org/abs/2504.09629) folds that drift into the next reconstruction target, allowing later layers to compensate for errors they will actually receive at inference time.
### 3. Allocate the encoded byte budget
For XS and S, each quantizable group can select among native Q2_K through Q8_0 representations. The schedule optimizer trades changes in the teacher distribution against **exact encoded byte cost**, including scale and minimum metadata. Sensitive groups keep more precision; robust groups carry more compression.
[ANNA](https://docs.thestage.ai/qlip/docs/source/anna_api.html) provides TheStageAI's automated constrained configuration search, while [RCO](https://arxiv.org/abs/2605.00649) supplies an exact-budget search route ([reference implementation](https://github.com/IST-DASLab/RCO)). For this model, [RCO](https://arxiv.org/abs/2605.00649) selected both the XS and S precision maps. M and L use fixed Q4_K_M and Q8_0 decoder qtype choices, respectively, while retaining the same reconstruction and scale-tuning stages.
Once the map is selected, PTQ is rerun from the original source weights. Every layer therefore sees the final upstream precision choices rather than a collection of independently prepared bank tensors.
### 4. Align the complete model
A short affine distillation pass freezes qtypes, packed codes, dense weights, and tensor layouts while tuning native FP16 scales and minima. The loss matches the teacher's next-token distribution—including high-probability tokens and the remaining tail mass—without changing file size or runtime layout.
Every shipping GGUF is hashed, load-tested, and evaluated on a held-out set of 3,072 sequences using next-token KL. Downstream harnesses use a deterministic HF mirror reconstructed from that exact GGUF; its source SHA-256 and evaluation IDs are recorded in [`release-manifest.json`](./release-manifest.json). The recommended tier is chosen from complete-model results, not from a local reconstruction proxy.
Technical file details
| Tier | Hub selector | GGUF file type | Whole-file BPW |
| --- | --- | --- | ---: |
| XS | `Q3_K_S` | `MOSTLY_Q2_K` | 2.889 |
| S | `Q4_K_S` | `MOSTLY_Q2_K` | 3.622 |
| M | `Q4_K_M` | `MOSTLY_Q4_K_M` | 4.538 |
| L | `Q8_0` | `MOSTLY_Q8_0` | 8.533 |
The Hub selector controls sidebar grouping and download discovery. For XS and S it approximates the whole-file size class; [`release-manifest.json`](./release-manifest.json) is authoritative for the internal tensor mix. M and L use fixed Q4_K_M and Q8_0 decoder qtype choices within the same production PTQ pipeline.
Runtime memory also includes KV cache and buffers, which grow with context length.
## TheStageAI Edge Stack
- **Portable local inference:** these GGUF files for llama.cpp-compatible runtimes.
- **Native Apple Silicon:** [edge-lm](https://github.com/TheStageAI/edge-lm) for compressed MLX models on Macs and iPhones.
- **Automated compression search:** [ANNA](https://docs.thestage.ai/qlip/docs/source/anna_api.html) for budget-constrained configuration discovery.
- **Custom deployment:** the [TheStageAI Platform](https://app.thestage.ai/) and [documentation](https://docs.thestage.ai/) for compression, compilation, and serving workflows.
**Have a device, latency, or memory target? [Talk to our team →](https://app.thestage.ai/contact)**
## Reproducibility
- **Release:** July 21, 2026.
- **Base model:** [`Qwen/Qwen3.5-4B`](https://huggingface.co/Qwen/Qwen3.5-4B) at revision [`851bf6e8`](https://huggingface.co/Qwen/Qwen3.5-4B/tree/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a).
- **Manifest:** [`release-manifest.json`](./release-manifest.json) records the exact base revision, byte sizes, GGUF file types, whole-file BPW, tensor inventories, SHA-256 digests, held-out KL values, and evaluation IDs.
- **Runtime gate:** export and load checks used [llama.cpp revision `bec4772f`](https://github.com/ggml-org/llama.cpp/commit/bec4772f6a2527d371557b5d2032641e5ff7619c).
## Citation
If you use this checkpoint, please cite the upstream base model and this release:
```bibtex
@misc{thestageai2026qwen3p54bgguf,
author = {{TheStageAI}},
title = {Qwen3.5 4B — TheStageAI GGUF Release},
year = {2026},
month = {jul},
howpublished = {Hugging Face model release},
url = {https://huggingface.co/TheStageAI/Qwen3.5-4B-GGUF},
note = {XS, S, M, and L deployment tiers},
}
```
### Methods and tools
- **Schedule selection:** [ANNA](https://docs.thestage.ai/qlip/docs/source/anna_api.html), TheStageAI's automated constrained compression configuration search.
- **Exact-budget optimization:** [RCO: Model Compression with Exact Budget Constraints via Riemannian Manifolds](https://arxiv.org/abs/2605.00649) ([code](https://github.com/IST-DASLab/RCO)).
- **Discrete PTQ:** [NeUQI: Near-Optimal Uniform Quantization Parameter Initialization for Low-Bit LLMs](https://arxiv.org/abs/2505.17595) and [QuantEase: Optimization-based Quantization for Language Models](https://arxiv.org/abs/2309.01885).
- **Sequential reconstruction:** [Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization](https://arxiv.org/abs/2504.09629).
## License
The model weights are released under the upstream model's **Apache-2.0** license. llama.cpp and other runtime software retain their own licenses.
