README.md

---
license: apache-2.0
base_model:
  - Qwen/Qwen3.5-4B
base_model_relation: quantized
library_name: llama.cpp
pipeline_tag: text-generation
thumbnail: https://huggingface.co/TheStageAI/Qwen3.5-4B-GGUF/resolve/main/assets/thestage-edge-models-header.png
tags:
  - gguf
  - llama.cpp
  - quantized
  - mixed-precision
  - local-inference
  - qwen3.5
---

<p align="center">
  <img src="./assets/thestage-edge-models-header.png" width="100%" alt="TheStageAI Edge Models — the right model at every memory budget">
</p>

<h1 align="center">Qwen3.5 4B — TheStageAI GGUF</h1>

<p align="center">
  Four GGUF checkpoints for local inference with llama.cpp, from 1.52 GB to 4.49 GB.
  <br>
  <strong>Start with M — 2.39 GB and 98% of the BF16 instruction-strict IFEval score in our evaluation.</strong>
</p>

<p align="center">
  <a href="https://github.com/TheStageAI/edge-lm"><img src="./assets/cta-edge-lm.svg" height="42" alt="Explore edge-lm on GitHub"></a>
  &nbsp;
  <a href="https://docs.thestage.ai/"><img src="./assets/cta-docs.svg" height="42" alt="Read TheStageAI documentation"></a>
  &nbsp;
  <a href="https://app.thestage.ai/"><img src="./assets/cta-platform.svg" height="42" alt="Open TheStageAI Platform"></a>
</p>

**Qwen 3.5 family:** [0.8B](https://huggingface.co/TheStageAI/Qwen3.5-0.8B-GGUF) · [2B](https://huggingface.co/TheStageAI/Qwen3.5-2B-GGUF) · **[4B](https://huggingface.co/TheStageAI/Qwen3.5-4B-GGUF)** · [9B](https://huggingface.co/TheStageAI/Qwen3.5-9B-GGUF)

## Start here

| Tier | Size | Use | Download |
| --- | ---: | --- | --- |
| XS | 1.52 GB | Smallest | [GGUF](./Qwen3.5-4B-XS-TS-Q3_K_S.gguf) |
| S | 1.90 GB | Adaptive compact | [GGUF](./Qwen3.5-4B-S-TS-Q4_K_S.gguf) |
| **M** | **2.39 GB** | **Recommended · Fixed Q4 reference** | [GGUF](./Qwen3.5-4B-M-TS-Q4_K_M.gguf) |
| L | 4.49 GB | Q8 fidelity reference | [GGUF](./Qwen3.5-4B-L-TS-Q8_0.gguf) |

Sizes use decimal MB/GB. Exact byte counts are recorded in [`release-manifest.json`](./release-manifest.json).

## Run with llama.cpp

This command selects the recommended **M** checkpoint by exact filename:

```bash
llama-cli \
  --hf-repo TheStageAI/Qwen3.5-4B-GGUF \
  --hf-file Qwen3.5-4B-M-TS-Q4_K_M.gguf
```

## Quality

| Variant | IFEval prompt / instruction strict (%) | MMLU-Pro (%) |
| --- | ---: | ---: |
| BF16 reference | 82.44 / 87.53 | 79.55 |
| XS | 70.43 / 78.30 | — |
| S | 77.82 / 83.93 | 74.39 |
| **M** | 80.22 / 86.09 | 78.86 |
| L | 81.70 / 87.05 | 79.59 |

Only complete 12,032-question MMLU-Pro runs are reported. `—` means that no complete release score is available; partial-subject results are not promoted to headline metrics.

The benchmarks intentionally exercise different modes: IFEval measures non-thinking instruction and format adherence, while MMLU-Pro includes long sampled reasoning.

> **Reasoning:** use S, M, or L for long-form reasoning. XS prioritizes minimum size; run Qwen XS with `--reasoning off`.

<details>
<summary><b>Evaluation protocol and XS reasoning note</b></summary>

- **IFEval:** 541 prompts, native chat template, `enable_thinking=false`, temperature 0.
- **MMLU-Pro:** 12,032 questions, native chat template, `enable_thinking=true`, temperature 1, top-p 0.95, 32,768-token output limit.
- The headline BF16-retention percentage uses instruction-strict IFEval and is capped at 100%; the uncapped raw scores are shown in the table.

XS prioritizes minimum footprint and ordinary chat or instruction following. For long-form reasoning, start with S, M, or L. Run Qwen XS with `--reasoning off`. In our long-thinking evaluation, XS generated substantially longer trajectories and reached the 32,768-token output limit more often than S. The XS MMLU-Pro cell is consequently marked `—`.

</details>

## How these checkpoints were built

This release starts from [`Qwen/Qwen3.5-4B`](https://huggingface.co/Qwen/Qwen3.5-4B) at revision [`851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a`](https://huggingface.co/Qwen/Qwen3.5-4B/tree/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a).

- **XS and S** use model-specific mixed-precision schedules selected for explicit size targets.
- **M** keeps fixed Q4_K precision across quantized decoder tensors.
- **L** keeps fixed Q8_0 precision as the high-fidelity reference.

All four checkpoints receive scale tuning without changing their stored quantized codes or tensor layout. The recommended tier is selected independently for each base model from end-to-end evaluation.

<details>
<summary><b>File types, Hub selectors, and effective BPW</b></summary>

| Variant | Hub selector | GGUF file type | Whole-file BPW |
| --- | --- | --- | ---: |
| XS | `Q3_K_S` | `MOSTLY_Q2_K` | 2.889 |
| S | `Q4_K_S` | `MOSTLY_Q2_K` | 3.622 |
| M | `Q4_K_M` | `MOSTLY_Q4_K_M` | 4.538 |
| L | `Q8_0` | `MOSTLY_Q8_0` | 8.533 |

XS and S are TheStageAI mixed-precision schedules. Their public Q-type suffixes provide approximate Hub size classes for discoverability; they do not describe every tensor in the payload. M and L use fixed Q4_K and Q8_0 assignments for quantized decoder tensors.

The exact tensor-type inventory and SHA-256 digest for every file are recorded in [`release-manifest.json`](./release-manifest.json). File size does not include KV cache or runtime buffers, which grow with context length. GGUF runtimes treat multimodal projectors as separate artifacts; this release contains the language-model files listed above.

</details>

## TheStageAI edge stack

These checkpoints provide a portable llama.cpp deployment path. For native compressed models on Apple Silicon, explore [edge-lm](https://github.com/TheStageAI/edge-lm). For custom compression, compilation, and deployment workflows, use the [TheStageAI Platform](https://app.thestage.ai/) and [documentation](https://docs.thestage.ai/).

Optimizing for a specific device, latency target, or memory budget? [Talk to the TheStageAI team →](https://app.thestage.ai/contact)

## Reproducibility

The exact base revision, byte sizes, GGUF file types, whole-file BPW, tensor inventories, SHA-256 digests, held-out KL values, and evaluation IDs are recorded in [`release-manifest.json`](./release-manifest.json). Use its SHA-256 digests to verify downloaded files. Export and load gates used [llama.cpp revision `bec4772f6a2527d371557b5d2032641e5ff7619c`](https://github.com/ggml-org/llama.cpp/commit/bec4772f6a2527d371557b5d2032641e5ff7619c).

## License

The model weights are released under the upstream model's **apache-2.0** license. llama.cpp and any surrounding runtime code retain their own licenses.