sdkyuan/qwen3.8-27B-qat-q2_0-gguf

Qwen3.8-27B-QAT-Q2_0

This is a 2-bit (Q2_0) quantization-aware-trained (QAT) checkpoint of Qwen3.8-27B for llama.cpp at 8.76 GB.

Quantized by Samuel Yuan <sdkyuan@mit.edu>.

This quantization was heavily focused on preserving code and reasoning performance. As such, multimodal and raw fact recall performance are slightly degraded.

Interestingly, being able to preserve reasoning and code performance at the expense of Wikitext recall performance during quantization via QAT seems to be in line with the "cognitive core" hypothesis.

A Q1_0 version and improved Q2_0 quant are in the works.

Unlike most community quants that use Post-Training Quantization (PTQ) (i.e. Unsloth), this is a quantization-aware-trained model.

Hopefully this quant makes Qwen3.8 27b-level coding and reasoning more accessible to those with <12-16 GB cards or Macs, since every little bit saved in model size means more room for KV cache and faster decode on memory bandwidth bound devices!

Experiments

We measure Top-1 agreement (how often the next-token argmax matches the FP16 teacher's) and mean KL divergence (nats/token, lower is better) on held-out chat-formatted slices and on wikitext-2.

Model	Size	Reasoning top-1 (↑) KL (↓)	Code top-1 (↑) KL (↓)	Tool calling top-1 (↑) KL (↓)	Wikitext-2 top-1 (↑) KL (↓)
Qwen3.8-27B-QAT-Q2_0	8.76 GB	94.7% 0.029	93.3% 0.064	78.7% 0.638	78.3% 0.302
Unsloth UD-Q2_K_XL	9.83 GB	94.4% 0.032	93.1% 0.061	82.5% 1.026	87.5% 0.098
Unsloth UD-IQ2_S	8.37 GB	92.9% 0.052	91.5% 0.089	78.9% 1.271	84.5% 0.148
Unsloth UD-IQ3_XXS	10.93 GB	95.7% 0.018	94.6% 0.038	84.0% 0.929	90.0% 0.064

Slices correspond to:

• Reasoning: nvidia/Nemotron-Post-Training-Dataset-v1 (math), open-thoughts/OpenThoughts3-1.2M, nvidia/OpenMathReasoning (cot)
• Code: nvidia/OpenCodeReasoning, open-r1/codeforces-cots, nvidia/Nemotron-Post-Training-Dataset-v1 (code), bigcode/the-stack-smol
• Tool calling: Salesforce/xlam-function-calling-60k, togethercomputer/glaive-function-calling-v2-formatted, togethercomputer/CoderForge-Preview-32B-SWE-Bench-Verified-Evaluation-trajectories, nvidia/Nemotron-Post-Training-Dataset-v1 (tool_calling)
• Wikitext-2: Salesforce/wikitext (wikitext-2-raw-v1, test)

So this quantization exceeds the reasoning and code performance of SOTA PTQ quants over 1 GB larger, but does indeed suffer lower Wikitext accuracy. The comparatively lower tool calling performance may be due to our data blend, and we are looking to release a improved version with more focus on that as well for local agentic use cases.

Methods

We applied the Efficient QAT (https://arxiv.org/abs/2407.11062) recipe with some improvements to the E2E-QP step (KD from teacher instead of CE on dataset) and data blend.

We have included a mmproj file for vision. Again, note that multimodal capabilities may be slightly degraded.

Run it

./llama-cli \
  -m qwen38-27b-qat-q2_0.gguf \
  --jinja -cnv \
  --reasoning-budget -1 \
  -c 32768 -ngl 99 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 --repeat-penalty 1.0 \
  --flash-attn on \
  --spec-type ngram-mod --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 \
  --no-mmproj

or

./llama-server \
  --model qwen38-27b-qat-q2_0.gguf \
  --jinja \
  --reasoning-budget -1 \
  --ctx-size 32768 \
  --host 127.0.0.1 \
  --port 8033 \
  -ngl 99 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.0 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --flash-attn on \
  --spec-type ngram-mod --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 \
  --no-mmproj

Requires a llama.cpp build with Q2_0 support.

License

apache-2.0 inherited from the base model.

Feel free to reach out if you have any quesions about the model.