---
license: apache-2.0
base_model: barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill
base_model_relation: quantized
pipeline_tag: text-generation
tags:
- gguf
- llama.cpp
- quantized
- moe
- qwen3
- reasoning
---

# Qwen3.6-29B-REAP-Opus-Reasoning-Distill-GGUF

GGUF quantizations of a REAP-pruned (205/256 experts) Qwen3.6-35B-A3B MoE, merged with an Opus-reasoning-distilled LoRA adapter. This is the **plain merge — no Multi-Token Prediction (MTP) head**.

A version with an MTP head grafted on (for self-speculative decoding) is available at [barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP-GGUF](https://huggingface.co/barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP-GGUF). See the benchmarks below to decide which one fits your hardware.

Converted with `llama.cpp`'s `convert_hf_to_gguf.py`.

## Highlights

- **The reasoning LoRA beats the unpruned base model on ARC-Challenge** (0.616 vs 0.532) — a net skill gain from distillation, not just recovered pruning loss. See [Quality benchmarks](#quality-benchmarks).
- **Identical speed to the MTP release** with speculative decoding off (219.06 vs 219.34 tok/s, full offload) — no downside to picking this file unless you specifically want the MTP speedup. See [Benchmarks](#benchmarks).

## Files

| Quant | Size | Notes |
|---|---:|---|
| BF16 | 56.5 GB | full precision reference |
| Q8_0 | 30.1 GB | near-lossless |
| Q6_K | 23.2 GB | |
| Q5_K_M | 20.2 GB | |
| Q4_K_M | 17.3 GB | most popular K-quant |
| Q4_K_S | 16.2 GB | |
| IQ4_XS | 15.3 GB | imatrix-based, smaller & often better than Q4_K_S |
| Q3_K_M | 13.7 GB | |
| IQ3_M | 12.6 GB | |
| IQ3_XXS | 11.2 GB | |
| Q2_K | 10.6 GB | |
| IQ2_M | 9.6 GB | |

All quants ≤Q4_K_M were calibrated with an importance matrix (imatrix) built from 512 samples of [barozp/opus-reasoning-distill-train](https://huggingface.co/datasets/barozp/opus-reasoning-distill-train). K-quants (`Q*_K*`) favor broad compatibility and fast CPU inference; IQ-quants (`IQ*`) require the imatrix and give better quality per bit at ≤4-bit, at some CPU-inference speed cost.

## Why choose this over the MTP version?

The MTP-grafted version carries an extra decoder layer used for self-speculative decoding. When that feature is left enabled on hardware that's memory-bandwidth-constrained (e.g. a small/weaker GPU with heavy CPU offload), the draft+verify overhead can compete with an already-scarce resource and result in **slower** generation than this plain model. This release removes that footgun entirely — no toggle to remember, always the same speed as the MTP file with speculative decoding off.

## Benchmarks

Measured with `llama-cli` (`Q4_K_M`, flash attention on, greedy decoding, 5 runs per config, mean ± std) on an NVIDIA RTX PRO 6000 Blackwell Server Edition (97 GB VRAM). Full methodology and the MTP comparison are on the [MTP-GGUF model card](https://huggingface.co/barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP-GGUF#benchmarks).

| `-ngl` | tok/s (generation) |
|---:|---:|
| 99 (full offload) | 219.06 ± 0.10 |
| 20 (partial offload) | 58.60 ± 0.52 |

This matches the MTP-GGUF file with speculative decoding disabled (`--spec-type none`) within measurement noise — confirming the MTP head, when unused, carries no VRAM/compute penalty. If your setup benefits from speculative decoding (compute-bound, strong GPU, full offload), the MTP release may be faster; see its model card for numbers (+39–67% observed in our tests).

## Quality benchmarks

Measured with `lm-evaluation-harness` (HF backend, bfloat16, chat template **disabled** — see note) on the underlying safetensors checkpoint, against the same base chain: unpruned `Qwen3.6-35B-A3B` → REAP 205/256 pruning only (no LoRA, [RangerX/Qwen3.6-35B-REAP-Pruned-ratio-0.2](https://huggingface.co/RangerX/Qwen3.6-35B-REAP-Pruned-ratio-0.2)) → this checkpoint.

| Task | Axis | Qwen3.6-35B-A3B | RangerX REAP-20% | **This model** |
|---|---|---:|---:|---:|
| wikitext perplexity (lower=better) | sanity | 7.85 | 10.06 | 10.06 |
| MMLU | out-of-calibration | 0.8409 | 0.8152 | **0.8257** |
| HellaSwag (0-shot, acc_norm) | out-of-calibration | 0.7420 | 0.7440 | 0.7340 |
| ARC-Challenge (0-shot, acc_norm) | out-of-calibration | 0.5320 | 0.5340 | **0.6160** |

**Key finding:** the reasoning LoRA doesn't just recover REAP's pruning loss on ARC-Challenge — it pushes the score *above the unpruned 256-expert base model* (0.616 vs 0.532), a genuine reasoning-skill transfer from the Opus chain-of-thought training data (ARC-Challenge appears in neither REAP's calibration mixture nor the LoRA's training data). MMLU sees a smaller but real gain (+1.3pp retained vs. the LoRA-less pruned checkpoint); wikitext perplexity is unaffected; HellaSwag is flat within measurement noise.

*Note: chat template was tested and found to badly corrupt loglikelihood-based multiple-choice scoring for this model family (MMLU dropped from 0.85 to 0.38 on the base model with it on) — all numbers above are with it off, applied consistently across every checkpoint. HellaSwag/ARC-Challenge are 0-shot, also applied consistently. Full methodology on the [safetensors model card](https://huggingface.co/barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill#quality-benchmarks).*

## Credits

- Base architecture: Qwen3.6-35B-A3B (MoE), pruned via [REAP](https://huggingface.co/papers) (Router-weighted Expert Activation Pruning) to 205/256 experts.
- Reasoning distillation: LoRA fine-tune on Opus-generated reasoning traces ([barozp/opus-reasoning-distill-train](https://huggingface.co/datasets/barozp/opus-reasoning-distill-train)).