---
license: apache-2.0
base_model: barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP
base_model_relation: quantized
pipeline_tag: text-generation
tags:
- gguf
- llama.cpp
- quantized
- moe
- qwen3
- reasoning
- mtp
- speculative-decoding
---

# Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP-GGUF

GGUF quantizations of [barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP](https://huggingface.co/barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP) — a REAP-pruned (205/256 experts) Qwen3.6-35B-A3B MoE, merged with an Opus-reasoning-distilled LoRA adapter, and grafted with a Multi-Token Prediction (MTP) head reverse-engineered from the original unpruned checkpoint.

Converted with `llama.cpp`'s `convert_hf_to_gguf.py`. The MTP head is preserved as native `nextn.*` / `blk.40.*` tensors — no manual tensor mapping was required.

A version without the MTP head is available at [barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-GGUF](https://huggingface.co/barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-GGUF), for backends/setups that don't benefit from self-speculative decoding (see benchmarks below).

## Highlights

- **+39% to +67% faster generation** via self-speculative decoding — measured with `llama-bench`/`llama-cli`, 5 runs per config, not estimated. See [Benchmarks](#benchmarks).
- **MTP costs zero measured quality** — identical scores to four decimal places vs. the non-MTP release. The speedup above is free. See [Quality benchmarks](#quality-benchmarks).
- **The reasoning LoRA beats the unpruned base model on ARC-Challenge** (0.616 vs 0.532) — a net skill gain from distillation, not just recovered pruning loss.

## Files

| Quant | Size | Notes |
|---|---:|---|
| BF16 | 57.9 GB | full precision reference |
| Q8_0 | 30.8 GB | near-lossless |
| Q6_K | 23.8 GB | |
| Q5_K_M | 20.7 GB | |
| Q4_K_M | 17.7 GB | most popular K-quant |
| Q4_K_S | 16.6 GB | |
| IQ4_XS | 15.7 GB | imatrix-based, smaller & often better than Q4_K_S |
| Q3_K_M | 14.0 GB | |
| IQ3_M | 12.9 GB | |
| IQ3_XXS | 11.5 GB | |
| Q2_K | 10.9 GB | |
| IQ2_M | 10.0 GB | |
| IQ2_XXS | 8.2 GB | |
| IQ1_M | 7.2 GB | extreme, quality loss is significant |

All quants ≤Q4_K_M were calibrated with an importance matrix (imatrix) built from 512 samples of [barozp/opus-reasoning-distill-train](https://huggingface.co/datasets/barozp/opus-reasoning-distill-train). K-quants (`Q*_K*`) favor broad compatibility and fast CPU inference; IQ-quants (`IQ*`) require the imatrix and give better quality per bit at ≤4-bit, at some CPU-inference speed cost.

## Using the MTP head (speculative decoding)

The GGUF ships the MTP decoder layer as `nextn.*` (front projection) and `blk.40.*` (the layer itself). Modern `llama.cpp` builds can use it for **self-speculative decoding** — the MTP head drafts a token, the base model verifies it in the same pass.

This is **not automatic on every build**. Enable it explicitly:

```bash
llama-cli -m Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP-Q4_K_M.gguf --spec-type draft-mtp -fa on -ngl 99 -p "..."
```

Without `--spec-type draft-mtp` (or your frontend's equivalent toggle), generation uses only the base model — behavior and speed then match the [non-MTP release](https://huggingface.co/barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-GGUF).

**Recommended draft settings:** max draft tokens `3`, min draft tokens `0`, draft probability `0`
(llama.cpp CLI: `--spec-draft-n-max 3 --spec-draft-n-min 0 --spec-draft-p-min 0`). Some frontends
ship stricter defaults (e.g. LM Studio defaults to draft probability `0.75`), which noticeably
suppresses acceptance — lowering it to `0` closes most of the gap to the measured rate below.

Measured MTP-head acceptance rate on held-out validation data: **~0.695** (69.5%).

## Benchmarks

Measured with `llama-cli` (`Q4_K_M`, flash attention on, greedy decoding, 5 runs per config, mean ± std) on an NVIDIA RTX PRO 6000 Blackwell Server Edition (97 GB VRAM).

| `-ngl` | Config | tok/s (generation) | Speedup vs. no-MTP baseline |
|---:|---|---:|---:|
| 99 (full offload) | No-MTP baseline | 219.06 ± 0.10 | — |
| 99 (full offload) | MTP, spec decoding off | 219.34 ± 0.10 | +0.1% (noise) |
| 99 (full offload) | **MTP, spec decoding on** | **304.56 ± 0.36** | **+39.0%** |
| 20 (partial offload) | No-MTP baseline | 58.60 ± 0.52 | — |
| 20 (partial offload) | MTP, spec decoding off | 56.80 ± 0.78 | −3.1% (noise) |
| 20 (partial offload) | **MTP, spec decoding on** | **97.60 ± 0.78** | **+66.6%** |

**Takeaways:**
- With speculative decoding off, the MTP file performs identically to the non-MTP file — the extra head carries no measurable VRAM/compute penalty when idle.
- With speculative decoding on, the MTP head gives a substantial, reproducible speedup (39–67% in this environment) — larger at partial GPU offload than at full offload.
- Speculative decoding's benefit depends on whether generation is compute-bound or memory-bandwidth-bound. On systems where the bottleneck is CPU↔GPU memory bandwidth (e.g. heavy CPU offload on a smaller/weaker GPU), the draft+verify overhead can compete with an already-scarce resource and speculative decoding may show little gain or even a small regression. If you observe a slowdown with spec decoding on, disable it (`--spec-type none`) or use the non-MTP release — both give identical baseline speed.

## Quality benchmarks

Measured with `lm-evaluation-harness` (HF backend, bfloat16, chat template **disabled** — see note) on the underlying safetensors checkpoint, against the same base chain: unpruned `Qwen3.6-35B-A3B` → REAP 205/256 pruning only (no LoRA, [RangerX/Qwen3.6-35B-REAP-Pruned-ratio-0.2](https://huggingface.co/RangerX/Qwen3.6-35B-REAP-Pruned-ratio-0.2)) → the [non-MTP merge](https://huggingface.co/barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill) → this checkpoint.

| Task | Axis | Qwen3.6-35B-A3B | RangerX REAP-20% | Ours, no MTP | **This model (+MTP)** |
|---|---|---:|---:|---:|---:|
| wikitext perplexity (lower=better) | sanity | 7.85 | 10.06 | 10.0635 | 10.0635 |
| MMLU | out-of-calibration | 0.8409 | 0.8152 | 0.8257 | **0.8257** |
| HellaSwag (0-shot, acc_norm) | out-of-calibration | 0.7420 | 0.7440 | 0.7340 | 0.7340 |
| ARC-Challenge (0-shot, acc_norm) | out-of-calibration | 0.5320 | 0.5340 | 0.6160 | **0.6160** |

**MTP head has zero measured effect on these scores** — this checkpoint and the non-MTP release score *identically to four decimal places* on every task, exactly as expected since `transformers` never loads the MTP tensors through the standard loading path (see note above). Use this release for the self-speculative-decoding speedup (benchmarks above), not for a quality difference — quality-wise the two releases are interchangeable.

**Separately, the reasoning LoRA** (present in both releases) pushes ARC-Challenge *above the unpruned 256-expert base model* (0.616 vs 0.532) — a genuine reasoning-skill transfer from the Opus chain-of-thought training data. MMLU sees a smaller but real gain (+1.3pp retained vs. the LoRA-less pruned checkpoint); wikitext perplexity is unaffected; HellaSwag is flat within measurement noise.

*Note: chat template was tested and found to badly corrupt loglikelihood-based multiple-choice scoring for this model family (MMLU dropped from 0.85 to 0.38 on the base model with it on) — all numbers above are with it off, applied consistently across every checkpoint. HellaSwag/ARC-Challenge are 0-shot, also applied consistently. Full methodology on the [safetensors model card](https://huggingface.co/barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP#quality-benchmarks).*

## Credits

- Base architecture: Qwen3.6-35B-A3B (MoE), pruned via [REAP](https://huggingface.co/papers) (Router-weighted Expert Activation Pruning) to 205/256 experts.
- Reasoning distillation: LoRA fine-tune on Opus-generated reasoning traces ([barozp/opus-reasoning-distill-train](https://huggingface.co/datasets/barozp/opus-reasoning-distill-train)).
- MTP head: reverse-engineered from the original unpruned checkpoint and grafted onto the pruned+merged model.