---
license: apache-2.0
base_model: barozp/Qwen3.8-27B-Opus-Distill
base_model_relation: quantized
library_name: llama.cpp
pipeline_tag: image-text-to-text
tags:
- gguf
- llama.cpp
- qwen
- reasoning
- opus-distill
- vision
- mtp
- imatrix
- quantized
---

# Qwen3.8-27B-Opus-Distill-GGUF

GGUF quantizations of
**[barozp/Qwen3.8-27B-Opus-Distill](https://huggingface.co/barozp/Qwen3.8-27B-Opus-Distill)** —
a Qwen3.8-27B fine-tuned with LoRA on Claude Opus reasoning traces (merged),
with the native vision tower and native MTP head carried over untouched.

## Highlights

- **Reasoning-distilled, not just quantized.** The LoRA was trained on 14,250
  Opus chain-of-thought traces and merged into the base weights. Quantization
  only converts the weights — the reasoning gains travel with them unchanged.
- **Full multimodal.** Native vision tower ships as a separate `mmproj` file
  (~0.9 GB). Text-only users can ignore it entirely.
- **Native MTP for self-speculative decoding.** The model was released with its
  MTP head trained in — unlike grafted MTP setups, no approximation involved.
  Free speedups on compute-bound hardware.
- **imatrix-calibrated.** All quants below Q3_K_M use an importance matrix
  built from the model's own reasoning-distillation data (see [Imatrix](#imatrix)).

## Quality benchmarks (of the source safetensors model)

Measured with `lm-evaluation-harness`: **0-shot, loglikelihood (multiple-choice),
chat template OFF, QUICK mode (`--limit 500`)**. Base and distill ran with the
identical harness, so the **Δ column is the meaningful signal**.

| Task | Metric | Base | Distill | Δ |
|---|---|---:|---:|---:|
| wikitext | word perplexity ↓ | 8.434 | 8.344 | −0.09 |
| mmlu | acc | 0.849 | 0.849 | −0.001 |
| hellaswag | acc_norm | 0.742 | 0.740 | −0.002 |
| arc_challenge | acc_norm | 0.588 | 0.630 | **+0.042** |
| gpqa_diamond | acc_norm | 0.232 | 0.495 | **+0.263** |

Reading the table:

- **Reasoning improved** (ARC +4.2pt, GPQA +26pt), **knowledge stayed flat**
  (MMLU −0.001) and **language modeling stayed flat** (wikitext −0.09 ppl).
- **GPQA caveat:** measured with *thinking disabled* (loglikelihood) — the base
  scores near random (25%) because it gets no chance to deliberate. The +26pt Δ
  is a valid same-protocol comparison, but do **not** compare 0.495 to Qwen's
  published 89.2 (measured with thinking ON, different harness).
- **ARC-Challenge is saturated** for modern models; treat it as continuity with
  the Qwen3.6 release — GPQA is the stronger reasoning signal here.

## Speed (MTP self-speculative decoding)

Not yet benchmarked for this exact model. On the Qwen3.6 sibling (same MTP
mechanism, grafted there), measured with llama.cpp: **+39% tok/s full offload,
+67% partial offload** with spec-decode ON. Native MTP (this model) is trained
in and typically does at least as well. Guidance:

- Compute-bound (full offload, strong GPU) → enable `--spec-type draft-mtp`.
- Memory-bandwidth-bound (partial offload) → keep spec off.

## Available quantizations

| File | Size | Bits/w | Use case |
|---|---:|---:|---|
| `Qwen3.8-27B-Opus-Distill-BF16.gguf` | 54.7 GB | 16.0 | reference / re-quantization source |
| `Qwen3.8-27B-Opus-Distill-Q8_0.gguf` | 29.0 GB | 8.5 | near-lossless |
| `Qwen3.8-27B-Opus-Distill-Q6_K.gguf` | 22.4 GB | 6.6 | high quality |
| `Qwen3.8-27B-Opus-Distill-Q5_K_M.gguf` | 19.5 GB | 5.7 | quality / balanced |
| `Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf` | 16.8 GB | 4.9 | **recommended all-rounder** |
| `Qwen3.8-27B-Opus-Distill-Q3_K_M.gguf` | 13.5 GB | 4.0 | tight VRAM |
| `Qwen3.8-27B-Opus-Distill-IQ3_XXS.gguf` | 11.4 GB | 3.3 | low-bit, imatrix |
| `Qwen3.8-27B-Opus-Distill-IQ2_XXS.gguf` | 8.7 GB | 2.5 | very low-bit, imatrix |
| `Qwen3.8-27B-Opus-Distill-IQ1_M.gguf` | 7.9 GB | 2.3 | extreme low-bit, imatrix |

K-quants (`Q8_0`–`Q3_K_M`) are plain `llama-quantize` passes, no imatrix needed.
IQ-quants (`IQ3_XXS` and below) **require** an importance matrix to run at all
in current `llama.cpp` and are built from the one in this repo (see below).

**Which one to pick:**

- Best quality with headroom → **Q6_K** or **Q8_0**
- Best quality/size balance → **Q4_K_M** (default recommendation)
- 24 GB card → Q4_K_M; 16 GB card → Q3_K_M (partial offload)
- Below that → IQ quants, accept the quality hit

## Imatrix

`imatrix.dat` in this repo (512 samples from
[barozp/opus-reasoning-distill-train](https://huggingface.co/datasets/barozp/opus-reasoning-distill-train),
context 512) was used to build the IQ quants above. It applies to any GGUF with
this same architecture — including the base
[Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) — so it can be reused
for re-quantization without recomputing it:

```bash
llama-quantize --imatrix imatrix.dat model-BF16.gguf model-IQ4_XS.gguf IQ4_XS
```

**Note on `IQ1_M`:** the MTP head (`blk.64`, the `nextn.*` decoder layer) is
never exercised by a normal forward pass, so the imatrix has no data for it.
`llama-quantize` pins that block to `q4_K` instead of failing, which is why
`IQ1_M` lands at ~2.3 bits/weight (7.9 GB) rather than the ~1.8 a "pure"
IQ1_M would suggest — the MTP head alone accounts for the difference, the rest
of the model is quantized normally.

## Vision (mmproj)

The vision tower is in `Qwen3.8-27B-Opus-Distill-mmproj-f16.gguf` (~0.9 GB) in
this repo. Load it alongside any quant for image/video input:

```bash
llama-server -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf --mmproj Qwen3.8-27B-Opus-Distill-mmproj-f16.gguf
```

Text-only usage does not need mmproj and runs fine without it.

## Quick start

```bash
# build llama.cpp with CUDA, then:

# text-only chat
llama-cli -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf -no-cnv

# multimodal server
llama-server -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf --mmproj Qwen3.8-27B-Opus-Distill-mmproj-f16.gguf

# with self-speculative decoding (compute-bound hardware)
llama-cli -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf -no-cnv --spec-type draft-mtp -fa on
```

## Training details (source safetensors model)

- **Base:** Qwen/Qwen3.8-27B — dense 27B, hybrid Gated-DeltaNet / full-attention, 64 layers
- **Method:** LoRA r=64, alpha=64, dropout 0.05, merged into base weights
- **LoRA targets:** attention q/k/v/o_proj on the 16 full-attention layers; FFN
  gate/up/down_proj on all 64 layers (Gated-DeltaNet projections untouched)
- **Data:** barozp/opus-reasoning-distill-train (14,250) + -validation (750, held out)
- **Run:** 1 epoch (891 steps), lr 1e-4 cosine + 3% warmup, effective batch 16,
  MAX_SEQ 4096, bf16, ~5h52m on A100 80GB
- **Final validation loss:** 0.4647
- **Vision + MTP:** carried over byte-for-byte from the base checkpoint — never trained

## Notes

- **Thinking mode is on by default** (same as the base model). The GGUF embeds
  the chat template; how thinking is toggled depends on the llama.cpp version /
  frontend (e.g., LM Studio exposes the setting in its UI).
- **Conversion:** llama.cpp `convert_hf_to_gguf.py` from the corrected multimodal
  config (nested `text_config` + `vision_config`).
- **No chaining:** every quant was produced directly from the BF16 GGUF, so
  errors do not accumulate across the ladder.

## Source chain

[Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) (base)
→ [barozp/Qwen3.8-27B-Opus-Distill](https://huggingface.co/barozp/Qwen3.8-27B-Opus-Distill) (LoRA finetune, safetensors)
→ **this repo** (GGUF quantizations)