Qwen3.8-27B-Opus-Distill-GGUF

GGUF quantizations of barozp/Qwen3.8-27B-Opus-Distill — a Qwen3.8-27B fine-tuned with LoRA on Claude Opus reasoning traces (merged), with the native vision tower and native MTP head carried over untouched.

Highlights

• Reasoning-distilled, not just quantized. The LoRA was trained on 14,250 Opus chain-of-thought traces and merged into the base weights. Quantization only converts the weights — the reasoning gains travel with them unchanged.
• Full multimodal. Native vision tower ships as a separate mmproj file (~0.9 GB). Text-only users can ignore it entirely.
• Native MTP for self-speculative decoding. The model was released with its MTP head trained in — unlike grafted MTP setups, no approximation involved. Free speedups on compute-bound hardware.

Quality benchmarks (of the source safetensors model)

Measured with lm-evaluation-harness: 0-shot, loglikelihood (multiple-choice), chat template OFF, QUICK mode (--limit 500). Base and distill ran with the identical harness, so the Δ column is the meaningful signal.

Task	Metric	Base	Distill	Δ
wikitext	word perplexity ↓	8.434	8.344	−0.09
mmlu	acc	0.849	0.849	−0.001
hellaswag	acc_norm	0.742	0.740	−0.002
arc_challenge	acc_norm	0.588	0.630	+0.042
gpqa_diamond	acc_norm	0.232	0.495	+0.263

Reading the table:

• Reasoning improved (ARC +4.2pt, GPQA +26pt), knowledge stayed flat (MMLU −0.001) and language modeling stayed flat (wikitext −0.09 ppl).
• GPQA caveat: measured with thinking disabled (loglikelihood) — the base scores near random (25%) because it gets no chance to deliberate. The +26pt Δ is a valid same-protocol comparison, but do not compare 0.495 to Qwen's published 89.2 (measured with thinking ON, different harness).
• ARC-Challenge is saturated for modern models; treat it as continuity with the Qwen3.6 release — GPQA is the stronger reasoning signal here.

Speed (MTP self-speculative decoding)

Not yet benchmarked for this exact model. On the Qwen3.6 sibling (same MTP mechanism, grafted there), measured with llama.cpp: +39% tok/s full offload, +67% partial offload with spec-decode ON. Native MTP (this model) is trained in and typically does at least as well. Guidance:

• Compute-bound (full offload, strong GPU) → enable --spec-type draft-mtp.
• Memory-bandwidth-bound (partial offload) → keep spec off.

Available quantizations

| File | Size (approx.) | Bits/w | Use case | |---|---|---:|---:|---| | Qwen3.8-27B-Opus-Distill-BF16.gguf | ~55.6 GB | 16.0 | reference / re-quantization source | | Qwen3.8-27B-Opus-Distill-Q8_0.gguf | ~29.0 GB | 8.5 | near-lossless | | Qwen3.8-27B-Opus-Distill-Q6_K.gguf | ~22.9 GB | 6.6 | high quality | | Qwen3.8-27B-Opus-Distill-Q5_K_M.gguf | ~19.8 GB | 5.7 | quality / balanced | | Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf | ~17.1 GB | 4.8 | recommended all-rounder | | Qwen3.8-27B-Opus-Distill-Q3_K_M.gguf | ~13.8 GB | 3.9 | tight VRAM | | Qwen3.8-27B-Opus-Distill-IQ3_XXS.gguf | ~11.9 GB | 3.4 | low-bit | | Qwen3.8-27B-Opus-Distill-IQ2_XXS.gguf | ~9.0 GB | 2.5 | very low-bit | | Qwen3.8-27B-Opus-Distill-IQ1_M.gguf | ~6.0 GB | 1.8 | extreme low-bit |

⚠️ IQ quants (IQ3_XXS and below) were produced without imatrix calibration in this first release. Their quality will be noticeably below the K-quants. Use them only when VRAM is the hard constraint. A future release may redo them with imatrix.

Which one to pick:

• Best quality with headroom → Q6_K or Q8_0
• Best quality/size balance → Q4_K_M (default recommendation)
• 24 GB card → Q4_K_M; 16 GB card → Q3_K_M (partial offload)
• Below that → IQ quants, accept the quality hit

Vision (mmproj)

The vision tower is in mmproj-BF16.gguf (~0.9 GB) in this repo. Load it for image/video input:

llama-server -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf --mmproj mmproj-BF16.gguf

Text-only usage does not need mmproj and runs fine without it.

Quick start

# build llama.cpp with CUDA, then:

# text-only chat
llama-cli -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf -no-cnv

# multimodal server
llama-server -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf --mmproj mmproj-BF16.gguf

# with self-speculative decoding (compute-bound hardware)
llama-cli -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf -no-cnv --spec-type draft-mtp -fa on

Training details (source safetensors model)

• Base: Qwen/Qwen3.8-27B — dense 27B, hybrid Gated-DeltaNet / full-attention, 64 layers
• Method: LoRA r=64, alpha=64, dropout 0.05, merged into base weights
• LoRA targets: attention q/k/v/o_proj on the 16 full-attention layers; FFN gate/up/down_proj on all 64 layers (Gated-DeltaNet projections untouched)
• Data: barozp/opus-reasoning-distill-train (14,250) + -validation (750, held out)
• Run: 1 epoch (891 steps), lr 1e-4 cosine + 3% warmup, effective batch 16, MAX_SEQ 4096, bf16, ~5h52m on A100 80GB
• Final validation loss: 0.4647
• Vision + MTP: carried over byte-for-byte from the base checkpoint — never trained

Notes

• Thinking mode is on by default (same as the base model). The GGUF embeds the chat template; how thinking is toggled depends on the llama.cpp version / frontend (e.g., LM Studio exposes the setting in its UI).
• Conversion: llama.cpp convert_hf_to_gguf.py from the corrected multimodal config (nested text_config + vision_config).
• No chaining: every quant was produced directly from the BF16 GGUF with llama-quantize, so errors do not accumulate across the ladder.

Source chain

Qwen/Qwen3.8-27B (base) → barozp/Qwen3.8-27B-Opus-Distill (LoRA finetune, safetensors) → this repo (GGUF quantizations)