pearsonkyle/Qwen3.8-27B-imatrix-MTP-GGUF

🧊 Qwen/Qwen3.8-27B · 32K imatrix · MTP · GGUF

imatrix (hybrid) @ 32K

📦 9.7 · 12.1 · 14.5 · 18.3 GiB IQ2_M · IQ3_M · IQ4_XS · Q5_K_M 🧬 calibrated at ctx 32,768 ⚡ MTP head bundled @ Q8_0 🏅 all 4 rungs solved the SWE issue 👁️ Text + Image · mmproj 629 MB 🏗️ llama.cpp f3e1828

🧊 What this is

imatrix-calibrated quantizations of Qwen/Qwen3.8-27B spanning 3.06 – 5.76 bits per weight, calibrated on real agentic-coding logs. Every rung bundles the model's own Multi-Token-Prediction draft head at Q8_0 (blk.64) — speculative decoding with no second file to download. IQ4_XS is the recommended default (KLD 0.010, 88% top-token agreement); Q5_K_M is the closest substitute for FP16. Plain GGUF — runs in vanilla llama.cpp / Ollama / LM Studio, no custom runtime.

👁️ Vision included (text + image)

Qwen3.8 is natively multimodal, so the vision tower ships alongside as mmproj-Qwen3.8-27B-Q8_0.gguf (629 MB, 27-layer encoder at Q8_0). Pair it with any of the four rungs via --mmproj and the model can see images — screenshots, diagrams, UI states. The text weights are unchanged; vision is purely additive, so drop the flag and you are back to the identical text-only model. Verified on the 2-bit rung — the hardest case: shown a test image it reported "a red circle on the left, a blue rectangle on the right, and a green triangle at the bottom center", correct in shape, colour and position for all three.

🧬 Why 32K calibration matters

An imatrix is activation statistics — it records what the model actually does while reading the calibration text. Pack that corpus into 7,500-token windows and the model never sees a long tool-call chain while the statistics are collected, so the channels carrying late-trajectory behaviour are under-weighted. Repacking for 32,768 took calibration windows from an 11,390-token maximum to a median of 13,004 and a p90 of 31,869 — 67% of sessions now exceed the old cap entirely, so a whole agentic trajectory fits in one calibration context instead of being cut mid-chain.

📉 2.8–5.2× smaller9.74–18.33 GiB on disk (incl. the bundled Q8_0 MTP head) vs 50.90 GiB FP16.

🧠 Hybrid-attention aware48 of 64 layers are linear attention. Their SSM tensors pass through with raw E[a²] — output-aware re-ranking is invalid for them.

🛠️ Standard GGUFLoads anywhere llama.cpp runs. No patches, kernels, or forks.

📊 Unified benchmark & quality table

• 📐 Static — measured against FP16 on the external holdout (code + math + tools, 90k tokens, disjoint from every calibration source) at eval-ctx 8192. KLD is median; top_p is top-token agreement with FP16.
• 🤖 Agentic — 25 held-out tool-use sessions (174 scored turns), plus one SWE-rebench issue solved end-to-end through the OpenAI Agents SDK.

Metric	FP16 (ref)	Q5_K_M	IQ4_XS	IQ3_M	IQ2_M
File	—	Q5_K_M.gguf	IQ4_XS.gguf	IQ3_M.gguf	IQ2_M.gguf
Method	—	imatrix	imatrix	imatrix	imatrix
Quality	—	⭐⭐⭐⭐	⭐⭐⭐	⭐⭐	⭐
BPW	16.000	5.763	4.549	3.816	3.062
Size (GiB)	50.90	18.33	14.47	12.14	9.74
🤖 SWE issue resolved	✅	✅	✅	✅	✅
🤖 Steps to solve	10	13	14	26	20
🤖 Malformed commands	0	0	0	0	0
🤖 Tool-selection acc	0.494	0.474	0.503	0.454	0.494
🤖 Param accuracy	0.256	0.265	0.274	0.239	0.260
📐 PPL	22.850	23.274	31.118	37.363	56.540
📐 KLD (med)	0.000	0.0036	0.0105	0.0354	0.1242
📐 same_top_p	100.0%	91.8%	88.2%	82.4%	72.0%

⚠️ The 🤖 rows are a band, not a ranking. At n≈174 turns the standard error is ±3.8pp, and the whole spread — FP16 included — is 0.454–0.503. That is a 4.9pp range inside one standard error. It supports "every rung, down to 3.06 bpw, tool-calls indistinguishably from FP16 here" and nothing finer. IQ2_M and FP16 both score 86/174 yet disagree on 24 turns (12 each way) — the tie is a coincidence, not equivalence.

📐 KLD and top_p are the trustworthy columns, strictly monotone across all six evals. IQ2_M gives up ~28% of top-token decisions — a memory-constrained option, not a drop-in.

📌 Sampling, methodology & all six eval distributions

Sampling. Static: eval-ctx 8192. Tool-call replay: greedy (temperature=0), ctx=32768, --no-stop-on-fail so every model is scored on the identical 174 turns (the default stop-on-fail halts a weak model early and scores it on fewer, easier turns — which makes models incomparable). Agentic: temperature=0.25, top_p=0.95, max_tokens=8096, 2,048-token reasoning budget, step cap 60. Run on an RTX PRO 6000 Blackwell.

"Tool errors" are classified, not counted. A non-zero exit is not an agent error — a grep that legitimately matched nothing and a pytest correctly reporting a failure both exit non-zero. Errors are split into malformed (shell syntax error, command not found, failed cd — the intended work never ran), timeout, and nonzero (the program ran and reported a result). Under that split every rung issues zero malformed commands; the only genuine tool failure in the whole sweep is one timeout on IQ3_M.

The reasoning budget is a cost control, not a quality lever. Uncapped, IQ2_M was observed emitting a 12,566-token single completion — a repetition loop, not reasoning. An unbudgeted control sweep was run alongside: at n=1 the 2k budget shows no reliable effect on whether the issue gets solved (IQ4_XS and Q5_K_M improved, IQ3_M got worse, FP16 rose).

Six eval distributions, each with its own FP16 baseline — never concatenated:

eval	what it is	FP16 PPL	IQ2_M	IQ3_M	IQ4_XS	Q5_K_M
external	code + math + tools, disjoint from calibration — the headline	22.850	0.1242	0.0354	0.0105	0.0036
general	combined_en_tiny, broad English (30k tok)	92.797	0.2199	0.0569	0.0154	0.0062
tools	held-out CLI + agent log sessions (150k tok)	21.652	0.0498	0.0115	0.0032	0.0014
agentic	held-out SWE trajectories (99k tok)	43.473	0.0169	0.0048	0.0013	0.0006
broad	held-out broad-instruct (100k tok)	88.193	0.2691	0.0702	0.0204	0.0076
cal8k	slice of the previous 8192-packed corpus — a fit probe, not a holdout	17.400	0.0840	0.0261	0.0062	0.0025

⚠️ The four chat-templated evals (tools, agentic, broad, cal8k) are quant-vs-quant only: llama-perplexity has no --parse-special, so chat markers tokenize as ordinary BPE and absolute PPL is off-distribution. KLD and top_p remain valid — which is why PPL goes non-monotone there while KLD does not.

---

📚 What it was calibrated on

An imatrix is only as good as the text it watched. Here is exactly what this one saw.

4,255,761

tokens

3,436

windows

6,786

tool calls

32,768

packing ctx

16.9 MB

corpus on disk

source	what it is	samples	windows	tokens	share	median window
logs	real Claude Code / opencode / qwen code sessions	108 sessions	125	2,000,139	47.0%	13,004
swe-trajectories	agent runs on real GitHub issues	61 sessions	62	681,012	16.0%	9,650
reasoning	<think> turns, re-cut so reasoning lands last	65 sessions	230	638,046	15.0%	2,022
broad-supplement	broad-instruct, non-coding	2,704 convos	2,704	537,851	12.6%	182
wiki	wiki.test.raw, raw prose	10 chunks	10	297,053	7.0%	29,641
redteam-refusals	attack prompts + generic refusals	305 convos	305	101,660	2.4%	231
			3,436	4,255,761	100%

Three things worth reading off that table:

• Nearly two-thirds is agentic (logs + swe-trajectories = 63%), and those are the long windows — median 13,004 and 9,650 tokens against a 182-token median for broad-instruct. 84 of the 125 log windows exceed 7,500 tokens and 50 exceed 16,384, which is only possible because packing is at ctx 32,768. Nothing was truncated: 0 sessions hit the 32,076-token cap.
• Tool-calling is dense, not decorative — 6,786 <tool_call> blocks and 5,827 <tool_response> blocks across 76 distinct tool schemas, and in the log windows 99.3% of tokens sit in tool-bearing turns.
• reasoning overlaps logs by design. It re-renders the same conversations with a reasoning turn placed last, because chat templates keep <think> only on a render's final assistant turn and scrub it from history. Without those extra windows the corpus would contain almost no reasoning at all.

The red-team slice ships attack prompts with generic refusals substituted — the targets' original harmful completions never enter any corpus. Refusal behaviour is what low-bit quantization erodes first, so the attack distribution belongs in calibration; the responses do not.

---

🔬 How they were made

• Hybrid imatrix at ctx 32,768 — activation energy E[a²] blended with weight-column energy ‖W[:,c]‖²·E[a²] per tensor, collected over the corpus above in 129 chunks of a full 32,768-token context (3h54m on an RTX PRO 6000 Blackwell). 496 tensors received statistics.
• SSM passthrough — Qwen3.8 is hybrid (48 linear-attention + 16 full-attention layers). 144 SSM tensors pass through with raw E[a²]; output-aware re-ranking is mathematically invalid for state-space tensors.
• Bundled MTP draft head — the trained MTP head (mtp_num_hidden_layers=1) is remapped to blk.64 and pinned to Q8_0 in every rung: 8 × Q8_0 + 7 × F32, zero low-bit tensors, verified per file. This is checked rather than assumed because llama-quantize silently accepts a --tensor-type pattern that matches nothing — a stale pin would quantize the draft head with the trunk and surface only as poor acceptance. The head gets no imatrix statistics; it sits outside the forward pass, which is why it is pinned rather than calibrated.
• The tool-call format trap — Qwen3.8's chat template emits tool calls as XML (<tool_call><function=NAME><parameter=KEY>), not the JSON form earlier Qwen3.x templates used. Calibrating on the JSON form would have collected statistics on a syntax this model never produces.
• output.weight is calibrated — llama-imatrix only collects tensors named blk.* unless you pass --process-output, so the largest quantized tensor (and the one that most directly shapes the distribution KLD measures) is otherwise quantized blind. llama-quantize prints did not find weights for output.weight and proceeds anyway, so the only symptom is a line in a log.
• Disjoint splits — calibration (train) and every eval holdout are disjoint by construction, so the static table measures generalization, not fit. The one exception is cal8k, explicitly labelled a fit probe.
• Toolchain: Quant-Tuner with vendored llama.cpp @ f3e1828. Calibration logs mined with LogMiner.

🔁 Reproduce

git clone https://github.com/pearsonkyle/Quant-Tuner && cd Quant-Tuner
git submodule update --init --recursive
cmake -S vendor/llama.cpp -B vendor/llama.cpp/build -DGGML_CUDA=ON \
      -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120 -DLLAMA_CURL=ON
cmake --build vendor/llama.cpp/build -j 64
uv sync --extra dev

PYTHONPATH=src .venv/bin/python scripts/exp060_repack_cal_32k.py --ctx 32768 \
    --wiki out/exp-001/wiki/wiki.test.raw
PYTHONPATH=src .venv/bin/python scripts/exp060_quants_qwen38.py \
    --run exp-060-32k --ctx 32768 --eval-ctx 8192 \
    --evals external general tools agentic broad cal8k

⚠️ llama-imatrix needs --no-ppl above ~17k ctx on this model. Its perplexity path computes all_logits + first*n_vocab with first = n_ctx/2 in int arithmetic (tools/imatrix/imatrix.cpp:911). With Qwen3.8's 248,320-token vocab that product overflows INT_MAX for any n_ctx > 2³²/248320 ≈ 17,296, and the process segfaults after the first pass. --no-ppl skips only the perplexity bookkeeping — the forward pass, and therefore every activation statistic, is unchanged.

---

🚀 Usage

Ollama

ollama run hf.co/pearsonkyle/Qwen3.8-27B-imatrix-MTP-GGUF:IQ4_XS
# also: :Q5_K_M · :IQ3_M · :IQ2_M

llama.cpp server, with MTP speculative decoding

./llama-server \
    --model Qwen3.8-27B-IQ4_XS.gguf \
    --ctx-size 32768 --n-gpu-layers 999 \
    --spec-type draft-mtp --spec-draft-n-max 1 \
    --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 \
    --host 0.0.0.0 --port 1234

The draft head is inside the GGUF — no --model-draft, nothing extra to fetch. Qwen3.8 exposes one nextn layer, so --spec-draft-n-max 1 is the right setting. Drop both --spec-* flags to run without speculative decoding.

🖼️ Vision (text + image)

The vision tower ships separately as mmproj-Qwen3.8-27B-Q8_0.gguf (629 MB) so you only download it if you need images. It pairs with any of the four rungs — the text weights are identical; the mmproj just adds the encoder + projector.

# one-shot from the CLI
./llama-mtmd-cli \
    -m Qwen3.8-27B-IQ4_XS.gguf \
    --mmproj mmproj-Qwen3.8-27B-Q8_0.gguf \
    --image screenshot.png \
    --jinja -ngl 999 --temp 0.2 -n 512 \
    -p "Describe this image."

# or serve it — /v1/chat/completions then accepts image_url content parts
./llama-server \
    -m Qwen3.8-27B-IQ4_XS.gguf \
    --mmproj mmproj-Qwen3.8-27B-Q8_0.gguf \
    --jinja --ctx-size 32768 --n-gpu-layers 999 \
    --host 0.0.0.0 --port 1234

--jinja is required — the chat template is Jinja-based and the multimodal path needs it. For grounding tasks (pointing at or locating things in an image) llama.cpp recommends --image-min-tokens 1024; the default tokenization is fine for description and Q&A.

OpenAI-compatible API

import json, urllib.request

def ask(content, max_tokens=512):
    body = {"messages": [{"role": "user", "content": content}], "max_tokens": max_tokens}
    req = urllib.request.Request("http://127.0.0.1:1234/v1/chat/completions",
                                 json.dumps(body).encode(),
                                 {"Content-Type": "application/json"})
    return json.loads(urllib.request.urlopen(req).read())["choices"][0]["message"]["content"]

print(ask("Write a Python function that reverses a linked list."))

---

🪪 License & attribution

• Inherits the license of the base model Qwen/Qwen3.8-27B (Apache-2.0).
• Calibration + quantization: Quant-Tuner with vendored llama.cpp @ f3e1828.
• Calibration logs mined with LogMiner.