pearsonkyle/Qwen3.8-27B-GPTQ-W4A16

⚡ Qwen/Qwen3.8-27B · W4A16 · vLLM

GPTQ @ 32K calibration

📦 18.1 GiB · 2.8× smaller compressed-tensors · int4 g128 sym 🧬 calibrated at ctx 32,768 ⚡ MTP bundled · 1.71× decode 👁️ Text + Image · vision 3/3 🏅 tool-calling = bf16, turn for turn 🏗️ vLLM 0.27.1

⚡ What this is

A compressed-tensors W4A16 (GPTQ, int4 group-128 symmetric) quantization of Qwen/Qwen3.8-27B that vLLM serves directly — no conversion, no custom runtime, quantization auto-detected from config.json. Calibrated on 4.26M tokens of real agentic-coding sessions packed at 32K context. Sibling of the GGUF ladder, same corpus: this one for vLLM, the GGUF for llama.cpp / Ollama / LM Studio.

⚡ Speculative decoding + vision, both bundled

The trained MTP draft head ships inside the checkpoint — one --speculative-config flag, no second file, 1.71× decode at 71.2% acceptance. The vision tower ships too, kept at bf16: shown a synthetic test image it named the colour, shape and position of all three shapes correctly, unprompted detail included ("inverted triangle, base horizontal at the top"). Both are additive — drop the flags and you are back to the identical text model.

📐 Read the size honestly

90.5% of parameters are int4, but the file is 5.2 bits per weight overall, not ~4.1. Qwen3.8's 248,320-token vocabulary with untied embeddings makes embed_tokens and lm_head 1.27 B parameters each, and both are deliberately kept at bf16 — 4.74 GiB, 26% of the download. That is why this is larger than the 14.5 GiB IQ4_XS GGUF despite quantizing the same trunk to the same width. The quantized trunk itself is 4.05 bpw. A quantized output head over a 248k vocab is the classic rare-token failure mode; the needle test below is the check that keeping it bf16 worked.

🎯 bf16 parity98/174 tool-call turns — identical to its own bf16 reference on the same stack.

🧠 Hybrid-attention aware48 of 64 layers are linear attention. The sequential GPTQ pipeline walked all 64 and quantized 496 modules — verified, not assumed.

🛠️ Standard formatPlain compressed-tensors — the same format Red Hat / Neural Magic publish. No patches or forks.

🚀 Quick start

vllm serve pearsonkyle/Qwen3.8-27B-GPTQ-W4A16 \
    --max-model-len 32768 \
    --enable-auto-tool-choice --tool-call-parser qwen3_xml

--tool-call-parser qwen3_xml is required for tool calling. Qwen3.8 emits XML tool calls (<tool_call><function=NAME>), not JSON. Without it tool_calls is always empty — which looks like a broken quant but is a serving flag.

---

Benchmarks

All on vLLM 0.27.1 (RTX PRO 6000 Blackwell) vs a bf16 reference served identically. That control is the only valid comparison — the same reference scores 0.563 here vs 0.494 on llama.cpp, so these do not compare across stacks.

Agentic — 25 held-out sessions / 174 turns, greedy, thinking off

	bf16 (ref)	W4A16
Tool-selection accuracy	0.563 (98/174)	0.563 (98/174)
Parameter accuracy	0.345	0.350
Schema-valid rate	0.948	0.948 (165/174)
Malformed tool calls	0	0

SWE-rebench (dask__dask-11393, end-to-end through the OpenAI Agents SDK, thinking off): ✅ resolved · 1/1 FAIL_TO_PASS · 34/34 PASS_TO_PASS · 5 steps, 622 output tokens · 0 malformed commands.

Long-context retrieval: 3/3 rare codes recovered exactly at ~29.7k tokens (PINEAPPLE-7742, ZEPHYR-3391-QX, MARMALADE-8156).

Vision: 3/3 on colour + shape + position for a synthetic three-shape image.

Static fidelity: median KLD vs bf16 across six held-out distributions ranges 0.0021 (agentic) to 0.0205 (general English) at eval-ctx 8192 — lowest exactly where the model was calibrated.

Per-distribution KLD table

eval	what it is	median KLD	top-1 agree	top-5 agree	ppl (bf16 → W4A16)
external	code + math + tools, disjoint from calibration — the headline	0.0141	87.8%	97.4%	15.186 → 16.094
general	broad English (combined_en_tiny)	0.0205	86.1%	97.2%	58.994 → 61.683
tools	held-out CLI + agent log sessions	0.0066	84.9%	93.5%	27.443 → 29.189
agentic	held-out SWE trajectories	0.0021	83.8%	92.1%	46.012 → 43.185
broad	held-out broad-instruct	0.0143	91.1%	99.4%	182.310 → 179.041
cal8k	slice of the previous 8192-packed corpus — a fit probe, not a holdout	0.0141	82.8%	93.2%	33.917 → 33.259

KLD is 3–10× lower on agentic/tool text than on general English — calibration spending fidelity where the model is used. Worst case for general chat: general, 0.0205.

Perplexity improves on three of six — that is noise, not a win. Trust KLD and top-1.

---

Reasoning level

Set per-request, no restart:

extra_body={"chat_template_kwargs": {"reasoning_effort": "medium"}}

Measured over 174 tool-call turns at max_tokens=512, tool-selection accuracy ranges 0.437 (xhigh) to 0.563 (off) — a 22-turn spread. Only the endpoints are reliable: replaying the ladder moved levels by up to 6 turns, and high/medium/low do not order consistently.

Per-level results

Reasoning level	What it injects	Tool-sel acc	Turns	Param acc	Schema
xhigh (default)	think carefully + validate assumptions + consider alternatives	0.437	76/174	0.226	0.862
high	think carefully + validate assumptions	0.483	84/174	0.258	0.845
medium	no instruction — native reasoning	0.511	89/174	0.273	0.862
low	keep thinking brief and focused	0.489	85/174	0.292	0.856
enable_thinking:false	pre-closed <think></think>	0.563	98/174	0.350	0.948

Read the endpoints, not the ordering. Run twice on the same weights, levels moved by up to 6 turns — so only the ends are safe: off is clearly best, xhigh clearly worst (22-turn spread), and high/medium/low are one blur that does not order consistently.

Tool-calling on a tight budget → enable_thinking: false. Reasoning competes with the answer for the same max_tokens; raise the budget before the reasoning level.

Why, and the bundled chat template

At 512 tokens xhigh is truncated before it can emit a <tool_call>, which the scorer records as "no tool call" — a serving artifact, not model quality. medium injects no instruction (native reasoning).

The stock template raises on reasoning_effort: "high" — the OpenAI-standard value — so a normal OpenAI client gets HTTP 400. The bundled chat_template_safe_v2.jinja fixes that plus three rendering bugs, and is byte-identical on 382/382 real holdout prefixes, so adopting it cannot change quality:

vllm serve ... --chat-template chat_template_safe_v2.jinja

---

MTP speculative decoding

The trained MTP draft head ships inside the checkpoint at bf16 — no second file to download.

vllm serve pearsonkyle/Qwen3.8-27B-GPTQ-W4A16 \
    --max-model-len 32768 \
    --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":2}'

draft-n	decode tok/s	vs baseline	acceptance
off (baseline)	75.0	1.00×	—
1	113.6	1.51×	82.1%
2	128.3	1.71×	71.2%

Best setting: num_speculative_tokens: 2 — 1.71× decode at 71.2% acceptance. Qwen3.8 has one nextn layer, so a deeper draft re-runs that head on its own guess and per-token acceptance drops — but it still nets more accepted tokens per step, which is what throughput follows. Optimise tokens-gained-per-step, not acceptance rate.

How it was made — quantization settings, what stayed bf16, calibration corpus

GPTQ W4A16 via llm-compressor 0.13.0 → compressed-tensors 0.18.0: int4, group 128, symmetric, static act-order. Sequential pipeline over all 64 layers (48 linear-attention + 16 full-attention), 496 modules quantized.

Kept bf16 on purpose: lm_head, embed_tokens, vision tower, MTP head. The 248,320-token vocab with untied embeddings makes the two vocab tensors 4.74 GiB, 26% of the download — which is why a "4-bit" 27B lands at 5.2 bpw overall (trunk alone: 4.05) and is bigger than the 14.5 GiB IQ4_XS GGUF. A quantized head over a 248k vocab is the classic rare-token failure mode; the needle test is the check that this worked.

Calibration: 4,255,761 tokens / 3,436 windows at ctx 32,768 — 63% real agentic sessions (CLI logs + SWE trajectories), 6,786 tool calls across 76 schemas, plus reasoning turns, broad-instruct and red-team refusals. GPTQ drew 128 × 32,768-token sequences by deterministic whole-corpus stride.

Built with Quant-Tuner; logs mined with LogMiner.

---

Limits

• Images tested, video not. Multi-image, high-res grounding and vision at long context are untested.
• Vision tower and MTP head are bf16, not quantized (+1.65 GiB) — which is why they work.
• SWE-rebench is one instance, not a pass rate.
• Agentic deltas are noise-bound. Replaying the ladder twice on the same weights moved levels by up to 6 turns (3.4pp) — vLLM is nondeterministic at temperature=0. Read the endpoints, not the third decimal.

License

Apache-2.0, inherited from Qwen/Qwen3.8-27B.