Qwen3.8-27B-GPTQ-INT4

Asymmetric INT4 weight-only (group_size=32) + GPTQ, long-sequence calibrated

A 4-bit quantization of Qwen/Qwen3.8-27B. ~2.1× smaller than BF16. Matches the BF16 base on every evaluated benchmark; all deltas are within sampling noise.

TL;DR

	value
Format	INT4, asymmetric (zero-point), group_size=32, weight-only
Method	GPTQ (Hessian error compensation) + imatrix-mse observer, actorder=static, block 128, dampening_frac: 0.01
Calibration	8 sequences × 262,144 tokens (the model's native context length), packed from Nemotron-Post-Training-v3 SFT datasets (instruction-following chat, math, science, agentic code, multilingual)
Disk size	~27 GB (vs ~55.6 GB BF16)
Accuracy vs BF16 base	No degradation beyond sampling noise on 5 benchmarks + long-context retrieval

What was quantized (and what was protected)

Component	Quantized	Protected (kept BF16)
language_model (64 hybrid-attention text layers)	block linears — attention self_attn.{q_proj,k_proj,v_proj,o_proj} and FFN mlp.{gate_proj,up_proj,down_proj} (INT4, asymmetric, group_size=32)	layer norms, embed_tokens, norm, lm_head
Linear-attention (DeltaNet) projections	—	all .*linear_attn.* modules
visual (frozen ViT encoder)	—	entire model.visual.*
MTP predictor	—	kept intact for speculative decoding

Quantization recipe

GPTQ (Hessian error compensation) with weight-only INT4: asymmetric (zero-point, stored as int8), group_size=32, imatrix-mse weight observer, actorder=static, block size 128, dampening_frac: 0.01. Targets all Linear layers except lm_head, embeddings, DeltaNet (.*linear_attn.*), vision (model.visual.*), and the MTP predictor. The full resolved recipe — including defaults — ships as recipe.yaml in this repo.

imatrix importance statistics were collected by forward hooks over the 8 packed calibration sequences (llm-compressor imatrix-mse observer). The 8-sequence budget was chosen to calibrate at the model's native 262,144-token context within a single 80GB GPU — the token count (~2.1M) is conventional, the document count is the trade-off.

Evaluation

Identical harness and hardware for base and quantized (lm-eval-harness, hf-multimodal backend, greedy). Generative tasks use the chat template with thinking enabled; ARC-Easy is loglikelihood.

task	protocol	n	base (BF16)	this	Δ
GSM8K	thinking, generative (250-sample subset)	250	95.60	94.80	−0.8 (within noise)
GPQA Diamond	thinking, generative	198	64.65	64.65	0.0
RealWorldQA	thinking, generative	100	68.00	70.00	+2.0 (n=100, within noise)
MathVision¹	thinking, generative	100	10.00	12.00	+2.0 (n=100, within noise)
ARC-Easy	loglikelihood, full	2376	82.58 / 72.81²	82.62 / 74.58	+0.04 / +1.77

¹ Letter-multiple-choice protocol applied identically to both models; absolutes are not comparable to the official boxed-answer protocol — the Δ is the meaningful comparison. ² acc_norm < acc inverts the usual ARC-Easy ordering for both models under this harness/tokenizer; the effect is identical for base and quantized.

Long-context needle-in-a-haystack (greedy, 3 needle depths per length; both models retrieve exactly):

Context (tokens)	Base	This model
8,192	3/3	3/3
32,768	3/3	3/3
131,072	3/3	3/3
262,144 (native)	3/3	3/3

Retrieval validated at the model's native 262,144-token context — the same length the model was calibrated at.

Not evaluated: agentic/tool-calling behavior; vision coverage beyond the two small-n tasks above.

Serving

vllm serve abhishekchohan/Qwen3.8-27B-GPTQ-INT4

Full-featured (thinking, tool calling, MTP speculative decoding):

vllm serve abhishekchohan/Qwen3.8-27B-GPTQ-INT4 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Requires a recent vLLM with qwen3_5 support (per the base model card, use the latest version). The checkpoint is compressed-tensors pack-quantized int4 — int4 block linears plus BF16 DeltaNet/vision/embeddings/lm_head, so int4 kernels (Marlin/compressed-tensors) engage for the quantized linears only. ~27 GB of weights: fits a 32GB+ GPU at moderate context (KV ≈2 GB at 32k). Full 262,144-token context needs ≈17 GB of KV on top — plan for ~48GB total, or cap --max-model-len on smaller cards. KV ≈65 KB/token (hybrid: only global-attention layers accumulate KV; DeltaNet layers carry fixed-size state). MTP predictor weights are included intact; the speculative-decoding flag above follows the base-model recipe — if your vLLM build rejects MTP for this architecture, drop the --speculative-config line and serve normally.

The base model supports YaRN extension to 1M context via --hf-overrides (see the base model card). This checkpoint was calibrated and validated only at the native 262,144 tokens; behavior beyond that length is untested and YaRN interacts with RoPE-bearing attention layers — run your own long-context eval before relying on it.