canada-quant/GLM-5.2-W4A16-MTP

GLM-5.2 — W4A16 (INT4) + BF16 MTP

An INT4 weight-only (W4A16) quantization of GLM-5.2 that preserves the BF16 multi-token-prediction (MTP) layer for speculative decoding. Quantized from zai-org/GLM-5.2 with llm-compressor (GPTQ).

Built for Hopper (H200), validated on Blackwell (8×RTX PRO 6000, SM120). Matches FP8 quality on half the GPUs (4×H200 vs 8) and is the fastest GLM-5.2 quant for interactive/agentic serving on Hopper in a matched, MTP-on head-to-head — with the lowest time-to-first-token by a wide margin. A complete, quality-validated RTX PRO 6000 recipe is in the Serving on Blackwell section below.

Why this model

• Half the footprint, FP8 quality. ~405 GB of weights (down from ~1.49 TB BF16) serve one replica on 4×H200 instead of 8 — freeing half the fleet, or two replicas per node — and eval matches the FP8 baseline within noise across reasoning, instruction-following, long-context, and agentic coding.
• Fastest interactive serving among GLM-5.2 quants on Hopper. In a matched benchmark (every model with MTP on, same box, same vLLM, same harness): +8% vs nvidia NVFP4 and +33% vs zai FP8 at concurrency 1, with TTFT of 215 ms vs 632/1258 ms.
• Honest trade-off. MTP's draft/verify overhead stops paying off once the batch saturates — at c32 the NVFP4/FP8 quants are ~11% faster. If your workload is fully-saturated batch serving, pick by that row.

Throughput — matched MTP-on comparison (8×H200, vLLM v0.23.0, same harness)

All three models served identically (--speculative-config '{"method":"mtp","num_speculative_tokens":5}', TP=8, fp8 KV); benchmarked with vllm bench serve (openai-chat endpoint, random 1024-in/512-out, num-prompts = 8×concurrency). Measured 2026-07-02 on AWS p5e.48xlarge.

concurrency	This (W4A16 + MTP)	nvidia NVFP4 + MTP	zai FP8 + MTP
1 — tok/s (TTFT)	125.7 (215 ms)	116.3 (632 ms)	94.4 (1258 ms)
8	495.7 (319 ms)	455.5 (422 ms)	394.2 (742 ms)
32	828.4 (413 ms)	925.2 (403 ms)	921.4 (412 ms)

MTP spec-decode acceptance was ~28% for all three models on this synthetic workload (higher, ~46–52%, on natural eval traffic) — the draft head performs the same across quants, so this is a clean quant-vs-quant comparison. Note NVFP4 is a Blackwell-native format measured here on Hopper, where it has no FP4 tensor cores; treat its column as a Hopper-deployment number.

Purpose

GLM-5.2 (744B-parameter MoE) in BF16 needs ~1.49 TB of weights — eight 141 GB H200s, fully occupied, to serve one replica. The goal of this artifact is a smaller-footprint variant that matches FP8 quality so the model runs on four H200s instead of eight (freeing half the fleet, or two replicas per node), while keeping the MTP draft head for speculative-decode speedups. It is a deployment-efficiency artifact, not a new model — all capability comes from the base GLM-5.2.

Details

Field	Value
Base model	zai-org/GLM-5.2 (BF16)
Architecture	GlmMoeDsaForCausalLM — 744B MoE, ~40B active, MLA + DeepSeek Sparse Attention, 1M context
Weight quantization	W4A16, INT4, asymmetric, group-size 128 (GPTQ, compressed-tensors), routed experts only
Kept in BF16	attention, dense layers (0–2), shared experts, router/gate, embeddings, lm_head, MTP layer 78
MTP	layer 78 preserved at BF16 for spec-decode (num_speculative_tokens=5)
Calibration	in-distribution chat/code set; calibrate_all_experts=True (visits every expert — see Method)
Size	~405 GB (from ~1488 GB BF16)
License	MIT (inherited from the base model)

The "FP8" sometimes seen in the filename refers to the fp8 KV-cache used at serving time, not the weights — the weights are INT4 (W4A16) and the MTP layer is BF16.

Evaluation — vs the FP8 baseline (same harness, 8×H200)

Measured against zai-org/GLM-5.2-FP8 under an identical setup (generative tasks via chat-completions with a 16,384-token generation budget for the reasoning CoT; matched serve config with --reasoning-parser). Quality is measured with speculative decoding off, where it is exact — MTP changes latency, not outputs.

Task	This (W4A16+MTP)	FP8 baseline
GSM8K (strict)	0.960	0.955
IFEval (prompt-strict / inst-strict)	0.909 / 0.911	0.891 / 0.903
MATH-500 (math-verify)	0.954	0.958
RULER @ 32K	0.832	0.831
RULER @ 64K	0.841	0.813
SWE-bench Verified (mini-SWE-agent + official grading)	82.0% (410/500)	82.2% (411/500)

Quantization preserves quality: scores track the FP8 baseline within run-to-run noise on reasoning, instruction-following, long-context retrieval, and agentic coding. (MMLU-Pro: FP8 full-set = 0.820; the W4A16 subset run was not completed — the verdict was already conclusive from the six tasks above. RULER used 50 samples per sub-task, not the full 500.)

Long context: serves at max_model_len=1,048,576 on 8×H200 and correctly retrieved a needle from a ~936K-token prompt (MLA + DSA compress the KV cache enough to fit 1M in the memory free after weights). On 4×H200 it serves 128K validated (single-stream engine ceiling ~239K at gpu-memory-utilization=0.92; 256K overflows the post-weights KV budget) and retrieved a 64K needle at both mid- and end-placement.

MTP: speculative-decode acceptance 46–52% aggregate (95% at draft position 0) on natural eval traffic on 8×H200, confirming the injected BF16 MTP layer is healthy. On 4×H200 (TP=4, 128K) aggregate acceptance is ~38% (7,848/20,765 draft tokens, mean accept-length ~2.9) — mildly lower under the tighter memory split but still a net speedup.

Serving (vLLM ≥ 0.23, Hopper / H200)

The asymmetric W4A16 MoE requires expert parallelism (--enable-expert-parallel); plain tensor-parallel trips a Marlin scale-sharding bug. The DSA indexer needs an nvcc ≥ 12.8 toolchain (CUDA_HOME). Validated on vLLM v0.23.0 (newer versions changed the DSA indexer layout — v0.24+ currently fails to load this checkpoint's per-layer indexer weights; pin v0.23.x until upstream support lands).

8×H200 (up to 1M context):

vllm serve <repo> \
  --tensor-parallel-size 8 --enable-expert-parallel \
  --kv-cache-dtype fp8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
  --max-model-len 1048576 --gpu-memory-utilization 0.90 --trust-remote-code

4×H200 (the footprint win, 128K validated / ~239K single-stream ceiling — 1M needs all 8):

vllm serve <repo> --tensor-parallel-size 4 --enable-expert-parallel \
  --kv-cache-dtype fp8 --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
  --max-model-len 32768 --gpu-memory-utilization 0.92 --trust-remote-code

Serving on Blackwell — 8×RTX PRO 6000 (SM120)

Validated end-to-end on 8× RTX PRO 6000 (96 GB, SM 12.0, PCIe): quality matches the H200 deployment (GSM8K 0.948, IFEval 0.909/0.920, MATH-500 0.954 math-verify, RULER@32K 0.918 / @64K 0.826 — all within margin of the H200 column above), zero token corruption, with MTP speculative decoding and cudagraphs both working. Throughput on this config:

concurrency	1	4	8	16	32	64
output tok/s	50	148	280	400	613	989

For reference, nvidia/GLM-5.2-NVFP4 on the same box (MTP off — a matched MTP-on NVFP4 run was not completed) measures 36 tok/s at c=1 and 447 at c=64. Single-stream speed is memory-bandwidth-bound on this hardware (~50 vs ~126 tok/s on H200); the sweet spot is concurrent/batch serving.

Why SM120 needs a different recipe: no SM120 sparse-MLA kernel supports GLM-5.2's DSA head layout, so DSA sparse attention is disabled and the model serves through dense TRITON_MLA. That takes four one-line patches on the official vllm/vllm-openai:glm52 image:

FROM vllm/vllm-openai:glm52

# 1. Disable DSA sparse attention (no SM120 sparse-MLA backend for this head size)
RUN sed -i 's/self\.is_v32 = hasattr(config, "index_topk")/self.is_v32 = False/g' \
    /usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v2.py

# 2-4. Skip the now-orphaned DSA indexer weights during load (deepseek_v2.py,
#      deepseek_mtp.py, glm4_moe_mtp.py): guard `param = params_dict[name]` with
#      `if name not in params_dict: continue`
RUN python3 - <<'EOF'
import re
base = '/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/'
for f in ('deepseek_v2.py', 'deepseek_mtp.py', 'glm4_moe_mtp.py'):
    p = base + f
    src = open(p).read()
    src = re.sub(r'(\s+)param = params_dict\[name\]',
                 r'\1if name not in params_dict:\n\1    continue\n\1param = params_dict[name]', src)
    open(p, 'w').write(src)
EOF

Then serve (docker build -t glm52-mtp-sm120 . first):

docker run -d --gpus all --ipc=host --shm-size 16g \
  -v /path/to/GLM-5.2-W4A16-MTP:/model:ro -p 8000:8000 \
  -e NCCL_P2P_DISABLE=1 -e VLLM_USE_DEEP_GEMM=0 -e VLLM_MOE_USE_DEEP_GEMM=0 \
  glm52-mtp-sm120 /model \
  --tensor-parallel-size 8 --enable-expert-parallel \
  --attention-backend TRITON_MLA \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
  --compilation-config '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[2,4,8,16,32,64],"max_cudagraph_capture_size":64}' \
  --max-model-len 131072 --max-num-seqs 64 --gpu-memory-utilization 0.92 \
  --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
  --disable-custom-all-reduce --trust-remote-code

Every deviation from the Hopper command is load-bearing:

• bf16 KV cache (no --kv-cache-dtype fp8) — fp8 KV overflows SM120's per-SM shared memory in the TRITON_MLA kernel (102,400 > 101,376 bytes).
• PIECEWISE cudagraph mode — FULL decode graphs + MTP produce degenerate output on TRITON_MLA (its decode kernel only handles single-token queries; MTP verify sends multi-token queries). PIECEWISE routes them through the prefill path correctly and still gives ~10× over eager at c=1.
• MTP num_speculative_tokens=1, with cudagraph capture sizes divisible by (1 + k) = 2.
• VLLM_USE_DEEP_GEMM=0 — DeepGEMM's attention path doesn't support SM120.
• --attention-backend TRITON_MLA — the dense-MLA backend that works once DSA is disabled.

A one-shot bootstrap script (HF download → image build → launch, idempotent) exists in the companion repository as scripts/bootstrap_sm120_glm52_w4a16_mtp.sh.

Method

1. GPTQ W4A16 (group-128, asymmetric) on the routed experts only, with attention/dense/MTP/embeddings/ lm_head held at BF16. calibrate_all_experts=True is required — calibrating only routed experts starves rarely-activated experts and produces a coherent-looking but degenerate model.
2. MTP preservation (Option-Y): GLM-5.2's MTP/nextn layer (index 78) isn't instantiated by from_pretrained, so quantization never sees it. It is injected back at BF16 from the source checkpoint after quantization and added to the ignore list so the serving stack treats it as unquantized.

The full recipe, evaluation methodology, and a log of the engineering walls hit and overcome are in the companion repository (calibration memory limits, MoE sequential-target OOMs, the MTP-loss-on-save issue, the asymmetric-MoE serving fix, and the Blackwell toolchain gaps).

Limitations

• The interactive edge is +8% vs NVFP4+MTP / +33% vs FP8+MTP at c1; at full saturation (c32) those quants are ~11% faster. Pick by your operating point.
• 1M-context serving requires all 8 H200s; 4×H200 serves up to ~128K (single-stream engine ceiling ~239K), with MTP acceptance ~38% (vs ~46–52% on 8×H200).
• Asymmetric weights require --enable-expert-parallel to serve correctly.
• Pin vLLM v0.23.x (v0.24+ DSA-indexer layout change breaks loading).
• On Blackwell SM120 (RTX PRO 6000) use the dedicated recipe above: DSA sparse attention must be disabled (dense TRITON_MLA), KV cache stays bf16, cudagraphs run in PIECEWISE mode, and MTP is limited to k=1. Quality is unaffected (validated at parity with H200); single-stream throughput is bandwidth-bound at ~50 tok/s, so size deployments for concurrent traffic. FULL-cudagraph + MTP on TRITON_MLA is an upstream kernel gap (vllm-project/vllm#21505), not fixable by configuration.

Acknowledgements

Built on zai-org/GLM-5.2 (MIT). Quantized with llm-compressor; served with vLLM.