README.md

metadata

language:
  - en
  - zh
license: mit
library_name: vllm
pipeline_tag: text-generation
tags:
  - glm-5.2
  - quantized
  - w4a16
  - int4
  - gptq
  - compressed-tensors
  - mtp
  - speculative-decoding
base_model: zai-org/GLM-5.2

GLM-5.2 — W4A16 (INT4) + BF16 MTP

An INT4 weight-only (W4A16) quantization of GLM-5.2 that preserves the BF16 multi-token-prediction (MTP) layer for speculative decoding. Quantized from zai-org/GLM-5.2 with llm-compressor (GPTQ).

Purpose

GLM-5.2 (744B-parameter MoE) in BF16 needs ~1.49 TB of weights — eight 141 GB H200s, fully occupied, to serve one replica. The goal of this artifact is a smaller-footprint variant that matches FP8 quality so the model runs on four H200s instead of eight (freeing half the fleet, or two replicas per node), while keeping the MTP draft head for speculative-decode speedups. It is a deployment-efficiency artifact, not a new model — all capability comes from the base GLM-5.2.

Details

Field	Value
Base model	zai-org/GLM-5.2 (BF16)
Architecture	GlmMoeDsaForCausalLM — 744B MoE, ~40B active, MLA + DeepSeek Sparse Attention, 1M context
Weight quantization	W4A16, INT4, asymmetric, group-size 128 (GPTQ, compressed-tensors), routed experts only
Kept in BF16	attention, dense layers (0–2), shared experts, router/gate, embeddings, lm_head, MTP layer 78
MTP	layer 78 preserved at BF16 for spec-decode (num_speculative_tokens=5)
Calibration	in-distribution chat/code set; calibrate_all_experts=True (visits every expert — see Method)
Size	~405 GB (from ~1488 GB BF16)
License	MIT (inherited from the base model)

The "FP8" sometimes seen in the filename refers to the fp8 KV-cache used at serving time, not the weights — the weights are INT4 (W4A16) and the MTP layer is BF16.

Evaluation — vs the FP8 baseline (same harness, 8×H200)

Measured against zai-org/GLM-5.2-FP8 under an identical setup (generative tasks via chat-completions with a 16,384-token generation budget for the reasoning CoT; matched serve config with --reasoning-parser).

Task	This (W4A16+MTP)	FP8 baseline
GSM8K (strict)	0.960	0.955
IFEval (prompt-strict / inst-strict)	0.909 / 0.911	0.891 / 0.903
MATH-500 (math-verify)	0.954	0.958
RULER @ 32K	0.832	0.831
RULER @ 64K	0.841	0.813
SWE-bench Verified (mini-SWE-agent + official grading)	82.0% (410/500)	82.2% (411/500)

Quantization preserves quality: scores track the FP8 baseline within run-to-run noise on reasoning, instruction-following, long-context retrieval, and agentic coding. (MMLU-Pro: FP8 full-set = 0.820; the W4A16 subset run was not completed — the verdict was already conclusive from the six tasks above. RULER used 50 samples per sub-task, not the full 500.)

Long context: serves at max_model_len=1,048,576 on 8×H200 and correctly retrieved a needle from a ~936K-token prompt (MLA + DSA compress the KV cache enough to fit 1M in the memory free after weights).

MTP: speculative-decode acceptance 46–52% aggregate (95% at draft position 0), confirming the injected BF16 MTP layer is healthy.

Throughput (8×H200, vLLM bench, output tok/s):

concurrency	This	FP8
1	132 (+48%)	89
8	466 (+32%)	354
32	825 (−13%)	953

Faster than FP8 at low/medium concurrency (MTP speculative decoding helps most in the interactive regime) and slightly slower at full saturation — honest trade-off, both directions shown.

Serving (vLLM ≥ 0.23, Hopper / H200)

The asymmetric W4A16 MoE requires expert parallelism (--enable-expert-parallel); plain tensor-parallel trips a Marlin scale-sharding bug. The DSA indexer needs an nvcc ≥ 12.8 toolchain (CUDA_HOME).

8×H200 (up to 1M context):

vllm serve <repo> \
  --tensor-parallel-size 8 --enable-expert-parallel \
  --kv-cache-dtype fp8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
  --max-model-len 1048576 --gpu-memory-utilization 0.90 --trust-remote-code

4×H200 (the footprint win, ≤ ~128K context — 1M needs all 8):

vllm serve <repo> --tensor-parallel-size 4 --enable-expert-parallel \
  --kv-cache-dtype fp8 --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
  --max-model-len 32768 --gpu-memory-utilization 0.92 --trust-remote-code

Validated on Hopper (H200). On Blackwell (sm100) the serving kernels need extra flags and are not yet recommended for this artifact.

Method

1. GPTQ W4A16 (group-128, asymmetric) on the routed experts only, with attention/dense/MTP/embeddings/ lm_head held at BF16. calibrate_all_experts=True is required — calibrating only routed experts starves rarely-activated experts and produces a coherent-looking but degenerate model.
2. MTP preservation (Option-Y): GLM-5.2's MTP/nextn layer (index 78) isn't instantiated by from_pretrained, so quantization never sees it. It is injected back at BF16 from the source checkpoint after quantization and added to the ignore list so the serving stack treats it as unquantized.

The full recipe, evaluation methodology, and a log of the engineering walls hit and overcome are in the companion repository (calibration memory limits, MoE sequential-target OOMs, the MTP-loss-on-save issue, the asymmetric-MoE serving fix, and the Blackwell toolchain gaps).

Limitations

• Throughput is ~13% below FP8 at very high concurrency (c32); the win is at low/medium concurrency.
• 1M-context serving requires all 8 H200s; 4×H200 is for ≤ ~128K.
• Asymmetric weights require --enable-expert-parallel to serve correctly.
• Recommended on Hopper; Blackwell serving needs additional kernel flags.

Acknowledgements

Built on zai-org/GLM-5.2 (MIT). Quantized with llm-compressor; served with vLLM.