README.md

---
language:
  - en
  - zh
license: mit
library_name: vllm
pipeline_tag: text-generation
tags:
  - glm-5.2
  - quantized
  - w4a16
  - int4
  - gptq
  - compressed-tensors
  - mtp
  - speculative-decoding
base_model: zai-org/GLM-5.2
---

# GLM-5.2 — W4A16 (INT4) + BF16 MTP

An **INT4 weight-only (W4A16) quantization of GLM-5.2** that preserves the BF16 multi-token-prediction (MTP)
layer for speculative decoding. Quantized from [zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2)
with [llm-compressor](https://github.com/vllm-project/llm-compressor) (GPTQ).

## Purpose

GLM-5.2 (744B-parameter MoE) in BF16 needs ~1.49 TB of weights — eight 141 GB H200s, fully occupied, to serve
one replica. The goal of this artifact is a **smaller-footprint variant that matches FP8 quality** so the model
runs on **four H200s instead of eight** (freeing half the fleet, or two replicas per node), while keeping the
MTP draft head for speculative-decode speedups. It is a deployment-efficiency artifact, not a new model — all
capability comes from the base GLM-5.2.

## Details

| Field | Value |
|---|---|
| Base model | [zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) (BF16) |
| Architecture | `GlmMoeDsaForCausalLM` — 744B MoE, ~40B active, MLA + DeepSeek Sparse Attention, 1M context |
| Weight quantization | **W4A16, INT4, asymmetric, group-size 128** (GPTQ, compressed-tensors), **routed experts only** |
| Kept in BF16 | attention, dense layers (0–2), shared experts, router/gate, embeddings, lm_head, **MTP layer 78** |
| MTP | layer 78 preserved at BF16 for spec-decode (`num_speculative_tokens=5`) |
| Calibration | in-distribution chat/code set; **`calibrate_all_experts=True`** (visits every expert — see Method) |
| Size | ~405 GB (from ~1488 GB BF16) |
| License | MIT (inherited from the base model) |

> The "FP8" sometimes seen in the filename refers to the **fp8 KV-cache used at serving time**, not the
> weights — the weights are INT4 (W4A16) and the MTP layer is BF16.

## Evaluation — vs the FP8 baseline (same harness, 8×H200)

Measured against `zai-org/GLM-5.2-FP8` under an identical setup (generative tasks via chat-completions with a
16,384-token generation budget for the reasoning CoT; matched serve config with `--reasoning-parser`).

| Task | This (W4A16+MTP) | FP8 baseline |
|---|---|---|
| GSM8K (strict) | 0.960 | 0.955 |
| IFEval (prompt-strict / inst-strict) | 0.909 / 0.911 | 0.891 / 0.903 |
| MATH-500 (math-verify) | 0.954 | 0.958 |
| RULER @ 32K | 0.832 | 0.831 |
| RULER @ 64K | 0.841 | 0.813 |
| SWE-bench Verified (mini-SWE-agent + official grading) | **82.0%** (410/500) | 82.2% (411/500) |

**Quantization preserves quality:** scores track the FP8 baseline within run-to-run noise on reasoning,
instruction-following, long-context retrieval, and agentic coding. (MMLU-Pro: FP8 full-set = 0.820; the W4A16
subset run was not completed — the verdict was already conclusive from the six tasks above. RULER used 50
samples per sub-task, not the full 500.)

**Long context:** serves at `max_model_len=1,048,576` on 8×H200 and correctly retrieved a needle from a
~936K-token prompt (MLA + DSA compress the KV cache enough to fit 1M in the memory free after weights).

**MTP:** speculative-decode acceptance ~46–52% aggregate (~95% at draft position 0), confirming the injected
BF16 MTP layer is healthy.

**Throughput (8×H200, vLLM bench, output tok/s):**

| concurrency | This | FP8 |
|---|---|---|
| 1 | 132 (+48%) | 89 |
| 8 | 466 (+32%) | 354 |
| 32 | 825 (−13%) | 953 |

Faster than FP8 at low/medium concurrency (MTP speculative decoding helps most in the interactive regime) and
slightly slower at full saturation — honest trade-off, both directions shown.

## Serving (vLLM ≥ 0.23, Hopper / H200)

The asymmetric W4A16 MoE **requires expert parallelism** (`--enable-expert-parallel`); plain tensor-parallel
trips a Marlin scale-sharding bug. The DSA indexer needs an nvcc ≥ 12.8 toolchain (`CUDA_HOME`).

**8×H200 (up to 1M context):**
```bash
vllm serve <repo> \
  --tensor-parallel-size 8 --enable-expert-parallel \
  --kv-cache-dtype fp8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
  --max-model-len 1048576 --gpu-memory-utilization 0.90 --trust-remote-code
```

**4×H200 (the footprint win, ≤ ~128K context — 1M needs all 8):**
```bash
vllm serve <repo> --tensor-parallel-size 4 --enable-expert-parallel \
  --kv-cache-dtype fp8 --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
  --max-model-len 32768 --gpu-memory-utilization 0.92 --trust-remote-code
```

Validated on Hopper (H200). On Blackwell (sm100) the serving kernels need extra flags and are not yet
recommended for this artifact.

## Method

1. **GPTQ W4A16** (group-128, asymmetric) on the routed experts only, with attention/dense/MTP/embeddings/
   lm_head held at BF16. `calibrate_all_experts=True` is required — calibrating only routed experts starves
   rarely-activated experts and produces a coherent-looking but degenerate model.
2. **MTP preservation (Option-Y):** GLM-5.2's MTP/nextn layer (index 78) isn't instantiated by
   `from_pretrained`, so quantization never sees it. It is injected back at BF16 from the source checkpoint
   after quantization and added to the `ignore` list so the serving stack treats it as unquantized.

The full recipe, evaluation methodology, and a log of the engineering walls hit and overcome are in the
companion repository (calibration memory limits, MoE sequential-target OOMs, the MTP-loss-on-save issue, the
asymmetric-MoE serving fix, and the Blackwell toolchain gaps).

## Limitations

- Throughput is ~13% below FP8 at very high concurrency (c32); the win is at low/medium concurrency.
- 1M-context serving requires all 8 H200s; 4×H200 is for ≤ ~128K.
- Asymmetric weights require `--enable-expert-parallel` to serve correctly.
- Recommended on Hopper; Blackwell serving needs additional kernel flags.

## Acknowledgements

Built on [zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) (MIT). Quantized with
[llm-compressor](https://github.com/vllm-project/llm-compressor); served with [vLLM](https://github.com/vllm-project/vllm).