---
license: mit
base_model: zai-org/GLM-5.2-FP8
base_model_relation: quantized
pipeline_tag: text-generation
library_name: vllm
language:
- en
- zh
tags:
- glm
- glm_moe_dsa
- mixture-of-experts
- w4a8
- int4
- fp8
- compressed-tensors
- quantized
- reasoning
- tool-calling
- vllm
---

# GLM-5.2-W4A8-CT

GLM-5.2-W4A8-CT is a 4-bit quantization of
[zai-org/GLM-5.2-FP8](https://huggingface.co/zai-org/GLM-5.2-FP8) for vLLM. It
stores the routed MoE experts as group-wise INT4 weights for dynamic FP8
activations using the `compressed-tensors` format. The dense linear layers use
block FP8 weights.

The quantized weights are the same as those in
[PhalaCloud/GLM-5.2-W4AFP8](https://huggingface.co/PhalaCloud/GLM-5.2-W4AFP8).
That repository uses SGLang's W4AFP8 serialization; this repository packages
the weights in vLLM's `compressed-tensors` serialization. This packaging step
does not recalibrate or requantize the expert values. The runtime-aware
AWQ-calibrated MTP layer is included.

## Requirements

- NVIDIA Hopper GPU (H100, H200, or GH200)
- A recent vLLM build with compressed-tensors W4A8 MoE support
- About 373 GiB of storage for the checkpoint

The checkpoint was validated with unmodified vLLM commit `34b560b72`
(`0.23.1rc1.dev786+g34b560b72.precompiled`) on one GH200 using UVA host-memory
offload.

## Serve with vLLM

The following configuration is intended for an eight-GPU Hopper server where
the weights fit in aggregate VRAM:

```bash
vllm serve PhalaCloud/GLM-5.2-W4A8-CT \
  --tensor-parallel-size 8 \
  --dtype bfloat16 \
  --enable-auto-tool-choice \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'
```

The server exposes the OpenAI-compatible chat-completions API. The validated
single-GH200 deployment uses vLLM's UVA offloader because the model is larger
than HBM. That setup is functional but substantially slower than keeping all
weights in GPU memory.

## Checkpoint format

| Component | Stored format |
| --- | --- |
| Routed MoE expert weights | Group-wise INT4, group size 128 |
| Routed MoE activations | Dynamic per-token FP8 E4M3 |
| Dense linear weights | Block FP8 |
| MTP layer | Included, with runtime-aware AWQ expert calibration |
| Serialization | Hugging Face safetensors, 41 shards |

At startup, vLLM converts the portable compressed-tensors INT4 representation
to the tiled layout required by its CUTLASS W4A8 kernels. This post-load step
can add noticeable cold-start time for heavily offloaded deployments.

## Validation

The converted checkpoint has passed these checks:

- all 41 safetensors shards load successfully in unmodified vLLM;
- native OpenAI-format tool calls work with the `glm47` parser;
- the bundled MTP layer produces accepted speculative tokens;
- HumanEval: 159/164 base tests passed (96.95%);
- HumanEval+: 151/164 combined tests passed (92.07%).

BFCL single-turn tool-calling evaluation is in progress. Results will be added
when the run completes.

## Conversion details

The routed-expert INT4 weights originate from quantizing GLM-5.2-FP8. For this
release, those already-quantized weights were converted from SGLang's W4AFP8
serialization to vLLM's `compressed-tensors` mixed-precision schema. The format
conversion repacked 58,368 expert INT4 tensors, renamed 654 FP8 scale tensors,
preserved sparse-attention indexer scales, and produced 176,997 tensors across
41 shards. The included `vllm_conversion_manifest.json` records the source
hashes and conversion counts.

## Limitations

- Only Hopper has been tested.
- A single-GPU deployment requires aggressive host-memory offload and is much
  slower than an all-VRAM deployment.
- Cold starts include a CUTLASS layout-conversion step.
- The full BFCL result is not yet available.

## License

MIT, inherited from [GLM-5.2](https://huggingface.co/zai-org/GLM-5.2-FP8).
See the original model card for architecture, training, and citation details.