dnhkng/GLM-5.2-AWQ-INT4-FP8-MTP-delta

GLM-5.2 AWQ + FP8 MTP Graft

This repository contains the script and vLLM patch needed to graft the official FP8 MTP layer from zai-org/GLM-5.2-FP8 onto cyankiwi/GLM-5.2-AWQ-INT4.

It does not contain model weights. The graft script downloads the official FP8 checkpoint shards that contain model.layers.78.*, extracts only those MTP tensors locally, and writes a merged local checkpoint.

Contents

• graft_glm52_awq_mtp.sh: builds a local AWQ checkpoint with the FP8 MTP layer.
• patches/vllm-awq-fp8-mtp-quant-config.patch: vLLM patch for mixed AWQ main weights plus FP8 MTP weights.

Why the vLLM patch is needed

The base AWQ model uses compressed-tensors quantization, while the grafted MTP layer uses FP8. The generated checkpoint records this as mtp_quantization_config in config.json.

Without the patch, vLLM applies the main AWQ quantization config to the MTP layer. The patch makes the GLM/DeepSeek MTP loader use mtp_quantization_config for the MTP decoder layer when it is present.

Patch and Build vLLM

Clone vLLM, apply the patch from this repo, and build/install vLLM in editable mode. Exact build dependencies vary by system; this is the minimal source flow:

git clone https://github.com/vllm-project/vllm.git
cd vllm

git apply /path/to/this-repo/patches/vllm-awq-fp8-mtp-quant-config.patch

python -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
python -m pip install -r requirements/build.txt
python -m pip install -e .

On Grace Hopper or other non-standard CUDA systems, use the local vLLM build settings appropriate for your machine. The important requirement is that this patch is applied before serving the grafted AWQ+FP8-MTP checkpoint.

Build the Grafted Checkpoint

Download the AWQ base model:

huggingface-cli download cyankiwi/GLM-5.2-AWQ-INT4 \
  --local-dir /models/GLM-5.2-AWQ-INT4

Then run the graft script:

./graft_glm52_awq_mtp.sh \
  --awq-dir /models/GLM-5.2-AWQ-INT4 \
  --download-fp8-mtp \
  --out-dir /models/GLM-5.2-AWQ-INT4-MTP-FP8

The script downloads only the official FP8 metadata plus the shards referenced by model.layers.78.*, extracts those tensors into compact local MTP shards, and merges them into the AWQ checkpoint index.

If you already have the full official FP8 checkpoint locally, you can avoid the download:

./graft_glm52_awq_mtp.sh \
  --awq-dir /models/GLM-5.2-AWQ-INT4 \
  --fp8-dir /models/GLM-5.2-FP8 \
  --extract-local-mtp \
  --out-dir /models/GLM-5.2-AWQ-INT4-MTP-FP8

Serve with vLLM

Example serving command:

vllm serve /models/GLM-5.2-AWQ-INT4-MTP-FP8 \
  --served-model-name glm52-awq-mtp3 \
  --trust-remote-code \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --spec-method mtp \
  --spec-tokens 3

For dual-GPU tensor parallel serving, add the usual vLLM flags for your system, for example:

  --tensor-parallel-size 2 \
  --distributed-executor-backend mp

For Grace Hopper systems with unified-memory offload, use the flags appropriate for your placement and memory budget.

Notes

• The base AWQ weights are unchanged.
• The grafted checkpoint adds model.layers.78.* and mtp_quantization_config.
• The practical default we found for GLM-5.2 AWQ+MTP serving was --spec-tokens 3.
• This repo intentionally avoids redistributing the extracted MTP weights.