malaiwah/GLM-5.2-EXL3-TR3-MTP78

GLM-5.2 MTP Layer-78 — EXL3 Trellis overlays (2/3/4 bpw × keep64/keep0)

Quantized MTP draft layer (layer 78) overlays for GLM-5.2-EXL3-TR3-3.0bpw, produced with Brandon Music's encode_tr3_v31.py (LDLQ-calibrated trellis, MCG codebook 0xCBAC1FED, exllamav3 v0.0.43 kernels, TP4 rank-sliced tensors) from a full-corpus calibration capture: 7,288,310 tokens of layer-78 MoE inputs + routed top-8 expert ids, captured during live serving over the complete 12,228-row / 4-axis owner corpus (reap_recall_calib.jsonl, sha256 cf247acc…). Capture dataset published separately: malaiwah/GLM-5.2-MTP78-calibration-capture.

Variants

folder	bpw	experts trellis-encoded	size	per-expert nmse (typ.)	status
3bpw-keep0/	3.0	256/256	3.7 GB	~2.2e-02	DEPLOY FLAGSHIP — validated
2bpw-keep0/	2.0	256/256	2.5 GB	~8.8e-02	encodes clean, unvalidated
4bpw-keep0/	4.0	256/256	4.9 GB	~5.7e-03	encodes clean, unvalidated
3bpw/	3.0	192/256 (worst-64 kept BF16)	2.6 GB	~2.2e-02	future-loader artifact*
2bpw/	2.0	192/256	1.8 GB	~8.8e-02	future-loader artifact*
4bpw/	4.0	192/256	3.7 GB	~5.7e-03	future-loader artifact*

* keep64 variants protect the 64 highest trellis-roundtrip-error experts in BF16 — but the current rank-sliced EXL3 runtime has no mixed BF16+trellis expert path (fails with KeyError …w2_weight); the base checkpoint's own convention is keep0 (nvfp4_keep_per_layer: 0). Published for future loaders.

Validation (3bpw-keep0)

Identical 20-prompt greedy bench (300 tok/req), vLLM 0.17.0rc1 RC2 image, 4× RTX PRO 6000 Blackwell, TP4, MTP-3:

draft	Mean acceptance length	notes
BF16 (19.3 GB)	3.054 (n=57 windows, 2.35–3.94)	baseline
3bpw-keep0 (3.7 GB)	3.06 (n=30 windows, 2.15–3.54)	parity at ~1/5 the size
online-FP8 ("quantization":"fp8")	n/a	CUDA IMA on this image

Draft VRAM at TP4: BF16 ≈ 4.8 GB/GPU → 3bpw-keep0 ≈ 0.95 GB/GPU — the reclaimed ~3.8 GB/GPU goes straight to KV cache.

Deploying (graft onto GLM-5.2-EXL3-TR3-3.0bpw)

1. Replace model-layer-078.safetensors with a merged shard: the 23 non-expert layer-78 tensors (BF16, from the original shard) + all tensors from 3bpw-keep0/tr3-layer-078.safetensors; rebuild model.safetensors.index.json accordingly (drop the 768 old expert .weight entries).
2. config.json: set hybrid_tr3_tail.moe_layers to [3, 78] and remove the model.layers.78* entry from quantization_config.ignore.
3. vLLM RC2 needs a 1-hunk patch: deepseek_mtp.py load_weights must apply quant_config.normalize_rank_sliced_weight_name() like deepseek_v2.load_weights already does (without it: KeyError …routed_experts.w2_rank0.mcg). Patch in tools/.
4. Config sensitivity on the RC2 image: the trellis draft validated on the cudagraph/DCP4 serving config; the BF16 draft conversely crashes there and needs DCP1/eager. Pick per draft type.

tools/ contains the graft builder, the vLLM patch, the capture hook, the bench driver, and per-variant layer-078.done.json (full per-expert routed counts + rel-RT-mse from the encode).

Credits

zai-org (GLM-5.2 base, MIT) · Brandon Music (EXL3 TR3 encoder + owner corpus + base checkpoint) · Josh Cartu / jcartu (MTP78 recipe, rank-sliced MTP runtime) · Luke Alonso (b12x) · turboderp-org (exllamav3, MIT) · Martin Vit + @yatesdr (RC2 image). Capture/encode/validation: malaiwah, 2026-07-24/25, 4× RTX PRO 6000 rental.