GLM-5.2-Alis-MLX-Dynamic-3.5bpw

Apple Silicon (MLX) mixed-precision quantization of zai-org/GLM-5.2 — a 744B-parameter (~40B active) Mixture-of-Experts model with DeepSeek-V3.2-style MLA + DeepSeek Sparse Attention (DSA, glm_moe_dsa).

This build targets the "golden spot" for a 512 GB M3 Ultra: the best quality that still runs a full 1M-token context comfortably — ~3.5 bits/weight, served with int8 quantization of the MLA latent KV cache.

New: the checkpoint now ships GLM-5.2's native MTP (nextn) layer for lossless self-speculative decoding (--mtp — see Native MTP). Fully backward-compatible: runtimes without MTP support simply strip the extra layer and behave exactly as before.

New (2026-07): quality-flagship sibling — GLM-5.2-Alis-MLX-Dynamic-4.5bpw promotes the routed experts to NVFP4 (4-bit float, gs16): strided PPL −4.8% (wikitext) / −1.1% (code) vs this build (this build is now DWQ-retuned — see below — which narrowed the gap from the earlier −7.9%/−3.8%), and MTP pays more on its cleaner target (accept-len 2.61–2.87, +15–16%), at ~424 GB with a ~500–600K-token context ceiling (requires mlx ≥ 0.30.3). This build remains the 1M-context choice — pick the 4.5 bpw sibling when quality matters more than context budget.

Update (2026-07): DWQ-retuned weights. main now carries quantization scales/biases re-tuned with layerwise DWQ (alis-dwq; upstream flag: mlx-lm#1499) against the 4.5 bpw sibling as teacher, with a 45%-ZH calibration mix. Same recipe, size class and speed; distributional fidelity improves:

KL / top-1 flip vs the 4.5 bpw reference (T=3072)	pre-DWQ	DWQ (main)
EN	0.246 / 13.7%	0.181 / 10.6%
code	0.115 / 9.3%	0.086 / 8.1%
ZH	0.371 / 21.6%	0.291 / 16.8%
overall	0.244 / 14.8%	0.186 / 11.9% (−24% KL)

The pre-DWQ weights remain on the pre-dwq branch. The Quality and Benchmarks figures below are measured on these DWQ weights — the retune moves strided PPL (wikitext 2.946→2.851, code 1.893→1.841) and mlx_lm.perplexity on tulu-3 (3.766→3.603). The gain is smaller than the 2-bit sibling's: at 3.5 bpw the build is already close to the teacher, so there is less to recover.

⚠️ Requires a patched mlx-lm with the glm_moe_dsa correctness fixes and int8 MLA-KV support.

This build uses IndexShare — the DSA indexer is stored only on the 21 "full" layers ([0,1,2,6,10,…,74]); the other 57 layers reuse it. Stock mlx-lm / omlx build an indexer on every layer, so loading fails with ValueError: Missing 285 parameters: …self_attn.indexer… (57 shared layers × 5 params). Use avlp12/mlx-lm, branch glm-5.2-dsa-inference (upstream PR ml-explore/mlx-lm#1419, open as of writing), which adds IndexShare + non-interleaved indexer RoPE + int8 MLA-KV. Serve with mlx_lm.server from that fork. On stock mlx-lm, --kv-bits is also a silent no-op for this architecture (see Correctness and Long context).

---

Quality

A 2-bit (≤256 GB) build of this model is bit-starved; lifting the budget to 512 GB and spending 3.5 bpw is the single biggest quality lever.

	this build (3.5 bpw)	2.56 bpw build	Δ
wikitext-2 PPL (prose)	2.851	3.774	−24%
code PPL (mlx_lm src)	1.841	2.069	−11%

Strided perplexity (ctx 2048 / stride 1024) from a fixed local harness — relative numbers for comparing builds, not directly comparable to perplexities other quantizers report on different corpora.

Base model	zai-org/GLM-5.2 (744B total / ~40B active)
Bits/weight	~3.535 (per-tensor mixed)
On-disk size	~333 GB (310 GiB) — incl. the 4.5 GB native MTP layer
Peak memory (load + short gen)	~331 GB (+4.5 GB with MTP attached)
Peak memory (1M ctx, int8 KV)	~376 GB (+4.5 GB with MTP attached)
Format	MLX (Apple Silicon)
Context	up to 1M tokens (DSA sparse attention)
Speculative decoding	native MTP (nextn layer 78 included)

---

Recipe

Bits are allocated by sensitivity — cheap bits on the robust expert bulk, full precision on the discrete routing/sparse-attention control paths.

Component	Bits	Why
Routed experts (gate/up/down)	3-bit g64	~96% of params — the bulk
Shared experts · MLA attn (incl. kv_a/q_a) · dense MLP	4-bit g64	on every token's critical path
Token embedding · LM head	6-bit g64	distribution-sensitive
Router (mlp.gate)	bf16	drives discrete top-8 routing — never quantized
DSA lightning indexer	fp16	drives discrete top-k selection

---

Long context (1M) — int8 KV

GLM-5.2's MLA stores the compressed latent (kv_lora 512 + rope 64 per layer), so the 1M-token KV cache is small: ~95 GB at fp16, ~48 GB at int8. Quantizing the latent cache to int8 is what makes 1M fit comfortably at 3.5 bpw.

Serve with int8 KV:

mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-3.5bpw \
  --kv-bits 8 --kv-group-size 64 --quantized-kv-start 4096 \
  --prompt "…"

# OpenAI-compatible server
mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-3.5bpw \
  --kv-bits 8 --quantized-kv-start 4096

The patched runtime quantizes only the MLA latent cache to int8 (the DSA indexer cache stays fp16) and dequantizes the latent on read inside MLA attention. At ≤128K context, fp16 KV is fine (a few GB); int8 is only needed to keep 1M comfortable.

Why int8 and not fp8? A common suggestion is to switch the KV latent to fp8 for long-context quality. Measured on the actual latent, that would hurt: the MLA latent is post-kv_a_layernorm (RMSNorm), so its distribution is tight and well-conditioned (measured std ≈ 0.006, max ≈ 0.03). Reconstruction fidelity on the real captured latent — int8-affine (g64) cos 0.99998 vs MXFP8 (g32) 0.9997 vs plain fp8 e4m3 0.994 — int8 is 100–200× lower MSE on both the latent and the decoupled RoPE key. fp8's wider dynamic range doesn't help a narrow, well-conditioned distribution where precision (mantissa bits) is the binding constraint, and MLX dequantizes the latent to fp16 for the MLA matmul anyway (no native fp8 GEMM on this path, so no throughput win either). At the same ~1-byte footprint, int8 is the better choice for long-context fidelity — hence this build keeps int8.

---

Native MTP — self-speculative decoding

GLM-5.2 ships a built-in MTP (multi-token-prediction / "nextn") layer that predicts token t+2 from the backbone's hidden state at t — DeepSeek-V3-style. Every public MLX build (including earlier revisions of this one) stripped it at conversion; this build restores it as model.layers.78.* (one extra shard, +4.51 GB), quantized to match the rest of the recipe (experts 3-bit g64, attention 4-bit, indexer/norms/eh_proj bf16 — measured: drafter precision does not affect acceptance, bf16 ≡ 3-bit).

Backward compatible: loaders without MTP support drop layer 78 in sanitize() and behave byte-identically to the previous revision. With the fork, --mtp turns it on:

mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-3.5bpw \
  --mtp --prompt "…"                        # k=2 chained drafts (default)

mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-3.5bpw \
  --mtp --mtp-num-draft-tokens 2            # OpenAI-compatible serving
  # optional: --mtp-hybrid  (adds conservative prompt-lookup drafting
  #            for repetition-heavy workloads: long quotes, JSON, boilerplate)

Measured on this build (M3 Ultra 512 GB, single request, greedy, short context; these MTP figures predate the 2026-07 DWQ retune — plain decode is format-identical under DWQ, and single-request MTP stays roughly neutral on the cleaner DWQ'd target, where draft acceptance holds or improves slightly):

	plain	--mtp (k=2)
decode	20.4 tok/s	20.7 tok/s (~+2%)
mean accepted length	—	~2.3 tokens/step

Set expectations accordingly: on this 3-bit-expert build, single-request MTP is roughly performance-neutral — the aggressively-quantized target accepts fewer drafts (its own outputs are noisier and harder to predict) and each verify pass streams extra MoE experts, which offsets the drafting gain. A 4-bit sibling of this quant measured +11% (22.0 → 24.6 tok/s, accept-len 2.46) with the identical code path, and the gain grows further with target precision and with multi-node serving (speculation amortizes fixed per-forward costs). It stays exactly lossless — at temperature 0 the output is the model's own greedy output; at temperature > 0 drafts are accepted only when they equal the target's sampled token (distribution-exact) — so enabling it costs nothing but the ~4.5 GB layer.

Long context: MTP works past index_topk (2048 tokens — the DSA-sparse regime). Draft acceptance holds there, and the fork's small-L gather path (commit 946c412) keeps the multi-token verify from paying the sparse-mask setup per iteration (before that fix, long-context MTP decoded at 0.26× plain). Measured on a 4-bit sibling at ~2.1K context, prefill-separated: --mtp k=2 ≈ 0.85× plain, k=1 ≈ 0.98× — so at very long contexts prefer --mtp-num-draft-tokens 1 (≈ neutral) or plain; the k=2 gain (+15%) lives in the ≤2048 dense regime. The remaining long-context gap is acceptance economics (accepted length ~1.8 vs ~2.1 break-even), not path cost.

Notes for other integrators (the three footguns, all verified against vLLM's deepseek_mtp semantics):

1. eh_proj input concat order is [enorm(embed(t+1)), hnorm(h_t)] — embedding first.
2. The hidden fed to the MTP is the backbone's pre-final-norm residual stream.
3. For chained multi-token drafting, feed back the MTP's shared_head.norm-normalized hidden — chaining the raw layer output halves chained acceptance (measured 0.27 → 0.75 conditional at position 2).

---

How it compares

Against other public GLM-5.2 MLX builds, this one is the smallest on disk and the only one that runs a full 1M-token context within 512 GB — it spends its budget on context headroom + an int8 MLA-KV cache instead of purely on weight bits. (For the author's own higher-quality tier, see the 4.5 bpw nvfp4 sibling, compared in detail on its card.)

	this build	mixed-3_6	Q4.8-INF	DQ4plus-q8
effective bpw	~3.5	~3.6	~4.8	~5.0
on-disk	333 GB¹	~360 GB	447 GB	465 GB
native MTP layer	✓ included	✗ stripped	✗ stripped	✗ stripped
DSA indexer	fp16	6-bit	(custom fmt)	8-bit
int8 MLA-KV	yes	no	no	no
1M ctx in 512 GB	✓ 136 GB free	tight	✗ over	✗ over

Per-component bit allocation was parsed from each build's config.json. The higher-bit builds (4.8–5.0 bpw) likely carry higher raw weight fidelity, but their footprint cannot hold a 1M KV cache on 512 GB. This build also keeps the DSA indexer at fp16 (the others quantize it) and bakes the indexer RoPE/eps long-context fixes.

¹ 328 GB base + 4.5 GB optional MTP layer (stripped automatically by runtimes without MTP support; the memory charts above show the base build — add +4.5 GB when serving with --mtp).

---

Benchmarks

Reproduced with mlx_lm.evaluate (0-shot) and mlx_lm.perplexity (seq 2048, 50 samples, seed 123), against the author's earlier GLM-5.1 quant under the same harness and settings:

	GLM-5.1 · 2.7 bpw	GLM-5.2 · 2.56 bpw	GLM-5.2 · 3.5 bpw (this)
Perplexity (lower)	4.165	3.564	3.603
HellaSwag (acc_norm)	0.606	0.652	0.610
PIQA (acc)	0.796	0.808	0.826
WinoGrande (acc)	0.660	0.780	0.770
Generation (tok/s)	18.35	22.87	21.29

Perplexity here is on allenai/tulu-3-sft-mixture (the mlx_lm.perplexity default) — a different corpus and method from the wikitext strided figure in Quality above, so values are not comparable across the two sections. Task accuracies use a 500-sample limit (CI ±0.02–0.04). GLM-5.1 is a different (older) base model, so cross-generation gaps reflect both the newer model and quantization.

---

Correctness (verified vs the HF reference)

glm_moe_dsa needs fixes beyond the stock mlx-lm port; this build was produced with a patched fork and validated:

• IndexShare — the DSA indexer runs only on "full" layers; "shared" layers reuse its top-k (index_topk_freq=4). The stock port built an indexer on every layer → missing-weights / wrong >2048-token output.
• Indexer RoPE / eps — non-interleaved (half-split) RoPE + LayerNorm eps 1e-6, distinct from the interleaved main attention. Baked into config.json (indexer_rope_traditional=false, indexer_norm_eps=1e-6); post-RoPE q matches the reference to ~1e-7.
• int8 MLA-KV — CacheList.to_quantized/offset + MLA dequant-on-read, so --kv-bits 8 actually engages for the latent cache (silently ignored on stock mlx-lm).
• Native MTP — layer 78 restored + --mtp self-speculative decoding in the fork; greedy output verified identical to non-speculative decoding, chained drafting matches vLLM's deepseek_mtp semantics (normed-hidden chaining).

Validation: full-attention logits match the HF reference to float precision at ≤index_topk context; long-context needle retrieval in the sparse-DSA regime; coherent prose + code generation; int8-KV verified coherent past the quantization threshold; peak memory measured.

---

Usage

# requires mlx-lm with the glm_moe_dsa fixes + int8 MLA-KV patch
mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-3.5bpw \
  --prompt "Write a quicksort in Python."

Hardware

Built for 512 GB Apple Silicon (M3 Ultra). Weights ~328 GB; with int8 KV a 1M-token context runs in ~376 GB, leaving comfortable headroom for the OS and other apps. For ≤256 GB machines, use the 2.56 bpw build instead.

Credits

• Base model: Zhipu / Z.ai — GLM-5.2 (MIT).
• MLX & mlx-lm: Apple ml-explore.
• Mixed-precision quantization, glm_moe_dsa correctness fixes, int8 MLA-KV cache support, and native-MTP restoration + speculative serving: Alis (avlp12).

Citation

Alis (avlp12) (2026). GLM-5.2-Alis-MLX-Dynamic-3.5bpw — 3.5 bpw MLX quantization of GLM-5.2. https://huggingface.co/avlp12/GLM-5.2-Alis-MLX-Dynamic-3.5bpw