GLM-5.2-Alis-MLX-Dynamic-2.56bpw

Apple Silicon (MLX) mixed-precision quantization of zai-org/GLM-5.2 — a 744B-parameter (~40B active) Mixture-of-Experts model with DeepSeek-V3.2-style MLA + DeepSeek Sparse Attention (DSA, glm_moe_dsa). Quantized to ~2.56 bits/weight so the full model runs in ≤256 GB of unified memory.

⚠️ Requires a patched mlx-lm with the glm_moe_dsa indexer fixes (see Correctness below). The stock port is incomplete for GLM-5.2; loading there fails or degrades long-context output.

New: the checkpoint now ships GLM-5.2's native MTP (nextn) layer for self-speculative decoding (--mtp — see Native MTP). Fully backward-compatible: runtimes without MTP support strip the extra layer and behave exactly as before.

Update (2026-07): DWQ-retuned weights. main now carries quantization scales/biases re-tuned with layerwise DWQ (alis-dwq; upstream flag: mlx-lm#1499) against the 4.5 bpw sibling as teacher, with a 45%-ZH calibration mix — low-bit damage concentrates in the model's Chinese mass, and this targets exactly that. Same recipe, size class and speed; distributional fidelity jumps:

KL / top-1 flip vs the 4.5 bpw reference (T=3072)	pre-DWQ	DWQ (main)
EN	0.727 / 24.9%	0.383 / 15.6%
code	0.252 / 12.7%	0.193 / 10.2%
ZH	0.987 / 35.7%	0.562 / 21.9%
overall	0.655 / 24.4%	0.379 / 15.9% (−42% KL)

The pre-DWQ weights remain available on the pre-dwq branch. The Quality and Benchmarks figures below are measured on these DWQ weights — the retune also moves perplexity (strided wikitext 4.34 → 3.77, code 2.20 → 2.07; mlx_lm.perplexity on tulu-3 3.85 → 3.56), not just the KL above.

Metrics

Base model	zai-org/GLM-5.2 (744B total / ~40B active)
Bits/weight	~2.56 (per-tensor mixed)
On-disk size	246.9 GB (48 shards, incl. the 4.5 GB native MTP layer)
Peak memory	~242 GB (weights, MTP-attached) · ~249 GB (8K) · ~263 GB (26K) · ~293 GB (64K) · ~344 GB (128K) — measured (int8 KV); the DSA prefill activation grows with context
Format	MLX (Apple Silicon)
Context	1M-capable architecture (DSA); machine-limited in practice — ≈26–32K prefill on a 256 GiB box (see Long context & memory)
Speculative decoding	native MTP (nextn layer 78 included)

Why this model

GLM-5.2 is a frontier agentic-coding MoE, but at 744B it is ~1.5 TB in bf16 — out of reach for consumer memory, and existing MLX builds start at ~360 GB (≥4-bit, 512 GB-class machines). This build uses Unsloth-style per-tensor mixed precision: the routed experts (~97% of params) go to 2-bit while the sensitive paths keep higher precision, landing under 256 GB while preserving long-context retrieval and coding quality.

Quality

This is the ≤256 GB option — the routed experts are 2-bit, so it is deliberately bit-starved. If you have a 512 GB machine, the 3.5 bpw build is still better on prose/code strided PPL (−24% wikitext, −11% code; the 3.5 bpw sibling is DWQ-retuned as well) and runs a full 1M context.

Strided perplexity from a fixed local harness — relative numbers for comparing these two builds, not directly comparable to perplexities other quantizers report on different corpora.

Benchmarks

Reproduced with mlx_lm.evaluate (0-shot) and mlx_lm.perplexity (seq 2048, 50 samples, seed 123), against the author's earlier GLM-5.1 quant under the same harness and settings:

	GLM-5.1 · 2.7 bpw	GLM-5.2 · 2.56 bpw (this)	GLM-5.2 · 3.5 bpw
Perplexity (lower)	4.165	3.564	3.603
HellaSwag (acc_norm)	0.606	0.652	0.610
PIQA (acc)	0.796	0.808	0.826
WinoGrande (acc)	0.660	0.780	0.770
Generation (tok/s)	18.35	22.87	21.29

Perplexity here is on allenai/tulu-3-sft-mixture (the mlx_lm.perplexity default) — a different corpus and method from the wikitext strided figure above, so values are not comparable across the two. Task accuracies use a 500-sample limit (CI ±0.02–0.04). GLM-5.1 is a different (older) base model, so cross-generation gaps reflect both the newer model and quantization.

Quantization recipe

Component	Bits	Notes
Routed experts (gate/up/down)	2-bit g64	~96% of params — the bulk
MLA attn · shared experts · dense MLP	4-bit g64	per-token critical path
Token embedding · LM head	6-bit g64	distribution-sensitive
Router (mlp.gate)	bf16	drives discrete top-8 routing
DSA lightning indexer	fp16	drives discrete top-k selection

Native MTP — self-speculative decoding

GLM-5.2 ships a built-in MTP (multi-token-prediction / "nextn") layer that predicts token t+2 from the backbone's hidden state at t — DeepSeek-V3-style. Public MLX builds (including earlier revisions of this one) stripped it at conversion; this build restores it as model.layers.78.* (one extra shard, +4.51 GB; experts 3-bit g64, attention 4-bit, indexer/norms/eh_proj bf16 — measured on the 3.5 bpw sibling: drafter precision does not affect acceptance, bf16 ≡ 3-bit).

Backward compatible: loaders without MTP support drop layer 78 in sanitize() and behave byte-identically to the previous revision. With the fork, --mtp turns it on:

mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw \
  --mtp --prompt "…"                        # k=2 chained drafts (default)

mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw \
  --mtp --mtp-num-draft-tokens 2            # OpenAI-compatible serving
  # optional: --mtp-hybrid  (adds conservative prompt-lookup drafting
  #            for repetition-heavy workloads: long quotes, JSON, boilerplate)

Reference numbers (M3 Ultra 512 GB, single request, greedy): the 3.5 bpw sibling measures ~performance-neutral (20.4 → 20.7 tok/s, accept-len ~2.3) and a 4-bit sibling measured +11% (22.0 → 24.6, accept-len 2.46) — draft acceptance drops as the target is quantized harder, because a noisier target's own outputs are harder for any drafter to predict. On this 2-bit-expert build expect neutral-at-best single-request speed (not yet benchmarked end-to-end here); the layer stays worth shipping because it is exactly lossless, and gains grow with target precision and multi-node serving.

Lossless: at temperature 0 the output is the model's own greedy output; at temperature > 0 drafts are accepted only when they equal the target's sampled token (distribution-exact).

Long context: MTP works past index_topk (2048 tokens — the DSA-sparse regime). Draft acceptance holds there, and the fork's small-L gather path (commit 946c412) keeps the multi-token verify from paying the sparse-mask setup per iteration (before that fix, long-context MTP decoded at 0.26× plain). Measured on a 4-bit sibling at ~2.1K context: --mtp k=2 ≈ 0.85× plain, k=1 ≈ 0.98× — at very long contexts prefer --mtp-num-draft-tokens 1 or plain; the k=2 gain lives in the ≤2048 dense regime.

Memory note: the MTP-capable fork attaches the module at load (+4.5 GB) even without --mtp, trimming the KV+activation budget by that much. To reclaim it, set "num_nextn_predict_layers": 0 in a local config.json copy (or use a runtime without MTP support, which strips the layer). See Long context & memory for the machine-level ceiling.

Notes for other integrators (verified against vLLM's deepseek_mtp semantics): eh_proj concat order is [enorm(embed(t+1)), hnorm(h_t)]; the hidden fed to the MTP is the backbone's pre-final-norm residual stream; for chained drafting feed back the shared_head.norm-normalized hidden (chaining the raw layer output halves chained acceptance — measured 0.27 → 0.75 conditional at position 2).

Long context & memory

GLM-5.2's MLA stores a compressed latent KV (kv_lora 512 + rope 64 per layer), so the KV cache itself is small — ~88 KB/token at fp16, ~44 KB/token with int8. Quantize it with --kv-bits (the patched fork engages int8 on the MLA latent; stock mlx-lm silently ignores it):

mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw \
  --kv-bits 8 --kv-group-size 64 --quantized-kv-start 4096 --prompt "…"

# OpenAI-compatible server
mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw \
  --kv-bits 8 --quantized-kv-start 4096

On a 256 GiB machine the KV is not the binding constraint — the DSA prefill activation is. Measured peak on this build (M3 Ultra, int8 KV, MTP-attached weights ≈ 242 GB):

prompt (prefill)	peak	fits 256 GiB (274.9 GB)?
8K	~249 GB	✓
26K	~263 GB	✓ (tight, ~12 GB headroom)
32K	~268 GB	✓ (very tight)
64K	~293 GB	✗
128K	~344 GB	✗

The DSA lightning indexer scores every past token per prefill chunk, so the activation peak climbs ~+20 GB per ~30K tokens — independent of KV bits (int8 vs fp16 differed by only ~1 GB at 26K, because the KV is tiny there). So on a 256 GiB box, keep prompts ≤ ~26–32K. --kv-bits 8 still helps where the KV is what grows — long decode/accumulated context, and 512 GiB machines with headroom for longer prefills. For genuinely long (100K–1M) context, use the 3.5 bpw build on a 512 GiB machine.

Correctness (verified vs the HF reference)

GLM-5.2's glm_moe_dsa needed fixes beyond the stock mlx-lm port; this build was produced with a patched fork and validated:

• IndexShare — the DSA indexer runs only on "full" layers; "shared" layers reuse its top-k (index_topk_freq=4). The stock port built an indexer on every layer → missing-weights / wrong >2048-token output.
• Indexer RoPE/eps — the indexer uses non-interleaved (half-split) RoPE + LayerNorm eps 1e-6, distinct from the interleaved main attention. Post-RoPE q matches the HF reference to ~1e-7. Recorded in config.json (indexer_rope_traditional=false, indexer_norm_eps=1e-6).
• Native MTP — layer 78 restored + --mtp self-speculative decoding in the fork; greedy output verified identical to non-speculative decoding (on the 3.5 bpw sibling), chained drafting matches vLLM's deepseek_mtp semantics (normed-hidden chaining).

Validation: full-attention logits match the HF reference to float precision at ≤index_topk context; needle retrieval succeeds through a 7,586-token prompt (sparse-DSA regime); coherent code generation; peak ≤256 GB.

Usage

# requires mlx-lm with the glm_moe_dsa indexer fixes
mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw \
  --prompt "Write a quicksort in Python."

# OpenAI-compatible server
mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw

# with native MTP speculative decoding (see the Native MTP section)
mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw --mtp

Hardware

Runs in ≤256 GB unified memory (Apple Silicon). A "256 GB" Mac is 256 GiB = 274.9 GB (not 256 decimal); the ~242 GB of (MTP-attached) weights leave ~33 GB. But the DSA sparse-attention prefill activation grows with context (measured +~20 GB at 26K, +~50 GB at 64K), so the practical prefill ceiling on a 256 GiB machine is ~26–32K tokens (peak ~263–268 GB) — not the architecture's 1M. For 100K+ context use a 512 GB (512 GiB) machine. See Long context & memory below.

Credits

• Base model: Zhipu / Z.ai — GLM-5.2 (MIT).
• MLX & mlx-lm: Apple ml-explore.
• Mixed-precision quantization, glm_moe_dsa correctness fixes, and native-MTP restoration + speculative serving: Alis (avlp12).

Citation

Alis (avlp12) (2026). GLM-5.2-Alis-MLX-Dynamic-2.56bpw — 2.56 bpw MLX quantization of GLM-5.2 for ≤256 GB Apple Silicon. https://huggingface.co/avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw