---
language:
- en
- zh
license: mit
library_name: mlx
pipeline_tag: text-generation
tags:
- glm_moe_dsa
- quantized
- glm
- moe
- apple-silicon
- mixed-precision
- 2-bit
- speculative-decoding
- mtp
- conversational
base_model: zai-org/GLM-5.2
---

# GLM-5.2-Alis-MLX-Dynamic-2.56bpw

Apple Silicon (MLX) **mixed-precision** quantization of [zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) — a 744B-parameter (\~40B active) Mixture-of-Experts model with DeepSeek-V3.2-style MLA + DeepSeek Sparse Attention (DSA, `glm_moe_dsa`). Quantized to **\~2.56 bits/weight** so the full model runs in **≤256 GB of unified memory**.

> ⚠️ **Requires a patched `mlx-lm`** with the `glm_moe_dsa` indexer fixes (see *Correctness* below). The stock port is incomplete for GLM-5.2; loading there fails or degrades long-context output.

**New:** the checkpoint now ships GLM-5.2's **native MTP (nextn) layer** for self-speculative decoding (`--mtp` — see [*Native MTP*](#native-mtp--self-speculative-decoding)). Fully backward-compatible: runtimes without MTP support strip the extra layer and behave exactly as before.

**Update (2026-07): DWQ-retuned weights.** `main` now carries quantization scales/biases re-tuned with **layerwise DWQ** ([alis-dwq](https://github.com/avlp12/alis-dwq); upstream flag: [mlx-lm#1499](https://github.com/ml-explore/mlx-lm/pull/1499)) against the [4.5 bpw sibling](https://huggingface.co/avlp12/GLM-5.2-Alis-MLX-Dynamic-4.5bpw) as teacher, with a 45%-ZH calibration mix — low-bit damage concentrates in the model's Chinese mass, and this targets exactly that. Same recipe, size class and speed; distributional fidelity jumps:

| KL / top-1 flip vs the 4.5 bpw reference (T=3072) | pre-DWQ | **DWQ (main)** |
|---|---|---|
| EN | 0.727 / 24.9% | **0.383 / 15.6%** |
| code | 0.252 / 12.7% | **0.193 / 10.2%** |
| ZH | 0.987 / 35.7% | **0.562 / 21.9%** |
| overall | 0.655 / 24.4% | **0.379 / 15.9% (−42% KL)** |

The pre-DWQ weights remain available on the [`pre-dwq`](https://huggingface.co/avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw/tree/pre-dwq) branch. **The *Quality* and *Benchmarks* figures below are measured on these DWQ weights** — the retune also moves perplexity (strided wikitext 4.34 → 3.77, code 2.20 → 2.07; `mlx_lm.perplexity` on tulu-3 3.85 → 3.56), not just the KL above.

## Metrics

| | |
|---|---|
| Base model | zai-org/GLM-5.2 (744B total / \~40B active) |
| Bits/weight | **\~2.56** (per-tensor mixed) |
| On-disk size | **246.9 GB** (48 shards, incl. the 4.5 GB native MTP layer) |
| Peak memory | \~242 GB (weights, MTP-attached) · \~249 GB (8K) · \~263 GB (26K) · \~293 GB (64K) · \~344 GB (128K) — **measured** (int8 KV); the DSA prefill activation grows with context |
| Format | MLX (Apple Silicon) |
| Context | 1M-capable architecture (DSA); **machine-limited in practice** — ≈26–32K prefill on a 256 GiB box (see *Long context & memory*) |
| Speculative decoding | native MTP (nextn layer 78 included) |

## Why this model

GLM-5.2 is a frontier agentic-coding MoE, but at 744B it is \~1.5 TB in bf16 — out of reach for consumer memory, and existing MLX builds start at \~360 GB (≥4-bit, 512 GB-class machines). This build uses **Unsloth-style per-tensor mixed precision**: the routed experts (\~97% of params) go to 2-bit while the sensitive paths keep higher precision, landing **under 256 GB** while preserving long-context retrieval and coding quality.

![On-disk footprint across GLM-5.2 MLX builds: this 2.56 bpw build (238 GB) is the only one that fits a ≤256 GB machine; golden 328, mixed-3_6 360, Q4.8-INF 447, DQ4plus 465 all need 512 GB-class](assets/field.png)

## Quality

This is the **≤256 GB** option — the routed experts are 2-bit, so it is deliberately bit-starved. If you have a 512 GB machine, the **[3.5 bpw build](https://huggingface.co/avlp12/GLM-5.2-Alis-MLX-Dynamic-3.5bpw)** is still better on prose/code strided PPL (−24% wikitext, −11% code; the 3.5 bpw sibling is DWQ-retuned as well) and runs a full 1M context.

![Perplexity: this 2.56 bpw build vs the 3.5 bpw build — 2.56 bpw is 24% higher (worse) on wikitext, 11% on code](assets/quality.png)

*Strided perplexity from a fixed local harness — relative numbers for comparing these two builds, not directly comparable to perplexities other quantizers report on different corpora.*

## Benchmarks

Reproduced with `mlx_lm.evaluate` (0-shot) and `mlx_lm.perplexity` (seq 2048, 50 samples, seed 123), against the author's earlier GLM-5.1 quant under the same harness and settings:

| | GLM-5.1 · 2.7 bpw | **GLM-5.2 · 2.56 bpw (this)** | GLM-5.2 · 3.5 bpw |
|---|---|---|---|
| Perplexity (lower) | 4.165 | **3.564** | 3.603 |
| HellaSwag (acc_norm) | 0.606 | **0.652** | 0.610 |
| PIQA (acc) | 0.796 | **0.808** | 0.826 |
| WinoGrande (acc) | 0.660 | **0.780** | 0.770 |
| Generation (tok/s) | 18.35 | **22.87** | 21.29 |

Perplexity here is on `allenai/tulu-3-sft-mixture` (the `mlx_lm.perplexity` default) — a different corpus and method from the wikitext strided figure above, so values are not comparable across the two. Task accuracies use a 500-sample limit (CI ±0.02–0.04). GLM-5.1 is a different (older) base model, so cross-generation gaps reflect both the newer model and quantization.

**Quantization recipe**

![Mixed-precision recipe: experts 2-bit, MLA/shared/dense 4-bit, embed/head 6-bit, router bf16, indexer fp16](assets/recipe.png)

| Component | Bits | Notes |
|---|---|---|
| Routed experts (gate/up/down) | 2-bit g64 | \~96% of params — the bulk |
| MLA attn · shared experts · dense MLP | 4-bit g64 | per-token critical path |
| Token embedding · LM head | 6-bit g64 | distribution-sensitive |
| Router (`mlp.gate`) | bf16 | drives discrete top-8 routing |
| DSA lightning indexer | fp16 | drives discrete top-k selection |

## Native MTP — self-speculative decoding

GLM-5.2 ships a **built-in MTP (multi-token-prediction / "nextn") layer** that predicts token *t+2* from the backbone's hidden state at *t* — DeepSeek-V3-style. Public MLX builds (including earlier revisions of this one) stripped it at conversion; this build **restores it as `model.layers.78.*`** (one extra shard, +4.51 GB; experts 3-bit g64, attention 4-bit, indexer/norms/`eh_proj` bf16 — measured on the 3.5 bpw sibling: drafter precision does not affect acceptance, bf16 ≡ 3-bit).

**Backward compatible:** loaders without MTP support drop layer 78 in `sanitize()` and behave **byte-identically** to the previous revision. With the [fork](https://github.com/avlp12/mlx-lm/tree/glm-5.2-dsa-inference), `--mtp` turns it on:

```bash
mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw \
  --mtp --prompt "…"                        # k=2 chained drafts (default)

mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw \
  --mtp --mtp-num-draft-tokens 2            # OpenAI-compatible serving
  # optional: --mtp-hybrid  (adds conservative prompt-lookup drafting
  #            for repetition-heavy workloads: long quotes, JSON, boilerplate)
```

Reference numbers (M3 Ultra 512 GB, single request, greedy): the [3.5 bpw sibling](https://huggingface.co/avlp12/GLM-5.2-Alis-MLX-Dynamic-3.5bpw) measures **~performance-neutral** (20.4 → 20.7 tok/s, accept-len ~2.3) and a 4-bit sibling measured **+11%** (22.0 → 24.6, accept-len 2.46) — draft acceptance drops as the target is quantized harder, because a noisier target's own outputs are harder for any drafter to predict. On this 2-bit-expert build expect **neutral-at-best single-request speed** (not yet benchmarked end-to-end here); the layer stays worth shipping because it is exactly lossless, and gains grow with target precision and multi-node serving.

**Lossless:** at temperature 0 the output is the model's own greedy output; at temperature > 0 drafts are accepted only when they equal the target's sampled token (distribution-exact).

**Long context:** MTP works past `index_topk` (2048 tokens — the DSA-sparse regime). Draft acceptance holds there, and the fork's **small-L gather path** (commit `946c412`) keeps the multi-token verify from paying the sparse-mask setup per iteration (before that fix, long-context MTP decoded at 0.26× plain). Measured on a 4-bit sibling at ~2.1K context: `--mtp` k=2 ≈ 0.85× plain, k=1 ≈ 0.98× — at very long contexts prefer `--mtp-num-draft-tokens 1` or plain; the k=2 gain lives in the ≤2048 dense regime.

**Memory note:** the MTP-capable fork attaches the module at load (+4.5 GB) even without `--mtp`, trimming the KV+activation budget by that much. To reclaim it, set `"num_nextn_predict_layers": 0` in a local `config.json` copy (or use a runtime without MTP support, which strips the layer). See *Long context & memory* for the machine-level ceiling.

**Notes for other integrators** (verified against vLLM's `deepseek_mtp` semantics): `eh_proj` concat order is `[enorm(embed(t+1)), hnorm(h_t)]`; the hidden fed to the MTP is the backbone's **pre-final-norm** residual stream; for chained drafting feed back the **`shared_head.norm`-normalized** hidden (chaining the raw layer output halves chained acceptance — measured 0.27 → 0.75 conditional at position 2).

## Long context & memory

GLM-5.2's MLA stores a **compressed latent** KV (kv_lora 512 + rope 64 per layer), so the KV cache itself is small — **\~88 KB/token at fp16, \~44 KB/token with int8**. Quantize it with `--kv-bits` (the patched fork engages int8 on the MLA latent; stock `mlx-lm` silently ignores it):

```bash
mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw \
  --kv-bits 8 --kv-group-size 64 --quantized-kv-start 4096 --prompt "…"

# OpenAI-compatible server
mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw \
  --kv-bits 8 --quantized-kv-start 4096
```

**On a 256 GiB machine the KV is *not* the binding constraint — the DSA prefill activation is.** Measured peak on this build (M3 Ultra, int8 KV, MTP-attached weights ≈ 242 GB):

| prompt (prefill) | peak | fits 256 GiB (274.9 GB)? |
|---|---|---|
| 8K | \~249 GB | ✓ |
| 26K | \~263 GB | ✓ (tight, \~12 GB headroom) |
| 32K | \~268 GB | ✓ (very tight) |
| 64K | \~293 GB | ✗ |
| 128K | \~344 GB | ✗ |

The DSA lightning indexer scores every past token per prefill chunk, so the activation peak climbs \~+20 GB per \~30K tokens — **independent of KV bits** (int8 vs fp16 differed by only \~1 GB at 26K, because the KV is tiny there). So on a **256 GiB box, keep prompts ≤ \~26–32K**. `--kv-bits 8` still helps where the *KV* is what grows — long **decode**/accumulated context, and **512 GiB** machines with headroom for longer prefills. For genuinely long (100K–1M) context, use the **[3.5 bpw build on a 512 GiB machine](https://huggingface.co/avlp12/GLM-5.2-Alis-MLX-Dynamic-3.5bpw)**.

## Correctness (verified vs the HF reference)

GLM-5.2's `glm_moe_dsa` needed fixes beyond the stock mlx-lm port; this build was produced with a patched fork and validated:

- **IndexShare** — the DSA indexer runs only on "full" layers; "shared" layers reuse its top-k (`index_topk_freq=4`). The stock port built an indexer on every layer → missing-weights / wrong >2048-token output.
- **Indexer RoPE/eps** — the indexer uses **non-interleaved (half-split) RoPE + LayerNorm eps 1e-6**, distinct from the interleaved main attention. Post-RoPE `q` matches the HF reference to \~1e-7. Recorded in `config.json` (`indexer_rope_traditional=false`, `indexer_norm_eps=1e-6`).
- **Native MTP** — layer 78 restored + `--mtp` self-speculative decoding in the fork; greedy output verified identical to non-speculative decoding (on the 3.5 bpw sibling), chained drafting matches vLLM's `deepseek_mtp` semantics (normed-hidden chaining).

**Validation:** full-attention logits match the HF reference to float precision at ≤index_topk context; **needle retrieval succeeds through a 7,586-token prompt** (sparse-DSA regime); coherent code generation; peak ≤256 GB.

## Usage

```bash
# requires mlx-lm with the glm_moe_dsa indexer fixes
mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw \
  --prompt "Write a quicksort in Python."

# OpenAI-compatible server
mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw

# with native MTP speculative decoding (see the Native MTP section)
mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw --mtp
```

## Hardware

Runs in **≤256 GB unified memory** (Apple Silicon). A "256 GB" Mac is **256 GiB = 274.9 GB** (not 256 decimal); the \~242 GB of (MTP-attached) weights leave \~33 GB. But the DSA sparse-attention **prefill activation grows with context** (measured +\~20 GB at 26K, +\~50 GB at 64K), so the **practical prefill ceiling on a 256 GiB machine is \~26–32K tokens** (peak \~263–268 GB) — not the architecture's 1M. For 100K+ context use a 512 GB (512 GiB) machine. See *Long context & memory* below.

## Credits

- Base model: **Zhipu / Z.ai — GLM-5.2** (MIT).
- **MLX** & **mlx-lm**: Apple ml-explore.
- Mixed-precision quantization, `glm_moe_dsa` correctness fixes, and native-MTP restoration + speculative serving: **Alis (avlp12)**.

## Citation

> **Alis (avlp12)** (2026). *GLM-5.2-Alis-MLX-Dynamic-2.56bpw* — 2.56 bpw MLX quantization of [GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) for ≤256 GB Apple Silicon.
> <https://huggingface.co/avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw>