Add int8 KV usage + Long-context/memory section; correct 256GB->256GiB(274.9GB) and add measured prefill-peak curve (DSA activation caps ~26-32K prompt on 256 GiB)

README.md

@@ -45,9 +45,9 @@ The pre-DWQ weights remain available on the [`pre-dwq`](https://huggingface.co/a
 | Base model | zai-org/GLM-5.2 (744B total / \~40B active) |
 | Bits/weight | **\~2.56** (per-tensor mixed) |
 | On-disk size | **246.9 GB** (48 shards, incl. the 4.5 GB native MTP layer) |
-| Peak memory | \~238 GB (short ctx) · \~245 GB (8K ctx) · +4.5 GB with MTP attached |
+| Peak memory | \~242 GB (weights, MTP-attached) · \~249 GB (8K) · \~263 GB (26K) · \~293 GB (64K) · \~344 GB (128K) — **measured** (int8 KV); the DSA prefill activation grows with context |
 | Format | MLX (Apple Silicon) |
-| Context | up to 1M tokens (DSA sparse attention) |
+| Context | 1M-capable architecture (DSA); **machine-limited in practice** — ≈26–32K prefill on a 256 GiB box (see *Long context & memory*) |
 | Speculative decoding | native MTP (nextn layer 78 included) |
 
 ## Why this model
@@ -112,10 +112,35 @@ Reference numbers (M3 Ultra 512 GB, single request, greedy): the [3.5 bpw siblin
 
 **Long context:** MTP works past `index_topk` (2048 tokens — the DSA-sparse regime). Draft acceptance holds there, and the fork's **small-L gather path** (commit `946c412`) keeps the multi-token verify from paying the sparse-mask setup per iteration (before that fix, long-context MTP decoded at 0.26× plain). Measured on a 4-bit sibling at ~2.1K context: `--mtp` k=2 ≈ 0.85× plain, k=1 ≈ 0.98× — at very long contexts prefer `--mtp-num-draft-tokens 1` or plain; the k=2 gain lives in the ≤2048 dense regime.
 
-**Memory note for 256 GB machines:** the MTP-capable fork attaches the module at load (+4.5 GB) even without `--mtp`. On a tight 256 GB box that shrinks short-context headroom from ~18 GB to ~13.5 GB — still workable, and `--mtp` gives you decode speedup for it; if you want the old footprint instead, either use a runtime without MTP support (it strips the layer) or set `"num_nextn_predict_layers": 0` in your local `config.json` copy.
+**Memory note:** the MTP-capable fork attaches the module at load (+4.5 GB) even without `--mtp`, trimming the KV+activation budget by that much. To reclaim it, set `"num_nextn_predict_layers": 0` in a local `config.json` copy (or use a runtime without MTP support, which strips the layer). See *Long context & memory* for the machine-level ceiling.
 
 **Notes for other integrators** (verified against vLLM's `deepseek_mtp` semantics): `eh_proj` concat order is `[enorm(embed(t+1)), hnorm(h_t)]`; the hidden fed to the MTP is the backbone's **pre-final-norm** residual stream; for chained drafting feed back the **`shared_head.norm`-normalized** hidden (chaining the raw layer output halves chained acceptance — measured 0.27 → 0.75 conditional at position 2).
 
+## Long context & memory
+
+GLM-5.2's MLA stores a **compressed latent** KV (kv_lora 512 + rope 64 per layer), so the KV cache itself is small — **\~88 KB/token at fp16, \~44 KB/token with int8**. Quantize it with `--kv-bits` (the patched fork engages int8 on the MLA latent; stock `mlx-lm` silently ignores it):
+
+```bash
+mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw \
+  --kv-bits 8 --kv-group-size 64 --quantized-kv-start 4096 --prompt "…"
+
+# OpenAI-compatible server
+mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw \
+  --kv-bits 8 --quantized-kv-start 4096
+```
+
+**On a 256 GiB machine the KV is *not* the binding constraint — the DSA prefill activation is.** Measured peak on this build (M3 Ultra, int8 KV, MTP-attached weights ≈ 242 GB):
+
+| prompt (prefill) | peak | fits 256 GiB (274.9 GB)? |
+|---|---|---|
+| 8K | \~249 GB | ✓ |
+| 26K | \~263 GB | ✓ (tight, \~12 GB headroom) |
+| 32K | \~268 GB | ✓ (very tight) |
+| 64K | \~293 GB | ✗ |
+| 128K | \~344 GB | ✗ |
+
+The DSA lightning indexer scores every past token per prefill chunk, so the activation peak climbs \~+20 GB per \~30K tokens — **independent of KV bits** (int8 vs fp16 differed by only \~1 GB at 26K, because the KV is tiny there). So on a **256 GiB box, keep prompts ≤ \~26–32K**. `--kv-bits 8` still helps where the *KV* is what grows — long **decode**/accumulated context, and **512 GiB** machines with headroom for longer prefills. For genuinely long (100K–1M) context, use the **[3.5 bpw build on a 512 GiB machine](https://huggingface.co/avlp12/GLM-5.2-Alis-MLX-Dynamic-3.5bpw)**.
+
 ## Correctness (verified vs the HF reference)
 
 GLM-5.2's `glm_moe_dsa` needed fixes beyond the stock mlx-lm port; this build was produced with a patched fork and validated:
@@ -142,9 +167,7 @@ mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw --mtp
 
 ## Hardware
 
-Runs in **≤256 GB unified memory** (Apple Silicon). On a 256 GB box the 238 GB of weights leave only \~18 GB for KV + OS (short/mid context; \~13.5 GB with the MTP layer attached — see the *Native MTP* memory note); on a 512 GB M3 Ultra there is ample room for a long-context KV cache.
-
-![Memory headroom: 238 GB weights are tight on a 256 GB machine (\~18 GB free, short/mid context) but roomy on 512 GB (\~274 GB free, 1M context)](assets/memory.png)
+Runs in **≤256 GB unified memory** (Apple Silicon). A "256 GB" Mac is **256 GiB = 274.9 GB** (not 256 decimal); the \~242 GB of (MTP-attached) weights leave \~33 GB. But the DSA sparse-attention **prefill activation grows with context** (measured +\~20 GB at 26K, +\~50 GB at 64K), so the **practical prefill ceiling on a 256 GiB machine is \~26–32K tokens** (peak \~263–268 GB) — not the architecture's 1M. For 100K+ context use a 512 GB (512 GiB) machine. See *Long context & memory* below.
 
 ## Credits