Add validated RTX PRO 6000 (SM120) serving recipe

README.md

@@ -23,9 +23,10 @@ An **INT4 weight-only (W4A16) quantization of GLM-5.2** that preserves the BF16
 layer for speculative decoding. Quantized from [zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2)
 with [llm-compressor](https://github.com/vllm-project/llm-compressor) (GPTQ).
 
-**Built for Hopper (H200).** Matches FP8 quality on **half the GPUs** (4×H200 vs 8) and is the **fastest
-GLM-5.2 quant for interactive/agentic serving on Hopper** in a matched, MTP-on head-to-head — with the
-lowest time-to-first-token by a wide margin.
+**Built for Hopper (H200), validated on Blackwell (8×RTX PRO 6000, SM120).** Matches FP8 quality on
+**half the GPUs** (4×H200 vs 8) and is the **fastest GLM-5.2 quant for interactive/agentic serving on
+Hopper** in a matched, MTP-on head-to-head — with the lowest time-to-first-token by a wide margin. A
+complete, quality-validated RTX PRO 6000 recipe is in the **Serving on Blackwell** section below.
 
 ## Why this model
 
@@ -134,6 +135,77 @@ vllm serve <repo> --tensor-parallel-size 4 --enable-expert-parallel \
   --max-model-len 32768 --gpu-memory-utilization 0.92 --trust-remote-code
 ```
 
+## Serving on Blackwell — 8×RTX PRO 6000 (SM120)
+
+Validated end-to-end on **8× RTX PRO 6000 (96 GB, SM 12.0, PCIe)**: quality matches the H200 deployment
+(GSM8K 0.948, IFEval 0.909/0.920, MATH-500 0.954 math-verify, RULER@32K 0.918 / @64K 0.826 — all within
+margin of the H200 column above), **zero token corruption**, with MTP speculative decoding and cudagraphs
+both working. Throughput on this config:
+
+| concurrency | 1 | 4 | 8 | 16 | 32 | 64 |
+|---|---|---|---|---|---|---|
+| output tok/s | 50 | 148 | 280 | 400 | 613 | **989** |
+
+For reference, `nvidia/GLM-5.2-NVFP4` on the same box (MTP **off** — a matched MTP-on NVFP4 run was not
+completed) measures 36 tok/s at c=1 and 447 at c=64. Single-stream speed is memory-bandwidth-bound on this
+hardware (~50 vs ~126 tok/s on H200); the sweet spot is concurrent/batch serving.
+
+**Why SM120 needs a different recipe:** no SM120 sparse-MLA kernel supports GLM-5.2's DSA head layout, so
+DSA sparse attention is disabled and the model serves through dense `TRITON_MLA`. That takes four one-line
+patches on the official `vllm/vllm-openai:glm52` image:
+
+```dockerfile
+FROM vllm/vllm-openai:glm52
+
+# 1. Disable DSA sparse attention (no SM120 sparse-MLA backend for this head size)
+RUN sed -i 's/self\.is_v32 = hasattr(config, "index_topk")/self.is_v32 = False/g' \
+    /usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v2.py
+
+# 2-4. Skip the now-orphaned DSA indexer weights during load (deepseek_v2.py,
+#      deepseek_mtp.py, glm4_moe_mtp.py): guard `param = params_dict[name]` with
+#      `if name not in params_dict: continue`
+RUN python3 - <<'EOF'
+import re
+base = '/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/'
+for f in ('deepseek_v2.py', 'deepseek_mtp.py', 'glm4_moe_mtp.py'):
+    p = base + f
+    src = open(p).read()
+    src = re.sub(r'(\s+)param = params_dict\[name\]',
+                 r'\1if name not in params_dict:\n\1    continue\n\1param = params_dict[name]', src)
+    open(p, 'w').write(src)
+EOF
+```
+
+Then serve (`docker build -t glm52-mtp-sm120 .` first):
+
+```bash
+docker run -d --gpus all --ipc=host --shm-size 16g \
+  -v /path/to/GLM-5.2-W4A16-MTP:/model:ro -p 8000:8000 \
+  -e NCCL_P2P_DISABLE=1 -e VLLM_USE_DEEP_GEMM=0 -e VLLM_MOE_USE_DEEP_GEMM=0 \
+  glm52-mtp-sm120 /model \
+  --tensor-parallel-size 8 --enable-expert-parallel \
+  --attention-backend TRITON_MLA \
+  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
+  --compilation-config '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[2,4,8,16,32,64],"max_cudagraph_capture_size":64}' \
+  --max-model-len 131072 --max-num-seqs 64 --gpu-memory-utilization 0.92 \
+  --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
+  --disable-custom-all-reduce --trust-remote-code
+```
+
+Every deviation from the Hopper command is load-bearing:
+
+- **bf16 KV cache** (no `--kv-cache-dtype fp8`) — fp8 KV overflows SM120's per-SM shared memory in the
+  TRITON_MLA kernel (102,400 > 101,376 bytes).
+- **`PIECEWISE` cudagraph mode** — FULL decode graphs + MTP produce degenerate output on TRITON_MLA (its
+  decode kernel only handles single-token queries; MTP verify sends multi-token queries). PIECEWISE routes
+  them through the prefill path correctly and still gives ~10× over eager at c=1.
+- **MTP `num_speculative_tokens=1`**, with cudagraph capture sizes divisible by (1 + k) = 2.
+- **`VLLM_USE_DEEP_GEMM=0`** — DeepGEMM's attention path doesn't support SM120.
+- **`--attention-backend TRITON_MLA`** — the dense-MLA backend that works once DSA is disabled.
+
+A one-shot bootstrap script (HF download → image build → launch, idempotent) exists in the companion
+repository as `scripts/bootstrap_sm120_glm52_w4a16_mtp.sh`.
+
 ## Method
 
 1. **GPTQ W4A16** (group-128, asymmetric) on the routed experts only, with attention/dense/MTP/embeddings/
@@ -154,8 +226,12 @@ asymmetric-MoE serving fix, and the Blackwell toolchain gaps).
 - 1M-context serving requires all 8 H200s; 4×H200 serves up to ~128K (single-stream engine ceiling ~239K),
   with MTP acceptance ~38% (vs ~46–52% on 8×H200).
 - Asymmetric weights require `--enable-expert-parallel` to serve correctly.
-- Pin vLLM v0.23.x (v0.24+ DSA-indexer layout change breaks loading). Blackwell serving needs additional
-  kernel flags; recommended on Hopper.
+- Pin vLLM v0.23.x (v0.24+ DSA-indexer layout change breaks loading).
+- On Blackwell SM120 (RTX PRO 6000) use the dedicated recipe above: DSA sparse attention must be disabled
+  (dense TRITON_MLA), KV cache stays bf16, cudagraphs run in PIECEWISE mode, and MTP is limited to k=1.
+  Quality is unaffected (validated at parity with H200); single-stream throughput is bandwidth-bound at
+  ~50 tok/s, so size deployments for concurrent traffic. FULL-cudagraph + MTP on TRITON_MLA is an upstream
+  kernel gap (vllm-project/vllm#21505), not fixable by configuration.
 
 ## Acknowledgements