Model card: add benchmarks + chart

README.md

@@ -43,7 +43,7 @@ If you want to put this model (or any other) to work as an autonomous agent with
 
 ## Files
 
-112 sharded `safetensors` (~169 GB total) + `config.json` carrying the `quantization_config` (ModelOpt NVFP4, weight-only). Load directly in vLLM.
+99 sharded `safetensors` (~169 GB total) + `config.json` carrying the `quantization_config` (ModelOpt NVFP4, weight-only). Load directly in vLLM.
 
 ## Usage (vLLM)
 
@@ -66,11 +66,30 @@ vllm serve LibertAIDAI/Hy3-NVFP4 \
   --trust-remote-code
 ```
 
-Requires a vLLM build with `HYV3ForCausalLM` support (vLLM ≥ 0.23) and a **CUDA-13** image on GB10. A full **2× GB10 deployment recipe** (multinode fabric, memory tuning, the exact kernel gotchas) is published separately. Verified serving on a 2× NVIDIA GB10 (Grace Blackwell, sm_121) cluster at tensor-parallel = 2 (coherent generation, correct arithmetic).
+Requires a vLLM build with `HYV3ForCausalLM` support (vLLM ≥ 0.23) and a **CUDA-13** image on GB10. A full **2× GB10 deployment recipe** (multinode fabric, memory tuning, the exact kernel gotchas) is in [`deploy/`](./deploy). Verified serving on a 2× NVIDIA GB10 (Grace Blackwell, sm_121) cluster at tensor-parallel = 2 — coherent generation, correct arithmetic, `hy_v3` tool calls, and 256K needle retrieval.
+
+## Benchmarks — 2× NVIDIA GB10 (DGX Spark)
+
+Served on **two GB10 boxes** (Grace Blackwell, sm_121, ~120 GB unified each) at **tensor-parallel 2** — marlin FP4 MoE, fp8 KV cache, prefix caching, `--enforce-eager --max-model-len 262144 --gpu-memory-utilization 0.90`.
+
+![Hy3-NVFP4 benchmarks on 2× GB10 / DGX Spark](bench-2xgb10.png)
+
+- **KV cache: 286,640 tokens** — a full **256K**-token request fits (weights ~84.5 GB/node).
+- **Throughput scales with concurrency** (128-tok completions, greedy):
+
+| Concurrency | 1 | 4 | 8 | 16 | 32 |
+|---|---|---|---|---|---|
+| Aggregate tok/s | 13.3 | 41.7 | 65.2 | 99.8 | **165.8** |
+| Per-request tok/s | 13.3 | 10.5 | 8.2 | 6.3 | 5.2 |
+
+- **Prefix caching:** a repeated ~6.3K-word prompt is served in **0.32 s vs 3.95 s cold — a 92% cut**.
+- **Long context:** ~120K-token prefill ≈ 138 s.
+
+> **Note on speed.** On GB10 (sm_121) CUDA graphs livelock during capture with the marlin MoE kernel, so `--enforce-eager` is required and single-stream tops out around 13 tok/s — aggregate throughput is where the two-box setup delivers. A native FP4 MoE path (vLLM's b12x / CuTe-DSL) will lift the single-stream ceiling once its sm_121 build ships. On **B200 (sm_100)** or **RTX 50-series (sm_120)** the native NVFP4 kernels run without this constraint.
 
 ## Provenance & method
 
-- Base: `tencent/Hy3` (BF16, 112 shards, 598 GB).
+- Base: `tencent/Hy3` (BF16, 99 shards, 598 GB).
 - Quantizer: shard-streaming ModelOpt `NVFP4QTensor` weight-only pass; expert FFN → NVFP4, everything else copied through in BF16.
 - Verification: per-expert round-trip cosine ≈ 0.995 vs the BF16 source.