card: correct throughput framing (best-measured TP2 config; ties W4A16, beats FP4 ~9-13%); note TP4 pending

README.md

@@ -20,7 +20,7 @@ pipeline_tag: text-generation
 
 A **W4A8** quantization of DeepSeek-V4-Flash: **INT4 group-quantized MoE expert weights** with **FP8 (e4m3) dynamic per-token activations**, plus FP8 block-quantized attention/dense layers. Produced as a **zero-cost config transformation** of [`canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP`](https://huggingface.co/canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP) — the INT4 weight bytes are **identical**; only the activation quantization scheme in `config.json` changed (experts `input_activations`: `null` → FP8 dynamic-token).
 
-> **⚠️ Honest headline first:** on H200 (Hopper / SM90) this checkpoint **does not make prefill or decode faster than the W4A16 base it was derived from.** It serves correctly and is footprint-neutral (same INT4 weights, same TP2), but W4A8 ≈ W4A16 in throughput. It is published as a **reproducible research artifact** documenting *why* the activation-precision lever doesn't move DeepSeek-V4-Flash performance on Hopper. See **[Investigation & findings](#investigation--findings)**.
+> **⚠️ Honest headline first:** on H200 (Hopper / SM90) this was the **fastest single-config in our sweep** — best TP2 prefill TTFT (1658 ms @24k) *and* highest per-GPU prefill throughput (7410 tok/s/GPU) of every cell tested. It **ties its W4A16 parent** (~2%, within run-to-run noise — the "W4A8 should be ~2× faster than W4A16" hypothesis was refuted), but it **beats the FP4-marlin config by ~9–13%** on the same 2×H200 footprint (int4→Marlin > nvfp4→Marlin). One caveat: it is **vLLM-only** (sglang can't load this checkpoint format), so it isn't a drop-in for an sglang deployment. See **[Investigation & findings](#investigation--findings)**.
 
 > **📦 This is a config / recipe repository — the weight shards are NOT included.** Because the W4A8 transformation reuses the base's INT4 weights **byte-for-byte**, duplicating ~159 GB here would be pure waste. This repo ships the W4A8 `config.json`, tokenizer, weight index, and this card. To get a runnable checkpoint, pull the weights from the base and drop in this `config.json` — see **[Getting the weights](#getting-the-weights)** (one command).
 
@@ -93,17 +93,25 @@ vllm serve ./dsv4-w4a8 \
 
 This checkpoint was built to test a hypothesis: *the DeepSeek-V4-Flash prefill bottleneck is the INT4→BF16 Marlin MoE GEMM, so a W4A8 path (native FP8 activation GEMM) should be ~1.5–2× faster.* **The hypothesis was refuted.** Full sweep on 2–8×H200 (TP2 unless noted), single-request prefill ladder (c=1), long-context (ISL up to 24k):
 
-### Headline: W4A8 gives no throughput advantage over W4A16
+### Headline: W4A8 leads the TP2 matrix, but ties W4A16
 
 | Config | Engine | TP | Prefill TTFT @24k | Prefill tok/s/GPU @24k |
 |---|---|---|---|---|
-| **W4A8** (this model) | vLLM | 2 | **1658 ms** | **7410** |
+| **W4A8** (this model) | vLLM | 2 | **1658 ms** ⭐ | **7410** ⭐ |
 | W4A16 (base) | vLLM | 2 | 1691 ms | 7267 |
 | FP4 (marlin) | vLLM | 2 | 1824 ms | 7090 |
 | FP4 (marlin) | sglang | 2 | 1894 ms | 6832 |
-| FP8 (native) | sglang | 4 | 892 ms | 6888 |
+| FP8 (native) | sglang | **4** | 892 ms | 6888 |
 
-**Per-GPU prefill throughput is flat at ~6.8–7.4k tok/s/GPU across every engine and every quantization.** W4A8 and W4A16 are a **wash** (1658 vs 1691 ms — within noise). The FP8-TP4 config's lower absolute TTFT (892 ms) is **pure tensor-parallel scaling** (2× the GPUs); per-GPU it is *also* a wash.
+**W4A8 is the fastest TP2 config and the highest per-GPU throughput of every cell measured.** Two things to read carefully:
+
+- **vs W4A16 (its parent): a tie** — 1658 vs 1691 ms is ~2%, within run-to-run noise. The specific hypothesis this checkpoint was built to test — *"FP8-activation MoE GEMM should be ~1.5–2× faster than W4A16"* — was **refuted**. At prefill batch-M the MoE is weight-bandwidth-bound, so activation precision doesn't move it and Marlin-W4A16 already matches Cutlass-W4A8.
+- **vs FP4-marlin: a real ~9–13% win** — int4→Marlin beats nvfp4→Marlin, so W4A8 (and W4A16) beat the FP4 base. FP4-marlin is what production currently runs, so W4A8/W4A16 are meaningfully faster than the deployed config *on the same 2-GPU footprint*.
+- The FP8-TP4 cell's low absolute TTFT (892 ms) is **tensor-parallel scaling** (2× the GPUs); **per-GPU, W4A8-TP2 still wins** (7410 > 6888).
+
+Per-GPU throughput spans a narrow ~6.8–7.4k tok/s/GPU band across all cells — the architecture sets a ceiling — but within that band W4A8 sits at the top.
+
+> **TP4 for this checkpoint is not yet benched** — see [To-do](#to-do). Given W4A8-TP2 already leads on both TTFT and per-GPU, W4A8-TP4 is the most likely config to beat the FP8-TP4 892 ms absolute latency.
 
 ### Why the activation-precision lever doesn't help
 
@@ -122,7 +130,11 @@ At prefill batch sizes, the DeepSeek-V4-Flash MoE (top-6 of 256 small experts) i
 
 ### Bottom line
 
-Use W4A8 for **GPU-footprint efficiency** (TP2, ~159 GB, native FP8 activation compute where a downstream kernel benefits) — but **not** expecting it to beat W4A16 on DeepSeek-V4-Flash prefill/decode on Hopper. On this architecture the two are equivalent; the real gains come from prefix caching and, eventually, Blackwell hardware.
+W4A8 is the **best-measured DeepSeek-V4-Flash config on Hopper** at TP2 — top prefill TTFT and top per-GPU throughput. It **ties** its W4A16 sibling (so the ~2× hypothesis failed), but it **beats the FP4-marlin config that ships in production by ~9–13%** on the same footprint. The practical catch is that this checkpoint format loads on **vLLM only**, so capturing that win over an sglang FP4 deployment means an engine switch, not a config swap. The dominant serving lever remains prefix caching (~55% radix hit in prod); larger absolute-latency wins beyond this need Blackwell (native NVFP4 + FP4 indexer).
+
+## To-do
+
+- **Bench TP4** for this checkpoint. W4A8-TP2 already leads the matrix on TTFT and per-GPU; W4A8-TP4 is the strongest candidate to beat the FP8-TP4 892 ms absolute TTFT while keeping INT4 weight footprint. (Not yet run.)
 
 ## Reproducibility