Update README.md

README.md

@@ -40,6 +40,13 @@ tensor-parallel 4, which is exactly why 4× 48 GB Ada cards are such an
 attractive target — and exactly why it is painful that the default serving
 stacks refuse to run there.
 
+> **262k is the checkpoint's context ceiling, not the recommended serving
+> length on 48 GB Ada.** After weights + activations + CUDA-graph pools, only
+> ~1 GiB/GPU is left for KV cache, so a full 262k context fits only at high
+> `--gpu-memory-utilization` and leaves almost no headroom. For concurrent or
+> agentic workloads the generated defaults cap `--max-model-len` lower (98304)
+> to keep KV *and* JIT headroom — see [Serving on 48 GB Ada](#serving-on-48-gb-ada-memory-concurrency-and-context) below.
+
 ## Measured performance
 
 4× RTX 6000 Ada (48 GB, PCIe-only, TP=4), `fp8_ds_mla` KV cache,
@@ -53,6 +60,12 @@ stacks refuse to run there.
 The BF16 MTP head drafts nearly perfectly on this checkpoint; a ~1.5–1.65×
 speedup from a single speculative token is about as good as MTP-k=1 gets.
 
+These figures are **single-stream throughput at the maximum-context operating
+point** (262k, high utilization, `--max-num-batched-tokens 2048`). That is the
+right point to *quote* peak tok/s, but not the right point to *run* a
+concurrent or agentic workload on 48 GB cards — see the next section for why,
+and for the defaults the setup script actually generates.
+
 ## Why Ada needs special handling
 
 The stock DeepSeek-V4 serving paths gate on newer architectures at three
@@ -96,8 +109,11 @@ Everything below is automated by
   (forcing the FlashInfer backend crashes on an SWA-cache shape mismatch),
   `--kv-cache-dtype fp8_ds_mla --block-size 256`, and the `deepseek_v4`
   tokenizer / reasoning-parser / tool-call-parser modes.
-- **MTP**: `--speculative-config '{"method":"mtp","num_speculative_tokens":1}'
-  --max-num-batched-tokens 2048`.
+- **MTP**: `--speculative-config '{"method":"mtp","num_speculative_tokens":1}'`.
+  `--max-num-batched-tokens` is `2048` at the peak-throughput point above; the
+  generated `serve.sh` raises it to `4096` for the agentic-serving defaults
+  (which enables chunked prefill — see the memory section, as its buffers come
+  out of the KV pool).
 
 ## The one required source patch (MTP only)
 
@@ -119,11 +135,54 @@ a single tiny layer, so the unfused fallback costs nothing measurable. The
 patcher is idempotent, backs up the original, and refuses to apply against
 any file that doesn't match the `dev145` layout.
 
+## Serving on 48 GB Ada: memory, concurrency, and context
+
+On these cards **KV cache is the thin slice** — weights, activations, and
+CUDA-graph pools consume almost the whole 48 GB, leaving on the order of
+~1 GiB/GPU for KV. Two failure modes sit on opposite sides of a narrow tuning
+window, and the generated defaults are the corner that clears both:
+
+- **JIT-OOM (too little headroom).** The Triton sparse-MLA path compiles a
+  kernel specialization *per prompt shape*, lazily, and loading each freshly
+  compiled cubin (`cuModuleLoadData`) needs driver-level free VRAM *outside*
+  the torch pool. At high utilization a never-before-seen prompt shape
+  arriving mid-run can find no room to load its kernel and crash a worker
+  (observed in `_tf32_hc_prenorm_gemm`), taking the server down. Two things
+  defuse this: the generated `warmup.sh` compiles the shape space at startup
+  while memory is free, and the systemd unit uses `Restart=always` (a JIT-OOM
+  can exit *cleanly* from systemd's view, so `on-failure` is not enough).
+- **KV-fit refusal (too little KV).** vLLM refuses to boot if a single full
+  `--max-model-len` sequence won't fit the KV pool. At the headroom-safe
+  utilizations below, full 262k does **not** fit; `--max-model-len 98304` does,
+  with room for the KV a long agentic trajectory actually uses.
+
+**Concurrency: use `--max-num-seqs 1` for agentic workloads.** With several
+30–100k-token trajectories in flight, they cannot co-reside in the KV pool, so
+vLLM preempts and *re-prefills* evicted sequences from scratch — prefix-cache
+hit rate collapses below 40%, generation throughput drops to single digits
+while prefill spikes, and wall-clock blows up roughly 10×. A single resident
+stream (prefix hit ~85–91%) decoding continuously is dramatically faster in
+practice. One stream that never evicts beats four that thrash.
+
+**Utilization.** `0.97` maximizes the KV pool and works for a *watched* run
+backed by warmup + auto-restart, but leaves only ~0.5–0.8 GiB/GPU free under a
+long trajectory. For **unattended or shared** operation prefer
+`--gpu-memory-utilization 0.95`: it restores ~2.4 GiB of headroom at identical
+seq=1 throughput. The setup script exposes both as parameters; pick per how
+closely you'll be watching.
+
+**Smaller Ada cards (RTX 4090, 24 GB).** This recipe is validated on 4× 48 GB
+(RTX 6000 Ada). A 159 GB checkpoint under TP=4 needs ~38 GiB/GPU for weights
+alone, so it does **not** fit four 24 GB 4090s — you would need more cards
+(e.g. TP=8) and an even tighter KV budget, and none of the memory figures here
+transfer directly. The `rtx-4090` tag reflects the SM89 architecture the stack
+targets, not a claim that this checkpoint fits on 4090-class VRAM.
+
 ## Quickstart
 
 ```bash
 # 1. Build the environment (venv, pinned wheels, patch, verification,
-#    generated serve.sh / canary.sh / systemd unit):
+#    generated serve.sh / warmup.sh / canary.sh / systemd unit):
 ./setup_deepseek_v4_sm89.sh ~/deepseek-v4-serve SinclairSchneider/DeepSeek-V4-Flash-W4A16-FP8-MTP-Ada
 #    (the original repo id works identically:
 #     ./setup_deepseek_v4_sm89.sh ~/deepseek-v4-serve canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP)
@@ -138,10 +197,14 @@ any file that doesn't match the `dev145` layout.
 ~/deepseek-v4-serve/canary.sh
 ```
 
-Expect ~80 s of weight loading plus ~5 s for the MTP drafter, brief Triton
-JIT compiles during warmup and the first requests, then steady-state
-throughput. For unattended operation, the setup script also generates a
-systemd unit (see [`deepseek-v4-flash.service`](./deepseek-v4-flash.service)).
+Expect ~80 s of weight loading plus ~5 s for the MTP drafter, then a Triton
+shape-warmup pass (`warmup.sh`, also generated) that compiles the kernel
+specializations up front, then steady-state throughput. For unattended
+operation the setup script generates a systemd unit
+(see [`deepseek-v4-flash.service`](./deepseek-v4-flash.service)) that runs the
+warmup automatically after every start (`ExecStartPost`) and uses
+`Restart=always` so a mid-run kernel-load OOM self-heals rather than leaving a
+dead port.
 
 ## Gotchas the scripts already handle