---
title: The Physical AI Inference Gap in Batch-1 LLM Decode
emoji: 🪜
colorFrom: green
colorTo: red
sdk: static
pinned: false
license: cc-by-4.0
short_description: Interactive companion to the batch-1 LLM decode paper
tags:
  - llm-inference
  - benchmark
  - cuda-graphs
  - quantization
  - physical-ai
---

# The Inverted Ladder

An interactive, editorial companion to **"Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode"** (Josef Chen, KAIKAKU).

Light, paper-style layout with four tabs:

- **Intro** — the workload and the headline inversion, plus a live console (pick model / GPU / context / rate → step time, analytic floor, R_floor, $/Mtok) and the animated **bandwidth-flow river** (useful vs wasted bandwidth per GPU).
- **Mechanism** — the **two clocks** (memory cost vs above-floor overhead, the slower one binds), the **empty-highway** kernel timeline with a CUDA Graphs toggle, and the **dexterous barrier** with a draggable control-rate target.
- **Methods** — the full **44-cell R_floor heatmap**, a **pre-registered falsification** panel (1.15× kill line, H100 1.259× CI [1.253, 1.267], L4 null 1.028×), the L4 quantisation matrix, the H100 attention-backend matrix, the cost inversion, and the measurement protocol.
- **Sandbox** — type a sentence and stream it token-by-token at each GPU's *measured* ms/token in real time. H100 + CUDA Graphs vs L4 + ExLlamaV2 vs L4 bf16.

## How it works

Static Space: a single self-contained `index.html` with all measurement data inlined. No backend, no build step, no cold start. SVG/Canvas visualisations only; one font import (Google Fonts).

## Data provenance

Every number is taken from the paper's released JSON artefacts. Step times are medians of 30 measured single-token decode steps after 5 warmup, batch 1, bf16, sdpa, on Modal cloud hosts. R_floor is recomputed from first principles (`W` from `weight_bytes`, `K` from `2·n_layers·n_kv_heads·head_dim·ctx·2` bytes) and matches the paper's tables to three decimals. Peak HBM: H100 3350, A100-80GB 2039, L40S 864, L4 300 GB/s.

The N=10 CUDA Graphs A/B on H100 Qwen-2.5-7B ctx 2048: 14.83 ms eager → 11.78 ms graphed, 1.259×, 95% bootstrap CI [1.253, 1.267].

## Notes

Only the H100 ($3.50/hr) and L4 ($0.30/hr) rate defaults are paper-sourced (Modal, May 2026); A100/L40S are editable placeholders. The cost figure is `step_time × rate`; it excludes idle, networking, storage and batching. The highway's 10-kernel split is schematic (the active fraction equals the measured R_floor). Not affiliated with NVIDIA; model names are trademarks of their respective owners.

Add the arXiv link in `index.html` (`ARXIV_URL`) once the paper URL is confirmed.