Upload folder using huggingface_hub

README.md

@@ -0,0 +1,243 @@
+---
+language:
+- en
+license: gemma
+library_name: vllm
+base_model: google/t5gemma-2-1b-1b
+datasets:
+- HuggingFaceH4/ultrachat_200k
+- Aeala/ShareGPT_Vicuna_unfiltered
+- ise-uiuc/Magicoder-Evol-Instruct-110K
+- Salesforce/xlam-function-calling-60k
+- glaiveai/glaive-function-calling-v2
+- TIGER-Lab/MathInstruct
+tags:
+- speculative-decoding
+- speculators
+- vllm
+- t5gemma2
+- dflash
+---
+
+# T5Gemma 2 1B-1B DFlash Large
+
+This repository contains a five-layer DFlash draft model that predicts a seven-token speculative block from selected verifier decoder states, trained for the frozen [google/t5gemma-2-1b-1b](https://huggingface.co/google/t5gemma-2-1b-1b) verifier.
+
+The checkpoint is a **speculative draft model**, not a standalone language model. It must be paired with the base verifier. This repository includes inference weights, complete optimizer and scheduler states, validation metrics, the exact training command, continuation state, TensorBoard events, and raw benchmark summaries.
+
+"Large" distinguishes this 200,000-example mixed-data run from the earlier development checkpoint.
+
+## Model details
+
+| Item | Value |
+|---|---|
+| Draft architecture | `DFlashDraftModel` |
+| Verifier | `google/t5gemma-2-1b-1b` |
+| Draft transformer layers | 5 |
+| Draft hidden / intermediate size | 1,152 / 6,912 |
+| Draft vocabulary | 32,000 tokens mapped from the verifier vocabulary |
+| Speculative block | 8 positions: one anchor and up to 7 proposed tokens |
+| Default proposals per step | 7 |
+| Runtime override | 1-7 proposed tokens |
+| Verifier decoder states | 2, 7, 13, 18, 21 |
+| Precision | BF16 |
+| Selected checkpoint | Epoch 5 (sixth/final epoch), global step 146,486 |
+
+## Usage with vLLM
+
+Install a compatible vLLM build and the [T5Gemma 2 vLLM plugin](https://github.com/d0rj/t5gemma2-vllm-plugin):
+
+```bash
+pip install "git+https://github.com/d0rj/t5gemma2-vllm-plugin.git"
+```
+
+The plugin is discovered through vLLM entry points. No `VLLM_PLUGINS` environment variable is required.
+
+### Serve with the checkpoint default (K=7)
+
+```bash
+vllm serve d0rj/t5gemma-2-1b-1b.dflash-large \
+  --host 127.0.0.1 \
+  --port 8000 \
+  --served-model-name t5gemma-2-1b-1b-dflash \
+  --trust-remote-code \
+  --no-enable-chunked-prefill \
+  --max-model-len 2048 \
+  --max-num-seqs 1
+```
+
+The plugin reads the verifier reference from `config.json` and loads `google/t5gemma-2-1b-1b` automatically.
+
+### Limit the number of draft tokens
+
+To override K, serve the verifier explicitly and reference this Hub repository. The adapter uses `method: "dflash"` for all DFlash-family schedulers, then dispatches to the concrete architecture recorded in the draft checkpoint.
+
+```bash
+vllm serve google/t5gemma-2-1b-1b \
+  --host 127.0.0.1 \
+  --port 8000 \
+  --served-model-name t5gemma-2-1b-1b-dflash-k3 \
+  --trust-remote-code \
+  --no-enable-chunked-prefill \
+  --max-model-len 2048 \
+  --max-num-seqs 1 \
+  --speculative-config '{"model":"d0rj/t5gemma-2-1b-1b.dflash-large","method":"dflash","num_speculative_tokens":3}'
+```
+
+Set `num_speculative_tokens` to an integer from 1 through 7. K=3 and K=7 were benchmarked below. CUDA Graph remains enabled when `--enforce-eager` is omitted.
+
+```bash
+curl http://127.0.0.1:8000/v1/completions \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "model": "t5gemma-2-1b-1b-dflash",
+    "prompt": "Solve step by step.\n\nQuestion: A train travels 60 miles in 2 hours. What is its average speed?\nAnswer:",
+    "max_tokens": 128,
+    "temperature": 0
+  }'
+```
+
+This draft is not intended to be loaded independently with Transformers `AutoModelForSeq2SeqLM`. The verifier supplies encoder execution, verification, embeddings, and target hidden states.
+
+## Training
+
+Training used online hidden-state extraction. The frozen verifier and trainable draft shared one RTX 5070 Ti 16 GB GPU under WSL2; hidden states were generated in process rather than stored for the full dataset.
+
+### Data mixture
+
+The mixture contained exactly 200,000 examples and was deterministically shuffled with seed 42.
+
+| Dataset | Examples | Share | Coverage |
+|---|---:|---:|---|
+| [UltraChat 200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) | 60,000 | 30% | General instruction following and dialogue |
+| [ShareGPT Vicuna unfiltered](https://huggingface.co/datasets/Aeala/ShareGPT_Vicuna_unfiltered) | 20,000 | 10% | Multi-turn conversation |
+| [Magicoder Evol Instruct](https://huggingface.co/datasets/ise-uiuc/Magicoder-Evol-Instruct-110K) | 50,000 | 25% | Code and software tasks |
+| [xLAM function calling](https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k) | 40,000 | 20% | Tool and function calling |
+| [Glaive function calling v2](https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2) | 20,000 | 10% | Tool use and function calling |
+| [MathInstruct](https://huggingface.co/datasets/TIGER-Lab/MathInstruct) | 10,000 | 5% | Mathematics and reasoning |
+
+The prepared data used a 90%/10% train/validation split. Encoder inputs were truncated to 2,048 tokens and decoder targets to 1,024 tokens. Exact retained counts and converters are in [training/data_config.json](training/data_config.json).
+
+### Hyperparameters
+
+| Parameter | Value |
+|---|---|
+| Epochs | 6 |
+| Optimizer | AdamW |
+| Learning rate | 6e-4 |
+| Weight decay | 0.01 |
+| Scheduler | Linear; automatic 1% warmup |
+| Packed token budget | 2,048 |
+| Online extraction microbatch | 32 |
+| Maximum anchors | 256 |
+| Block size | 8 |
+| Hidden-state dtype | BF16 |
+| Noise standard deviation | 0.05 |
+| Loss | KL divergence |
+| Verifier | Frozen |
+| Dataloader workers | 0 |
+| Seed | 42 |
+| Training revision | `a58573acefd975901bae8e83b6e8b5e6ace717ef` |
+| Software | Speculators 0.7.0.dev72; Transformers 5.12.1; PyTorch 2.12.0 |
+
+
+The launcher is [training/train.sh](training/train.sh). [train_command.txt](train_command.txt) records the resolved command and software versions.
+
+### TensorBoard
+
+The requested cumulative comparison logs are included: `dflash_t5gemma2_mixture_200k`.
+
+```bash
+tensorboard --logdir tensorboard --port 6006
+```
+
+## Validation metrics
+
+These are teacher-forced metrics from the selected epoch-5 checkpoint. Position accuracy is not equivalent to runtime acceptance because later runtime positions depend on all earlier draft tokens being accepted.
+
+| Metric | Value |
+|---|---:|
+| Validation loss | 1.2283 |
+| Full-block token accuracy | 34.51% |
+| Position 1 accuracy | 67.90% |
+| Position 2 accuracy | 49.17% |
+| Position 3 accuracy | 36.79% |
+| Position 4 accuracy | 28.76% |
+| Position 5 accuracy | 23.30% |
+| Position 6 accuracy | 19.33% |
+| Position 7 accuracy | 16.21% |
+
+Unrounded values are available in [val_metrics.json](val_metrics.json).
+
+## vLLM performance
+
+The raw verifier and all drafts used identical saved prompt snapshots on one RTX 5070 Ti 16 GB under WSL2. The vLLM development build was based on commit `e4b3da3feb20c1854a4b23e431cfb787ee268f72`.
+
+Common settings: greedy decoding, seed 0, maximum model length 2,048, maximum output 128, `--max-num-batched-tokens 4096`, GPU memory utilization 0.85, chunked prefill disabled, CUDA Graph enabled, and 8 excluded warm-up requests. BS1 used concurrency/max-num-seqs 1; BS4 used 4. Dataset sizes: MATH-500 500, GSM8K 1,319, HumanEval 164, MBPP 500, MT-Bench 160 independent turns, and UltraChat `test_sft` 5,000.
+
+`Base tok/s` is raw verifier throughput; `Draft tok/s` is end-to-end verified throughput. Speedup is their ratio. Latency, TTFT, and ITL are mean/p95 milliseconds. Acceptance rate is accepted tokens divided by all proposed draft tokens. Acceptance length is `1 + accepted_tokens / draft_steps`. Values below 1.0x are genuine slowdowns for this setup.
+
+### Batch size 1
+
+| Dataset | K | Base tok/s | Draft tok/s | Speedup | Latency mean/p95 ms | TTFT mean/p95 ms | ITL mean/p95 ms | Accept rate | Accept p1/p2/p3 | Accept len |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| math500 | 3 | 113.45 | 64.68 | 0.570x | 1970.50/2394.33 | 176.98/348.85 | 14.18/17.33 | 6.32% | 16.82%/1.89%/0.26% | 1.190 |
+| math500 | 7 | 113.45 | 74.16 | 0.654x | 1716.92/2122.72 | 138.95/232.84 | 12.49/15.42 | 2.73% | 16.89%/1.92%/0.26% | 1.191 |
+| gsm8k | 3 | 115.56 | 65.68 | 0.568x | 1943.86/2365.31 | 178.53/355.62 | 13.93/16.93 | 6.84% | 18.35%/1.95%/0.21% | 1.205 |
+| gsm8k | 7 | 115.56 | 73.26 | 0.634x | 1743.27/2099.80 | 152.56/246.42 | 12.55/15.04 | 2.75% | 17.19%/1.83%/0.20% | 1.192 |
+| humaneval | 3 | 93.66 | 58.98 | 0.630x | 1178.81/2207.20 | 162.99/323.05 | 15.05/20.25 | 5.99% | 16.43%/1.45%/0.10% | 1.180 |
+| humaneval | 7 | 93.66 | 64.81 | 0.692x | 1022.86/2083.64 | 157.66/238.51 | 13.24/16.24 | 2.71% | 17.46%/1.41%/0.08% | 1.189 |
+| mbpp | 3 | 117.61 | 67.35 | 0.573x | 1830.21/2291.14 | 181.88/374.19 | 13.37/16.29 | 6.52% | 18.39%/1.10%/0.06% | 1.196 |
+| mbpp | 7 | 117.61 | 74.82 | 0.636x | 1635.87/2016.28 | 152.67/244.47 | 12.10/14.41 | 2.84% | 18.66%/1.14%/0.06% | 1.199 |
+| mt_bench | 3 | 116.89 | 65.85 | 0.563x | 1907.75/2463.49 | 175.25/376.67 | 13.83/17.45 | 6.04% | 16.21%/1.76%/0.15% | 1.181 |
+| mt_bench | 7 | 116.89 | 73.27 | 0.627x | 1714.64/2135.27 | 139.66/237.62 | 12.69/15.55 | 2.35% | 14.88%/1.46%/0.09% | 1.164 |
+| ultrachat | 3 | 62.83 | 52.06 | 0.829x | 2442.49/3377.26 | 168.36/252.91 | 18.01/24.71 | 7.08% | 18.36%/2.59%/0.30% | 1.212 |
+| ultrachat | 7 | 62.83 | 51.13 | 0.814x | 2487.86/3443.12 | 172.78/258.77 | 18.33/25.19 | 3.06% | 18.49%/2.58%/0.32% | 1.215 |
+
+### Batch size 4
+
+| Dataset | K | Base tok/s | Draft tok/s | Speedup | Latency mean/p95 ms | TTFT mean/p95 ms | ITL mean/p95 ms | Accept rate | Accept p1/p2/p3 | Accept len |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| math500 | 3 | 266.74 | 185.18 | 0.694x | 2740.52/3396.06 | 232.73/422.78 | 19.89/24.71 | 6.29% | 16.64%/1.97%/0.24% | 1.189 |
+| math500 | 7 | 266.74 | 188.44 | 0.706x | 2690.23/3411.27 | 223.15/410.68 | 19.61/24.83 | 2.74% | 16.86%/2.02%/0.26% | 1.192 |
+| gsm8k | 3 | 280.58 | 188.79 | 0.673x | 2702.38/3317.10 | 235.61/424.62 | 19.48/24.18 | 6.81% | 18.37%/1.86%/0.20% | 1.204 |
+| gsm8k | 7 | 280.58 | 188.46 | 0.672x | 2705.30/3477.26 | 301.18/834.45 | 18.98/23.59 | 2.74% | 17.14%/1.81%/0.20% | 1.192 |
+| humaneval | 3 | 194.21 | 147.26 | 0.758x | 1856.84/3610.00 | 250.89/497.85 | 24.55/42.01 | 5.95% | 16.41%/1.35%/0.08% | 1.179 |
+| humaneval | 7 | 194.21 | 141.91 | 0.731x | 1877.28/3978.09 | 343.90/982.06 | 22.96/33.21 | 2.68% | 17.40%/1.31%/0.09% | 1.188 |
+| mbpp | 3 | 275.24 | 189.94 | 0.690x | 2595.66/3320.97 | 234.11/439.53 | 19.14/23.96 | 6.46% | 18.25%/1.11%/0.04% | 1.194 |
+| mbpp | 7 | 275.24 | 170.29 | 0.619x | 2875.58/4074.32 | 634.04/1885.92 | 18.20/24.69 | 2.85% | 18.78%/1.13%/0.05% | 1.200 |
+| mt_bench | 3 | 309.23 | 195.62 | 0.633x | 2573.23/3166.33 | 221.35/448.16 | 18.68/23.10 | 6.07% | 16.45%/1.71%/0.06% | 1.182 |
+| mt_bench | 7 | 309.23 | 180.37 | 0.583x | 2753.90/3791.87 | 381.83/1026.59 | 21.71/25.58 | 2.48% | 15.47%/1.60%/0.12% | 1.173 |
+| ultrachat | 3 | 155.46 | 135.15 | 0.869x | 3763.95/4515.90 | 257.60/405.56 | 27.76/33.28 | 7.09% | 18.40%/2.57%/0.30% | 1.213 |
+| ultrachat | 7 | 155.46 | 134.03 | 0.862x | 3794.92/4543.27 | 255.94/399.63 | 28.03/33.56 | 3.07% | 18.54%/2.58%/0.31% | 1.215 |
+
+Full results, all quantiles, request counts, prompt hashes, baseline rows, and complete suite comparisons are included in [benchmarks/bs1](benchmarks/bs1) and [benchmarks/bs4](benchmarks/bs4).
+
+## Repository contents
+
+- `model.safetensors`: selected draft weights.
+- `config.json`, `config.py`: architecture and verifier linkage.
+- `optimizer_state_dict.pt`: complete AdamW state.
+- `scheduler_state_dict.pt`: scheduler state.
+- `training_state.json`: epoch/global-step continuation metadata.
+- `train_command.txt`: exact training command and package versions.
+- `val_metrics.json`: selected-checkpoint validation metrics.
+- `training/`: data metadata and launcher.
+- `tensorboard/`: cumulative TensorBoard events.
+- `benchmarks/`: raw bs1 and bs4 reports.
+
+For optimizer-level continuation, restore the root checkpoint files as an epoch directory in a compatible Speculators checkpoint tree and point `checkpoint_best` to it.
+
+## Limitations
+
+- Performance depends on hardware, workload, batch size, and vLLM version; universal acceleration is not claimed.
+- The plugin path was validated for single-GPU text serving. Tensor/pipeline parallelism, quantized serving, and the vision path require separate validation.
+- Teacher-forced validation accuracy does not guarantee equal runtime acceptance.
+- K=3 reduces draft work but changes block length relative to eight-position training; K=7 matches the trained block shape.
+- Generated outputs retain the behavior and limitations of the verifier and training data.
+
+## License and attribution
+
+This checkpoint is derived from T5Gemma 2 and is distributed under the Gemma license. Using the verifier requires accepting Google's Gemma terms. Each dataset retains its own license and conditions.
+
+Training used the [vLLM Speculators project](https://github.com/vllm-project/speculators) with T5Gemma 2, DFlare, DSpark, online-extraction, and logging extensions from the accompanying development branch.

benchmarks/bs1/comparison.md

@@ -0,0 +1,44 @@
+| Dataset | Run | Req | Output tok/s | Speedup | Latency mean/p95 ms | TTFT mean/p95 ms | ITL mean/p95 ms | TPOT mean/p95 ms | Accept rate | Accept p1/p2/p3 | Accept len |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| math500 | baseline | 500 | 113.45 | 1.000x | 1121.85/1314.59 | 169.94/335.47 | 7.54/8.68 | 7.54/8.69 | — | -/-/- | — |
+| math500 | dflash_k3 | 500 | 64.68 | 0.570x | 1970.50/2394.33 | 176.98/348.85 | 14.18/17.33 | 14.18/17.33 | 6.32% | 16.82%/1.89%/0.26% | 1.190 |
+| math500 | dflash_k7 | 500 | 74.16 | 0.654x | 1716.92/2122.72 | 138.95/232.84 | 12.49/15.42 | 12.49/15.42 | 2.73% | 16.89%/1.92%/0.26% | 1.191 |
+| math500 | dflare_k3 | 500 | 93.74 | 0.826x | 1359.56/1919.75 | 150.99/254.30 | 9.56/13.64 | 9.56/13.64 | 39.18% | 64.52%/35.08%/17.94% | 2.175 |
+| math500 | dflare_k7 | 500 | 97.45 | 0.859x | 1306.61/1831.94 | 149.97/241.51 | 9.16/13.16 | 9.17/13.16 | 18.55% | 64.82%/35.30%/17.18% | 2.298 |
+| math500 | dspark_k3 | 500 | 131.01 | 1.155x | 972.82/1383.58 | 139.16/243.83 | 6.59/9.67 | 6.59/9.67 | 44.80% | 65.41%/42.19%/26.82% | 2.344 |
+| math500 | dspark_k7 | 500 | 136.48 | 1.203x | 932.87/1415.98 | 142.46/239.27 | 6.26/10.08 | 6.26/10.08 | 22.84% | 64.00%/39.61%/24.04% | 2.599 |
+| gsm8k | baseline | 1319 | 115.56 | 1.000x | 1105.36/1276.34 | 164.78/332.90 | 7.42/7.93 | 7.42/7.93 | — | -/-/- | — |
+| gsm8k | dflash_k3 | 1319 | 65.68 | 0.568x | 1943.86/2365.31 | 178.53/355.62 | 13.93/16.93 | 13.93/16.93 | 6.84% | 18.35%/1.95%/0.21% | 1.205 |
+| gsm8k | dflash_k7 | 1319 | 73.26 | 0.634x | 1743.27/2099.80 | 152.56/246.42 | 12.55/15.04 | 12.55/15.04 | 2.75% | 17.19%/1.83%/0.20% | 1.192 |
+| gsm8k | dflare_k3 | 1319 | 80.95 | 0.701x | 1577.24/2166.57 | 150.18/254.45 | 11.26/15.61 | 11.26/15.61 | 27.89% | 53.39%/21.58%/8.69% | 1.837 |
+| gsm8k | dflare_k7 | 1319 | 81.92 | 0.709x | 1558.94/2191.44 | 152.78/255.01 | 11.09/15.86 | 11.09/15.86 | 12.80% | 54.17%/21.95%/9.32% | 1.896 |
+| gsm8k | dspark_k3 | 1319 | 111.95 | 0.969x | 1140.44/1627.45 | 137.90/232.76 | 7.91/11.64 | 7.91/11.64 | 31.19% | 53.50%/25.78%/14.30% | 1.936 |
+| gsm8k | dspark_k7 | 1319 | 111.09 | 0.961x | 1149.49/1690.77 | 142.42/238.71 | 7.94/11.94 | 7.94/11.94 | 14.63% | 53.36%/25.01%/13.27% | 2.024 |
+| humaneval | baseline | 164 | 93.66 | 1.000x | 754.68/1367.14 | 189.28/342.50 | 8.11/9.87 | 8.11/9.87 | — | -/-/- | — |
+| humaneval | dflash_k3 | 164 | 58.98 | 0.630x | 1178.81/2207.20 | 162.99/323.05 | 15.05/20.25 | 15.06/20.27 | 5.99% | 16.43%/1.45%/0.10% | 1.180 |
+| humaneval | dflash_k7 | 164 | 64.81 | 0.692x | 1022.86/2083.64 | 157.66/238.51 | 13.24/16.24 | 13.24/16.24 | 2.71% | 17.46%/1.41%/0.08% | 1.189 |
+| humaneval | dflare_k3 | 164 | 99.28 | 1.060x | 700.23/1298.74 | 163.54/255.47 | 8.18/11.82 | 8.19/11.83 | 58.63% | 77.59%/56.43%/41.86% | 2.759 |
+| humaneval | dflare_k7 | 164 | 114.43 | 1.222x | 579.29/1124.12 | 153.89/249.90 | 7.29/11.29 | 7.29/11.30 | 32.60% | 76.90%/55.26%/40.09% | 3.282 |
+| humaneval | dspark_k3 | 164 | 130.99 | 1.399x | 530.74/905.70 | 146.04/236.92 | 6.10/8.98 | 6.11/8.99 | 60.43% | 77.12%/58.94%/45.24% | 2.813 |
+| humaneval | dspark_k7 | 164 | 145.41 | 1.553x | 455.86/852.21 | 143.63/241.35 | 5.51/9.66 | 5.51/9.66 | 34.97% | 73.45%/55.20%/39.37% | 3.448 |
+| mbpp | baseline | 500 | 117.61 | 1.000x | 1045.16/1252.24 | 170.20/343.01 | 7.19/7.64 | 7.19/7.64 | — | -/-/- | — |
+| mbpp | dflash_k3 | 500 | 67.35 | 0.573x | 1830.21/2291.14 | 181.88/374.19 | 13.37/16.29 | 13.37/16.29 | 6.52% | 18.39%/1.10%/0.06% | 1.196 |
+| mbpp | dflash_k7 | 500 | 74.82 | 0.636x | 1635.87/2016.28 | 152.67/244.47 | 12.10/14.41 | 12.10/14.41 | 2.84% | 18.66%/1.14%/0.06% | 1.199 |
+| mbpp | dflare_k3 | 500 | 103.43 | 0.879x | 1191.91/1663.97 | 151.14/261.67 | 8.62/12.05 | 8.62/12.05 | 46.96% | 68.67%/44.38%/27.84% | 2.409 |
+| mbpp | dflare_k7 | 500 | 115.48 | 0.982x | 1059.83/1511.53 | 149.30/254.47 | 7.82/11.13 | 7.83/11.13 | 24.50% | 67.47%/42.50%/25.01% | 2.715 |
+| mbpp | dspark_k3 | 500 | 146.73 | 1.248x | 840.11/1144.96 | 133.81/238.02 | 5.95/8.07 | 5.95/8.07 | 52.37% | 71.03%/51.06%/35.03% | 2.571 |
+| mbpp | dspark_k7 | 500 | 158.24 | 1.346x | 773.38/1069.82 | 144.88/240.50 | 5.46/7.70 | 5.46/7.70 | 28.85% | 68.24%/46.95%/30.63% | 3.020 |
+| mt_bench | baseline | 160 | 116.89 | 1.000x | 1074.68/1319.75 | 160.69/327.47 | 7.31/8.34 | 7.31/8.34 | — | -/-/- | — |
+| mt_bench | dflash_k3 | 160 | 65.85 | 0.563x | 1907.75/2463.49 | 175.25/376.67 | 13.83/17.45 | 13.84/17.45 | 6.04% | 16.21%/1.76%/0.15% | 1.181 |
+| mt_bench | dflash_k7 | 160 | 73.27 | 0.627x | 1714.64/2135.27 | 139.66/237.62 | 12.69/15.55 | 12.69/15.55 | 2.35% | 14.88%/1.46%/0.09% | 1.164 |
+| mt_bench | dflare_k3 | 160 | 85.55 | 0.732x | 1468.51/2127.07 | 159.73/265.74 | 10.65/15.71 | 10.66/15.71 | 32.31% | 57.74%/27.25%/11.94% | 1.969 |
+| mt_bench | dflare_k7 | 160 | 89.12 | 0.762x | 1409.59/2124.10 | 146.91/259.67 | 10.23/15.34 | 10.24/15.34 | 14.63% | 57.43%/27.01%/11.06% | 2.024 |
+| mt_bench | dspark_k3 | 160 | 121.09 | 1.036x | 1037.41/1565.51 | 141.77/240.41 | 7.31/11.71 | 7.32/11.71 | 37.06% | 59.00%/32.88%/19.30% | 2.112 |
+| mt_bench | dspark_k7 | 160 | 122.24 | 1.046x | 1027.68/1596.33 | 141.25/255.56 | 7.28/11.79 | 7.29/11.79 | 17.42% | 56.41%/30.24%/17.08% | 2.220 |
+| ultrachat | baseline | 5000 | 62.83 | 1.000x | 2024.82/3256.73 | 166.96/289.88 | 14.71/23.86 | 14.71/23.87 | — | -/-/- | — |
+| ultrachat | dflash_k3 | 5000 | 52.06 | 0.829x | 2442.49/3377.26 | 168.36/252.91 | 18.01/24.71 | 18.01/24.71 | 7.08% | 18.36%/2.59%/0.30% | 1.212 |
+| ultrachat | dflash_k7 | 5000 | 51.13 | 0.814x | 2487.86/3443.12 | 172.78/258.77 | 18.33/25.19 | 18.33/25.19 | 3.06% | 18.49%/2.58%/0.32% | 1.215 |
+| ultrachat | dflare_k3 | 5000 | 62.40 | 0.993x | 2037.94/2756.50 | 185.96/279.43 | 14.67/19.80 | 14.67/19.80 | 31.13% | 57.22%/26.06%/10.11% | 1.934 |
+| ultrachat | dflare_k7 | 5000 | 66.03 | 1.051x | 1925.98/2635.48 | 180.71/272.82 | 13.82/18.90 | 13.82/18.90 | 14.39% | 57.70%/26.66%/10.39% | 2.008 |
+| ultrachat | dspark_k3 | 5000 | 78.10 | 1.243x | 1628.21/2287.42 | 171.98/258.65 | 11.54/16.24 | 11.54/16.25 | 32.41% | 55.57%/28.18%/13.47% | 1.972 |
+| ultrachat | dspark_k7 | 5000 | 78.70 | 1.253x | 1616.26/2290.96 | 176.72/263.78 | 11.40/16.28 | 11.40/16.28 | 15.45% | 55.33%/28.03%/13.31% | 2.081 |

benchmarks/bs1/results.json

@@ -0,0 +1,1304 @@
+[
+  {
+    "dataset": "math500",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 500,
+    "output_tokens": 63642,
+    "request_throughput_rps": 0.8912881111809688,
+    "output_throughput_tps": 113.44671594355843,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1121.8503840199992,
+    "latency_p50_ms": 1100.7910255029856,
+    "latency_p95_ms": 1314.5946230510162,
+    "latency_p99_ms": 1484.1045933025453,
+    "ttft_mean_ms": 169.94078358601837,
+    "ttft_p50_ms": 146.2457435009128,
+    "ttft_p95_ms": 335.47060909950216,
+    "ttft_p99_ms": 373.49712165247183,
+    "itl_mean_ms": 7.536181297069743,
+    "itl_p50_ms": 7.382753629891218,
+    "itl_p95_ms": 8.68439770515185,
+    "itl_p99_ms": 10.087323529295983,
+    "tpot_mean_ms": 7.537555813759378,
+    "tpot_p50_ms": 7.383992137781786,
+    "tpot_p95_ms": 8.685839803156412,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "math500",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 63731,
+    "request_throughput_rps": 0.5074555001107937,
+    "output_throughput_tps": 64.68129295512199,
+    "speedup_vs_baseline": 0.5701468959868523,
+    "latency_mean_ms": 1970.4961424658832,
+    "latency_p50_ms": 1974.9981015011144,
+    "latency_p95_ms": 2394.326527546944,
+    "latency_p99_ms": 2618.456352452631,
+    "ttft_mean_ms": 176.97923488401284,
+    "ttft_p50_ms": 151.66605199920014,
+    "ttft_p95_ms": 348.84992924853566,
+    "ttft_p99_ms": 401.735668064066,
+    "itl_mean_ms": 14.179071923655801,
+    "itl_p50_ms": 14.188702799193972,
+    "itl_p95_ms": 17.33207021927915,
+    "itl_p99_ms": 18.87767345473462,
+    "tpot_mean_ms": 14.180381967095736,
+    "tpot_p50_ms": 14.189951897641887,
+    "tpot_p95_ms": 17.33332515668878,
+    "acceptance_rate": 0.06323912607737021,
+    "acceptance_pos_1": 0.16818250150330727,
+    "acceptance_pos_2": 0.01894167167769092,
+    "acceptance_pos_3": 0.0025932050511124475,
+    "mean_acceptance_length": 1.1897173782321107
+  },
+  {
+    "dataset": "math500",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 63668,
+    "request_throughput_rps": 0.5823973894271881,
+    "output_throughput_tps": 74.16015398010043,
+    "speedup_vs_baseline": 0.6537003152827826,
+    "latency_mean_ms": 1716.9249164580651,
+    "latency_p50_ms": 1721.4581165007985,
+    "latency_p95_ms": 2122.7248207490447,
+    "latency_p99_ms": 2282.573311009855,
+    "ttft_mean_ms": 138.95278630002576,
+    "ttft_p50_ms": 141.47450450036558,
+    "ttft_p95_ms": 232.83745254921087,
+    "ttft_p99_ms": 255.96303990992965,
+    "itl_mean_ms": 12.488242557152846,
+    "itl_p50_ms": 12.48854895669593,
+    "itl_p95_ms": 15.423414866535039,
+    "itl_p99_ms": 16.76060266095041,
+    "tpot_mean_ms": 12.489462153205425,
+    "tpot_p50_ms": 12.489741562990455,
+    "tpot_p95_ms": 15.424737617707956,
+    "acceptance_rate": 0.027334900964792862,
+    "acceptance_pos_1": 0.16893291649402992,
+    "acceptance_pos_2": 0.019209763079588684,
+    "acceptance_pos_3": 0.002561301743945158,
+    "mean_acceptance_length": 1.19134430675355
+  },
+  {
+    "dataset": "math500",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 63731,
+    "request_throughput_rps": 0.7354717914893614,
+    "output_throughput_tps": 93.74470548681697,
+    "speedup_vs_baseline": 0.8263324742997098,
+    "latency_mean_ms": 1359.560892554051,
+    "latency_p50_ms": 1336.2468844989053,
+    "latency_p95_ms": 1919.7459352993976,
+    "latency_p99_ms": 2183.3393649194472,
+    "ttft_mean_ms": 150.9933998281267,
+    "ttft_p50_ms": 150.4645309996704,
+    "ttft_p95_ms": 254.30402324909662,
+    "ttft_p99_ms": 281.55977436035755,
+    "itl_mean_ms": 9.55804924402514,
+    "itl_p50_ms": 9.37427223622698,
+    "itl_p95_ms": 13.641124985851773,
+    "itl_p99_ms": 15.580885431589866,
+    "tpot_mean_ms": 9.559225141953267,
+    "tpot_p50_ms": 9.37540769291087,
+    "tpot_p95_ms": 13.642259077954161,
+    "acceptance_rate": 0.39181146804128403,
+    "acceptance_pos_1": 0.6452053391595262,
+    "acceptance_pos_2": 0.35083467039907146,
+    "acceptance_pos_3": 0.1793943945652545,
+    "mean_acceptance_length": 2.175434404123852
+  },
+  {
+    "dataset": "math500",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 63668,
+    "request_throughput_rps": 0.7652758346384362,
+    "output_throughput_tps": 97.4471636795199,
+    "speedup_vs_baseline": 0.8589685727703342,
+    "latency_mean_ms": 1306.6064661140117,
+    "latency_p50_ms": 1288.1172004999826,
+    "latency_p95_ms": 1831.9392699944729,
+    "latency_p99_ms": 2117.9882052937314,
+    "ttft_mean_ms": 149.97463030007202,
+    "ttft_p50_ms": 150.71226700456464,
+    "ttft_p95_ms": 241.5094937001413,
+    "ttft_p99_ms": 269.5821494147821,
+    "itl_mean_ms": 9.164062260873436,
+    "itl_p50_ms": 9.109615248048863,
+    "itl_p95_ms": 13.162089144880586,
+    "itl_p99_ms": 16.025888632277603,
+    "tpot_mean_ms": 9.165267968224493,
+    "tpot_p50_ms": 9.110725452762747,
+    "tpot_p95_ms": 13.163240382294413,
+    "acceptance_rate": 0.18547955359716675,
+    "acceptance_pos_1": 0.6482415681752667,
+    "acceptance_pos_2": 0.3530196021908331,
+    "acceptance_pos_3": 0.17180743730181608,
+    "mean_acceptance_length": 2.2983568751801675
+  },
+  {
+    "dataset": "math500",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 63731,
+    "request_throughput_rps": 1.0278341396966633,
+    "output_throughput_tps": 131.0097951140161,
+    "speedup_vs_baseline": 1.1548134648445494,
+    "latency_mean_ms": 972.8192880541756,
+    "latency_p50_ms": 951.3454184962029,
+    "latency_p95_ms": 1383.5792890022267,
+    "latency_p99_ms": 1658.223661600423,
+    "ttft_mean_ms": 139.16257599797973,
+    "ttft_p50_ms": 140.1768340037961,
+    "ttft_p95_ms": 243.82559089899584,
+    "ttft_p99_ms": 256.6295453281782,
+    "itl_mean_ms": 6.592793092238339,
+    "itl_p50_ms": 6.475244685019803,
+    "itl_p95_ms": 9.666808936191497,
+    "itl_p99_ms": 11.624004104496413,
+    "tpot_mean_ms": 6.593887856670312,
+    "tpot_p50_ms": 6.476409696829716,
+    "tpot_p95_ms": 9.667887905099029,
+    "acceptance_rate": 0.44804447383243945,
+    "acceptance_pos_1": 0.6541032988024392,
+    "acceptance_pos_2": 0.4218646682830064,
+    "acceptance_pos_3": 0.26816545441187273,
+    "mean_acceptance_length": 2.344133421497318
+  },
+  {
+    "dataset": "math500",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 63668,
+    "request_throughput_rps": 1.0718413712400758,
+    "output_throughput_tps": 136.4839928482263,
+    "speedup_vs_baseline": 1.2030669351074852,
+    "latency_mean_ms": 932.8721254997654,
+    "latency_p50_ms": 924.5265510035097,
+    "latency_p95_ms": 1415.9799854001901,
+    "latency_p99_ms": 1651.539169616298,
+    "ttft_mean_ms": 142.46345183774247,
+    "ttft_p50_ms": 146.44532399688615,
+    "ttft_p95_ms": 239.27307030317024,
+    "ttft_p99_ms": 256.7968881978595,
+    "itl_mean_ms": 6.260414953927999,
+    "itl_p50_ms": 6.1309666968776675,
+    "itl_p95_ms": 10.081072708306051,
+    "itl_p99_ms": 12.369931453538051,
+    "tpot_mean_ms": 6.261500149276782,
+    "tpot_p50_ms": 6.13199001179062,
+    "tpot_p95_ms": 10.082133980709683,
+    "acceptance_rate": 0.2283637249725866,
+    "acceptance_pos_1": 0.6400113714819478,
+    "acceptance_pos_2": 0.3961336961377574,
+    "acceptance_pos_3": 0.24038500588880315,
+    "mean_acceptance_length": 2.598546074808106
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 1319,
+    "output_tokens": 168500,
+    "request_throughput_rps": 0.9045831802614913,
+    "output_throughput_tps": 115.55895820626328,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1105.3625852046998,
+    "latency_p50_ms": 1089.5358249981655,
+    "latency_p95_ms": 1276.3373936999415,
+    "latency_p99_ms": 1319.7392648432285,
+    "ttft_mean_ms": 164.77733511295938,
+    "ttft_p50_ms": 141.7763389981701,
+    "ttft_p95_ms": 332.90244539603,
+    "ttft_p99_ms": 382.7308631595223,
+    "itl_mean_ms": 7.419506798811708,
+    "itl_p50_ms": 7.384766740153826,
+    "itl_p95_ms": 7.931435276391204,
+    "itl_p99_ms": 8.172192777608757,
+    "tpot_mean_ms": 7.420847337932145,
+    "tpot_p50_ms": 7.386314905524496,
+    "tpot_p95_ms": 7.932668355122615,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 1319,
+    "output_tokens": 168418,
+    "request_throughput_rps": 0.5144087930838571,
+    "output_throughput_tps": 65.6828658935535,
+    "speedup_vs_baseline": 0.568392679486734,
+    "latency_mean_ms": 1943.8572523131418,
+    "latency_p50_ms": 1934.8288309993222,
+    "latency_p95_ms": 2365.305382902443,
+    "latency_p99_ms": 2508.7120192195284,
+    "ttft_mean_ms": 178.53159811208383,
+    "ttft_p50_ms": 159.649838002224,
+    "ttft_p95_ms": 355.616971301788,
+    "ttft_p99_ms": 411.2498142791445,
+    "itl_mean_ms": 13.929846219481163,
+    "itl_p50_ms": 13.876137322875465,
+    "itl_p95_ms": 16.931091559855826,
+    "itl_p99_ms": 17.821733611478884,
+    "tpot_mean_ms": 13.931159616121034,
+    "tpot_p50_ms": 13.87738181893644,
+    "tpot_p95_ms": 16.93262100001677,
+    "acceptance_rate": 0.06835341539008165,
+    "acceptance_pos_1": 0.1835041448500868,
+    "acceptance_pos_2": 0.019489077905896417,
+    "acceptance_pos_3": 0.0020670234142617415,
+    "mean_acceptance_length": 1.2050602461702449
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 1319,
+    "output_tokens": 168457,
+    "request_throughput_rps": 0.5735945671017055,
+    "output_throughput_tps": 73.25702804416376,
+    "speedup_vs_baseline": 0.6339363834814603,
+    "latency_mean_ms": 1743.2748062024025,
+    "latency_p50_ms": 1739.7538450004504,
+    "latency_p95_ms": 2099.799393000285,
+    "latency_p99_ms": 2201.536506499797,
+    "ttft_mean_ms": 152.56252339044147,
+    "ttft_p50_ms": 159.50948600038828,
+    "ttft_p95_ms": 246.41776080115957,
+    "ttft_p99_ms": 261.92414222023217,
+    "itl_mean_ms": 12.548829301304638,
+    "itl_p50_ms": 12.529087370075011,
+    "itl_p95_ms": 15.037010089759487,
+    "itl_p99_ms": 15.820388422991211,
+    "tpot_mean_ms": 12.550042645359513,
+    "tpot_p50_ms": 12.530304661416404,
+    "tpot_p95_ms": 15.038574778736802,
+    "acceptance_rate": 0.02749077227835654,
+    "acceptance_pos_1": 0.1718565178355114,
+    "acceptance_pos_2": 0.018334307172683095,
+    "acceptance_pos_3": 0.002037936980717981,
+    "mean_acceptance_length": 1.1924354059484958
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 1319,
+    "output_tokens": 168418,
+    "request_throughput_rps": 0.6339721362775607,
+    "output_throughput_tps": 80.94944597998045,
+    "speedup_vs_baseline": 0.7005034247149607,
+    "latency_mean_ms": 1577.243275963595,
+    "latency_p50_ms": 1558.0307409982197,
+    "latency_p95_ms": 2166.574809700978,
+    "latency_p99_ms": 2443.7027309426035,
+    "ttft_mean_ms": 150.18097911596254,
+    "ttft_p50_ms": 151.24057199864183,
+    "ttft_p95_ms": 254.45422550219513,
+    "ttft_p99_ms": 278.5744088418869,
+    "itl_mean_ms": 11.259462841455447,
+    "itl_p50_ms": 11.12902188976914,
+    "itl_p95_ms": 15.608968344093311,
+    "itl_p99_ms": 17.811628303623607,
+    "tpot_mean_ms": 11.260660320716259,
+    "tpot_p50_ms": 11.130129110223226,
+    "tpot_p95_ms": 15.610202330715275,
+    "acceptance_rate": 0.2788578880838151,
+    "acceptance_pos_1": 0.5339278493601303,
+    "acceptance_pos_2": 0.21577434619629957,
+    "acceptance_pos_3": 0.08687146869501547,
+    "mean_acceptance_length": 1.8365736642514454
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 1319,
+    "output_tokens": 168457,
+    "request_throughput_rps": 0.6414136486780062,
+    "output_throughput_tps": 81.91858909427664,
+    "speedup_vs_baseline": 0.7088899931761125,
+    "latency_mean_ms": 1558.9424188529392,
+    "latency_p50_ms": 1528.7051639970741,
+    "latency_p95_ms": 2191.441884898086,
+    "latency_p99_ms": 2472.421863719936,
+    "ttft_mean_ms": 152.77644680597203,
+    "ttft_p50_ms": 155.94628499820828,
+    "ttft_p95_ms": 255.01179830098403,
+    "ttft_p99_ms": 283.681260881276,
+    "itl_mean_ms": 11.093297977502198,
+    "itl_p50_ms": 10.81332909192518,
+    "itl_p95_ms": 15.859892152744344,
+    "itl_p99_ms": 18.10012607823774,
+    "tpot_mean_ms": 11.094531792060318,
+    "tpot_p50_ms": 10.814917770089934,
+    "tpot_p95_ms": 15.861106548793142,
+    "acceptance_rate": 0.12798578221940327,
+    "acceptance_pos_1": 0.5417164364111695,
+    "acceptance_pos_2": 0.21953528205359357,
+    "acceptance_pos_3": 0.0932478420590025,
+    "mean_acceptance_length": 1.8959004755358229
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 1319,
+    "output_tokens": 168418,
+    "request_throughput_rps": 0.8767773042557421,
+    "output_throughput_tps": 111.95229721618163,
+    "speedup_vs_baseline": 0.9687894296896996,
+    "latency_mean_ms": 1140.4352353411668,
+    "latency_p50_ms": 1110.0252579999506,
+    "latency_p95_ms": 1627.445946199441,
+    "latency_p99_ms": 1797.3359449025884,
+    "ttft_mean_ms": 137.89601687177677,
+    "ttft_p50_ms": 139.47895800083643,
+    "ttft_p95_ms": 232.75966649744078,
+    "ttft_p99_ms": 255.23558505970868,
+    "itl_mean_ms": 7.9109000767037285,
+    "itl_p50_ms": 7.7173600157683415,
+    "itl_p95_ms": 11.63812698740246,
+    "itl_p99_ms": 13.068145703495471,
+    "tpot_mean_ms": 7.912022636927084,
+    "tpot_p50_ms": 7.718454007901284,
+    "tpot_p95_ms": 11.63953219130328,
+    "acceptance_rate": 0.3119374812973321,
+    "acceptance_pos_1": 0.5350344129088691,
+    "acceptance_pos_2": 0.257809083166448,
+    "acceptance_pos_3": 0.14296894781667932,
+    "mean_acceptance_length": 1.9358124438919964
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 1319,
+    "output_tokens": 168457,
+    "request_throughput_rps": 0.8698276010199105,
+    "output_throughput_tps": 111.09063547006147,
+    "speedup_vs_baseline": 0.9613329610654137,
+    "latency_mean_ms": 1149.4913456012362,
+    "latency_p50_ms": 1122.8163880005013,
+    "latency_p95_ms": 1690.7677308932762,
+    "latency_p99_ms": 1912.2929262730754,
+    "ttft_mean_ms": 142.42033671721288,
+    "ttft_p50_ms": 144.59181799611542,
+    "ttft_p95_ms": 238.7138173042331,
+    "ttft_p99_ms": 260.84847668505967,
+    "itl_mean_ms": 7.943716780175663,
+    "itl_p50_ms": 7.752916023642096,
+    "itl_p95_ms": 11.936723438570985,
+    "itl_p99_ms": 13.552413253823115,
+    "tpot_mean_ms": 7.944849474277738,
+    "tpot_p50_ms": 7.753948110208927,
+    "tpot_p95_ms": 11.937846825916434,
+    "acceptance_rate": 0.14634874499637113,
+    "acceptance_pos_1": 0.5336472057746123,
+    "acceptance_pos_2": 0.25006905994403145,
+    "acceptance_pos_3": 0.13272720721586337,
+    "mean_acceptance_length": 2.024441214974598
+  },
+  {
+    "dataset": "humaneval",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 164,
+    "output_tokens": 11593,
+    "request_throughput_rps": 1.3248947198163699,
+    "output_throughput_tps": 93.65551516360473,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 754.6755473965566,
+    "latency_p50_ms": 647.226564502489,
+    "latency_p95_ms": 1367.1402201009187,
+    "latency_p99_ms": 1480.0008050996985,
+    "ttft_mean_ms": 189.28278684156462,
+    "ttft_p50_ms": 179.46013799883076,
+    "ttft_p95_ms": 342.49747914982436,
+    "ttft_p99_ms": 379.2299685735634,
+    "itl_mean_ms": 8.10896273002792,
+    "itl_p50_ms": 7.874335342868083,
+    "itl_p95_ms": 9.866437620078749,
+    "itl_p99_ms": 14.497655275340552,
+    "tpot_mean_ms": 8.11360654911452,
+    "tpot_p50_ms": 7.876261433408698,
+    "tpot_p95_ms": 9.867678297241653,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 164,
+    "output_tokens": 11403,
+    "request_throughput_rps": 0.8482418297936308,
+    "output_throughput_tps": 58.97866820205349,
+    "speedup_vs_baseline": 0.6297404706922488,
+    "latency_mean_ms": 1178.8075736159687,
+    "latency_p50_ms": 1028.3337114997266,
+    "latency_p95_ms": 2207.2047492016277,
+    "latency_p99_ms": 2550.0725987079936,
+    "ttft_mean_ms": 162.98695540243034,
+    "ttft_p50_ms": 142.10437850124435,
+    "ttft_p95_ms": 323.0461064536938,
+    "ttft_p99_ms": 404.1426902490639,
+    "itl_mean_ms": 15.05167389736459,
+    "itl_p50_ms": 15.00903958145443,
+    "itl_p95_ms": 20.248163725045735,
+    "itl_p99_ms": 21.988641405840404,
+    "tpot_mean_ms": 15.056187308266047,
+    "tpot_p50_ms": 15.011203893487512,
+    "tpot_p95_ms": 20.267178071169052,
+    "acceptance_rate": 0.0599447417200014,
+    "acceptance_pos_1": 0.1643059490084986,
+    "acceptance_pos_2": 0.01447906830343091,
+    "acceptance_pos_3": 0.0010492078480747037,
+    "mean_acceptance_length": 1.1798342251600042
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 164,
+    "output_tokens": 10873,
+    "request_throughput_rps": 0.9775628464767565,
+    "output_throughput_tps": 64.81122457159618,
+    "speedup_vs_baseline": 0.6920171701407963,
+    "latency_mean_ms": 1022.8616001402481,
+    "latency_p50_ms": 861.6272605004269,
+    "latency_p95_ms": 2083.6442281001837,
+    "latency_p99_ms": 2187.8588047894846,
+    "ttft_mean_ms": 157.66290578664288,
+    "ttft_p50_ms": 154.99908150104602,
+    "ttft_p95_ms": 238.5054721988126,
+    "ttft_p99_ms": 254.32537110971683,
+    "itl_mean_ms": 13.23754715845204,
+    "itl_p50_ms": 13.002521015674134,
+    "itl_p95_ms": 16.242605144630183,
+    "itl_p99_ms": 17.38454934863588,
+    "tpot_mean_ms": 13.2414967604306,
+    "tpot_p50_ms": 13.004417010839589,
+    "tpot_p95_ms": 16.244994796315652,
+    "acceptance_rate": 0.027065165688917075,
+    "acceptance_pos_1": 0.17458379578246394,
+    "acceptance_pos_2": 0.014095449500554939,
+    "acceptance_pos_3": 0.0007769145394006659,
+    "mean_acceptance_length": 1.1894561598224196
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 164,
+    "output_tokens": 11403,
+    "request_throughput_rps": 1.4279144513142106,
+    "output_throughput_tps": 99.28358834351184,
+    "speedup_vs_baseline": 1.060093344957588,
+    "latency_mean_ms": 700.2340810428306,
+    "latency_p50_ms": 654.7297254983278,
+    "latency_p95_ms": 1298.7420633009606,
+    "latency_p99_ms": 1572.5833712578244,
+    "ttft_mean_ms": 163.544509548978,
+    "ttft_p50_ms": 175.85190250065352,
+    "ttft_p95_ms": 255.46592105038144,
+    "ttft_p99_ms": 272.46247330847837,
+    "itl_mean_ms": 8.18464679387673,
+    "itl_p50_ms": 8.052460137035897,
+    "itl_p95_ms": 11.823310047215534,
+    "itl_p99_ms": 14.873981783527022,
+    "tpot_mean_ms": 8.188209899387681,
+    "tpot_p50_ms": 8.053541158448532,
+    "tpot_p95_ms": 11.828740774528667,
+    "acceptance_rate": 0.5862826458384184,
+    "acceptance_pos_1": 0.7759336099585062,
+    "acceptance_pos_2": 0.5643153526970954,
+    "acceptance_pos_3": 0.4185989748596534,
+    "mean_acceptance_length": 2.758847937515255
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 164,
+    "output_tokens": 10873,
+    "request_throughput_rps": 1.7259982556670077,
+    "output_throughput_tps": 114.43157947480107,
+    "speedup_vs_baseline": 1.2218349263778336,
+    "latency_mean_ms": 579.2890396647502,
+    "latency_p50_ms": 542.1822514981613,
+    "latency_p95_ms": 1124.121437451322,
+    "latency_p99_ms": 1337.6976848527559,
+    "ttft_mean_ms": 153.88535212816083,
+    "ttft_p50_ms": 160.26985099961166,
+    "ttft_p95_ms": 249.8991386499256,
+    "ttft_p99_ms": 257.3446220774349,
+    "itl_mean_ms": 7.2881835467627765,
+    "itl_p50_ms": 6.791492409145974,
+    "itl_p95_ms": 11.28842372509098,
+    "itl_p99_ms": 14.967644061684823,
+    "tpot_mean_ms": 7.292224562039036,
+    "tpot_p50_ms": 6.792956900041677,
+    "tpot_p95_ms": 11.300806915223802,
+    "acceptance_rate": 0.3260095527572731,
+    "acceptance_pos_1": 0.7689969604863222,
+    "acceptance_pos_2": 0.5525835866261398,
+    "acceptance_pos_3": 0.40091185410334346,
+    "mean_acceptance_length": 3.282066869300912
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 164,
+    "output_tokens": 11403,
+    "request_throughput_rps": 1.8838722899407694,
+    "output_throughput_tps": 130.98655928167435,
+    "speedup_vs_baseline": 1.3985995277785492,
+    "latency_mean_ms": 530.7444518413565,
+    "latency_p50_ms": 495.157746001496,
+    "latency_p95_ms": 905.6995584996912,
+    "latency_p99_ms": 1086.2262809240203,
+    "ttft_mean_ms": 146.0434188045882,
+    "ttft_p50_ms": 152.46622500490048,
+    "ttft_p95_ms": 236.91825044916186,
+    "ttft_p99_ms": 255.82899200926477,
+    "itl_mean_ms": 6.104835246645276,
+    "itl_p50_ms": 5.68536281673191,
+    "itl_p95_ms": 8.98480516036281,
+    "itl_p99_ms": 11.694451736160172,
+    "tpot_mean_ms": 6.108340038664795,
+    "tpot_p50_ms": 5.68948486237622,
+    "tpot_p95_ms": 8.989236841355433,
+    "acceptance_rate": 0.6043326693227091,
+    "acceptance_pos_1": 0.7711653386454184,
+    "acceptance_pos_2": 0.5893924302788844,
+    "acceptance_pos_3": 0.4524402390438247,
+    "mean_acceptance_length": 2.812998007968128
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 164,
+    "output_tokens": 10873,
+    "request_throughput_rps": 2.193269418244654,
+    "output_throughput_tps": 145.4110877108178,
+    "speedup_vs_baseline": 1.5526163884401514,
+    "latency_mean_ms": 455.86265411590676,
+    "latency_p50_ms": 409.3146830055048,
+    "latency_p95_ms": 852.2122731519629,
+    "latency_p99_ms": 1123.0342373249005,
+    "ttft_mean_ms": 143.6314728172608,
+    "ttft_p50_ms": 148.64660549210384,
+    "ttft_p95_ms": 241.35335180908442,
+    "ttft_p99_ms": 261.36555448989384,
+    "itl_mean_ms": 5.509904282925857,
+    "itl_p50_ms": 5.137632333503879,
+    "itl_p95_ms": 9.656399889036864,
+    "itl_p99_ms": 11.480592018691826,
+    "tpot_mean_ms": 5.513652760595837,
+    "tpot_p50_ms": 5.139975778089672,
+    "tpot_p95_ms": 9.664061358919506,
+    "acceptance_rate": 0.3497128270580728,
+    "acceptance_pos_1": 0.7345245692405871,
+    "acceptance_pos_2": 0.5520102105934908,
+    "acceptance_pos_3": 0.3937460114869177,
+    "mean_acceptance_length": 3.4479897894065092
+  },
+  {
+    "dataset": "mbpp",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 500,
+    "output_tokens": 61467,
+    "request_throughput_rps": 0.9566788994866338,
+    "output_throughput_tps": 117.60836382948983,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1045.163285173956,
+    "latency_p50_ms": 1051.2518215000455,
+    "latency_p95_ms": 1252.2429954937252,
+    "latency_p99_ms": 1313.4473929634257,
+    "ttft_mean_ms": 170.204570235961,
+    "ttft_p50_ms": 147.96629649936222,
+    "ttft_p95_ms": 343.0072944531275,
+    "ttft_p99_ms": 394.77327711974795,
+    "itl_mean_ms": 7.1850096596128905,
+    "itl_p50_ms": 7.149241377965513,
+    "itl_p95_ms": 7.642400291320152,
+    "itl_p99_ms": 7.887752700210706,
+    "tpot_mean_ms": 7.1880918593224585,
+    "tpot_p50_ms": 7.1504635433104955,
+    "tpot_p95_ms": 7.643652335044196,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 61643,
+    "request_throughput_rps": 0.5463233439436157,
+    "output_throughput_tps": 67.3540197814326,
+    "speedup_vs_baseline": 0.5726975326268746,
+    "latency_mean_ms": 1830.2107265740342,
+    "latency_p50_ms": 1857.0996774942614,
+    "latency_p95_ms": 2291.14192195193,
+    "latency_p99_ms": 2455.683664069438,
+    "ttft_mean_ms": 181.87522897598683,
+    "ttft_p50_ms": 156.96376850246452,
+    "ttft_p95_ms": 374.19471385073825,
+    "ttft_p99_ms": 413.4018152903445,
+    "itl_mean_ms": 13.36878509883122,
+    "itl_p50_ms": 13.397268279545893,
+    "itl_p95_ms": 16.285712324037817,
+    "itl_p99_ms": 17.16560508802913,
+    "tpot_mean_ms": 13.37178172838754,
+    "tpot_p50_ms": 13.398526082703144,
+    "tpot_p95_ms": 16.287178615374877,
+    "acceptance_rate": 0.0651825815270455,
+    "acceptance_pos_1": 0.18390939269673892,
+    "acceptance_pos_2": 0.011013473930872876,
+    "acceptance_pos_3": 0.0006248779535247023,
+    "mean_acceptance_length": 1.1955477445811364
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 61199,
+    "request_throughput_rps": 0.6112544218481429,
+    "output_throughput_tps": 74.81631872536899,
+    "speedup_vs_baseline": 0.636147942962957,
+    "latency_mean_ms": 1635.86529272798,
+    "latency_p50_ms": 1672.3464854994745,
+    "latency_p95_ms": 2016.2770743999317,
+    "latency_p99_ms": 2129.984438859737,
+    "ttft_mean_ms": 152.66953481407472,
+    "ttft_p50_ms": 159.74267549972865,
+    "ttft_p95_ms": 244.47033859860304,
+    "ttft_p99_ms": 260.42180475840723,
+    "itl_mean_ms": 12.100660442193382,
+    "itl_p50_ms": 12.01978580708575,
+    "itl_p95_ms": 14.406062782671242,
+    "itl_p99_ms": 15.092129389210932,
+    "tpot_mean_ms": 12.10357721618688,
+    "tpot_p50_ms": 12.02091915748017,
+    "tpot_p95_ms": 14.40738571140975,
+    "acceptance_rate": 0.028364658688842314,
+    "acceptance_pos_1": 0.18660277646316453,
+    "acceptance_pos_2": 0.011358258400378608,
+    "acceptance_pos_3": 0.0005718567597412841,
+    "mean_acceptance_length": 1.1985526108218962
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 61643,
+    "request_throughput_rps": 0.8389172370608934,
+    "output_throughput_tps": 103.4267504882893,
+    "speedup_vs_baseline": 0.8794166258297648,
+    "latency_mean_ms": 1191.9062847682944,
+    "latency_p50_ms": 1208.0225355002767,
+    "latency_p95_ms": 1663.9658735984995,
+    "latency_p99_ms": 1887.62577743968,
+    "ttft_mean_ms": 151.1433905701415,
+    "ttft_p50_ms": 151.9770585000515,
+    "ttft_p95_ms": 261.6679305512662,
+    "ttft_p99_ms": 283.6371392303044,
+    "itl_mean_ms": 8.617537094703247,
+    "itl_p50_ms": 8.440096641743008,
+    "itl_p95_ms": 12.05171109251937,
+    "itl_p99_ms": 14.760888378997743,
+    "tpot_mean_ms": 8.620249662427044,
+    "tpot_p50_ms": 8.441339763779492,
+    "tpot_p95_ms": 12.052845620474532,
+    "acceptance_rate": 0.4696401115367576,
+    "acceptance_pos_1": 0.6867328590415136,
+    "acceptance_pos_2": 0.4438276913454773,
+    "acceptance_pos_3": 0.278359784223282,
+    "mean_acceptance_length": 2.408920334610273
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 61199,
+    "request_throughput_rps": 0.943455812538664,
+    "output_throughput_tps": 115.47710454310739,
+    "speedup_vs_baseline": 0.9818783357153716,
+    "latency_mean_ms": 1059.8270873299334,
+    "latency_p50_ms": 1066.4042809985403,
+    "latency_p95_ms": 1511.5341901015197,
+    "latency_p99_ms": 1678.6425156925543,
+    "ttft_mean_ms": 149.29563258778944,
+    "ttft_p50_ms": 151.24238849966787,
+    "ttft_p95_ms": 254.46768340189007,
+    "ttft_p99_ms": 276.2093989884306,
+    "itl_mean_ms": 7.8246956861011725,
+    "itl_p50_ms": 7.491708066904327,
+    "itl_p95_ms": 11.13285018701698,
+    "itl_p99_ms": 19.920353018048132,
+    "tpot_mean_ms": 7.827419804285574,
+    "tpot_p50_ms": 7.492821476362439,
+    "tpot_p95_ms": 11.13404580474156,
+    "acceptance_rate": 0.24501954356323288,
+    "acceptance_pos_1": 0.6746690203000882,
+    "acceptance_pos_2": 0.42497793468667255,
+    "acceptance_pos_3": 0.2500882612533098,
+    "mean_acceptance_length": 2.71513680494263
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 61643,
+    "request_throughput_rps": 1.1901763850957827,
+    "output_throughput_tps": 146.73208581291865,
+    "speedup_vs_baseline": 1.247633085225578,
+    "latency_mean_ms": 840.1123063238338,
+    "latency_p50_ms": 843.1062914969516,
+    "latency_p95_ms": 1144.961133646211,
+    "latency_p99_ms": 1246.0605967608715,
+    "ttft_mean_ms": 133.81045833000098,
+    "ttft_p50_ms": 131.7662695000763,
+    "ttft_p95_ms": 238.02317654808573,
+    "ttft_p99_ms": 263.0950807044428,
+    "itl_mean_ms": 5.951950751595078,
+    "itl_p50_ms": 5.73346276769729,
+    "itl_p95_ms": 8.067283917319386,
+    "itl_p99_ms": 14.154150555114027,
+    "tpot_mean_ms": 5.954451005112758,
+    "tpot_p50_ms": 5.734595043312457,
+    "tpot_p95_ms": 8.068427130726949,
+    "acceptance_rate": 0.5237361111111111,
+    "acceptance_pos_1": 0.7102916666666667,
+    "acceptance_pos_2": 0.510625,
+    "acceptance_pos_3": 0.35029166666666667,
+    "mean_acceptance_length": 2.5712083333333333
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 61199,
+    "request_throughput_rps": 1.292868654843097,
+    "output_throughput_tps": 158.2445376154854,
+    "speedup_vs_baseline": 1.3455211216517766,
+    "latency_mean_ms": 773.3765653301089,
+    "latency_p50_ms": 777.3538839974208,
+    "latency_p95_ms": 1069.8221424441717,
+    "latency_p99_ms": 1181.5079208668612,
+    "ttft_mean_ms": 144.87902384001063,
+    "ttft_p50_ms": 148.97060450311983,
+    "ttft_p95_ms": 240.50168554604164,
+    "ttft_p99_ms": 261.4171255609835,
+    "itl_mean_ms": 5.4609269159301945,
+    "itl_p50_ms": 5.171082055140708,
+    "itl_p95_ms": 7.696279004271422,
+    "itl_p99_ms": 16.670999059686896,
+    "tpot_mean_ms": 5.463527012393008,
+    "tpot_p50_ms": 5.172086559067843,
+    "tpot_p95_ms": 7.697418791287318,
+    "acceptance_rate": 0.2885384335766015,
+    "acceptance_pos_1": 0.6823742415345468,
+    "acceptance_pos_2": 0.46951458211000197,
+    "acceptance_pos_3": 0.3062732432961441,
+    "mean_acceptance_length": 3.0197690350362105
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 0.9303956789559812,
+    "output_throughput_tps": 116.89258711483208,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1074.6847601811623,
+    "latency_p50_ms": 1056.2882739977795,
+    "latency_p95_ms": 1319.7490790018492,
+    "latency_p99_ms": 1393.648768519415,
+    "ttft_mean_ms": 160.69353549364678,
+    "ttft_p50_ms": 136.07312599924626,
+    "ttft_p95_ms": 327.46918089833343,
+    "ttft_p99_ms": 382.2414645081152,
+    "itl_mean_ms": 7.309607731920508,
+    "itl_p50_ms": 7.198581275588367,
+    "itl_p95_ms": 8.339637732306286,
+    "itl_p99_ms": 9.805146286736829,
+    "tpot_mean_ms": 7.3133077895495875,
+    "tpot_p50_ms": 7.1998062007842805,
+    "tpot_p95_ms": 8.34087489329106,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 0.5241432036444518,
+    "output_throughput_tps": 65.85204174787981,
+    "speedup_vs_baseline": 0.5633551568431672,
+    "latency_mean_ms": 1907.7480007938448,
+    "latency_p50_ms": 1930.5097770011344,
+    "latency_p95_ms": 2463.4897702966555,
+    "latency_p99_ms": 2646.199334648336,
+    "ttft_mean_ms": 175.24922163756855,
+    "ttft_p50_ms": 140.2629744989099,
+    "ttft_p95_ms": 376.67113924690057,
+    "ttft_p99_ms": 423.97999386870646,
+    "itl_mean_ms": 13.833307178800341,
+    "itl_p50_ms": 13.946147039380268,
+    "itl_p95_ms": 17.445409657068915,
+    "itl_p99_ms": 18.298039104984053,
+    "tpot_mean_ms": 13.837227284098185,
+    "tpot_p50_ms": 13.947568574815339,
+    "tpot_p95_ms": 17.446681594068878,
+    "acceptance_rate": 0.060423199037646175,
+    "acceptance_pos_1": 0.16210140211796722,
+    "acceptance_pos_2": 0.017630006507720522,
+    "acceptance_pos_3": 0.0015381884872507838,
+    "mean_acceptance_length": 1.1812695971129386
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 0.583168838359363,
+    "output_throughput_tps": 73.26787492937446,
+    "speedup_vs_baseline": 0.6267965893970513,
+    "latency_mean_ms": 1714.6447665063306,
+    "latency_p50_ms": 1721.8866950006486,
+    "latency_p95_ms": 2135.267027699956,
+    "latency_p99_ms": 2242.278299470854,
+    "ttft_mean_ms": 139.6615898878281,
+    "ttft_p50_ms": 138.0474095003592,
+    "ttft_p95_ms": 237.62467689830373,
+    "ttft_p99_ms": 245.49128495971672,
+    "itl_mean_ms": 12.686621501440076,
+    "itl_p50_ms": 12.566233303133897,
+    "itl_p95_ms": 15.546194449997866,
+    "itl_p99_ms": 16.16330862709806,
+    "tpot_mean_ms": 12.6899270927419,
+    "tpot_p50_ms": 12.567349094479857,
+    "tpot_p95_ms": 15.547512893692872,
+    "acceptance_rate": 0.02348325183475921,
+    "acceptance_pos_1": 0.14881334188582424,
+    "acceptance_pos_2": 0.01463642194880168,
+    "acceptance_pos_3": 0.0008746865706455186,
+    "mean_acceptance_length": 1.1643827628433145
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 0.6809063012234507,
+    "output_throughput_tps": 85.5473654199613,
+    "speedup_vs_baseline": 0.7318459410597347,
+    "latency_mean_ms": 1468.511645206172,
+    "latency_p50_ms": 1454.6153324990883,
+    "latency_p95_ms": 2127.0678361986943,
+    "latency_p99_ms": 2281.048237952054,
+    "ttft_mean_ms": 159.72944469986032,
+    "ttft_p50_ms": 168.77598499922897,
+    "ttft_p95_ms": 265.74251724632626,
+    "ttft_p99_ms": 281.86643974164326,
+    "itl_mean_ms": 10.654948740516124,
+    "itl_p50_ms": 10.288291291322736,
+    "itl_p95_ms": 15.705922906309695,
+    "itl_p99_ms": 16.95837595336085,
+    "tpot_mean_ms": 10.6584103940507,
+    "tpot_p50_ms": 10.289786913365493,
+    "tpot_p95_ms": 15.707090863808236,
+    "acceptance_rate": 0.3230895107183767,
+    "acceptance_pos_1": 0.5774177712322042,
+    "acceptance_pos_2": 0.27245949926362295,
+    "acceptance_pos_3": 0.1193912616593029,
+    "mean_acceptance_length": 1.9692685321551302
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 0.7093667936140545,
+    "output_throughput_tps": 89.12307053268577,
+    "speedup_vs_baseline": 0.7624356063326214,
+    "latency_mean_ms": 1409.5885398314294,
+    "latency_p50_ms": 1421.8769274993974,
+    "latency_p95_ms": 2124.1017438478584,
+    "latency_p99_ms": 2515.584632036989,
+    "ttft_mean_ms": 146.91454585622523,
+    "ttft_p50_ms": 142.44566849811235,
+    "ttft_p95_ms": 259.6700968468212,
+    "ttft_p99_ms": 287.97775861195987,
+    "itl_mean_ms": 10.234430485377915,
+    "itl_p50_ms": 10.088829999998364,
+    "itl_p95_ms": 15.341923362584064,
+    "itl_p99_ms": 19.70016743978345,
+    "tpot_mean_ms": 10.238094617695932,
+    "tpot_p50_ms": 10.089903972448514,
+    "tpot_p95_ms": 15.343091349977053,
+    "acceptance_rate": 0.14626535131870344,
+    "acceptance_pos_1": 0.5742903160861688,
+    "acceptance_pos_2": 0.2700825447956513,
+    "acceptance_pos_3": 0.1106301590497282,
+    "mean_acceptance_length": 2.023857459230924
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 0.9638367060855711,
+    "output_throughput_tps": 121.09403416082594,
+    "speedup_vs_baseline": 1.035942801418763,
+    "latency_mean_ms": 1037.4081412996475,
+    "latency_p50_ms": 1001.3265734996821,
+    "latency_p95_ms": 1565.5059430991967,
+    "latency_p99_ms": 1781.7731402603386,
+    "ttft_mean_ms": 141.7672610684349,
+    "ttft_p50_ms": 145.46368399896892,
+    "ttft_p95_ms": 240.4056666531687,
+    "ttft_p99_ms": 261.25688084066496,
+    "itl_mean_ms": 7.313367654694055,
+    "itl_p50_ms": 6.893495897653395,
+    "itl_p95_ms": 11.711175241387847,
+    "itl_p99_ms": 16.168680668779416,
+    "tpot_mean_ms": 7.316552068142107,
+    "tpot_p50_ms": 6.894642385820525,
+    "tpot_p95_ms": 11.712829905159229,
+    "acceptance_rate": 0.3706431026018655,
+    "acceptance_pos_1": 0.5900483904902167,
+    "acceptance_pos_2": 0.3288449400378708,
+    "acceptance_pos_3": 0.19303597727750893,
+    "mean_acceptance_length": 2.1119293078055965
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 0.9729636803231215,
+    "output_throughput_tps": 122.24072438659618,
+    "speedup_vs_baseline": 1.045752578531864,
+    "latency_mean_ms": 1027.6755360433526,
+    "latency_p50_ms": 1009.7822754978552,
+    "latency_p95_ms": 1596.3315117551247,
+    "latency_p99_ms": 2005.884019333607,
+    "ttft_mean_ms": 141.24648624401743,
+    "ttft_p50_ms": 139.84237199474592,
+    "ttft_p95_ms": 255.56160285341318,
+    "ttft_p99_ms": 265.59654699100065,
+    "itl_mean_ms": 7.283154806994812,
+    "itl_p50_ms": 6.8463791023933425,
+    "itl_p95_ms": 11.787724323593851,
+    "itl_p99_ms": 15.836048513204252,
+    "tpot_mean_ms": 7.286351986039958,
+    "tpot_p50_ms": 6.847493661481403,
+    "tpot_p95_ms": 11.788784734661597,
+    "acceptance_rate": 0.1742194735272419,
+    "acceptance_pos_1": 0.5641138336446544,
+    "acceptance_pos_2": 0.30238435336776176,
+    "acceptance_pos_3": 0.17075046698165036,
+    "mean_acceptance_length": 2.2195363146906932
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 5000,
+    "output_tokens": 636089,
+    "request_throughput_rps": 0.4938431604479032,
+    "output_throughput_tps": 62.825640417229266,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 2024.8159037780163,
+    "latency_p50_ms": 1899.8980629985454,
+    "latency_p95_ms": 3256.729267303672,
+    "latency_p99_ms": 3405.7137782224895,
+    "ttft_mean_ms": 166.95593686518114,
+    "ttft_p50_ms": 163.98859200126026,
+    "ttft_p95_ms": 289.8830948040995,
+    "ttft_p99_ms": 336.2988253685762,
+    "itl_mean_ms": 14.712897944777563,
+    "itl_p50_ms": 13.827794673245338,
+    "itl_p95_ms": 23.864679124793426,
+    "itl_p99_ms": 24.76342741740255,
+    "tpot_mean_ms": 14.714261969753975,
+    "tpot_p50_ms": 13.829047236223541,
+    "tpot_p95_ms": 23.866155100384233,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 5000,
+    "output_tokens": 635846,
+    "request_throughput_rps": 0.4093959318736204,
+    "output_throughput_tps": 52.06255313962281,
+    "speedup_vs_baseline": 0.82868320631309,
+    "latency_mean_ms": 2442.487533897996,
+    "latency_p50_ms": 2383.8188690006064,
+    "latency_p95_ms": 3377.258205899534,
+    "latency_p99_ms": 3705.6293327211833,
+    "ttft_mean_ms": 168.35537181059328,
+    "ttft_p50_ms": 170.54413749974628,
+    "ttft_p95_ms": 252.9078775001381,
+    "ttft_p99_ms": 299.23500963023804,
+    "itl_mean_ms": 18.0134018722681,
+    "itl_p50_ms": 17.54899170471861,
+    "itl_p95_ms": 24.705050003148795,
+    "itl_p99_ms": 27.191358835904875,
+    "tpot_mean_ms": 18.01468435680956,
+    "tpot_p50_ms": 17.550207216533355,
+    "tpot_p95_ms": 24.706307322046488,
+    "acceptance_rate": 0.07081888392688648,
+    "acceptance_pos_1": 0.18361221407447353,
+    "acceptance_pos_2": 0.02588322896495871,
+    "acceptance_pos_3": 0.002961208741227203,
+    "mean_acceptance_length": 1.2124566517806594
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 5000,
+    "output_tokens": 636040,
+    "request_throughput_rps": 0.4019325528965641,
+    "output_throughput_tps": 51.12903618886613,
+    "speedup_vs_baseline": 0.8138243533900298,
+    "latency_mean_ms": 2487.8614642295834,
+    "latency_p50_ms": 2432.8269440011354,
+    "latency_p95_ms": 3443.123125650527,
+    "latency_p99_ms": 3724.6332480803176,
+    "ttft_mean_ms": 172.7844437341555,
+    "ttft_p50_ms": 173.04360800153518,
+    "ttft_p95_ms": 258.7694342015311,
+    "ttft_p99_ms": 299.8475944703024,
+    "itl_mean_ms": 18.332019165246155,
+    "itl_p50_ms": 17.85771458268332,
+    "itl_p95_ms": 25.186547632286487,
+    "itl_p99_ms": 27.352591655754214,
+    "tpot_mean_ms": 18.33325328096626,
+    "tpot_p50_ms": 17.85883874804062,
+    "tpot_p95_ms": 25.18777078149849,
+    "acceptance_rate": 0.03064899424451109,
+    "acceptance_pos_1": 0.1849223980874338,
+    "acceptance_pos_2": 0.025778901811864367,
+    "acceptance_pos_3": 0.0031517369145264325,
+    "mean_acceptance_length": 1.2145429597115776
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 5000,
+    "output_tokens": 635846,
+    "request_throughput_rps": 0.4906630866625573,
+    "output_throughput_tps": 62.397232200408084,
+    "speedup_vs_baseline": 0.9931809972174402,
+    "latency_mean_ms": 2037.9430946188047,
+    "latency_p50_ms": 2007.9161120011122,
+    "latency_p95_ms": 2756.5027899490815,
+    "latency_p99_ms": 3044.030925800253,
+    "ttft_mean_ms": 185.9597727056098,
+    "ttft_p50_ms": 186.23321399718407,
+    "ttft_p95_ms": 279.4342198045342,
+    "ttft_p99_ms": 323.5009970532701,
+    "itl_mean_ms": 14.673263611031553,
+    "itl_p50_ms": 14.47747526376982,
+    "itl_p95_ms": 19.803011401590926,
+    "itl_p99_ms": 21.86518714472933,
+    "tpot_mean_ms": 14.674543237540032,
+    "tpot_p50_ms": 14.478983830686477,
+    "tpot_p95_ms": 19.804314228371027,
+    "acceptance_rate": 0.3113198824188073,
+    "acceptance_pos_1": 0.572229825914399,
+    "acceptance_pos_2": 0.26063458484348756,
+    "acceptance_pos_3": 0.10109523649853532,
+    "mean_acceptance_length": 1.933959647256422
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 5000,
+    "output_tokens": 636040,
+    "request_throughput_rps": 0.5191022628219358,
+    "output_throughput_tps": 66.03396064905282,
+    "speedup_vs_baseline": 1.0510670517724432,
+    "latency_mean_ms": 1925.9773102255406,
+    "latency_p50_ms": 1903.7692794991017,
+    "latency_p95_ms": 2635.482599895113,
+    "latency_p99_ms": 2962.525904481372,
+    "ttft_mean_ms": 180.70931879136043,
+    "ttft_p50_ms": 182.9802385000221,
+    "ttft_p95_ms": 272.8241580512986,
+    "ttft_p99_ms": 313.5787184005314,
+    "itl_mean_ms": 13.822185940257995,
+    "itl_p50_ms": 13.62802470079417,
+    "itl_p95_ms": 18.901379390565385,
+    "itl_p99_ms": 21.488975903824134,
+    "tpot_mean_ms": 13.823438931487352,
+    "tpot_p50_ms": 13.629216417345067,
+    "tpot_p95_ms": 18.902619697261734,
+    "acceptance_rate": 0.14394009500546442,
+    "acceptance_pos_1": 0.5769995228510126,
+    "acceptance_pos_2": 0.26659988687461095,
+    "acceptance_pos_3": 0.10386364282712356,
+    "mean_acceptance_length": 2.007580665038251
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 5000,
+    "output_tokens": 635846,
+    "request_throughput_rps": 0.6141321767148702,
+    "output_throughput_tps": 78.09869760708868,
+    "speedup_vs_baseline": 1.2431022921283417,
+    "latency_mean_ms": 1628.2056611145947,
+    "latency_p50_ms": 1592.831803496665,
+    "latency_p95_ms": 2287.4158730010095,
+    "latency_p99_ms": 2571.3276440004856,
+    "ttft_mean_ms": 171.98177616532774,
+    "ttft_p50_ms": 172.63183150498662,
+    "ttft_p95_ms": 258.64595805287536,
+    "ttft_p99_ms": 302.3406934656669,
+    "itl_mean_ms": 11.538623067112681,
+    "itl_p50_ms": 11.238307917321478,
+    "itl_p95_ms": 16.24375355278012,
+    "itl_p99_ms": 18.46682838556656,
+    "tpot_mean_ms": 11.5398444087042,
+    "tpot_p50_ms": 11.239705511848223,
+    "tpot_p95_ms": 16.245154050035826,
+    "acceptance_rate": 0.3240773825244086,
+    "acceptance_pos_1": 0.5557454318356237,
+    "acceptance_pos_2": 0.28176742365161084,
+    "acceptance_pos_3": 0.13471929208599118,
+    "mean_acceptance_length": 1.9722321475732256
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 5000,
+    "output_tokens": 636040,
+    "request_throughput_rps": 0.6186701811110291,
+    "output_throughput_tps": 78.69979639877178,
+    "speedup_vs_baseline": 1.2526700225595981,
+    "latency_mean_ms": 1616.259658270469,
+    "latency_p50_ms": 1584.57970800373,
+    "latency_p95_ms": 2290.956548342365,
+    "latency_p99_ms": 2594.4233495666413,
+    "ttft_mean_ms": 176.71817260388633,
+    "ttft_p50_ms": 176.76159800612368,
+    "ttft_p95_ms": 263.7752173024638,
+    "ttft_p99_ms": 303.1083872308955,
+    "itl_mean_ms": 11.399967330431618,
+    "itl_p50_ms": 11.174435653507444,
+    "itl_p95_ms": 16.282335170467917,
+    "itl_p99_ms": 18.624601763757692,
+    "tpot_mean_ms": 11.401195973910607,
+    "tpot_p50_ms": 11.175635023563679,
+    "tpot_p95_ms": 16.283561415453793,
+    "acceptance_rate": 0.15448514765416219,
+    "acceptance_pos_1": 0.5533425372792525,
+    "acceptance_pos_2": 0.28025688169439844,
+    "acceptance_pos_3": 0.13306681148704796,
+    "mean_acceptance_length": 2.0813960335791357
+  }
+]

benchmarks/bs1/run_config.json

@@ -0,0 +1,108 @@
+{
+  "args": {
+    "base_model": "google/t5gemma-2-1b-1b",
+    "dflash_model": "/mnt/d/Projects/dflash_t5/dflash-output/dflash_t5gemma2_mixture_200k/checkpoints/5",
+    "dflare_model": "/mnt/d/Projects/dflash_t5/dflash-output/dflare_t5gemma2_mixture_200k/checkpoints/5",
+    "dspark_model": "/mnt/d/Projects/dflash_t5/dflash-output/dspark_t5gemma2_mixture_200k/checkpoints/5",
+    "speculative_token_counts": [
+      3,
+      7
+    ],
+    "methods": [
+      "dflash",
+      "dflare",
+      "dspark"
+    ],
+    "datasets": [
+      "math500",
+      "gsm8k",
+      "humaneval",
+      "mbpp",
+      "mt_bench",
+      "ultrachat"
+    ],
+    "num_prompts": 0,
+    "ultrachat_num_prompts": 5000,
+    "warmup": 8,
+    "concurrency": 1,
+    "max_num_seqs": 1,
+    "max_num_batched_tokens": 4096,
+    "max_model_len": 2048,
+    "max_tokens": 128,
+    "gpu_memory_utilization": 0.85,
+    "host": "127.0.0.1",
+    "port": 8011,
+    "temperature": 0.0,
+    "top_p": 1.0,
+    "seed": 0,
+    "timeout_s": 180.0,
+    "server_timeout_s": 600.0,
+    "enforce_eager": false,
+    "include_baseline": true
+  },
+  "runs": [
+    {
+      "name": "baseline",
+      "checkpoint": null,
+      "num_speculative_tokens": null
+    },
+    {
+      "name": "dflash_k3",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dflash_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 3
+    },
+    {
+      "name": "dflash_k7",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dflash_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 7
+    },
+    {
+      "name": "dflare_k3",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dflare_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 3
+    },
+    {
+      "name": "dflare_k7",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dflare_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 7
+    },
+    {
+      "name": "dspark_k3",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dspark_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 3
+    },
+    {
+      "name": "dspark_k7",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dspark_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 7
+    }
+  ],
+  "prompt_sets": {
+    "math500": {
+      "count": 500,
+      "sha256": "78a67699bc700588d0e23da6457ea3afc00d5635897d456a6245f1e964449c4a"
+    },
+    "gsm8k": {
+      "count": 1319,
+      "sha256": "d591bbef95c9d5b17f6f26999bde24def2f6fbf25d7e20953b9ac89c65ffbb8e"
+    },
+    "humaneval": {
+      "count": 164,
+      "sha256": "de1066c5b81dc0f99eab434315cee2c3c68c5a522b8c25d53c73f9c9042b44c8"
+    },
+    "mbpp": {
+      "count": 500,
+      "sha256": "c596be7f77834627fd0a05755691e30dbf4f6724425842de6d03348614e5fecd"
+    },
+    "mt_bench": {
+      "count": 160,
+      "sha256": "8761bfece58bd26e5b93c33e9aad480728ea9950a7d257aaff267db5c7b98010"
+    },
+    "ultrachat": {
+      "count": 5000,
+      "sha256": "5dac9e9dc0d63372c701dbebbb18f3f22a568e9b62cc0ff1b4fb88fc41bf5bf7"
+    }
+  },
+  "benchmark_fingerprint": "436353e1f7cd491994a7f381976fb001a63b77a4fea4411f0b645d604defc27e",
+  "output_dir": "/mnt/d/Projects/dflash_t5/benchmark-results/t5gemma2_speculators_suite_bs1_fixed"
+}

benchmarks/bs4/comparison.md

@@ -0,0 +1,44 @@
+| Dataset | Run | Req | Output tok/s | Speedup | Latency mean/p95 ms | TTFT mean/p95 ms | ITL mean/p95 ms | TPOT mean/p95 ms | Accept rate | Accept p1/p2/p3 | Accept len |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| math500 | baseline | 500 | 266.74 | 1.000x | 1901.95/2310.23 | 211.01/379.08 | 13.40/16.31 | 13.40/16.31 | — | -/-/- | — |
+| math500 | dflash_k3 | 500 | 185.18 | 0.694x | 2740.52/3396.06 | 232.73/422.78 | 19.89/24.71 | 19.90/24.71 | 6.29% | 16.64%/1.97%/0.24% | 1.189 |
+| math500 | dflash_k7 | 500 | 188.44 | 0.706x | 2690.23/3411.27 | 223.15/410.68 | 19.61/24.83 | 19.61/24.83 | 2.74% | 16.86%/2.02%/0.26% | 1.192 |
+| math500 | dflare_k3 | 500 | 217.77 | 0.816x | 2328.94/3331.75 | 279.51/506.23 | 16.27/23.21 | 16.27/23.21 | 39.35% | 64.89%/35.28%/17.89% | 2.181 |
+| math500 | dflare_k7 | 500 | 225.99 | 0.847x | 2249.20/3338.34 | 263.03/475.01 | 15.76/24.25 | 15.76/24.25 | 18.69% | 64.67%/35.48%/17.53% | 2.309 |
+| math500 | dspark_k3 | 500 | 286.34 | 1.073x | 1774.14/2512.93 | 237.01/434.45 | 12.18/17.88 | 12.18/17.88 | 44.99% | 65.67%/42.25%/27.05% | 2.350 |
+| math500 | dspark_k7 | 500 | 294.75 | 1.105x | 1725.25/2646.27 | 247.76/461.03 | 11.72/18.88 | 11.72/18.88 | 22.89% | 63.98%/39.64%/24.22% | 2.602 |
+| gsm8k | baseline | 1319 | 280.58 | 1.000x | 1819.24/2167.76 | 217.05/398.45 | 12.64/15.22 | 12.64/15.22 | — | -/-/- | — |
+| gsm8k | dflash_k3 | 1319 | 188.79 | 0.673x | 2702.38/3317.10 | 235.61/424.62 | 19.48/24.18 | 19.48/24.19 | 6.81% | 18.37%/1.86%/0.20% | 1.204 |
+| gsm8k | dflash_k7 | 1319 | 188.46 | 0.672x | 2705.30/3477.26 | 301.18/834.45 | 18.98/23.59 | 18.98/23.59 | 2.74% | 17.14%/1.81%/0.20% | 1.192 |
+| gsm8k | dflare_k3 | 1319 | 201.03 | 0.716x | 2536.44/3534.03 | 258.19/484.66 | 17.98/25.82 | 17.98/25.82 | 27.95% | 53.64%/21.51%/8.69% | 1.838 |
+| gsm8k | dflare_k7 | 1319 | 154.41 | 0.550x | 3304.27/5885.10 | 1205.81/4089.58 | 16.56/27.05 | 16.56/27.06 | 12.83% | 54.22%/21.89%/9.37% | 1.898 |
+| gsm8k | dspark_k3 | 1319 | 267.82 | 0.954x | 1904.86/2717.59 | 232.41/436.56 | 13.20/19.22 | 13.20/19.23 | 31.15% | 53.33%/25.75%/14.35% | 1.934 |
+| gsm8k | dspark_k7 | 1319 | 256.27 | 0.913x | 1991.49/2946.94 | 240.23/457.41 | 13.82/21.07 | 13.82/21.07 | 14.63% | 53.19%/24.90%/13.31% | 2.024 |
+| humaneval | baseline | 164 | 194.21 | 1.000x | 1403.54/2734.28 | 218.21/420.29 | 17.84/29.90 | 17.85/29.91 | — | -/-/- | — |
+| humaneval | dflash_k3 | 164 | 147.26 | 0.758x | 1856.84/3610.00 | 250.89/497.85 | 24.55/42.01 | 24.56/42.02 | 5.95% | 16.41%/1.35%/0.08% | 1.179 |
+| humaneval | dflash_k7 | 164 | 141.91 | 0.731x | 1877.28/3978.09 | 343.90/982.06 | 22.96/33.21 | 22.97/33.23 | 2.68% | 17.40%/1.31%/0.09% | 1.188 |
+| humaneval | dflare_k3 | 164 | 194.56 | 1.002x | 1401.65/2758.01 | 307.73/627.48 | 17.78/33.04 | 17.78/33.05 | 58.74% | 77.35%/56.42%/42.44% | 2.762 |
+| humaneval | dflare_k7 | 164 | 206.56 | 1.064x | 1296.23/2684.54 | 395.19/956.47 | 16.23/32.95 | 16.23/32.96 | 33.05% | 77.91%/56.44%/40.45% | 3.313 |
+| humaneval | dspark_k3 | 164 | 234.79 | 1.209x | 1172.97/2107.93 | 265.30/502.88 | 14.78/29.63 | 14.78/29.64 | 59.91% | 76.53%/58.74%/44.47% | 2.797 |
+| humaneval | dspark_k7 | 164 | 210.60 | 1.084x | 1276.63/2616.22 | 564.26/1681.50 | 12.49/25.28 | 12.49/25.28 | 35.09% | 74.01%/55.03%/39.85% | 3.457 |
+| mbpp | baseline | 500 | 275.24 | 1.000x | 1788.02/2216.35 | 221.07/398.67 | 13.20/15.66 | 13.21/15.67 | — | -/-/- | — |
+| mbpp | dflash_k3 | 500 | 189.94 | 0.690x | 2595.66/3320.97 | 234.11/439.53 | 19.14/23.96 | 19.15/23.96 | 6.46% | 18.25%/1.11%/0.04% | 1.194 |
+| mbpp | dflash_k7 | 500 | 170.29 | 0.619x | 2875.58/4074.32 | 634.04/1885.92 | 18.20/24.69 | 18.21/24.70 | 2.85% | 18.78%/1.13%/0.05% | 1.200 |
+| mbpp | dflare_k3 | 500 | 244.12 | 0.887x | 2016.51/2791.48 | 280.07/557.88 | 14.26/19.44 | 14.27/19.63 | 47.05% | 68.89%/44.50%/27.76% | 2.411 |
+| mbpp | dflare_k7 | 500 | 184.84 | 0.672x | 2657.85/4488.95 | 1190.47/3259.46 | 12.45/22.86 | 12.46/22.86 | 24.66% | 67.13%/42.74%/25.45% | 2.726 |
+| mbpp | dspark_k3 | 500 | 310.80 | 1.129x | 1592.68/2178.77 | 240.73/455.66 | 11.35/15.68 | 11.35/15.68 | 52.29% | 71.25%/50.82%/34.79% | 2.569 |
+| mbpp | dspark_k7 | 500 | 274.50 | 0.997x | 1794.33/3171.44 | 643.14/2119.01 | 10.10/14.88 | 10.11/14.88 | 28.83% | 68.32%/47.02%/30.55% | 3.018 |
+| mt_bench | baseline | 160 | 309.23 | 1.000x | 1607.62/2284.11 | 320.49/629.17 | 11.01/15.18 | 11.01/15.19 | — | -/-/- | — |
+| mt_bench | dflash_k3 | 160 | 195.62 | 0.633x | 2573.23/3166.33 | 221.35/448.16 | 18.68/23.10 | 18.68/23.11 | 6.07% | 16.45%/1.71%/0.06% | 1.182 |
+| mt_bench | dflash_k7 | 160 | 180.37 | 0.583x | 2753.90/3791.87 | 381.83/1026.59 | 21.71/25.58 | 21.71/25.58 | 2.48% | 15.47%/1.60%/0.12% | 1.173 |
+| mt_bench | dflare_k3 | 160 | 225.31 | 0.729x | 2231.25/3333.71 | 248.97/520.00 | 15.78/23.60 | 15.78/23.60 | 32.30% | 57.50%/27.57%/11.81% | 1.969 |
+| mt_bench | dflare_k7 | 160 | 133.91 | 0.433x | 3752.53/6717.97 | 1934.01/4820.78 | 14.32/29.83 | 14.32/29.83 | 14.58% | 56.90%/26.94%/11.54% | 2.021 |
+| mt_bench | dspark_k3 | 160 | 282.93 | 0.915x | 1771.67/2604.84 | 252.12/503.80 | 12.36/19.15 | 12.36/19.15 | 36.54% | 58.03%/32.54%/19.06% | 2.096 |
+| mt_bench | dspark_k7 | 160 | 224.25 | 0.725x | 2225.17/3657.94 | 838.84/2441.89 | 11.43/22.87 | 11.43/23.00 | 17.22% | 55.30%/29.97%/16.74% | 2.205 |
+| ultrachat | baseline | 5000 | 155.46 | 1.000x | 3269.57/3934.88 | 231.24/386.99 | 24.09/28.91 | 24.09/28.91 | — | -/-/- | — |
+| ultrachat | dflash_k3 | 5000 | 135.15 | 0.869x | 3763.95/4515.90 | 257.60/405.56 | 27.76/33.28 | 27.77/33.28 | 7.09% | 18.40%/2.57%/0.30% | 1.213 |
+| ultrachat | dflash_k7 | 5000 | 134.03 | 0.862x | 3794.92/4543.27 | 255.94/399.63 | 28.03/33.56 | 28.03/33.56 | 3.07% | 18.54%/2.58%/0.31% | 1.215 |
+| ultrachat | dflare_k3 | 5000 | 161.25 | 1.037x | 3155.82/4013.42 | 282.90/456.06 | 22.75/29.20 | 22.75/29.20 | 31.06% | 57.10%/26.00%/10.07% | 1.932 |
+| ultrachat | dflare_k7 | 5000 | 157.88 | 1.016x | 3221.08/4123.29 | 352.64/612.31 | 22.73/29.63 | 22.73/29.63 | 14.37% | 57.63%/26.59%/10.33% | 2.006 |
+| ultrachat | dspark_k3 | 5000 | 190.92 | 1.228x | 2665.16/3441.95 | 264.37/429.11 | 19.01/24.80 | 19.01/24.80 | 32.40% | 55.58%/28.18%/13.44% | 1.972 |
+| ultrachat | dspark_k7 | 5000 | 189.60 | 1.220x | 2683.34/3469.80 | 287.66/433.69 | 18.97/25.07 | 18.97/25.07 | 15.40% | 55.21%/27.94%/13.27% | 2.078 |

benchmarks/bs4/results.json

@@ -0,0 +1,1304 @@
+[
+  {
+    "dataset": "math500",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 500,
+    "output_tokens": 63527,
+    "request_throughput_rps": 2.0994279500650657,
+    "output_throughput_tps": 266.7407187675668,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1901.9527513722423,
+    "latency_p50_ms": 1894.7456854948541,
+    "latency_p95_ms": 2310.2332794354875,
+    "latency_p99_ms": 2623.0439431988634,
+    "ttft_mean_ms": 211.01336648262804,
+    "ttft_p50_ms": 200.41962849791162,
+    "ttft_p95_ms": 379.0829671488606,
+    "ttft_p99_ms": 495.6532387045445,
+    "itl_mean_ms": 13.40169340837536,
+    "itl_p50_ms": 13.301524035513445,
+    "itl_p95_ms": 16.309320616836498,
+    "itl_p99_ms": 19.31773654766166,
+    "tpot_mean_ms": 13.403480814341385,
+    "tpot_p50_ms": 13.303263444846724,
+    "tpot_p95_ms": 16.311841278199893,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "math500",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 63537,
+    "request_throughput_rps": 1.4572988369276767,
+    "output_throughput_tps": 185.18479240374757,
+    "speedup_vs_baseline": 0.694250181447229,
+    "latency_mean_ms": 2740.524441106001,
+    "latency_p50_ms": 2728.2846120000104,
+    "latency_p95_ms": 3396.0575988000182,
+    "latency_p99_ms": 3908.6018861800358,
+    "ttft_mean_ms": 232.72850664999817,
+    "ttft_p50_ms": 208.99405350002098,
+    "ttft_p95_ms": 422.7843856499816,
+    "ttft_p99_ms": 635.657571979995,
+    "itl_mean_ms": 19.894394063291394,
+    "itl_p50_ms": 19.884664330708496,
+    "itl_p95_ms": 24.709495220472647,
+    "itl_p99_ms": 27.384374742046813,
+    "tpot_mean_ms": 19.89611259958302,
+    "tpot_p50_ms": 19.88607477952736,
+    "tpot_p95_ms": 24.71110046456739,
+    "acceptance_rate": 0.06285151149944762,
+    "acceptance_pos_1": 0.16639047905995782,
+    "acceptance_pos_2": 0.019716028924374813,
+    "acceptance_pos_3": 0.002448026514010244,
+    "mean_acceptance_length": 1.1885545344983428
+  },
+  {
+    "dataset": "math500",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 63468,
+    "request_throughput_rps": 1.484521663636696,
+    "output_throughput_tps": 188.43924189538765,
+    "speedup_vs_baseline": 0.7064509789358043,
+    "latency_mean_ms": 2690.2326844700183,
+    "latency_p50_ms": 2693.1952875002025,
+    "latency_p95_ms": 3411.2693664001654,
+    "latency_p99_ms": 3642.2530803891277,
+    "ttft_mean_ms": 223.1485508320311,
+    "ttft_p50_ms": 201.97454650042346,
+    "ttft_p95_ms": 410.6845789998714,
+    "ttft_p99_ms": 603.7728062999304,
+    "itl_mean_ms": 19.605501851902595,
+    "itl_p50_ms": 19.610514098427146,
+    "itl_p95_ms": 24.830368157877164,
+    "itl_p99_ms": 27.14365097669306,
+    "tpot_mean_ms": 19.607220136180974,
+    "tpot_p50_ms": 19.612673795280877,
+    "tpot_p95_ms": 24.8317818901585,
+    "acceptance_rate": 0.02742245654217219,
+    "acceptance_pos_1": 0.1686266354080012,
+    "acceptance_pos_2": 0.020192089540951374,
+    "acceptance_pos_3": 0.0026090902215836044,
+    "mean_acceptance_length": 1.1919571957952053
+  },
+  {
+    "dataset": "math500",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 63495,
+    "request_throughput_rps": 1.7148645616582274,
+    "output_throughput_tps": 217.7706506849783,
+    "speedup_vs_baseline": 0.8164132258889946,
+    "latency_mean_ms": 2328.9442384679533,
+    "latency_p50_ms": 2273.6253690000012,
+    "latency_p95_ms": 3331.7489405495494,
+    "latency_p99_ms": 3864.866585709678,
+    "ttft_mean_ms": 279.50968451999506,
+    "ttft_p50_ms": 263.16054300059477,
+    "ttft_p95_ms": 506.23001639960404,
+    "ttft_p99_ms": 776.1183005902055,
+    "itl_mean_ms": 16.27184360810761,
+    "itl_p50_ms": 15.999782673234437,
+    "itl_p95_ms": 23.205805455507797,
+    "itl_p99_ms": 27.393199366532944,
+    "tpot_mean_ms": 16.273605324281704,
+    "tpot_p50_ms": 16.000888346459,
+    "tpot_p95_ms": 23.207303030310158,
+    "acceptance_rate": 0.3935291759720552,
+    "acceptance_pos_1": 0.6488919429651263,
+    "acceptance_pos_2": 0.35275725820305787,
+    "acceptance_pos_3": 0.17893832674798144,
+    "mean_acceptance_length": 2.1805875279161655
+  },
+  {
+    "dataset": "math500",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 63622,
+    "request_throughput_rps": 1.7760438758773052,
+    "output_throughput_tps": 225.9909269421318,
+    "speedup_vs_baseline": 0.8472307039820806,
+    "latency_mean_ms": 2249.2024364179815,
+    "latency_p50_ms": 2185.3041670019593,
+    "latency_p95_ms": 3338.340588499702,
+    "latency_p99_ms": 4000.3402635519888,
+    "ttft_mean_ms": 263.0280147520243,
+    "ttft_p50_ms": 252.19799500155204,
+    "ttft_p95_ms": 475.0075314999774,
+    "ttft_p99_ms": 595.4660286509533,
+    "itl_mean_ms": 15.760428864173193,
+    "itl_p50_ms": 15.345385657476402,
+    "itl_p95_ms": 24.247727375981995,
+    "itl_p99_ms": 29.25197200714273,
+    "tpot_mean_ms": 15.762062569323362,
+    "tpot_p50_ms": 15.347322909449966,
+    "tpot_p95_ms": 24.248864364582953,
+    "acceptance_rate": 0.18693377898042707,
+    "acceptance_pos_1": 0.6466625620223824,
+    "acceptance_pos_2": 0.3548223534098729,
+    "acceptance_pos_3": 0.17529245590525516,
+    "mean_acceptance_length": 2.3085364528629895
+  },
+  {
+    "dataset": "math500",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 63636,
+    "request_throughput_rps": 2.2498034619335257,
+    "output_throughput_tps": 286.33698620720367,
+    "speedup_vs_baseline": 1.0734656018405375,
+    "latency_mean_ms": 1774.1428698560267,
+    "latency_p50_ms": 1733.4975750000012,
+    "latency_p95_ms": 2512.932257099783,
+    "latency_p99_ms": 2941.1046552174953,
+    "ttft_mean_ms": 237.00501048202568,
+    "ttft_p50_ms": 212.508828000864,
+    "ttft_p95_ms": 434.44543489968055,
+    "ttft_p99_ms": 581.4404782277412,
+    "itl_mean_ms": 12.18272455456093,
+    "itl_p50_ms": 11.912186838580821,
+    "itl_p95_ms": 17.877204798435372,
+    "itl_p99_ms": 22.473681951484796,
+    "tpot_mean_ms": 12.18441765344298,
+    "tpot_p50_ms": 11.913581936995776,
+    "tpot_p95_ms": 17.87863581495454,
+    "acceptance_rate": 0.4499175952574226,
+    "acceptance_pos_1": 0.6567411999114456,
+    "acceptance_pos_2": 0.42247804590067156,
+    "acceptance_pos_3": 0.27053353996015056,
+    "mean_acceptance_length": 2.3497527857722678
+  },
+  {
+    "dataset": "math500",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 63621,
+    "request_throughput_rps": 2.316449613127783,
+    "output_throughput_tps": 294.7496816736054,
+    "speedup_vs_baseline": 1.1050044516467135,
+    "latency_mean_ms": 1725.2525323180525,
+    "latency_p50_ms": 1675.7616530012456,
+    "latency_p95_ms": 2646.2700289983313,
+    "latency_p99_ms": 3227.1719797111646,
+    "ttft_mean_ms": 247.76108907210437,
+    "ttft_p50_ms": 221.10979600074643,
+    "ttft_p95_ms": 461.0275230510521,
+    "ttft_p99_ms": 670.7492681724033,
+    "itl_mean_ms": 11.723149998862587,
+    "itl_p50_ms": 11.28927224801989,
+    "itl_p95_ms": 18.88152986101506,
+    "itl_p99_ms": 22.771947493768472,
+    "tpot_mean_ms": 11.72477114787681,
+    "tpot_p50_ms": 11.290766421259073,
+    "tpot_p95_ms": 18.882649400401576,
+    "acceptance_rate": 0.22890383017299992,
+    "acceptance_pos_1": 0.6397510474718301,
+    "acceptance_pos_2": 0.3964121547410812,
+    "acceptance_pos_3": 0.24219989423585406,
+    "mean_acceptance_length": 2.6023268112109994
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 1319,
+    "output_tokens": 168502,
+    "request_throughput_rps": 2.196360834095345,
+    "output_throughput_tps": 280.5846802628763,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1819.2423071760666,
+    "latency_p50_ms": 1798.3721370110288,
+    "latency_p95_ms": 2167.7596873778384,
+    "latency_p99_ms": 2310.5863884446435,
+    "ttft_mean_ms": 217.04894016683772,
+    "ttft_p50_ms": 191.49014502181672,
+    "ttft_p95_ms": 398.45379750477116,
+    "ttft_p99_ms": 646.2297889695037,
+    "itl_mean_ms": 12.638841870584082,
+    "itl_p50_ms": 12.576321425240883,
+    "itl_p95_ms": 15.217476899920795,
+    "itl_p99_ms": 16.315808897562086,
+    "tpot_mean_ms": 12.640636354922657,
+    "tpot_p50_ms": 12.58000173247566,
+    "tpot_p95_ms": 15.218959284321164,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 1319,
+    "output_tokens": 168356,
+    "request_throughput_rps": 1.4790952868779552,
+    "output_throughput_tps": 188.79042162064067,
+    "speedup_vs_baseline": 0.6728465055318247,
+    "latency_mean_ms": 2702.3829456360854,
+    "latency_p50_ms": 2683.6698380000144,
+    "latency_p95_ms": 3317.1016436999594,
+    "latency_p99_ms": 3597.6961572799746,
+    "ttft_mean_ms": 235.6068064935536,
+    "ttft_p50_ms": 221.15605599992705,
+    "ttft_p95_ms": 424.62412700004944,
+    "ttft_p99_ms": 526.131548720104,
+    "itl_mean_ms": 19.477842147847973,
+    "itl_p50_ms": 19.355762133858388,
+    "itl_p95_ms": 24.183653737845873,
+    "itl_p99_ms": 26.07432056251997,
+    "tpot_mean_ms": 19.479491279517458,
+    "tpot_p50_ms": 19.357246826771867,
+    "tpot_p95_ms": 24.18510518785972,
+    "acceptance_rate": 0.0680806946304376,
+    "acceptance_pos_1": 0.18367244085417656,
+    "acceptance_pos_2": 0.018568116693305686,
+    "acceptance_pos_3": 0.002001526343830547,
+    "mean_acceptance_length": 1.2042420838913128
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 1319,
+    "output_tokens": 168367,
+    "request_throughput_rps": 1.4764153266187332,
+    "output_throughput_tps": 188.46066663898122,
+    "speedup_vs_baseline": 0.6716712632436481,
+    "latency_mean_ms": 2705.295749132689,
+    "latency_p50_ms": 2679.983797999739,
+    "latency_p95_ms": 3477.261450399509,
+    "latency_p99_ms": 4079.732661799678,
+    "ttft_mean_ms": 301.1807209363258,
+    "ttft_p50_ms": 224.2310860001453,
+    "ttft_p95_ms": 834.4522548001792,
+    "ttft_p99_ms": 1980.7203211395963,
+    "itl_mean_ms": 18.978770912763007,
+    "itl_p50_ms": 19.034192118114312,
+    "itl_p95_ms": 23.592571637004227,
+    "itl_p99_ms": 25.352459187087952,
+    "tpot_mean_ms": 18.980617877378993,
+    "tpot_p50_ms": 19.03604929134238,
+    "tpot_p95_ms": 23.594471511024295,
+    "acceptance_rate": 0.027380290750817878,
+    "acceptance_pos_1": 0.17135507958901572,
+    "acceptance_pos_2": 0.018069627919570206,
+    "acceptance_pos_3": 0.001987944081056817,
+    "mean_acceptance_length": 1.1916620352557252
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 1319,
+    "output_tokens": 168415,
+    "request_throughput_rps": 1.574435488473932,
+    "output_throughput_tps": 201.0299869532504,
+    "speedup_vs_baseline": 0.7164681505950642,
+    "latency_mean_ms": 2536.437202144023,
+    "latency_p50_ms": 2473.5267859996384,
+    "latency_p95_ms": 3534.031745998982,
+    "latency_p99_ms": 4064.8575092010615,
+    "ttft_mean_ms": 258.1910506959681,
+    "ttft_p50_ms": 238.57910900005663,
+    "ttft_p95_ms": 484.66444300011057,
+    "ttft_p99_ms": 598.7782853190582,
+    "itl_mean_ms": 17.977036765777857,
+    "itl_p50_ms": 17.445515787392107,
+    "itl_p95_ms": 25.81660302204809,
+    "itl_p99_ms": 30.753264190233132,
+    "tpot_mean_ms": 17.978638165785107,
+    "tpot_p50_ms": 17.446966267718064,
+    "tpot_p95_ms": 25.818113115744246,
+    "acceptance_rate": 0.2794753137068277,
+    "acceptance_pos_1": 0.536425000273502,
+    "acceptance_pos_2": 0.21512575623311125,
+    "acceptance_pos_3": 0.08687518461386984,
+    "mean_acceptance_length": 1.838425941120483
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 1319,
+    "output_tokens": 168400,
+    "request_throughput_rps": 1.2094482721851378,
+    "output_throughput_tps": 154.41325931461503,
+    "speedup_vs_baseline": 0.5503267647041427,
+    "latency_mean_ms": 3304.2710242100056,
+    "latency_p50_ms": 3064.5563209982356,
+    "latency_p95_ms": 5885.097530001435,
+    "latency_p99_ms": 7240.322633720641,
+    "ttft_mean_ms": 1205.8129131272765,
+    "ttft_p50_ms": 677.296675999969,
+    "ttft_p95_ms": 4089.5798977991944,
+    "ttft_p99_ms": 5644.379561060411,
+    "itl_mean_ms": 16.556715234828605,
+    "itl_p50_ms": 16.74034803936521,
+    "itl_p95_ms": 27.050037795280637,
+    "itl_p99_ms": 35.51621275700378,
+    "tpot_mean_ms": 16.558478447758667,
+    "tpot_p50_ms": 16.743846314973666,
+    "tpot_p95_ms": 27.058657145679476,
+    "acceptance_rate": 0.12825457741085033,
+    "acceptance_pos_1": 0.5421524916756024,
+    "acceptance_pos_2": 0.21891754613691516,
+    "acceptance_pos_3": 0.09374118178226762,
+    "mean_acceptance_length": 1.8977820418759523
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 1319,
+    "output_tokens": 168413,
+    "request_throughput_rps": 2.0975172075679485,
+    "output_throughput_tps": 267.81589497963677,
+    "speedup_vs_baseline": 0.954492222200882,
+    "latency_mean_ms": 1904.8592331402729,
+    "latency_p50_ms": 1849.926056998811,
+    "latency_p95_ms": 2717.593596396909,
+    "latency_p99_ms": 3128.5920352824137,
+    "ttft_mean_ms": 232.41419941998183,
+    "ttft_p50_ms": 211.30282399826683,
+    "ttft_p95_ms": 436.5587686021171,
+    "ttft_p99_ms": 608.768046818439,
+    "itl_mean_ms": 13.20091798661006,
+    "itl_p50_ms": 12.809261669308327,
+    "itl_p95_ms": 19.22371915434054,
+    "itl_p99_ms": 22.673929599071368,
+    "tpot_mean_ms": 13.202611622505504,
+    "tpot_p50_ms": 12.810728181104036,
+    "tpot_p95_ms": 19.225053650402195,
+    "acceptance_rate": 0.31145141140312554,
+    "acceptance_pos_1": 0.5333057414177645,
+    "acceptance_pos_2": 0.2575245453082247,
+    "acceptance_pos_3": 0.14352394748338737,
+    "mean_acceptance_length": 1.9343542342093767
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 1319,
+    "output_tokens": 168420,
+    "request_throughput_rps": 2.0069879067415446,
+    "output_throughput_tps": 256.2675536417065,
+    "speedup_vs_baseline": 0.9133340900922055,
+    "latency_mean_ms": 1991.4858750174596,
+    "latency_p50_ms": 1940.2353619989299,
+    "latency_p95_ms": 2946.936268898934,
+    "latency_p99_ms": 3409.506298801133,
+    "ttft_mean_ms": 240.2313964965955,
+    "ttft_p50_ms": 217.0051639986923,
+    "ttft_p95_ms": 457.4054478005564,
+    "ttft_p99_ms": 621.9962791216665,
+    "itl_mean_ms": 13.815387803215751,
+    "itl_p50_ms": 13.491749685057384,
+    "itl_p95_ms": 21.069445844088154,
+    "itl_p99_ms": 24.68408968393126,
+    "tpot_mean_ms": 13.817049546266604,
+    "tpot_p50_ms": 13.493896850398981,
+    "tpot_p95_ms": 21.07079229370886,
+    "acceptance_rate": 0.1463064478504878,
+    "acceptance_pos_1": 0.5319253673998655,
+    "acceptance_pos_2": 0.2490394774757468,
+    "acceptance_pos_3": 0.13306838920372682,
+    "mean_acceptance_length": 2.0241451349534145
+  },
+  {
+    "dataset": "humaneval",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 164,
+    "output_tokens": 11229,
+    "request_throughput_rps": 2.83651766504482,
+    "output_throughput_tps": 194.21498085846514,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1403.542471638946,
+    "latency_p50_ms": 1171.1717744910857,
+    "latency_p95_ms": 2734.2781511018984,
+    "latency_p99_ms": 3048.235554893327,
+    "ttft_mean_ms": 218.20821551103487,
+    "ttft_p50_ms": 209.54365949728526,
+    "ttft_p95_ms": 420.2904565172503,
+    "ttft_p99_ms": 467.85162955609854,
+    "itl_mean_ms": 17.843567054063506,
+    "itl_p50_ms": 17.024532066210227,
+    "itl_p95_ms": 29.900638077389125,
+    "itl_p99_ms": 38.85421750062092,
+    "tpot_mean_ms": 17.849692097307134,
+    "tpot_p50_ms": 17.028038263717008,
+    "tpot_p95_ms": 29.90535920137036,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 164,
+    "output_tokens": 11292,
+    "request_throughput_rps": 2.1388107405014773,
+    "output_throughput_tps": 147.26494440087,
+    "speedup_vs_baseline": 0.7582573895686753,
+    "latency_mean_ms": 1856.8446467317099,
+    "latency_p50_ms": 1529.8299214999815,
+    "latency_p95_ms": 3609.9989535499276,
+    "latency_p99_ms": 4076.3032318901037,
+    "ttft_mean_ms": 250.89464046951855,
+    "ttft_p50_ms": 233.70097149984304,
+    "ttft_p95_ms": 497.8459723500346,
+    "ttft_p99_ms": 638.7824368799534,
+    "itl_mean_ms": 24.549073372278432,
+    "itl_p50_ms": 22.60172811232828,
+    "itl_p95_ms": 42.012251182637584,
+    "itl_p99_ms": 51.31418752062339,
+    "tpot_mean_ms": 24.555336995750512,
+    "tpot_p50_ms": 22.603792561644838,
+    "tpot_p95_ms": 42.01500447592228,
+    "acceptance_rate": 0.059506172839506176,
+    "acceptance_pos_1": 0.16412698412698412,
+    "acceptance_pos_2": 0.013544973544973546,
+    "acceptance_pos_3": 0.0008465608465608466,
+    "mean_acceptance_length": 1.1785185185185185
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 164,
+    "output_tokens": 11052,
+    "request_throughput_rps": 2.1057562192267367,
+    "output_throughput_tps": 141.90742521276766,
+    "speedup_vs_baseline": 0.7306718801274306,
+    "latency_mean_ms": 1877.2794746829145,
+    "latency_p50_ms": 1504.3869174996871,
+    "latency_p95_ms": 3978.088593899155,
+    "latency_p99_ms": 4819.554965540591,
+    "ttft_mean_ms": 343.89938129273037,
+    "ttft_p50_ms": 254.9212224994335,
+    "ttft_p95_ms": 982.0552947493524,
+    "ttft_p99_ms": 1265.2674350103007,
+    "itl_mean_ms": 22.96417710011784,
+    "itl_p50_ms": 22.33989236389506,
+    "itl_p95_ms": 33.208631611878445,
+    "itl_p99_ms": 43.04405845282055,
+    "tpot_mean_ms": 22.970011297649936,
+    "tpot_p50_ms": 22.34099295422598,
+    "tpot_p95_ms": 33.23066878742338,
+    "acceptance_rate": 0.02684898226160627,
+    "acceptance_pos_1": 0.17398888040989863,
+    "acceptance_pos_2": 0.013081870707511174,
+    "acceptance_pos_3": 0.0008721247138340783,
+    "mean_acceptance_length": 1.1879428758312438
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 164,
+    "output_tokens": 11277,
+    "request_throughput_rps": 2.8294040460667382,
+    "output_throughput_tps": 194.55603309447932,
+    "speedup_vs_baseline": 1.0017560552461333,
+    "latency_mean_ms": 1401.6537050121503,
+    "latency_p50_ms": 1262.2241380004198,
+    "latency_p95_ms": 2758.0062766009178,
+    "latency_p99_ms": 3165.725670439934,
+    "ttft_mean_ms": 307.73021359141796,
+    "ttft_p50_ms": 282.3090209994916,
+    "ttft_p95_ms": 627.4767150008302,
+    "ttft_p99_ms": 760.1846476996616,
+    "itl_mean_ms": 17.775962665717213,
+    "itl_p50_ms": 16.24701220755945,
+    "itl_p95_ms": 33.04292447846012,
+    "itl_p99_ms": 49.24142361158533,
+    "tpot_mean_ms": 17.781789938797445,
+    "tpot_p50_ms": 16.250892179999738,
+    "tpot_p95_ms": 33.04544840657803,
+    "acceptance_rate": 0.5873682476943346,
+    "acceptance_pos_1": 0.7734683794466403,
+    "acceptance_pos_2": 0.5642292490118577,
+    "acceptance_pos_3": 0.4244071146245059,
+    "mean_acceptance_length": 2.762104743083004
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 164,
+    "output_tokens": 11047,
+    "request_throughput_rps": 3.0665775841960174,
+    "output_throughput_tps": 206.5639181256915,
+    "speedup_vs_baseline": 1.0635838554402026,
+    "latency_mean_ms": 1296.2304299939337,
+    "latency_p50_ms": 1225.3228209992812,
+    "latency_p95_ms": 2684.5432901496665,
+    "latency_p99_ms": 3137.9315633195806,
+    "ttft_mean_ms": 395.1852547253976,
+    "ttft_p50_ms": 324.23020049827755,
+    "ttft_p95_ms": 956.4745560996016,
+    "ttft_p99_ms": 1385.3407927481385,
+    "itl_mean_ms": 16.226717747252554,
+    "itl_p50_ms": 13.57385855245207,
+    "itl_p95_ms": 32.95489575833981,
+    "itl_p99_ms": 75.28418661784087,
+    "tpot_mean_ms": 16.23359719581637,
+    "tpot_p50_ms": 13.576510465236616,
+    "tpot_p95_ms": 32.96468897523092,
+    "acceptance_rate": 0.33047302832995556,
+    "acceptance_pos_1": 0.7790522185330516,
+    "acceptance_pos_2": 0.5644431029278599,
+    "acceptance_pos_3": 0.4044672502263809,
+    "mean_acceptance_length": 3.3133111983096892
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 164,
+    "output_tokens": 11346,
+    "request_throughput_rps": 3.393797033660498,
+    "output_throughput_tps": 234.79281185312203,
+    "speedup_vs_baseline": 1.2089325489480554,
+    "latency_mean_ms": 1172.9745371464626,
+    "latency_p50_ms": 1033.1892974991206,
+    "latency_p95_ms": 2107.933460299682,
+    "latency_p99_ms": 3156.528969858848,
+    "ttft_mean_ms": 265.29695418294733,
+    "ttft_p50_ms": 228.4132034983486,
+    "ttft_p95_ms": 502.8849432006609,
+    "ttft_p99_ms": 699.91048091968,
+    "itl_mean_ms": 14.77773082701616,
+    "itl_p50_ms": 13.023184389006781,
+    "itl_p95_ms": 29.633302174848463,
+    "itl_p99_ms": 45.321116141877226,
+    "tpot_mean_ms": 14.783925134182368,
+    "tpot_p50_ms": 13.039449575903792,
+    "tpot_p95_ms": 29.64001845995613,
+    "acceptance_rate": 0.5991372158619546,
+    "acceptance_pos_1": 0.7653061224489796,
+    "acceptance_pos_2": 0.587356893977103,
+    "acceptance_pos_3": 0.44474863115978097,
+    "mean_acceptance_length": 2.7974116475858635
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 164,
+    "output_tokens": 11058,
+    "request_throughput_rps": 3.1234019762512726,
+    "output_throughput_tps": 210.6010917889425,
+    "speedup_vs_baseline": 1.08437099372071,
+    "latency_mean_ms": 1276.6294960669388,
+    "latency_p50_ms": 1098.64930799813,
+    "latency_p95_ms": 2616.217871799927,
+    "latency_p99_ms": 3188.0119062716053,
+    "ttft_mean_ms": 564.2550010548891,
+    "ttft_p50_ms": 336.2399385005119,
+    "ttft_p95_ms": 1681.4955957521306,
+    "ttft_p99_ms": 2938.6766085621284,
+    "itl_mean_ms": 12.486812538079404,
+    "itl_p50_ms": 11.807729195672675,
+    "itl_p95_ms": 25.278697270681064,
+    "itl_p99_ms": 43.8447163140126,
+    "tpot_mean_ms": 12.494207252966246,
+    "tpot_p50_ms": 11.83652314342243,
+    "tpot_p95_ms": 25.281766504248417,
+    "acceptance_rate": 0.35094729280775794,
+    "acceptance_pos_1": 0.740100565681961,
+    "acceptance_pos_2": 0.5502828409805154,
+    "acceptance_pos_3": 0.39849151477058453,
+    "mean_acceptance_length": 3.4566310496543053
+  },
+  {
+    "dataset": "mbpp",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 500,
+    "output_tokens": 61587,
+    "request_throughput_rps": 2.23460184339787,
+    "output_throughput_tps": 275.2448474586892,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1788.0222868800047,
+    "latency_p50_ms": 1819.359868997708,
+    "latency_p95_ms": 2216.346881000209,
+    "latency_p99_ms": 2362.3936853421037,
+    "ttft_mean_ms": 221.07102080830373,
+    "ttft_p50_ms": 204.21197451651096,
+    "ttft_p95_ms": 398.6740397042012,
+    "ttft_p99_ms": 655.3067808030755,
+    "itl_mean_ms": 13.201886877023522,
+    "itl_p50_ms": 12.736825452678412,
+    "itl_p95_ms": 15.664009772456069,
+    "itl_p99_ms": 17.321856500478695,
+    "tpot_mean_ms": 13.205948736768402,
+    "tpot_p50_ms": 12.738972996026057,
+    "tpot_p95_ms": 15.67098670536386,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 61808,
+    "request_throughput_rps": 1.5365610042290248,
+    "output_throughput_tps": 189.94352509877513,
+    "speedup_vs_baseline": 0.690089303587357,
+    "latency_mean_ms": 2595.6573488480017,
+    "latency_p50_ms": 2617.603001500015,
+    "latency_p95_ms": 3320.9663893999277,
+    "latency_p99_ms": 3516.577585000016,
+    "ttft_mean_ms": 234.1106791739985,
+    "ttft_p50_ms": 218.79009800011318,
+    "ttft_p95_ms": 439.53066730014143,
+    "ttft_p99_ms": 544.9614302001191,
+    "itl_mean_ms": 19.14316073130201,
+    "itl_p50_ms": 18.9773965472434,
+    "itl_p95_ms": 23.96081993622043,
+    "itl_p99_ms": 25.695518167638486,
+    "tpot_mean_ms": 19.146981896371145,
+    "tpot_p50_ms": 18.978798267717032,
+    "tpot_p95_ms": 23.96214876692921,
+    "acceptance_rate": 0.06462320419065401,
+    "acceptance_pos_1": 0.1824529329391629,
+    "acceptance_pos_2": 0.01106659405632488,
+    "acceptance_pos_3": 0.0003500855764742493,
+    "mean_acceptance_length": 1.1938696125719621
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 61377,
+    "request_throughput_rps": 1.387242813763742,
+    "output_throughput_tps": 170.2896043607544,
+    "speedup_vs_baseline": 0.6186840768611036,
+    "latency_mean_ms": 2875.578809605966,
+    "latency_p50_ms": 2855.830152499948,
+    "latency_p95_ms": 4074.322934499559,
+    "latency_p99_ms": 4764.013334749542,
+    "ttft_mean_ms": 634.0442261159515,
+    "ttft_p50_ms": 392.3641729988958,
+    "ttft_p95_ms": 1885.916530999566,
+    "ttft_p99_ms": 2897.4566687989864,
+    "itl_mean_ms": 18.201249949891572,
+    "itl_p50_ms": 18.463196299214943,
+    "itl_p95_ms": 24.69457239645406,
+    "itl_p99_ms": 30.63269140497326,
+    "tpot_mean_ms": 18.205769807911185,
+    "tpot_p50_ms": 18.464951303145412,
+    "tpot_p95_ms": 24.695845573618083,
+    "acceptance_rate": 0.028501713363674443,
+    "acceptance_pos_1": 0.1877840964993408,
+    "acceptance_pos_2": 0.011255632735787795,
+    "acceptance_pos_3": 0.0004525866309844743,
+    "mean_acceptance_length": 1.199511993545721
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 61640,
+    "request_throughput_rps": 1.9801835796759275,
+    "output_throughput_tps": 244.11703170244834,
+    "speedup_vs_baseline": 0.8869086340992713,
+    "latency_mean_ms": 2016.50626224402,
+    "latency_p50_ms": 2021.2945930006754,
+    "latency_p95_ms": 2791.478140551044,
+    "latency_p99_ms": 3086.5239088800263,
+    "ttft_mean_ms": 280.0661946479813,
+    "ttft_p50_ms": 256.2331505005204,
+    "ttft_p95_ms": 557.8820319009537,
+    "ttft_p99_ms": 769.5084362501256,
+    "itl_mean_ms": 14.264965995399256,
+    "itl_p50_ms": 14.109097578741029,
+    "itl_p95_ms": 19.439541319448956,
+    "itl_p99_ms": 23.81797452189929,
+    "tpot_mean_ms": 14.269046870264877,
+    "tpot_p50_ms": 14.11056187795813,
+    "tpot_p95_ms": 19.634308889327635,
+    "acceptance_rate": 0.47048386886358,
+    "acceptance_pos_1": 0.6888967163711792,
+    "acceptance_pos_2": 0.44499236820476695,
+    "acceptance_pos_3": 0.2775625220147939,
+    "mean_acceptance_length": 2.41145160659074
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 61475,
+    "request_throughput_rps": 1.503343533132173,
+    "output_throughput_tps": 184.83608739860065,
+    "speedup_vs_baseline": 0.6715333242572042,
+    "latency_mean_ms": 2657.845550606042,
+    "latency_p50_ms": 2483.5515449995,
+    "latency_p95_ms": 4488.94517174831,
+    "latency_p99_ms": 5523.38156365822,
+    "ttft_mean_ms": 1190.47147037393,
+    "ttft_p50_ms": 932.4905619996571,
+    "ttft_p95_ms": 3259.4635773521077,
+    "ttft_p99_ms": 4279.743841777054,
+    "itl_mean_ms": 12.451201825429687,
+    "itl_p50_ms": 12.228331582672416,
+    "itl_p95_ms": 22.859363721256052,
+    "itl_p99_ms": 36.967242600596585,
+    "tpot_mean_ms": 12.456394792296264,
+    "tpot_p50_ms": 12.230752885830222,
+    "tpot_p95_ms": 22.860772319684717,
+    "acceptance_rate": 0.24657171832973235,
+    "acceptance_pos_1": 0.6712818025486132,
+    "acceptance_pos_2": 0.42739979716918736,
+    "acceptance_pos_3": 0.2544644825609595,
+    "mean_acceptance_length": 2.7260020283081268
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 61981,
+    "request_throughput_rps": 2.507209609073491,
+    "output_throughput_tps": 310.7987175599681,
+    "speedup_vs_baseline": 1.1291717916958104,
+    "latency_mean_ms": 1592.6791832899762,
+    "latency_p50_ms": 1603.4026140005153,
+    "latency_p95_ms": 2178.7681314988727,
+    "latency_p99_ms": 2552.1267430388243,
+    "ttft_mean_ms": 240.7260946480019,
+    "ttft_p50_ms": 224.12168300070334,
+    "ttft_p95_ms": 455.65863744814123,
+    "ttft_p99_ms": 601.2736888401558,
+    "itl_mean_ms": 11.346454396871485,
+    "itl_p50_ms": 10.845360450802424,
+    "itl_p95_ms": 15.682580235835754,
+    "itl_p99_ms": 23.395322236705972,
+    "tpot_mean_ms": 11.350096012527779,
+    "tpot_p50_ms": 10.847383575174785,
+    "tpot_p95_ms": 15.684506253152122,
+    "acceptance_rate": 0.5228963898667771,
+    "acceptance_pos_1": 0.7125284738041002,
+    "acceptance_pos_2": 0.5082211638020294,
+    "acceptance_pos_3": 0.3479395319942017,
+    "mean_acceptance_length": 2.568689169600331
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 61632,
+    "request_throughput_rps": 2.2269171937357544,
+    "output_throughput_tps": 274.498720968644,
+    "speedup_vs_baseline": 0.997289226312739,
+    "latency_mean_ms": 1794.3305799321024,
+    "latency_p50_ms": 1694.5109794996824,
+    "latency_p95_ms": 3171.435100350026,
+    "latency_p99_ms": 4215.3635372184735,
+    "ttft_mean_ms": 643.143013108056,
+    "ttft_p50_ms": 362.4557425009698,
+    "ttft_p95_ms": 2119.0080873015154,
+    "ttft_p99_ms": 3187.935962892621,
+    "itl_mean_ms": 10.101565495090881,
+    "itl_p50_ms": 9.322213830710384,
+    "itl_p95_ms": 14.88066484486098,
+    "itl_p99_ms": 19.67402437656759,
+    "tpot_mean_ms": 10.105878412845934,
+    "tpot_p50_ms": 9.323771708668072,
+    "tpot_p95_ms": 14.882234868113521,
+    "acceptance_rate": 0.28831800034716193,
+    "acceptance_pos_1": 0.6831591737545565,
+    "acceptance_pos_2": 0.47018226002430136,
+    "acceptance_pos_3": 0.3054678007290401,
+    "mean_acceptance_length": 3.0182260024301337
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 2.461309935234548,
+    "output_throughput_tps": 309.2328269880305,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1607.6186366983165,
+    "latency_p50_ms": 1548.2021450006869,
+    "latency_p95_ms": 2284.110534451611,
+    "latency_p99_ms": 2534.602298826794,
+    "ttft_mean_ms": 320.49049263077904,
+    "ttft_p50_ms": 293.138627501321,
+    "ttft_p95_ms": 629.1677434302974,
+    "ttft_p99_ms": 759.140422597702,
+    "itl_mean_ms": 11.007379475699068,
+    "itl_p50_ms": 9.985256696983395,
+    "itl_p95_ms": 15.183904158686165,
+    "itl_p99_ms": 26.580996584276846,
+    "tpot_mean_ms": 11.014195983013959,
+    "tpot_p50_ms": 9.988513000022554,
+    "tpot_p95_ms": 15.18576166809785,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 160,
+    "output_tokens": 20228,
+    "request_throughput_rps": 1.5473009629136512,
+    "output_throughput_tps": 195.61752423635835,
+    "speedup_vs_baseline": 0.632589774318915,
+    "latency_mean_ms": 2573.229441274998,
+    "latency_p50_ms": 2591.1725164999098,
+    "latency_p95_ms": 3166.3267970500783,
+    "latency_p99_ms": 3608.086452589978,
+    "ttft_mean_ms": 221.35167090626453,
+    "ttft_p50_ms": 203.15486550020978,
+    "ttft_p95_ms": 448.1615023500803,
+    "ttft_p99_ms": 696.169536259867,
+    "itl_mean_ms": 18.676105907776027,
+    "itl_p50_ms": 18.64073133858227,
+    "itl_p95_ms": 23.103904855904904,
+    "itl_p99_ms": 25.284028265118394,
+    "tpot_mean_ms": 18.68020851087627,
+    "tpot_p50_ms": 18.641842641731216,
+    "tpot_p95_ms": 23.105400170866087,
+    "acceptance_rate": 0.060748671490479834,
+    "acceptance_pos_1": 0.16453909053473734,
+    "acceptance_pos_2": 0.01705982704864992,
+    "acceptance_pos_3": 0.0006470968880522384,
+    "mean_acceptance_length": 1.1822460144714395
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 1.4356652700068615,
+    "output_throughput_tps": 180.37339536048705,
+    "speedup_vs_baseline": 0.5832931681844656,
+    "latency_mean_ms": 2753.9001531249596,
+    "latency_p50_ms": 2798.7496705000012,
+    "latency_p95_ms": 3791.8684454991308,
+    "latency_p99_ms": 4289.662929459572,
+    "ttft_mean_ms": 381.8320665374472,
+    "ttft_p50_ms": 295.44119050024165,
+    "ttft_p95_ms": 1026.5876399000251,
+    "ttft_p99_ms": 1367.1522384703346,
+    "itl_mean_ms": 21.709309200685574,
+    "itl_p50_ms": 18.78574032282878,
+    "itl_p95_ms": 25.575840727953157,
+    "itl_p99_ms": 59.525236419324514,
+    "tpot_mean_ms": 21.713930513288513,
+    "tpot_p50_ms": 18.787575688973476,
+    "tpot_p95_ms": 25.57992160196576,
+    "acceptance_rate": 0.024768061794215188,
+    "acceptance_pos_1": 0.15474581251836614,
+    "acceptance_pos_2": 0.016044666470761092,
+    "acceptance_pos_3": 0.0012342051131354686,
+    "mean_acceptance_length": 1.1733764325595064
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 160,
+    "output_tokens": 20228,
+    "request_throughput_rps": 1.7821635622658862,
+    "output_throughput_tps": 225.31002835946467,
+    "speedup_vs_baseline": 0.7286096710818666,
+    "latency_mean_ms": 2231.2470228500047,
+    "latency_p50_ms": 2197.780867999427,
+    "latency_p95_ms": 3333.7097092000463,
+    "latency_p99_ms": 3700.461037129698,
+    "ttft_mean_ms": 248.96705823127832,
+    "ttft_p50_ms": 215.34929249992274,
+    "ttft_p95_ms": 520.0022110006103,
+    "ttft_p99_ms": 662.6540237599145,
+    "itl_mean_ms": 15.77862160127276,
+    "itl_p50_ms": 15.232473944878599,
+    "itl_p95_ms": 23.602753981101593,
+    "itl_p99_ms": 27.57565280062838,
+    "tpot_mean_ms": 15.782222927801888,
+    "tpot_p50_ms": 15.23403437795065,
+    "tpot_p95_ms": 23.60411872952008,
+    "acceptance_rate": 0.3229694606887589,
+    "acceptance_pos_1": 0.5750487329434698,
+    "acceptance_pos_2": 0.27573099415204677,
+    "acceptance_pos_3": 0.11812865497076024,
+    "mean_acceptance_length": 1.9689083820662767
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 1.0658824478410331,
+    "output_throughput_tps": 133.9148060406278,
+    "speedup_vs_baseline": 0.4330549487419434,
+    "latency_mean_ms": 3752.528322143644,
+    "latency_p50_ms": 3464.8722160018224,
+    "latency_p95_ms": 6717.973978601185,
+    "latency_p99_ms": 7876.092720710149,
+    "ttft_mean_ms": 1934.0097326686418,
+    "ttft_p50_ms": 1588.4214640009304,
+    "ttft_p95_ms": 4820.7839886515985,
+    "ttft_p99_ms": 6494.002312830161,
+    "itl_mean_ms": 14.317043586959333,
+    "itl_p50_ms": 13.67209011023508,
+    "itl_p95_ms": 29.82729992639314,
+    "itl_p99_ms": 39.377268431499964,
+    "tpot_mean_ms": 14.323458562974045,
+    "tpot_p50_ms": 13.673784755900577,
+    "tpot_p95_ms": 29.829240646451908,
+    "acceptance_rate": 0.14578666207134242,
+    "acceptance_pos_1": 0.5689585846401287,
+    "acceptance_pos_2": 0.26940088459991957,
+    "acceptance_pos_3": 0.11540008041817451,
+    "mean_acceptance_length": 2.020506634499397
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 160,
+    "output_tokens": 20228,
+    "request_throughput_rps": 2.2379435718508462,
+    "output_throughput_tps": 282.9320160712432,
+    "speedup_vs_baseline": 0.9149481923605564,
+    "latency_mean_ms": 1771.671463362577,
+    "latency_p50_ms": 1775.1809370001865,
+    "latency_p95_ms": 2604.8437047013063,
+    "latency_p99_ms": 3118.5169013906484,
+    "ttft_mean_ms": 252.11589151258522,
+    "ttft_p50_ms": 227.79171349975513,
+    "ttft_p95_ms": 503.8048422973585,
+    "ttft_p99_ms": 627.0037127103565,
+    "itl_mean_ms": 12.360936149822532,
+    "itl_p50_ms": 11.83748445670406,
+    "itl_p95_ms": 19.148658472828007,
+    "itl_p99_ms": 25.009972698654565,
+    "tpot_mean_ms": 12.364664553850584,
+    "tpot_p50_ms": 11.838727370086703,
+    "tpot_p95_ms": 19.150851622824714,
+    "acceptance_rate": 0.36540854417254254,
+    "acceptance_pos_1": 0.5802571547075902,
+    "acceptance_pos_2": 0.3253836582330983,
+    "acceptance_pos_3": 0.19058481957693904,
+    "mean_acceptance_length": 2.0962256325176276
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 1.784863821370322,
+    "output_throughput_tps": 224.24582835741384,
+    "speedup_vs_baseline": 0.7251682511898833,
+    "latency_mean_ms": 2225.1736813125035,
+    "latency_p50_ms": 2117.7182369992806,
+    "latency_p95_ms": 3657.939555099983,
+    "latency_p99_ms": 4345.6770745698295,
+    "ttft_mean_ms": 838.8437911749861,
+    "ttft_p50_ms": 589.9439045006147,
+    "ttft_p95_ms": 2441.8948530525054,
+    "ttft_p99_ms": 2773.190703780456,
+    "itl_mean_ms": 11.426459321042842,
+    "itl_p50_ms": 10.71108703149011,
+    "itl_p95_ms": 22.870212017662954,
+    "itl_p99_ms": 25.594833969671022,
+    "tpot_mean_ms": 11.431539335517765,
+    "tpot_p50_ms": 10.713255098423582,
+    "tpot_p95_ms": 23.0010987298119,
+    "acceptance_rate": 0.17215237263282687,
+    "acceptance_pos_1": 0.5529591613889495,
+    "acceptance_pos_2": 0.2997379340467351,
+    "acceptance_pos_3": 0.16739462764795807,
+    "mean_acceptance_length": 2.205066608429788
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 5000,
+    "output_tokens": 635569,
+    "request_throughput_rps": 1.2230023684575178,
+    "output_throughput_tps": 155.4604784636352,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 3269.5651201336645,
+    "latency_p50_ms": 3276.8055605120026,
+    "latency_p95_ms": 3934.875892131822,
+    "latency_p99_ms": 4087.044628756121,
+    "ttft_mean_ms": 231.24218063543086,
+    "ttft_p50_ms": 218.61417350010015,
+    "ttft_p95_ms": 386.9901933081565,
+    "ttft_p99_ms": 553.9115706225857,
+    "itl_mean_ms": 24.08620830909222,
+    "itl_p50_ms": 24.040817992012823,
+    "itl_p95_ms": 28.90904858166638,
+    "itl_p99_ms": 29.91989293644905,
+    "tpot_mean_ms": 24.087996070253364,
+    "tpot_p50_ms": 24.042644452736308,
+    "tpot_p95_ms": 28.910471471294663,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 5000,
+    "output_tokens": 636175,
+    "request_throughput_rps": 1.0622074332559788,
+    "output_throughput_tps": 135.14996277032446,
+    "speedup_vs_baseline": 0.8693525461002507,
+    "latency_mean_ms": 3763.952480385199,
+    "latency_p50_ms": 3769.2712649998157,
+    "latency_p95_ms": 4515.89870830021,
+    "latency_p99_ms": 4805.0974385998825,
+    "ttft_mean_ms": 257.6036252754003,
+    "ttft_p50_ms": 246.59032999988995,
+    "ttft_p95_ms": 405.558623099887,
+    "ttft_p99_ms": 540.4809878199559,
+    "itl_mean_ms": 27.764084011389272,
+    "itl_p50_ms": 27.68626928346698,
+    "itl_p95_ms": 33.282436218505644,
+    "itl_p99_ms": 35.5445708588995,
+    "tpot_mean_ms": 27.765764208304795,
+    "tpot_p50_ms": 27.687906570868,
+    "tpot_p95_ms": 33.28350510590754,
+    "acceptance_rate": 0.07093823412968715,
+    "acceptance_pos_1": 0.1840484911746139,
+    "acceptance_pos_2": 0.02574222849873457,
+    "acceptance_pos_3": 0.0030239827157130055,
+    "mean_acceptance_length": 1.2128147023890614
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 5000,
+    "output_tokens": 636064,
+    "request_throughput_rps": 1.0536069317356047,
+    "output_throughput_tps": 134.03228788549515,
+    "speedup_vs_baseline": 0.8621630990081349,
+    "latency_mean_ms": 3794.9240800698,
+    "latency_p50_ms": 3797.6578120005797,
+    "latency_p95_ms": 4543.265116648946,
+    "latency_p99_ms": 4838.748624158453,
+    "ttft_mean_ms": 255.93729505000047,
+    "ttft_p50_ms": 246.4688734989977,
+    "ttft_p95_ms": 399.62789619985415,
+    "ttft_p99_ms": 533.8578004602641,
+    "itl_mean_ms": 28.02829548006744,
+    "itl_p50_ms": 27.958994283470407,
+    "itl_p95_ms": 33.55774176338532,
+    "itl_p99_ms": 35.83546665536011,
+    "tpot_mean_ms": 28.030316558857468,
+    "tpot_p50_ms": 27.961111295281434,
+    "tpot_p95_ms": 33.55949413779617,
+    "acceptance_rate": 0.030697806232490035,
+    "acceptance_pos_1": 0.1853871675887231,
+    "acceptance_pos_2": 0.025839366107025186,
+    "acceptance_pos_3": 0.0030756573256390638,
+    "mean_acceptance_length": 1.2148846436274303
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 5000,
+    "output_tokens": 636228,
+    "request_throughput_rps": 1.267241365085764,
+    "output_throughput_tps": 161.25088784515708,
+    "speedup_vs_baseline": 1.0372468259376697,
+    "latency_mean_ms": 3155.8181194270032,
+    "latency_p50_ms": 3158.2490724995296,
+    "latency_p95_ms": 4013.4220384479713,
+    "latency_p99_ms": 4389.7799070399205,
+    "ttft_mean_ms": 282.9019767206311,
+    "ttft_p50_ms": 270.00340999984473,
+    "ttft_p95_ms": 456.05508385024234,
+    "ttft_p99_ms": 595.7626636705755,
+    "itl_mean_ms": 22.751411578135976,
+    "itl_p50_ms": 22.677721858262814,
+    "itl_p95_ms": 29.19536946929717,
+    "itl_p99_ms": 32.0268499520443,
+    "tpot_mean_ms": 22.753117775499057,
+    "tpot_p50_ms": 22.679262984259076,
+    "tpot_p95_ms": 29.196810057103683,
+    "acceptance_rate": 0.31057058500760176,
+    "acceptance_pos_1": 0.5710008489248474,
+    "acceptance_pos_2": 0.2600387645252867,
+    "acceptance_pos_3": 0.10067214157267132,
+    "mean_acceptance_length": 1.9317117550228053
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 5000,
+    "output_tokens": 635796,
+    "request_throughput_rps": 1.2415631421695674,
+    "output_throughput_tps": 157.87617590776844,
+    "speedup_vs_baseline": 1.0155389811481785,
+    "latency_mean_ms": 3221.0815614180046,
+    "latency_p50_ms": 3164.543153499835,
+    "latency_p95_ms": 4123.292088050039,
+    "latency_p99_ms": 4616.265497698114,
+    "ttft_mean_ms": 352.63900740198807,
+    "ttft_p50_ms": 287.9059334991325,
+    "ttft_p95_ms": 612.3089999979129,
+    "ttft_p99_ms": 1196.3210188611995,
+    "itl_mean_ms": 22.7251370702789,
+    "itl_p50_ms": 22.450031862195097,
+    "itl_p95_ms": 29.62520190592225,
+    "itl_p99_ms": 33.215207786221775,
+    "tpot_mean_ms": 22.72683404563422,
+    "tpot_p50_ms": 22.451526275598177,
+    "tpot_p95_ms": 29.626681563398265,
+    "acceptance_rate": 0.1436652803641537,
+    "acceptance_pos_1": 0.5763152990672803,
+    "acceptance_pos_2": 0.26589619110489227,
+    "acceptance_pos_3": 0.10326562455582894,
+    "mean_acceptance_length": 2.005656962549076
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 5000,
+    "output_tokens": 636215,
+    "request_throughput_rps": 1.5004265390600509,
+    "output_throughput_tps": 190.91877410961806,
+    "speedup_vs_baseline": 1.2280855944636573,
+    "latency_mean_ms": 2665.1578789277846,
+    "latency_p50_ms": 2665.9706919999735,
+    "latency_p95_ms": 3441.952741001842,
+    "latency_p99_ms": 3755.308294759525,
+    "ttft_mean_ms": 264.3723180442066,
+    "ttft_p50_ms": 253.26144649989146,
+    "ttft_p95_ms": 429.1125219519018,
+    "ttft_p99_ms": 559.1942446103716,
+    "itl_mean_ms": 19.008812154313944,
+    "itl_p50_ms": 18.913669216526902,
+    "itl_p95_ms": 24.80109081614243,
+    "itl_p99_ms": 27.068464386929747,
+    "tpot_mean_ms": 19.010452079968594,
+    "tpot_p50_ms": 18.915089492132775,
+    "tpot_p95_ms": 24.80313036299977,
+    "acceptance_rate": 0.32401200895393567,
+    "acceptance_pos_1": 0.5557804733415505,
+    "acceptance_pos_2": 0.28182543551086175,
+    "acceptance_pos_3": 0.1344301180093948,
+    "mean_acceptance_length": 1.9720360268618071
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 5000,
+    "output_tokens": 636157,
+    "request_throughput_rps": 1.4902242443540412,
+    "output_throughput_tps": 189.60331692310675,
+    "speedup_vs_baseline": 1.2196239121151176,
+    "latency_mean_ms": 2683.3432791144196,
+    "latency_p50_ms": 2638.018617002672,
+    "latency_p95_ms": 3469.7974141512536,
+    "latency_p99_ms": 3857.7436888694515,
+    "ttft_mean_ms": 287.66377756739894,
+    "ttft_p50_ms": 259.45143649914826,
+    "ttft_p95_ms": 433.6856615469515,
+    "ttft_p99_ms": 568.5856753819111,
+    "itl_mean_ms": 18.969870524426348,
+    "itl_p50_ms": 18.7173821692899,
+    "itl_p95_ms": 25.069323533594044,
+    "itl_p99_ms": 28.087401625863414,
+    "tpot_mean_ms": 18.971464480602076,
+    "tpot_p50_ms": 18.718672405515193,
+    "tpot_p95_ms": 25.071671481768842,
+    "acceptance_rate": 0.15400452166690387,
+    "acceptance_pos_1": 0.5521141942543566,
+    "acceptance_pos_2": 0.27944601414362474,
+    "acceptance_pos_3": 0.13266262718646393,
+    "mean_acceptance_length": 2.078031651668327
+  }
+]

benchmarks/bs4/run_config.json

@@ -0,0 +1,108 @@
+{
+  "args": {
+    "base_model": "google/t5gemma-2-1b-1b",
+    "dflash_model": "/mnt/d/Projects/dflash_t5/dflash-output/dflash_t5gemma2_mixture_200k/checkpoints/5",
+    "dflare_model": "/mnt/d/Projects/dflash_t5/dflash-output/dflare_t5gemma2_mixture_200k/checkpoints/5",
+    "dspark_model": "/mnt/d/Projects/dflash_t5/dflash-output/dspark_t5gemma2_mixture_200k/checkpoints/5",
+    "speculative_token_counts": [
+      3,
+      7
+    ],
+    "methods": [
+      "dflash",
+      "dflare",
+      "dspark"
+    ],
+    "datasets": [
+      "math500",
+      "gsm8k",
+      "humaneval",
+      "mbpp",
+      "mt_bench",
+      "ultrachat"
+    ],
+    "num_prompts": 0,
+    "ultrachat_num_prompts": 5000,
+    "warmup": 8,
+    "concurrency": 4,
+    "max_num_seqs": 4,
+    "max_num_batched_tokens": 4096,
+    "max_model_len": 2048,
+    "max_tokens": 128,
+    "gpu_memory_utilization": 0.85,
+    "host": "127.0.0.1",
+    "port": 8011,
+    "temperature": 0.0,
+    "top_p": 1.0,
+    "seed": 0,
+    "timeout_s": 180.0,
+    "server_timeout_s": 600.0,
+    "enforce_eager": false,
+    "include_baseline": true
+  },
+  "runs": [
+    {
+      "name": "baseline",
+      "checkpoint": null,
+      "num_speculative_tokens": null
+    },
+    {
+      "name": "dflash_k3",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dflash_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 3
+    },
+    {
+      "name": "dflash_k7",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dflash_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 7
+    },
+    {
+      "name": "dflare_k3",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dflare_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 3
+    },
+    {
+      "name": "dflare_k7",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dflare_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 7
+    },
+    {
+      "name": "dspark_k3",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dspark_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 3
+    },
+    {
+      "name": "dspark_k7",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dspark_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 7
+    }
+  ],
+  "prompt_sets": {
+    "math500": {
+      "count": 500,
+      "sha256": "78a67699bc700588d0e23da6457ea3afc00d5635897d456a6245f1e964449c4a"
+    },
+    "gsm8k": {
+      "count": 1319,
+      "sha256": "d591bbef95c9d5b17f6f26999bde24def2f6fbf25d7e20953b9ac89c65ffbb8e"
+    },
+    "humaneval": {
+      "count": 164,
+      "sha256": "de1066c5b81dc0f99eab434315cee2c3c68c5a522b8c25d53c73f9c9042b44c8"
+    },
+    "mbpp": {
+      "count": 500,
+      "sha256": "c596be7f77834627fd0a05755691e30dbf4f6724425842de6d03348614e5fecd"
+    },
+    "mt_bench": {
+      "count": 160,
+      "sha256": "8761bfece58bd26e5b93c33e9aad480728ea9950a7d257aaff267db5c7b98010"
+    },
+    "ultrachat": {
+      "count": 5000,
+      "sha256": "5dac9e9dc0d63372c701dbebbb18f3f22a568e9b62cc0ff1b4fb88fc41bf5bf7"
+    }
+  },
+  "benchmark_fingerprint": "f9d631623c44c4276ee1b29d5e2e5ce03c216e6d5c95ffda6333d549323e0c89",
+  "output_dir": "/mnt/d/Projects/dflash_t5/benchmark-results/t5gemma2_speculators_suite_bs4_fixed"
+}

config.json

@@ -0,0 +1,87 @@
+{
+  "architectures": [
+    "DFlashDraftModel"
+  ],
+  "auto_map": {
+    "": "config.DFlashSpeculatorConfig"
+  },
+  "aux_hidden_state_layer_ids": [
+    2,
+    7,
+    13,
+    18,
+    21
+  ],
+  "block_size": 8,
+  "draft_vocab_size": 32000,
+  "dtype": "bfloat16",
+  "mask_token_id": 4,
+  "sliding_window_non_causal": false,
+  "speculators_config": {
+    "algorithm": "dflash",
+    "default_proposal_method": "greedy",
+    "proposal_methods": [
+      {
+        "accept_tolerance": 0.0,
+        "proposal_type": "greedy",
+        "speculative_tokens": 7,
+        "verifier_accept_k": 1
+      }
+    ],
+    "verifier": {
+      "architectures": [
+        "T5Gemma2ForConditionalGeneration"
+      ],
+      "name_or_path": "google/t5gemma-2-1b-1b"
+    }
+  },
+  "speculators_model_type": "dflash",
+  "speculators_version": "0.7.0.dev72",
+  "target_hidden_size": null,
+  "tie_word_embeddings": false,
+  "transformer_layer_config": {
+    "architectures": [
+      "T5Gemma2ForConditionalGeneration"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bos_token_id": 1,
+    "eos_token_id": 2,
+    "head_dim": 256,
+    "hidden_act": "silu",
+    "hidden_size": 1152,
+    "initializer_range": 0.02,
+    "intermediate_size": 6912,
+    "layer_types": [
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 32768,
+    "mlp_bias": false,
+    "model_type": "llama",
+    "num_attention_heads": 4,
+    "num_hidden_layers": 5,
+    "num_key_value_heads": 1,
+    "pad_token_id": null,
+    "pretraining_tp": 1,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "factor": 8.0,
+      "rope_theta": 1000000,
+      "rope_type": "linear"
+    },
+    "rope_theta": 1000000,
+    "sliding_window": 2048,
+    "tie_word_embeddings": false,
+    "use_cache": true,
+    "verifier_final_logit_softcapping": null,
+    "verifier_hidden_states_are_normalized": true,
+    "verifier_is_encoder_decoder": true,
+    "verifier_model_type": "t5gemma2",
+    "vocab_size": 262144
+  },
+  "transformers_version": "5.12.1"
+}

config.py

@@ -0,0 +1,94 @@
+from typing import Any, Literal
+
+from pydantic import Field, field_serializer, field_validator
+from transformers import AutoConfig, PretrainedConfig
+from transformers.models.qwen3.modeling_qwen3 import (
+    Qwen3Config,
+)
+
+from speculators import SpeculatorModelConfig
+
+__all__ = [
+    "DFlashSpeculatorConfig",
+]
+
+
+@SpeculatorModelConfig.register("dflash")
+class DFlashSpeculatorConfig(SpeculatorModelConfig):
+    """
+    Configuration for DFlash speculator with vocabulary mapping.
+
+    DFlash features vocabulary mapping between draft (64K) and target (128K)
+    vocabularies, enabling cross-tokenizer speculation.
+
+    :param transformer_layer_config: Configuration for the transformer decoder layer
+    :param draft_vocab_size: Size of draft model vocabulary for speculation
+    """
+
+    speculators_model_type: Literal["dflash"] = "dflash"
+    architectures: list[str] = Field(
+        default_factory=lambda: ["DFlashSpeculator"],
+        description="Model architectures that can load these weights",
+    )
+
+    transformer_layer_config: PretrainedConfig = Field(
+        default_factory=Qwen3Config,
+        description="Configuration for the transformer decoder layer",
+    )
+
+    draft_vocab_size: int = Field(
+        default=32000,
+        description="Size of draft model vocabulary for speculation",
+    )
+
+    block_size: int = Field(
+        default=8,
+        description=(
+            "Default size of the draft block predicted with a forward pass of the model"
+        ),
+    )
+
+    target_hidden_size: int | None = Field(
+        default=None,
+        description="Hidden size of the target model (if different from draft model)",
+    )
+
+    aux_hidden_state_layer_ids: list[int] | None = Field(
+        default=None,
+        description="Layer IDs of the DFlash auxiliary hidden state layers",
+    )
+
+    mask_token_id: int | None = Field(
+        default=None,
+        description="Token ID used for masking",
+    )
+
+    sliding_window_non_causal: bool = Field(
+        default=False,
+        description="Use non-causal (bidirectional) masking within draft blocks for "
+        "sliding window attention layers. Full attention layers are always "
+        "bidirectional.",
+    )
+
+    @field_serializer("transformer_layer_config")
+    def serialize_transformer_config(self, value: PretrainedConfig) -> dict:
+        """Serialize transformer config to dict."""
+        return value.to_diff_dict()
+
+    @field_validator("transformer_layer_config", mode="before")
+    @classmethod
+    def validate_transformer_config(cls, value: Any) -> PretrainedConfig:
+        """Validate and convert transformer config."""
+        if isinstance(value, dict):
+            config_class: type[PretrainedConfig] = Qwen3Config
+            if "model_type" in value:
+                config_class = AutoConfig.for_model(
+                    model_type=value["model_type"]
+                ).__class__
+            return config_class(**value)
+        return value
+
+    @property
+    def target_vocab_size(self) -> int:
+        """Get target vocabulary size from transformer config."""
+        return self.transformer_layer_config.vocab_size

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f0e86b5328a43fd64100e7da36b854bb2805cf74f9502ddbe3a9a1e8708a1406
+size 959906136

optimizer_state_dict.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61d19ff5abc33e37952ccb110d8f31202f46b039c4c8127fd08a867ba4de17d9
+size 563401351

scheduler_state_dict.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6c2768020750de480fd72c1d5a77315c1d840eb9814df19d857f400239b6598b
+size 1531

tensorboard/dflash_t5gemma2_mixture_200k/events.out.tfevents.1783579670.DESKTOP-OU3A33R.176605.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2beb9af1f92260009dfd788480e9ee5ed0e20bec971b29a0c8bce9f9a4b6d2c3
+size 19292160

tensorboard/dflash_t5gemma2_mixture_200k/events.out.tfevents.1783592272.DESKTOP-OU3A33R.431.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7e1fea84388bcc9257bb2b7efd2bcc4b07b15293e8164dfa821dffe607297da6
+size 29485

tensorboard/dflash_t5gemma2_mixture_200k/events.out.tfevents.1783592392.DESKTOP-OU3A33R.873.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4db6f5f57a1cfd6437f77bfffab1f298703a50520ba10b6f3c3110586c13b52
+size 148582

tensorboard/dflash_t5gemma2_mixture_200k/events.out.tfevents.1783592508.DESKTOP-OU3A33R.1385.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7493048a4a509e0431030137e630a2997ce1a0d0705c28070626a05db08fb02a
+size 153939710

train_command.txt

@@ -0,0 +1,9 @@
+# Timestamp: 2026-07-09T10:21:35.298654+00:00
+# Git SHA: a58573acefd975901bae8e83b6e8b5e6ace717ef
+# World size: 1
+# speculators: 0.7.0.dev72
+# vllm: not installed
+# transformers: 5.12.1
+# torch: 2.12.0
+# compressed-tensors: not installed
+scripts/train_t5gemma_online.py --verifier-name-or-path google/t5gemma-2-1b-1b --data-path /home/pc/dflash-output/t5gemma2_mixture_200k --save-path ../dflash-output/dflash_t5gemma2_mixture_200k/checkpoints --draft-vocab-size 32000 --speculator-type dflash --draft-arch llama --draft-hidden-act silu --draft-attn-impl eager --block-size 8 --max-anchors 256 --num-layers 5 --target-layer-ids 2 7 13 18 21 --total-seq-len 2048 --epochs 6 --lr 6e-4 --loss-fn kl_div --logger tensorboard --log-dir /home/pc/dflash-output/tensorboard/t5gemma2_mixture_200k --run-name dflash_t5gemma2_mixture_200k --on-missing raise --num-workers 0 --prefetch-factor 1

training/data_config.json

@@ -0,0 +1,46 @@
+{
+  "model": "google/t5gemma-2-1b-1b",
+  "mixture": "t5gemma2_200k",
+  "samples": 200000,
+  "seed": 42,
+  "sources": [
+    {
+      "source": "ultrachat",
+      "converter": "auto",
+      "requested": 60000,
+      "kept": 60000
+    },
+    {
+      "source": "sharegpt",
+      "converter": "auto",
+      "requested": 20000,
+      "kept": 20000
+    },
+    {
+      "source": "hf:ise-uiuc/Magicoder-Evol-Instruct-110K::train",
+      "converter": "instruction_response",
+      "requested": 50000,
+      "kept": 50000
+    },
+    {
+      "source": "hf:Salesforce/xlam-function-calling-60k::train",
+      "converter": "xlam_function_calling",
+      "requested": 40000,
+      "kept": 40000
+    },
+    {
+      "source": "hf:glaiveai/glaive-function-calling-v2::train",
+      "converter": "glaive_function_calling",
+      "requested": 20000,
+      "kept": 20000
+    },
+    {
+      "source": "hf:TIGER-Lab/MathInstruct::train",
+      "converter": "instruction_output",
+      "requested": 10000,
+      "kept": 10000
+    }
+  ],
+  "decoder_start_token_id": 2,
+  "eos_token_id": 1
+}

training/train.sh

@@ -0,0 +1,70 @@
+#!/bin/bash
+set -euo pipefail
+
+MODEL="${MODEL:-google/t5gemma-2-1b-1b}"
+MIXTURE="${MIXTURE:-t5gemma2_200k}"
+DATASET="${DATASET:-}"
+DATA_DIR="${DATA_DIR:-$HOME/dflash-output/t5gemma2_mixture_200k}"
+OUTPUT_DIR="${OUTPUT_DIR:-../dflash-output/dflash_t5gemma2_mixture_200k}"
+LOG_DIR="${LOG_DIR:-$HOME/dflash-output/tensorboard/t5gemma2_mixture_200k}"
+RUN_NAME="${RUN_NAME:-dflash_t5gemma2_mixture_200k}"
+MAX_SAMPLES="${MAX_SAMPLES:-200000}"
+SEED="${SEED:-42}"
+# Number of samples processed by the frozen verifier in one online extraction
+# microbatch. Increase to 8/16 for higher GPU utilization if VRAM allows;
+# reduce to 1/2 if full-length samples OOM.
+T5GEMMA_ONLINE_EXTRACTOR_BATCH_SIZE="${T5GEMMA_ONLINE_EXTRACTOR_BATCH_SIZE:-32}"
+# Use 5 verifier layers, evenly spaced across the decoder, matching the DFlash
+# paper's setup (5 target hidden states between the early and late layers).
+TARGET_LAYER_IDS=(2 7 13 18 21)
+
+# The Arrow dataset is shared by DFlash/DFlare/DSpark runs. It stores only
+# encoder/decoder token IDs and masks; hidden states are extracted online.
+if [[ ! -f "$DATA_DIR/dataset_info.json" ]]; then
+    PREPARE_ARGS=(
+        scripts/prepare_t5gemma_data.py
+        --model "$MODEL"
+        --output "$DATA_DIR"
+        --max-samples "$MAX_SAMPLES"
+        --seed "$SEED"
+        --encoder-seq-length 2048
+        --decoder-seq-length 1024
+    )
+    if [[ -n "$MIXTURE" ]]; then
+        PREPARE_ARGS+=(--mixture "$MIXTURE")
+    else
+        PREPARE_ARGS+=(--data "$DATASET")
+    fi
+    python "${PREPARE_ARGS[@]}"
+else
+    echo "Prepared dataset already exists at $DATA_DIR; reusing it."
+fi
+
+# The verifier is kept frozen on the same GPU as the drafter. num_workers must
+# remain zero: worker processes cannot share this in-process CUDA model.
+PYTORCH_ALLOC_CONF=expandable_segments:True \
+TORCHDYNAMO_DISABLE=1 \
+T5GEMMA_ONLINE_EXTRACTOR_BATCH_SIZE="$T5GEMMA_ONLINE_EXTRACTOR_BATCH_SIZE" \
+python scripts/train_t5gemma_online.py \
+    --verifier-name-or-path "$MODEL" \
+    --data-path "$DATA_DIR" \
+    --save-path "$OUTPUT_DIR/checkpoints" \
+    --draft-vocab-size 32000 \
+    --speculator-type dflash \
+    --draft-arch llama \
+    --draft-hidden-act silu \
+    --draft-attn-impl eager \
+    --block-size 8 \
+    --max-anchors 256 \
+    --num-layers 5 \
+    --target-layer-ids "${TARGET_LAYER_IDS[@]}" \
+    --total-seq-len 2048 \
+    --epochs 6 \
+    --lr 6e-4 \
+    --loss-fn kl_div \
+    --logger tensorboard \
+    --log-dir "$LOG_DIR" \
+    --run-name "$RUN_NAME" \
+    --on-missing raise \
+    --num-workers 0 \
+    --prefetch-factor 1

training_state.json

@@ -0,0 +1 @@
+{"epoch": 5, "local_step": 0, "global_step": 146486}

val_metrics.json

@@ -0,0 +1 @@
+{"loss_epoch": 1.2283039277097254, "full_acc_epoch": 0.3451195259581689, "position_1_acc_epoch": 0.6789729837541473, "position_2_acc_epoch": 0.4917242393918339, "position_3_acc_epoch": 0.36788837073739433, "position_4_acc_epoch": 0.2876449677914155, "position_5_acc_epoch": 0.23299416992991076, "position_6_acc_epoch": 0.19331019995240967, "position_7_acc_epoch": 0.16208325787625819}