Upload folder using huggingface_hub

.gitattributes

@@ -1,35 +1,4 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
 *.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
 *.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text

MANIFEST.md

@@ -0,0 +1,26 @@
+# Release Manifest
+
+Source checkpoint:
+
+```text
+training/output/dflash-onpolicy-20k-b4-l1-v28k/checkpoints/checkpoint_best
+```
+
+Files:
+
+```text
+d88d201dd9282be01c1daef19f65f3e65a03beef9d5cd69156b95ff2544eadd9  config.json
+b71d94e8ac2f7dc32c610046209d18cac058655df6092af0844f655805350a17  config.py
+21f3f87933e9692044f5c62121c1bac39602ea8226a076a91a431cc5831108ac  val_metrics.json
+e7ffbde19080c85333332bd25f214de07274e172e01fa140aaee806d5c635efc  model.safetensors
+```
+
+Primary benchmark artifacts copied from `results/`:
+
+```text
+dflash20k-b4-l1-v28k-elyza100-streaming.json
+dflash20k-b4-l1-v28k-elyza100-streaming.csv
+elyza100-baseline-vs-dflash-summary.json
+elyza100-baseline-vs-dflash-speedup.png
+dflash20k-b4-l1-elyza100-tps-histogram.png
+```

README.md

@@ -0,0 +1,115 @@
+---
+license: apache-2.0
+base_model:
+- kel-jp/llm-jp-4-8b-instruct-NVFP4
+library_name: speculators
+inference: false
+language:
+- ja
+- en
+tags:
+- speculative-decoding
+- speculative-decoding-draft
+- dflash
+- draft-model
+- speculator
+- speculators
+- vllm
+- llm-jp
+- japanese
+- nvfp4
+- dgx-spark
+---
+
+# llm-jp-4-8b-instruct-NVFP4 DFlash Drafter
+
+This repository contains a DFlash draft model for speculative decoding with
+[`kel-jp/llm-jp-4-8b-instruct-NVFP4`](https://huggingface.co/kel-jp/llm-jp-4-8b-instruct-NVFP4).
+
+This is not a standalone text-generation model. Use it as the draft/speculator
+model while serving the NVFP4 model as the verifier in stock vLLM.
+
+## Intended Use
+
+- Verifier model: `kel-jp/llm-jp-4-8b-instruct-NVFP4`
+- Drafting method: `dflash`
+- Draft architecture: block size 4, 1 transformer layer, 28k draft vocabulary
+- Proposal setting used in the released config: greedy, `speculative_tokens=3`
+- Runtime target: stock vLLM with DFlash support
+
+## vLLM Example
+
+```bash
+vllm serve kel-jp/llm-jp-4-8b-instruct-NVFP4 \
+  --trust-remote-code \
+  --reasoning-parser llmjp4 \
+  --speculative-config '{"method":"dflash","model":"kel-jp/llm-jp-4-8b-instruct-NVFP4-speculator.dflash","num_speculative_tokens":3}'
+```
+
+If you use an environment wrapper, set the same JSON as `SPECULATIVE_CONFIG`.
+The verifier model still needs the serving requirements documented on the NVFP4
+model page, including the LLM-jp remote code/plugin setup.
+
+## Benchmark
+
+Representative Japanese streaming decode benchmark on ELYZA-100 prompts:
+
+| Setup | Mean decode tok/s | Notes |
+| --- | ---: | --- |
+| Baseline NVFP4 | 29.73 | no speculative decoding |
+| NVFP4 + this DFlash drafter | 48.76 paired / 49.04 standalone run | stock vLLM, DFlash |
+
+Paired speedup summary:
+
+- Ratio of mean decode throughput: 1.64x
+- Mean paired request speedup: 1.64x
+- Median paired request speedup: 1.63x
+- p05/p95 paired speedup: 1.37x / 1.97x
+- Requests at least 1.5x faster: 70 / 100
+- Requests at least 2.0x faster: 5 / 100
+- Requests slower than baseline: 0 / 100
+
+The raw benchmark artifacts are included under [`benchmark/`](benchmark/):
+
+- `dflash20k-b4-l1-v28k-elyza100-streaming.json`
+- `dflash20k-b4-l1-v28k-elyza100-streaming.csv`
+- `elyza100-baseline-vs-dflash-summary.json`
+- `elyza100-baseline-vs-dflash-speedup.png`
+- `dflash20k-b4-l1-elyza100-tps-histogram.png`
+
+![Decode speedup distribution](benchmark/elyza100-baseline-vs-dflash-speedup.png)
+
+## Validation Metrics
+
+The released checkpoint's held-out validation metrics:
+
+```json
+{
+  "loss_epoch": 2.047496609403255,
+  "full_acc_epoch": 0.4021220020978507,
+  "position_1_acc_epoch": 0.5539027137301571,
+  "position_2_acc_epoch": 0.38056573750461736,
+  "position_3_acc_epoch": 0.27111557473966125
+}
+```
+
+## Files
+
+- `model.safetensors`: released drafter weights
+- `config.json`: DFlash/speculators configuration consumed by vLLM
+- `config.py`: custom config class required by the drafter
+- `val_metrics.json`: validation metrics from the selected checkpoint
+- `benchmark/`: benchmark summaries, per-request CSV/JSON, and plots
+
+## Limitations
+
+- This artifact is useful only with a compatible verifier model and DFlash-capable
+  vLLM runtime.
+- The published benchmark is throughput-oriented and uses Japanese ELYZA-100
+  prompts. It is not a general quality evaluation of the verifier model.
+- Speedup depends on prompt mix, max token settings, batching, hardware, and vLLM
+  version.
+
+## License
+
+Apache-2.0, following the verifier model release.

UPLOAD.md

@@ -0,0 +1,63 @@
+# Publishing Instructions
+
+Recommended Hub repository:
+
+```text
+kel-jp/llm-jp-4-8b-instruct-NVFP4-speculator.dflash
+```
+
+Publish this as a new model repository, not as a new revision of
+`kel-jp/llm-jp-4-8b-instruct-NVFP4`. The NVFP4 repository is the verifier model;
+this repository is a companion draft model used through vLLM speculative decoding.
+
+## CLI Upload
+
+```bash
+cd /home/dgx/ege/kel-jp
+hf auth login
+HF_XET_HIGH_PERFORMANCE=1 hf upload \
+  kel-jp/llm-jp-4-8b-instruct-NVFP4-speculator.dflash \
+  release/llm-jp-4-8b-instruct-NVFP4-speculator.dflash \
+  .
+```
+
+If the repo does not exist yet, create it first:
+
+```bash
+hf repos create kel-jp/llm-jp-4-8b-instruct-NVFP4-speculator.dflash --type model
+```
+
+To create it private instead:
+
+```bash
+hf repos create kel-jp/llm-jp-4-8b-instruct-NVFP4-speculator.dflash --type model --private
+```
+
+## Python Upload
+
+```python
+from huggingface_hub import HfApi, create_repo
+
+repo_id = "kel-jp/llm-jp-4-8b-instruct-NVFP4-speculator.dflash"
+local_dir = "/home/dgx/ege/kel-jp/release/llm-jp-4-8b-instruct-NVFP4-speculator.dflash"
+
+create_repo(repo_id, repo_type="model", exist_ok=True)
+
+HfApi().upload_folder(
+    repo_id=repo_id,
+    repo_type="model",
+    folder_path=local_dir,
+    commit_message="Release DFlash drafter for llm-jp-4-8b-instruct-NVFP4",
+)
+```
+
+## Suggested First Release Tag
+
+After upload, tag the initial checkpoint as:
+
+```text
+b4-l1-v28k-elyza100
+```
+
+Use later tags or branches for future drafters with different block sizes,
+layer counts, vocab widths, verifier revisions, or benchmark suites.

benchmark/dflash20k-b4-l1-v28k-elyza100-streaming.csv

@@ -0,0 +1,101 @@
+prompt_tokens,completion_tokens,ttft_ms,decode_seconds,decode_tokens_per_second,e2e_seconds,e2e_tokens_per_second,index,prompt_characters
+89,128,240.89696002192795,2.545692512008827,49.888193252288445,2.786589472030755,45.934286799238755,0,28
+100,128,257.2139200055972,3.243633327947464,39.15362408745634,3.500847247953061,36.562577837362475,1,50
+86,128,147.0602560439147,2.0904240959789604,60.753222393623915,2.237484352022875,57.20710398902999,2,24
+223,128,246.14515202119946,2.5264487200183794,50.26818830467934,2.772593872039579,46.16615555953764,3,323
+87,128,149.6418240130879,2.5420003199833445,49.960654607955405,2.6916421439964324,47.55461281712256,4,15
+138,128,184.6838559722528,2.7219324320321903,46.65802813671683,2.906616288004443,44.037460509752826,5,107
+197,128,191.15900801261887,2.903922384022735,43.73395125804633,3.095081392035354,41.35593988881372,6,286
+129,128,151.36272000381723,2.6518156320089474,47.89171557292166,2.8031783520127647,45.662453089398404,7,77
+253,128,189.83232002938166,2.5254691839800216,50.2876854746863,2.7153015040094033,47.140253047772305,8,359
+107,128,181.43943999893963,2.6863999040215276,47.27516547699455,2.8678393440204673,44.63290465237668,9,76
+115,128,185.43846398824826,2.791916431975551,45.48846754346985,2.977354895963799,42.99117991392999,10,93
+136,128,187.10769596509635,3.0453063839813694,41.70352141513028,3.232414079946466,39.598887034336855,11,126
+239,128,158.63804804394022,2.879851776000578,44.099491876061926,3.038489824044518,42.12619011822783,12,301
+113,128,151.9024320296012,2.936355519981589,43.25089354329829,3.0882579520111904,41.44731495522955,13,76
+239,128,158.4336319938302,2.4920913279638626,50.961214211906125,2.6505249599576928,48.29232017571459,14,323
+122,128,219.40715197706595,2.4481918240198866,51.875020067450556,2.6675989759969525,47.983224297108976,15,120
+152,128,183.11686400556937,2.5510446240077727,49.78352742433762,2.734161488013342,46.81508409841789,16,182
+141,128,154.21734401024878,2.864302864007186,44.338886643546424,3.018520208017435,42.40488424096735,17,140
+196,128,159.43828801391646,2.804585279955063,45.282987437641715,2.9640235679689795,43.18454191229954,18,191
+118,128,153.91444804845378,2.8666789119597524,44.302136339775416,3.020593360008206,42.37578010157986,19,75
+269,128,159.85811199061573,2.6029292000457644,48.791184945701595,2.76278731203638,46.330023104693666,20,368
+151,128,218.98252802202478,2.907417631999124,43.68137504644474,3.1264001600211486,40.941656041603494,21,137
+83,128,216.05988795636222,2.797060272016097,45.40481350030384,3.013120159972459,42.4808813469855,22,8
+257,128,162.47585602104664,2.4971475200145505,50.858028603476335,2.659623376035597,48.12711497174283,23,379
+264,128,191.5586399845779,2.643744495988358,48.03792506904921,2.8353031359729357,45.14508462111115,24,480
+115,128,187.85404798109084,2.101830688014161,60.42351590174534,2.2896847359952517,55.90289265057381,25,82
+167,128,154.14873597910628,2.2119666560320184,57.414970362989806,2.3661153920111246,54.09710804138084,26,145
+86,128,146.7720319633372,2.3094228000263683,54.992095859861585,2.4561948319897056,52.11312976190501,27,11
+95,128,181.20713595999405,2.489829232043121,51.00751423654283,2.671036368003115,47.92147405154714,28,33
+138,128,184.11780800670385,2.6619341599871404,47.70966987425922,2.8460519679938443,44.974582839478515,29,155
+89,128,151.55057597439736,2.448815823998302,51.86180142883953,2.6003663999726996,49.22383245735825,30,23
+122,128,153.96367997163907,2.561781839991454,49.57486934188887,2.715745519963093,47.13254576288117,31,97
+94,128,150.61763202538714,2.6968333919649012,47.09226768638768,2.8474510239902884,44.95248519520684,32,61
+132,128,186.14015995990485,2.2456431520404294,56.553954213342244,2.4317833120003343,52.636268769650314,33,136
+114,128,154.66494398424402,2.8738129920093343,44.192158763679046,3.0284779359935783,42.26545568607749,34,82
+97,128,179.51884801732376,2.5915289919939823,49.005818724136,2.771047840011306,46.191912731278485,35,49
+116,128,155.2071999758482,2.419067424023524,52.49957018096119,2.574274623999372,49.72274473231618,36,101
+127,128,185.88307203026488,3.1969690079567954,39.72512704499646,3.3828520799870603,37.837894466993546,37,118
+99,128,181.75656005041674,2.491517999966163,50.97294099489739,2.67327456001658,47.88135192488651,38,44
+89,128,183.00852802349254,2.558840816025622,49.63184861075325,2.7418493440491147,46.683819546033796,39,21
+131,128,153.34939199965447,3.088830400025472,41.11588645299292,3.2421797920251265,39.47961193109799,40,99
+164,128,152.6076800073497,2.217368912009988,57.27508819670325,2.3699765920173377,54.00897225362284,41,255
+119,128,186.0543679795228,2.4219749600160867,52.436545421244354,2.6080293279956095,49.079202686103955,42,98
+109,128,182.9020160366781,2.386754415987525,53.210334146361454,2.5696564320242032,49.812106554326476,43,69
+116,128,184.15585602633655,2.352350127999671,53.988561689154196,2.5365059840260074,50.46311769264393,44,63
+99,128,150.65185603452846,2.1394616479519755,59.36072755572516,2.290113503986504,55.89242619511418,45,51
+87,128,216.31857595639303,2.6657414400251582,47.64152970469688,2.8820600159815513,44.412676797227164,46,16
+121,128,156.02417598711327,2.62969036801951,48.29465915245645,2.7857145440066233,45.94871368833822,47,83
+103,128,150.48937598476186,2.4914338079979643,50.97466350191864,2.641923183982726,48.44955401278499,48,52
+128,128,183.87961603002623,2.286609119968489,55.54075635006221,2.470488735998515,51.81161044568184,49,82
+116,128,154.44545599166304,2.423129935981706,52.41155173486282,2.577575391973369,49.659071233607776,50,89
+202,128,191.38249597745016,2.990655552013777,42.465605881788605,3.1820380479912274,40.22579179428872,51,202
+133,128,153.75451202271506,2.5296551039791666,50.20447245959658,2.6834096160018817,47.70050730857567,52,73
+97,128,148.5070079797879,2.6631859360495582,47.687244920039525,2.811692944029346,45.52417442018663,53,42
+208,128,194.71399998292327,2.8161441440461203,45.09712340843875,3.0108581440290436,42.51279664365525,54,254
+113,128,154.7356799710542,3.018775232019834,42.07004173511305,3.1735109119908884,40.33387738367654,55,85
+127,128,155.87502403650433,2.9150032319594175,43.56770469672264,3.070878255995922,41.68188685112432,56,107
+144,128,153.51360000204295,2.4251393599552102,52.36812452804591,2.578652959957253,49.638319691581096,57,121
+230,128,160.93422402627766,2.8165897599537857,45.08998854063994,2.9775239839800633,42.98873852525685,58,346
+124,128,188.24977602344006,3.1212593279778957,40.6887049921215,3.309509104001336,38.67643084747603,59,93
+170,128,186.31780799478292,2.6343118080403656,48.20993460697192,2.8206296160351485,45.37993902933088,60,207
+148,128,152.77222398435697,2.597186352009885,48.89907106654803,2.749958575994242,46.546155682989465,61,122
+123,128,154.4233439490199,2.4211461600498296,52.45449535247646,2.5755695039988495,49.69774638240831,62,88
+98,128,181.36867199791595,2.244091311993543,56.59306255554249,2.425459983991459,52.77349486069721,63,43
+196,128,190.48356800340116,2.921752463968005,43.46706354018864,3.112236031971406,41.127986015546526,64,177
+280,128,196.39590400038287,2.8607529280125163,44.3939071970931,3.057148832012899,41.869077049716864,65,494
+100,128,182.66371201025322,2.8731081599835306,44.203000001478536,3.055771871993784,41.88794365610954,66,56
+111,128,150.77420801389962,2.348080655967351,54.086728101628665,2.4988548639812507,51.223463133055496,67,68
+222,128,160.28342401841655,2.040870528027881,62.22834729389802,2.2011539520462975,58.151316440635654,68,333
+86,128,182.60081601329148,2.1359057759982534,59.45955174012501,2.318506592011545,55.20795172246922,69,20
+116,128,152.41059195250273,2.3862265280331485,53.222105490831154,2.5386371199856512,50.42075489730627,70,70
+122,128,155.00452800188214,2.5244410719606094,50.308165799792405,2.6794455999624915,47.77107622628794,71,100
+123,128,188.91860800795257,3.0515632319729775,41.61801357066705,3.24048183998093,39.50029851139461,72,100
+304,128,162.21892804605886,2.8239016799489036,44.973237171025715,2.9861206079949625,42.864979953353554,73,516
+96,128,181.1951840063557,2.3167113279923797,54.8190870677254,2.4979065119987354,51.242910567369066,74,51
+98,128,185.27556798653677,2.910698832012713,43.632133494271905,3.0959743999992497,41.3440111132802,75,38
+146,128,153.3910240395926,2.8828183519653976,44.05411111436007,3.03620937600499,42.15783042222897,76,133
+140,128,185.16944005386904,2.3894321440020576,53.15070374306082,2.5746015840559267,49.71643022076985,77,158
+89,128,150.22780798608437,2.3469663040013984,54.11240876508312,2.4971941119874828,51.257529154642505,78,17
+163,128,188.56855999911204,3.0211055999970995,42.037590476851236,3.2096741599962115,39.87943748163866,79,193
+154,128,154.23041599569842,2.458247360016685,51.66282371158044,2.6124777760123834,48.99563210653443,80,151
+119,128,183.22851200355217,2.7403013600269333,46.34526766017872,2.9235298720304854,43.782689284135785,81,75
+274,128,190.3801920125261,2.998549023992382,42.35381812464329,3.188929216004908,40.138865220833736,82,413
+96,128,149.46897595655173,2.839366831991356,44.72828187224054,2.9888358079479076,42.82603937614191,83,53
+187,128,192.14686402119696,3.0270581119693816,41.95492630214976,3.2192049759905785,39.761369951477924,84,243
+175,128,154.42934399470687,2.2495230400236323,56.45641219956823,2.403952384018339,53.24564698159326,85,193
+95,128,183.63060802221298,2.5285595359746367,50.226224928909055,2.7121901439968497,47.19433122464318,86,54
+135,128,153.55235198512673,2.1443000319995917,59.226786412706716,2.2978523839847185,55.70418748050059,87,153
+132,128,184.29516803007573,2.5293376480112784,50.21077359911013,2.713632816041354,47.16924089484086,88,83
+151,128,182.26567999226972,2.9498485919903032,43.05305714498091,3.132114271982573,40.86696361783067,89,154
+98,128,180.34731200896204,2.209185392013751,57.48725320161337,2.389532704022713,53.56695883865306,90,47
+120,128,184.92633604910225,2.5972612960031256,48.89766008350327,2.782187632052228,46.00696176108843,91,91
+129,128,154.7068800427951,3.1600337119889446,40.1894446626221,3.3147405920317397,38.61538978576407,92,149
+126,128,154.17582396185026,2.7378541760263033,46.38669258284846,2.8920299999881536,44.259568538543625,93,129
+176,128,184.01340796845034,2.775731760018971,45.75370063825332,2.959745167987421,43.246966456587856,94,181
+151,128,184.33799996273592,2.530220592045225,50.19325208216075,2.714558592007961,47.15315424645828,95,182
+113,128,154.31780798826367,2.4937886080006137,50.92652985604173,2.6481064159888774,48.336426069267766,96,72
+159,128,154.63102399371564,2.0045958719565533,63.35441560898941,2.159226895950269,59.28047684107214,97,144
+135,128,154.29542399942875,3.126877536007669,40.615597680921944,3.2811729600070976,39.01043972998093,98,146
+183,128,160.44097597477958,2.536386896041222,50.07122541053215,2.6968278720160015,47.46317009261484,99,214

benchmark/dflash20k-b4-l1-v28k-elyza100-streaming.json

@@ -0,0 +1,1120 @@
+{
+  "dataset": "training/data/benchmarks/elyza-tasks-100/test.csv",
+  "model": "llm-jp-4-8b-instruct-NVFP4",
+  "temperature": 0,
+  "max_tokens": 128,
+  "summary": {
+    "count": 100,
+    "mean": 49.038942024401024,
+    "median": 48.95244489534201,
+    "stdev": 5.541487810280493,
+    "min": 39.15362408745634,
+    "p05": 41.11588645299292,
+    "p25": 44.338886643546424,
+    "p75": 52.41155173486282,
+    "p95": 59.36072755572516,
+    "max": 63.35441560898941
+  },
+  "records": [
+    {
+      "prompt_tokens": 89,
+      "completion_tokens": 128,
+      "ttft_ms": 240.89696002192795,
+      "decode_seconds": 2.545692512008827,
+      "decode_tokens_per_second": 49.888193252288445,
+      "e2e_seconds": 2.786589472030755,
+      "e2e_tokens_per_second": 45.934286799238755,
+      "index": 0,
+      "prompt_characters": 28
+    },
+    {
+      "prompt_tokens": 100,
+      "completion_tokens": 128,
+      "ttft_ms": 257.2139200055972,
+      "decode_seconds": 3.243633327947464,
+      "decode_tokens_per_second": 39.15362408745634,
+      "e2e_seconds": 3.500847247953061,
+      "e2e_tokens_per_second": 36.562577837362475,
+      "index": 1,
+      "prompt_characters": 50
+    },
+    {
+      "prompt_tokens": 86,
+      "completion_tokens": 128,
+      "ttft_ms": 147.0602560439147,
+      "decode_seconds": 2.0904240959789604,
+      "decode_tokens_per_second": 60.753222393623915,
+      "e2e_seconds": 2.237484352022875,
+      "e2e_tokens_per_second": 57.20710398902999,
+      "index": 2,
+      "prompt_characters": 24
+    },
+    {
+      "prompt_tokens": 223,
+      "completion_tokens": 128,
+      "ttft_ms": 246.14515202119946,
+      "decode_seconds": 2.5264487200183794,
+      "decode_tokens_per_second": 50.26818830467934,
+      "e2e_seconds": 2.772593872039579,
+      "e2e_tokens_per_second": 46.16615555953764,
+      "index": 3,
+      "prompt_characters": 323
+    },
+    {
+      "prompt_tokens": 87,
+      "completion_tokens": 128,
+      "ttft_ms": 149.6418240130879,
+      "decode_seconds": 2.5420003199833445,
+      "decode_tokens_per_second": 49.960654607955405,
+      "e2e_seconds": 2.6916421439964324,
+      "e2e_tokens_per_second": 47.55461281712256,
+      "index": 4,
+      "prompt_characters": 15
+    },
+    {
+      "prompt_tokens": 138,
+      "completion_tokens": 128,
+      "ttft_ms": 184.6838559722528,
+      "decode_seconds": 2.7219324320321903,
+      "decode_tokens_per_second": 46.65802813671683,
+      "e2e_seconds": 2.906616288004443,
+      "e2e_tokens_per_second": 44.037460509752826,
+      "index": 5,
+      "prompt_characters": 107
+    },
+    {
+      "prompt_tokens": 197,
+      "completion_tokens": 128,
+      "ttft_ms": 191.15900801261887,
+      "decode_seconds": 2.903922384022735,
+      "decode_tokens_per_second": 43.73395125804633,
+      "e2e_seconds": 3.095081392035354,
+      "e2e_tokens_per_second": 41.35593988881372,
+      "index": 6,
+      "prompt_characters": 286
+    },
+    {
+      "prompt_tokens": 129,
+      "completion_tokens": 128,
+      "ttft_ms": 151.36272000381723,
+      "decode_seconds": 2.6518156320089474,
+      "decode_tokens_per_second": 47.89171557292166,
+      "e2e_seconds": 2.8031783520127647,
+      "e2e_tokens_per_second": 45.662453089398404,
+      "index": 7,
+      "prompt_characters": 77
+    },
+    {
+      "prompt_tokens": 253,
+      "completion_tokens": 128,
+      "ttft_ms": 189.83232002938166,
+      "decode_seconds": 2.5254691839800216,
+      "decode_tokens_per_second": 50.2876854746863,
+      "e2e_seconds": 2.7153015040094033,
+      "e2e_tokens_per_second": 47.140253047772305,
+      "index": 8,
+      "prompt_characters": 359
+    },
+    {
+      "prompt_tokens": 107,
+      "completion_tokens": 128,
+      "ttft_ms": 181.43943999893963,
+      "decode_seconds": 2.6863999040215276,
+      "decode_tokens_per_second": 47.27516547699455,
+      "e2e_seconds": 2.8678393440204673,
+      "e2e_tokens_per_second": 44.63290465237668,
+      "index": 9,
+      "prompt_characters": 76
+    },
+    {
+      "prompt_tokens": 115,
+      "completion_tokens": 128,
+      "ttft_ms": 185.43846398824826,
+      "decode_seconds": 2.791916431975551,
+      "decode_tokens_per_second": 45.48846754346985,
+      "e2e_seconds": 2.977354895963799,
+      "e2e_tokens_per_second": 42.99117991392999,
+      "index": 10,
+      "prompt_characters": 93
+    },
+    {
+      "prompt_tokens": 136,
+      "completion_tokens": 128,
+      "ttft_ms": 187.10769596509635,
+      "decode_seconds": 3.0453063839813694,
+      "decode_tokens_per_second": 41.70352141513028,
+      "e2e_seconds": 3.232414079946466,
+      "e2e_tokens_per_second": 39.598887034336855,
+      "index": 11,
+      "prompt_characters": 126
+    },
+    {
+      "prompt_tokens": 239,
+      "completion_tokens": 128,
+      "ttft_ms": 158.63804804394022,
+      "decode_seconds": 2.879851776000578,
+      "decode_tokens_per_second": 44.099491876061926,
+      "e2e_seconds": 3.038489824044518,
+      "e2e_tokens_per_second": 42.12619011822783,
+      "index": 12,
+      "prompt_characters": 301
+    },
+    {
+      "prompt_tokens": 113,
+      "completion_tokens": 128,
+      "ttft_ms": 151.9024320296012,
+      "decode_seconds": 2.936355519981589,
+      "decode_tokens_per_second": 43.25089354329829,
+      "e2e_seconds": 3.0882579520111904,
+      "e2e_tokens_per_second": 41.44731495522955,
+      "index": 13,
+      "prompt_characters": 76
+    },
+    {
+      "prompt_tokens": 239,
+      "completion_tokens": 128,
+      "ttft_ms": 158.4336319938302,
+      "decode_seconds": 2.4920913279638626,
+      "decode_tokens_per_second": 50.961214211906125,
+      "e2e_seconds": 2.6505249599576928,
+      "e2e_tokens_per_second": 48.29232017571459,
+      "index": 14,
+      "prompt_characters": 323
+    },
+    {
+      "prompt_tokens": 122,
+      "completion_tokens": 128,
+      "ttft_ms": 219.40715197706595,
+      "decode_seconds": 2.4481918240198866,
+      "decode_tokens_per_second": 51.875020067450556,
+      "e2e_seconds": 2.6675989759969525,
+      "e2e_tokens_per_second": 47.983224297108976,
+      "index": 15,
+      "prompt_characters": 120
+    },
+    {
+      "prompt_tokens": 152,
+      "completion_tokens": 128,
+      "ttft_ms": 183.11686400556937,
+      "decode_seconds": 2.5510446240077727,
+      "decode_tokens_per_second": 49.78352742433762,
+      "e2e_seconds": 2.734161488013342,
+      "e2e_tokens_per_second": 46.81508409841789,
+      "index": 16,
+      "prompt_characters": 182
+    },
+    {
+      "prompt_tokens": 141,
+      "completion_tokens": 128,
+      "ttft_ms": 154.21734401024878,
+      "decode_seconds": 2.864302864007186,
+      "decode_tokens_per_second": 44.338886643546424,
+      "e2e_seconds": 3.018520208017435,
+      "e2e_tokens_per_second": 42.40488424096735,
+      "index": 17,
+      "prompt_characters": 140
+    },
+    {
+      "prompt_tokens": 196,
+      "completion_tokens": 128,
+      "ttft_ms": 159.43828801391646,
+      "decode_seconds": 2.804585279955063,
+      "decode_tokens_per_second": 45.282987437641715,
+      "e2e_seconds": 2.9640235679689795,
+      "e2e_tokens_per_second": 43.18454191229954,
+      "index": 18,
+      "prompt_characters": 191
+    },
+    {
+      "prompt_tokens": 118,
+      "completion_tokens": 128,
+      "ttft_ms": 153.91444804845378,
+      "decode_seconds": 2.8666789119597524,
+      "decode_tokens_per_second": 44.302136339775416,
+      "e2e_seconds": 3.020593360008206,
+      "e2e_tokens_per_second": 42.37578010157986,
+      "index": 19,
+      "prompt_characters": 75
+    },
+    {
+      "prompt_tokens": 269,
+      "completion_tokens": 128,
+      "ttft_ms": 159.85811199061573,
+      "decode_seconds": 2.6029292000457644,
+      "decode_tokens_per_second": 48.791184945701595,
+      "e2e_seconds": 2.76278731203638,
+      "e2e_tokens_per_second": 46.330023104693666,
+      "index": 20,
+      "prompt_characters": 368
+    },
+    {
+      "prompt_tokens": 151,
+      "completion_tokens": 128,
+      "ttft_ms": 218.98252802202478,
+      "decode_seconds": 2.907417631999124,
+      "decode_tokens_per_second": 43.68137504644474,
+      "e2e_seconds": 3.1264001600211486,
+      "e2e_tokens_per_second": 40.941656041603494,
+      "index": 21,
+      "prompt_characters": 137
+    },
+    {
+      "prompt_tokens": 83,
+      "completion_tokens": 128,
+      "ttft_ms": 216.05988795636222,
+      "decode_seconds": 2.797060272016097,
+      "decode_tokens_per_second": 45.40481350030384,
+      "e2e_seconds": 3.013120159972459,
+      "e2e_tokens_per_second": 42.4808813469855,
+      "index": 22,
+      "prompt_characters": 8
+    },
+    {
+      "prompt_tokens": 257,
+      "completion_tokens": 128,
+      "ttft_ms": 162.47585602104664,
+      "decode_seconds": 2.4971475200145505,
+      "decode_tokens_per_second": 50.858028603476335,
+      "e2e_seconds": 2.659623376035597,
+      "e2e_tokens_per_second": 48.12711497174283,
+      "index": 23,
+      "prompt_characters": 379
+    },
+    {
+      "prompt_tokens": 264,
+      "completion_tokens": 128,
+      "ttft_ms": 191.5586399845779,
+      "decode_seconds": 2.643744495988358,
+      "decode_tokens_per_second": 48.03792506904921,
+      "e2e_seconds": 2.8353031359729357,
+      "e2e_tokens_per_second": 45.14508462111115,
+      "index": 24,
+      "prompt_characters": 480
+    },
+    {
+      "prompt_tokens": 115,
+      "completion_tokens": 128,
+      "ttft_ms": 187.85404798109084,
+      "decode_seconds": 2.101830688014161,
+      "decode_tokens_per_second": 60.42351590174534,
+      "e2e_seconds": 2.2896847359952517,
+      "e2e_tokens_per_second": 55.90289265057381,
+      "index": 25,
+      "prompt_characters": 82
+    },
+    {
+      "prompt_tokens": 167,
+      "completion_tokens": 128,
+      "ttft_ms": 154.14873597910628,
+      "decode_seconds": 2.2119666560320184,
+      "decode_tokens_per_second": 57.414970362989806,
+      "e2e_seconds": 2.3661153920111246,
+      "e2e_tokens_per_second": 54.09710804138084,
+      "index": 26,
+      "prompt_characters": 145
+    },
+    {
+      "prompt_tokens": 86,
+      "completion_tokens": 128,
+      "ttft_ms": 146.7720319633372,
+      "decode_seconds": 2.3094228000263683,
+      "decode_tokens_per_second": 54.992095859861585,
+      "e2e_seconds": 2.4561948319897056,
+      "e2e_tokens_per_second": 52.11312976190501,
+      "index": 27,
+      "prompt_characters": 11
+    },
+    {
+      "prompt_tokens": 95,
+      "completion_tokens": 128,
+      "ttft_ms": 181.20713595999405,
+      "decode_seconds": 2.489829232043121,
+      "decode_tokens_per_second": 51.00751423654283,
+      "e2e_seconds": 2.671036368003115,
+      "e2e_tokens_per_second": 47.92147405154714,
+      "index": 28,
+      "prompt_characters": 33
+    },
+    {
+      "prompt_tokens": 138,
+      "completion_tokens": 128,
+      "ttft_ms": 184.11780800670385,
+      "decode_seconds": 2.6619341599871404,
+      "decode_tokens_per_second": 47.70966987425922,
+      "e2e_seconds": 2.8460519679938443,
+      "e2e_tokens_per_second": 44.974582839478515,
+      "index": 29,
+      "prompt_characters": 155
+    },
+    {
+      "prompt_tokens": 89,
+      "completion_tokens": 128,
+      "ttft_ms": 151.55057597439736,
+      "decode_seconds": 2.448815823998302,
+      "decode_tokens_per_second": 51.86180142883953,
+      "e2e_seconds": 2.6003663999726996,
+      "e2e_tokens_per_second": 49.22383245735825,
+      "index": 30,
+      "prompt_characters": 23
+    },
+    {
+      "prompt_tokens": 122,
+      "completion_tokens": 128,
+      "ttft_ms": 153.96367997163907,
+      "decode_seconds": 2.561781839991454,
+      "decode_tokens_per_second": 49.57486934188887,
+      "e2e_seconds": 2.715745519963093,
+      "e2e_tokens_per_second": 47.13254576288117,
+      "index": 31,
+      "prompt_characters": 97
+    },
+    {
+      "prompt_tokens": 94,
+      "completion_tokens": 128,
+      "ttft_ms": 150.61763202538714,
+      "decode_seconds": 2.6968333919649012,
+      "decode_tokens_per_second": 47.09226768638768,
+      "e2e_seconds": 2.8474510239902884,
+      "e2e_tokens_per_second": 44.95248519520684,
+      "index": 32,
+      "prompt_characters": 61
+    },
+    {
+      "prompt_tokens": 132,
+      "completion_tokens": 128,
+      "ttft_ms": 186.14015995990485,
+      "decode_seconds": 2.2456431520404294,
+      "decode_tokens_per_second": 56.553954213342244,
+      "e2e_seconds": 2.4317833120003343,
+      "e2e_tokens_per_second": 52.636268769650314,
+      "index": 33,
+      "prompt_characters": 136
+    },
+    {
+      "prompt_tokens": 114,
+      "completion_tokens": 128,
+      "ttft_ms": 154.66494398424402,
+      "decode_seconds": 2.8738129920093343,
+      "decode_tokens_per_second": 44.192158763679046,
+      "e2e_seconds": 3.0284779359935783,
+      "e2e_tokens_per_second": 42.26545568607749,
+      "index": 34,
+      "prompt_characters": 82
+    },
+    {
+      "prompt_tokens": 97,
+      "completion_tokens": 128,
+      "ttft_ms": 179.51884801732376,
+      "decode_seconds": 2.5915289919939823,
+      "decode_tokens_per_second": 49.005818724136,
+      "e2e_seconds": 2.771047840011306,
+      "e2e_tokens_per_second": 46.191912731278485,
+      "index": 35,
+      "prompt_characters": 49
+    },
+    {
+      "prompt_tokens": 116,
+      "completion_tokens": 128,
+      "ttft_ms": 155.2071999758482,
+      "decode_seconds": 2.419067424023524,
+      "decode_tokens_per_second": 52.49957018096119,
+      "e2e_seconds": 2.574274623999372,
+      "e2e_tokens_per_second": 49.72274473231618,
+      "index": 36,
+      "prompt_characters": 101
+    },
+    {
+      "prompt_tokens": 127,
+      "completion_tokens": 128,
+      "ttft_ms": 185.88307203026488,
+      "decode_seconds": 3.1969690079567954,
+      "decode_tokens_per_second": 39.72512704499646,
+      "e2e_seconds": 3.3828520799870603,
+      "e2e_tokens_per_second": 37.837894466993546,
+      "index": 37,
+      "prompt_characters": 118
+    },
+    {
+      "prompt_tokens": 99,
+      "completion_tokens": 128,
+      "ttft_ms": 181.75656005041674,
+      "decode_seconds": 2.491517999966163,
+      "decode_tokens_per_second": 50.97294099489739,
+      "e2e_seconds": 2.67327456001658,
+      "e2e_tokens_per_second": 47.88135192488651,
+      "index": 38,
+      "prompt_characters": 44
+    },
+    {
+      "prompt_tokens": 89,
+      "completion_tokens": 128,
+      "ttft_ms": 183.00852802349254,
+      "decode_seconds": 2.558840816025622,
+      "decode_tokens_per_second": 49.63184861075325,
+      "e2e_seconds": 2.7418493440491147,
+      "e2e_tokens_per_second": 46.683819546033796,
+      "index": 39,
+      "prompt_characters": 21
+    },
+    {
+      "prompt_tokens": 131,
+      "completion_tokens": 128,
+      "ttft_ms": 153.34939199965447,
+      "decode_seconds": 3.088830400025472,
+      "decode_tokens_per_second": 41.11588645299292,
+      "e2e_seconds": 3.2421797920251265,
+      "e2e_tokens_per_second": 39.47961193109799,
+      "index": 40,
+      "prompt_characters": 99
+    },
+    {
+      "prompt_tokens": 164,
+      "completion_tokens": 128,
+      "ttft_ms": 152.6076800073497,
+      "decode_seconds": 2.217368912009988,
+      "decode_tokens_per_second": 57.27508819670325,
+      "e2e_seconds": 2.3699765920173377,
+      "e2e_tokens_per_second": 54.00897225362284,
+      "index": 41,
+      "prompt_characters": 255
+    },
+    {
+      "prompt_tokens": 119,
+      "completion_tokens": 128,
+      "ttft_ms": 186.0543679795228,
+      "decode_seconds": 2.4219749600160867,
+      "decode_tokens_per_second": 52.436545421244354,
+      "e2e_seconds": 2.6080293279956095,
+      "e2e_tokens_per_second": 49.079202686103955,
+      "index": 42,
+      "prompt_characters": 98
+    },
+    {
+      "prompt_tokens": 109,
+      "completion_tokens": 128,
+      "ttft_ms": 182.9020160366781,
+      "decode_seconds": 2.386754415987525,
+      "decode_tokens_per_second": 53.210334146361454,
+      "e2e_seconds": 2.5696564320242032,
+      "e2e_tokens_per_second": 49.812106554326476,
+      "index": 43,
+      "prompt_characters": 69
+    },
+    {
+      "prompt_tokens": 116,
+      "completion_tokens": 128,
+      "ttft_ms": 184.15585602633655,
+      "decode_seconds": 2.352350127999671,
+      "decode_tokens_per_second": 53.988561689154196,
+      "e2e_seconds": 2.5365059840260074,
+      "e2e_tokens_per_second": 50.46311769264393,
+      "index": 44,
+      "prompt_characters": 63
+    },
+    {
+      "prompt_tokens": 99,
+      "completion_tokens": 128,
+      "ttft_ms": 150.65185603452846,
+      "decode_seconds": 2.1394616479519755,
+      "decode_tokens_per_second": 59.36072755572516,
+      "e2e_seconds": 2.290113503986504,
+      "e2e_tokens_per_second": 55.89242619511418,
+      "index": 45,
+      "prompt_characters": 51
+    },
+    {
+      "prompt_tokens": 87,
+      "completion_tokens": 128,
+      "ttft_ms": 216.31857595639303,
+      "decode_seconds": 2.6657414400251582,
+      "decode_tokens_per_second": 47.64152970469688,
+      "e2e_seconds": 2.8820600159815513,
+      "e2e_tokens_per_second": 44.412676797227164,
+      "index": 46,
+      "prompt_characters": 16
+    },
+    {
+      "prompt_tokens": 121,
+      "completion_tokens": 128,
+      "ttft_ms": 156.02417598711327,
+      "decode_seconds": 2.62969036801951,
+      "decode_tokens_per_second": 48.29465915245645,
+      "e2e_seconds": 2.7857145440066233,
+      "e2e_tokens_per_second": 45.94871368833822,
+      "index": 47,
+      "prompt_characters": 83
+    },
+    {
+      "prompt_tokens": 103,
+      "completion_tokens": 128,
+      "ttft_ms": 150.48937598476186,
+      "decode_seconds": 2.4914338079979643,
+      "decode_tokens_per_second": 50.97466350191864,
+      "e2e_seconds": 2.641923183982726,
+      "e2e_tokens_per_second": 48.44955401278499,
+      "index": 48,
+      "prompt_characters": 52
+    },
+    {
+      "prompt_tokens": 128,
+      "completion_tokens": 128,
+      "ttft_ms": 183.87961603002623,
+      "decode_seconds": 2.286609119968489,
+      "decode_tokens_per_second": 55.54075635006221,
+      "e2e_seconds": 2.470488735998515,
+      "e2e_tokens_per_second": 51.81161044568184,
+      "index": 49,
+      "prompt_characters": 82
+    },
+    {
+      "prompt_tokens": 116,
+      "completion_tokens": 128,
+      "ttft_ms": 154.44545599166304,
+      "decode_seconds": 2.423129935981706,
+      "decode_tokens_per_second": 52.41155173486282,
+      "e2e_seconds": 2.577575391973369,
+      "e2e_tokens_per_second": 49.659071233607776,
+      "index": 50,
+      "prompt_characters": 89
+    },
+    {
+      "prompt_tokens": 202,
+      "completion_tokens": 128,
+      "ttft_ms": 191.38249597745016,
+      "decode_seconds": 2.990655552013777,
+      "decode_tokens_per_second": 42.465605881788605,
+      "e2e_seconds": 3.1820380479912274,
+      "e2e_tokens_per_second": 40.22579179428872,
+      "index": 51,
+      "prompt_characters": 202
+    },
+    {
+      "prompt_tokens": 133,
+      "completion_tokens": 128,
+      "ttft_ms": 153.75451202271506,
+      "decode_seconds": 2.5296551039791666,
+      "decode_tokens_per_second": 50.20447245959658,
+      "e2e_seconds": 2.6834096160018817,
+      "e2e_tokens_per_second": 47.70050730857567,
+      "index": 52,
+      "prompt_characters": 73
+    },
+    {
+      "prompt_tokens": 97,
+      "completion_tokens": 128,
+      "ttft_ms": 148.5070079797879,
+      "decode_seconds": 2.6631859360495582,
+      "decode_tokens_per_second": 47.687244920039525,
+      "e2e_seconds": 2.811692944029346,
+      "e2e_tokens_per_second": 45.52417442018663,
+      "index": 53,
+      "prompt_characters": 42
+    },
+    {
+      "prompt_tokens": 208,
+      "completion_tokens": 128,
+      "ttft_ms": 194.71399998292327,
+      "decode_seconds": 2.8161441440461203,
+      "decode_tokens_per_second": 45.09712340843875,
+      "e2e_seconds": 3.0108581440290436,
+      "e2e_tokens_per_second": 42.51279664365525,
+      "index": 54,
+      "prompt_characters": 254
+    },
+    {
+      "prompt_tokens": 113,
+      "completion_tokens": 128,
+      "ttft_ms": 154.7356799710542,
+      "decode_seconds": 3.018775232019834,
+      "decode_tokens_per_second": 42.07004173511305,
+      "e2e_seconds": 3.1735109119908884,
+      "e2e_tokens_per_second": 40.33387738367654,
+      "index": 55,
+      "prompt_characters": 85
+    },
+    {
+      "prompt_tokens": 127,
+      "completion_tokens": 128,
+      "ttft_ms": 155.87502403650433,
+      "decode_seconds": 2.9150032319594175,
+      "decode_tokens_per_second": 43.56770469672264,
+      "e2e_seconds": 3.070878255995922,
+      "e2e_tokens_per_second": 41.68188685112432,
+      "index": 56,
+      "prompt_characters": 107
+    },
+    {
+      "prompt_tokens": 144,
+      "completion_tokens": 128,
+      "ttft_ms": 153.51360000204295,
+      "decode_seconds": 2.4251393599552102,
+      "decode_tokens_per_second": 52.36812452804591,
+      "e2e_seconds": 2.578652959957253,
+      "e2e_tokens_per_second": 49.638319691581096,
+      "index": 57,
+      "prompt_characters": 121
+    },
+    {
+      "prompt_tokens": 230,
+      "completion_tokens": 128,
+      "ttft_ms": 160.93422402627766,
+      "decode_seconds": 2.8165897599537857,
+      "decode_tokens_per_second": 45.08998854063994,
+      "e2e_seconds": 2.9775239839800633,
+      "e2e_tokens_per_second": 42.98873852525685,
+      "index": 58,
+      "prompt_characters": 346
+    },
+    {
+      "prompt_tokens": 124,
+      "completion_tokens": 128,
+      "ttft_ms": 188.24977602344006,
+      "decode_seconds": 3.1212593279778957,
+      "decode_tokens_per_second": 40.6887049921215,
+      "e2e_seconds": 3.309509104001336,
+      "e2e_tokens_per_second": 38.67643084747603,
+      "index": 59,
+      "prompt_characters": 93
+    },
+    {
+      "prompt_tokens": 170,
+      "completion_tokens": 128,
+      "ttft_ms": 186.31780799478292,
+      "decode_seconds": 2.6343118080403656,
+      "decode_tokens_per_second": 48.20993460697192,
+      "e2e_seconds": 2.8206296160351485,
+      "e2e_tokens_per_second": 45.37993902933088,
+      "index": 60,
+      "prompt_characters": 207
+    },
+    {
+      "prompt_tokens": 148,
+      "completion_tokens": 128,
+      "ttft_ms": 152.77222398435697,
+      "decode_seconds": 2.597186352009885,
+      "decode_tokens_per_second": 48.89907106654803,
+      "e2e_seconds": 2.749958575994242,
+      "e2e_tokens_per_second": 46.546155682989465,
+      "index": 61,
+      "prompt_characters": 122
+    },
+    {
+      "prompt_tokens": 123,
+      "completion_tokens": 128,
+      "ttft_ms": 154.4233439490199,
+      "decode_seconds": 2.4211461600498296,
+      "decode_tokens_per_second": 52.45449535247646,
+      "e2e_seconds": 2.5755695039988495,
+      "e2e_tokens_per_second": 49.69774638240831,
+      "index": 62,
+      "prompt_characters": 88
+    },
+    {
+      "prompt_tokens": 98,
+      "completion_tokens": 128,
+      "ttft_ms": 181.36867199791595,
+      "decode_seconds": 2.244091311993543,
+      "decode_tokens_per_second": 56.59306255554249,
+      "e2e_seconds": 2.425459983991459,
+      "e2e_tokens_per_second": 52.77349486069721,
+      "index": 63,
+      "prompt_characters": 43
+    },
+    {
+      "prompt_tokens": 196,
+      "completion_tokens": 128,
+      "ttft_ms": 190.48356800340116,
+      "decode_seconds": 2.921752463968005,
+      "decode_tokens_per_second": 43.46706354018864,
+      "e2e_seconds": 3.112236031971406,
+      "e2e_tokens_per_second": 41.127986015546526,
+      "index": 64,
+      "prompt_characters": 177
+    },
+    {
+      "prompt_tokens": 280,
+      "completion_tokens": 128,
+      "ttft_ms": 196.39590400038287,
+      "decode_seconds": 2.8607529280125163,
+      "decode_tokens_per_second": 44.3939071970931,
+      "e2e_seconds": 3.057148832012899,
+      "e2e_tokens_per_second": 41.869077049716864,
+      "index": 65,
+      "prompt_characters": 494
+    },
+    {
+      "prompt_tokens": 100,
+      "completion_tokens": 128,
+      "ttft_ms": 182.66371201025322,
+      "decode_seconds": 2.8731081599835306,
+      "decode_tokens_per_second": 44.203000001478536,
+      "e2e_seconds": 3.055771871993784,
+      "e2e_tokens_per_second": 41.88794365610954,
+      "index": 66,
+      "prompt_characters": 56
+    },
+    {
+      "prompt_tokens": 111,
+      "completion_tokens": 128,
+      "ttft_ms": 150.77420801389962,
+      "decode_seconds": 2.348080655967351,
+      "decode_tokens_per_second": 54.086728101628665,
+      "e2e_seconds": 2.4988548639812507,
+      "e2e_tokens_per_second": 51.223463133055496,
+      "index": 67,
+      "prompt_characters": 68
+    },
+    {
+      "prompt_tokens": 222,
+      "completion_tokens": 128,
+      "ttft_ms": 160.28342401841655,
+      "decode_seconds": 2.040870528027881,
+      "decode_tokens_per_second": 62.22834729389802,
+      "e2e_seconds": 2.2011539520462975,
+      "e2e_tokens_per_second": 58.151316440635654,
+      "index": 68,
+      "prompt_characters": 333
+    },
+    {
+      "prompt_tokens": 86,
+      "completion_tokens": 128,
+      "ttft_ms": 182.60081601329148,
+      "decode_seconds": 2.1359057759982534,
+      "decode_tokens_per_second": 59.45955174012501,
+      "e2e_seconds": 2.318506592011545,
+      "e2e_tokens_per_second": 55.20795172246922,
+      "index": 69,
+      "prompt_characters": 20
+    },
+    {
+      "prompt_tokens": 116,
+      "completion_tokens": 128,
+      "ttft_ms": 152.41059195250273,
+      "decode_seconds": 2.3862265280331485,
+      "decode_tokens_per_second": 53.222105490831154,
+      "e2e_seconds": 2.5386371199856512,
+      "e2e_tokens_per_second": 50.42075489730627,
+      "index": 70,
+      "prompt_characters": 70
+    },
+    {
+      "prompt_tokens": 122,
+      "completion_tokens": 128,
+      "ttft_ms": 155.00452800188214,
+      "decode_seconds": 2.5244410719606094,
+      "decode_tokens_per_second": 50.308165799792405,
+      "e2e_seconds": 2.6794455999624915,
+      "e2e_tokens_per_second": 47.77107622628794,
+      "index": 71,
+      "prompt_characters": 100
+    },
+    {
+      "prompt_tokens": 123,
+      "completion_tokens": 128,
+      "ttft_ms": 188.91860800795257,
+      "decode_seconds": 3.0515632319729775,
+      "decode_tokens_per_second": 41.61801357066705,
+      "e2e_seconds": 3.24048183998093,
+      "e2e_tokens_per_second": 39.50029851139461,
+      "index": 72,
+      "prompt_characters": 100
+    },
+    {
+      "prompt_tokens": 304,
+      "completion_tokens": 128,
+      "ttft_ms": 162.21892804605886,
+      "decode_seconds": 2.8239016799489036,
+      "decode_tokens_per_second": 44.973237171025715,
+      "e2e_seconds": 2.9861206079949625,
+      "e2e_tokens_per_second": 42.864979953353554,
+      "index": 73,
+      "prompt_characters": 516
+    },
+    {
+      "prompt_tokens": 96,
+      "completion_tokens": 128,
+      "ttft_ms": 181.1951840063557,
+      "decode_seconds": 2.3167113279923797,
+      "decode_tokens_per_second": 54.8190870677254,
+      "e2e_seconds": 2.4979065119987354,
+      "e2e_tokens_per_second": 51.242910567369066,
+      "index": 74,
+      "prompt_characters": 51
+    },
+    {
+      "prompt_tokens": 98,
+      "completion_tokens": 128,
+      "ttft_ms": 185.27556798653677,
+      "decode_seconds": 2.910698832012713,
+      "decode_tokens_per_second": 43.632133494271905,
+      "e2e_seconds": 3.0959743999992497,
+      "e2e_tokens_per_second": 41.3440111132802,
+      "index": 75,
+      "prompt_characters": 38
+    },
+    {
+      "prompt_tokens": 146,
+      "completion_tokens": 128,
+      "ttft_ms": 153.3910240395926,
+      "decode_seconds": 2.8828183519653976,
+      "decode_tokens_per_second": 44.05411111436007,
+      "e2e_seconds": 3.03620937600499,
+      "e2e_tokens_per_second": 42.15783042222897,
+      "index": 76,
+      "prompt_characters": 133
+    },
+    {
+      "prompt_tokens": 140,
+      "completion_tokens": 128,
+      "ttft_ms": 185.16944005386904,
+      "decode_seconds": 2.3894321440020576,
+      "decode_tokens_per_second": 53.15070374306082,
+      "e2e_seconds": 2.5746015840559267,
+      "e2e_tokens_per_second": 49.71643022076985,
+      "index": 77,
+      "prompt_characters": 158
+    },
+    {
+      "prompt_tokens": 89,
+      "completion_tokens": 128,
+      "ttft_ms": 150.22780798608437,
+      "decode_seconds": 2.3469663040013984,
+      "decode_tokens_per_second": 54.11240876508312,
+      "e2e_seconds": 2.4971941119874828,
+      "e2e_tokens_per_second": 51.257529154642505,
+      "index": 78,
+      "prompt_characters": 17
+    },
+    {
+      "prompt_tokens": 163,
+      "completion_tokens": 128,
+      "ttft_ms": 188.56855999911204,
+      "decode_seconds": 3.0211055999970995,
+      "decode_tokens_per_second": 42.037590476851236,
+      "e2e_seconds": 3.2096741599962115,
+      "e2e_tokens_per_second": 39.87943748163866,
+      "index": 79,
+      "prompt_characters": 193
+    },
+    {
+      "prompt_tokens": 154,
+      "completion_tokens": 128,
+      "ttft_ms": 154.23041599569842,
+      "decode_seconds": 2.458247360016685,
+      "decode_tokens_per_second": 51.66282371158044,
+      "e2e_seconds": 2.6124777760123834,
+      "e2e_tokens_per_second": 48.99563210653443,
+      "index": 80,
+      "prompt_characters": 151
+    },
+    {
+      "prompt_tokens": 119,
+      "completion_tokens": 128,
+      "ttft_ms": 183.22851200355217,
+      "decode_seconds": 2.7403013600269333,
+      "decode_tokens_per_second": 46.34526766017872,
+      "e2e_seconds": 2.9235298720304854,
+      "e2e_tokens_per_second": 43.782689284135785,
+      "index": 81,
+      "prompt_characters": 75
+    },
+    {
+      "prompt_tokens": 274,
+      "completion_tokens": 128,
+      "ttft_ms": 190.3801920125261,
+      "decode_seconds": 2.998549023992382,
+      "decode_tokens_per_second": 42.35381812464329,
+      "e2e_seconds": 3.188929216004908,
+      "e2e_tokens_per_second": 40.138865220833736,
+      "index": 82,
+      "prompt_characters": 413
+    },
+    {
+      "prompt_tokens": 96,
+      "completion_tokens": 128,
+      "ttft_ms": 149.46897595655173,
+      "decode_seconds": 2.839366831991356,
+      "decode_tokens_per_second": 44.72828187224054,
+      "e2e_seconds": 2.9888358079479076,
+      "e2e_tokens_per_second": 42.82603937614191,
+      "index": 83,
+      "prompt_characters": 53
+    },
+    {
+      "prompt_tokens": 187,
+      "completion_tokens": 128,
+      "ttft_ms": 192.14686402119696,
+      "decode_seconds": 3.0270581119693816,
+      "decode_tokens_per_second": 41.95492630214976,
+      "e2e_seconds": 3.2192049759905785,
+      "e2e_tokens_per_second": 39.761369951477924,
+      "index": 84,
+      "prompt_characters": 243
+    },
+    {
+      "prompt_tokens": 175,
+      "completion_tokens": 128,
+      "ttft_ms": 154.42934399470687,
+      "decode_seconds": 2.2495230400236323,
+      "decode_tokens_per_second": 56.45641219956823,
+      "e2e_seconds": 2.403952384018339,
+      "e2e_tokens_per_second": 53.24564698159326,
+      "index": 85,
+      "prompt_characters": 193
+    },
+    {
+      "prompt_tokens": 95,
+      "completion_tokens": 128,
+      "ttft_ms": 183.63060802221298,
+      "decode_seconds": 2.5285595359746367,
+      "decode_tokens_per_second": 50.226224928909055,
+      "e2e_seconds": 2.7121901439968497,
+      "e2e_tokens_per_second": 47.19433122464318,
+      "index": 86,
+      "prompt_characters": 54
+    },
+    {
+      "prompt_tokens": 135,
+      "completion_tokens": 128,
+      "ttft_ms": 153.55235198512673,
+      "decode_seconds": 2.1443000319995917,
+      "decode_tokens_per_second": 59.226786412706716,
+      "e2e_seconds": 2.2978523839847185,
+      "e2e_tokens_per_second": 55.70418748050059,
+      "index": 87,
+      "prompt_characters": 153
+    },
+    {
+      "prompt_tokens": 132,
+      "completion_tokens": 128,
+      "ttft_ms": 184.29516803007573,
+      "decode_seconds": 2.5293376480112784,
+      "decode_tokens_per_second": 50.21077359911013,
+      "e2e_seconds": 2.713632816041354,
+      "e2e_tokens_per_second": 47.16924089484086,
+      "index": 88,
+      "prompt_characters": 83
+    },
+    {
+      "prompt_tokens": 151,
+      "completion_tokens": 128,
+      "ttft_ms": 182.26567999226972,
+      "decode_seconds": 2.9498485919903032,
+      "decode_tokens_per_second": 43.05305714498091,
+      "e2e_seconds": 3.132114271982573,
+      "e2e_tokens_per_second": 40.86696361783067,
+      "index": 89,
+      "prompt_characters": 154
+    },
+    {
+      "prompt_tokens": 98,
+      "completion_tokens": 128,
+      "ttft_ms": 180.34731200896204,
+      "decode_seconds": 2.209185392013751,
+      "decode_tokens_per_second": 57.48725320161337,
+      "e2e_seconds": 2.389532704022713,
+      "e2e_tokens_per_second": 53.56695883865306,
+      "index": 90,
+      "prompt_characters": 47
+    },
+    {
+      "prompt_tokens": 120,
+      "completion_tokens": 128,
+      "ttft_ms": 184.92633604910225,
+      "decode_seconds": 2.5972612960031256,
+      "decode_tokens_per_second": 48.89766008350327,
+      "e2e_seconds": 2.782187632052228,
+      "e2e_tokens_per_second": 46.00696176108843,
+      "index": 91,
+      "prompt_characters": 91
+    },
+    {
+      "prompt_tokens": 129,
+      "completion_tokens": 128,
+      "ttft_ms": 154.7068800427951,
+      "decode_seconds": 3.1600337119889446,
+      "decode_tokens_per_second": 40.1894446626221,
+      "e2e_seconds": 3.3147405920317397,
+      "e2e_tokens_per_second": 38.61538978576407,
+      "index": 92,
+      "prompt_characters": 149
+    },
+    {
+      "prompt_tokens": 126,
+      "completion_tokens": 128,
+      "ttft_ms": 154.17582396185026,
+      "decode_seconds": 2.7378541760263033,
+      "decode_tokens_per_second": 46.38669258284846,
+      "e2e_seconds": 2.8920299999881536,
+      "e2e_tokens_per_second": 44.259568538543625,
+      "index": 93,
+      "prompt_characters": 129
+    },
+    {
+      "prompt_tokens": 176,
+      "completion_tokens": 128,
+      "ttft_ms": 184.01340796845034,
+      "decode_seconds": 2.775731760018971,
+      "decode_tokens_per_second": 45.75370063825332,
+      "e2e_seconds": 2.959745167987421,
+      "e2e_tokens_per_second": 43.246966456587856,
+      "index": 94,
+      "prompt_characters": 181
+    },
+    {
+      "prompt_tokens": 151,
+      "completion_tokens": 128,
+      "ttft_ms": 184.33799996273592,
+      "decode_seconds": 2.530220592045225,
+      "decode_tokens_per_second": 50.19325208216075,
+      "e2e_seconds": 2.714558592007961,
+      "e2e_tokens_per_second": 47.15315424645828,
+      "index": 95,
+      "prompt_characters": 182
+    },
+    {
+      "prompt_tokens": 113,
+      "completion_tokens": 128,
+      "ttft_ms": 154.31780798826367,
+      "decode_seconds": 2.4937886080006137,
+      "decode_tokens_per_second": 50.92652985604173,
+      "e2e_seconds": 2.6481064159888774,
+      "e2e_tokens_per_second": 48.336426069267766,
+      "index": 96,
+      "prompt_characters": 72
+    },
+    {
+      "prompt_tokens": 159,
+      "completion_tokens": 128,
+      "ttft_ms": 154.63102399371564,
+      "decode_seconds": 2.0045958719565533,
+      "decode_tokens_per_second": 63.35441560898941,
+      "e2e_seconds": 2.159226895950269,
+      "e2e_tokens_per_second": 59.28047684107214,
+      "index": 97,
+      "prompt_characters": 144
+    },
+    {
+      "prompt_tokens": 135,
+      "completion_tokens": 128,
+      "ttft_ms": 154.29542399942875,
+      "decode_seconds": 3.126877536007669,
+      "decode_tokens_per_second": 40.615597680921944,
+      "e2e_seconds": 3.2811729600070976,
+      "e2e_tokens_per_second": 39.01043972998093,
+      "index": 98,
+      "prompt_characters": 146
+    },
+    {
+      "prompt_tokens": 183,
+      "completion_tokens": 128,
+      "ttft_ms": 160.44097597477958,
+      "decode_seconds": 2.536386896041222,
+      "decode_tokens_per_second": 50.07122541053215,
+      "e2e_seconds": 2.6968278720160015,
+      "e2e_tokens_per_second": 47.46317009261484,
+      "index": 99,
+      "prompt_characters": 214
+    }
+  ]
+}

benchmark/elyza100-baseline-vs-dflash-summary.json

@@ -0,0 +1,15 @@
+{
+  "count": 100,
+  "baseline_mean_tps": 29.72816958251134,
+  "speculative_mean_tps": 48.76273364465015,
+  "ratio_of_mean_tps": 1.640287119235776,
+  "paired_speedup_mean": 1.6404260527171066,
+  "paired_speedup_median": 1.6296921802007382,
+  "paired_speedup_p05": 1.3683736759954253,
+  "paired_speedup_p95": 1.9736926728405424,
+  "paired_speedup_min": 1.2828180903828805,
+  "paired_speedup_max": 2.3186871408906584,
+  "requests_at_least_1_5x": 70,
+  "requests_at_least_2x": 5,
+  "requests_slower_than_baseline": 0
+}

config.json

@@ -0,0 +1,71 @@
+{
+  "architectures": [
+    "DFlashDraftModel"
+  ],
+  "auto_map": {
+    "": "config.DFlashSpeculatorConfig"
+  },
+  "aux_hidden_state_layer_ids": [
+    2,
+    16,
+    29
+  ],
+  "block_size": 4,
+  "draft_vocab_size": 28000,
+  "dtype": "bfloat16",
+  "mask_token_id": 3,
+  "max_anchors": 256,
+  "sliding_window_non_causal": false,
+  "speculators_config": {
+    "algorithm": "dflash",
+    "default_proposal_method": "greedy",
+    "proposal_methods": [
+      {
+        "accept_tolerance": 0.0,
+        "proposal_type": "greedy",
+        "speculative_tokens": 3,
+        "verifier_accept_k": 1
+      }
+    ],
+    "verifier": {
+      "architectures": [],
+      "name_or_path": "kel-jp/llm-jp-4-8b-instruct-NVFP4"
+    }
+  },
+  "speculators_model_type": "dflash",
+  "speculators_version": "0.6.0",
+  "target_hidden_size": null,
+  "tie_word_embeddings": false,
+  "transformer_layer_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bos_token_id": 1,
+    "eos_token_id": 2,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 4096,
+    "initializer_range": 0.02,
+    "intermediate_size": 14336,
+    "layer_types": [
+      "full_attention"
+    ],
+    "max_position_embeddings": 65536,
+    "mlp_bias": false,
+    "model_type": "llama",
+    "num_attention_heads": 32,
+    "num_hidden_layers": 1,
+    "num_key_value_heads": 8,
+    "pad_token_id": null,
+    "pretraining_tp": 1,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "rope_theta": 500000,
+      "rope_type": "default"
+    },
+    "sliding_window": 2048,
+    "tie_word_embeddings": false,
+    "use_cache": true,
+    "vocab_size": 196608
+  },
+  "transformers_version": "5.10.4"
+}

config.py

@@ -0,0 +1,102 @@
+from typing import Any, Literal
+
+from pydantic import Field, field_serializer, field_validator
+from transformers import AutoConfig, PretrainedConfig
+from transformers.models.qwen3.modeling_qwen3 import (
+    Qwen3Config,
+)
+
+from speculators import SpeculatorModelConfig
+
+__all__ = [
+    "DFlashSpeculatorConfig",
+]
+
+
+@SpeculatorModelConfig.register("dflash")
+class DFlashSpeculatorConfig(SpeculatorModelConfig):
+    """
+    Configuration for DFlash speculator with vocabulary mapping.
+
+    DFlash features vocabulary mapping between draft (64K) and target (128K)
+    vocabularies, enabling cross-tokenizer speculation.
+
+    :param transformer_layer_config: Configuration for the transformer decoder layer
+    :param draft_vocab_size: Size of draft model vocabulary for speculation
+    """
+
+    speculators_model_type: Literal["dflash"] = "dflash"
+    architectures: list[str] = Field(
+        default_factory=lambda: ["DFlashSpeculator"],
+        description="Model architectures that can load these weights",
+    )
+
+    transformer_layer_config: PretrainedConfig = Field(
+        default_factory=Qwen3Config,
+        description="Configuration for the transformer decoder layer",
+    )
+
+    draft_vocab_size: int = Field(
+        default=32000,
+        description="Size of draft model vocabulary for speculation",
+    )
+
+    block_size: int = Field(
+        default=8,
+        description=(
+            "Default size of the draft block predicted with a forward pass of the model"
+        ),
+    )
+
+    max_anchors: int = Field(
+        default=256,
+        description=(
+            "Maximum number of anchor positions to sample during training "
+            "(controls memory usage and training efficiency)"
+        ),
+    )
+
+    target_hidden_size: int | None = Field(
+        default=None,
+        description="Hidden size of the target model (if different from draft model)",
+    )
+
+    aux_hidden_state_layer_ids: list[int] | None = Field(
+        default=None,
+        description="Layer IDs of the DFlash auxiliary hidden state layers",
+    )
+
+    mask_token_id: int | None = Field(
+        default=None,
+        description="Token ID used for masking",
+    )
+
+    sliding_window_non_causal: bool = Field(
+        default=False,
+        description="Use non-causal (bidirectional) masking within draft blocks for "
+        "sliding window attention layers. Full attention layers are always "
+        "bidirectional.",
+    )
+
+    @field_serializer("transformer_layer_config")
+    def serialize_transformer_config(self, value: PretrainedConfig) -> dict:
+        """Serialize transformer config to dict."""
+        return value.to_diff_dict()
+
+    @field_validator("transformer_layer_config", mode="before")
+    @classmethod
+    def validate_transformer_config(cls, value: Any) -> PretrainedConfig:
+        """Validate and convert transformer config."""
+        if isinstance(value, dict):
+            config_class: type[PretrainedConfig] = Qwen3Config
+            if "model_type" in value:
+                config_class = AutoConfig.for_model(
+                    model_type=value["model_type"]
+                ).__class__
+            return config_class(**value)
+        return value
+
+    @property
+    def target_vocab_size(self) -> int:
+        """Get target vocabulary size from transformer config."""
+        return self.transformer_layer_config.vocab_size

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7ffbde19080c85333332bd25f214de07274e172e01fa140aaee806d5c635efc
+size 2377315376

val_metrics.json

@@ -0,0 +1 @@
+{"loss_epoch": 2.047496609403255, "full_acc_epoch": 0.4021220020978507, "position_1_acc_epoch": 0.5539027137301571, "position_2_acc_epoch": 0.38056573750461736, "position_3_acc_epoch": 0.27111557473966125}