| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3-8B |
| tags: |
| - speculative-decoding |
| - draft-model |
| - dflash |
| - domino |
| - sglang |
| - specforge |
| --- |
| |
| # Qwen3-8B Domino speculative-decoding draft (10-epoch) |
|
|
| A **Domino** draft model for `Qwen/Qwen3-8B`, trained with the SpecForge |
| **DataFlow online-disaggregated** pipeline (frozen Qwen3-8B target + live |
| spec-capture through sglang/Mooncake -> DP domino trainer). |
|
|
| ## This checkpoint |
| - `training_state.pt` — torch checkpoint; the trained draft weights are under |
| the key **`draft_state_dict`** (~1.1B params, bf16), plus training metadata |
| (`global_step`, `epoch`, `strategy`, `run_id`, ...). |
| - `config.json` — the draft config (`DFlashDraftModel`, `projector_type=domino`, |
| 5 layers, hidden 4096, `target_layer_ids=[1,9,17,25,33]`, `block_size=16`, |
| vocab 151936, `mask_token_id=151669`). |
| - **Step 44,000** — the latest periodic checkpoint (save-interval 4000) of a |
| completed 10-epoch run (~epoch 9.3). |
|
|
| ## Training |
| - Data: `perfectblend` (47,160 filtered prompts), 10 epochs, batch 2, DP=5, |
| LR 6e-4, warmup 0.04, grad-clip 1.0, num_anchors 256, loss-decay-gamma 7.0. |
| - Target `Qwen/Qwen3-8B` frozen (no gradients); only the draft is trained. |
| - Final training-time metrics: loss ~0.1, token-match acc ~0.95. |
| |
| > Note: the training-time `acc` is a proxy. Real serving accept-length (τ) |
| > requires exporting to an sglang-loadable draft and running the spec-decode |
| > benchmark. |
| |
| ## Load the draft weights |
| ```python |
| import torch |
| sd = torch.load("training_state.pt", map_location="cpu", weights_only=False) |
| draft_weights = sd["draft_state_dict"] # load into a DFlashDraftModel(config) |
| ``` |
| |
