File size: 1,649 Bytes
fd7f390 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | ---
license: apache-2.0
base_model: Qwen/Qwen3-8B
tags:
- speculative-decoding
- draft-model
- dflash
- domino
- sglang
- specforge
---
# Qwen3-8B Domino speculative-decoding draft (10-epoch)
A **Domino** draft model for `Qwen/Qwen3-8B`, trained with the SpecForge
**DataFlow online-disaggregated** pipeline (frozen Qwen3-8B target + live
spec-capture through sglang/Mooncake -> DP domino trainer).
## This checkpoint
- `training_state.pt` — torch checkpoint; the trained draft weights are under
the key **`draft_state_dict`** (~1.1B params, bf16), plus training metadata
(`global_step`, `epoch`, `strategy`, `run_id`, ...).
- `config.json` — the draft config (`DFlashDraftModel`, `projector_type=domino`,
5 layers, hidden 4096, `target_layer_ids=[1,9,17,25,33]`, `block_size=16`,
vocab 151936, `mask_token_id=151669`).
- **Step 44,000** — the latest periodic checkpoint (save-interval 4000) of a
completed 10-epoch run (~epoch 9.3).
## Training
- Data: `perfectblend` (47,160 filtered prompts), 10 epochs, batch 2, DP=5,
LR 6e-4, warmup 0.04, grad-clip 1.0, num_anchors 256, loss-decay-gamma 7.0.
- Target `Qwen/Qwen3-8B` frozen (no gradients); only the draft is trained.
- Final training-time metrics: loss ~0.1, token-match acc ~0.95.
> Note: the training-time `acc` is a proxy. Real serving accept-length (τ)
> requires exporting to an sglang-loadable draft and running the spec-decode
> benchmark.
## Load the draft weights
```python
import torch
sd = torch.load("training_state.pt", map_location="cpu", weights_only=False)
draft_weights = sd["draft_state_dict"] # load into a DFlashDraftModel(config)
```
|