---
license: apache-2.0
base_model: Qwen/Qwen3-8B
tags:
  - speculative-decoding
  - draft-model
  - dflash
  - domino
  - sglang
  - specforge
---

# Qwen3-8B Domino speculative-decoding draft (10-epoch)

A **Domino** draft model for `Qwen/Qwen3-8B`, trained with the SpecForge
**DataFlow online-disaggregated** pipeline (frozen Qwen3-8B target + live
spec-capture through sglang/Mooncake -> DP domino trainer).

## This checkpoint
- `training_state.pt` — torch checkpoint; the trained draft weights are under
  the key **`draft_state_dict`** (~1.1B params, bf16), plus training metadata
  (`global_step`, `epoch`, `strategy`, `run_id`, ...).
- `config.json` — the draft config (`DFlashDraftModel`, `projector_type=domino`,
  5 layers, hidden 4096, `target_layer_ids=[1,9,17,25,33]`, `block_size=16`,
  vocab 151936, `mask_token_id=151669`).
- **Step 44,000** — the latest periodic checkpoint (save-interval 4000) of a
  completed 10-epoch run (~epoch 9.3).

## Training
- Data: `perfectblend` (47,160 filtered prompts), 10 epochs, batch 2, DP=5,
  LR 6e-4, warmup 0.04, grad-clip 1.0, num_anchors 256, loss-decay-gamma 7.0.
- Target `Qwen/Qwen3-8B` frozen (no gradients); only the draft is trained.
- Final training-time metrics: loss ~0.1, token-match acc ~0.95.

> Note: the training-time `acc` is a proxy. Real serving accept-length (τ)
> requires exporting to an sglang-loadable draft and running the spec-decode
> benchmark.

## Load the draft weights
```python
import torch
sd = torch.load("training_state.pt", map_location="cpu", weights_only=False)
draft_weights = sd["draft_state_dict"]  # load into a DFlashDraftModel(config)
```