---
language: en
license: apache-2.0
tags:
  - speculative-decoding
  - domino
  - dflash
  - qwen3
  - draft-model
pipeline_tag: text-generation
base_model: Qwen/Qwen3-0.6B
---

# Qwen3-0.6B-Domino-b8

Draft model for [Domino](https://arxiv.org/abs/2605.29707) speculative decoding,
trained on Qwen3-0.6B as the verifier with block size 8.

## Model Description

Domino decouples causal dependency modeling from expensive autoregressive draft
execution. A parallel DFlash backbone produces preliminary draft distributions,
then a lightweight GRU correction head refines them with prefix-dependent causal
information. The head is a low-rank residual update in draft vocabulary space
(paper §4.1.2).

| Parameter | Value |
|-----------|-------|
| Verifier | `Qwen/Qwen3-0.6B` |
| Verifier layers | 28, hidden=1024 |
| Draft layers | 3, hidden=1024, 16 attn heads, 8 KV heads |
| Auxiliary layers | [1, 13, 25] |
| Block size | 8 |
| Draft vocab size | 32,000 |
| Target vocab size | 151,936 |
| Projector | `domino` |
| GRU hidden dim | 1024 |
| Bottleneck dim | 256 |
| Shift label | True |
| Pure draft prefix len | 0 |
| Mask token ID | 151669 |

## Inference Benchmark

Evaluated on [164 HumanEval prompts](https://huggingface.co/datasets/openai/openai_humaneval),
greedy T=0, max 256 new tokens, RTX 4060 8 GB, verifier `Qwen/Qwen3-0.6B` (bf16),
patched vLLM inference backend.

Three models with identical training budget:

| Checkpoint | Projector | Shift | First trained prediction |
|-----------|-----------|-------|-------------------------|
| [DFlash shift](https://huggingface.co/Eros483/Qwen3-0.6B-DFlash-shift-b8) | dflash | True | None (pos0 masked) |
| [DFlash noshift](https://huggingface.co/Eros483/Qwen3-0.6B-DFlash-noshift-b8) | dflash | False | pos1 (a+1, 21.5%) |
| [Domino](https://huggingface.co/Eros483/Qwen3-0.6B-Domino-b8) | domino | True | pos0 (a+1, 41.0%) |

### Aggregate

| Model | Accept len | TPS | vs DFlash noshift |
|-------|-----------|-----|-------------------|
| DFlash shift | 1.070 | 47.3 | — |
| DFlash noshift | 1.079 | 45.1 | baseline |
| **Domino** | **1.378** | **53.1** | **+27.7% accept, +17.8% TPS** |

### Per-position unconditional acceptance

| Model | pos_0 | pos_1 | pos_2 | pos_3 | pos_4+ |
|-------|-------|-------|-------|-------|--------|
| DFlash shift | 0.065 | 0.004 | 0.001 | 0.000 | 0 |
| DFlash noshift | 0.076 | 0.003 | 0.000 | 0 | 0 |
| **Domino** | **0.319** | **0.049** | **0.008** | **0.002** | 0 |

Fair comparison: Domino pos_0 vs DFlash noshift pos_1 — both predict token a+1 with full
training signal. Domino's base-anchored curriculum produces a 9.7× stronger backbone
(31.9% vs 3.3%) at inference.

### Domino conditional per-position

| Position | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|----------|---|---|---|---|---|---|---|---|
| Conditional acceptance | 31.9% | 15.2% | 16.5% | 23.4% | 29.8% | 41.2% | 71.4% | 80.0% |

GRU correction compounds: once the prefix chain survives, the growing causal context
drives corrections to 80% at position 7.

## Training

| Parameter | Value |
|-----------|-------|
| Dataset | ShareGPT, 5,000 samples |
| Sequence length | 4,096 |
| Epochs | 2 |
| LR | 6e-4, linear schedule, 50 warmup steps |
| Base-anchored curriculum | lambda_base 1.0 → 0, decay ratio 0.5 |
| Loss | Position-weighted cross-entropy (gamma=7) |
| Teacher forcing | Ground-truth embeddings for GRU prefix |
| Hardware | Single RTX 4060 Laptop, 8 GB VRAM |

Training was done with [vllm-project/speculators](https://github.com/vllm-project/speculators) (commit `cd63eca`),
using vLLM 0.24.0 pre-generated hidden states for the auxiliary layers.

Training validation (ShareGPT held-out, epoch 2, greedy accuracy):

| Position | Base acc | Final acc |
|----------|----------|-----------|
| 0 | 41.0% | 41.0% |
| 1 | 31.8% | 31.8% |
| 2 | 27.8% | 27.8% |
| 3 | 25.7% | 25.7% |
| 4 | 24.3% | 24.3% |
| 5 | 23.1% | 23.1% |
| 6 | 22.4% | 22.4% |
| 7 | 21.5% | 21.5% |
| **Overall** | 12.1% | 27.2% |

## Related Repos

- [Domino paper & reference implementation](https://github.com/jianuo-huang/Domino)
- [SGLang SpecForge training code](https://github.com/sgl-project/SpecForge)
- [Speculators training framework](https://github.com/vllm-project/speculators)
- [Companion DFlash (shift) checkpoint](https://huggingface.co/Eros483/Qwen3-0.6B-DFlash-shift-b8)
- [Companion DFlash (noshift) checkpoint](https://huggingface.co/Eros483/Qwen3-0.6B-DFlash-noshift-b8)
- [vLLM/Speculators Training Integration](https://github.com/vllm-project/speculators/pull/685)
- [vLLM Inference Integration](https://github.com/vllm-project/vllm/pull/47627)