Qwen3-0.6B-Domino-b8

Draft model for Domino speculative decoding, trained on Qwen3-0.6B as the verifier with block size 8.

Model Description

Domino decouples causal dependency modeling from expensive autoregressive draft execution. A parallel DFlash backbone produces preliminary draft distributions, then a lightweight GRU correction head refines them with prefix-dependent causal information. The head is a low-rank residual update in draft vocabulary space (paper §4.1.2).

Parameter	Value
Verifier	`Qwen/Qwen3-0.6B`
Verifier layers	28, hidden=1024
Draft layers	3, hidden=1024, 16 attn heads, 8 KV heads
Auxiliary layers	[1, 13, 25]
Block size	8
Draft vocab size	32,000
Target vocab size	151,936
Projector	`domino`
GRU hidden dim	1024
Bottleneck dim	256
Shift label	True
Pure draft prefix len	0
Mask token ID	151669

Inference Benchmark

Evaluated on 164 HumanEval prompts, greedy T=0, max 256 new tokens, RTX 4060 8 GB, verifier Qwen/Qwen3-0.6B (bf16), patched vLLM inference backend.

Three models with identical training budget:

Checkpoint	Projector	Shift	First trained prediction
DFlash shift	dflash	True	None (pos0 masked)
DFlash noshift	dflash	False	pos1 (a+1, 21.5%)
Domino	domino	True	pos0 (a+1, 41.0%)

Aggregate

Model	Accept len	TPS	vs DFlash noshift
DFlash shift	1.070	47.3	—
DFlash noshift	1.079	45.1	baseline
Domino	1.378	53.1	+27.7% accept, +17.8% TPS

Per-position unconditional acceptance

Model	pos_0	pos_1	pos_2	pos_3
DFlash shift	0.065	0.004	0.001	0.000
DFlash noshift	0.076	0.003	0.000	0
Domino	0.319	0.049	0.008	0.002

Fair comparison: Domino pos_0 vs DFlash noshift pos_1 — both predict token a+1 with full training signal. Domino's base-anchored curriculum produces a 9.7× stronger backbone (31.9% vs 3.3%) at inference.

Domino conditional per-position

Position	0	1	2	3	4	5	6	7
Conditional acceptance	31.9%	15.2%	16.5%	23.4%	29.8%	41.2%	71.4%	80.0%

GRU correction compounds: once the prefix chain survives, the growing causal context drives corrections to 80% at position 7.

Training

Parameter	Value
Dataset	ShareGPT, 5,000 samples
Sequence length	4,096
Epochs	2
LR	6e-4, linear schedule, 50 warmup steps
Base-anchored curriculum	lambda_base 1.0 → 0, decay ratio 0.5
Loss	Position-weighted cross-entropy (gamma=7)
Teacher forcing	Ground-truth embeddings for GRU prefix
Hardware	Single RTX 4060 Laptop, 8 GB VRAM