---
library_name: speculators
base_model:
- Qwen/Qwen3-4B
license: apache-2.0
tags:
- speculative-decoding
- dflash2
- speculators
---

# RedHatAI/Qwen3-4B-speculator.dflash2

This is a DFlash2 speculator model for [Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B).

## Training Details

This model was trained using the [Speculators](https://github.com/vllm-project/speculators) library on [inference-optimization/Qwen3-8B-Regenerated-Collection](https://huggingface.co/datasets/inference-optimization/Qwen3-8B-Regenerated-Collection). DFlash2 extends DFlash with local dynamic convolutions and a candidate selector for improved token prediction.
<details>
  <summary>Commands</summary>

  Using the [Speculators](https://github.com/vllm-project/speculators) library and the helper scripts provided in the repo.

  ### Prepare data
  ```bash
  # In virtual environment with speculators installed
  python scripts/prepare_data.py \
    --model Qwen/Qwen3-4B \
    --data ./data/qwen3_8b_regen/autoif_train_Qwen3-8B.jsonl \
    --data ./data/qwen3_8b_regen/evol_codealpaca_train_Qwen3-8B.jsonl \
    --data ./data/qwen3_8b_regen/lmsys_arena_train_Qwen3-8B.jsonl \
    --data ./data/qwen3_8b_regen/magpie_output.jsonl \
    --data ./data/qwen3_8b_regen/metamathqa_train_Qwen3-8B.jsonl \
    --data ./data/qwen3_8b_regen/nemotron_chat_Qwen3-8B.jsonl \
    --data ./data/qwen3_8b_regen/nemotron_math_Qwen3-8B.jsonl \
    --data ./data/qwen3_8b_regen/nemotron_stem_Qwen3-8B.jsonl \
    --data ./data/qwen3_8b_regen/orca_math_train_Qwen3-8B.jsonl \
    --data ./data/qwen3_8b_regen/tulu3_Qwen3-8B.jsonl \
    --data ./data/qwen3_8b_regen/ultrachat_output.jsonl \
    --data ./data/qwen3_8b_regen/ultrafeedback_train_sft_Qwen3-8B.jsonl \
    --data ./data/qwen3_8b_regen/ultrainteract_train_Qwen3-8B.jsonl \
    --output ./output/dflash2_qwen3_4b_8spec \
    --seq-length 8192
  ```

  ### Launch vLLM
  ```bash
  # In (separate) virtual environment with vllm installed
  CUDA_VISIBLE_DEVICES=0 python scripts/launch_vllm.py \
    Qwen/Qwen3-4B \
    --target-layer-ids 1 9 17 25 33 \
    -- --port 8000 \
    --max-model-len 8194 \
    --enforce-eager
  ```

  ### Launch training
  Must be run once vLLM has finished launching and is running in the background.
  ```bash
  # In virtual environment with speculators installed
  CUDA_VISIBLE_DEVICES=1,2,3,4 torchrun \
    --standalone \
    --nproc_per_node 4 \
    scripts/train.py \
    --verifier-name-or-path Qwen/Qwen3-4B \
    --speculator-type dflash2 \
    --data-path ./output/dflash2_qwen3_4b_8spec \
    --vllm-endpoint http://localhost:8000/v1 \
    --save-path ./output/dflash2_qwen3_4b_8spec/checkpoints \
    --block-size 9 \
    --max-anchors 512 \
    --target-layer-ids 1 9 17 25 33 \
    --num-layers 5 \
    --conv-kernel-size 2 \
    --conv-group-size 16 \
    --selector-rank 256 \
    --selector-top-k 16 \
    --loss-fn '{"ce": 0.1, "tv": 0.9}' \
    --dflash-decay-gamma 4.0 \
    --per-position-loss-weight fixed-exp-decay \
    --optimizer adamw \
    --lr 6e-4 \
    --weight-decay 0.0 \
    --scheduler-type cosine \
    --scheduler-warmup-ratio 0.04 \
    --epochs 1 \
    --total-seq-len 8192 \
    --seed 42 \
    --fsdp-shard \
    --on-missing generate \
    --on-generate delete \
    --checkpoint-freq 0.1
  ```

</details>


## Model Specifications

| | |
|---|---|
| **Base Model** | Qwen/Qwen3-4B |
| **Chat Template** | Qwen/Qwen3-4B (use `/chat/completions` endpoint) |
| **Format** | Safetensors |
| **License** | Apache 2.0 |
| **Validation Hardware** | Nvidia H100 |

## Deployment

  ```bash
  pip install -U "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/52816/head"

  # Deploy with speculative decoding
  vllm serve Qwen/Qwen3-4B \
      --speculative-config '{
          "model": "RedHatAI/Qwen3-4B-speculator.dflash2",
          "num_speculative_tokens": 8,
          "method": "dflash2"
      }'
  ```

## Acceptance Rates

Per-position token acceptance rates across datasets:

| Dataset | Pos 0 | Pos 1 | Pos 2 | Pos 3 | Pos 4 | Pos 5 | Pos 6 | Avg. Length |
|---------|-------|-------|-------|-------|-------|-------|-------|-------------|
| HumanEval | 89.3% | 77.9% | 67.0% | 56.9% | 48.2% | 40.6% | 33.9% | 5.14 |
| math_reasoning | 92.5% | 84.4% | 76.2% | 68.4% | 61.0% | 54.3% | 47.8% | 5.85 |
| qa | 80.1% | 63.7% | 50.7% | 41.4% | 33.9% | 28.1% | 23.5% | 4.21 |
| question | 81.7% | 64.8% | 51.7% | 41.8% | 34.0% | 27.9% | 23.3% | 4.25 |
| rag | 82.3% | 65.8% | 52.6% | 42.0% | 33.6% | 27.1% | 22.1% | 4.25 |
| summarization | 74.5% | 52.2% | 35.5% | 23.3% | 15.7% | 10.3% | 6.6% | 3.18 |
| tool_call | 81.0% | 63.6% | 49.9% | 38.7% | 30.5% | 24.1% | 19.5% | 4.07 |
| translation | 84.6% | 69.6% | 56.5% | 44.9% | 35.4% | 28.3% | 22.6% | 4.42 |
| writing | 81.6% | 64.8% | 51.6% | 41.6% | 33.9% | 27.8% | 23.2% | 4.24 |