Koopah/Qwen3.6-35B-A3B-NVFP4-DSPARK

Qwen3.6-35B-A3B-NVFP4 · DSPARK draft model

Overview

A DSPARK speculative-decoding draft for unsloth/Qwen3.6-35B-A3B-NVFP4 (the Unsloth NVFP4 quantization of Qwen3.6-35B-A3B, a hybrid attention/gated-DeltaNet MoE). The draft consumes hidden states captured from that exact checkpoint, so it is matched to the NVFP4 weights as served — not to the bf16 original.

DSPARK drafts a whole block of tokens in a single forward pass and the target verifies them with the lossless ratio-acceptance rule (min(1, q/p)) — output quality is identical to running the target model alone, at any sampling temperature.

Supported targets: measured on both NVFP4 and NVFP4-Fast (acceptance actually improves on Fast — see the -Fast section). fp8 and bf16 variants of Qwen3.6-35B-A3B are untested but likely compatible: the draft reads the same hidden-state interface, and the Fast result shows cleaner activations help rather than hurt.

🚧 Training is still in progress. This is the best-scoring checkpoint from an ongoing multi-week run, published early. Acceptance is still climbing — expect updated weights.

Performance at a glance (single GPU)

Benchmark protocol matched to the sglang DSpark release post: one batch of B simultaneous requests, the post's fixed prompt (16 concatenated GSM8K questions), output forced to exactly 1024 tokens, temperature 0.7, prompt caching disabled, every point averaged over 3 rounds. Per-user decode speed is the median per-request rate from first token to completion; aggregate = per-user × B (steady state). One RTX PRO 6000 Blackwell 96 GB (Max-Q).

sglang (fork branch below)

* benchmarked without confidence-scheduled block truncation — no engine currently wires the draft's confidence head into inference (see known issues), so every round verifies the full drafted block. These numbers are a floor.

config	solo (bs1)	bs16	bs96
DSpark k=8	422 tok/s	202/user · 3,237 agg	56/user · 5,376 agg
DSpark k=6	387	189 · 3,021	58 · 5,606
DSpark k=4	337	173 · 2,765	59 · 5,702
MTP (steps 4, k=6)	290	158 · 2,531	57 · 5,453
MTP (steps 3, k=4)	273	161 · 2,579	57 · 5,472
no speculation	156	96 · 1,539	43 · 4,128

vLLM 0.26

* same disclaimer: measured without confidence-scheduled truncation — a floor, not a ceiling.

config	solo (bs1)	bs16	bs96
DSpark k=8	568 tok/s	259/user · 4,144 agg	71/user · 6,835 agg
DSpark k=6	511	246 · 3,934	63 · 6,067
DSpark k=4	427	217 · 3,475	63 · 6,058
MTP (k=4)	341	186 · 2,976	61 · 5,818
no speculation	185	115 · 1,838	50 · 4,790

Headlines: 3.1× solo / 1.4× at-capacity over plain decoding on vLLM (568 tok/s single-user, 6,835 tok/s aggregate), ahead of the built-in MTP head at every batch size on both engines. The k=4/6/8 family doubles as a static simulation of confidence scheduling: a wired confidence head would pick the verify budget per request per step, so the best-k envelope is a floor on what it would deliver (see known issues — no engine wires the head yet; all numbers verify blindly).

Continuous serving (closed loop)

The same GSM8K workload served as a continuous request stream (EOS-terminated generations, prompts sampled per request — closer to production traffic than the one-batch protocol above; sglang fork, radix cache on as in production):

config	solo (bs1)	bs16	bs96
DSpark k=8	443 tok/s	128/user · 2,046 agg	54/user · 5,205 agg
DSpark k=5	369	115 · 1,845	53 · 5,057
MTP (steps 4, k=6)	284	103 · 1,647	34 · 3,226
no speculation	152	70 · 1,114	28 · 2,681

Under sustained load with request churn, DSpark's margin over MTP is larger than in the one-batch numbers (+61% aggregate at capacity): continuous serving mixes prefill into every decode window, and DSpark's faster turnarounds keep queues shorter.

Evaluation detail

Serving-faithful rollout on held-out prompts never seen in training (summarization-heavy workload), sampling at temperature 1.0, top-p 0.95, top-k 20, ratio acceptance, no confidence threshold, bonus token included:

metric	value
Mean accepted length (8 draft positions)	4.24 (best eval) · ~4.1 typical
Per-position acceptance @0…@7	0.812 / 0.784 / 0.780 / 0.792 / 0.824 / 0.757 / 0.780 / 0.766
Eval progression across the run so far	3.36 → 3.75 → 4.06 → 4.24

Per-position acceptance is nearly flat across all 8 positions — deep draft positions hold up as well as shallow ones, which is what makes the long block pay off.

Spec-Bench (6 domains, 480 prompts)

Measured end-to-end on the live sglang server (fork branch below), ratio acceptance, 8 draft positions, this checkpoint. Two conditions: temp 0 (greedy target — the convention most published tables use) and temp 1.0 + top-p 0.95 + top-k 20 (the sampled production serving condition):

domain	temp 0	temp 1.0 (serving)
rag	5.43	4.96
math_reasoning	4.50	4.44
translation	3.89	3.50
mt_bench	3.69	3.35
summarization	3.28	3.12
qa	3.07	2.93
macro-avg	3.98	3.72

Both columns will keep moving up while training continues.

The -Fast variant

The draft also works on unsloth/Qwen3.6-35B-A3B-NVFP4-Fast — without retraining, acceptance transfers intact (mean acceptance length 7.4 vs 6.2 on the training target, per-position 0.968/0.896/0.841/0.809/…, while the built-in MTP head is unchanged between the two targets — so the gain is the Fast recipe handing the draft cleaner capture-layer activations, not an easier target overall). Same benchmark protocol as above, vLLM 0.26:

* same disclaimer: measured without confidence-scheduled truncation — a floor, not a ceiling.

config	solo (bs1)	bs16	bs96
DSpark k=8	395 tok/s	248/user · 3,960 agg	74/user · 7,104 agg
built-in MTP head	270	186 · 2,979	72 · 6,864
no speculation	212	124 · 1,978	54 · 5,203

Note: the -Fast variant currently serves via vLLM ≥ 0.26 only — the pinned sglang fork branch predates upstream support for the Fast recipe's quantization layout and fails to load it.

Deployment (sglang)

Use the pre-patched engine — upstream engines have several known issues with this exact target (detailed below):

• Patched fork branch: KoopahTManiac/sglang@dspark-hybrid-nvfp4 — last known-good upstream commit for NVFP4-on-Blackwell, plus four commits on top: the hybrid-GDN state-commit fix for DSPARK, a backport of upstream's DSPARK grammar-constrained decoding (#31753) — so JSON-schema / regex / EBNF output and tool_choice work through speculative decoding — and a backport of the NEXTN mamba-track guard (#27998), so the built-in MTP head also serves on this branch for A/B comparisons. Diff it yourself.
• A ready-to-build Dockerfile in this repo: a thin overlay on the official lmsysorg/sglang:nightly-dev-20260714-2f79d334 image (the last engine era before the SM120 NVFP4 regression, with era-matched kernels and the flashinfer JIT cache prebaked) that swaps in the fork branch's Python tree — all fork commits are pure Python. The default command carries the full tuned flag set and env; validated on an RTX PRO 6000 Blackwell: containerized serving matches native throughput within 1% (after first-minute JIT warmup).

# prebuilt image (or build it yourself from ./docker):
docker pull koopahtmaniac/sglang-dspark:latest   # pinned: :pin-0c01971ee
docker run -d --runtime nvidia --gpus all -p 30000:30000 koopahtmaniac/sglang-dspark
# or with local model/draft directories instead of HF downloads:
docker run -d --runtime nvidia --gpus all \
  -v /path/to/target-model:/models:ro -v /path/to/draft:/draft:ro \
  -e MODEL=/models -e DRAFT=/draft -p 30000:30000 koopahtmaniac/sglang-dspark

Full launch command as benchmarked (single ~96 GB Blackwell). The last four flags before the parsers are the high-concurrency tuning — --mem-fraction-static 0.74 pairs with the GDN state pool (--max-mamba-cache-size 540, admits ~96 concurrent) — drop those four for low-concurrency use:

python -m sglang.launch_server \
  --model-path unsloth/Qwen3.6-35B-A3B-NVFP4 \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path Koopah/Qwen3.6-35B-A3B-NVFP4-DSPARK \
  --tp 1 \
  --context-length 32768 \
  --kv-cache-dtype fp8_e4m3 \
  --linear-attn-decode-backend flashinfer \
  --mamba-ssm-dtype bfloat16 \
  --chunked-prefill-size 16384 \
  --enable-mixed-chunk \
  --num-continuous-decode-steps 2 \
  --enforce-shared-experts-fusion \
  --enable-fused-qk-norm-rope \
  --mem-fraction-static 0.74 \
  --max-mamba-cache-size 540 \
  --cuda-graph-max-bs-decode 96 \
  --max-running-requests 96 \
  --grammar-backend xgrammar \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --trust-remote-code

TTFT at these settings (open loop, ~1k-token prompts): p50 ≈ 105 ms from 2 up to 10 req/s, p99 ≤ 215 ms at 10 req/s.

Deployment (vLLM)

vLLM 0.26 is currently the fastest way to serve this draft (see the chart and table in Performance at a glance). Requirements and tuning:

• vLLM ≥ 0.26.0 — older versions fail to load this draft (KV page-size unification and draft-config handling both fixed in 0.26).
• --mamba-ssm-cache-dtype bfloat16 matters: without it the GDN recurrent states default to fp32 and DSpark's deep verify staging caps admission at ~47 concurrent requests (high-batch aggregate drops ~25%).
• Use the default KV-cache dtype (fp8 KV crashes on this hybrid target, see known issues).

vllm serve unsloth/Qwen3.6-35B-A3B-NVFP4 \
  --trust-remote-code \
  --served-model-name qwen3.6-35b-a3b \
  --max-num-seqs 96 --max-num-batched-tokens 16384 \
  --gpu-memory-utilization 0.9 \
  --mamba-ssm-cache-dtype bfloat16 \
  --speculative-config '{"method": "dspark", "model": "Koopah/Qwen3.6-35B-A3B-NVFP4-DSPARK", "num_speculative_tokens": 8}'

⚠️ Known issues in upstream engines (as of July 2026) — why the fork exists

1. Hybrid-GDN state commit (affects this target!): upstream sglang's DSPARK worker does not yet commit the target's gated-DeltaNet state after speculative verification. On hybrid-linear-attention targets like Qwen3.6-35B-A3B this silently collapses acceptance and corrupts outputs. The fix is already included in the fork branch and Docker image above (a port of DFlash's _update_target_mamba_state_after_verify into the DSPARK worker); an upstream PR will follow. Dense targets are unaffected.
2. NVFP4 on SM120 (consumer Blackwell): sglang's FP4 quantization refactor (9756f768a) routes nvfp4-pack-quantized compressed-tensors MoE onto a FlashInfer kernel path with no working SM12x support — garbage output at any temperature. Upstream work is in flight (sglang #32119, flashinfer #3932); until it lands, use the pinned fork branch above (0c01971ee base, the last known-good).
3. vLLM: fp8 KV cache crashes on this hybrid target — use the default KV dtype.
4. Confidence head unused by both engines: this draft ships a trained confidence head for dynamic block truncation (measured on-policy calibration: ECE 0.010, AUC 0.925), but vLLM explicitly skips its weights ("not wired into inference yet") and sglang's confidence-scheduling only supports MLA-attention targets. All benchmark numbers on this card therefore verify the full block every round — once either engine wires the head, expect the mid-batch throughput gap vs MTP to close and the peaks to rise.

Model Specifications

Draft architecture	Qwen3DSparkModel, 6 transformer layers, hidden 2048
Block size	8 (runtime knob — can be served at smaller blocks)
Target capture layers	[1, 6, 11, 16, 22, 27, 32, 37]
Heads	low-rank Markov head (rank 256) + confidence head
Parameters	≈1.5 B total (≈0.5 B trainable core; embeddings & LM head are frozen copies of the target's)
Vocabulary	target's 248,077 tokens + 1 mask token (id 248077), padded to 248,320
Precision	BF16

Training Details

• Data: ~1.34 M prompts (open-perfectblend mix) with responses regenerated by the target itself in both thinking and non-thinking modes, sampled at the target's serving configuration — the draft learns the target's actual output distribution rather than generic text.
• Stack: DeepSpec, adapted for online single-GPU training against the live NVFP4 target.
• Initialization: warm-started from the z-lab DFlash backbone for this target, converted to DSPARK format (Markov head soft-started, confidence head fresh), then trained from step one under the DSPARK objective.
• Evaluation during training: rollout-based accept-length eval on held-out prompts every 100 steps at the serving sampling configuration; this repo tracks the best-scoring checkpoint.

References

• DSpark: block-diffusion speculative decoding (DeepSeek, 2026) — merged in sglang and vLLM
• DeepSpec training framework
• z-lab DFlash (backbone initialization)