Card: measured CUDA + SPEED-Bench + Apple Silicon results, recipe, findings, credits

README.md

@@ -0,0 +1,98 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3.6-27B
+tags:
+  - speculative-decoding
+  - dspark
+  - dflash
+  - draft-model
+  - llama.cpp
+---
+
+# Qwen3.6-27B DSpark drafter (block-15, warm-started)
+
+A DSpark speculative-decoding drafter head for **Qwen3.6-27B**, trained with
+[DeepSpec](https://github.com/deepseek-ai/DeepSpec) in online mode against the target's own
+regenerated responses, warm-started from the z-lab DFlash head for this target. Served with
+llama.cpp (`--spec-type draft-dspark`) it delivers **1.6-2.7× decode speedup** on an RTX Pro 6000
+Blackwell at Q8_0, with **zero quality change** (speculative decoding is lossless: every accepted
+token is verified by the target).
+
+## Measured results (llama.cpp, RTX Pro 6000 Blackwell, target Qwen3.6-27B-Q8_0, greedy, 200-token completions)
+
+| metric | value |
+|---|---|
+| draft acceptance (code prompt) | 0.292 |
+| draft acceptance (counting) | 0.305 |
+| mean accepted length | 5.35-5.53 (block size 15) |
+| target baseline decode | 51.0 t/s |
+| with DSpark | 129.5-136.1 t/s (**2.54-2.67×**) |
+
+### SPEED-Bench (qualitative set, coding+writing, temperature 0, concurrency 1)
+
+| output len | coding | writing | baseline | speedup |
+|---|---|---|---|---|
+| 256 | 99.7 t/s | 97.8 t/s | 50.9 / 50.8 t/s | **1.96× / 1.93×** |
+| 1024 | 112.1 t/s | 81.0 t/s | 50.7 / 50.6 t/s | **2.21× / 1.60×** |
+
+Aggregate draft acceptance across the diverse SPEED-Bench prompts: 0.19-0.20: the honest
+"varied real prompts" number, vs 0.29-0.31 on narrow greedy probes. Both are reported; when you
+compare drafter heads, make sure you compare the same kind of probe.
+
+### Apple Silicon (M3 Max 128GB, Metal, same GGUFs, same probes)
+
+| metric | value |
+|---|---|
+| draft acceptance (code prompt) | 0.2922 (161/551: identical accepted count to CUDA) |
+| draft acceptance (counting) | 0.2945 |
+| target baseline decode | 12.3-12.4 t/s |
+| with DSpark | 17.1-17.3 t/s (**1.39×**) |
+
+Acceptance is backend-independent (matches CUDA to the third decimal on byte-identical GGUFs).
+The economics flip with target scale: this same pipeline's 0.8B head is a net *slowdown* on
+Metal (draft overhead dominates a fast tiny target), while at 27B the head pays for itself:
+1.39× on a MacBook is the difference between 12 and 17 tokens per second where it is actually
+felt.
+
+## Recipe (reproducible)
+
+- Data: 12,000 prompts (mlabonne/open-perfectblend), responses regenerated by the target itself
+  (llama-server, Q8_0 GGUF, 16 parallel slots, thinking disabled, max 2048 tokens), 11.4K valid.
+- Training: DeepSpec online mode (no precomputed cache; target forward runs per epoch on-GPU),
+  `Qwen3DSparkTrainer`, block_size 15, 5 draft layers, target extract layers [1, 16, 31, 46, 61],
+  markov_rank 256, num_anchors 128, lr 6e-4, global batch 128 (micro-batch 1), 3 epochs
+  (269 steps), bf16, single 96GB GPU, ~5.5h. Loss 2.04 → 1.04.
+- Warm-start: drafter initialized from z-lab's DFlash head for this target (shape-tolerant
+  load), the same trick fal used: it is worth several points of acceptance at this data scale
+  vs cold start.
+- Conversion: llama.cpp branch [satindergrewal/llama.cpp:dspark-qwen35](https://github.com/satindergrewal/llama.cpp/tree/dspark-qwen35)
+  (qwen3.5/3.6-family DSpark support on top of PR #25173):
+  `convert_hf_to_gguf.py <ckpt> --target-model-dir <hf-target> --outfile head.gguf --outtype bf16`
+- Serve: `llama-server -m Qwen3.6-27B-Q8_0.gguf -md head.gguf --spec-type draft-dspark --spec-draft-n-max 15 -fa on --jinja`
+
+## Findings worth more than the head
+
+1. **Warm-starting from a published DFlash/EAGLE head is the single highest-leverage trick at
+   small data scale.** This 12K-sample head reaches mean accepted length 5.4; our identically-sized
+   cold-start 0.8B head managed 1.8-2.05. The community scaling curve says ~250K+ samples from
+   scratch to match this.
+2. **Bigger targets make speculation pay.** At 27B the verify pass amortizes (mainline batch
+   scaling is essentially linear), so acceptance 0.19-0.31 already buys 1.6-2.7×. The same head
+   quality on a 0.8B target is a net slowdown: speculation value scales with target cost.
+3. **Trainer fix shipped along the way:** DeepSpec's checkpoint saver asserted micro-step
+   alignment at end-of-data, which crashes any run whose dataset×epochs isn't divisible by the
+   global batch (and online mode's valid-sample counts vary per epoch, so this is unavoidable).
+   Patched to align the final save down to the last completed optimizer step:
+   [satindergrewal/DeepSpec](https://github.com/satindergrewal/DeepSpec) `qwen35-lane`.
+
+## Files
+
+`model.safetensors` (drafter, bf16), `Qwen3.6-27B-DSpark.gguf` (block-15 head, ready for
+llama.cpp `-md`), training config, this card.
+
+## Credits
+
+DeepSeek (DeepSpec toolkit and the DSpark method), z-lab (DFlash warm-start weights), wjinxu
+(llama.cpp DSpark PR #25173), ruixiang63 (DFlash/EAGLE3 foundation in llama.cpp), Ofir408
+(DeepSpec online-training mode PR #23), hikarioyama (warm-start recipe), Qwen team (target model),
+mlabonne (open-perfectblend prompts).