File size: 3,012 Bytes
eb03982 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 | ---
library_name: transformers
pipeline_tag: text-generation
tags:
- speculative-decoding
- dspark
- dflash
- specforge
- sglang
inference: false
---
# Kimi K3 DSpark speculator
## Overview
A DSpark speculator for the Kimi K3 target, enabling faster inference through speculative decoding. DSpark extends the DFlash parallel-draft backbone with a Markov logit-bias head and a per-position confidence head. This checkpoint was trained with [SpecForge](https://github.com/sgl-project/SpecForge/) using hidden states from a live SGLang target engine.
## Model Specifications
- **Base model:** `moonshotai/Kimi-K3`.
- **Format:** Safetensors (single-file BF16, 2,249,289,601 parameters).
- **Draft:** 5 layers (Qwen3-style GQA), hidden 7168, 64 heads / 16 KV heads, head_dim 64, FFN 14336, rope_theta 10000, `block_size=7`.
- **Vocabulary:** 163,840 rows; `mask_token_id=163824`, `bos=163584`, `eos=163586`, `pad=163839`.
- **DSpark heads:** Markov rank 256 (vanilla) and confidence head (with-Markov).
- **Aux hidden-state layers:** `[7, 23, 51, 67, 83]` (full-attention layers of the 93-layer target).
- **Trained context:** sequence length 4096.
- **Target weights:** target embedding and unembedding weights are not included in this checkpoint.
## Evaluation Results
| Dataset | Accept length |
|---|---:|
| GSM8K | 5.6660 |
| MATH500 | 4.0803 |
| HumanEval | 5.3613 |
| MBPP | 4.9220 |
| MT-Bench | 3.3425 |
| AIME26 | 2.6326 |
## Serving with SGLang
```bash
sglang serve \
--trust-remote-code \
--model-path moonshotai/Kimi-K3 \
--tp-size 8 \
--dcp-size 8 \
--mem-fraction-static 0.85 \
--max-mamba-cache-size 160 \
--max-running-requests 32 \
--cuda-graph-max-bs-decode 32 \
--reasoning-parser kimi_k3 \
--tool-call-parser kimi_k3 \
--host 0.0.0.0 \
--port 30000 \
--speculative-algorithm DSPARK \
--speculative-draft-model-path RadixArk/Kimi-K3-DSpark \
--speculative-dspark-block-size 7
```
## Training Details
- **Framework:** SpecForge online distillation, with hidden states captured from a frozen Kimi K3 target served by a live SGLang engine. Draft trained from random initialization.
- **Data:** 700K regenerated OpenPerfectBlend samples. `thinking_effort=max`, right-truncated at 4096.
- **Schedule:** 10 epochs, AdamW (fp32 master weights), peak learning rate `6e-4`, cosine decay to 0, 4% warmup, and gradient clipping at 1.0. Global batch 512, seed 42.
- **Loss:** `0.1 CE + 0.9 L1 distillation + 1.0 confidence BCE`, decay gamma 4.0, with 512 sampled anchors per sequence and `block_size=7`.
- **Topology:** 4 nodes × 4 GB300 (16 ranks) — 2 × TP8 target replicas, DP2 sampler, FSDP16 `SHARD_GRAD_OP` on the draft, TP-batch scatter. Batch 8 per replica × 32 accumulation steps × 2 replicas = global batch 512.
## Known Limitations
- This DSpark checkpoint was trained with a maximum context length of 4,096 tokens, which may lead to reduced acceptance lengths in extreme long-context and agentic use cases. Training for these scenarios is ongoing.
|