splats

/

Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-oQ8e-fp16

like 1

📎 PNG

Hi @splats , head-to-head eval of the Reasoning-Distilled-oQ8e checkpoint against the vanilla MLX-oQ8 baseline, on the oMLX harness, identical hardware and seed, ten benchmarks, both thinking modes.
Sharing the numbers because the trade-off is non-trivial and the maintainer view matters more than mine.

TL;DR

The distillation trades 3.6 pts of mean thinking-mode accuracy for 1.58x
faster wall-clock and 45% fewer output tokens. That trade is Pareto
positive on four benchmarks and Pareto negative on three. The remaining
three are roughly neutral on cost.
The most load-bearing finding is on HumanEval no-think: +14.0 pts,
93.3% vs 79.3%, 26 problems gained against 3 lost. That asymmetry says
reasoning is partially internalized, not just relocated.

Setup

• Distilled : splats/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-oQ8e-fp16
• Baseline : Qwen3.6-35B-A3B-MLX-oQ8-FP16
• Eval harness : oMLX 0.3.8, identical hardware/seed (MacStudio M3 Ultra), 10 benchmarks × {think, no-think}
• Token counts are character-based estimates (≈ chars / 4); wall-clock is end-to-end including sampling.

Full results

* MathQA no-think: both models output ~1-char answers; the aggregate ratio is dominated by a few outliers and not meaningful.

Pareto wins (thinking mode, acc parity or better AND cheaper)

MMLU-Pro: +4.0 pts at 42% fewer tokens, 1.71x faster. Cleanest win.
ARC-Challenge: 0.0 pts at 66% fewer tokens, 3.37x faster. Identical 97.3% accuracy at one-third the compute.
BBQ: +1.0 pts at 71% fewer tokens, 2.92x faster. Most efficient thinking trace in the entire eval.
MathQA: 0.0 pts at 31% fewer tokens, 1.49x faster. Modest but free.

Trade-off boundary

Long-form code generation in thinking mode is where the distilled underperforms.
HumanEval thinking: 95.1% vs 82.9%, delta -12.2 pts
MBPP thinking: 93.5% vs 78.5%, delta -15.0 pts
LiveCodeBench thinking: 52.0% vs 47.0%, delta -5.0 pts

The pattern points to compressed reasoning traces. Shorter traces help on multi-choice and short-form code, do not help on long-horizon code where extended deliberation pays off. TruthfulQA also regresses by 5 pts in both modes; mechanism less clear, possibly an alignment artifact from the teacher distribution.

Interpretation

The invariant being tested: does Claude-4.7-Opus distillation transfer reasoning into the student weights, or does it just shape inference-time output style?
The data points to genuine transfer, bounded to specific regimes:
Multi-choice and short-form reasoning: transfer succeeds. No-think gains on MMLU, MMLU-Pro, HumanEval are evidence.
Compressed thinking trace: transfer succeeds for short reasoning, yielding the four Pareto wins.
Long-horizon code generation: transfer incomplete. The teacher's long-trace behavior in thinking mode is not fully recovered.

Default suggestion for downstream users: the distilled is the better default for no-think serving paths and for any thinking workload outside long-form code. For HumanEval-style or MBPP-style production code generation with thinking budget available, vanilla oQ8 holds the edge.

Questions

Distillation setup: teacher-trace dataset size, response-based vs reasoning-trace distillation, training duration?
MBPP and HumanEval thinking-mode regression: observed during training or only at eval time?
oQ8e vs standard oQ8: calibration dataset, group size, anything notable?
Intermediate checkpoints: planned release? The no-think reasoning gain emerging across training would be a useful curve.

Raw JSONs and harness output available on request.

Thanks for the model and sorry for my AI assisted english translation.
Feel free to reuse my benchmark results.
I'll add Kimi reasoning soon.

∵ RCR Regis ∴