senorperez/qwen3.6-35b-a3b-puzzle-75pct

Qwen3.6-35B-A3B, FFN-pruned to 75% width (s75)

A 26.61B research checkpoint produced by pruning Qwen/Qwen3.6-35B-A3B's expert FFN widths to ~75% (per-layer, mean 384 of 512) and healing with Muon + coarse-tail KL distillation.

⚠️ Experimental — it does not meet expectations

Published for reproducibility, not for use. It is worse than the parent on every benchmark, and the capability loss was not recovered by any amount of healing we tried:

	params	MMLU-Pro	IFEval	GPQA-D
parent Qwen3.6-35B-A3B	35B	77.78	84.10	80.81
this model	26.61B	73.65	79.30	68.69

The central finding of the project was that the knowledge ceiling is set at prune time: doubling the healing token budget cut KL to the parent by 20.9% and moved MMLU-Pro by +0.38pp (not significant). Frontier reasoning degrades hardest (GPQA retention 85%), retrieval least (99.3%).

Loading

This checkpoint uses per-layer FFN widths (moe_intermediate_sizes), which stock transformers does not read — it will build 40 uniform layers and fail on shape mismatch. Patch first:

import perlayer_moe          # included in this repo
perlayer_moe.patch()         # must precede from_pretrained
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained("senorperez/qwen3.6-35b-a3b-puzzle-75pct", dtype="bfloat16")

vLLM is not supported.

Details

Method: NVIDIA Puzzle-style FFN-only compression (arXiv 2411.19146). Whitened covariance-aware channel selection, per-layer width allocation by DP, then blockwise local distillation followed by global KD. Full write-up, all negative results, and the measurement bugs that invalidated earlier conclusions: https://github.com/sootaugur/puzzle (private).

Tip: reduce active experts by truncating the router's top-k rather than lowering num_experts_per_tok — the latter renormalises the surviving gates and costs 2–5× more perplexity at identical FLOPs (+2.74% vs +0.57% at k=6).