senorperez/qwen3.6-35b-a3b-puzzle-60pct

Qwen3.6-35B-A3B, FFN-pruned to 60% width (s60)

A 21.68B research checkpoint produced by pruning Qwen/Qwen3.6-35B-A3B's expert FFN widths to ~60% (per-layer, mean 305.6 of 512) and healing with Muon + coarse-tail KL distillation.

⚠️ Experimental — it does not meet expectations

Published for reproducibility, not for use. It is worse than the parent on every benchmark, and the capability loss was not recovered by any amount of healing we tried:

	params	MMLU-Pro	IFEval	GPQA-D
parent Qwen3.6-35B-A3B	35B	77.78	84.10	80.81
this model	21.68B	67.80	71.90	56.57

Width→retention is linear from 75% to 60% with no knee: GPQA Diamond lost 12.12pp from parent to s75, and another 12.12pp from s75 to this model. It does fit in 16 GiB when 4-bit quantized (12.35 GiB), which was the only reason it was built.

Loading

This checkpoint uses per-layer FFN widths (moe_intermediate_sizes), which stock transformers does not read — it will build 40 uniform layers and fail on shape mismatch. Patch first:

import perlayer_moe          # included in this repo
perlayer_moe.patch()         # must precede from_pretrained
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained("senorperez/qwen3.6-35b-a3b-puzzle-60pct", dtype="bfloat16")

vLLM is not supported.

Details

Method: NVIDIA Puzzle-style FFN-only compression (arXiv 2411.19146). Whitened covariance-aware channel selection, per-layer width allocation by DP, then blockwise local distillation followed by global KD. Full write-up, all negative results, and the measurement bugs that invalidated earlier conclusions: https://github.com/sootaugur/puzzle (private).

Tip: reduce active experts by truncating the router's top-k rather than lowering num_experts_per_tok — the latter renormalises the surviving gates and costs 2–5× more perplexity at identical FLOPs (+2.74% vs +0.57% at k=6).