True2456/Step-3.7-p15-ream-shared8-head8

Step-3.7-REAM-4bit-MLX — experimental, rejected as a deploy candidate

This is not the recommended deployment build. It's uploaded for research transparency, not as a suggested download. The recommended deploy build from this same project is the pruned shared8-head8 variant, not this one.

What this is

An alternative to pruning: instead of dropping low-saliency MoE experts, each pruned expert is merged into its most-similar kept expert (router-row cosine similarity), with the kept expert becoming a saliency-weighted average of itself plus everything it absorbed (REAM = merge-not-prune). Output expert count is identical to the pruned model (245/288) -- this changes the kept experts' values, not the count. 1,806 experts merged (43 per layer x 42 layers), 4.68 bits/weight.

Why it's rejected

Perplexity looked like a large win (-0.194 NLL overall vs the pruned deploy model, ~2x the combined gains of two other real improvements in the same project). It wasn't real. Two tests confirmed this:

1. Math/factual accuracy (24 items, single-right-answer): pruned model 24/24, REAM 23/24 -- REAM one item behind, within noise. PPL predicted no gain here (low-entropy category) and accuracy confirmed it.
2. Tool-call accuracy (15 items, exact function name + arguments): 14/15 for both models -- an exact tie, despite REAM showing a 41% lower perplexity (-0.53 NLL) on this exact category. This was the strongest signal in the whole project against trusting PPL: its largest, most task-relevant gain converted to zero real capability improvement.

The mechanism: averaging expert weights flattens the output distribution, which perplexity rewards independent of whether the underlying capability improved. Gains scaled with category entropy (biggest on tool_use/coding, zero on reasoning_math) -- the textbook signature of smoothing, not learning.

Full writeup with tables: docs/REAM-RESULT.md in the source repo (github search: Step-3.7 REAP pruning pipeline).

Kept for the record, not for deployment

The merge code is correct and the method could plausibly help a genuinely redundant MoE with a real low-saliency tail -- this model, measured directly, didn't have one. Uploaded as a negative result: perplexity moved -0.194 while real capability was flat-to-worse, which is exactly the trap a PPL-only evaluation pipeline would have missed.