Revised SOTA Approach for Parameter Golf

Why the Original MTP Approach Was Wrong

Multi-Token Prediction (Meta FAIR, 2404.19737) is not effective at small model scales:

Paper's own Table 1: gains appear only at 300M+ parameters
Follow-up paper (2505.22757) explicitly shows MTP with subword tokenization fails for SLMs
At 8M parameters, MTP adds compute overhead without meaningful sample efficiency gains
The only exception: byte-level tokenization with reverse curriculum — but we use SP8192

Revised Technique Stack (ordered by expected impact)

1. QAT-Fused Cooldown (Replace GPTQ Post-Hoc) ★★★★★

Paper: Compute-Optimal QAT (2509.22935, Sep 2025)

Current SOTA: Trains in FP16/BF16 → GPTQ quantization after training ends Novel: Start INT6 QAT (Straight-Through Estimator) during the LR warmdown phase

Why this works at 8M scale:

Model is 225× overtrained relative to Chinchilla (36B tokens / 8M params)
More training tokens = more accumulated quantization error for post-hoc GPTQ
QAT during warmdown lets the optimizer actively adapt weights to quantization noise
The paper shows QAT-cooldown fusion consistently improves over separate phases at 4-6 bit
Zero extra artifact cost — same INT6 weights, just better quality

Implementation:

# At training fraction >= 0.65 (during warmdown), enable fake quantization
if frac >= 0.65:
    for name, param in model.named_parameters():
        if param.ndim == 2 and param.numel() > 65536:
            # Symmetric per-row INT6 fake quantization
            scale = param.abs().amax(dim=1, keepdim=True) / 31  # 2^5 - 1
            param.data = (param / scale).round().clamp(-31, 31) * scale

Expected gain: -0.003 to -0.008 BPB (better quantization quality)

2. INT4 Mixed Precision → Pack More Parameters ★★★★★

Current: INT6 matrices (6 bits) + INT8 embeddings → ~15.99 MB Novel: INT4 for MLP weights (highest redundancy) + INT6 for attention + INT8 embeddings

Why this works:

MLP weights are 4× expansion = largest matrices, most redundant
At INT4: MLP weights use 33% less space than INT6
Freed budget (~1-2 MB) can be used for: wider model, more layers, or bigger embeddings
With QAT-fused cooldown, INT4 quality is much better than post-hoc INT4

Parameter budget (INT4 MLP + INT6 attn + INT8 embed):

Current 11L × 512d (INT6 uniform):
  MLP: 11 × (512×2048 + 2048×512) × 6/8 = ~8.6 MB
  Attn: 11 × (512×512×4) × 6/8 = ~5.1 MB  
  Embed: 8192×512 × 1 = ~4.2 MB
  Total: ~17.9 MB → compressed to ~15.99 MB

With INT4 MLP:
  MLP: 11 × (512×2048 + 2048×512) × 4/8 = ~5.7 MB  (saves ~2.9 MB)
  Attn: 11 × (512×512×4) × 6/8 = ~5.1 MB
  Embed: 8192×512 × 1 = ~4.2 MB  
  Total: ~15.0 MB → room for 12 layers or 576-dim model

Expected gain: -0.005 to -0.015 BPB (more model capacity)

3. NuMuon Optimizer (Nuclear-Norm Constrained Muon) ★★★★

Paper: NuMuon (2603.03597, Mar 2025)

What it does: Adds a nuclear-norm penalty to Muon that forces weights into low-rank structure during training. This makes post-training quantization/compression dramatically more effective.

Results: On Llama-1.8B with FineWeb-Edu:

40% SVD compression: NuMuon retains 97% quality vs 91% for Muon
80% compression: NuMuon retains 89% vs 73% for Muon
The low-rank weight structure is free at inference — just better weight matrices

Why it helps Parameter Golf:

Current GPTQ with Brotli-11 compression benefits from low-entropy weight distributions
NuMuon-trained weights have lower effective rank → lower entropy → better compression
Even at 18% slower per step, the compression gains are worth it

Trade-off: ~18% fewer training steps (3700 vs 4500). But each step produces weights that compress 20-40% better.

Expected gain: -0.002 to -0.005 BPB (better compression ratio → more effective params)

4. Wider Model with Depth Recurrence ★★★

Current: 11 physical layers × 512d, loop 3 layers → 17 virtual layers Novel: Reduce to 9 physical layers × 576d, loop 3 layers → 15 virtual layers

Why: At extreme depth recurrence, width > depth for the stored parameters. A wider model captures more features per layer, and the recurrence provides depth. 576d × 9L has similar param count to 512d × 11L but each layer is more expressive.

Expected gain: -0.001 to -0.003 BPB

5. Progressive Depth Recurrence During QAT ★★★

Current: Looping enabled at frac=0.35, fixed thereafter Novel: Start with 1 loop, progressively increase to 3 loops during training

At QAT-cooldown fusion time, the model has adapted to 3-loop depth. This means the quantized model's recurrent behavior is well-trained.

Expected gain: -0.001 to -0.002 BPB

What NOT to Do at 8M Scale

Technique	Why Skip
Multi-Token Prediction	Gains only at 300M+; proven ineffective for SLMs
BitNet 1.58-bit	Needs 2× hidden dim to match FP16; net zero at 16MB
Knowledge Distillation	10-min constraint too tight for online distillation
SOAP optimizer	Muon dominates at large batch sizes
Byte-level tokenization	Too many tokens per document; SP8192 is better

Combined Expected Improvement

Technique	Expected Δ BPB
QAT-Fused Cooldown	-0.003 to -0.008
INT4 MLP Mixed Precision	-0.005 to -0.015
NuMuon Optimizer	-0.002 to -0.005
Wider model (576d)	-0.001 to -0.003
Progressive recurrence	-0.001 to -0.002
Total	-0.012 to -0.033

Conservative target: 1.0810 - 0.012 = 1.069 BPB Optimistic target: 1.0810 - 0.033 = 1.048 BPB

Key Insight

The biggest lever is packing more parameters into 16MB via better quantization (QAT + INT4). At 8M params, the model is capacity-limited. Going from 8M to 12M params (via INT4 MLP) is like scaling up 50% — that's worth ~0.01 BPB on FineWeb scaling curves.