REVISED_APPROACH.md

# Revised SOTA Approach for Parameter Golf

## Why the Original MTP Approach Was Wrong

Multi-Token Prediction (Meta FAIR, 2404.19737) is **not effective at small model scales**:
- Paper's own Table 1: gains appear only at 300M+ parameters
- Follow-up paper (2505.22757) explicitly shows MTP with subword tokenization fails for SLMs
- At 8M parameters, MTP adds compute overhead without meaningful sample efficiency gains
- The only exception: byte-level tokenization with reverse curriculum — but we use SP8192

## Revised Technique Stack (ordered by expected impact)

### 1. QAT-Fused Cooldown (Replace GPTQ Post-Hoc) ★★★★★
**Paper**: Compute-Optimal QAT (2509.22935, Sep 2025)

**Current SOTA**: Trains in FP16/BF16 → GPTQ quantization after training ends
**Novel**: Start INT6 QAT (Straight-Through Estimator) during the LR warmdown phase

**Why this works at 8M scale**:
- Model is 225× overtrained relative to Chinchilla (36B tokens / 8M params)
- More training tokens = more accumulated quantization error for post-hoc GPTQ
- QAT during warmdown lets the optimizer **actively adapt weights** to quantization noise
- The paper shows QAT-cooldown fusion consistently improves over separate phases at 4-6 bit
- Zero extra artifact cost — same INT6 weights, just better quality

**Implementation**:
```python
# At training fraction >= 0.65 (during warmdown), enable fake quantization
if frac >= 0.65:
    for name, param in model.named_parameters():
        if param.ndim == 2 and param.numel() > 65536:
            # Symmetric per-row INT6 fake quantization
            scale = param.abs().amax(dim=1, keepdim=True) / 31  # 2^5 - 1
            param.data = (param / scale).round().clamp(-31, 31) * scale
```

**Expected gain**: -0.003 to -0.008 BPB (better quantization quality)

### 2. INT4 Mixed Precision → Pack More Parameters ★★★★★
**Current**: INT6 matrices (6 bits) + INT8 embeddings → ~15.99 MB
**Novel**: INT4 for MLP weights (highest redundancy) + INT6 for attention + INT8 embeddings

**Why this works**:
- MLP weights are 4× expansion = largest matrices, most redundant
- At INT4: MLP weights use 33% less space than INT6
- Freed budget (~1-2 MB) can be used for: wider model, more layers, or bigger embeddings
- With QAT-fused cooldown, INT4 quality is much better than post-hoc INT4

**Parameter budget (INT4 MLP + INT6 attn + INT8 embed)**:
```
Current 11L × 512d (INT6 uniform):
  MLP: 11 × (512×2048 + 2048×512) × 6/8 = ~8.6 MB
  Attn: 11 × (512×512×4) × 6/8 = ~5.1 MB  
  Embed: 8192×512 × 1 = ~4.2 MB
  Total: ~17.9 MB → compressed to ~15.99 MB

With INT4 MLP:
  MLP: 11 × (512×2048 + 2048×512) × 4/8 = ~5.7 MB  (saves ~2.9 MB)
  Attn: 11 × (512×512×4) × 6/8 = ~5.1 MB
  Embed: 8192×512 × 1 = ~4.2 MB  
  Total: ~15.0 MB → room for 12 layers or 576-dim model
```

**Expected gain**: -0.005 to -0.015 BPB (more model capacity)

### 3. NuMuon Optimizer (Nuclear-Norm Constrained Muon) ★★★★
**Paper**: NuMuon (2603.03597, Mar 2025)

**What it does**: Adds a nuclear-norm penalty to Muon that forces weights into low-rank structure during training. This makes post-training quantization/compression dramatically more effective.

**Results**: On Llama-1.8B with FineWeb-Edu:
- 40% SVD compression: NuMuon retains 97% quality vs 91% for Muon
- 80% compression: NuMuon retains 89% vs 73% for Muon
- The low-rank weight structure is **free at inference** — just better weight matrices

**Why it helps Parameter Golf**:
- Current GPTQ with Brotli-11 compression benefits from low-entropy weight distributions
- NuMuon-trained weights have lower effective rank → lower entropy → better compression
- Even at 18% slower per step, the compression gains are worth it

**Trade-off**: ~18% fewer training steps (3700 vs 4500). But each step produces weights that compress 20-40% better.

**Expected gain**: -0.002 to -0.005 BPB (better compression ratio → more effective params)

### 4. Wider Model with Depth Recurrence ★★★
**Current**: 11 physical layers × 512d, loop 3 layers → 17 virtual layers
**Novel**: Reduce to 9 physical layers × 576d, loop 3 layers → 15 virtual layers

**Why**: At extreme depth recurrence, width > depth for the stored parameters. A wider model captures more features per layer, and the recurrence provides depth. 576d × 9L has similar param count to 512d × 11L but each layer is more expressive.

**Expected gain**: -0.001 to -0.003 BPB

### 5. Progressive Depth Recurrence During QAT ★★★
**Current**: Looping enabled at frac=0.35, fixed thereafter
**Novel**: Start with 1 loop, progressively increase to 3 loops during training

At QAT-cooldown fusion time, the model has adapted to 3-loop depth. This means the quantized model's recurrent behavior is well-trained.

**Expected gain**: -0.001 to -0.002 BPB

## What NOT to Do at 8M Scale

| Technique | Why Skip |
|-----------|----------|
| Multi-Token Prediction | Gains only at 300M+; proven ineffective for SLMs |
| BitNet 1.58-bit | Needs 2× hidden dim to match FP16; net zero at 16MB |
| Knowledge Distillation | 10-min constraint too tight for online distillation |
| SOAP optimizer | Muon dominates at large batch sizes |
| Byte-level tokenization | Too many tokens per document; SP8192 is better |

## Combined Expected Improvement

| Technique | Expected Δ BPB |
|-----------|---------------|
| QAT-Fused Cooldown | -0.003 to -0.008 |
| INT4 MLP Mixed Precision | -0.005 to -0.015 |
| NuMuon Optimizer | -0.002 to -0.005 |
| Wider model (576d) | -0.001 to -0.003 |
| Progressive recurrence | -0.001 to -0.002 |
| **Total** | **-0.012 to -0.033** |

**Conservative target**: 1.0810 - 0.012 = **1.069 BPB**
**Optimistic target**: 1.0810 - 0.033 = **1.048 BPB**

## Key Insight

The biggest lever is **packing more parameters into 16MB** via better quantization (QAT + INT4). At 8M params, the model is capacity-limited. Going from 8M to 12M params (via INT4 MLP) is like scaling up 50% — that's worth ~0.01 BPB on FineWeb scaling curves.