Upload REVISED_APPROACH.md with huggingface_hub

REVISED_APPROACH.md

@@ -0,0 +1,127 @@
+# Revised SOTA Approach for Parameter Golf
+
+## Why the Original MTP Approach Was Wrong
+
+Multi-Token Prediction (Meta FAIR, 2404.19737) is **not effective at small model scales**:
+- Paper's own Table 1: gains appear only at 300M+ parameters
+- Follow-up paper (2505.22757) explicitly shows MTP with subword tokenization fails for SLMs
+- At 8M parameters, MTP adds compute overhead without meaningful sample efficiency gains
+- The only exception: byte-level tokenization with reverse curriculum — but we use SP8192
+
+## Revised Technique Stack (ordered by expected impact)
+
+### 1. QAT-Fused Cooldown (Replace GPTQ Post-Hoc) ★★★★★
+**Paper**: Compute-Optimal QAT (2509.22935, Sep 2025)
+
+**Current SOTA**: Trains in FP16/BF16 → GPTQ quantization after training ends
+**Novel**: Start INT6 QAT (Straight-Through Estimator) during the LR warmdown phase
+
+**Why this works at 8M scale**:
+- Model is 225× overtrained relative to Chinchilla (36B tokens / 8M params)
+- More training tokens = more accumulated quantization error for post-hoc GPTQ
+- QAT during warmdown lets the optimizer **actively adapt weights** to quantization noise
+- The paper shows QAT-cooldown fusion consistently improves over separate phases at 4-6 bit
+- Zero extra artifact cost — same INT6 weights, just better quality
+
+**Implementation**:
+```python
+# At training fraction >= 0.65 (during warmdown), enable fake quantization
+if frac >= 0.65:
+    for name, param in model.named_parameters():
+        if param.ndim == 2 and param.numel() > 65536:
+            # Symmetric per-row INT6 fake quantization
+            scale = param.abs().amax(dim=1, keepdim=True) / 31  # 2^5 - 1
+            param.data = (param / scale).round().clamp(-31, 31) * scale
+```
+
+**Expected gain**: -0.003 to -0.008 BPB (better quantization quality)
+
+### 2. INT4 Mixed Precision → Pack More Parameters ★★★★★
+**Current**: INT6 matrices (6 bits) + INT8 embeddings → ~15.99 MB
+**Novel**: INT4 for MLP weights (highest redundancy) + INT6 for attention + INT8 embeddings
+
+**Why this works**:
+- MLP weights are 4× expansion = largest matrices, most redundant
+- At INT4: MLP weights use 33% less space than INT6
+- Freed budget (~1-2 MB) can be used for: wider model, more layers, or bigger embeddings
+- With QAT-fused cooldown, INT4 quality is much better than post-hoc INT4
+
+**Parameter budget (INT4 MLP + INT6 attn + INT8 embed)**:
+```
+Current 11L × 512d (INT6 uniform):
+  MLP: 11 × (512×2048 + 2048×512) × 6/8 = ~8.6 MB
+  Attn: 11 × (512×512×4) × 6/8 = ~5.1 MB  
+  Embed: 8192×512 × 1 = ~4.2 MB
+  Total: ~17.9 MB → compressed to ~15.99 MB
+
+With INT4 MLP:
+  MLP: 11 × (512×2048 + 2048×512) × 4/8 = ~5.7 MB  (saves ~2.9 MB)
+  Attn: 11 × (512×512×4) × 6/8 = ~5.1 MB
+  Embed: 8192×512 × 1 = ~4.2 MB  
+  Total: ~15.0 MB → room for 12 layers or 576-dim model
+```
+
+**Expected gain**: -0.005 to -0.015 BPB (more model capacity)
+
+### 3. NuMuon Optimizer (Nuclear-Norm Constrained Muon) ★★★★
+**Paper**: NuMuon (2603.03597, Mar 2025)
+
+**What it does**: Adds a nuclear-norm penalty to Muon that forces weights into low-rank structure during training. This makes post-training quantization/compression dramatically more effective.
+
+**Results**: On Llama-1.8B with FineWeb-Edu:
+- 40% SVD compression: NuMuon retains 97% quality vs 91% for Muon
+- 80% compression: NuMuon retains 89% vs 73% for Muon
+- The low-rank weight structure is **free at inference** — just better weight matrices
+
+**Why it helps Parameter Golf**:
+- Current GPTQ with Brotli-11 compression benefits from low-entropy weight distributions
+- NuMuon-trained weights have lower effective rank → lower entropy → better compression
+- Even at 18% slower per step, the compression gains are worth it
+
+**Trade-off**: ~18% fewer training steps (3700 vs 4500). But each step produces weights that compress 20-40% better.
+
+**Expected gain**: -0.002 to -0.005 BPB (better compression ratio → more effective params)
+
+### 4. Wider Model with Depth Recurrence ★★★
+**Current**: 11 physical layers × 512d, loop 3 layers → 17 virtual layers
+**Novel**: Reduce to 9 physical layers × 576d, loop 3 layers → 15 virtual layers
+
+**Why**: At extreme depth recurrence, width > depth for the stored parameters. A wider model captures more features per layer, and the recurrence provides depth. 576d × 9L has similar param count to 512d × 11L but each layer is more expressive.
+
+**Expected gain**: -0.001 to -0.003 BPB
+
+### 5. Progressive Depth Recurrence During QAT ★★★
+**Current**: Looping enabled at frac=0.35, fixed thereafter
+**Novel**: Start with 1 loop, progressively increase to 3 loops during training
+
+At QAT-cooldown fusion time, the model has adapted to 3-loop depth. This means the quantized model's recurrent behavior is well-trained.
+
+**Expected gain**: -0.001 to -0.002 BPB
+
+## What NOT to Do at 8M Scale
+
+| Technique | Why Skip |
+|-----------|----------|
+| Multi-Token Prediction | Gains only at 300M+; proven ineffective for SLMs |
+| BitNet 1.58-bit | Needs 2× hidden dim to match FP16; net zero at 16MB |
+| Knowledge Distillation | 10-min constraint too tight for online distillation |
+| SOAP optimizer | Muon dominates at large batch sizes |
+| Byte-level tokenization | Too many tokens per document; SP8192 is better |
+
+## Combined Expected Improvement
+
+| Technique | Expected Δ BPB |
+|-----------|---------------|
+| QAT-Fused Cooldown | -0.003 to -0.008 |
+| INT4 MLP Mixed Precision | -0.005 to -0.015 |
+| NuMuon Optimizer | -0.002 to -0.005 |
+| Wider model (576d) | -0.001 to -0.003 |
+| Progressive recurrence | -0.001 to -0.002 |
+| **Total** | **-0.012 to -0.033** |
+
+**Conservative target**: 1.0810 - 0.012 = **1.069 BPB**
+**Optimistic target**: 1.0810 - 0.033 = **1.048 BPB**
+
+## Key Insight
+
+The biggest lever is **packing more parameters into 16MB** via better quantization (QAT + INT4). At 8M params, the model is capacity-limited. Going from 8M to 12M params (via INT4 MLP) is like scaling up 50% — that's worth ~0.01 BPB on FineWeb scaling curves.