# Parameter Golf SOTA Analysis & Novel Optimization Plan

## Current SOTA: 1.0810 BPB (PR #1493)

### Techniques Already Used (the current stack):
1. **SP8192 tokenizer** - SentencePiece 8192 vocab (up from 1024 baseline)
2. **11 layers x 512d x 8H/4KV** - GQA architecture
3. **MLP 4x** with LeakyReLU(0.5)^2 activation
4. **Depth recurrence** - loops layers 3-5, creating 17 virtual from 11 physical layers
5. **Parallel residuals** (from layer 7) - GPT-J style parallel attention+MLP
6. **XSA** (cross-sequence attention) on all 11 layers
7. **Partial RoPE** (16/64 dims) - saves RoPE compute, rest is free dimensions
8. **Layerwise LN scale** (1/sqrt(layer_idx+1))
9. **QK-Gain 5.25** - learnable per-head query scaling
10. **U-Net skip connections** with sigmoid skip gates
11. **Resid mixing** (learnable convex combination of current+initial hidden state)
12. **MuonEq-R optimizer** (row-normalized Muon + Newton-Schulz)
13. **EMA** (decay 0.9965)
14. **GPTQ SDClip** (int6 matrices, int8 embeddings) with byte-shuffle + Brotli-11
15. **Score-first TTT** at eval (SGD lr=0.005, mom=0.9, 3 epochs, 32K chunks)
16. **Sliding window eval** (stride=64)
17. **LZMA code compression**
18. **Orthogonal weight init** for large matrices
19. **WD=0.095** (high weight decay drives RMS down → better compression ratio)
20. **Warmdown 72%** (cosine decay over final 72% of training)
21. **Sequence length 2048** (up from 1024 baseline)

### Architecture Details:
- Encoder indices: [0,1,2,3,4,5,3,4]
- Decoder indices: [5,3,4,5,6,7,8,9,10]
- Parallel residuals: layers 7-10
- XSA on all layers
- looping activated at frac=0.35 of training

## Opportunities for Improvement (ordered by expected impact)

### 1. In-Place TTT During Training (NOT just eval) ★★★★★
**Current**: TTT only at eval time (score-first SGD adaptation)
**Novel**: Use In-Place TTT (arxiv 2604.06169) during TRAINING to improve learning.
- Repurpose W_down in MLP as fast weights
- Chunk-wise updates (512-1024 tokens)
- NTP-aligned objective (not reconstruction)
- Key insight: This effectively gives the model "memory" that adapts per-document DURING training, improving sample efficiency
- Zero extra stored params (W_down is already part of the model)
- At eval time, the model naturally does better TTT because it was trained with TTT objective

### 2. Multi-Resolution Recurrence (SpiralFormer) ★★★★
**Current**: Simple depth recurrence (loop layers 3-5 at same resolution)
**Novel**: Apply SpiralFormer multi-resolution schedule to the recurrent loops.
- Early loop iterations at coarsened resolution (e.g., r=0.5, L/2 tokens)
- Later iterations at full resolution
- Saves FLOPs on early passes → can afford MORE loop iterations in the same time budget
- Coarse-to-fine schedule naturally induces hierarchical processing
- SpiralFormer-L at 410M reduced FLOPs by 3-10% while IMPROVING perplexity

### 3. Multi-Token Prediction (MTP) Auxiliary Loss ★★★★
**Current**: Standard next-token prediction only
**Novel**: Add n=2 or n=4 multi-token prediction heads during training (arxiv 2404.19737)
- Shared trunk, independent output heads for predicting tokens t+1, t+2, ..., t+n
- At training: loss = avg(CE(head_i, token_{t+i})) for i=1..n
- At eval: discard extra heads, use only head_1
- ZERO extra params in artifact (heads discarded)
- 20-30% improved sample efficiency → more learning per training step
- "Reinforces choice points" - model learns to plan ahead

### 4. Sigma-MoE FFN Within Shared Block ★★★
**Current**: Dense MLP in all layers
**Novel**: Replace MLP in the recurrent (shared) layers with Sigma-MoE
- Many small experts (e.g., 32 experts, top-2 active)
- Routing uses Sigma function for smooth gating
- Total params in MoE > dense MLP, but only top-K activated → same FLOPs
- Since shared layers are looped, the MoE weights are stored ONCE
- This multiplies effective parameter capacity of the recurrent core
- MoEUT (arxiv 2405.16039) showed MoE outperforms dense at same param count

### 5. Improved GPTQ with Hessian-Weighted Quantization ★★★
**Current**: SDClip uses k * std(row) clipping
**Novel**: Use actual Hessian diagonal (H_diag = E[x_i^2]) to weight quantization
- GPTQ already uses Hessian; but SDClip approximates with std
- Full Hessian-aware quantization (AQLM/QuIP# style) could squeeze more bits
- Better rate-distortion tradeoff → lower BPB for same artifact size

### 6. Larger Vocabulary (SP16384 or SP32768) ★★★
**Current**: SP8192 (moved from 1024 → 8192 was a huge win)
**Novel**: Continue the trend to SP16384 or SP32768
- More tokens per byte = fewer tokens per document = more text seen in 10 min
- Embeddings get more expensive but with int8 GPTQ they're compressed
- The BPB metric rewards efficient tokenization
- But: diminishing returns and embedding size grows linearly with vocab

### 7. Progressive Depth Recurrence Schedule ★★
**Current**: Fixed recurrence (3 loops of layers 3-5)
**Novel**: Progressive recurrence - start with no loops, gradually increase
- Early training: no recurrence (all 11 layers unique)
- Mid training: loop 1 time
- Late training: loop 2+ times
- Avoids early training instability from too-deep recurrence
- Already partially done (enable_looping_at=0.35), but could be more gradual

### 8. Curriculum Learning on Data Quality ★★
**Novel**: Start training on all FineWeb data, progressively filter to higher quality
- Similar to data annealing from Llama papers
- Easy to implement: just re-rank training shards by quality score
- Could improve BPB if val set favors high-quality text patterns

## Implementation Plan

**Phase 1: Core Innovations** (highest impact, 3 techniques)
1. Multi-Token Prediction auxiliary loss (n=2)
2. SpiralFormer multi-resolution recurrence
3. In-Place TTT during training (chunk-wise, NTP-aligned)

**Phase 2: Architecture Refinements**
4. Sigma-MoE in shared layers
5. SP16384 tokenizer exploration
6. Progressive recurrence schedule

**Phase 3: Compression Optimization**
7. Hessian-weighted GPTQ refinement
8. INT4/INT5 experiments for matrices (if artifact space allows)