Upload ANALYSIS.md with huggingface_hub

ANALYSIS.md

@@ -0,0 +1,120 @@
+# Parameter Golf SOTA Analysis & Novel Optimization Plan
+
+## Current SOTA: 1.0810 BPB (PR #1493)
+
+### Techniques Already Used (the current stack):
+1. **SP8192 tokenizer** - SentencePiece 8192 vocab (up from 1024 baseline)
+2. **11 layers x 512d x 8H/4KV** - GQA architecture
+3. **MLP 4x** with LeakyReLU(0.5)^2 activation
+4. **Depth recurrence** - loops layers 3-5, creating 17 virtual from 11 physical layers
+5. **Parallel residuals** (from layer 7) - GPT-J style parallel attention+MLP
+6. **XSA** (cross-sequence attention) on all 11 layers
+7. **Partial RoPE** (16/64 dims) - saves RoPE compute, rest is free dimensions
+8. **Layerwise LN scale** (1/sqrt(layer_idx+1))
+9. **QK-Gain 5.25** - learnable per-head query scaling
+10. **U-Net skip connections** with sigmoid skip gates
+11. **Resid mixing** (learnable convex combination of current+initial hidden state)
+12. **MuonEq-R optimizer** (row-normalized Muon + Newton-Schulz)
+13. **EMA** (decay 0.9965)
+14. **GPTQ SDClip** (int6 matrices, int8 embeddings) with byte-shuffle + Brotli-11
+15. **Score-first TTT** at eval (SGD lr=0.005, mom=0.9, 3 epochs, 32K chunks)
+16. **Sliding window eval** (stride=64)
+17. **LZMA code compression**
+18. **Orthogonal weight init** for large matrices
+19. **WD=0.095** (high weight decay drives RMS down → better compression ratio)
+20. **Warmdown 72%** (cosine decay over final 72% of training)
+21. **Sequence length 2048** (up from 1024 baseline)
+
+### Architecture Details:
+- Encoder indices: [0,1,2,3,4,5,3,4]
+- Decoder indices: [5,3,4,5,6,7,8,9,10]
+- Parallel residuals: layers 7-10
+- XSA on all layers
+- looping activated at frac=0.35 of training
+
+## Opportunities for Improvement (ordered by expected impact)
+
+### 1. In-Place TTT During Training (NOT just eval) ★★★★★
+**Current**: TTT only at eval time (score-first SGD adaptation)
+**Novel**: Use In-Place TTT (arxiv 2604.06169) during TRAINING to improve learning.
+- Repurpose W_down in MLP as fast weights
+- Chunk-wise updates (512-1024 tokens)
+- NTP-aligned objective (not reconstruction)
+- Key insight: This effectively gives the model "memory" that adapts per-document DURING training, improving sample efficiency
+- Zero extra stored params (W_down is already part of the model)
+- At eval time, the model naturally does better TTT because it was trained with TTT objective
+
+### 2. Multi-Resolution Recurrence (SpiralFormer) ★★★★
+**Current**: Simple depth recurrence (loop layers 3-5 at same resolution)
+**Novel**: Apply SpiralFormer multi-resolution schedule to the recurrent loops.
+- Early loop iterations at coarsened resolution (e.g., r=0.5, L/2 tokens)
+- Later iterations at full resolution
+- Saves FLOPs on early passes → can afford MORE loop iterations in the same time budget
+- Coarse-to-fine schedule naturally induces hierarchical processing
+- SpiralFormer-L at 410M reduced FLOPs by 3-10% while IMPROVING perplexity
+
+### 3. Multi-Token Prediction (MTP) Auxiliary Loss ★★★★
+**Current**: Standard next-token prediction only
+**Novel**: Add n=2 or n=4 multi-token prediction heads during training (arxiv 2404.19737)
+- Shared trunk, independent output heads for predicting tokens t+1, t+2, ..., t+n
+- At training: loss = avg(CE(head_i, token_{t+i})) for i=1..n
+- At eval: discard extra heads, use only head_1
+- ZERO extra params in artifact (heads discarded)
+- 20-30% improved sample efficiency → more learning per training step
+- "Reinforces choice points" - model learns to plan ahead
+
+### 4. Sigma-MoE FFN Within Shared Block ★★★
+**Current**: Dense MLP in all layers
+**Novel**: Replace MLP in the recurrent (shared) layers with Sigma-MoE
+- Many small experts (e.g., 32 experts, top-2 active)
+- Routing uses Sigma function for smooth gating
+- Total params in MoE > dense MLP, but only top-K activated → same FLOPs
+- Since shared layers are looped, the MoE weights are stored ONCE
+- This multiplies effective parameter capacity of the recurrent core
+- MoEUT (arxiv 2405.16039) showed MoE outperforms dense at same param count
+
+### 5. Improved GPTQ with Hessian-Weighted Quantization ★★★
+**Current**: SDClip uses k * std(row) clipping
+**Novel**: Use actual Hessian diagonal (H_diag = E[x_i^2]) to weight quantization
+- GPTQ already uses Hessian; but SDClip approximates with std
+- Full Hessian-aware quantization (AQLM/QuIP# style) could squeeze more bits
+- Better rate-distortion tradeoff → lower BPB for same artifact size
+
+### 6. Larger Vocabulary (SP16384 or SP32768) ★★★
+**Current**: SP8192 (moved from 1024 → 8192 was a huge win)
+**Novel**: Continue the trend to SP16384 or SP32768
+- More tokens per byte = fewer tokens per document = more text seen in 10 min
+- Embeddings get more expensive but with int8 GPTQ they're compressed
+- The BPB metric rewards efficient tokenization
+- But: diminishing returns and embedding size grows linearly with vocab
+
+### 7. Progressive Depth Recurrence Schedule ★★
+**Current**: Fixed recurrence (3 loops of layers 3-5)
+**Novel**: Progressive recurrence - start with no loops, gradually increase
+- Early training: no recurrence (all 11 layers unique)
+- Mid training: loop 1 time
+- Late training: loop 2+ times
+- Avoids early training instability from too-deep recurrence
+- Already partially done (enable_looping_at=0.35), but could be more gradual
+
+### 8. Curriculum Learning on Data Quality ★★
+**Novel**: Start training on all FineWeb data, progressively filter to higher quality
+- Similar to data annealing from Llama papers
+- Easy to implement: just re-rank training shards by quality score
+- Could improve BPB if val set favors high-quality text patterns
+
+## Implementation Plan
+
+**Phase 1: Core Innovations** (highest impact, 3 techniques)
+1. Multi-Token Prediction auxiliary loss (n=2)
+2. SpiralFormer multi-resolution recurrence
+3. In-Place TTT during training (chunk-wise, NTP-aligned)
+
+**Phase 2: Architecture Refinements**
+4. Sigma-MoE in shared layers
+5. SP16384 tokenizer exploration
+6. Progressive recurrence schedule
+
+**Phase 3: Compression Optimization**
+7. Hessian-weighted GPTQ refinement
+8. INT4/INT5 experiments for matrices (if artifact space allows)