# FSI_Edge Gold Standard Training Recipe
## Frontier Lab Techniques → Refined for FSI_Edge

Based on deconstructed training pipelines from OpenAI (GPT-5), Anthropic (Claude),
Google DeepMind (Gemini), and Meta (Code Llama).

## ====== THE GOLD STANDARD PIPELINE ======

## STAGE 0: DATA CURATION (The Secret Sauce)
The labs spend 60%+ of their time here. This is what makes or breaks the model.

### 0a. Source Acquisition
```
Sources:
├── Public code: GitHub Archive, GitLab, Bitbucket (deduped)
├── Licensed data: The Stack v2, CodeSearchNet, BigQuery
├── Synthetic: Code exercises generated by teacher models
├── Execution traces: Runtime logs, debug output, test results
├── Documentation: MDN, Python docs, Rust book, Go docs
├── Q&A: StackOverflow (filtered), CodeForces, LeetCode discussions
└── NLP: FineWeb-Edu (top educational content), arXiv papers
```

### 0b. Decontamination (Critical for benchmarks!)
```
1. n-gram overlap removal (13-gram minimum) with HumanEval/MBPP/CodeContests
2. Edit distance < 0.8 removal against held-out test sets
3. URL/commit hash matching against known benchmark sources
4. All evaluation files removed from training corpus
5. Cross-contamination check: family code → filter eval dataset overlaps
```

### 0c. Quality Filtering (The OpenAI/Anthropic approach)
```
1. Model-based filtering: Classifier trained on high-quality code
   - Features: documentation coverage, test coverage, stars, commit frequency
   - Threshold: p(high_quality) > 0.85
2. Heuristic filtering:
   - Min/max file length: 30-50000 chars
   - Min comment ratio: > 5%
   - No binary files, no generated code, no minified JS
3. Deduplication:
   - Exact: SHA256 hash match removal
   - Near-exact: MinHash LSH (Jaccard > 0.8)
   - Semantic: Code embedding similarity (cosine > 0.95)
4. PII removal: Regex-based + classifier for emails, tokens, keys
```

### 0d. Data Enrichment (The Gemma 3 approach)
```
For each code sample, augment with:
├── AST annotations: node types, depth, scope boundaries
├── Docstring extraction: function descriptions as NLP pairs
├── Test case generation: auto-generate test harnesses
├── Execution traces: run code, capture stdout/stdout + errors
├── Dependency info: import graph, module relationships
└── Difficulty score: cyclomatic complexity + comment ratio
```

### 0e. Synthetic Data Generation (The Code Llama Self-Instruct method)
```
Pipeline:
1. Prompt teacher with: "Write a programming problem about {concept}"
2. Generate unit tests for the problem
3. Generate K solutions (K=10)
4. Execute tests against each solution
5. Keep ONLY solutions that pass ALL tests
6. Generate step-by-step reasoning for each solution
7. Format as: problem → reasoning → solution → tests

Target: 500K high-quality synthetic problems
```

## STAGE 1: PRE-TRAINING (Next-Token Prediction)

### 1a. Curriculum Design (Progressive)
```
Phase    Tokens     Context    LR        Data Mix
─────────────────────────────────────────────────────
1a       500B       4K         3e-4      100% code (syntax acquisition)
1b       1T         8K         3e-4      70% code / 30% NLP
1c       1.5T       16K        2e-4      50% code / 40% NLP / 10% math
1d       1T         16K        1e-4      30% code / 50% NLP / 20% reasoning
```

### 1b. FIM (Fill-in-the-Middle) Training
```
- 80% of tokens use FIM objective (Code Llama method)
- PSM mode: Prefix-Suffix-Middle (60%)
- SPM mode: Suffix-Prefix-Middle (40%)
- Special tokens: <FIM_PREFIX>, <FIM_SUFFIX>, <FIM_MIDDLE>
- Benefits: learns code structure bidirectional, better infilling
```

### 1c. Training Hyperparameters (from frontier research)
```
Parameter          Value          Source
─────────────────────────────────────────────
Optimizer          AdamW           OpenAI/Meta
β1, β2             0.9, 0.95      All labs
Weight decay       0.1            Standard
LR schedule        Cosine decay   All labs
Warmup steps       2000           Standard
Batch size         4M tokens      Meta (Code Llama)
Grad clipping      1.0            Standard
Precision          BF16           All labs (H100 native)
```

## STAGE 2: CODE SPECIALIZATION (Continued Pre-training)

### 2a. Code-Focused Continued Training (CodeGemma method)
```
Additional 500B-1T tokens of pure code:
├── Fill-in-the-Middle (80% FIM rate)
├── Line completion objective
├── Repository-level packing (co-locate related files)
├── Multi-language mixture:
│   Python: 30%
│   JavaScript/TS: 15%
│   Java: 12%
│   C++: 10%
│   Rust: 8%
│   Go: 8%
│   SQL: 7%
│   Others: 10%
└── Execution trace tagging
```

### 2b. Mid-Training Intervention (Google DeepMind technique)
```
Branch off at 80% of pretraining → inject code-heavy data
├── Prevents catastrophic forgetting of code
├── Earlier branching = better code retention
├── Gradual mixture shift over 100K steps
└── Keep NLP in mixture (30%) to maintain language understanding
```

## STAGE 3: SUPERVISED FINE-TUNING (SFT)

### 3a. Instruction Data (Multi-source curated)
```
Sources:
├── OpenOrca: High-quality GPT-4 reasoning traces
├── CodeAlpaca: Self-instruct code tasks
├── OASST1: Diverse human-assistant conversations
├── Anthropic HH-RLHF: Helpful & harmless dialogue
├── Self-generated: Using teacher models + execution validation
│   - 100K coding problems with verified solutions
│   - 50K debugging tasks (introduce bug → identify → fix)
│   - 50K code review pairs (bad code → good code explanation)
│   - 30K test generation tasks
│   - 20K API usage examples
```

### 3b. SFT Formulation (The Anthropic "why" approach)
```
Instead of just showing correct code, teach:
├── Reasoning traces: "Let me think about this..."
├── Error analysis: "The bug is caused by..."
├── Alternatives: "We could also do this by..."
├── Trade-offs: "This approach is O(n) vs O(n²) because..."
├── Edge cases: "We need to handle empty input when..."
```

### 3c. SFT Hyperparameters
```
Parameter          Value
──────────────────────────
LR                 2e-5
Batch size         128
Epochs             2-3
Max length         8K (extend to 16K in final phase)
Dropout            0.1
Label smoothing    0.1
Warmup             200 steps
```

## STAGE 4: REINFORCEMENT LEARNING WITH EXECUTION FEEDBACK

### 4a. GRPO (Group Relative Policy Optimization) - The DeepSeek method
```
For each prompt:
1. Generate K = 8 solutions at temperature 1.0
2. Execute each solution against unit tests
3. Score: pass@1 = 1 if any solution passes, 0 otherwise
4. Compute advantages: A_i = (score_i - mean(scores)) / std(scores)
5. Policy gradient with KL penalty against reference model

Advantage: No critic network needed (DeepSeek innovation)
```

### 4b. Multi-Objective Rewards (The Gemini approach)
```
Reward = w1 * ExecutionCorrectness + w2 * Efficiency + w3 * Readability + w4 * Safety

ExecutionCorrectness: Pass rate over held-out test suite (0.0-1.0)
Efficiency: Log-normalized runtime percentile (0.0-1.0)
Readability: LLM-as-judge evaluates code quality (0.0-1.0)
Safety: Whether code contains vulnerabilities (0 or 1)

Weight schedule:
─ Phase 1 (0-1K steps): w1=1.0, w2=0, w3=0, w4=0 (just get correct)
─ Phase 2 (1K-5K steps): w1=1.0, w2=0.3, w3=0.1, w4=0.1
─ Phase 3 (5K+ steps): w1=1.0, w2=0.5, w3=0.3, w4=0.2
```

### 4c. RL Training Protocol (OpenAI/Anthropic hybrid)
```
Step 1: On-policy sample from current policy
Step 2: Execute code in sandbox (isolated Docker container)
Step 3: Score using test suite
Step 4: Compute GRPO advantage
Step 5: Update policy with KL(zero || current) < ε
Step 6: Every 100 steps, evaluate on held-out set
Step 7: If accuracy plateaus, increase temperature or sample count

Total: 10K-20K RL steps
```

## STAGE 5: PREFERENCE ALIGNMENT

### 5a. DPO (Direct Preference Optimization)
```
Dataset: Code preferences
- Pair A: Code that compiles + passes tests + is clean
- Pair B: Code that fails tests OR is buggy OR is messy
- Preference: Human-validated or automatically determined by test results

DPO loss: -E[log σ(β(log π(y_w|x) - π_ref(y_w|x) - (log π(y_l|x) - π_ref(y_l|x))))]
β = 0.1
```

### 5b. Constitutional AI (Anthropic method, adapted for code)
```
Code Constitution:
1. "Write correct, idiomatic code that handles edge cases"
2. "Include clear error messages for invalid inputs"
3. "Prefer readability over cleverness"
4. "Add comments explaining non-obvious logic"
5. "Use appropriate data structures for the problem"
6. "Consider time and space complexity trade-offs"
7. "Write defensive code that validates inputs"

SDF: Generate documents discussing these principles → train on them
      before RL phase (this is Anthropic's key insight)
```

## STAGE 6: LONG-CONTEXT ADAPTATION

### 6a. Context Extension (Code Llama method)
```
1. Start at 16K context (trained)
2. Increase RoPE base frequency: θ = 10^6 → 10^7
3. Continue training on long sequences:
   - 32K: 10K steps, 2M tokens/batch, LR=2e-5
   - 64K: 5K steps, 1M tokens/batch, LR=1e-5
   - 128K: 5K steps, 0.5M tokens/batch, LR=5e-6
4. Pack repository-level files into single sequences
```

## FINAL ENSEMBLE: THE COMPLETE TIMELINE

```
Week  1-2:   Data curation, dedup, decontamination, synthetic gen
Week  3-5:   Stage 1a-1b: Pretraining (two phases)
Week  6:     Stage 1c-1d: Continued pretraining with curriculum
Week  7:     Stage 2: Code specialization (FIM training)
Week  8:     Mid-training intervention checkpoint
Week  9:     Stage 3: SFT on curated instruction data
Week  10:    Stage 4a-4b: GRPO RL with execution feedback
Week  11:    Stage 4c: Multi-objective refinement
Week  12:    Stage 5: DPO + Constitutional alignment
Week  12-13: Stage 6: Long-context adaptation
Week  13:    Final eval, failure analysis, targeted retraining
```

## EVALUATION GATE (Must-pass for #1 in class)

```
Target Benchmarks:
────────────────────────────────────────────────
HumanEval pass@1:      ≥ 80%
HumanEval+ pass@1:     ≥ 75%
MBPP pass@1:           ≥ 75%
CodeContests:          ≥ 40%
SWE-bench Lite:        ≥ 35%
CruxEval:              ≥ 60%
RepoBench:             ≥ 45%
MultiPL-E (5 langs):   ≥ 65% avg
CodeForce rating:      ≥ 1200
BigCodeBench:          ≥ 55%
────────────────────────────────────────────────
```

## The Mission

Build the #1 tiny coding model ever created. Solo-developed.
From-scratch novel architecture. 80%+ accuracy across all code benchmarks.
Natural language precision that understands, writes, and debugs code like a senior engineer.
Small enough to fit in your palm. Fast enough for real-time. Accurate enough for production.

No excuses. No shortcuts. Just the best engineering.