# FSI_Edge CPU Gold Standard Training Recipe ## From-scratch novel architecture training on ARM CPU Adapted from frontier lab techniques (OpenAI, Anthropic, Google DeepMind, Meta) for single-CPU training with the FSI_Edge novel architecture. ## ====== CORE PHILOSOPHY ====== CPU training trades parallelism for precision. Without GPU memory limits, we maximize throughput via: - Large batch sizes (system RAM instead of VRAM) - Deep gradient accumulation (simulate huge batches) - Full FP32 precision (no quantization noise) - All CPU cores (8 threads on ARM Cortex-A520) - Pre-allocated tensors (no dynamic growth in hot loops) ## ====== ENVIRONMENT ====== ``` Hardware: ARM Cortex-A520, 8 cores RAM: ~100GB available Storage: 5-100GB OS: Linux aarch64 Python: 3.12 Torch: 2.12.1+cpu (OpenMP + torch.compile available) ``` ### Thread Configuration ```python torch.set_num_threads(8) torch.set_num_interop_threads(8) os.environ['TOKENIZERS_PARALLELISM'] = 'false' os.environ['OMP_NUM_THREADS'] = '8' os.environ['OPENBLAS_NUM_THREADS'] = '8' ``` ### Critical: Disable tokenizer parallelism to avoid deadlocks with DataLoader ## ====== DATA PIPELINE (CPU-Optimized) ====== ### Tokenizer - BPE tokenizer with 32K vocab, trained on synthetic code - FIM special tokens: `<|fim_prefix|>`, `<|fim_middle|>`, `<|fim_suffix|>` - Cold-start tokens: `<|thought|>`, `<|answer|>`, `<|code|>`, `<|explain|>` ### Dataset Configuration ```python # CPU-optimized settings: batch_size: 8-64 # Large batches use system RAM (no VRAM limit) max_length: 128-256 # Shorter sequences for CPU speed num_workers: 4 # Parallel data loading with 8 cores prefetch_factor: 2 # Prefetch next batches pin_memory: false # Not needed on CPU (no GPU transfer) ``` ### Data Mix (CPU Training Schedule) ``` Phase 1 (0-10K steps): 100% code (syntax acquisition) Phase 2 (10K-50K steps): 70% code / 30% NLP Phase 3 (50K+ steps): 50% code / 40% NLP / 10% reasoning (cold-start) ``` ### Synthetic Data Generation ```bash # Generate code data (multi-language) python data/prepare_data.py --samples 100000 --output /FSI_Edge/data/train # Generate cold-start reasoning data python data/generate_coldstart.py --num 5000 --output /FSI_Edge/data/cold_start.jsonl ``` ## ====== MODEL ARCHITECTURE (CPU-Sized) ====== ### Recommended Sizes for CPU Training | Config | Params | d_model | Layers | Heads | d_ff | Steps/Day | Quality | |----------|--------|---------|--------|-------|-------|-----------|---------| | 4K-tiny | 4.7K | 64 | 2 | 4 | 256 | ~86K | Minimal | | 5K-small | 5.3K | 64 | 4 | 8 | 256 | ~14K | Basic | | 28K-med | 28K | 128 | 4 | 8 | 512 | ~3K | Moderate| **Default choice for sustained training: 4K-tiny (fastest iteration)** ### Head Distribution ```python # Auto-balanced by FSIEdgeConfig.__post_init__: # local_heads + struct_heads + global_heads == n_heads # For n_heads=4: local=2, struct=1, global=1 # For n_heads=8: local=4, struct=2, global=2 ``` ### Helix Memory (CPU-Optimized) ``` - Pre-allocated slot buffer (no dynamic growth) - Active mask instead of dynamic sizing - No .item() calls (torch.compile compatible) - Per-token loop preserved but streamlined - Compression: cluster-by-cluster merge when near limit ``` The per-token loop is the bottleneck (~0.25s per layer per forward). This is the fundamental cost of O(log L) memory scaling on CPU. ## ====== TRAINING LOOP (CPU Gold Standard) ====== ### Optimizer ```python AdamW(lr=3e-4, betas=(0.9, 0.95), weight_decay=0.1, eps=1e-8) ``` - AdamW per-param state uses ~2x memory of params (acceptable on CPU) - Consider SGD+Momentum if memory is constrained (less state) ### Learning Rate Schedule ```python # Cosine decay with linear warmup warmup_steps = 100 # Short warmup for CPU T_max = total_steps # Cosine period = total steps min_lr = 1e-5 # Floor at 3% of peak # Phase transitions: # 0-100: Linear warmup 3e-6 -> 3e-4 # 100-T_max: Cosine decay 3e-4 -> 1e-5 ``` ### Gradient Accumulation (CPU Advantage) ```python grad_accum_steps = 4-8 # Simulate larger batch on CPU effective_batch = batch_size * grad_accum_steps # e.g., 8*4=32 ``` CPU advantage: no VRAM limit means we can accumulate more gradients before each optimizer step. ### Effective Token Budget ``` Tokens per step: batch_size * max_length = 1 * 128 = 128 tokens With grad_accum=4: 512 tokens per optimizer step Target: 1M+ tokens for meaningful training Steps needed: 1M / 512 ≈ 2000 optimizer steps ``` ### Training Loop Structure ```python for step, batch in enumerate(loader): loss = model(**batch).loss loss_adjusted = loss / grad_accum_steps loss_adjusted.backward() if (step + 1) % grad_accum_steps == 0: torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) optimizer.step() scheduler.step() optimizer.zero_grad() # Log every N steps if step % log_interval == 0: log(step, loss, lr, grad_norm) # Checkpoint every N steps if step % save_interval == 0: save_checkpoint(model, optimizer, scheduler, step, loss) ``` ## ====== 9-STAGE TRAINING (CPU Adaptation) ====== ### Stage 0: Data Filtering (1 pass, offline) ``` Run once, before training: python data/prepare_data.py --samples 100000 python data/generate_coldstart.py --num 5000 ``` 4-stage filtering: dedup → quality → heuristic → grounding ### Stage 1: Pretraining (Next-Token Prediction) ``` Objective: Language modeling on code + NLP Duration: 10K-100K steps (depending on target size) Curriculum: Short (128) -> Medium (256) -> Long (512) Monitor: Loss should drop from ~10.5 to ~5.0 for 4K model ``` ### Stage 1b: Code Specialization (FIM) ``` Objective: Fill-in-the-Middle on code 80% FIM rate (Code Llama method) PSM (60%) + SPM (40%) corruption 5K-10K steps ``` ### Stage 2: Supervised Fine-Tuning ``` Objective: Learn instruction-following format Cold-start data: 5000 examples with reasoning traces 3 epochs over cold-start data LR: 1e-5 (lower than pretrain) ``` ### Stage 2b: Cold-Start Reasoning ``` Objective: Chain-of-thought before RL 5000 examples per epoch 3 epochs total ``` ### Stage 3: MCPO RL (CPU-Simplified) ``` Note: Full MCPO requires code execution for rewards. On CPU: Use format-based rewards (structural token statistics). RL steps: 500-2000 (lighter than GPU target of 20K) ``` ### Stage 4: DPO Preference Alignment ``` Pairwise preferences from execution results 1000-5000 steps LR: 5e-7 (very small) ``` ### Stage 5: Long-Context Adaptation ``` Context: 128 -> 256 -> 512 (CPU practical limit) RoPE base frequency adjustment ``` ## ====== MONITORING & CHECKPOINTING ====== ### Logging ```python # Every N steps, log to console + file: { 'step': int, 'loss': float, # Current loss 'avg_loss': float, # Moving average (100 steps) 'lr': float, # Current learning rate 'grad_norm': float, # Gradient L2 norm 'tokens_per_sec': float, 'elapsed_hours': float, 'loss_history': List[float], # All losses for plotting } ``` ### Checkpoint Format ```python { 'step': int, 'model': model.state_dict(), 'optimizer': optimizer.state_dict(), 'scheduler': scheduler.state_dict(), 'loss': float, 'config': model.config, 'args': training_args, } ``` ### Recovery - If training is interrupted, resume from latest checkpoint - Verify checkpoint integrity with checksums - Keep last 3 checkpoints (disk permitting) ## ====== BENCHMARK GATES ====== ### Training Progress Checkpoints ``` Step Expected Loss Notes ────── ───────────── ────────────────────── 0 10.4-10.5 Random init (ceiling) 1K ~8.0 Word-level patterns learned 5K ~6.0 Syntax acquired 10K ~5.0 Basic code structure 25K ~4.0 Functional patterns emerging 50K ~3.5 Approaching reasonable completion 100K ~3.0 Good language model ``` ### Evaluation on CPU Subset ``` Small HumanEval subset (5-10 problems): pass@1: Should improve from 0% -> 5-10% after 50K steps Target for 4K model on CPU: ~5% (limited by model capacity) Note: The 4K model is a proof of concept. Full 800M training requires cloud GPUs (see GOLD_STANDARD_TRAINING.md). ``` ## ====== COMPLETE LAUNCH COMMAND ====== ```bash # Quick smoke test (10 steps): timeout 60 python3 -c " import sys, os, torch, time os.environ['TOKENIZERS_PARALLELISM'] = 'false' sys.path.insert(0, '/FSI_Edge') torch.set_num_threads(8) from src.model import FSIEdgeModel, FSIEdgeConfig from src.data import CodeDataset, collate_fn from torch.utils.data import DataLoader from torch.optim import AdamW ds = CodeDataset('/FSI_Edge/data/train', '/FSI_Edge/fsi_edge_tokenizer', max_length=128) loader = DataLoader(ds, batch_size=1, shuffle=True, collate_fn=collate_fn) config = FSIEdgeConfig(d_model=64, n_layers=2, n_heads=4, kv_heads=2, d_ff=256, max_seq_len=128, window_size=32, local_heads=2, struct_heads=1, global_heads=1) model = FSIEdgeModel(config) opt = AdamW(model.parameters(), lr=3e-4) for step, batch in enumerate(loader): if step >= 10: break out = model(**{k:v.to('cpu') for k,v in batch.items()}) out.loss.backward(); opt.step(); opt.zero_grad() print(f'step {step+1} loss {out.loss.item():.4f}') " # Sustained training: # (Run with nohup or tmux) python3 /FSI_Edge/training/train.py \ --model-size 4K \ --batch-size 1 \ --max-steps 100000 \ --device cpu \ --stages stage1,stage1b,stage2,stage2b,stage3,stage3b,stage4,stage5 \ --output-dir /FSI_Edge/output \ --no-wandb ``` ## ====== DATA FLOW DIAGRAM ====== ``` Raw Templates → DataFilterPipeline → JSONL shards → CodeDataset ↓ ColdStartGenerator → cold_start.jsonl → ColdStartDataset ↓ All datasets → DataLoader (num_workers=4) → collate_fn → model(**batch) ↓ loss → backward → optimizer.step ↓ checkpoint (every 5000 steps) ↓ evaluation → GGUF export ``` ## ====== THE MISSION ====== Train a from-scratch novel architecture model on ARM CPU. Each step proves the architecture works. Each checkpoint is a milestone. The code is production-ready. The cloud path is clear. This CPU run validates the pipeline end-to-end before scaling to H100s. No excuses. No shortcuts. Just the best engineering.