---
license: mit
language:
- en
pipeline_tag: text-generation
tags:
- code-generation
- from-scratch
- novel-architecture
- helix-memory
- cpu-training
---

# FSI_Edge: From-Scratch Novel Architecture Coding Model

A tiny but capable code generation model trained from scratch on ARM CPU, with a novel DNA-inspired architecture.

## Architecture

- **Helix Memory** — DNA helix-inspired curved memory for O(log L) context scaling
- **HCA** (Hybrid Concentrated Attention) — 3-tier code attention (local + structural + global)
- **EA-FFN** (Execution-Augmented FFN) — learns execution traces
- **RoPE-S** — RoPE with structural bias for code structure
- **PPN** (Prefix-Preserving Norm) — stabilizes deep training
- **MoD** (Mixture-of-Depths) — dynamic routing to save compute

## Training Stages

1. **Stage 1** — Pretraining (next-token prediction on code + NLP)
2. **Stage 1b** — FIM (Fill-in-Middle code infilling)
3. **Stage 2** — SFT (Supervised Fine-Tuning)
4. **Stage 2b** — Cold-Start Reasoning (chain-of-thought)
5. **Stage 3** — MCPO RL (Monte Carlo Policy Optimization)
6. **Stage 4** — DPO (Direct Preference Optimization)
7. **Stage 5** — Long-Context Extension

## Quick Start

```bash
# Clone from HuggingFace
git clone https://huggingface.co/FerrellSyntheticIntelligence/FSI-Edge
cd FSI-Edge
pip install -r requirements.txt

# Train on CPU
python training/run_cpu.py --model-size 4K --steps 1000

# Resume training from checkpoint (step 4132)
python training/run_cpu.py --model-size 4K --steps 10000 \
  --resume checkpoints/cpu_ckpt_004132.pt --lr 2e-4

# Or on Colab T4 GPU (100x faster)
# Upload scripts/fsi_edge_colab.ipynb to Google Colab
```

## Checkpoints

`checkpoints/` contains trained checkpoints from ARM CPU training:
- `cpu_best.pt` — best model weights (19MB)
- `cpu_latest.pt` — latest model weights (19MB)
- `cpu_ckpt_004132.pt` — full training state (52MB, step 4132)

## Tokenizer

Trained BPE tokenizer (32K vocab) at `fsi_edge_tokenizer/`.

## Results

| Steps | Best Loss | Platform |
|-------|-----------|----------|
| 0     | 10.44     | ARM CPU  |
| 1000  | ~6.0      | ARM CPU  |
| 2000  | ~1.0      | ARM CPU  |
| 4132  | 0.70      | ARM CPU  |

## Colab Training

Open `scripts/fsi_edge_colab.ipynb` in Google Colab with T4 GPU for 100x faster training.

## Mission

Train a from-scratch novel architecture model. Each step proves the architecture.
The code is production-ready for cloud GPU scaling (H100s).