🧠 I trained my own French LLM from scratch — alone, with a 1080 Ti, and the power went out ⚡🇫🇷

---

👤 Context — Who does this and why

20 years old, solo, GTX 1080 Ti 11GB. No team, no cloud budget, no supervisor.

The real question is: why not grab a HuggingFace model + LoRA like everyone else? Short answer: because I wanted to understand every single step. Not fine-tune a black box. Actually understand it.

So I built the entire pipeline myself:

• 🕷️ Collection — my crawler RDTvlokipBot
• 🧹 Cleaning — RDTextract, my custom HTML→Markdown extractor
• 🔤 Tokenization — ByteLevel BPE tokenizer from scratch (32k vocab)
• 🏗️ Architecture — GPT-2 like with zero HuggingFace dependency
• 🎓 Training — custom multi-phase trainer

4 months building the dataset before touching the model. The complete pipeline, fully mastered — from the first byte to the generated token.

---

🏗️ Architecture — LLaMA-style, not vanilla GPT-2

Final choices

After going through 7 different sizes (7.68M → 13.3M → 44M → 109M → 12.5M → 4M → 15M final), here's what stuck:

config = {
    "n_embd": 256,
    "n_layer": 8,
    "n_head": 4,
    "head_dim": 64,        # Multiple of 64, keeping it clean
    "vocab_size": 32000,   # Custom BPE 32k
    "context_length": 512,
    "n_params": "~15M"
}

Why 15M? Chinchilla ratio. The Chinchilla law says an optimal model sees ~20 tokens per parameter. With 271M tokens of dataset: 271M / 15M = 18.1 tokens/param. Right in the zone. Not random.

Modern architecture — not old-school GPT-2

This is where it really differs from vanilla GPT-2:

Bottom line: LLaMA-style architecture, not GPT-2 style. The "GPT-2 like" label refers to the approach (decoder-only autoregressive) — not the internal components.

RoPE — why this choice 🌀

RoPE encodes positions via rotation in vector space rather than adding a fixed vector. The relationship between two tokens becomes a function of their relative distance, not their absolute position.

def rotary_embedding(x, cos, sin):
    x1 = x[..., :x.shape[-1] // 2]
    x2 = x[..., x.shape[-1] // 2:]
    rotated = torch.cat([-x2, x1], dim=-1)
    return x * cos + rotated * sin

# rope_theta = 10000.0
# rope_scaling = null (no extended context)

Concrete advantages:

• ✅ No learned positional parameters (fewer params, same performance)
• ✅ Better generalization on long sequences
• ✅ Standard used by LLaMA, Mistral, GPT-NeoX, Gemma

Might as well start with the right standard from day one.

SwiGLU — the detail that changes param count

SwiGLU modifies the internal FFN. The n_inner auto-adjusts to compensate:

# SwiGLU: effective n_inner = 768 instead of 1024
# Maintains parameter parity vs a classic FFN
n_inner = int(n_embd * 4 * 2/3)  # ~682 → rounded to 768

---

📚 The dataset — French Wikipedia rewritten by AI

Not the raw dump

Raw French Wikipedia is clean but encyclopedically rigid — wikitexte syntax, templates, uneven structure. Bad for training an LLM that needs to generate fluid text.

Approach chosen: rewrite every article with an AI into a structured, pedagogical, uniform markdown style. With one hard constraint: "don't add anything you don't know" — facts are preserved (distances, dates, structures), only the style changes.

Result after correct_markdown.py (apostrophe normalization, indentation, tables):

152,948 .txt files
1.09 billion characters
271M tokens

Custom tokenizer — not GPT-2's

The original GPT-2 tokenizer was trained on English. On French, one word often becomes 2-3 tokens instead of one. Suboptimal.

Custom ByteLevel BPE tokenizer from scratch:

Vocab size        : 32,000 tokens
Coverage          : 99% (31,671 / 32,000 unique tokens used)
Chars/token ratio : 3.997  ← optimal for French
UNK rate          : 0%

This work on tokenization is also what eventually led to AG-BPE (Attention-Guided BPE) — but that's another story.

Technical preprocessing

# Chunking with overlap to maximize samples
chunk_size = 512
stride = 400       # 112-token overlap between chunks
# → 450 useful tokens + 50 shared context

# Train/val split with shuffle
# seed=42, avoids alphabetical bias from Wikipedia filenames
# Multi-processing numpy int32 → ~85% RAM saved vs Python lists

---

⚙️ Training — 3 phases, not 1

Multi-phase pipeline (research-level, personal project)

Rather than a classic CLM on all data at once, 18 epochs total across 3 phases:

Phase 1 — Denoising CLM (3 epochs) 15% token corruption → the model learns to reconstruct. Forces robust representations before optimizing pure generation.

Phase 2 — Curriculum CLM (10 epochs) 5 cumulative length buckets. Start on short sequences, progressively expand. The model ramps up in difficulty gradually.

Phase 3 — CLM + Contrastive SimCSE-style (5 epochs) Dropout as data augmentation → two views of the same text → contrastive loss. 2 forward passes per batch = 2x slower, but much richer representations.

training_config = {
    "optimizer": "AdamW",
    "learning_rate": 3e-4,
    "lr_scheduler": "cosine_with_warmup",
    "warmup_steps": 1000,
    "weight_decay": 0.1,
    "grad_clip": 1.0,
    "precision": "fp16",
    "gradient_checkpointing": True,
    "flash_attention": True      # SDPA
}

The bugs — there were many

Training from scratch on your own machine means debugging constantly:

• 🐛 torch.cuda.amp deprecated → torch.amp
• 🐛 autocast missing device_type='cuda'
• 🐛 Weight tying applied before weight init → embeddings overwritten
• 🐛 Critical bug: causal mask broken with Flash Attention when attention_mask provided → the model saw the future during training. Artificially low PPL of 1.13. Generation looping "ïsïsïs" / "Sen Sen Sen". Hours of debugging.
• 🐛 Critical bug 2: enable_padding() permanently enabled → dataset ballooned to 30 GB instead of 3 GB. 8.5M padding chunks instead of 700k real chunks. ignore_index=0 missing from the loss.

Every bug found and fixed. GPU temp: <70°C throughout. 4.5 steps/s average. ~3h per epoch in phase 1.

---

⚡ The power outage — epoch 10, bucket 4/5

Here's the honest moment.

The final run was live: 15M params, 271M tokens, 18 epochs planned. Epoch 10 running, curriculum bucket 4/5. Stats at the time of the outage:

Epoch      : 10 / 18
Train loss : 2.8991
Val loss   : 2.8829
PPL        : 17.87
Bucket     : 4/5
Estimated time remaining : ~43h

Power cuts. Brutal.

What saved everything: automatic checkpoint at each epoch. Zero real progress lost — just the inability to finish the remaining 8 epochs (bucket 5 + 5 contrastive epochs).

Verdict: "The power went out during training, no big deal — I think the model is already well trained." Direct test with epoch 10 checkpoint.

---

🔮 What's next — Lambda Labs + expanded dataset

Why Lambda Labs

Vast.ai and runpod.io rejected — those are individuals renting their GPU. Unreliable, no real guarantees.

Real ML cloud comparison:

Estimated cost for the full run on A100: ~€4 instead of 35-50h on a 1080 Ti. And no more power outages.

Why expand the dataset before relaunching

271M tokens hits the Chinchilla sweet spot for 15M params. But generation tests are honest:

The model learns form (markdown, grammar, style) but not substance (strong conceptual associations). The only truly coherent prompt: "The solar system" — massively overrepresented in Wikipedia through thousands of asteroid/planet articles.

One data style = one-style model. Target sources for v2:

• 📝 French tech blogs (developpez.com, linuxfr.org)
• 💬 Forums (forum.ubuntu-fr.org, Stack Overflow FR)
• 📚 Project Gutenberg FR (public domain literature)
• 📖 French technical docs (Python, Django...)

Goal: 1B clean tokens, multi-style, quality 9/10. Not to beat Mistral. To become the best small French model on a specific domain.

---

✅ Advantages

• 🔬 100% mastered pipeline — collection → cleaning → tokenization → training → generation
• 🌀 Modern architecture — RoPE, RMSNorm, SwiGLU, Flash Attention. Not 2019 GPT-2.
• 🇫🇷 Native French focus — FR-optimized tokenizer, AI-rewritten FR dataset
• 📐 Chinchilla ratio respected — 18.1 tokens/param, justified choice
• 🎓 3-phase pipeline — research-level on a solo personal project

❌ Disadvantages

• 💸 Limited local compute — 1080 Ti, not a cluster
• 🐌 Contrastive phase = 2x slower — 2 forward passes per batch
• 📉 Wikipedia only = single style — model hallucinates outside its comfort zone
• ⚡ Power outage risk — lived experience

⚠️ Limits

• 15M params is experimental — GPT-2 small is already 117M
• Without instruction fine-tuning it's a pure completion model
• Dataset diversity will directly dictate the quality of conceptual associations

---

📋 Quick reference

---

🗂️ Summary

A 15M parameter French LLM, built from zero solo on a 1080 Ti. LLaMA-style architecture (RoPE, RMSNorm, SwiGLU, Flash Attention), custom ByteLevel BPE tokenizer optimized for French, French Wikipedia dataset rewritten by AI for style uniformity. 3-phase training pipeline (denoising → curriculum → contrastive). Interrupted at epoch 10/18 by a power outage at 273M tokens. The model learns form but not yet substance — next run on Lambda Labs with a multi-source dataset.

---

🏁 Conclusion

This project isn't a compute flex — it's the opposite. It's proving you can build the entire chain from scratch, alone, with consumer hardware. Every bug debugged, every architectural choice justified, every byte of the dataset controlled. The power outage at 273M tokens is just an anecdote — the real learning happened over the 4 months before. Next stop: Lambda Labs with an expanded dataset. The goal hasn't changed: the best small French model on a specific domain. 🚀

---

❓ Q&A

— Why not use a pre-trained model + LoRA like everyone else?

Because LoRA on an existing model means adapting a black box. The goal here was to understand every component — why RoPE over absolute embeddings, why SwiGLU over GELU, how curriculum affects loss curves. You don't learn that by fine-tuning Mistral.

— Is the 3-phase pipeline actually useful at 15M params?

Honestly, hard to measure without a full ablation study. What's certain: the denoising phase forces robust representations early, and SimCSE-style contrastive learning at this scale stays rare in personal projects. Maybe over-engineered for 15M — but learning was also the objective.

— How will the model be evaluated on the next run?

PPL on a multi-source val set (not just Wikipedia), generation tests on out-of-domain prompts, and a before/after comparison after dataset diversification. The goal is to see if mixing 5-6 source styles actually changes the coherence of conceptual associations.

---

💡 Did you know?

The Chinchilla law (DeepMind, 2022) completely changed LLM training practices. Before it, the trend was to maximize model size with a fixed compute budget. Chinchilla proved that a smaller model trained on more tokens systematically beats a larger undertrained one. GPT-3 (175B params, 300B tokens) was significantly undertrained by these standards. LLaMA 1 was the first major public model to truly apply this logic — and it's exactly this principle that guided the 15M param choice in this project.

---

Théo CHARLET

IT Systems & Networks Student - AI/ML Specialization

Creator of AG-BPE (Attention-Guided Byte-Pair Encoding)

🔗 LinkedIn: https://www.linkedin.com/in/théo-charlet

🚀 Seeking internship opportunities

🔗 Website : https://rdtvlokip.fr