| ---
|
| language: fr
|
| license: apache-2.0
|
| tags:
|
| - text-generation
|
| - from-scratch
|
| - looped-transformer
|
| - adaptive-computation
|
| - french
|
| pipeline_tag: text-generation
|
| ---
|
|
|
| # Cadence-15M-fr 🔁
|
|
|
| > ⚠️ **Preliminary results — 1 seed, variance not controlled.** Every number below is a single training run evaluated once (50 prompts × 200 tokens, fixed seed). Directions are consistent, magnitudes are not validated. Treat as a lab notebook, not a benchmark.
|
|
|
| **Cadence** is a 15M-parameter French language model trained **from scratch** (no 🤗 Transformers, no pretrained weights) on a single GTX 1080 Ti. It is the **looped / recurrent-depth** reference of a three-model family — the plain loop, without adaptive halting.
|
|
|
| ## Identity
|
|
|
| - **Architecture:** decoder-only, LLaMA-style (RoPE, RMSNorm, SwiGLU, QK-Norm, Flash/SDPA attention), **looped**: the 8 transformer blocks are applied **R=4** times (effective depth 32) with **zero added parameters** beyond a small per-iteration depth embedding. Zero-init residual projections for stable unrolling.
|
| - **Size:** 15.01M parameters · `n_embd 256 / n_layer 8 / n_head 4` · context 768.
|
| - **Language:** French, from scratch.
|
| - **Tokenizer:** custom ByteLevel BPE, 32k vocab (`bpe_tokenizer_32k.json`).
|
| - **Data:** French corpus (AI-rewritten Wikipedia + filtered web via RDTextract), ~425M tokens total. **These runs used 20% of the train split, 2 epochs** (fast research protocol).
|
|
|
| ## What Cadence is good at
|
|
|
| Cadence is the **perplexity reference**: the plain loop gives the cleanest hard-metric win of the family.
|
|
|
| - **Val perplexity 28.9** (vs 31.2 baseline, −7%) — the best of the four models.
|
| - Better in-domain coherence and fewer invented names than the vanilla baseline.
|
| - Trade-off: it is **weaker out-of-domain** than the baseline (see table).
|
|
|
| ## Robust evaluation (50×200, 1 seed)
|
|
|
| Same table on all three model cards, so the trade-off is visible everywhere. Auto = held-out corpus prompts (in-domain); Fixed = generic hand-written prompts (out-of-domain).
|
|
|
| | Model | Val PPL ↓ | Coherence auto ↑ | Coherence fixed ↑ | Invented names auto ↓ | Prompt overlap auto ↑ |
|
| |---|---|---|---|---|---|
|
| | Baseline (vanilla) | 31.2 | 35.2 | 40.7 | 0.137 | 0.139 |
|
| | **Cadence** (looped R=4) | **28.9** | 39.3 | 32.5 | 0.121 | 0.147 |
|
| | Focal (absolute halt) | 29.1 | **44.1** | 29.1 | 0.103 | **0.176** |
|
| | Nomade (percentile halt) | 31.0 | 36.3 | **41.8** | **0.094** | 0.126 |
|
|
|
| **No model wins everything — that is the result.** Recurrence helps in-domain and hurts out-of-domain; the halting variants trade one for the other. Factuality is *not* improved by any of them (15M capacity ceiling).
|
|
|
| ## Lineage (this is not novel)
|
|
|
| - **ACT** — Adaptive Computation Time, Graves 2016 · [arXiv:1603.08983](https://arxiv.org/abs/1603.08983)
|
| - **Universal Transformer** — Dehghani et al. 2018 (the looped/weight-shared ancestor) · [arXiv:1807.03819](https://arxiv.org/abs/1807.03819)
|
| - **Recurrent-Depth Transformer** — Geiping et al. 2025 (Huginn-3.5B; the canonical RDT this family follows) · [arXiv:2502.05171](https://arxiv.org/abs/2502.05171)
|
| - **CALM** — Confident Adaptive Language Modeling, Schuster et al. 2022 · [arXiv:2207.07061](https://arxiv.org/abs/2207.07061)
|
| - **LoopViT** — Shu et al. 2026 (parameter-free entropy exit + weight-tied loop) · [arXiv:2602.02156](https://arxiv.org/abs/2602.02156)
|
|
|
| Direct basis — the two from-scratch looped studies this work reproduces in French: **Kohli et al. 2026** ([arXiv:2604.07822](https://arxiv.org/abs/2604.07822), COLM) and **Chen 2026** ([arXiv:2603.21676](https://arxiv.org/abs/2603.21676)).
|
|
|
| Cadence is a **Universal-Transformer-style looped transformer** applied to French at 15M. No claim of novelty.
|
|
|
| ## Limitations
|
|
|
| - **1 seed, no variance control** — magnitudes may reorder with more seeds.
|
| - **20% of data, 2 epochs** — a short research protocol, not a full training run.
|
| - **15M capacity** — hallucinations are expected; the model learns *form and coherence*, not *facts*. It will not reliably tell you the capital of France.
|
| - No instruction tuning — a pure completion model.
|
|
|
| ## Related
|
|
|
| - ✍️ Write-up (Article 3): *Teaching a 15M French LLM to think deeper* — [Hugging Face blog](https://huggingface.co/blog/RDTvlokip/teaching-a-15m-french-llm-to-think-deeper)
|
| - 🧪 **Focal** (in-domain variant): [RDTvlokip/Focal-15M-fr](https://huggingface.co/RDTvlokip/Focal-15M-fr)
|
| - 🧭 **Nomade** (out-of-domain variant): [RDTvlokip/Nomade-15M-fr](https://huggingface.co/RDTvlokip/Nomade-15M-fr)
|
|
|
| ## Usage
|
|
|
| This model uses **custom from-scratch code** (not 🤗 Transformers). The code ships with the repo (`model/`, `utils/`), so a snapshot download is self-contained. Requires `torch`, `huggingface_hub`, `tokenizers`.
|
|
|
| ```python
|
| import sys, torch
|
| from huggingface_hub import snapshot_download
|
|
|
| repo = snapshot_download("RDTvlokip/Cadence-15M-fr") # code + weights + tokenizer
|
| sys.path.insert(0, repo)
|
| from model.gpt2 import GPT2
|
| from utils.tokenizer import GPT2Tokenizer
|
|
|
| model = GPT2.from_pretrained(f"{repo}/best_model.pt", device="cuda"); model.eval()
|
| tok = GPT2Tokenizer(f"{repo}/bpe_tokenizer_32k.json")
|
|
|
| ids = tok.encode("La capitale de la France est", add_special_tokens=True)
|
| if ids and ids[-1] == tok.eos_token_id:
|
| ids = ids[:-1] # drop trailing <eos> (keeps it on-topic)
|
| out = model.generate(
|
| torch.tensor([ids], device="cuda"),
|
| max_length=80, temperature=0.8, top_k=40, top_p=0.9,
|
| repetition_penalty=1.3, eos_token_id=tok.eos_token_id,
|
| )
|
| print(tok.decode(out[0].tolist(), skip_special_tokens=True))
|
| ```
|
|
|
| **Théo CHARLET** — TSSR Graduate, AI/ML · Creator of AG-BPE · [rdtvlokip.fr](https://rdtvlokip.fr)
|
|
|
