gpt2small-en-it-nanochat-lr2e4-bs6-wsd-fastdecay-webwiki-step11000

This repo stages the best saved checkpoint from the local NanoChat EN/IT GPT-2-small-like run 202605181849_stable-config-recipe-v5-gpt2small-lr2e4-batchmaxpossible-bs6-wsd-fastdecay-webwiki.

What this is

• model family: GPT-2-small-like decoder-only LM
• parameters: ~136M
• languages: English + Italian
• context length: 2500
• selected checkpoint: step_11000.pt
• selection reason: lowest recorded validation loss among saved checkpoints in best_validation.json

Best validation

• step: 11000
• validation loss: 3.7920523576
• validation perplexity: 44.3473235090
• validation batches: 128

Why this checkpoint

This run improved cleanly through step_11000, then degraded sharply:

• step_12000: validation loss 4.1425
• step_13000: validation loss 4.6568
• step_14000: validation loss 4.3794
• step_15000: validation loss 8.2670
• step_16000: validation loss 8.1599

So this release intentionally publishes step_11000.pt instead of the latest saved checkpoint. The later tail is not "more trained"; it is materially worse on validation.

Important caveat

The collapse pattern is real in the telemetry, but the exact root cause is still an operator analysis, not a formal proof. The most plausible reading is that this web/wiki recipe spent too long at peak_lr=2e-4 under a very long WSD plateau and eventually destabilized on this dataset/tokenizer combination.

Training/data provenance

• training config: training_config.yaml
• tokenizer: tokenizer.json + tokenizer_meta.json
• packed dataset root used by the run: /mnt/apps/llm-nanochat/datasets/202605141153_fineweb50_wiki50_50en_50it_score100_2500context_5Btokens_tok_20260515_en50it50_webwiki_stratified_500M
• tokenizer root used by the run: /mnt/apps/llm-nanochat/tokenizers/tokenizer_20260515_en50it50_webwiki_stratified_500M

Included files

• step_11000.pt
• step_11000.safetensors
• step_11000.safetensors.json
• training_config.yaml
• tokenizer.json
• tokenizer_meta.json
• best_validation.json
• eval_summary.json
• probe_step11000_summary.json
• full run telemetry snapshots: eval_metrics.jsonl, metrics.jsonl, probe_generations.jsonl

Probe reading at step 11000

The run does include usable probe telemetry at step 11000. The exported JSON is a bit awkward because the useful fields live inside nested diagnostics rather than the flat keys used by some older summaries.

• EN factual prompt The capital of Italy is -> Rome: weak but improved (rank=20, prob=0.0061646)
• EN simple continuation A small language model should -> be: strong (rank=1, prob=0.4258)
• IT factual prompt La capitale d'Italia è -> Roma: still weak (rank=57, prob=0.0022430)
• IT simple continuation Un piccolo modello linguistico dovrebbe -> essere: strong (rank=1, prob=0.5234)

These probes are useful as directional evidence, but they do not change the main selection rule: this checkpoint is published because it is the best validation checkpoint in the run.

Usage

This project uses a custom NanoChat inference/training stack. The easiest local UI in the source repo is the Chainlit checkpoint tester documented in the repo README.

Limitations

• factual recall is still limited
• generations may become repetitive
• the model was selected by validation loss inside this run family, not by broad downstream benchmark performance
• dataset redistribution for the full training corpus may have separate licensing constraints; this repo contains model artifacts, not the raw/prepared training corpus