gpt2small-en-it-nanochat-lr2e4-bs6-wsd-shortfastdecay11k-final5e6-webwiki-step11000

This repo stages step_11000.pt from the local NanoChat EN/IT GPT-2-small-like WSD short-fast-decay web/wiki run 20260603_fresh-gpt2small-lr2e4-bs6-wsd-shortfastdecay11k-final5e6-webwiki.

What this is

• model family: GPT-2-small-like decoder-only LM
• parameters: ~136M
• languages: English + Italian
• context length: 2500
• selected checkpoint: step_11000.pt
• estimated tokens seen at this checkpoint: ~2.64B
• selection reason: best full repo-native CPU benchmark result among the checked saved checkpoints from this run family
• status relative to comparable public checkpoints: best and most stable recent short-fast candidate from this run family, but still slightly behind the overall comparable champion

Best in-run validation

• best saved validation step for the run: 11000
• validation loss: 3.8305816711
• validation perplexity: 46.0893392775
• validation batches: 128
• this release checkpoint matches the best online validation checkpoint

Important caveat: this label is intentionally narrow. It means the best and most stable recent short-fast candidate in this tracked slice, not the overall benchmark champion and not a claim of polished generation quality.

Benchmark summary

Repo-native benchmark suite: configs/eval/20260521_pretrain_minimal_en_it_webwiki_step11000.yaml

Release checkpoint metrics:

• val_loss_mixed: 5.2270
• ppl_mixed: 186.2382
• val_loss_en: 4.9931
• ppl_en: 147.3916
• val_loss_it: 4.0279
• ppl_it: 56.1452
• loop_rate: 0.400
• repeated_4gram_rate: 0.900
• distinct_2: 0.4990
• cloze_en_contains: 0.02
• cloze_it_contains: 0.10

Ranking inside the checked saved checkpoints from this run:

1. step_11000 -> mixed=5.2270
2. step_7000 -> mixed=5.2277
3. step_10000 -> mixed=5.2677
4. step_8000 -> mixed=5.3010
5. step_9000 -> mixed=5.3963

Cross-run reading

Comparable champion still ahead:

• champion: 20260530_fresh-gpt2small-lr2e4-bs6-wsd-earlydecay7000-final5e6-webwiki step_7000
• champion repo: https://huggingface.co/nazdef/gpt2small-en-it-nanochat-lr2e4-bs6-wsd-earlydecay7000-final5e6-webwiki-step7000
• champion val_loss_mixed: 5.2158
• this release val_loss_mixed: 5.2270
• delta vs champion: +0.0112

Operational reading:

• step_11000 is the best checkpoint of the short-fast-decay run
• it is also the most stable-looking checkpoint among the recent short-fast/new candidates in this tracked slice
• the overall crown still stays with earlydecay7000 step_7000
• this release is therefore comparative and useful, not a claim that the benchmark champion has changed

Source/domain losses for this checkpoint

• source_loss_books_en: 4.9168
• source_loss_books_it: 4.8002
• source_loss_code: 7.8108
• source_loss_web_en: 5.8082
• source_loss_web_it: 6.2641
• source_loss_wiki_en: 4.0474
• source_loss_wiki_it: 3.7621

Probe reading at this checkpoint

• EN factual prompt The capital of Italy is -> Rome: rank=29, prob=0.0038452
• EN procedural prompt A small language model should -> be: rank=1, prob=0.4707031
• IT factual prompt La capitale d'Italia è -> Roma: rank=222, prob=0.0004597
• IT procedural prompt Un piccolo modello linguistico dovrebbe -> essere: rank=1, prob=0.3085938

These probes are directional evidence only. The main selection rule here remains the repo-native benchmark result and the cross-run comparison context above.

Training/data provenance

• training config: training_config.yaml
• tokenizer: tokenizer.json + tokenizer_meta.json
• packed dataset root used by the run: /mnt/apps/llm-nanochat/datasets/202605141153_fineweb50_wiki50_50en_50it_score100_2500context_5Btokens_tok_20260515_en50it50_webwiki_stratified_500M
• tokenizer root used by the run: /mnt/apps/llm-nanochat/tokenizers/tokenizer_20260515_en50it50_webwiki_stratified_500M
• source commit for release packaging: 032e34b8ae96bba41f37978f2e311e9b98ab7fa5
• estimated tokens seen at this checkpoint: ~2.64B

Included files

• step_11000.pt
• step_11000.safetensors
• step_11000.safetensors.json
• training_config.yaml
• tokenizer.json
• tokenizer_meta.json
• best_validation.json
• eval_summary.json
• comparison.json
• benchmark_report.md
• benchmark_metrics.json
• benchmark_scores.json
• benchmark_source_losses.json
• probe_step_11000_summary.json
• eval_metrics.jsonl
• metrics.jsonl
• probe_generations.jsonl
• release note: 2026-06-06_shortfastdecay11k_release_step11000.md

Usage

This project uses a custom NanoChat inference/training stack. The easiest local UI in the source repo is the Chainlit checkpoint tester documented in the repo README.

Limitations

• mixed quality is still in the weak/intermediate band
• generations remain repetitive and often unstable under free-form continuation
• factual recall is still weak in both languages
• the overall comparable benchmark champion is still earlydecay7000 step_7000
• dataset redistribution for the full training corpus may have separate licensing constraints; this repo contains model artifacts, not the raw/prepared training corpus