gpt2small-en-it-nanochat-lr3e4-bs6-wsd-shortfastdecay8k-final5e6-webwiki-step4000

This repo stages step_4000.pt from the local NanoChat EN/IT GPT-2-small-like WSD short-fast-decay web/wiki run 20260605_fresh-gpt2small-lr3e4-bs6-wsd-shortfastdecay8k-final5e6-webwiki.

What this is

• model family: GPT-2-small-like decoder-only LM
• parameters: ~136M
• languages: English + Italian
• context length: 2500
• selected checkpoint: step_4000.pt
• B tokens seen: ~0.96B
• selection reason: best full repo-native CPU benchmark result among the checked saved checkpoints from this run family
• status relative to comparable public checkpoints: current best comparable published checkpoint in this tracked GPT-2-small EN/IT web/wiki slice by val_loss_mixed

Best in-run validation

• best saved validation step for the run: 8000
• validation loss: 3.8823011749
• validation perplexity: 48.535776
• validation batches: 128
• this release checkpoint does not match the best online validation checkpoint

Important caveat: this run split into two different winners:

• step_8000 won the run's internal online validation
• step_4000 won the external repo-native benchmark used for checkpoint selection

Operationally, that means the run kept improving on its own internal validation while moving away from the best comparable checkpoint on the benchmark we actually use to rank releases.

Benchmark summary

Repo-native benchmark suite: configs/eval/20260521_pretrain_minimal_en_it_webwiki_step11000.yaml

Release checkpoint metrics:

• val_loss_mixed: 5.1440
• ppl_mixed: 171.3970
• val_loss_en: 4.7796
• ppl_en: 119.0571
• val_loss_it: 4.0479
• ppl_it: 57.2766
• loop_rate: 0.725
• repeated_4gram_rate: 0.900
• distinct_2: 0.4251
• cloze_en_contains: 0.00
• cloze_it_contains: 0.08

Ranking inside the checked saved checkpoints from this run:

1. step_4000 -> mixed=5.1440
2. step_7000 -> mixed=5.3313
3. step_5000 -> mixed=5.3651
4. step_8000 -> mixed=5.3930
5. step_6000 -> mixed=5.5364

Cross-run reading

This checkpoint takes the current comparable crown in the tracked public slice:

• previous comparable champion:
  • 20260530_fresh-gpt2small-lr2e4-bs6-wsd-earlydecay7000-final5e6-webwiki step_7000
  • repo: https://huggingface.co/nazdef/gpt2small-en-it-nanochat-lr2e4-bs6-wsd-earlydecay7000-final5e6-webwiki-step7000
  • val_loss_mixed = 5.2158
• this release:
  • val_loss_mixed = 5.1440
• delta vs previous champion:
  • -0.0718

Operational reading:

• step_4000 is the best checkpoint of the lr3e4 / 8k run under the repo-native benchmark
• it also beats the previous comparable public champion on the same primary metric
• however it is not the cleanest-looking checkpoint from its own run in terms of repetition/diversity behavior
• the final step_8000 is less loopy and more diverse on surface metrics, but loses clearly on the primary benchmark metric

So this release is a benchmark-first winner, not a claim that free-form generation quality is suddenly solved.

Source/domain losses for this checkpoint

• source_loss_books_en: 4.9947
• source_loss_books_it: 5.0274
• source_loss_code: 8.6152
• source_loss_web_en: 6.1538
• source_loss_web_it: 6.0197
• source_loss_wiki_en: 3.8654
• source_loss_wiki_it: 3.5642

Probe reading at this checkpoint

• EN factual prompt The capital of Italy is -> Rome: rank=2708, prob=0.0000197
• EN procedural prompt A small language model should -> be: rank=1, prob=0.34375
• IT factual prompt La capitale d'Italia è -> Roma: rank=559, prob=0.0001488
• IT procedural prompt Un piccolo modello linguistico dovrebbe -> essere: rank=1, prob=0.1904297

These probes are directional evidence only. The main selection rule here remains the repo-native benchmark result and the comparable cross-run ranking context above.

Training/data provenance

• training config: training_config.yaml
• tokenizer: tokenizer.json + tokenizer_meta.json
• packed dataset root used by the run: /mnt/apps/llm-nanochat/datasets/202605141153_fineweb50_wiki50_50en_50it_score100_2500context_5Btokens_tok_20260515_en50it50_webwiki_stratified_500M
• tokenizer root used by the run: /mnt/apps/llm-nanochat/tokenizers/tokenizer_20260515_en50it50_webwiki_stratified_500M
• source commit for release packaging: bffb58ef99b4bb27ea6772f5853c16d43607e4eb
• estimated tokens seen at this checkpoint: ~0.96B

Included files

• step_4000.pt
• step_4000.safetensors
• step_4000.safetensors.json
• training_config.yaml
• tokenizer.json
• tokenizer_meta.json
• best_validation.json
• eval_summary.json
• comparison.json
• benchmark_report.md
• benchmark_metrics.json
• benchmark_scores.json
• benchmark_source_losses.json
• probe_step_4000_summary.json
• eval_metrics.jsonl
• metrics.jsonl
• probe_generations.jsonl
• release note: 2026-06-07_shortfastdecay8k_release_step4000.md

Usage

This project uses a custom NanoChat inference/training stack. The easiest local UI in the source repo is the Chainlit checkpoint tester documented in the repo README.

Limitations

• generations remain weak, repetitive, and often unstable under free-form continuation
• this checkpoint wins on the main benchmark metric, but not on repetition/diversity surface metrics inside its own run
• factual probes remain weak in both languages
• dataset redistribution for the full training corpus may have separate licensing constraints; this repo contains model artifacts, not the raw/prepared training corpus