gpt2small-en-it-nanochat-lr2e4-bs6-wsd-shortfastdecay11k-final5e6-webwiki-step7000

This repo stages step_7000.pt from the local NanoChat EN/IT GPT-2-small-like WSD short-fast-decay web/wiki run 20260603_fresh-gpt2small-lr2e4-bs6-wsd-shortfastdecay11k-final5e6-webwiki.

What this is

• model family: GPT-2-small-like decoder-only LM
• parameters: ~136M
• languages: English + Italian
• context length: 2500
• selected checkpoint: step_7000.pt
• estimated tokens seen at this checkpoint: ~1.68B
• selection reason: near-top early checkpoint intentionally published for direct comparison with the final short-fast winner
• status relative to comparable public checkpoints: secondary comparison checkpoint from the same run family; the recommended winner remains step_11000

Best in-run validation

• best saved validation step for the run: 11000
• validation loss: 3.8305816711
• validation perplexity: 46.0893392775
• validation batches: 128
• this release checkpoint does not match the best online validation checkpoint

Important caveat: this repo is intentionally a comparison release, not the final recommended checkpoint from the run.

Benchmark summary

Repo-native benchmark suite: configs/eval/20260521_pretrain_minimal_en_it_webwiki_step11000.yaml

Release checkpoint metrics:

• val_loss_mixed: 5.2277
• ppl_mixed: 186.3665
• val_loss_en: 5.0606
• ppl_en: 157.6905
• val_loss_it: 4.0280
• ppl_it: 56.1479
• loop_rate: 0.600
• repeated_4gram_rate: 0.925
• distinct_2: 0.4338
• cloze_en_contains: 0.00
• cloze_it_contains: 0.08

Ranking inside the checked saved checkpoints from this run:

1. step_11000 -> mixed=5.2270
2. step_7000 -> mixed=5.2277
3. step_10000 -> mixed=5.2677
4. step_8000 -> mixed=5.3010
5. step_9000 -> mixed=5.3963

Cross-run reading

Comparable champion still ahead:

• champion: 20260530_fresh-gpt2small-lr2e4-bs6-wsd-earlydecay7000-final5e6-webwiki step_7000
• champion repo: https://huggingface.co/nazdef/gpt2small-en-it-nanochat-lr2e4-bs6-wsd-earlydecay7000-final5e6-webwiki-step7000
• champion val_loss_mixed: 5.2158
• this release val_loss_mixed: 5.2277
• delta vs champion: +0.0119

Operational reading:

• step_11000 is the best checkpoint of the short-fast-decay run
• it is also the most stable-looking checkpoint among the recent short-fast/new candidates in this tracked slice
• the overall crown still stays with earlydecay7000 step_7000
• this release is therefore comparative and useful, not a claim that the benchmark champion has changed

Source/domain losses for this checkpoint

• source_loss_books_en: 4.7184
• source_loss_books_it: 4.8091
• source_loss_code: 8.3822
• source_loss_web_en: 6.0977
• source_loss_web_it: 6.1020
• source_loss_wiki_en: 4.1120
• source_loss_wiki_it: 3.9270

Probe reading at this checkpoint

• EN factual prompt The capital of Italy is -> Rome: rank=152, prob=0.0009003
• EN procedural prompt A small language model should -> be: rank=1, prob=0.5898438
• IT factual prompt La capitale d'Italia è -> Roma: rank=820, prob=0.0000634
• IT procedural prompt Un piccolo modello linguistico dovrebbe -> essere: rank=1, prob=0.2177734

These probes are directional evidence only. The main selection rule here remains the repo-native benchmark result and the cross-run comparison context above.

Training/data provenance

• training config: training_config.yaml
• tokenizer: tokenizer.json + tokenizer_meta.json
• packed dataset root used by the run: /mnt/apps/llm-nanochat/datasets/202605141153_fineweb50_wiki50_50en_50it_score100_2500context_5Btokens_tok_20260515_en50it50_webwiki_stratified_500M
• tokenizer root used by the run: /mnt/apps/llm-nanochat/tokenizers/tokenizer_20260515_en50it50_webwiki_stratified_500M
• source commit for release packaging: 032e34b8ae96bba41f37978f2e311e9b98ab7fa5
• estimated tokens seen at this checkpoint: ~1.68B

Included files

• step_7000.pt
• step_7000.safetensors
• step_7000.safetensors.json
• training_config.yaml
• tokenizer.json
• tokenizer_meta.json
• best_validation.json
• eval_summary.json
• comparison.json
• benchmark_report.md
• benchmark_metrics.json
• benchmark_scores.json
• benchmark_source_losses.json
• probe_step_7000_summary.json
• eval_metrics.jsonl
• metrics.jsonl
• probe_generations.jsonl
• release note: 2026-06-06_shortfastdecay11k_release_step7000.md

Usage

This project uses a custom NanoChat inference/training stack. The easiest local UI in the source repo is the Chainlit checkpoint tester documented in the repo README.

Limitations

• mixed quality is still in the weak/intermediate band
• generations remain repetitive and often unstable under free-form continuation
• factual recall is still weak in both languages
• the overall comparable benchmark champion is still earlydecay7000 step_7000
• dataset redistribution for the full training corpus may have separate licensing constraints; this repo contains model artifacts, not the raw/prepared training corpus