---
language:
- en
- it
license: other
library_name: custom
pipeline_tag: text-generation
tags:
- nanochat
- gpt2-small
- bilingual
- english
- italian
- pretraining
- webwiki
- wsd
- short-fast-decay
- benchmark-selected
- recent-best
- stable-recent-candidate
---

# gpt2small-en-it-nanochat-lr2e4-bs6-wsd-shortfastdecay11k-final5e6-webwiki-step11000

This repo stages `step_11000.pt` from the local NanoChat EN/IT GPT-2-small-like WSD short-fast-decay web/wiki run `20260603_fresh-gpt2small-lr2e4-bs6-wsd-shortfastdecay11k-final5e6-webwiki`.

## What this is

- model family: GPT-2-small-like decoder-only LM
- parameters: ~136M
- languages: English + Italian
- context length: 2500
- selected checkpoint: `step_11000.pt`
- estimated tokens seen at this checkpoint: `~2.64B`
- selection reason: best full repo-native CPU benchmark result among the checked saved checkpoints from this run family
- status relative to comparable public checkpoints: best and most stable recent short-fast candidate from this run family, but still slightly behind the overall comparable champion

## Best in-run validation

- best saved validation step for the run: `11000`
- validation loss: `3.8305816711`
- validation perplexity: `46.0893392775`
- validation batches: `128`
- this release checkpoint matches the best online validation checkpoint

Important caveat: this label is intentionally narrow. It means the best and most stable recent short-fast candidate in this tracked slice, not the overall benchmark champion and not a claim of polished generation quality.

## Benchmark summary

Repo-native benchmark suite: `configs/eval/20260521_pretrain_minimal_en_it_webwiki_step11000.yaml`

Release checkpoint metrics:

- `val_loss_mixed`: `5.2270`
- `ppl_mixed`: `186.2382`
- `val_loss_en`: `4.9931`
- `ppl_en`: `147.3916`
- `val_loss_it`: `4.0279`
- `ppl_it`: `56.1452`
- `loop_rate`: `0.400`
- `repeated_4gram_rate`: `0.900`
- `distinct_2`: `0.4990`
- `cloze_en_contains`: `0.02`
- `cloze_it_contains`: `0.10`

Ranking inside the checked saved checkpoints from this run:

1. `step_11000` -> `mixed=5.2270`
2. `step_7000` -> `mixed=5.2277`
3. `step_10000` -> `mixed=5.2677`
4. `step_8000` -> `mixed=5.3010`
5. `step_9000` -> `mixed=5.3963`

## Cross-run reading

Comparable champion still ahead:

- champion: `20260530_fresh-gpt2small-lr2e4-bs6-wsd-earlydecay7000-final5e6-webwiki step_7000`
- champion repo: https://huggingface.co/nazdef/gpt2small-en-it-nanochat-lr2e4-bs6-wsd-earlydecay7000-final5e6-webwiki-step7000
- champion `val_loss_mixed`: `5.2158`
- this release `val_loss_mixed`: `5.2270`
- delta vs champion: `+0.0112`

Operational reading:

- `step_11000` is the best checkpoint of the short-fast-decay run
- it is also the most stable-looking checkpoint among the recent short-fast/new candidates in this tracked slice
- the overall crown still stays with `earlydecay7000 step_7000`
- this release is therefore comparative and useful, not a claim that the benchmark champion has changed

## Source/domain losses for this checkpoint

- `source_loss_books_en`: `4.9168`
- `source_loss_books_it`: `4.8002`
- `source_loss_code`: `7.8108`
- `source_loss_web_en`: `5.8082`
- `source_loss_web_it`: `6.2641`
- `source_loss_wiki_en`: `4.0474`
- `source_loss_wiki_it`: `3.7621`

## Probe reading at this checkpoint

- EN factual prompt `The capital of Italy is -> Rome`: `rank=29`, `prob=0.0038452`
- EN procedural prompt `A small language model should -> be`: `rank=1`, `prob=0.4707031`
- IT factual prompt `La capitale d'Italia è -> Roma`: `rank=222`, `prob=0.0004597`
- IT procedural prompt `Un piccolo modello linguistico dovrebbe -> essere`: `rank=1`, `prob=0.3085938`

These probes are directional evidence only. The main selection rule here remains the repo-native benchmark result and the cross-run comparison context above.

## Training/data provenance

- training config: `training_config.yaml`
- tokenizer: `tokenizer.json` + `tokenizer_meta.json`
- packed dataset root used by the run: `/mnt/apps/llm-nanochat/datasets/202605141153_fineweb50_wiki50_50en_50it_score100_2500context_5Btokens_tok_20260515_en50it50_webwiki_stratified_500M`
- tokenizer root used by the run: `/mnt/apps/llm-nanochat/tokenizers/tokenizer_20260515_en50it50_webwiki_stratified_500M`
- source commit for release packaging: `032e34b8ae96bba41f37978f2e311e9b98ab7fa5`
- estimated tokens seen at this checkpoint: `~2.64B`

## Included files

- `step_11000.pt`
- `step_11000.safetensors`
- `step_11000.safetensors.json`
- `training_config.yaml`
- `tokenizer.json`
- `tokenizer_meta.json`
- `best_validation.json`
- `eval_summary.json`
- `comparison.json`
- `benchmark_report.md`
- `benchmark_metrics.json`
- `benchmark_scores.json`
- `benchmark_source_losses.json`
- `probe_step_11000_summary.json`
- `eval_metrics.jsonl`
- `metrics.jsonl`
- `probe_generations.jsonl`
- release note: `2026-06-06_shortfastdecay11k_release_step11000.md`

## Usage

This project uses a custom NanoChat inference/training stack. The easiest local UI in the source repo is the Chainlit checkpoint tester documented in the repo README.

## Limitations

- mixed quality is still in the weak/intermediate band
- generations remain repetitive and often unstable under free-form continuation
- factual recall is still weak in both languages
- the overall comparable benchmark champion is still `earlydecay7000 step_7000`
- dataset redistribution for the full training corpus may have separate licensing constraints; this repo contains model artifacts, not the raw/prepared training corpus