Publish step_23100 decay-only checkpoint candidate

README.md

@@ -0,0 +1,97 @@
+---
+language: [en, it]
+license: cc-by-sa-4.0
+library_name: transformers
+pipeline_tag: text-generation
+datasets:
+- epfml/FineWeb-HQ
+- epfml/FineWeb2-HQ
+- google/wiki40b
+tags:
+- 1gpu-llm
+- single-gpu
+- continual-pretraining
+- decay-only
+- gpt2preln
+- bilingual
+- english
+- italian
+- checkpoint-release
+- gpt2
+- causal-lm
+- llm-nanochat
+- medium
+---
+
+# GPT2Medium EN/IT NanoChat — 22k decay-only checkpoint `step_23100`
+
+This is an **ordinary, non-official checkpoint release** from a decay-only continual-pretraining branch started from `step_22000`. It is a candidate for comparison in the future `1gpu-llm-medium-v2` selection; it is not the definitive v2 release.
+
+## Released checkpoint
+
+- checkpoint: `step_23100.pt`
+- branch: `20260713_resume-gpt2medium-gpt2preln-k20-wsddecayonly-cpt14700-step22000-lr5e5-final1e5-webwiki-d1800`
+- parent checkpoint: `step_22000.pt`
+- decay schedule: `1800` steps, final target `step_23800`
+- languages: English + Italian
+- context window: `2500` tokens
+- architecture: GPT-2-style decoder with pre-layernorm blocks
+- architecture identifiers: `architecture: gpt2`, `block_type: gpt2_prelayernorm`
+- parameter count: approximately `337.7M` native training parameters; approximately `337.6M` in the Transformers export
+- hardware: single RTX 4060 Ti 16GB
+
+## Selection and position
+
+`step_23100` is the scalar winner of the branch, reached after `1100` of the `1800` decay steps:
+
+- parent `step_22000`: `val_loss_mixed = 4.5058`
+- winner `step_23100`: `val_loss_mixed = 4.4675`
+- improvement: `-0.0383`
+- endpoint `step_23800`: `val_loss_mixed = 4.4918`
+
+The endpoint must therefore not be promoted automatically. The current global medium scalar champion remains the no-decay-branch CPT checkpoint `step_34000` at `4.4401`. This checkpoint is a branch winner and comparison candidate, not an official family release.
+
+## Training data
+
+The model was trained on the bilingual EN/IT web + wiki corpus:
+
+- English FineWeb-HQ (`epfml/FineWeb-HQ`)
+- Italian FineWeb2-HQ (`epfml/FineWeb2-HQ`)
+- English and Italian Wiki40B (`google/wiki40b`)
+- local dataset: `202605141153_fineweb50_wiki50_50en_50it_score100_2500context_5Btokens_tok_20260515_en50it50_webwiki_stratified_500M`
+
+## Quick start
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+repo_id = "nazdef/20260713_resume-gpt2medium-step22000-d1800-step23100"
+tokenizer = AutoTokenizer.from_pretrained(repo_id)
+model = AutoModelForCausalLM.from_pretrained(repo_id)
+
+prompt = "La capitale d'Italia è"
+prompt_ids = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
+bos = torch.tensor([[tokenizer.bos_token_id]], dtype=prompt_ids["input_ids"].dtype)
+input_ids = torch.cat([bos, prompt_ids["input_ids"]], dim=1)
+attention_mask = torch.ones_like(input_ids)
+outputs = model.generate(
+    input_ids=input_ids,
+    attention_mask=attention_mask,
+    do_sample=True,
+    max_new_tokens=64,
+    temperature=0.8,
+    top_k=50,
+    top_p=0.95,
+    repetition_penalty=1.1,
+    eos_token_id=tokenizer.eos_token_id,
+    pad_token_id=tokenizer.pad_token_id,
+)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+This is a base pretraining checkpoint, not an instruction-tuned chat model.
+
+## License
+
+This release uses **CC-BY-SA-4.0** as the practical downstream posture for the mixed training corpus. The corpus combines FineWeb-HQ/FineWeb2-HQ web data and Wiki40B slices, whose upstream terms and attribution/share-alike obligations may apply to downstream use and redistribution. Users are responsible for checking that their intended use and derivative packaging comply with the upstream dataset terms.

benchmark_summary_22100to23800.json

@@ -0,0 +1,14 @@
+{
+  "comparison_path": "/mnt/apps/llm-nanochat/evals/20260714_1407_gpt2medium_wsddo22000_d1800_22100to23800_cpu_full_benchmark/comparison.json",
+  "metadata_path": "/mnt/apps/llm-nanochat/evals/20260714_1407_gpt2medium_wsddo22000_d1800_22100to23800_cpu_full_benchmark/eval_metadata.json",
+  "num_checkpoints": 18,
+  "out_dir": "/mnt/apps/llm-nanochat/evals/20260714_1407_gpt2medium_wsddo22000_d1800_22100to23800_cpu_full_benchmark",
+  "recommended_checkpoint": {
+    "checkpoint_name": "step_23100",
+    "checkpoint_path": "/mnt/apps/llm-nanochat/checkpoints/20260713_resume-gpt2medium-gpt2preln-k20-wsddecayonly-cpt14700-step22000-lr5e5-final1e5-webwiki-d1800/step_23100.pt",
+    "direction": "min",
+    "value": 4.467542012532552
+  },
+  "report_path": "/mnt/apps/llm-nanochat/evals/20260714_1407_gpt2medium_wsddo22000_d1800_22100to23800_cpu_full_benchmark/report.md",
+  "suite": "pretrain_minimal_en_it_webwiki_step11000"
+}

config.json

@@ -0,0 +1,26 @@
+{
+  "activation_function": "gelu",
+  "architecture": "gpt2",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.0,
+  "block_type": "gpt2_prelayernorm",
+  "causal_mask_mode": "buffered_upper_triangular",
+  "embd_pdrop": 0.0,
+  "init_strategy": "gpt2_std_0.02_residual_scale",
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 2500,
+  "n_embd": 1024,
+  "n_head": 16,
+  "n_layer": 24,
+  "n_positions": 2500,
+  "norm_order": "preln",
+  "norm_type": "layernorm",
+  "positional_encoding": "learned_absolute",
+  "resid_pdrop": 0.0,
+  "tie_word_embeddings": true,
+  "use_cache": true,
+  "vocab_size": 32000
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef314714ddc2ff4b0778757d75e6534520fd4d4ab77a3fcc7e11e25ef8635c3b
+size 1350587904

special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<bos>",
+  "eos_token": "<eos>",
+  "pad_token": "<pad>",
+  "unk_token": "<unk>"
+}

step_23100.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:26b1a4f177caa855dbe160797fa596a665fc9ddc90780fc7c9bebf90b8006428
+size 4052410819

step_23100.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:12f55e2a7e2a16cb3bd7d89a3a7bafbb8f30677053e0f789d16a19e35029ec23
+size 1481791472

tokenizer_config.json

@@ -0,0 +1,9 @@
+{
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<eos>",
+  "model_max_length": 2500,
+  "pad_token": "<pad>",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<unk>"
+}

tokenizer_meta.json

@@ -0,0 +1,10 @@
+{
+  "vocab_size_requested": 32000,
+  "vocab_size_actual": 32000,
+  "special_tokens": [
+    "<pad>",
+    "<bos>",
+    "<eos>",
+    "<unk>"
+  ]
+}

training_config.yaml

@@ -0,0 +1,62 @@
+# WSD-decay-only continuation from the medium continual-pretraining checkpoint at step_22000.
+# Intention: branch from a checkpoint that was still inside the stable plateau of the parent CPT run,
+# then apply a shorter explicit decay-only cooldown for dense checkpoint comparison.
+
+resume_from: /mnt/apps/llm-nanochat/checkpoints/20260703_continual-pretraining-gpt2medium-gpt2preln-k20-step14700-lr5e5-w500-s18500-d2000-final1e5-webwiki/step_22000.pt
+dataset_dir: /mnt/apps/llm-nanochat/datasets/202605141153_fineweb50_wiki50_50en_50it_score100_2500context_5Btokens_tok_20260515_en50it50_webwiki_stratified_500M
+output_dir: /mnt/apps/llm-nanochat/artifacts/runs/20260713_resume-gpt2medium-gpt2preln-k20-wsddecayonly-cpt14700-step22000-lr5e5-final1e5-webwiki-d1800
+tokenizer_dir: /mnt/apps/llm-nanochat/tokenizers/tokenizer_20260515_en50it50_webwiki_stratified_500M
+seed: 1337
+
+model:
+  architecture: gpt2
+  block_type: gpt2_prelayernorm
+  tie_word_embeddings: true
+  vocab_size: 32000
+  dim: 1024
+  n_layers: 24
+  n_heads: 16
+
+training:
+  sequence_length: 2500
+  max_steps: 23800
+  batch_size: 2
+  grad_accum_steps: 48
+
+  learning_rate: 5.0e-05
+  peak_lr: 5.0e-05
+  lr_schedule: wsd-decay-only
+
+  warmup_steps: 0
+  stable_steps: 0
+  decay_steps: 1800
+  final_lr: 1.0e-05
+
+  adamw_betas:
+    - 0.9
+    - 0.95
+  adamw_eps: 1.0e-08
+  weight_decay: 0.1
+  clip_grad_norm: 1.0
+
+  save_every_steps: 100
+  checkpoint_dir: /mnt/apps/llm-nanochat/checkpoints/20260713_resume-gpt2medium-gpt2preln-k20-wsddecayonly-cpt14700-step22000-lr5e5-final1e5-webwiki-d1800
+  precision: bf16
+
+evaluation:
+  validation_every_steps: 100
+  validation_max_batches: 128
+  probe_every_steps: 1000
+  probe_tokenizer_dir: /mnt/apps/llm-nanochat/tokenizers/tokenizer_20260515_en50it50_webwiki_stratified_500M
+  probe_max_new_tokens: 32
+  probe_prompts:
+    en:
+      - prompt: "The capital of Italy is"
+        expected_next_text: " Rome"
+      - prompt: "A small language model should"
+        expected_next_text: " be"
+    it:
+      - prompt: "La capitale d'Italia è"
+        expected_next_text: " Roma"
+      - prompt: "Un piccolo modello linguistico dovrebbe"
+        expected_next_text: " essere"