---
language: [en, it]
license: cc-by-sa-4.0
library_name: transformers
pipeline_tag: text-generation
datasets:
- epfml/FineWeb-HQ
- epfml/FineWeb2-HQ
- google/wiki40b
tags:
- 1gpu-llm
- single-gpu
- continual-pretraining
- decay-only
- gpt2preln
- bilingual
- english
- italian
- checkpoint-release
- gpt2
- causal-lm
- llm-nanochat
- medium
---

# GPT2Medium EN/IT NanoChat — 22k decay-only checkpoint `step_23100`

This is an **ordinary, non-official checkpoint release** from a decay-only continual-pretraining branch started from `step_22000`. It is a candidate for comparison in the future `1gpu-llm-medium-v2` selection; it is not the definitive v2 release.

## Released checkpoint

- checkpoint: `step_23100.pt`
- branch: `20260713_resume-gpt2medium-gpt2preln-k20-wsddecayonly-cpt14700-step22000-lr5e5-final1e5-webwiki-d1800`
- parent checkpoint: `step_22000.pt`
- decay schedule: `1800` steps, final target `step_23800`
- languages: English + Italian
- context window: `2500` tokens
- architecture: GPT-2-style decoder with pre-layernorm blocks
- architecture identifiers: `architecture: gpt2`, `block_type: gpt2_prelayernorm`
- parameter count: approximately `337.7M` native training parameters; approximately `337.6M` in the Transformers export
- hardware: single RTX 4060 Ti 16GB

## Selection and position

`step_23100` is the scalar winner of the branch, reached after `1100` of the `1800` decay steps:

- parent `step_22000`: `val_loss_mixed = 4.5058`
- winner `step_23100`: `val_loss_mixed = 4.4675`
- improvement: `-0.0383`
- endpoint `step_23800`: `val_loss_mixed = 4.4918`

The endpoint must therefore not be promoted automatically. The current global medium scalar champion remains the no-decay-branch CPT checkpoint `step_34000` at `4.4401`. This checkpoint is a branch winner and comparison candidate, not an official family release.

## Training data

The model was trained on the bilingual EN/IT web + wiki corpus:

- English FineWeb-HQ (`epfml/FineWeb-HQ`)
- Italian FineWeb2-HQ (`epfml/FineWeb2-HQ`)
- English and Italian Wiki40B (`google/wiki40b`)
- local dataset: `202605141153_fineweb50_wiki50_50en_50it_score100_2500context_5Btokens_tok_20260515_en50it50_webwiki_stratified_500M`

## Quick start

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

repo_id = "nazdef/20260713_resume-gpt2medium-step22000-d1800-step23100"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)

prompt = "La capitale d'Italia è"
prompt_ids = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
bos = torch.tensor([[tokenizer.bos_token_id]], dtype=prompt_ids["input_ids"].dtype)
input_ids = torch.cat([bos, prompt_ids["input_ids"]], dim=1)
attention_mask = torch.ones_like(input_ids)
outputs = model.generate(
    input_ids=input_ids,
    attention_mask=attention_mask,
    do_sample=True,
    max_new_tokens=64,
    temperature=0.8,
    top_k=50,
    top_p=0.95,
    repetition_penalty=1.1,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

This is a base pretraining checkpoint, not an instruction-tuned chat model.

## License

This release uses **CC-BY-SA-4.0** as the practical downstream posture for the mixed training corpus. The corpus combines FineWeb-HQ/FineWeb2-HQ web data and Wiki40B slices, whose upstream terms and attribution/share-alike obligations may apply to downstream use and redistribution. Users are responsible for checking that their intended use and derivative packaging comply with the upstream dataset terms.