Xerneas3318/modern-1024x24-base-stable

modern-1024x24 — base (end of WSD stable phase)

Pretrained base captured at the end of the WSD stable phase (step 15000, just before LR decay begins at step 15259). Learning rate is still at maximum here, so this checkpoint is the right starting point for continued pretraining or your own annealing/decay schedule. For downstream use you probably want the decayed modern-1024x24-base instead.

~481M parameters, trained from scratch on a single H100 80GB.

Architecture

Parameters	480.9M
Layers	24
Model dim	1024
Heads	16 (GQA, kv_group=4 → 4 KV heads)
Context	2048
Vocab	50304 (GPT-2 BPE, padded from 50257)

Modern components: RMSNorm (pre-norm), RoPE, SwiGLU FFN, grouped-query attention, QK-norm, value embeddings, U-net style skip connections between layer halves, and a final logit softcap (15 * tanh(logits/15)).

Optimizer: Muon (Newton-Schulz orthogonalisation) on hidden 2-D matrices, AdamW on embeddings / lm_head / norms — embeddings are excluded from Muon by identity.

Training

Data	FineWeb-Edu sample-10BT (82%) + codeparrot-clean Python (18%)
Tokens	~10.0B (19,073 steps x 524,288 tokens)
Schedule	WSD — 3% warmup, stable to step 15,259, linear decay to 0
LR	Muon 0.02, AdamW 1.2e-3
Precision	bf16 autocast
Hardware	1x H100 80GB, ~132k tok/s, ~21h

Results

Validation loss at this checkpoint: 3.0997 (FineWeb-Edu held-out shard).

Learning rates were still at maximum when this was saved (Muon 0.02, AdamW 1.2e-3) — decay had not yet started. For reference, the fully decayed model reaches 2.7618, so the decay phase alone is worth -0.34 val loss.

checkpoint	step	val loss
end of stable (this model)	15000	3.0997
end of decay (modern-1024x24-base)	19072	2.7618

Usage

This is a raw PyTorch checkpoint, not a transformers model. You need model.py from xerneas3318/modern-lm.

import torch
from model import GPT, GPT2Config, build_enc

ckpt = torch.load("model.pt", map_location="cpu", weights_only=False)
model = GPT(GPT2Config(**ckpt["config"]))
model.load_state_dict(ckpt["model"])
model.eval()

enc = build_enc()   # GPT-2 BPE + reserved chat/tool special tokens

Tokenizer

GPT-2 BPE with reserved special tokens in the padded vocab region (IDs 50257+): <|im_start|>, <|im_end|>, <pad>, <think>, </think>, <tool_call>, </tool_call>, <tool_result>, </tool_result>. These are untrained during pretraining (they never occur in the corpus) and are learned at SFT.

Limitations

Trained on ~10B tokens at 481M parameters — roughly compute-optimal for its size, but small by modern standards. Expect factual unreliability and limited reasoning. No safety tuning of any kind. The base models are next-token predictors and are not instruction-following.