---
license: other
language: en
library_name: transformers
pipeline_tag: text-generation
tags:
- nanochat
- nemotron
- from-scratch
- perlmutter
- gpt2-tokenizer
---
# d24-midtrain-v1base-mathheavy-3.7B

v1-base math-heavy-midtrained BASE LM (pre-SFT).

nanochat-style **depth-24** decoder — 24 layers × 1536 hidden × 12 heads, SwiGLU / RoPE / RMSNorm, tied embeddings, GPT-2 BPE vocab (50304), **0.757B params**, 2048-token context.

**Lineage.** v1 pretrain (5.84B ClimbMix) → math-heavy midtrain 3.7B (FineMath/OpenMath/MetaMath/OpenThoughts + ClimbMix anchor).

**Metrics.** Base checkpoint (pre-SFT) — evaluate after SFT. Corresponding SFT: `d24-sft-v1base-mathheavy-3.7B` (GSM8K 5.46%).

## Use (base LM)

This is a **base language model** (post-midtrain, **pre-SFT**) — use it for text continuation, not chat. EOS is the GPT-2 `<|endoftext|>` (`50256`). For a chat model, use the `d24-sft-*` checkpoints.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
mid = "sfanm/d24-midtrain-v1base-mathheavy-3.7B"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype="bfloat16", device_map="auto")
inputs = tok("The derivative of x**2 is", return_tensors="pt").to(model.device)
print(tok.decode(model.generate(**inputs, max_new_tokens=128)[0], skip_special_tokens=True))
```

*Research checkpoint from a from-scratch nanochat-d24 replication (pretrain → midtrain → SFT → RL) on NERSC Perlmutter. Trained on third-party corpora (ClimbMix, FineMath, OpenMath, MetaMath, OpenThoughts, OLMo-3 Dolmino, SmolTalk, …) — see those datasets' licenses; provided as-is for research.*