---
license: other
language:
- en
library_name: transformers
pipeline_tag: text-generation
base_model: sfanm/d24-v6-pretrain
tags:
- d24-v6
- base-model
- midtraining
- olmo-3
- no-pretraining-replay
- gpt2-tokenizer
---

# D24 v6 — midtrain

The loadable model at this repository's root is the terminal replay-free
midtraining checkpoint in the public D24 v6 lineage. Starting from
`sfanm/d24-v6-pretrain`, it consumed
100,000,595,968 OLMo-3/Dolmino midtraining tokens with an explicit pretraining
replay fraction of zero and ended at optimizer iteration 23,842.

## Architecture

| Field | Value |
|---|---:|
| Parameters | 756,819,456 |
| Layers / hidden size | 24 / 1,536 |
| Attention heads | 12 (MHA) |
| FFN size | 4,096 (SwiGLU/SiLU) |
| Position encoding | RoPE, theta 10,000 |
| Normalization | RMSNorm |
| Embeddings | Tied |
| Tokenizer | GPT-2 BPE, vocabulary padded to 50,304 |
| Context | 2,048 tokens |
| Published weights | BF16 |

The Transformers class is `LlamaForCausalLM`, but this model uses the D24
architecture and GPT-2 tokenizer.

## Training

Midtraining used sequence length 2,048, global batch 2,048, microbatch 4,
AdamW at peak LR `4e-4`, weight decay 0.1, gradient clip 1.0, and a WSD schedule
with 1,000 warmup steps and a final 4,768-step cosine decay to `4e-5`. Final
held-out language-model loss was 1.365992 (perplexity 3.920).

## Loading

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sfanm/d24-v6-midtrain"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto"
)
```

This remains a base language model and has no chat template. All 12 retained,
resumable Megatron distributed checkpoints are published under `megatron/`,
from `iter_0002000` through the terminal `iter_0023842`.

This experimental research checkpoint has not undergone safety alignment or a
comprehensive evaluation. Review the licenses and terms of all upstream data
before redistribution or downstream use.