dakopi's picture
Update README.md
e577d06 verified
|
Raw
History Blame Contribute Delete
1.9 kB
---
license: apache-2.0
base_model: allenai/Olmo-3-1025-7B
language:
- en
library_name: transformers
---
This repo is a collection of SFT checkpoints produced by sweeping unique training samples vs epochs, following the setup from the paper:
Data Repetition Beats Data Scaling in Long-CoT Supervised Fine-Tuning
https://arxiv.org/abs/2602.11149
### Default model
The repo root contains the weights and config for the default variant trained with 16 epochs on 800 samples.
Calling `from_pretrained(repo_id)` loads this checkpoint.
### Variants
Each subfolder follows:
s{N}_e{M}
where:
- s{N} means N unique samples
- e{M} means M epochs
Example names:
- s3200_e8 means 3200 unique samples trained for 8 epochs
- s12800_e1 means 12800 unique samples trained for 1 epoch
## How to load
Load the default model (root):
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "dakopi/olmo3-7b_data-repetition"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)
```
Load a specific variant (subfolder):
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "dakopi/olmo3-7b_data-repetition"
variant = "s6400_e4"
tokenizer = AutoTokenizer.from_pretrained(repo_id, subfolder=variant)
model = AutoModelForCausalLM.from_pretrained(repo_id, subfolder=variant)
```
## Reproducibility and code
Official training and evaluation code:
https://github.com/dkopi/data-repetition
## Citation
```
@misc{kopiczko2026datarepetitionbeatsdata,
title = {Data Repetition Beats Data Scaling in Long-CoT Supervised Fine-Tuning},
author = {Dawid J. Kopiczko and Sagar Vaze and Tijmen Blankevoort and Yuki M. Asano},
year = {2026},
eprint = {2602.11149},
archivePrefix= {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2602.11149}
}
```