---
language: [ro, en]
license: cc-by-nc-4.0
tags: [romanian, bilingual, nanochat, pretrained]
---

# stefaniancu/rost-d6-sft

the instruction-tuned chat model, depth d6, checkpoint step 000010.

Trained from scratch on a bilingual Romanian/English mixture. This is a research
model at GPT-2 scale, not a product.

| | |
|---|---|
| architecture | nanochat GPT, d6, 2048 context |
| tokenizer | [stefaniancu/rost-tok-bilingual](https://huggingface.co/stefaniancu/rost-tok-bilingual), 32,768 vocab, bilingual |
| pretraining mixture | 30% Romanian / 70% English, interleaved at document level |
| Romanian source | FineWeb2-ro, educational score >= 3, diacritic-normalized |
| English source | ClimbMix |
| stage | sft |

The tokenizer is included under `tokenizer/` so this repo is self-contained.

## Licence

**CC-BY-NC-4.0, non-commercial.** Both halves of the training data carry
non-commercial terms: ClimbMix is CC-BY-NC-4.0 ("for research and development
only"), and the OpenLLM-Ro SFT datasets are CC-BY-NC-4.0. The model inherits
that. Note that the ClimbMix mirror used is tagged MIT, which does not override
the upstream terms.

## Known characteristics

The Romanian pretraining data is roughly **one third health content** by
character count (measured across three shards of the source), with history,
geography, finance and education each near 10%. Entertainment, gaming and
software are each under 1%. Expect fluency to be uneven across domains
accordingly.

## Reproduce

```bash
bash runs/speedrun_ro.sh
```

Source and full experiment log: https://github.com/StefanIancu/rost