---
language:
- ja
license: apache-2.0
tags:
- japanese
- language-model
- 100m
- wsd
- z-loss
- ema
- pretrain
- research
library_name: pytorch
datasets:
- diet-records
- aozora-bunko
---

# HinoMoto-100M v12 (WSD + z-loss + EMA, seed=2)

Sister model of v15. Same architecture/training as v15 but seed=2.

Use this for seed-variance studies. See `FIshota/hinomoto-100m-v15-wsd-zloss-ema` for primary.

## Quick facts

- 100M (12 layer, d_model=512)
- Tokenizer: same as v15 (byte-BPE 9506)
- Training: 20000 steps WSD + z-loss + EMA decay=0.999
- **seed=2** (vs v15 seed=0)

## PPL

| Eval | PPL |
|---|---|
| 51 prompts | 16.35 |
| 200 prompts | 31.02 |
| 789 prompts | 48.49 |

## Use with v15 / v16 for 3-seed variance estimate

| Seed | Model | 51-prompt PPL |
|---|---|---|
| 0 | v15 | 15.94 |
| 1 | v11/v16 (bit-identical) | 16.50 |
| 2 | **v12 (this model)** | 16.35 |

## License

Apache-2.0