HinoMoto-100M v12 (WSD + z-loss + EMA, seed=2)
Sister model of v15. Same architecture/training as v15 but seed=2.
Use this for seed-variance studies. See FIshota/hinomoto-100m-v15-wsd-zloss-ema for primary.
Quick facts
- 100M (12 layer, d_model=512)
- Tokenizer: same as v15 (byte-BPE 9506)
- Training: 20000 steps WSD + z-loss + EMA decay=0.999
- seed=2 (vs v15 seed=0)
PPL
| Eval | PPL |
|---|---|
| 51 prompts | 16.35 |
| 200 prompts | 31.02 |
| 789 prompts | 48.49 |
Use with v15 / v16 for 3-seed variance estimate
| Seed | Model | 51-prompt PPL |
|---|---|---|
| 0 | v15 | 15.94 |
| 1 | v11/v16 (bit-identical) | 16.50 |
| 2 | v12 (this model) | 16.35 |
License
Apache-2.0
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support