---
language: en
tags:
- causal-lm
- gqa
- qk-norm
- rope
- swiglu
- refresh-gate
- rose-x1
license: apache-2.0
---

# Rose-Mini

Rose X1 SLM (49.4M params).

## Architecture
| Property | Value |
|---|---|
| Layers | 14 |
| Hidden | 512 |
| Heads | 8 (kv=2) |
| QK Norm | True |
| Refresh Gate | [4, 9] |
| Params | 49.443M |

## Training
- Tokens: 12,206,861,568
- Val PPL: 6.14
- Optimizer: muon_adamw