| --- |
| license: apache-2.0 |
| tags: |
| - babylm |
| - masked-language-model |
| - gpt-bert |
| - strict |
| --- |
| |
| # BAM-B0 |
|
|
| BAM-B0 (BabyLM Aligned MNTP) is a 96.3M-parameter GPT-BERT/MNTP masked language |
| model trained for the BabyLM 2026 Strict track. |
|
|
| ## Model |
|
|
| - Architecture: custom GPT-BERT masked language model |
| - Parameters: 96,333,280 |
| - Tokenizer: 16k BabyLM Strict tokenizer |
| - Objective: 15 masked-next-token-prediction steps to 1 causal language modeling step |
| - Initialization: random initialization |
|
|
| ## Evaluation |
|
|
| The B0-only Strict submission artifact contains final predictions, B0's own |
| 28-checkpoint fast-evaluation trajectory, GlobalPIQA, filtered Entity |
| Tracking predictions, Reading predictions, and 28-stage AoA surprisals. It |
| was checked with the 2026 leaderboard validation code. The JSON is retained |
| separately for the official BabyLM submission and is intentionally not |
| included in this model repository. |
|
|
| The model uses custom configuration and modeling files included in this |
| repository; load it with `trust_remote_code=True`. |
|
|
