Add BAM-B0 model card
Browse files
README.md
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
tags:
|
| 4 |
+
- babylm
|
| 5 |
+
- masked-language-model
|
| 6 |
+
- gpt-bert
|
| 7 |
+
- strict
|
| 8 |
+
---
|
| 9 |
+
|
| 10 |
+
# BAM-B0
|
| 11 |
+
|
| 12 |
+
BAM-B0 (BabyLM Aligned MNTP) is a 96.3M-parameter GPT-BERT/MNTP masked language
|
| 13 |
+
model trained for the BabyLM 2026 Strict track.
|
| 14 |
+
|
| 15 |
+
## Model
|
| 16 |
+
|
| 17 |
+
- Architecture: custom GPT-BERT masked language model
|
| 18 |
+
- Parameters: 96,333,280
|
| 19 |
+
- Tokenizer: 16k BabyLM Strict tokenizer
|
| 20 |
+
- Objective: 15 masked-next-token-prediction steps to 1 causal language modeling step
|
| 21 |
+
- Initialization: random initialization
|
| 22 |
+
|
| 23 |
+
## Evaluation
|
| 24 |
+
|
| 25 |
+
The B0-only Strict submission artifact contains final predictions, B0's own
|
| 26 |
+
28-checkpoint fast-evaluation trajectory, GlobalPIQA, filtered Entity
|
| 27 |
+
Tracking predictions, Reading predictions, and 28-stage AoA surprisals. It
|
| 28 |
+
was checked with the 2026 leaderboard validation code. The JSON is retained
|
| 29 |
+
separately for the official BabyLM submission and is intentionally not
|
| 30 |
+
included in this model repository.
|
| 31 |
+
|
| 32 |
+
The model uses custom configuration and modeling files included in this
|
| 33 |
+
repository; load it with `trust_remote_code=True`.
|