poopoobabylm/babylm-2026-strict-small-exp022

BabyLM 2026 Strict-Small — exp_022 (best run)

Best model of the babylm-2026-v3 run for the BabyLM 2026 Strict-Small track. A GPT-BERT / LTG-BERT hybrid trained with a data2vec-text style EMA self-distillation auxiliary on top of a hybrid MLM+CLM ("both") objective.

It was the highest-scoring of all 59 experiments in the run.

Headline result

Field	Value
Experiment	exp_022 — data2vec-text EMA self-distillation on a GPT-BERT (LTG-BERT) hybrid
aggregate_relative	1.5220 (1.0 = GPT-2-Strict-Small anchor; gpt-bert-mixed ref ≈ 1.73)
Best in run	Yes — highest aggregate across all 59 experiments
Training time	7560 s (126 min) on one L40S; final loss 2.695
Submission backend	both (hybrid MLM+CLM)
Checkpoint	exp_022_step9914 (EMA weights — these are what was exported)

Per-task scores (parity layer, externally comparable)

Task	Backend	exp_022	GPT-2-SS anchor	gpt-bert-mixed ref
BLiMP	mntp	72.86	65.36	71.40
BLiMP-Supplement	mntp	60.80	59.60	69.20
EWoK	mntp	50.86	49.86	51.23
EntityTracking	mntp	28.11	21.68	41.78
COMPS	mntp	52.48	51.81	53.70
Reading SPR (ΔR²)	causal	4.92	1.64	3.37
Reading ET (ΔR²)	causal	11.05	9.63	9.50

Seed caveat. With seed=42 this run is seed-fragile: EntityTracking is 28.11 at seed 42 but collapses to ~15.7 / 16.2 at seeds 0 / 1 (2 of 3 seeds collapse). A more seed-robust alternative from the same run is exp_016 (I-JEPA, aggregate_relative=1.4723, ET 27.05 / 26.33 across seeds).

Architecture

LTG-BERT / GPT-BERT hybrid (model_type: ltgbert, LTGBertForMaskedLM):

• hidden size 384, 12 layers, 6 attention heads, intermediate size 1280
• vocabulary 16,384 (custom BPE tokenizer), max position 512, relative position buckets (size 32)
• ~39M parameters (fp32 model.safetensors ≈ 150 MB)

Training

• Objective: token-level data2vec-text EMA self-distillation + hybrid MLM/CLM ("both" backend). EMA teacher τ=0.999; top-K=8 layer-norm'd layers as Smooth-L1 (β=4) targets at masked positions; aux weight λ=0.1, loss-dropout 0.5, disabled at seq=512.
• Optimizer LAMB, lr 7e-3, batch size 128, ~9,914 steps, seq length schedule 128 → 256 → 512, seed 42.
• Data: BabyLM 2026 strict-small corpus (bpe16k tokenizer).

Usage

This model ships custom modeling code (modeling_ltgbert.py, configuration_ltgbert.py), so load it with trust_remote_code=True:

from transformers import AutoModelForMaskedLM, AutoTokenizer

repo = "poopoobabylm/babylm-2026-strict-small-exp022"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True)

Notes

• Evaluation backend is hard-coded per task to match the BabyLM eval pipeline; the submission backend is "both".