remg1997/modernbert-small-factorized-linear-dual-babylm2026

ModernBERT-Small (Factorized Linear Embeddings, Tied) — BabyLM 2026 Strict-Small

This model is a ModernBERT-Small masked language model (RoPE, GeGLU, alternating local/global attention, 384 hidden size, 16 layers, 6 attention heads) trained from scratch on the BabyLM 2026 Strict-Small 10M-word corpus, as part of an ablation study on parameter-efficient token embedding layers for developmentally-plausible pretraining under the BabyLM Challenge's strict-small compute and data budget.

Embedding design

Standard transformer token embedding tables scale as vocab_size x hidden_size, which for this model's 30,522-token vocabulary and 384 hidden size would be an 11.7M-parameter dense lookup table -- a large fraction of the model's total parameter budget under the strict-small constraint. Like this sweep's other factorized-linear checkpoint, this model uses an ALBERT-style linear factorization: tokens are first embedded into a smaller 128-dimensional bottleneck space, then linearly projected up to the model's 384-dimensional hidden size, replacing one large vocab_size x hidden_size matrix with two much smaller ones (vocab_size x 128 and 128 x 384).

This checkpoint is the tied ("dual-use") variant: tie_word_embeddings=true, so the same factorized embedding weights are reused in reverse for the MLM output projection, rather than training a separate untied decoder as its sibling checkpoint in this sweep does. This roughly halves the embedding-related parameter count relative to the untied variant, at the cost of constraining the input and output representations to share the same factorized subspace -- a direct ablation of whether tying helps or hurts under a fixed factorized-embedding budget.

This is one variant in a broader comparison of embedding-layer parameterizations (dense, linear and MLP-style factorization, tensor-train decomposition, deterministic Fourier expansion, and compositional/frequency-adaptive variants) evaluated under identical data, tokenizer, optimizer, and training budget, to isolate the effect of the embedding layer's parameterization on downstream BabyLM evaluation performance.

Training data

BabyLM 2026 Strict-Small corpus (~10M words), tokenized with a byte-level BPE tokenizer trained on the same corpus (vocab size 30,522). No external data, synthetic augmentation, or human annotation beyond the corpus as officially released.

Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer

model = AutoModelForMaskedLM.from_pretrained(
    "remg1997/modernbert-small-factorized-linear-dual-babylm2026",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "remg1997/modernbert-small-factorized-linear-dual-babylm2026"
)

Each chck_{N}M branch of this repository corresponds to a BabyLM-Challenge compliance checkpoint (one per N million words of training data seen); main points at the final, fully-trained checkpoint. Note: this run's W&B logging only captured checkpoints from chck_20M through chck_100M -- earlier milestones (chck_1M-chck_10M) were not preserved, so this repository has fewer checkpoint branches than this sweep's other models.

Evaluation

Evaluated with the official babylm-eval harness (BLiMP, EWoK, entity tracking, COMPS, Global PIQA, reading-time correlation, GLUE/SuperGLUE fine-tuning, and Age-of-Acquisition word-surprisal correlation) under the strict-small track. Age-of-Acquisition results are unavailable for this checkpoint since that metric requires the full chck_1M-chck_100M checkpoint set, which this run does not have.