---
license: mit
tags:
  - babylm
  - babylm-2026
  - modernbert
  - masked-language-model
  - strict-small
language:
  - en
library_name: transformers
pipeline_tag: fill-mask
---

# ModernBERT-Small (Compositional + Shuffled-Frequency Residual, Control) — BabyLM 2026 Strict-Small

This model is a `ModernBERT-Small` masked language model (RoPE, GeGLU, alternating local/global
attention, 384 hidden size, 16 layers, 6 attention heads) trained from scratch on the
**BabyLM 2026 Strict-Small** 10M-word corpus, as part of an ablation study on
parameter-efficient token embedding layers for developmentally-plausible pretraining under the
BabyLM Challenge's strict-small compute and data budget.

## Embedding design

Standard transformer token embedding tables scale as `vocab_size x hidden_size`, which for this
model's 30,522-token vocabulary and 384 hidden size would be an 11.7M-parameter dense lookup
table. This checkpoint's base representation is the same **compositional byte-n-gram embedding**
used across this ablation sweep: each vocabulary token is decomposed into its raw UTF-8 byte
sequence, every byte n-gram (n = 1 to 4) is extracted and hashed into one of 8,192 shared
buckets, and the token's representation is the mean of its buckets' embeddings (a
128-dimensional `EmbeddingBag` table, ~1.0M parameters) rather than a dedicated per-token row.

On top of that shared composed representation, this variant adds a **frequency-gated per-token
residual** -- a learned `vocab_size x 128` embedding added to the composed vector, scaled per
token by `frequency / (frequency + tau)` (tau = 100) so common tokens lean more on their own
residual and rare tokens rely mostly on the composition. Critically, this checkpoint is the
**shuffled-frequency control**: the frequency values used to compute each token's gating weight
are randomly permuted across the vocabulary (fixed seed 42) before being applied, so a token's
residual is gated by an unrelated token's frequency rather than its own. This isolates whether
any benefit from the frequency-gated residual comes from the *correct* frequency signal, or
merely from having *some* per-token residual-plus-gating mechanism -- a null-effect control
against the true frequency-gated variant in this sweep.

This is one variant in a broader comparison of embedding-layer parameterizations (dense, linear
and MLP-style factorization, tensor-train decomposition, deterministic Fourier expansion, and
compositional/frequency-adaptive variants) evaluated under identical data, tokenizer, optimizer,
and training budget, to isolate the effect of the embedding layer's parameterization on
downstream BabyLM evaluation performance.

## Training data

BabyLM 2026 Strict-Small corpus (~10M words), tokenized with a byte-level BPE tokenizer trained
on the same corpus (vocab size 30,522). No external data, synthetic augmentation, or human
annotation beyond the corpus as officially released.

## Usage

```python
from transformers import AutoModelForMaskedLM, AutoTokenizer

model = AutoModelForMaskedLM.from_pretrained(
    "remg1997/modernbert-small-phase6-composition-frequency-shuffled-babylm2026",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "remg1997/modernbert-small-phase6-composition-frequency-shuffled-babylm2026"
)
```

Each `chck_{N}M` branch of this repository corresponds to a BabyLM-Challenge compliance
checkpoint (one per `N` million words of training data seen); `main` points at the final,
fully-trained checkpoint.

## Evaluation

Evaluated with the official [`babylm-eval`](https://github.com/babylm-org/babylm-eval) harness
(BLiMP, EWoK, entity tracking, COMPS, Global PIQA, reading-time correlation, GLUE/SuperGLUE
fine-tuning, and Age-of-Acquisition word-surprisal correlation) under the strict-small track.