Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,78 @@
+---
+license: mit
+tags:
+  - babylm
+  - babylm-2026
+  - modernbert
+  - masked-language-model
+  - strict-small
+language:
+  - en
+library_name: transformers
+pipeline_tag: fill-mask
+---
+
+# ModernBERT-Small (Compositional + Shuffled-Frequency Residual, Control) — BabyLM 2026 Strict-Small
+
+This model is a `ModernBERT-Small` masked language model (RoPE, GeGLU, alternating local/global
+attention, 384 hidden size, 16 layers, 6 attention heads) trained from scratch on the
+**BabyLM 2026 Strict-Small** 10M-word corpus, as part of an ablation study on
+parameter-efficient token embedding layers for developmentally-plausible pretraining under the
+BabyLM Challenge's strict-small compute and data budget.
+
+## Embedding design
+
+Standard transformer token embedding tables scale as `vocab_size x hidden_size`, which for this
+model's 30,522-token vocabulary and 384 hidden size would be an 11.7M-parameter dense lookup
+table. This checkpoint's base representation is the same **compositional byte-n-gram embedding**
+used across this ablation sweep: each vocabulary token is decomposed into its raw UTF-8 byte
+sequence, every byte n-gram (n = 1 to 4) is extracted and hashed into one of 8,192 shared
+buckets, and the token's representation is the mean of its buckets' embeddings (a
+128-dimensional `EmbeddingBag` table, ~1.0M parameters) rather than a dedicated per-token row.
+
+On top of that shared composed representation, this variant adds a **frequency-gated per-token
+residual** -- a learned `vocab_size x 128` embedding added to the composed vector, scaled per
+token by `frequency / (frequency + tau)` (tau = 100) so common tokens lean more on their own
+residual and rare tokens rely mostly on the composition. Critically, this checkpoint is the
+**shuffled-frequency control**: the frequency values used to compute each token's gating weight
+are randomly permuted across the vocabulary (fixed seed 42) before being applied, so a token's
+residual is gated by an unrelated token's frequency rather than its own. This isolates whether
+any benefit from the frequency-gated residual comes from the *correct* frequency signal, or
+merely from having *some* per-token residual-plus-gating mechanism -- a null-effect control
+against the true frequency-gated variant in this sweep.
+
+This is one variant in a broader comparison of embedding-layer parameterizations (dense, linear
+and MLP-style factorization, tensor-train decomposition, deterministic Fourier expansion, and
+compositional/frequency-adaptive variants) evaluated under identical data, tokenizer, optimizer,
+and training budget, to isolate the effect of the embedding layer's parameterization on
+downstream BabyLM evaluation performance.
+
+## Training data
+
+BabyLM 2026 Strict-Small corpus (~10M words), tokenized with a byte-level BPE tokenizer trained
+on the same corpus (vocab size 30,522). No external data, synthetic augmentation, or human
+annotation beyond the corpus as officially released.
+
+## Usage
+
+```python
+from transformers import AutoModelForMaskedLM, AutoTokenizer
+
+model = AutoModelForMaskedLM.from_pretrained(
+    "remg1997/modernbert-small-phase6-composition-frequency-shuffled-babylm2026",
+    trust_remote_code=True,
+)
+tokenizer = AutoTokenizer.from_pretrained(
+    "remg1997/modernbert-small-phase6-composition-frequency-shuffled-babylm2026"
+)
+```
+
+Each `chck_{N}M` branch of this repository corresponds to a BabyLM-Challenge compliance
+checkpoint (one per `N` million words of training data seen); `main` points at the final,
+fully-trained checkpoint.
+
+## Evaluation
+
+Evaluated with the official [`babylm-eval`](https://github.com/babylm-org/babylm-eval) harness
+(BLiMP, EWoK, entity tracking, COMPS, Global PIQA, reading-time correlation, GLUE/SuperGLUE
+fine-tuning, and Age-of-Acquisition word-surprisal correlation) under the strict-small track.