Add model card (license + description) for leaderboard submission

README.md

@@ -0,0 +1,68 @@
+---
+license: mit
+tags:
+  - babylm
+  - babylm-2026
+  - modernbert
+  - masked-language-model
+  - strict-small
+language:
+  - en
+library_name: transformers
+pipeline_tag: fill-mask
+---
+
+# ModernBERT-Small (Factorized Linear Embeddings) — BabyLM 2026 Strict-Small
+
+This model is a `ModernBERT-Small` masked language model (RoPE, GeGLU, alternating local/global
+attention, 384 hidden size, 16 layers, 6 attention heads) trained from scratch on the
+**BabyLM 2026 Strict-Small** 10M-word corpus, as part of an ablation study on
+parameter-efficient token embedding layers for developmentally-plausible pretraining under the
+BabyLM Challenge's strict-small compute and data budget.
+
+## Embedding design
+
+Standard transformer token embedding tables scale as `vocab_size x hidden_size`, which for this
+model's 30,522-token vocabulary and 384 hidden size would be an 11.7M-parameter dense lookup
+table -- a large fraction of the model's total parameter budget under the strict-small
+constraint. This checkpoint instead uses an **ALBERT-style linear factorization**: tokens are
+first embedded into a smaller 128-dimensional bottleneck space, then linearly projected up to the
+model's 384-dimensional hidden size, replacing one large `vocab_size x hidden_size` matrix with
+two much smaller ones (`vocab_size x 128` and `128 x 384`). The output projection (`tie_word_
+embeddings=false`) uses its own separate decoder rather than sharing the input embedding weights.
+
+This is one variant in a broader comparison of embedding-layer parameterizations (dense, linear
+and MLP-style factorization, tensor-train decomposition, deterministic Fourier expansion, and
+compositional/frequency-adaptive variants) evaluated under identical data, tokenizer, optimizer,
+and training budget, to isolate the effect of the embedding layer's parameterization on
+downstream BabyLM evaluation performance.
+
+## Training data
+
+BabyLM 2026 Strict-Small corpus (~10M words), tokenized with a byte-level BPE tokenizer trained
+on the same corpus (vocab size 30,522). No external data, synthetic augmentation, or human
+annotation beyond the corpus as officially released.
+
+## Usage
+
+```python
+from transformers import AutoModelForMaskedLM, AutoTokenizer
+
+model = AutoModelForMaskedLM.from_pretrained(
+    "remg1997/modernbert-small-modernbert-small-factorized-linear-babylm2026",
+    trust_remote_code=True,
+)
+tokenizer = AutoTokenizer.from_pretrained(
+    "remg1997/modernbert-small-modernbert-small-factorized-linear-babylm2026"
+)
+```
+
+Each `chck_{N}M` branch of this repository corresponds to a BabyLM-Challenge compliance
+checkpoint (one per `N` million words of training data seen); `main` points at the final,
+fully-trained checkpoint.
+
+## Evaluation
+
+Evaluated with the official [`babylm-eval`](https://github.com/babylm-org/babylm-eval) harness
+(BLiMP, EWoK, entity tracking, COMPS, Global PIQA, reading-time correlation, GLUE/SuperGLUE
+fine-tuning, and Age-of-Acquisition word-surprisal correlation) under the strict-small track.