---
language:
- en
- nl
- zh
license: mit
tags:
- causal-lm
- babylm
- ipa
- phonology
---

# babylm2026-pl-ru-ipa-add

LLaMA-style causal language model trained for BabyLM 2026 with IPA phonological
feature augmentation.

## Architecture

- **Base**: LLaMA-style transformer (RMSNorm, RoPE, SwiGLU FFN, weight-tied LM head)
- **IPA variant**: `ipa_add` — gated fusion of panphon IPA feature vectors at the embedding layer
- **Hidden size**: 512  |  **Layers**: 6  |  **Heads**: 8
- **Vocab size**: 32000  |  **Context**: 512 tokens
- **IPA dim**: 24 (panphon binary articulatory features)
- **Training split**: pl_ru

## IPA Fusion Variants

| Variant | Description |
|---------|-------------|
| `baseline` | Token embeddings only |
| `ipa_add` | `embed + W·ipa` |
| `ipa_gate` | `embed + σ(W_g·ipa) ⊙ (W·ipa)` |
| `ipa_full` | Gated fusion + MSE auxiliary reconstruction loss |

## Usage

```python
from transformers import AutoConfig, AutoModelForCausalLM
import torch

config = AutoConfig.from_pretrained("pakphum/babylm2026-pl-ru-ipa-add", trust_remote_code=True)
model  = AutoModelForCausalLM.from_pretrained("pakphum/babylm2026-pl-ru-ipa-add", trust_remote_code=True)
model.eval()

# input_ids from your tokenizer
input_ids = torch.tensor([[1, 2, 3, 4]])

# Optional: supply IPA vectors [B, T, ipa_dim=24]. Zeros = no phonological signal.
ipa_vectors = torch.zeros(1, 4, config.ipa_dim)

with torch.no_grad():
    out = model(input_ids, ipa_vectors=ipa_vectors)
    # out.logits: [B, T, vocab_size]
```

## Citation

BabyLM 2026 shared task.