---
language: tr
license: apache-2.0
tags:
  - tokenizer
  - morphel
  - morphology-aware
  - turkish
  - xnli
  - nlp-research
  - low-resource
  - ablation-random-baseline
datasets:
  - facebook/xnli
---

# NIRVLab — MorpheL Tokenizer for Turkish XNLI — ABLATION: Random-segmentation null baseline

**This is NOT the real MorpheL algorithm.** Every algorithmic component is
disabled: no MI scoring (Eq.1-4), no vowel-consonant plausible-boundary
pre-filter, no TopK candidate ranking (Eq.5), no Gumbel sampling (Eq.6-9).
Boundary count/position come from a plain uniform RNG. Exists purely as a
worst-case sanity control for MorpheL's own pipeline (Proposal: *MorpheL:
Morphology-Aware Tokenizer Adaptation for Pretrained Models in Low-Resource
Languages*) — if the real MI+Gumbel pipeline doesn't clearly beat this, the
added complexity isn't earning its keep.

Trained on the Turkish (`tr`) subset of
[facebook/xnli](https://huggingface.co/datasets/facebook/xnli) — all splits.

## Algorithm (ablated)

Words are segmented by drawing a uniform-random number of cuts (0-2) and
uniform-random cut positions — no MI, no vowel filter, no Gumbel, no learned
signal of any kind. Segmentation is cached at induction time with a fixed
seed for reproducibility, but carries no linguistic information.

**Vocab induction also differs from the real MorpheL pipeline here**: the
shared span-recombination step (counting every contiguous merge of a word's
pieces, e.g. [a,b,c] -> a,b,c,ab,bc,abc) was REMOVED for this ablation. That
step lets a frequent whole word win a vocab slot regardless of how it was cut
(the full-span "abc" always carries the word's true corpus frequency), which
would silently undo the random segmentation and make this not a true
know-nothing floor. Only the literal pieces produced by random cutting are
counted here.

## Training Config

| Parameter | Value |
|---|---|
| Algorithm | RANDOM (ablation — MI+Gumbel+vowel-filter all OFF) |
| Vocabulary size | 32,054 |
| random_max_cuts | 2 |
| seed | 42 |
| min_frequency | 2 |
| Special tokens | `<s>, <pad>, </s>, <unk>, <mask>` |
| Corpus | `facebook/xnli/tr` — all splits (800,404 sentences) |
| Vowel set | N/A — no plausible-boundary filter in this ablation |

## Evaluation Metrics (vs Baselines, vocab_size=32000, same corpus)

| Metric | BPE | WordPiece | Unigram | MorpheL (full) | **Random (this ablation)** |
|---|---|---|---|---|
| Fertility ↓ | — | — | — | — | **3.0540** |
| Tokens/char ↓ | — | — | — | — | **0.4077** |
| Avg seq len ↓ | — | — | — | — | **33.83** |
| Vocab coverage ↑ | — | — | — | — | **1.0000** |
| OOV rate ↓ | — | — | — | — | **0.0000** |

*Fill baseline columns after running baseline notebooks.*

## Usage

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("NIRVLab/xnli-morphel-tr-32k-ablation-random-baseline")
```

> **Note**: this checkpoint uses RANDOM segmentation (ablation control) — do
> not use it for actual MorpheL comparisons other than as the worst-case
> sanity baseline. No temperature / Gumbel parameter applies here.