---
tags:
- kabyle
- tokenizer
- sentencepiece
- low-resource
language: kab
license: apache-2.0
---

# Kabyle Tokenizer for T5

SentencePiece tokenizer trained on 787,648 Kabyle sentences from Tatoeba, designed for T5-style models.

## Vocabulary
- Size: 32,000 tokens
- Type: BPE (Byte Pair Encoding)
- Character coverage: 99.99%

## Special Tokens
- `<unk>`: Unknown token
- `<pad>`: Padding token
- `</s>`: End of sequence
- `<s>`: Beginning of sequence
- `<mask>`: Mask token (for T5)

## Usage

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("boffire/kabyle-tokenizer-T5")
tokens = tokenizer.tokenize("Aqcic-nni yeɣra adlis.")
print(tokens)  # ['▁Aqcic', '-', 'nni', '▁yeɣra', '▁adlis', '.']
```

## Training Data
- Source: Tatoeba Kabyle corpus
- Sentences: 787,648
- Cleaning: Greek/Cyrillic contamination removed (ε→ɛ, γ→ɣ, Σ→Ɛ, Γ→Ɣ, Ԑ→Ɛ, ԑ→ɛ)

## Comparison with T5-original

| Phrase | T5-original tokens | Kabyle-SPM tokens |
|--------|-------------------|-------------------|
| Aqcic-nni yeɣra adlis. | 19 | 6 |
| Tettmeslayeḍ taqbaylit? | 18 | 4 |
| Ur zmireɣ ara ad qqimeɣ argaz-a. | ~20 | 10 |

## Kabyle Characters Preserved
- ɛ / Ɛ (open e)
- ɣ / Ɣ (gamma)
- č / Č (c with caron)
- ǧ / Ǧ (g with caron)
- ḍ / Ḍ (d with dot below)
- ḥ / Ḥ (h with dot below)
- ṛ / Ṛ (r with dot below)
- ṣ / Ṣ (s with dot below)
- ṭ / Ṭ (t with dot below)
- ẓ / Ẓ (z with dot below)

## Limitations
- Optimized for short sentences (Tatoeba style)
- May split rare compound words (e.g., "tebirt" → "teb" + "irt")
- Requires T5 model with resized embeddings for full compatibility