---
language: ar
tags:
  - arabic
  - diacritization
  - tashkeel
  - nlp
  - classical-arabic
license: mit
datasets:
  - shamela
metrics:
  - wer
  - character_error_rate
---

# visionworks/turath1.0

A lightweight Transformer model for Arabic diacritization (tashkeel), trained on the Shamela classical Islamic text corpus. Designed for speed and accuracy on classical and modern standard Arabic.

## Model details

| Property | Value |
|----------|-------|
| Architecture | Transformer Encoder (6 layers) |
| Parameters | ~8M |
| Embed dim | 256 |
| Attention heads | 4 |
| FFN dim | 1024 |
| Max sequence length | 256 characters |
| Vocab size | 130 |
| Output classes | 15 diacritic labels |
| Training domain | Shamela classical Islamic corpus |

## Benchmark results

Evaluated on 1,000 held-out sentences from the Shamela test split, compared against publicly available Arabic diacritization systems.

| Model | DER ↓ | WER ↓ | Latency (CPU) |
|-------|-------|-------|---------------|
| flan-t5-small | **3.1%** | **29.2%** | 90ms |
| catt-eo | 5.2% | 30.1% | 31ms |
| **visionworks:turath1.0** | 7.1% | 29.7% | **5ms** |
| mishkal | 20.6% | 85.3% | 36ms |
| camel (MLE) | 33.7% | 81.0% | 6ms |

- **DER** (Diacritic Error Rate): fraction of characters with incorrect diacritics
- **WER** (Word Error Rate): fraction of words with at least one incorrect diacritic
- Latency measured on CPU (single sentence, no batching)

![Accuracy benchmark](bench_accuracy.png)

![Latency benchmark](bench_latency.png)

Despite being 17× faster than flan-t5-small, turath1.0 achieves nearly identical WER (29.7% vs 29.2%), making it well-suited for high-throughput or resource-constrained environments. Mishkal and CAMeL MLE score poorly on this benchmark due to domain mismatch (tuned for modern standard Arabic, not classical texts).

## Files

| File | Description |
|------|-------------|
| `best_model.pt` | Model weights + training args (PyTorch checkpoint) |
| `vocab.json` | Character vocabulary (130 tokens) |
| `inference.py` | Self-contained inference script |

## Usage

### Install dependencies

```bash
pip install torch
```

### Quick start

```python
import json
import torch
import torch.nn as nn
from huggingface_hub import hf_hub_download

# ── Download files ────────────────────────────────────────────────────────────
ckpt_path  = hf_hub_download("visionworks/turath1.0", "best_model.pt")
vocab_path = hf_hub_download("visionworks/turath1.0", "vocab.json")

# ── Constants ─────────────────────────────────────────────────────────────────
TASHKEEL_SET = set('ًٌٍَُِّْٰٕٓٔ')
LABEL_TO_DIAC = {
    0: '',    1: 'َ',  2: 'ُ',  3: 'ِ',
    4: 'ً',  5: 'ٌ',  6: 'ٍ',  7: 'ْ',  8: 'ّ',
    9: 'َّ', 10: 'ُّ', 11: 'ِّ', 12: 'ًّ', 13: 'ٌّ', 14: 'ٍّ',
}

# ── Model definition ──────────────────────────────────────────────────────────
class TransformerDiacritizer(nn.Module):
    def __init__(self, vocab_size, embed_dim, n_heads, n_layers,
                 ffn_dim, dropout=0.0, max_len=256):
        super().__init__()
        self.embed     = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        self.pos_embed = nn.Embedding(max_len + 2, embed_dim)
        self.drop      = nn.Dropout(dropout)
        self.layers    = nn.ModuleList([
            nn.TransformerEncoderLayer(
                d_model=embed_dim, nhead=n_heads, dim_feedforward=ffn_dim,
                dropout=dropout, batch_first=True, norm_first=True,
            ) for _ in range(n_layers)
        ])
        self.norm = nn.LayerNorm(embed_dim)
        self.fc   = nn.Linear(embed_dim, 15)

    def forward(self, ids, lengths):
        B, T     = ids.shape
        pos      = torch.arange(T, device=ids.device).unsqueeze(0).expand(B, -1)
        x        = self.drop(self.embed(ids) + self.pos_embed(pos))
        pad_mask = torch.arange(T, device=ids.device)[None, :] >= lengths[:, None]
        for layer in self.layers:
            x = layer(x, src_key_padding_mask=pad_mask)
        return self.fc(self.norm(x))

# ── Load ──────────────────────────────────────────────────────────────────────
with open(vocab_path, encoding='utf-8') as f:
    vocab = json.load(f)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
ckpt   = torch.load(ckpt_path, map_location=device, weights_only=False)
a      = ckpt['args']

model = TransformerDiacritizer(
    vocab_size = len(vocab),
    embed_dim  = a['embed_dim'],
    n_heads    = a['n_heads'],
    n_layers   = a['n_layers'],
    ffn_dim    = a['ffn_dim'],
).to(device)
model.load_state_dict(ckpt['model_state'])
model.eval()

# ── Inference ─────────────────────────────────────────────────────────────────
def diacritize(text: str, max_len: int = 256) -> str:
    raw   = ''.join(c for c in text if c not in TASHKEEL_SET)
    chars = list(raw[:max_len])
    if not chars:
        return text
    unk     = vocab.get('<UNK>', 1)
    ids     = torch.tensor([[vocab.get(c, unk) for c in chars]], dtype=torch.long, device=device)
    lengths = torch.tensor([len(chars)])
    with torch.no_grad():
        pred = model(ids, lengths).argmax(-1)[0].cpu().tolist()
    return ''.join(c + LABEL_TO_DIAC.get(l, '') for c, l in zip(chars, pred))

# ── Example ───────────────────────────────────────────────────────────────────
print(diacritize("الحمد لله رب العالمين"))
# → الْحَمْدُ لِلَّهِ رَبِّ الْعَالَمِينَ
```

### Command-line inference

Download `inference.py`, `best_model.pt`, and `vocab.json` into the same directory, then:

```bash
# Single sentence
python inference.py --checkpoint best_model.pt --text "ذهب الولد إلى المدرسة"

# File (one sentence per line)
python inference.py --checkpoint best_model.pt --file input.txt

# Interactive mode
python inference.py --checkpoint best_model.pt
```

## Training

Trained on a subset of the [Shamela](https://shamela.ws) classical Islamic library, preprocessed into character-aligned diacritic label sequences. Training used a cross-entropy loss over 15 diacritic classes (no diacritic, fatha, damma, kasra, tanwin variants, shadda combinations).

## Limitations

- Max input length is 256 characters; longer texts should be chunked at sentence boundaries.
- Primarily trained on classical Islamic Arabic (Shamela corpus); performance on dialectal or heavily colloquial Arabic may be lower.
- Does not handle hamza placement or orthographic normalization.

## License

MIT