visionworks/turath1.0

A lightweight Transformer model for Arabic diacritization (tashkeel), trained on the Shamela classical Islamic text corpus. Designed for speed and accuracy on classical and modern standard Arabic.

Model details

Property	Value
Architecture	Transformer Encoder (6 layers)
Parameters	~8M
Embed dim	256
Attention heads	4
FFN dim	1024
Max sequence length	256 characters
Vocab size	130
Output classes	15 diacritic labels
Training domain	Shamela classical Islamic corpus

Benchmark results

Evaluated on 1,000 held-out sentences from the Shamela test split, compared against publicly available Arabic diacritization systems.

Model	DER ↓	WER ↓	Latency (CPU)
flan-t5-small	3.1%	29.2%	90ms
catt-eo	5.2%	30.1%	31ms
visionworks:turath1.0	7.1%	29.7%	5ms
mishkal	20.6%	85.3%	36ms
camel (MLE)	33.7%	81.0%	6ms

• DER (Diacritic Error Rate): fraction of characters with incorrect diacritics
• WER (Word Error Rate): fraction of words with at least one incorrect diacritic
• Latency measured on CPU (single sentence, no batching)

Despite being 17× faster than flan-t5-small, turath1.0 achieves nearly identical WER (29.7% vs 29.2%), making it well-suited for high-throughput or resource-constrained environments. Mishkal and CAMeL MLE score poorly on this benchmark due to domain mismatch (tuned for modern standard Arabic, not classical texts).

Files

File	Description
best_model.pt	Model weights + training args (PyTorch checkpoint)
vocab.json	Character vocabulary (130 tokens)
inference.py	Self-contained inference script

Usage

Install dependencies

pip install torch

Quick start

import json
import torch
import torch.nn as nn
from huggingface_hub import hf_hub_download

# ── Download files ────────────────────────────────────────────────────────────
ckpt_path  = hf_hub_download("visionworks/turath1.0", "best_model.pt")
vocab_path = hf_hub_download("visionworks/turath1.0", "vocab.json")

# ── Constants ─────────────────────────────────────────────────────────────────
TASHKEEL_SET = set('ًٌٍَُِّْٰٕٓٔ')
LABEL_TO_DIAC = {
    0: '',    1: 'َ',  2: 'ُ',  3: 'ِ',
    4: 'ً',  5: 'ٌ',  6: 'ٍ',  7: 'ْ',  8: 'ّ',
    9: 'َّ', 10: 'ُّ', 11: 'ِّ', 12: 'ًّ', 13: 'ٌّ', 14: 'ٍّ',
}

# ── Model definition ──────────────────────────────────────────────────────────
class TransformerDiacritizer(nn.Module):
    def __init__(self, vocab_size, embed_dim, n_heads, n_layers,
                 ffn_dim, dropout=0.0, max_len=256):
        super().__init__()
        self.embed     = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        self.pos_embed = nn.Embedding(max_len + 2, embed_dim)
        self.drop      = nn.Dropout(dropout)
        self.layers    = nn.ModuleList([
            nn.TransformerEncoderLayer(
                d_model=embed_dim, nhead=n_heads, dim_feedforward=ffn_dim,
                dropout=dropout, batch_first=True, norm_first=True,
            ) for _ in range(n_layers)
        ])
        self.norm = nn.LayerNorm(embed_dim)
        self.fc   = nn.Linear(embed_dim, 15)

    def forward(self, ids, lengths):
        B, T     = ids.shape
        pos      = torch.arange(T, device=ids.device).unsqueeze(0).expand(B, -1)
        x        = self.drop(self.embed(ids) + self.pos_embed(pos))
        pad_mask = torch.arange(T, device=ids.device)[None, :] >= lengths[:, None]
        for layer in self.layers:
            x = layer(x, src_key_padding_mask=pad_mask)
        return self.fc(self.norm(x))

# ── Load ──────────────────────────────────────────────────────────────────────
with open(vocab_path, encoding='utf-8') as f:
    vocab = json.load(f)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
ckpt   = torch.load(ckpt_path, map_location=device, weights_only=False)
a      = ckpt['args']

model = TransformerDiacritizer(
    vocab_size = len(vocab),
    embed_dim  = a['embed_dim'],
    n_heads    = a['n_heads'],
    n_layers   = a['n_layers'],
    ffn_dim    = a['ffn_dim'],
).to(device)
model.load_state_dict(ckpt['model_state'])
model.eval()

# ── Inference ─────────────────────────────────────────────────────────────────
def diacritize(text: str, max_len: int = 256) -> str:
    raw   = ''.join(c for c in text if c not in TASHKEEL_SET)
    chars = list(raw[:max_len])
    if not chars:
        return text
    unk     = vocab.get('<UNK>', 1)
    ids     = torch.tensor([[vocab.get(c, unk) for c in chars]], dtype=torch.long, device=device)
    lengths = torch.tensor([len(chars)])
    with torch.no_grad():
        pred = model(ids, lengths).argmax(-1)[0].cpu().tolist()
    return ''.join(c + LABEL_TO_DIAC.get(l, '') for c, l in zip(chars, pred))

# ── Example ───────────────────────────────────────────────────────────────────
print(diacritize("الحمد لله رب العالمين"))
# → الْحَمْدُ لِلَّهِ رَبِّ الْعَالَمِينَ

Command-line inference

Download inference.py, best_model.pt, and vocab.json into the same directory, then:

# Single sentence
python inference.py --checkpoint best_model.pt --text "ذهب الولد إلى المدرسة"

# File (one sentence per line)
python inference.py --checkpoint best_model.pt --file input.txt

# Interactive mode
python inference.py --checkpoint best_model.pt

Training

Trained on a subset of the Shamela classical Islamic library, preprocessed into character-aligned diacritic label sequences. Training used a cross-entropy loss over 15 diacritic classes (no diacritic, fatha, damma, kasra, tanwin variants, shadda combinations).

Limitations

• Max input length is 256 characters; longer texts should be chunked at sentence boundaries.
• Primarily trained on classical Islamic Arabic (Shamela corpus); performance on dialectal or heavily colloquial Arabic may be lower.
• Does not handle hamza placement or orthographic normalization.

License

MIT