Cohere-ar-tashkeel: Restoring Arabic Diacritics — and Grammar — from Speech

Written Arabic usually drops its short vowels. The reader restores them from context — but machines, and language learners, often can't. Diacritization (تشكيل) puts those vowels back: the fatḥa, ḍamma, kasra, sukūn, shadda, tanwīn, and the all-important case-endings (إعراب) that carry grammatical meaning.

Almost every diacritizer today works from text. That is a fundamentally hard problem, because the same unvoweled skeleton can be read many valid ways. Cohere-ar-tashkeel takes a different route: it restores diacritics from the audio — it listens to how the words are pronounced and writes the vowels down. Built on Cohere Labs' Arabic speech model, it reached 95.3% diacritic accuracy and, in its first day on the Hub, 1,772 downloads.

This post covers what it does, how it was fine-tuned, how to deploy it, and two applications we're especially excited about: Qur'anic recitation checking and automatic إعراب (grammatical) annotation.

---

Highlights

• 🎧 Speech → fully-diacritized Arabic text, including case-endings (iʿrāb).
• 🏗️ Built on CohereLabs/cohere-transcribe-arabic-07-2026 (~2B params) via lightweight LoRA fine-tuning.
• 📈 95.30% diacritic accuracy, 6.62% DER (with case-endings) / 5.67% (without) on a held-out natural-speech set.
• 🧠 The purpose-built win is on case-endings (iʿrāb) — the part text-only systems get most wrong.
• 🚀 Self-contained transformers model; runs in bf16 on a single GPU, or 4-bit for lighter deployment.
• 👉 Model: https://huggingface.co/Omartificial-Intelligence-Space/Cohere-ar-tashkeel

---

Why diacritize from speech?

Consider the bare skeleton علم. Depending on the vowels, it can be:

A text-only diacritizer must guess which one was intended. It has nothing but surrounding text to go on, and for case-endings — which depend on syntactic role — it is guessing at grammar. This is exactly why text-only systems, however good, hit a ceiling: the information simply isn't in the input.

Speech carries that information. When a word is spoken, its vowels and its final case-ending are acoustically present. Cohere-ar-tashkeel exploits this: it hears عَلَّمَ vs عَلِمَ and transcribes the vowels it actually heard. The ambiguity that stumps text models is resolved by the audio itself.

---

Results

We evaluate on a held-out set of 260 natural Arabic speech clips, comparing three fine-tuning strategies (described below). The released model is "Mixed."

When the model hears a letter correctly, it places the right diacritic 95.3% of the time. Only about 1 diacritic in 15 is wrong even including the tricky grammatical endings. Diacritized WER is strict by construction (one wrong vowel fails the word), so it reads high — DER and diacritic accuracy are the meaningful diacritization metrics.

---

The إعراب (iʿrāb) advantage

Case-endings are where Arabic diacritization is hardest. The final vowel of a word (رَفْع ḍamma, نَصْب fatḥa, جَرّ kasra, جَزْم sukūn) encodes its grammatical role — subject, object, genitive — and is precisely what a text model cannot recover reliably.

Because our model hears these endings, it shrinks the "case-ending penalty" — the extra error contributed by word-final marks — by 21% versus a natural-speech-only baseline:

That is the headline capability: grammar you can hear, transcribed.

---

How we fine-tuned it

The base model is a strong Arabic/English speech recognizer; our job was to teach it to emit fully-diacritized output without hurting its transcription quality. We did this with parameter-efficient LoRA fine-tuning and a small set of controlled experiments.

Setup

• Adapter: LoRA, rank r = 16, alpha = 32, dropout 0.05, applied to all linear projections (attention q/k/v/o, the conformer feed-forwards, and the relative-position projections) — ~33M trainable params (1.6%) of the ~2B base.
• Precision & memory: bf16, gradient checkpointing, AdamW (β = 0.9/0.98), cosine LR schedule with warmup, on a single 32 GB GPU.
• Objective: standard next-token cross-entropy on the diacritized target, with the decoder prompted for Arabic + punctuation so the model learns to always produce harakāt.
• Selection: dev loss tracked each epoch on a held-out natural-speech set; best checkpoint kept.

The experiments

We compared three recipes, all evaluated on the same dev set:

Lessons worth sharing

• Mix, don't stage. Sequentially pre-training then adapting let the model drift to the read-speech (MSA) domain, hurting dialectal recognition. Keeping the target-domain data in the mix — oversampled — preserved it while still importing the diacritization signal.
• Augmentation matters for single-voice data. Read-speech corpora are often one narrator; gain/speed/noise augmentation reduced over-fitting to that voice.
• Don't trust dev-loss alone. Augmentation inflates token cross-entropy, yet the augmented model had the best DER. We only saw this by evaluating the real metric — a reminder to score what you actually care about, not a proxy.
• Negative results are results. The two-stage drift and the dev-loss/DER mismatch shaped the final recipe as much as the wins did.

---

Deployment guide

The released model is self-contained — the adapter is merged into the base, so it loads exactly like a normal transformers speech model.

Quick start

import torch, soundfile as sf, librosa
from transformers import AutoProcessor, CohereAsrForConditionalGeneration

repo = "NAMAA-Space/Cohere-Speech-Tashkeel-2B"
proc = AutoProcessor.from_pretrained(repo)
model = CohereAsrForConditionalGeneration.from_pretrained(
    repo, dtype=torch.bfloat16, device_map="auto"
).eval()

def diacritize(path):
    wav, sr = sf.read(path, dtype="float32")
    if wav.ndim > 1:
        wav = wav.mean(1)
    if sr != 16000:
        wav = librosa.resample(wav, orig_sr=sr, target_sr=16000)
    feats = proc.feature_extractor([wav], sampling_rate=16000, return_tensors="pt",
                                   padding="longest", return_attention_mask=True)
    aci = feats.pop("audio_chunk_index")
    prompt = proc.get_decoder_prompt_ids(language="ar", punctuation=True)
    dii = torch.tensor([prompt], dtype=torch.long, device=model.device)
    with torch.no_grad():
        out = model.generate(
            input_features=feats["input_features"].to(model.device, torch.bfloat16),
            attention_mask=feats["attention_mask"].to(model.device),
            decoder_input_ids=dii, max_new_tokens=448)
    res = proc.decode(out, skip_special_tokens=True, audio_chunk_index=aci, language="ar")
    return res[0] if isinstance(res, (list, tuple)) else res  # one utterance in, one out

print(diacritize("ayah.wav"))

Batched inference

For throughput, feed a list of waveforms — the feature extractor pads to the longest, and you replicate the prompt across the batch:

prompt = proc.get_decoder_prompt_ids(language="ar", punctuation=True)
feats = proc.feature_extractor(wavs, sampling_rate=16000, return_tensors="pt",
                               padding="longest", return_attention_mask=True)
aci = feats.pop("audio_chunk_index")
dii = torch.tensor([prompt] * feats["input_features"].shape[0], device=model.device)
out = model.generate(input_features=feats["input_features"].to(model.device, torch.bfloat16),
                     attention_mask=feats["attention_mask"].to(model.device),
                     decoder_input_ids=dii, max_new_tokens=448)
# decode reassembles chunks and returns one string per input utterance
texts = proc.decode(out, skip_special_tokens=True, audio_chunk_index=aci, language="ar")

Lighter deployment (4-bit)

For memory-constrained GPUs, load in NF4 4-bit — the same scheme is ~near-lossless on the base model:

import torch
from transformers import BitsAndBytesConfig, CohereAsrForConditionalGeneration

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16,
                         bnb_4bit_use_double_quant=True)
model = CohereAsrForConditionalGeneration.from_pretrained(
    repo, quantization_config=bnb, device_map="auto")

Practical tips

• Audio: 16 kHz mono. Resample anything else (as above).
• Long files: split into ≤ ~30 s segments (ideally on silence via a VAD), diacritize each, and stitch. A noise gate helps on noisy input.
• Serving: wrap diacritize() in a FastAPI endpoint or a transformers pipeline behind a queue. The model uses the custom CohereAsr architecture in transformers, so no trust_remote_code is needed; for a dedicated high-throughput server, keep it on transformers/TGI rather than assuming vLLM support.
• Hardware: comfortably fits a single 16–24 GB GPU in bf16; less in 4-bit.

---

Application 1 — Qur'anic recitation checking

Qur'anic recitation (تلاوة) is judged partly by correct vowels and case-endings. A لحن جليّ (clear error) — pronouncing a ḍamma where the text has a fatḥa, or getting a case-ending wrong — changes the grammar and is treated seriously. These are exactly the errors a speech diacritizer can surface.

The workflow:

1. Transcribe the reciter's audio → diacritized text (what they actually voiced).
2. Align it, letter-by-letter, to the canonical diacritized text of the āyah.
3. Diff the harakāt — where the reciter's vowel/case-ending differs from the reference, flag it.

This flags vowel and case-ending mismatches — the model's strongest suit. A few honest caveats:

• It targets laḥn jaliyy (harakāt / iʿrāb / letter errors), not the fine timing rules of tajwīd (madd length, ghunna nasalization, qalqala) — those are laḥn khafiyy and need dedicated acoustic modeling.
• Treat it as a teaching aid / first-pass screener with a human in the loop, never as a religious authority. For sacred text especially, surface flags for a qualified teacher (مُقْرِئ) to confirm.

Even so, an automatic "you said كِتابُ but the āyah is كِتابَ" is genuinely useful for learners and self-study apps.

Application 2 — Automatic إعراب (grammatical) annotation

Because the model recovers case-endings from speech reliably, its output is a practical bootstrap for iʿrāb — labeling each word's grammatical case:

• ـُ (ḍamma) → رفع (nominative — e.g. subject)
• ـَ (fatḥa) → نصب (accusative — e.g. object)
• ـِ (kasra) → جرّ (genitive — e.g. after a preposition)
• ـْ (sukūn) → جزم / بناء (jussive / indeclinable)

Where this helps:

• Building training data for text-based syntactic parsers and iʿrāb taggers — from spoken corpora, which are abundant, instead of hand-diacritized text, which is scarce.
• Educational tools that show learners the grammatical function of each word from a spoken example.
• Linguistic research on how case-endings are realized (or dropped) across dialects and registers — since the model transcribes what was actually said.

A note on scope: audible case-endings give you the morphological case marker; full syntactic parse (identifying why a word is منصوب) still needs a grammar model on top. This model gives that pipeline a high-quality, speech-grounded first layer.

---

Other use cases

• TTS front-ends — diacritized text is what high-quality Arabic TTS needs; generate it from reference audio.
• Pronunciation feedback for Arabic learners (MSA and Qur'anic).
• Diacritized subtitles / captions for lectures, sermons, and audiobooks.
• ASR post-editing — enrich existing transcripts with vowels for readability and search.

---

Limitations & responsible use

• Diacritization is inferred from pronunciation: on noisy audio or unclear articulation, letters and their diacritics degrade together. Use a VAD / noise gate.
• Optimized for Arabic; no timestamps or speaker diarization (inherited from the base).
• For religious or high-stakes use (Qur'an, legal, educational grading), keep a human in the loop — present the model as an assistant, not an arbiter.

Roadmap

• Wider dialectal coverage for natural-speech evaluation.
• A dedicated tajwīd layer (madd/ghunna/qalqala) to complement the harakāt-level checking.
• Confidence scores per diacritic, to drive better recitation-feedback UX.

---

Links & citation

• Model: https://huggingface.co/NAMAA-Space/Cohere-Speech-Tashkeel-2B
• Base model: CohereLabs/cohere-transcribe-arabic-07-2026

@misc{cohere_ar_tashkeel_2026,
  title  = {Cohere-ar-tashkeel: Arabic Speech Diacritization},
  author = {Omer Nacar},
  year   = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Omartificial-Intelligence-Space/Cohere-ar-tashkeel}},
  note   = {Built on CohereLabs/cohere-transcribe-arabic-07-2026}
}

Built on Cohere Labs' Arabic speech model. Released under Apache 2.0.