---
license: other
license_name: free-non-commercial
license_link: LICENSE
extra_gated_heading: "Usage agreement: free, non-commercial use only"
extra_gated_button_content: "I agree to these terms"
extra_gated_prompt: "This model is shared freely for the sake of Allah. By requesting access you agree: (1) you may use, fine-tune and redistribute it and its outputs ONLY in applications that are FREE to end users; (2) you may NOT sell it, place it behind a paid subscription or paywall, monetize it with ads, or earn any revenue from an app or service that uses this model or its outputs; (3) these terms pass on to anyone you share it with. وَما أَسأَلُكُم عَلَيهِ مِن أَجرٍ إِن أَجرِيَ إِلّا عَلىٰ رَبِّ العالَمِين"
extra_gated_fields:
  Full name: text
  Organization or app (if any): text
  "I will use this model and its outputs ONLY in free apps (no sales, subscriptions, ads, or revenue)": checkbox
  "I will keep it free to end users and pass these terms on to anyone I share it with": checkbox
language:
- ar
library_name: icefall
pipeline_tag: automatic-speech-recognition
datasets:
- Muno459/arabic-broad-phonemes
- Muno459/quran-phonemes
- obadx/muaalem-annotated-v3
tags:
- arabic
- quran
- phoneme
- ctc
- zipformer
- streaming
- dialect
- tajweed
- pronunciation
- noise-robust
- onnx
---

# zipformer_p-arabic-v2

A streaming phoneme recognizer for Arabic. It maps audio to phoneme units (consonant plus haraka)
representing what was actually pronounced, including dialect and tajweed.

**Scope.** This is a general Arabic model. It covers everyday speech (MSA, broadcast, and Egyptian,
Gulf, Levantine and Maghrebi dialect) as well as Qur'anic recitation with tajweed. Qur'an-only models
typically degrade sharply on ordinary Arabic; this model is trained for both domains in one.

## Comparison with the models it replaces

| | **this model** | [`zipformer_p-quran`](https://huggingface.co/Muno459/zipformer_p-quran)<br>*(Qur'an specialist)* | [`zipformer_p-arabic`](https://huggingface.co/Muno459/zipformer_p-arabic)<br>*(v1, broad)* |
|---|---|---|---|
| Qur'an, real phone audio (held-out PER) | **8.14%** | 13.39% | 34.72% |
| Qur'an, clean recitation (PER) | **3.31%** | 5.82% | not applicable |
| General Arabic (khutbah, CER) | **4.7%** | 22.4% | 6.9% |
| General Arabic (khutbah, WER) | **9.4%** | 36.9% | 11.4% |

The Qur'an specialist is accurate on clean recitation but loses ground on real phone audio, and its
tajweed markers transfer poorly to everyday speech, which is what the general Arabic rows show.
Training both domains jointly retained Qur'an accuracy and improved general Arabic.

## Specification

| | |
|---|---|
| Architecture | Zipformer2 (icefall/k2), causal streaming, CTC |
| Parameters | 64.7M |
| Vocabulary | 250 phoneme units plus blank |
| Features | 80-bin kaldi fbank (povey), 16 kHz |
| Training | from scratch, 10 epochs, 2,039 h unique audio, last 3 epochs averaged |
| Streaming | `decode_chunk_len` 48, chunk 24, left context 256 |
| Formats | PyTorch, ONNX fp32, ONNX int8, CoreML fp16 and fp32 |

## Benchmarks

### Official Qur'an-Lab benchmark (600 clips)

Scored with the official `score.py` normalisation. Phonemes are converted to text by nearest-ayah
retrieval, the same method used for the published `zipformer_p-quran` entry.

| Metric | **this model** | `zipformer_p-quran` |
|---|---|---|
| Overall WER | **5.64** | 5.83 |
| Phone WER (`tlog_holdout`) | **7.56** | 7.76 |
| EveryAyah WER | 7.72 | **7.64** |
| QUL WER | **2.19** | 2.66 |
| WER (alef-insensitive) | **4.40** | 4.59 |

### Phoneme error rate

| Test set | **this model** | `zipformer_p-quran` |
|---|---|---|
| Held-out real-phone Qur'an (793 clips, deterministic gold) | **8.14%** | 13.39% |
| Clean Qur'an recitation (400 clips) | **3.31%** | 5.82% |
| Benchmark clips with gold phonemes (492) | **8.26%** | 11.54% |

PER is the appropriate metric for this model, and the two tables measure different things. Nearest-ayah
retrieval constrains output to one of about 9,112 ayat, so it corrects most acoustic errors before WER
is computed: retrieval selects the correct ayah 99.8% of the time for this model against 99.6% for the
specialist. With that little headroom, leaderboard WER compresses a 28% relative PER improvement into a
3% WER improvement. Use PER to compare acoustic quality and WER to compare end-to-end retrieval systems.

### General Arabic (non-Qur'an)

Reverberant YouTube khutbah (MSA sermon), phonemes converted to text by an LLM, scored against
ElevenLabs Scribe:

| | WER | CER |
|---|---|---|
| **this model** | **9.4%** | **4.7%** |
| `zipformer_p-arabic` (v1) | 11.4% | 6.9% |
| `zipformer_p-quran` | 36.9% | 22.4% |

## Noise robustness

v1 was trained on clean, un-augmented audio. v2 applies three augmentations on GPU every batch,
ramped in over the first 6,000 steps:

| Augmentation | Setting | Purpose |
|---|---|---|
| MUSAN additive noise | 1,983 clips (noise, music, babble), p=0.5, SNR 10 to 20 dB | rooms, traffic, crowds, PA systems, phone audio |
| Speed perturbation | 0.9x, 1.0x, 1.1x | fast and slow delivery |
| SpecAugment | 2 frequency masks (max 27 bins), 2 time masks (max 15% total) | reduces over-fitting to specific frames and bands |

The effect is largest where conditions are worst. On 793 held-out real phone recordings v2 scores
8.14% PER against 13.39% for the Qur'an specialist, a 39% relative reduction. On clean studio
recitation the margin narrows to 3.31% against 5.82%.

MUSAN is narrowband: median 99% energy bandwidth is about 3.2 kHz, with under 0.6% of energy above
6 kHz. The model is therefore well hardened against low and mid frequency interference, which covers
most real-world noise, but has seen comparatively little high frequency hiss.

## Training data

2,039 h of unique audio.

**Broad Arabic, 1,527 h** (MSA, broadcast, and dialect coverage)

| Source | Hours | Share | Contribution |
|---|---|---|---|
| MASC | 859.8 h | 56.3% | dialectal YouTube speech (Egyptian, Gulf, Levantine) |
| SADA | 254.7 h | 16.7% | Saudi broadcast and drama, dialectal |
| MGB2 | 225.7 h | 14.8% | Al Jazeera broadcast MSA |
| `mostafa_mahmoud` | 67.2 h | 4.4% | Egyptian colloquial broadcast speech |
| CommonVoice | 64.8 h | 4.2% | read MSA |
| ArVoice | 49.1 h | 3.2% | read and expressive |
| FLEURS | 5.8 h | 0.4% | read MSA |

**Qur'an, 512 h**

| Source | Hours | Share | Contribution |
|---|---|---|---|
| EveryAyah | 392.0 h | 76.6% | multi-reciter studio recitation |
| `tlog` (tier S) | 85.2 h | 16.6% | real-phone recitation: phones, rooms, background noise |
| muaalem | 27.4 h | 5.4% | tajweed-annotated recitation |
| `tlog` (tiers A and B) | 6.8 h | 1.3% | lower-confidence labels, excluded from this run |

EveryAyah and muaalem are repeated 4x per epoch. The effective training mix is 3,290 h per epoch:

| Component | Effective hours | Share of epoch |
|---|---|---|
| Clean studio Qur'an (EveryAyah) | 1,568 h | 47.7% |
| Broad Arabic | 1,527 h | 46.4% |
| Tajweed-annotated Qur'an (muaalem) | 110 h | 3.3% |
| Real-phone Qur'an (`tlog` tier S) | 85 h | 2.6% |

Real-phone Qur'an is a small fraction of the mix, so the noise robustness reported above comes
predominantly from the augmentation and from the acoustic variety of the broad Arabic corpora rather
than from volume of in-domain noisy recitation.

For comparison, `zipformer_p-quran` was trained almost entirely on Qur'an, dominated by clean studio
recitation. Applications tuned against that model may need their thresholds recalibrated for v2, since
clean studio audio accounts for a smaller share of this model's training distribution.

### Labels

The audio above comes from public and third-party corpora. The phoneme labelling is ours. No public
Arabic phoneme corpus exists at this scale, so every phoneme label used in training was produced for
this project and released as two datasets:

- **[`quran-phonemes`](https://huggingface.co/datasets/Muno459/quran-phonemes)**, deterministic.
  Generated rule-based with `quran-transcript` (the generator behind the muaalem gold), so tajweed is
  exact rather than inferred: madd lengths, ghunna, idgham and qalqala follow Hafs rules.
- **[`arabic-broad-phonemes`](https://huggingface.co/datasets/Muno459/arabic-broad-phonemes)**,
  LLM-phonemized and source-aware. Dialectal sources are prompted for dialectal vowels and MSA sources
  for MSA, preserving dialect rather than normalising it. Being LLM-generated it carries some noise,
  mainly vowel-length artifacts.
- **Quality tiering.** The `tlog` real-phone clips are tiered S, A and B by teacher-label confidence
  and weighted accordingly, so noisier labels count for less.

The `tlog_holdout` evaluation set is excluded from training, so the real-phone figures above are
measured on unseen clips of the same domain.

The 41-symbol Qur'an alphabet is a superset of the 37-symbol broad alphabet, so both are covered by one
250-unit tokenizer.

## Files

| File | Size | Contents |
|---|---|---|
| `zipformer_p_arabic_v2.pt` | 259 MB | PyTorch weights (64.7M params) plus `blank_id` |
| `zipformer_p_arabic_v2.onnx` | 263 MB | cache-aware streaming CTC, sherpa-onnx compatible |
| `zipformer_p_arabic_v2.int8.onnx` | 73 MB | same graph, dynamic INT8, for mobile and edge |
| `zipformer_p_arabic_v2.streaming.ane.mlpackage` | 130 MB | CoreML fp16, streaming, Apple Neural Engine, iOS 17+ |
| `zipformer_p_arabic_v2.streaming.mlpackage` | 258 MB | CoreML fp32, streaming, iOS 17+ (reference and CPU/GPU fallback) |
| `tokens.txt` | 3 KB | symbol table for decoding |
| `phoneme_units.json` | 3 KB | training-time tokenizer (greedy longest match) |
| `decode_with_confidence.py` | | per-phoneme confidence, alternatives and timings |
| `LICENSE` | | free / non-commercial terms |

### Symbol table

Decode with `tokens.txt`. It is authoritative and is the format sherpa-onnx expects.

`phoneme_units.json` is a training-time artifact whose indices do not describe the model's output
layer. It numbers `<blank>` as 0 and the phonemes 1 to 250, whereas the model emits phonemes at 0 to
249 with blank at 250. A symbol table built from the JSON is therefore shifted by one symbol. The model
loads, decodes and emits well-formed Arabic in that state, so the failure is silent and appears only as
degraded accuracy. Thanks to the QuranTracker team for identifying and reporting this.

Blank at 250 is intentional and matches `zipformer_p-quran`. It will not change without a corresponding
`tokens.txt` published alongside, so read the blank id from the file rather than hard-coding it.

### ONNX

The ONNX graphs are chunk-by-chunk streaming with cache states, so they are not bit-identical to
full-utterance PyTorch decoding. Accuracy on held-out real-phone Qur'an:

| Runtime | PER |
|---|---|
| PyTorch, offline full-utterance | 8.83% |
| ONNX fp32, streaming | 10.51% |
| ONNX int8, streaming | 10.74% |

The 1.7 point difference is the expected cost of limited lookahead relative to full-utterance context.
INT8 costs a further 0.23 points while being 3.6x smaller, making it the default choice on device.

Streaming configuration is stored in the ONNX metadata (`decode_chunk_len=48`, `T=61`,
`left_context_len=256,128,64,32,64,128`), so sherpa-onnx reads it automatically.

### CoreML

Both CoreML builds are cache-aware streaming, produced from the same icefall wrapper as the streaming
ONNX, so CoreML and ONNX perform the identical per-chunk computation. Nothing is padded to a fixed
utterance length.

Use `.streaming.ane.mlpackage` (fp16), since the Apple Neural Engine executes fp16 only. The fp32
package is a reference and CPU/GPU fallback.

Interface, per call:

| | |
|---|---|
| Input `x` | `(1, 61, 80)`, one chunk of 80-bin kaldi fbank (povey, 16 kHz) |
| Input `state_0 .. state_97` | 98 cache tensors, zero-initialised on the first call |
| Output `log_probs` | `(1, 12, 251)` |
| Output `new_state_0 .. new_state_97` | carry into the next call |

fp16 precision was verified rather than assumed:

| Check | Result |
|---|---|
| Peak activation across every leaf module, on real audio | 1,042.7, which is 0.016x the fp16 ceiling of 65,504. No module exceeds the limit. |
| fp16 against fp32 decoding, 15 real phone clips | 15/15 identical phoneme strings, 0 blank outputs |

A FastConformer model elsewhere in this project does blank on ANE in fp16, because its
`RelPositionalEncoding` multiplies activations by sqrt(512) and lifts peaks near 5,200 to about
118,000. Zipformer2 has no such multiply; it uses `CompactRelPositionalEncoding`, and the measured
headroom above is roughly 63x, so that failure mode does not apply here.

coremltools cannot execute a model outside macOS, and these packages were built on Windows and WSL.
They are conversion-verified and precision-verified, but not runtime-verified on Apple hardware. The
ONNX files were measured end to end against gold labels. When deploying on device, check the first
outputs against the ONNX or PyTorch reference.

## Per-phoneme confidence, alternatives and timings

Greedy CTC keeps the argmax and discards the rest of the 251-way posterior. For recitation grading that
discarded information is useful, because low model certainty and a genuine mispronunciation are
indistinguishable in a plain transcript. `decode_with_confidence.py` retains it, so every emitted
phoneme carries a confidence, its runner-up candidates and a time span.

```
$ python decode_with_confidence.py clip.wav --topk 2

ءِننننَرَببَهُ۾۾۾بِهِميَومَءِذِللَخَبِۦۦر

         time  ph       conf  margin  alternatives
  1.20- 1.24  ننننَ   1.000   1.000  <blank>:0.00  نَ:0.00
  4.52- 4.60  ءِ      0.790   0.581  <blank>:0.21  ا:0.00
  5.44- 5.48  خَ      1.000   0.999  حَ:0.00  <blank>:0.00
  5.72- 5.80  بِ      0.894   0.789  <blank>:0.10  رِ:0.00
```

Each entry gives `phoneme`, `confidence` (0 to 1), `margin` (confidence minus runner-up), `start` and
`end` in seconds, and `alternatives`. Pass `--json out.json` for machine-readable output.

CTC posteriors are peaky. On the clip above the mean confidence is 0.983 and no value falls below 0.5,
so a fixed `confidence < 0.5` gate will rarely fire and high confidence is not evidence of correctness.
Two signals carry more information:

- **`margin` rather than `confidence`.** A phoneme at 0.79 with `<blank>` at 0.21 is a contested
  position; one at 1.000 with all alternatives at 0.00 is not.
- **The identity of the runner-up.** `<blank>` competing indicates uncertainty that anything was said
  at that position (onset, pause, clipped sound). A competing phoneme, for example خَ against حَ,
  indicates a sound was heard but its identity is ambiguous, which is the useful case for pronunciation
  feedback.

Thresholds are not calibrated probabilities and should be tuned on your own audio. For true calibration,
temperature-scale the logits against a labelled set of your own recordings.

Timing resolution is one encoder frame, 40 ms. This separates phonemes reliably but is coarse for
grading a precise 2-harakah hold, so treat spans as boundaries rather than fine duration measurements.

## Usage

icefall `Zipformer2`, 80-bin kaldi fbank (povey window) at 16 kHz, CTC greedy decode to phoneme units.

Use `torchaudio.compliance.kaldi.fbank`, not `torchaudio.transforms.MelSpectrogram`. Slaney-mel
features roughly double the error rate (measured: 11.4% to 22.1% WER on identical audio) without
producing any error, and are the most common cause of poor results with this model.

```python
import torch, torchaudio.compliance.kaldi as kaldi

feats = kaldi.fbank(wav, num_mel_bins=80, frame_length=25.0, frame_shift=10.0,
                    sample_frequency=16000, dither=0.0, snip_edges=False,
                    window_type="povey", use_energy=False, low_freq=20.0,
                    high_freq=-400.0, remove_dc_offset=True, preemphasis_coefficient=0.97)
```

For the 1000 ms look-ahead profile use `chunk_size=(24,)` and `left_context_frames=(256,)`.

The ONNX files use the standard sherpa-onnx online zipformer2-CTC format (`model_type=zipformer2`), so
they load directly in sherpa-onnx, and in plain onnxruntime by feeding one chunk plus the cache tensors
and carrying the returned states forward.

When streaming ends, flush the buffer by passing one final chunk of silence (at least the right-context
length, about 1.05 s). Audio still held in the right-context buffer is not emitted otherwise, which
causes short final words to be dropped.

## Limitations

- **Phonemes, not text.** Converting Arabic phonemes to orthography is a separate problem. Evaluate
  this model by PER rather than word-level text WER.
- **Retrieval is a Qur'an-lexicon constraint.** The leaderboard WER above uses nearest-ayah matching
  and does not transfer to general Arabic.
- **Dialectal orthography** is out of scope. The model transcribes dialect faithfully at the phonetic
  level, but reconstructing fluent dialectal text from it is a separate task.
- Broad-Arabic labels are LLM-generated and carry some vowel-length noise.
- Emphatic consonants (ط ض ظ ص) degrade on reciters absent from training. Measured on a held-out unseen
  reciter, ط reaches 24.4% error against 2.6% for ت, while on reciters seen in training ط is 9.3%. The
  cause is speaker generalisation rather than noise, and reciter diversity is the effective remedy.
- The CoreML builds have not been executed on Apple hardware, since coremltools cannot run a model
  outside macOS. They are conversion-verified and precision-verified only.

## License

Shared freely for the sake of Allah under free, non-commercial terms, not Apache-2.0. Access is gated:
free apps only, no sales, no paid subscriptions or paywalls, no ad revenue, and no monetization of any
kind. These terms pass on to anyone you share it with. See `LICENSE`.