README.md

---
license: apache-2.0
language:
- ar
base_model: openbmb/VoxCPM2
pipeline_tag: text-to-speech
tags:
- text-to-speech
- arabic
- najdi
- saudi
- voxcpm
- diacritics
- tashkeel
---

# VoxCPM2 Najdi Arabic TTS (diacritized) — final

Full fine-tune of **VoxCPM2** (2.29B) for **Saudi Najdi Arabic** TTS, trained on
*diacritized* text so that harakat control pronunciation at inference time.

**This checkpoint: step 18264 — final, end of epoch 1** (val loss 0.7848).

## Why diacritics, and why a full fine-tune

VoxCPM2's tokenizer has no diacritic entries, so every harakat falls through to a
byte-fallback pair (`<0xD9><0x8E>` = fatha) whose embedding rows carry a
fusha/Quranic association from pretraining. Token count inflates x1.96 on marked
text.

A LoRA run on this data could not fix that: `apply_lora_to_named_linear_modules`
wraps **Linear only** (q/k/v/o_proj) and freezes everything else, so
`nn.Embedding` is structurally untouchable — attention can only reinterpret
those rows in context, never change them. Full fine-tuning unfreezes everything
except the AudioVAE, so the rows themselves retrain.

## Training data

292,217 rows / 515.8 h, derived from a 377 h / ~1,032-speaker Najdi YouTube corpus:

| portion | rows | note |
|---|---|---|
| diacritized | 204,552 | coverage mean 0.804 |
| long-form (>25 s), bare | 3,835 | diacritization degrades past ~25 s |
| bare **duplicates** of diacritized clips | 83,830 | same audio, marks stripped |

**70.5%** of rows carry marks; 45.0% carry a same-speaker `ref_audio`.

The bare duplicates are the load-bearing part. If bare and marked text sat on
disjoint audio, the model could satisfy both by keying on the voice and learn
nothing about marks. Identical audio carrying both forms leaves one consistent
reading: **marks are optional modifiers of the same speech**.

Diacritics come from
[NAMAA-Space/Cohere-Speech-Tashkeel-2B](https://huggingface.co/NAMAA-Space/Cohere-Speech-Tashkeel-2B),
an **acoustic** diacritizer, so harakat reflect how each word was actually said
rather than what a text model guesses — it stays dialectal
(نِتْسَلَّى, حَتْلَاقُوا, حَيَاخُذ). Because it runs ~10% letter WER its output is
**not** used as the transcript: the corpus transcript is the letter backbone and
only marks transfer word-by-word, leaving misheard words bare. Word-final marks
(i'rab / pausal) are stripped; shadda is kept, being gemination rather than a
case ending.

Transcripts are Cohere ASR text, not raw auto-captions.

## Training

Stock `openbmb/VoxCPM2`, one epoch, 18,264 steps at effective batch 16
(4 x 4 grad-accum), lr 1e-5, weight_decay 0.01, warmup 100, max_grad_norm 1.0,
lambdas 1.0/1.0. Single H200.

Validation loss: 0.9689 (step 0) -> 0.8033 (2k) -> 0.7852 (8k) -> **0.7834 (15k, best)**
-> 0.7848 (18k). Essentially flat past step 8,000.

## Inference

Output is **48 kHz** (`model.tts_model.sample_rate`), not 16 kHz — writing it
with a 16 kHz header plays ~3x slow and about an octave and a half low.

Short text works through the standard CLI:

```bash
voxcpm clone --model-path <this-repo> \
  --text "وِش رَايِك نِتْقَابَل بَاكِر بِالْمَقْهَى اللِّي جَنْب الْبَيْت" \
  --reference-audio ref.wav --no-denoiser --output out.wav
```

**Long text must be chunked.** A single autoregressive pass past ~20 s degenerates
into syllable babble in its tail. Measured on a 47 s single-shot generation:
transcribe-back WER 0.41-0.48, with output like
`بي بي بييسوشايشاي ... بيطا بيطا ويطبي طبيطبير`. Insertions and deletions were
both zero, so this is accumulated drift, not a stop-head failure.

Splitting on sentence boundaries fixes it:

| split | chunks | transcribe-back WER |
|---|---|---|
| single-shot 47 s | 1 | 0.410-0.476 |
| comma + word fallback | 8 | 0.105 |
| full stops / question marks only | 16 | **0.069** |

Chunks up to ~15 s measured clean (WER 0.040-0.100). `cfg_value` 2.0 beat 1.5 and
1.3 (0.250 vs 0.500) — do not lower it.

## Known limitations

- **Inference text must be diacritized in the same convention as training**:
  pausal, word-final marks stripped, dialectal vowels. Text carrying standard MSA
  i'rab endings is off-distribution. There is no text-side diacritizer in this
  stack — the one used to build the data is acoustic and needs audio.
- Mark coverage is ~80%, so sentences are internally mixed (some words bare).
  That is realistic input, not a defect.
- Reference audio in training is capped at 10.0 s; longer references are
  untested territory.
- Trained on Najdi. The gaf (ق pronounced /g/) is a deliberate register target,
  not an error.