Rabe3/VoxCPM_mid_najdi

VoxCPM2 Najdi Arabic TTS (diacritized) — final

Full fine-tune of VoxCPM2 (2.29B) for Saudi Najdi Arabic TTS, trained on diacritized text so that harakat control pronunciation at inference time.

This checkpoint: step 18264 — final, end of epoch 1 (val loss 0.7848).

Why diacritics, and why a full fine-tune

VoxCPM2's tokenizer has no diacritic entries, so every harakat falls through to a byte-fallback pair (<0xD9><0x8E> = fatha) whose embedding rows carry a fusha/Quranic association from pretraining. Token count inflates x1.96 on marked text.

A LoRA run on this data could not fix that: apply_lora_to_named_linear_modules wraps Linear only (q/k/v/o_proj) and freezes everything else, so nn.Embedding is structurally untouchable — attention can only reinterpret those rows in context, never change them. Full fine-tuning unfreezes everything except the AudioVAE, so the rows themselves retrain.

Training data

292,217 rows / 515.8 h, derived from a 377 h / ~1,032-speaker Najdi YouTube corpus:

portion	rows	note
diacritized	204,552	coverage mean 0.804
long-form (>25 s), bare	3,835	diacritization degrades past ~25 s
bare duplicates of diacritized clips	83,830	same audio, marks stripped

70.5% of rows carry marks; 45.0% carry a same-speaker ref_audio.

The bare duplicates are the load-bearing part. If bare and marked text sat on disjoint audio, the model could satisfy both by keying on the voice and learn nothing about marks. Identical audio carrying both forms leaves one consistent reading: marks are optional modifiers of the same speech.

Diacritics come from NAMAA-Space/Cohere-Speech-Tashkeel-2B, an acoustic diacritizer, so harakat reflect how each word was actually said rather than what a text model guesses — it stays dialectal (نِتْسَلَّى, حَتْلَاقُوا, حَيَاخُذ). Because it runs ~10% letter WER its output is not used as the transcript: the corpus transcript is the letter backbone and only marks transfer word-by-word, leaving misheard words bare. Word-final marks (i'rab / pausal) are stripped; shadda is kept, being gemination rather than a case ending.

Transcripts are Cohere ASR text, not raw auto-captions.

Training

Stock openbmb/VoxCPM2, one epoch, 18,264 steps at effective batch 16 (4 x 4 grad-accum), lr 1e-5, weight_decay 0.01, warmup 100, max_grad_norm 1.0, lambdas 1.0/1.0. Single H200.

Validation loss: 0.9689 (step 0) -> 0.8033 (2k) -> 0.7852 (8k) -> 0.7834 (15k, best) -> 0.7848 (18k). Essentially flat past step 8,000.

Inference

Output is 48 kHz (model.tts_model.sample_rate), not 16 kHz — writing it with a 16 kHz header plays ~3x slow and about an octave and a half low.

Short text works through the standard CLI:

voxcpm clone --model-path <this-repo> \
  --text "وِش رَايِك نِتْقَابَل بَاكِر بِالْمَقْهَى اللِّي جَنْب الْبَيْت" \
  --reference-audio ref.wav --no-denoiser --output out.wav

Long text must be chunked. A single autoregressive pass past ~20 s degenerates into syllable babble in its tail. Measured on a 47 s single-shot generation: transcribe-back WER 0.41-0.48, with output like بي بي بييسوشايشاي ... بيطا بيطا ويطبي طبيطبير. Insertions and deletions were both zero, so this is accumulated drift, not a stop-head failure.

Splitting on sentence boundaries fixes it:

split	chunks	transcribe-back WER
single-shot 47 s	1	0.410-0.476
comma + word fallback	8	0.105
full stops / question marks only	16	0.069

Chunks up to ~15 s measured clean (WER 0.040-0.100). cfg_value 2.0 beat 1.5 and 1.3 (0.250 vs 0.500) — do not lower it.

Known limitations

• Inference text must be diacritized in the same convention as training: pausal, word-final marks stripped, dialectal vowels. Text carrying standard MSA i'rab endings is off-distribution. There is no text-side diacritizer in this stack — the one used to build the data is acoustic and needs audio.
• Mark coverage is ~80%, so sentences are internally mixed (some words bare). That is realistic input, not a defect.
• Reference audio in training is capped at 10.0 s; longer references are untested territory.
• Trained on Najdi. The gaf (ق pronounced /g/) is a deliberate register target, not an error.