Upload folder using huggingface_hub

README.md

@@ -0,0 +1,116 @@
+---
+license: apache-2.0
+language:
+- ar
+base_model: openbmb/VoxCPM2
+pipeline_tag: text-to-speech
+tags:
+- text-to-speech
+- arabic
+- najdi
+- saudi
+- voxcpm
+- diacritics
+- tashkeel
+---
+
+# VoxCPM2 Najdi Arabic TTS (diacritized) — final
+
+Full fine-tune of **VoxCPM2** (2.29B) for **Saudi Najdi Arabic** TTS, trained on
+*diacritized* text so that harakat control pronunciation at inference time.
+
+**This checkpoint: step 18264 — final, end of epoch 1** (val loss 0.7848).
+
+## Why diacritics, and why a full fine-tune
+
+VoxCPM2's tokenizer has no diacritic entries, so every harakat falls through to a
+byte-fallback pair (`<0xD9><0x8E>` = fatha) whose embedding rows carry a
+fusha/Quranic association from pretraining. Token count inflates x1.96 on marked
+text.
+
+A LoRA run on this data could not fix that: `apply_lora_to_named_linear_modules`
+wraps **Linear only** (q/k/v/o_proj) and freezes everything else, so
+`nn.Embedding` is structurally untouchable — attention can only reinterpret
+those rows in context, never change them. Full fine-tuning unfreezes everything
+except the AudioVAE, so the rows themselves retrain.
+
+## Training data
+
+292,217 rows / 515.8 h, derived from a 377 h / ~1,032-speaker Najdi YouTube corpus:
+
+| portion | rows | note |
+|---|---|---|
+| diacritized | 204,552 | coverage mean 0.804 |
+| long-form (>25 s), bare | 3,835 | diacritization degrades past ~25 s |
+| bare **duplicates** of diacritized clips | 83,830 | same audio, marks stripped |
+
+**70.5%** of rows carry marks; 45.0% carry a same-speaker `ref_audio`.
+
+The bare duplicates are the load-bearing part. If bare and marked text sat on
+disjoint audio, the model could satisfy both by keying on the voice and learn
+nothing about marks. Identical audio carrying both forms leaves one consistent
+reading: **marks are optional modifiers of the same speech**.
+
+Diacritics come from
+[NAMAA-Space/Cohere-Speech-Tashkeel-2B](https://huggingface.co/NAMAA-Space/Cohere-Speech-Tashkeel-2B),
+an **acoustic** diacritizer, so harakat reflect how each word was actually said
+rather than what a text model guesses — it stays dialectal
+(نِتْسَلَّى, حَتْلَاقُوا, حَيَاخُذ). Because it runs ~10% letter WER its output is
+**not** used as the transcript: the corpus transcript is the letter backbone and
+only marks transfer word-by-word, leaving misheard words bare. Word-final marks
+(i'rab / pausal) are stripped; shadda is kept, being gemination rather than a
+case ending.
+
+Transcripts are Cohere ASR text, not raw auto-captions.
+
+## Training
+
+Stock `openbmb/VoxCPM2`, one epoch, 18,264 steps at effective batch 16
+(4 x 4 grad-accum), lr 1e-5, weight_decay 0.01, warmup 100, max_grad_norm 1.0,
+lambdas 1.0/1.0. Single H200.
+
+Validation loss: 0.9689 (step 0) -> 0.8033 (2k) -> 0.7852 (8k) -> **0.7834 (15k, best)**
+-> 0.7848 (18k). Essentially flat past step 8,000.
+
+## Inference
+
+Output is **48 kHz** (`model.tts_model.sample_rate`), not 16 kHz — writing it
+with a 16 kHz header plays ~3x slow and about an octave and a half low.
+
+Short text works through the standard CLI:
+
+```bash
+voxcpm clone --model-path <this-repo> \
+  --text "وِش رَايِك نِتْقَابَل بَاكِر بِالْمَقْهَى اللِّي جَنْب الْبَيْت" \
+  --reference-audio ref.wav --no-denoiser --output out.wav
+```
+
+**Long text must be chunked.** A single autoregressive pass past ~20 s degenerates
+into syllable babble in its tail. Measured on a 47 s single-shot generation:
+transcribe-back WER 0.41-0.48, with output like
+`بي بي بييسوشايشاي ... بيطا بيطا ويطبي طبيطبير`. Insertions and deletions were
+both zero, so this is accumulated drift, not a stop-head failure.
+
+Splitting on sentence boundaries fixes it:
+
+| split | chunks | transcribe-back WER |
+|---|---|---|
+| single-shot 47 s | 1 | 0.410-0.476 |
+| comma + word fallback | 8 | 0.105 |
+| full stops / question marks only | 16 | **0.069** |
+
+Chunks up to ~15 s measured clean (WER 0.040-0.100). `cfg_value` 2.0 beat 1.5 and
+1.3 (0.250 vs 0.500) — do not lower it.
+
+## Known limitations
+
+- **Inference text must be diacritized in the same convention as training**:
+  pausal, word-final marks stripped, dialectal vowels. Text carrying standard MSA
+  i'rab endings is off-distribution. There is no text-side diacritizer in this
+  stack — the one used to build the data is acoustic and needs audio.
+- Mark coverage is ~80%, so sentences are internally mixed (some words bare).
+  That is realistic input, not a defect.
+- Reference audio in training is capped at 10.0 s; longer references are
+  untested territory.
+- Trained on Najdi. The gaf (ق pronounced /g/) is a deliberate register target,
+  not an error.

audiovae.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:94b5d51e107e0507d4acc976cfdadb64edd6fd06d1f751dadbf2fd1594274bf1
+size 376951122

config.json

@@ -0,0 +1,67 @@
+{
+    "architecture": "voxcpm2",
+    "lm_config": {
+        "bos_token_id": 1,
+        "eos_token_id": 2,
+        "hidden_size": 2048,
+        "intermediate_size": 6144,
+        "max_position_embeddings": 32768,
+        "num_attention_heads": 16,
+        "num_hidden_layers": 28,
+        "num_key_value_heads": 2,
+        "rms_norm_eps": 1e-05,
+        "rope_theta": 10000,
+        "kv_channels": 128,
+        "rope_scaling": {
+            "type": "longrope", 
+            "long_factor": [0.9977997200264581, 1.014658295992452, 1.0349680404997148, 1.059429246056193, 1.0888815016813513, 1.1243301355211495, 1.166977103606075, 1.2182568066927284, 1.2798772354275727, 1.3538666751582975, 1.4426259039919596, 1.5489853358570191, 1.6762658237220625, 1.8283407612492941, 2.0096956085876183, 2.225478927469756, 2.481536379650452, 2.784415934557119, 3.1413289096347365, 3.560047844772632, 4.048719380066383, 4.615569542115128, 5.2684819496549835, 6.014438591970396, 6.858830049237097, 7.804668263503327, 8.851768731513417, 9.99600492938444, 11.228766118181639, 12.536757560834843, 13.902257701387796, 15.303885189125953, 16.717837610115794, 18.119465097853947, 19.484965238406907, 20.792956681060105, 22.02571786985731, 23.16995406772833, 24.217054535738416, 25.16289275000465, 26.007284207271347, 26.753240849586767, 27.40615325712662, 27.973003419175363, 28.461674954469114, 28.880393889607006, 29.237306864684626, 29.540186419591297, 29.79624387177199, 30.01202719065413, 30.193382037992453, 30.34545697551969, 30.47273746338473, 30.579096895249787, 30.66785612408345, 30.741845563814174, 30.80346599254902, 30.85474569563567, 30.897392663720595, 30.932841297560394, 30.962293553185553, 30.986754758742034, 31.007064503249293, 31.02392307921529],
+            "short_factor": [0.9977997200264581, 1.014658295992452, 1.0349680404997148, 1.059429246056193, 1.0888815016813513, 1.1243301355211495, 1.166977103606075, 1.2182568066927284, 1.2798772354275727, 1.3538666751582975, 1.4426259039919596, 1.5489853358570191, 1.6762658237220625, 1.8283407612492941, 2.0096956085876183, 2.225478927469756, 2.481536379650452, 2.784415934557119, 3.1413289096347365, 3.560047844772632, 4.048719380066383, 4.615569542115128, 5.2684819496549835, 6.014438591970396, 6.858830049237097, 7.804668263503327, 8.851768731513417, 9.99600492938444, 11.228766118181639, 12.536757560834843, 13.902257701387796, 15.303885189125953, 16.717837610115794, 18.119465097853947, 19.484965238406907, 20.792956681060105, 22.02571786985731, 23.16995406772833, 24.217054535738416, 25.16289275000465, 26.007284207271347, 26.753240849586767, 27.40615325712662, 27.973003419175363, 28.461674954469114, 28.880393889607006, 29.237306864684626, 29.540186419591297, 29.79624387177199, 30.01202719065413, 30.193382037992453, 30.34545697551969, 30.47273746338473, 30.579096895249787, 30.66785612408345, 30.741845563814174, 30.80346599254902, 30.85474569563567, 30.897392663720595, 30.932841297560394, 30.962293553185553, 30.986754758742034, 31.007064503249293, 31.02392307921529],
+            "original_max_position_embeddings": 32768
+        },    
+        "vocab_size": 73448,
+        "use_mup": false,
+        "scale_emb": 12,
+        "dim_model_base": 256,
+        "scale_depth": 1.4
+    },
+    "patch_size": 4,
+    "feat_dim": 64,
+    "scalar_quantization_latent_dim": 512,
+    "scalar_quantization_scale": 9,
+    "residual_lm_num_layers": 8,
+    "residual_lm_no_rope": true,
+    "encoder_config": {
+        "hidden_dim": 1024,
+        "ffn_dim": 4096,
+        "num_heads": 16,
+        "num_layers": 12,
+        "kv_channels": 128
+    },
+    "dit_config": {
+        "hidden_dim": 1024,
+        "ffn_dim": 4096,
+        "num_heads": 16,
+        "num_layers": 12,
+        "kv_channels": 128,
+        "mean_mode": false,
+        "cfm_config": {
+            "sigma_min": 1e-06,
+            "solver": "euler",
+            "t_scheduler": "log-norm",
+            "inference_cfg_rate": 2.0
+        }
+    },
+    "audio_vae_config": {
+        "encoder_dim": 128,
+        "encoder_rates": [2, 5, 8, 8],
+        "latent_dim": 64,
+        "decoder_dim": 2048,
+        "decoder_rates": [8, 6, 5, 2, 2, 2],
+        "sr_bin_boundaries": [20000, 30000, 40000],
+        "sample_rate": 16000,
+        "out_sample_rate": 48000
+    },
+    "max_length": 8192,
+    "device": "cuda",
+    "dtype": "bfloat16"
+}

merge_info.json

@@ -0,0 +1,30 @@
+{
+  "base_model": "/workspace/VoxCPM/models",
+  "lora_ckpt": "/workspace/VoxCPM/checkpoints/finetune_lora/step_0005000",
+  "lora_config": {
+    "enable_lm": true,
+    "enable_dit": true,
+    "enable_proj": false,
+    "r": 64,
+    "alpha": 64,
+    "dropout": 0.0,
+    "target_modules_lm": [
+      "q_proj",
+      "v_proj",
+      "k_proj",
+      "o_proj"
+    ],
+    "target_modules_dit": [
+      "q_proj",
+      "v_proj",
+      "k_proj",
+      "o_proj"
+    ],
+    "target_proj_modules": [
+      "enc_to_lm_proj",
+      "lm_to_dit_proj",
+      "res_to_dit_proj",
+      "fusion_concat_proj"
+    ]
+  }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:032c1fe16c5ce64dbc3537eeba3951f741de9bfe14a0101d7ccac1a53aed4516
+size 9160089192

special_tokens_map.json

@@ -0,0 +1,81 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|tool_call|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|execute_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|execute_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ],
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,218 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "<|audio_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "<|audio_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "<|audio_prompt_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "104": {
+      "content": "<|audio_prompt_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "105": {
+      "content": "<|background|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "106": {
+      "content": "<|/background|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "107": {
+      "content": "<|characters|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "108": {
+      "content": "<|/characters|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "109": {
+      "content": "<|speaker_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110": {
+      "content": "<|/speaker_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "111": {
+      "content": "<|span|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "112": {
+      "content": "<|/span|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "73440": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "73441": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "73442": {
+      "content": "<|tool_call|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "73443": {
+      "content": "<|execute_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "73444": {
+      "content": "<|execute_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "73445": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "73446": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "73447": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_end|>",
+    "<|im_start|>",
+    "<|tool_call|>",
+    "<|execute_start|>",
+    "<|execute_end|>",
+    "<|fim_prefix|>",
+    "<|fim_middle|>",
+    "<|fim_suffix|>"
+  ],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "legacy": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": null,
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "VoxCPM2Tokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false,
+  "chat_template": "{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
+  "auto_map": {
+    "AutoTokenizer": [
+      "tokenization_voxcpm2.VoxCPM2Tokenizer",
+      null
+    ]
+  }
+}