Oshara/xtts-v2-nepali

XTTS v2 — Nepali (नेपाली) fine-tune

A fine-tune of Coqui XTTS v2 that adds Nepali (ne) — a language the base model does not ship. XTTS v2 is a zero-shot voice-cloning TTS model: given a few seconds of any voice and a line of text, it speaks that line in that voice. This checkpoint teaches it to do so fluently in Nepali.

• Base model: coqui/XTTS-v2
• Method: full GPT fine-tune (not frozen), restored from the base checkpoint
• Data: 4,140 single-speaker Nepali clips (3,933 train / 207 eval, 95/5)
• Language route: a new ne code registered in config.json, over the existing Devanagari tokenizer
• Output: 24 kHz

Checkpoints

Two checkpoints are provided as self-contained folders. Epoch 10 is recommended — it generalises best; later epochs overfit the single training speaker (eval mel-CE rises while text-CE stays flat).

Folder	When to use
epoch-10/	Recommended. Best generalisation.
epoch-20/	More adapted to the training speaker, less general.

Each folder is drop-in XTTS format: model.pth, config.json, vocab.json, speakers_xtts.pth.

Usage

from huggingface_hub import snapshot_download
from TTS.tts.configs.xtts_config import XttsConfig
from TTS.tts.models.xtts import Xtts
import torchaudio, torch

d = snapshot_download("Oshara/xtts-v2-nepali", allow_patterns=["epoch-10/*"]) + "/epoch-10"

config = XttsConfig(); config.load_json(f"{d}/config.json")
model = Xtts.init_from_config(config)
model.load_checkpoint(config, checkpoint_path=f"{d}/model.pth",
                      vocab_path=f"{d}/vocab.json",
                      speaker_file_path=f"{d}/speakers_xtts.pth", eval=True)
model.cuda()

out = model.synthesize(
    "नेपालका हिमालहरू संसारभर प्रसिद्ध छन्।",
    config, speaker_wav="reference_voice.wav", language="ne",
    temperature=0.65, repetition_penalty=5.0,
)
torchaudio.save("out.wav", torch.tensor(out["wav"]).unsqueeze(0), 24000)

Tip: for long or numeral-heavy text, normalise first (strip zero-width chars, split on the danda ।, expand Nepali digits to words, insert short inter-sentence silence). This prevents the autoregressive GPT from drifting into noise between sentences.

Evaluation

Evaluated on NepTTS-Bench (205 sentences) plus voice-cloning and prosody metrics. The intelligibility and quality numbers place it among the strongest Nepali systems.

Metric	Score	Note
Whisper round-trip CER	0.380	2nd best on the benchmark
MMS-1b round-trip CER	0.199	2nd best
XLS-R Nepali CER	0.183	mid-pack
SCOREQ auto-MOS	4.21	3rd overall
SQUIM MOS (est.)	4.69	reference-free naturalness
SQUIM STOI / PESQ / SI-SDR	0.99 / 3.78 / 27.6 dB	clean signal
Speaker similarity (SECS, WavLM-SV)	0.923	high cloning fidelity (same-speaker threshold ≈ 0.86)
Pitch expressiveness (F0 std)	3.58 st	somewhat flatter than natural (~4.95 st) — the main soft spot

Limitations

• Single-speaker training corpus; timbre diversity comes from XTTS's zero-shot cloning, not the fine-tune.
• Slightly reduced pitch expressiveness vs. natural Nepali speech (flatter intonation).
• Nepali is routed over the existing Devanagari tokenizer; no vocabulary was added.

License

Inherits the Coqui Public Model License (CPML) from the XTTS v2 base model.