kingabzpro/wav2vec2-large-xls-r-300m-Urdu

Urdu ASR XLS-R 300M

A fine-tuned XLS-R 300M CTC model for Urdu automatic speech recognition. It transcribes 16 kHz mono audio and includes an optional 5-gram KenLM decoder.

Best reported result: 39.89% WER / 16.70% CER with KenLM decoding on the Urdu Common Voice 8.0 test set. See the Kaggle evaluation notebook for a reproducible example.

⚡ Quick start

Install the required packages:

pip install -U torch torchaudio transformers pyctcdecode kenlm huggingface_hub

Note: After installing the packages in a notebook environment, restart the kernel before running the inference code.

Transcribe a local audio file:

import torch
import torchaudio
from transformers import AutoModelForCTC, AutoProcessor

MODEL_ID = "kingabzpro/wav2vec2-large-xls-r-300m-Urdu"
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
INFERENCE_DTYPE = torch.float16 if DEVICE.type == "cuda" else torch.float32

processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForCTC.from_pretrained(MODEL_ID).eval().to(
    device=DEVICE, dtype=INFERENCE_DTYPE
)

waveform, sample_rate = torchaudio.load("audio.wav")

# Convert stereo (or multi-channel) audio to mono and resample to 16 kHz.
waveform = waveform.mean(dim=0)
if sample_rate != 16_000:
    waveform = torchaudio.functional.resample(waveform, sample_rate, 16_000)

inputs = processor(
    waveform.numpy(), sampling_rate=16_000, return_tensors="pt", padding=True
).input_values.to(device=DEVICE, dtype=INFERENCE_DTYPE)

with torch.inference_mode():
    predicted_ids = model(inputs).logits.argmax(dim=-1)

transcription = processor.batch_decode(predicted_ids)[0]
print(transcription)

🧠 Language-model decoding

Why use it? The included 5-gram KenLM language model reduces the reported full-test WER from 56.07% (greedy CTC) to 39.89%.

Show the complete KenLM decoding example

The repository contains a 5-gram KenLM language model.

import json
import torch
import torchaudio
from huggingface_hub import hf_hub_download
from pyctcdecode import build_ctcdecoder
from transformers import AutoModelForCTC, AutoProcessor

MODEL_ID = "kingabzpro/wav2vec2-large-xls-r-300m-Urdu"
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
INFERENCE_DTYPE = torch.float16 if DEVICE.type == "cuda" else torch.float32

processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForCTC.from_pretrained(MODEL_ID).eval().to(
    device=DEVICE, dtype=INFERENCE_DTYPE
)

kenlm_path = hf_hub_download(MODEL_ID, "language_model/5gram.bin")
unigrams_path = hf_hub_download(MODEL_ID, "language_model/unigrams.txt")
attrs_path = hf_hub_download(MODEL_ID, "language_model/attrs.json")

with open(unigrams_path, encoding="utf-8") as file:
    unigrams = [line.strip() for line in file if line.strip()]
with open(attrs_path, encoding="utf-8") as file:
    attrs = json.load(file)

# Keep only acoustic tokens. The matching ID list is then applied to logits.
vocab_items = sorted(processor.tokenizer.get_vocab().items(), key=lambda item: item[1])
blank_id = processor.tokenizer.pad_token_id
delimiter = processor.tokenizer.word_delimiter_token
decoder_pairs = [
    (token, token_id)
    for token, token_id in vocab_items
    if token_id == blank_id or token == delimiter or len(token) == 1
]
kept_token_ids = [token_id for _, token_id in decoder_pairs]
labels = [
    "" if token_id == blank_id else " " if token == delimiter else token
    for token, token_id in decoder_pairs
]
decoder = build_ctcdecoder(
    labels,
    kenlm_model_path=kenlm_path,
    unigrams=unigrams,
    alpha=attrs.get("alpha", 0.5),
    beta=attrs.get("beta", 1.0),
)

waveform, sample_rate = torchaudio.load("audio.wav")
waveform = waveform.mean(dim=0)
if sample_rate != 16_000:
    waveform = torchaudio.functional.resample(waveform, sample_rate, 16_000)

inputs = processor(
    waveform.numpy(), sampling_rate=16_000, return_tensors="pt"
).input_values.to(device=DEVICE, dtype=INFERENCE_DTYPE)
with torch.inference_mode():
    logits = model(inputs).logits[0].float().cpu().numpy()

transcription = decoder.decode(logits[:, kept_token_ids])
print(transcription)

🧪 Kaggle evaluation

The Kaggle notebook evaluates a five-sample streaming smoke test from fixie-ai/common_voice_17_0 (ur, test).

Show the core evaluation code

from datasets import Audio, load_dataset

stream = load_dataset(
    "fixie-ai/common_voice_17_0", "ur", split="test", streaming=True
).cast_column("audio", Audio(sampling_rate=16_000))

example = next(iter(stream))
audio = example["audio"]
samples = audio.get_all_samples().data if hasattr(audio, "get_all_samples") else audio["array"]
waveform = samples.detach().cpu().numpy() if torch.is_tensor(samples) else samples
if waveform.ndim == 2:
    waveform = waveform.mean(axis=0 if waveform.shape[0] <= waveform.shape[-1] else 1)

inputs = processor(waveform, sampling_rate=16_000, return_tensors="pt")
with torch.inference_mode():
    logits = model(inputs.input_values.to(device=DEVICE, dtype=INFERENCE_DTYPE)).logits[0]

prediction = decoder.decode(logits.float().cpu().numpy()[:, kept_token_ids])
print("Reference: ", example["sentence"])
print("Prediction:", prediction)

Recorded notebook output

Single-sample inference:

Reference:  بے ذوق نہیں اگرچہ فطرت
Prediction: بھی ذوق نہیں اگھرچے فطرت

Five-sample streaming smoke-test results:

Sample	Duration (s)	WER	CER
1	2.92	0.00%	0.00%
2	2.88	0.00%	0.00%
3	5.40	22.22%	3.03%
4	4.36	33.33%	12.50%
5	5.69	42.11%	24.00%
Mean	—	19.53%	7.91%

Important: This is a five-sample smoke test—not a benchmark. Do not compare it directly with the full Common Voice 8.0 test-set results below.

📊 Evaluation

Full Common Voice 8.0 test set

Decoder	Test WER	Test CER
Greedy CTC	56.07%	23.70%
5-gram language model	39.89%	16.70%

Results are reported on the Urdu test split of Mozilla Common Voice 8.0. The language-model row is the model-card score; compare each result only with the same decoding strategy.

To reproduce language-model evaluation from this repository:

python eval.py --model_id kingabzpro/wav2vec2-large-xls-r-300m-Urdu --dataset mozilla-foundation/common_voice_8_0 --config ur --split test

🏗️ Training

The model was trained from facebook/wav2vec2-xls-r-300m on Urdu Mozilla Common Voice 8.0.

Hyperparameter	Value
Learning rate	1e-4
Train batch size	32
Evaluation batch size	8
Gradient accumulation	2
Effective train batch size	64
Epochs	200
LR scheduler	Linear, 1,000 warm-up steps
Optimizer	Adam (β₁=0.9, β₂=0.999, ε=1e-8)

Training checkpoints

Training loss	Epoch	Step	Validation loss	WER	CER
3.6398	30.77	400	3.3517	1.0000	1.0000
2.9225	61.54	800	2.5123	1.0000	0.8310
1.2568	92.31	1,200	0.9699	0.6273	0.2575
0.8974	123.08	1,600	0.9715	0.5888	0.2457
0.7151	153.85	2,000	0.9984	0.5588	0.2353
0.6416	184.62	2,400	0.9889	0.5607	0.2370

⚠️ Intended use and limitations

Use this model for Urdu speech transcription and prototyping. Accuracy varies with recording quality, speaker accent, code-switching, background noise, domain-specific vocabulary, and utterance length. Review transcripts before using them in consequential or user-facing workflows.

Historical training environment

• Transformers 4.17.0.dev0
• PyTorch 1.10.2+cu102
• Datasets 1.18.2.dev0
• Tokenizers 0.11.0