rinhoooo/phowhisper-large-vien-cs-asr

PhoWhisper-large — Vietnamese-English Code-Switching ASR

Fine-tuned PhoWhisper-large on the first labeled Vietnamese-English code-switching speech corpus, using LoRA + speed perturbation + SpecAugment. Achieves 4.56% WER-N on the holdout test set — a 78.2% relative reduction over the untuned baseline (20.95% WER-N).

Paper: "A Vietnamese-English Code-Switching Speech Dataset and Benchmark Evaluation for Automatic Speech Recognition" — Procedia Computer Science, KES 2026

---

Performance

Holdout test set (Vietnamese-English code-switching)

Metric	Raw	Normalized
WER	5.76%	4.56%
CER	2.76%	2.44%
SER	37.26%	31.48%

Normalized = punctuation removed, lowercased, language tags stripped.

Cross-domain: VIVOS (monolingual Vietnamese)

Condition	WER-N
PhoWhisper-large untuned	4.61%
This model (fine-tuned)	7.20%

Modest 2.59pp degradation on monolingual Vietnamese — no catastrophic forgetting.

---

Usage

from transformers import WhisperProcessor, WhisperForConditionalGeneration
from peft import PeftModel
import torch

base_model_id = "vinai/PhoWhisper-large"
adapter_id    = "rinhoooo/phowhisper-large-vien-cs-asr"

processor = WhisperProcessor.from_pretrained(base_model_id)
model     = WhisperForConditionalGeneration.from_pretrained(
    base_model_id, torch_dtype=torch.float16, device_map="auto"
)
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

# Inference
inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
    predicted_ids = model.generate(**inputs)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
print(transcription)

---

Training details

Hyperparameter	Value
Base model	vinai/PhoWhisper-large
LoRA rank r	32
LoRA alpha α	64
LoRA dropout	0.1
Target modules	q_proj, k_proj, v_proj, out_proj, fc1, fc2
Trainable params	57.7M (3.6% of total)
Optimizer	AdamW
Learning rate	5e-5
LR schedule	Cosine annealing (2 restarts)
Effective batch size	32
Max epochs	5
Early stopping patience	3
Augmentation	Speed perturbation ×{0.9, 1.0} + SpecAugment
Hardware	NVIDIA A100 80 GB

---

Citation

@inproceedings{[author]2026viencsasr,
  title     = {A Vietnamese-English Code-Switching Speech Dataset and Benchmark
               Evaluation for Automatic Speech Recognition},
  author    = {[Author] and [Author] and [Author]},
  booktitle = {Procedia Computer Science (KES 2026)},
  year      = {2026}
}