whisper-tiny-grpo-2277b09-cv22-tiny-sftanneal

A Whisper model fine-tuned with GRPO (Group Relative Policy Optimization) using a blended error-rate reward. Trained with whisper-rl.

Best validation (overall across 77 languages): WER 0.765, CER 0.344

Training data

Fine-tuned on cv22_index — all 77 Common Voice locales, streamed and decoded on the fly from the full training split. Each clip's language is pinned from its Common Voice locale during training.

Performance per language

Validation WER and CER at the best checkpoint, per Common Voice locale (also in the Evaluation Results metadata above):

Language	WER	CER
af	0.867	0.338
am	1.000	0.870
ar	1.706	0.400
as	0.906	0.543
az	0.814	0.277
ba	1.000	0.493
be	0.926	0.288
bg	0.800	0.242
bn	0.906	0.521
br	0.951	0.487
ca	0.680	0.298
cs	0.688	0.161
cy	0.775	0.441
da	0.885	0.388
de	0.224	0.083
el	0.500	0.123
en	0.244	0.101
es	0.478	0.161
et	1.130	0.322
eu	0.917	0.200
fa	0.967	0.497
fi	0.538	0.143
fr	0.375	0.120
gl	0.562	0.115
ha	0.806	0.204
he	1.000	0.393
hi	0.434	0.228
ht	0.905	0.400
hu	1.056	0.771
hy-AM	1.160	0.772
id	0.923	0.432
is	0.903	0.478
it	0.579	0.140
ja	1.000	0.583
ka	1.000	0.939
kk	1.000	0.467
ko	0.435	0.232
lo	1.000	0.868
lt	1.000	0.470
lv	0.833	0.222
mk	0.727	0.226
ml	0.907	0.500
mn	0.941	0.511
mr	0.457	0.234
mt	1.031	0.356
ne-NP	0.520	0.277
nl	0.476	0.151
nn-NO	0.889	0.371
oc	0.880	0.312
pa-IN	0.829	0.519
pl	0.462	0.162
ps	0.875	0.500
pt	0.278	0.113
ro	0.444	0.139
ru	0.576	0.225
sk	0.870	0.307
sl	0.417	0.103
sq	0.926	0.509
sr	0.444	0.226
sv-SE	0.517	0.215
sw	0.933	0.314
ta	0.455	0.246
te	0.857	0.559
tg	0.700	0.252
th	0.655	0.256
tk	1.000	0.483
tt	0.875	0.354
uk	0.583	0.174
ur	0.800	0.252
uz	1.000	0.342
vi	0.435	0.250
yi	1.095	0.585
yo	1.000	0.587
yue	1.000	0.451
zh-CN	1.000	0.312
zh-HK	1.000	0.643
zh-TW	0.667	0.167

How it was trained

Instead of cross-entropy against a single reference, for each audio clip the policy samples a group of num_generations transcriptions, scores each by a negated blend of word error rate, character error rate, and length / repetition penalties, and is nudged toward the better candidates with a clipped policy-gradient objective regularized by a per-token KL penalty to the frozen base model. Advantages are the group-relative, standardized rewards (A = (r - mean) / (std + eps)), so no value network is needed. The clip's language is pinned from its Common Voice locale, and the policy's own greedy transcriptions are scored as validation WER and CER.

Hyperparameters

Field	Value
Base model	openai/whisper-tiny
Dataset	/data/cv22_index
Learning rate	1e-05
Sampling temperature	0.7
Group size (generations/clip)	8
Reward weights	{'cer': 1, 'wer': 1, 'length': 0.5, 'repetition': 0.5}
KL penalty (β)	0.04
Batch size (clips/step)	16
Max optimizer steps	1000000
Warmup steps	20

Training curves

Pulled from the Weights & Biases run (static snapshot):

Usage

from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="wrice/whisper-tiny-grpo-2277b09-cv22-tiny-sftanneal")
print(asr("audio.wav")["text"])

Limitations

A proof-of-concept GRPO recipe, not a tuned production system. WER and CER are reported on a held-out Common Voice validation slice after text normalization; real-world performance varies by domain, accent, language, and audio quality.