whisper-tiny-grpo-d46ed86-cv22-tiny-sftanneal2

A Whisper model fine-tuned with GRPO (Group Relative Policy Optimization) using a blended error-rate reward. Trained with whisper-rl.

Best validation (overall across 77 languages): WER 0.661, CER 0.298

Training data

Fine-tuned on cv22_index — all 77 Common Voice locales, streamed and decoded on the fly from the full training split. Each clip's language is pinned from its Common Voice locale during training.

Performance per language

Validation WER and CER at the best checkpoint, per Common Voice locale (also in the Evaluation Results metadata above):

Language	WER	CER
af	0.822	0.295
am	1.027	0.774
ar	0.529	0.253
as	0.625	0.321
az	0.767	0.233
ba	0.781	0.321
be	0.815	0.226
bg	0.800	0.194
bn	0.438	0.260
br	0.927	0.413
ca	0.600	0.267
cs	0.562	0.125
cy	0.525	0.211
da	0.654	0.287
de	0.204	0.083
el	0.545	0.138
en	0.293	0.180
es	0.565	0.198
et	1.019	0.287
eu	0.833	0.179
fa	1.067	0.497
fi	0.500	0.128
fr	0.425	0.146
gl	0.406	0.110
ha	0.645	0.159
he	0.912	0.491
hi	0.377	0.179
ht	0.810	0.400
hu	1.056	0.814
hy-AM	1.000	0.475
id	0.538	0.270
is	0.968	1.586
it	0.579	0.128
ja	1.333	0.604
ka	1.000	0.528
kk	0.700	0.243
ko	0.348	0.146
lo	0.879	0.635
lt	1.000	0.476
lv	0.833	0.222
mk	0.727	0.194
ml	0.558	0.337
mn	0.794	0.326
mr	0.200	0.091
mt	0.750	0.198
ne-NP	0.600	0.295
nl	0.571	0.246
nn-NO	0.944	0.405
oc	0.800	0.291
pa-IN	0.371	0.169
pl	0.538	0.231
ps	0.833	0.471
pt	0.278	0.103
ro	0.389	0.102
ru	0.545	0.182
sk	0.783	0.235
sl	0.750	0.141
sq	0.963	0.409
sr	0.444	0.226
sv-SE	0.414	0.174
sw	0.733	0.220
ta	0.309	0.162
te	0.464	0.237
tg	0.700	0.200
th	0.759	0.293
tk	1.000	0.469
tt	0.938	0.329
uk	0.667	0.180
ur	0.600	0.252
uz	0.864	0.295
vi	0.478	0.283
yi	1.000	0.463
yo	0.968	0.487
yue	1.000	0.294
zh-CN	1.000	0.292
zh-HK	1.000	0.464
zh-TW	1.000	0.417

How it was trained

Instead of cross-entropy against a single reference, for each audio clip the policy samples a group of num_generations transcriptions, scores each by a negated blend of word error rate, character error rate, and length / repetition penalties, and is nudged toward the better candidates with a clipped policy-gradient objective regularized by a per-token KL penalty to the frozen base model. Advantages are the group-relative, standardized rewards (A = (r - mean) / (std + eps)), so no value network is needed. The clip's language is pinned from its Common Voice locale, and the policy's own greedy transcriptions are scored as validation WER and CER.

Hyperparameters

Field	Value
Base model	openai/whisper-tiny
Dataset	/data/cv22_index
Learning rate	1e-05
Sampling temperature	0.7
Group size (generations/clip)	8
Reward weights	{'cer': 1, 'wer': 1, 'length': 0.5, 'repetition': 0.5}
KL penalty (β)	0.04
Batch size (clips/step)	16
Max optimizer steps	1000000
Warmup steps	20

Training curves

Pulled from the Weights & Biases run (static snapshot):

Usage

from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="wrice/whisper-tiny-grpo-d46ed86-cv22-tiny-sftanneal2")
print(asr("audio.wav")["text"])

Limitations

A proof-of-concept GRPO recipe, not a tuned production system. WER and CER are reported on a held-out Common Voice validation slice after text normalization; real-world performance varies by domain, accent, language, and audio quality.