Link the project website (chuvash-asr.vercel.app)

736c18f verified 20 days ago

6.1 kB

language:
  - cv
license: cc-by-nc-4.0
library_name: transformers
pipeline_tag: automatic-speech-recognition
base_model: bond005/whisper-podlodka-turbo
datasets:
  - SpeechCollector/ChuvashAsrDataset
tags:
  - whisper
  - chuvash
  - chv
  - speech-recognition
  - asr
metrics:
  - wer
  - cer
model-index:
  - name: whisper-chuvash-turbo
    results:
      - task:
          type: automatic-speech-recognition
          name: Automatic Speech Recognition
        dataset:
          type: SpeechCollector/ChuvashAsrDataset
          name: ChuvashAsrDataset (Common Voice test split)
          split: test
        metrics:
          - type: wer
            value: 16.02
            name: WER (orthographic, beam-5)
          - type: cer
            value: 3.63
            name: CER (orthographic, beam-5)
          - type: wer
            value: 11.13
            name: WER (Whisper-normalized, beam-5)

whisper-chuvash-turbo

Whisper fine-tuned for Chuvash (chv) — the most accurate — and the fastest — model of the family in the Aisar family by SpeechCollector. The family was trained on ChuvashAsrDataset (~49.5 h of labeled Chuvash speech) and evaluated on its Common Voice test split (1,288 utterances).

Base model: bond005/whisper-podlodka-turbo (809M parameters)
WER / CER (orthographic, beam-5): 16.02% / 3.63%
WER / CER (Whisper-normalized): 11.13% / 2.6%
Checkpoint size: ~1.6 GB
Inference speed: ~13x real time (measured, beam-5, V100 fp32; ~22x greedy)

The model was initialized from bond005/whisper-podlodka-turbo, a Russian-adapted large-v3-turbo. That starting point pays off for Chuvash, which carries many Russian loanwords. Despite being the largest model of the family by parameter count, it is also the fastest: large-v3-turbo keeps only 4 decoder layers.

Beyond recognition, this model produced the pseudo-labels for the raw corpora (ChuvashRaw, ChuvashConversationRaw, ChuvashNewsRaw — about 1,080 hours in total). None of that data was used to train the models, so the evaluation is leak-free.

The Aisar family

Model	Base	Params	WER	CER	WER (norm.)
whisper-chuvash-small	openai/whisper-small	244M	20.3%	5.35%	15.4%
whisper-chuvash-medium	openai/whisper-medium	769M	17.28%	3.83%	12.36%
whisper-chuvash-turbo	bond005/whisper-podlodka-turbo	809M	16.02%	3.63%	11.13%

Orthographic = text compared as-is, with casing and punctuation. Normalized = same beam-5 run scored after Whisper-style normalization (lowercase, ё→е, punctuation stripped). Speed estimates for small/medium are derived from OpenAI's official relative speeds (small ~4x, medium ~2x, turbo ~8x vs large) anchored to the measured turbo RTF.

How to use

Chuvash has no language token in the Whisper vocabulary, so generation uses language="russian" — this is intentional and gives the best results. The numbers above were obtained with beam-5; num_beams=1 is ~~1.7x faster at a small quality cost (~~+0.3 pp normalized WER for turbo).

import torch
from transformers import pipeline

asr = pipeline(
    "automatic-speech-recognition",
    model="SpeechCollector/whisper-chuvash-turbo",
    torch_dtype=torch.float16,   # use torch.float32 on CPU
    device="cuda:0",             # or "cpu"
    chunk_length_s=30,           # handles audio longer than 30 s
    stride_length_s=(5, 5),
)

out = asr(
    "speech.wav",  # any sample rate — the pipeline resamples to 16 kHz
    generate_kwargs={
        "language": "russian",
        "task": "transcribe",
        "num_beams": 5,
    },
)
print(out["text"])

For long spontaneous audio (conversations, broadcast), add a temperature fallback to generate_kwargs — it tames repetition loops on hard segments:

"temperature": (0.0, 0.2, 0.4, 0.6, 0.8, 1.0),
"compression_ratio_threshold": 2.4,
"logprob_threshold": -1.0,
"no_speech_threshold": 0.6,

Training data

ChuvashAsrDataset: 36,781 utterances / ~49.5 h, 16 kHz mono. Sources: alexantonov/chuvash_voice (38.7 h, CC0), Common Voice 25.0 (5.6 h, CC0), ftyers Turkic_TTS (5.2 h, CC-BY-SA). Splits: train 34,251 / val 1,242 / test 1,288 — validation and test contain Common Voice recordings only.

Limitations

Evaluated on read speech (Common Voice); expect higher error rates on spontaneous or noisy audio.
Fine-tuned almost exclusively on monolingual Chuvash, so the model performs best on Chuvash-dominant audio and degrades on Chuvash–Russian code-switched speech. Bilingual fine-tuning is a natural next step.
Output is orthographic Chuvash (ӑ, ӗ, ҫ, ӳ); casing and punctuation come from the model and are not guaranteed.

License and acceptable use

License: CC BY-NC 4.0 — free for research and other non-commercial use.
Commercial use: possible under a separate agreement with the SpeechCollector community — reach out on Telegram: @Michaelya.
Surveillance: please do not use this model to build systems for surveillance, covert monitoring, or tracking of individuals.

SpeechCollector
/

whisper-chuvash-turbo