Add model card and evaluation results

e15b122 verified 11 days ago

4.89 kB

language:
  - sna
license: apache-2.0
library_name: qwen-asr
pipeline_tag: automatic-speech-recognition
tags:
  - automatic-speech-recognition
  - audio
  - speech
  - shona
  - qwen3-asr
  - transformers
base_model: Qwen/Qwen3-ASR-1.7B
datasets:
  - manassehzw/sna-dataset-annotated
metrics:
  - wer
  - cer
model-index:
  - name: Shona Qwen3-ASR 1.7B
    results:
      - task:
          type: automatic-speech-recognition
          name: Automatic Speech Recognition
        dataset:
          name: Shona Annotated Validation Probe
          type: manassehzw/sna-dataset-annotated
          split: validation
        metrics:
          - name: Normalized Word Error Rate
            type: wer
            value: 0.2519838056680162
          - name: Normalized Character Error Rate
            type: cer
            value: 0.050787599076953946

manassehzw/sna-qwen-asr-1.7b

Shona Qwen3-ASR 1.7B is a Shona (sna) automatic speech recognition model fine-tuned from Qwen/Qwen3-ASR-1.7B on manassehzw/sna-dataset-annotated.

Overview

This release contains checkpoint 760 from run 20260803T140000Z. It was selected by generated validation WER and occurs at approximately two epochs. Qwen3-ASR adapted rapidly to Shona and substantially outperformed its zero-shot behavior on the project dataset. Evaluation on FLEURS also revealed a meaningful domain and speaker-generalization gap, so the in-domain score should not be treated as a universal Shona ASR result.

Model Details

Curated by: Manasseh Changachirere (Harare Institute of Technology)
Base model: Qwen/Qwen3-ASR-1.7B
Base model revision: 7278e1e70fe206f11671096ffdd38061171dd6e5
Training dataset: manassehzw/sna-dataset-annotated
Dataset revision: f91b1a79cbac15520d3c808b56e5192bf903f280
Training run: 20260803T140000Z
Released checkpoint: checkpoint-760
Checkpoint epoch: approximately 1.996
Selection criterion: lowest generated WER among retained checkpoints
Framework: qwen-asr==0.0.6, Transformers, PyTorch

Evaluation

Text for the project evaluations was normalized with Unicode NFKC, case folding, punctuation and symbol removal, and whitespace collapse.

Evaluation	Checkpoint	Examples	WER	CER
In-domain validation generation probe	760	256	25.20%	5.08%
Full in-domain test set	855	1,565	25.51%	4.97%
PazaBench v2 / FLEURS `sn_zw` test	855	925	52.85%	13.57%
Zero-shot base model, full in-domain test set	base	1,565	106.58%	31.64%

The full-test and FLEURS numbers are included as nearby-run context and were measured on checkpoint 855, not the released checkpoint. Checkpoints 760 and 855 were effectively tied on the 256-example generation probe: 25.198% versus 25.215% WER, a difference of one word error. A full external evaluation of checkpoint 760 has not yet been recorded.

Training Summary

Training examples: 12,170
Validation examples: 1,504
Held-out test examples: 1,565
Training audio: approximately 62.63 hours
Learning rate: 2e-5
Microbatch size: 4
Gradient accumulation: 8
Effective batch size: 32
Precision: bfloat16
Scheduler: linear with 2% warmup
Planned epochs: 3
Released checkpoint: approximately 2 epochs / 760 optimizer steps
Checkpoint evaluation loss: 0.17747

Example Usage

Install the official Qwen ASR package:

pip install -U qwen-asr

import torch
from qwen_asr import Qwen3ASRModel

model = Qwen3ASRModel.from_pretrained(
    "manassehzw/sna-qwen-asr-1.7b",
    dtype=torch.bfloat16,
    device_map="cuda:0",
    max_inference_batch_size=16,
    max_new_tokens=256,
)

results = model.transcribe(audio="sample.wav", language=None)
print(results[0].text)

The recorded evaluations used language=None. Although the fine-tuning targets identify the output as Shona, Shona was not in the base model's original list of supported language arguments, so automatic language handling is the tested path.

Limitations

External FLEURS performance is substantially weaker than in-domain performance.
The training corpus is relatively small and may not cover Zimbabwe's full range of speakers, accents, recording conditions, and conversational domains.
Code-switching was not separately quantified for this checkpoint.
Long-form, streaming, noisy, telephone, and far-field behavior require further evaluation.
Orthographically close substitutions can produce a low CER while still being penalized heavily by WER.

License

The model is released under Apache-2.0, matching the base Qwen3-ASR model.