Add model card and evaluation results

README.md

@@ -0,0 +1,142 @@
+---
+language:
+- sna
+license: apache-2.0
+library_name: qwen-asr
+pipeline_tag: automatic-speech-recognition
+tags:
+- automatic-speech-recognition
+- audio
+- speech
+- shona
+- qwen3-asr
+- transformers
+base_model: Qwen/Qwen3-ASR-1.7B
+datasets:
+- manassehzw/sna-dataset-annotated
+metrics:
+- wer
+- cer
+model-index:
+- name: Shona Qwen3-ASR 1.7B
+  results:
+  - task:
+      type: automatic-speech-recognition
+      name: Automatic Speech Recognition
+    dataset:
+      name: Shona Annotated Validation Probe
+      type: manassehzw/sna-dataset-annotated
+      split: validation
+    metrics:
+    - name: Normalized Word Error Rate
+      type: wer
+      value: 0.2519838056680162
+    - name: Normalized Character Error Rate
+      type: cer
+      value: 0.050787599076953946
+---
+
+# manassehzw/sna-qwen-asr-1.7b
+
+Shona Qwen3-ASR 1.7B is a Shona (`sna`) automatic speech recognition model
+fine-tuned from [`Qwen/Qwen3-ASR-1.7B`](https://huggingface.co/Qwen/Qwen3-ASR-1.7B)
+on [`manassehzw/sna-dataset-annotated`](https://huggingface.co/datasets/manassehzw/sna-dataset-annotated).
+
+## Overview
+
+This release contains checkpoint 760 from run `20260803T140000Z`. It was selected
+by generated validation WER and occurs at approximately two epochs. Qwen3-ASR
+adapted rapidly to Shona and substantially outperformed its zero-shot behavior on
+the project dataset. Evaluation on FLEURS also revealed a meaningful domain and
+speaker-generalization gap, so the in-domain score should not be treated as a
+universal Shona ASR result.
+
+## Model Details
+
+- **Curated by:** [Manasseh Changachirere (Harare Institute of Technology)](https://www.manasseh.dev/)
+- **Base model:** [`Qwen/Qwen3-ASR-1.7B`](https://huggingface.co/Qwen/Qwen3-ASR-1.7B)
+- **Base model revision:** `7278e1e70fe206f11671096ffdd38061171dd6e5`
+- **Training dataset:** [`manassehzw/sna-dataset-annotated`](https://huggingface.co/datasets/manassehzw/sna-dataset-annotated)
+- **Dataset revision:** `f91b1a79cbac15520d3c808b56e5192bf903f280`
+- **Training run:** `20260803T140000Z`
+- **Released checkpoint:** `checkpoint-760`
+- **Checkpoint epoch:** approximately `1.996`
+- **Selection criterion:** lowest generated WER among retained checkpoints
+- **Framework:** `qwen-asr==0.0.6`, Transformers, PyTorch
+
+## Evaluation
+
+Text for the project evaluations was normalized with Unicode NFKC, case folding,
+punctuation and symbol removal, and whitespace collapse.
+
+| Evaluation | Checkpoint | Examples | WER | CER |
+|---|---:|---:|---:|---:|
+| In-domain validation generation probe | 760 | 256 | **25.20%** | **5.08%** |
+| Full in-domain test set | 855 | 1,565 | 25.51% | 4.97% |
+| PazaBench v2 / FLEURS `sn_zw` test | 855 | 925 | 52.85% | 13.57% |
+| Zero-shot base model, full in-domain test set | base | 1,565 | 106.58% | 31.64% |
+
+The full-test and FLEURS numbers are included as nearby-run context and were
+measured on checkpoint 855, not the released checkpoint. Checkpoints 760 and 855
+were effectively tied on the 256-example generation probe: 25.198% versus
+25.215% WER, a difference of one word error. A full external evaluation of
+checkpoint 760 has not yet been recorded.
+
+## Training Summary
+
+- **Training examples:** 12,170
+- **Validation examples:** 1,504
+- **Held-out test examples:** 1,565
+- **Training audio:** approximately 62.63 hours
+- **Learning rate:** `2e-5`
+- **Microbatch size:** 4
+- **Gradient accumulation:** 8
+- **Effective batch size:** 32
+- **Precision:** bfloat16
+- **Scheduler:** linear with 2% warmup
+- **Planned epochs:** 3
+- **Released checkpoint:** approximately 2 epochs / 760 optimizer steps
+- **Checkpoint evaluation loss:** `0.17747`
+
+## Example Usage
+
+Install the official Qwen ASR package:
+
+```bash
+pip install -U qwen-asr
+```
+
+```python
+import torch
+from qwen_asr import Qwen3ASRModel
+
+model = Qwen3ASRModel.from_pretrained(
+    "manassehzw/sna-qwen-asr-1.7b",
+    dtype=torch.bfloat16,
+    device_map="cuda:0",
+    max_inference_batch_size=16,
+    max_new_tokens=256,
+)
+
+results = model.transcribe(audio="sample.wav", language=None)
+print(results[0].text)
+```
+
+The recorded evaluations used `language=None`. Although the fine-tuning targets
+identify the output as Shona, Shona was not in the base model's original list of
+supported language arguments, so automatic language handling is the tested path.
+
+## Limitations
+
+- External FLEURS performance is substantially weaker than in-domain performance.
+- The training corpus is relatively small and may not cover Zimbabwe's full range
+  of speakers, accents, recording conditions, and conversational domains.
+- Code-switching was not separately quantified for this checkpoint.
+- Long-form, streaming, noisy, telephone, and far-field behavior require further
+  evaluation.
+- Orthographically close substitutions can produce a low CER while still being
+  penalized heavily by WER.
+
+## License
+
+The model is released under Apache-2.0, matching the base Qwen3-ASR model.