inesassia/fonbench-mms-1b-fon-ft

FonBench MMS-1b Fon — Full Fine-Tuning (FT-2)

This model is the best-performing ASR model of the FonBench benchmark for the Fon language (tonal Gbe language of Benin, ~8M speakers), obtained by full fine-tuning of facebook/mms-1b-all.

Model Description

• Developed by: Inès Assia HOUNKPONOU (ESGIS Bénin)
• Model type: Automatic Speech Recognition (Wav2Vec2 + CTC head)
• Language: Fon (fon, ISO 639-3)
• License: CC-BY-4.0
• Finetuned from model: facebook/mms-1b-all
• Architecture: Wav2Vec2ForCTC (encoder-only with CTC head)
• Parameters: 967M (all trained)

Intended Uses

Direct use

• Transcription of recorded Fon speech in conditions similar to the training corpus (read or semi-spontaneous speech, studio-quality audio at 16 kHz).
• Educational and research applications related to Fon language technology.
• Component in pipelines for documentation, sub-titling, or accessibility tools targeting Fon speakers.

Downstream use

• Further fine-tuning on domain-specific Fon corpora (broadcast news, conversational speech, medical, legal, etc.).
• Use as a teacher model for distillation to smaller architectures.
• Adaptation to closely related Gbe languages (ewe, mina, gen) as a transfer learning starting point.

Out-of-scope use

• ⚠️ Not suitable for highly noisy or far-field audio (training was on clean studio recordings).
• ⚠️ Not suitable for spontaneous code-switched speech (Fon-French mixing), as the training corpus contains primarily monolingual Fon utterances.
• ⚠️ Not suitable for other Gbe or African languages without further adaptation.
• ⚠️ Not suitable for safety-critical applications (medical transcription, legal evidence, etc.) without independent validation by native speakers.

How to Use

from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import torch, librosa

model = Wav2Vec2ForCTC.from_pretrained("inesassia/fonbench-mms-1b-fon-ft")
processor = Wav2Vec2Processor.from_pretrained("inesassia/fonbench-mms-1b-fon-ft")

# Audio must be 16 kHz mono
audio, sr = librosa.load("audio.wav", sr=16000)

inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
    logits = model(inputs.input_values).logits
pred_ids = torch.argmax(logits, dim=-1)
transcription = processor.batch_decode(pred_ids)[0]
print(transcription)

Training Details

Training data

• Dataset: alaleye/fon (CC-BY-4.0)
• Total corpus size: 10,402 examples
• Speakers: 27 unique speakers
• Training split: 5,488 examples, 20 speakers
• Validation split: 1,373 examples (random 80/20 split of train, same speakers)
• Test split: 3,541 examples, 7 unseen speakers (strict speaker stratification between train+val and test)
• Speech characteristics: Read and semi-spontaneous speech, studio-quality audio at 16 kHz mono

Training procedure

Preprocessing

• Audio resampled to 16 kHz mono.
• Tokenizer with a vocabulary of 53 characters specific to Fon (constructed from the corpus).

Hyperparameters

Hyperparameter	Value
Strategy	Full fine-tuning
Optimizer	AdamW (8-bit, bitsandbytes)
Learning rate	5e-5
LR scheduler	Linear with warmup (warmup_ratio=0.1)
Batch size (per device)	2
Gradient accumulation steps	8
Effective batch size	16
Epochs	3 (early stopping triggered)
Early stopping patience	2
Precision	fp16 mixed precision
Gradient checkpointing	Enabled
Random seed	42

Compute infrastructure

• Hardware: Single NVIDIA T4 GPU (16 GB), Google Colab
• Training duration: ~2 hours
• Total parameters trained: 966,679,987 (100%)

Experiment tracking

All training runs are logged on Weights & Biases: FonBench project

Evaluation

Testing data

• Test set of 3,541 examples from the alaleye/fon dataset.
• 7 unseen speakers: antoine, cyrielle, frejus, helmut, melissa, miguel, mikael.
• Strict speaker stratification: no overlap with training or validation speakers.

Metrics

• WER (Word Error Rate): standard ASR metric measuring word-level errors.
• CER (Character Error Rate): sub-word reliability metric.

Results

Metric	Value
WER (test)	19.27 %
CER (test)	6.02 %
Loss (test)	0.207

Comparison with the FonBench benchmark

Configuration	Strategy	Params trained	WER	CER
MMS-1b zero-shot	None	0	83.44 %	31.88 %
LoRA-1	LoRA r=8	2.0M (0.21%)	32.20 %	9.08 %
LoRA-2	LoRA r=32	7.9M (0.82%)	28.75 %	8.18 %
FT-1	Full FT (LR 1e-5)	967M (100%)	24.32 %	7.17 %
This model (FT-2)	Full FT (LR 5e-5)	967M (100%)	19.27 %	6.02 %

Result: This model achieves a 64.2-point WER reduction compared to zero-shot baseline.

Limitations and Biases

Linguistic limitations

• Tonal information is not captured: The training corpus does not include systematic tonal marking (no combining diacritics such as \u0300, \u0301). Tonal distinctions are partially expressed through IPA vowel variants (ɔ, ɛ, ɖ) but the model has no explicit way to recover tonal contrasts not encoded in the orthography. Word-level tonal disambiguation is therefore not reliable with this model.
• Limited dialectal coverage: The dataset focuses on specific varieties of Fon. Performance on regional dialects or non-standard speech may be substantially lower.
• Code-switching not supported: Fon-French code-switching, common in everyday spoken Fon in Benin, is not represented in the training data.

Data limitations

• Corpus size: 5,488 training examples is modest by ASR standards.
• Acoustic conditions: The corpus consists primarily of clean, studio-quality recordings. Performance on noisy, far-field, or low-quality audio will degrade.
• Speaker diversity: 20 training speakers limit acoustic diversity. Generalization to populations with different age groups, accents, or social registers may be limited.
• Validation set: The validation split shares speakers with the train split (random 80/20 of train, no speaker stratification between train and validation). The test set, however, contains 7 strictly unseen speakers, which guarantees scientific validity of the reported metrics.

Architectural limitations

• The Wav2Vec2 + CTC architecture assumes monotonic alignment between audio and text, which is suitable for transcription but not for translation or paraphrasing.
• The model produces lowercase characters only (matching the corpus convention). Post-processing is required if uppercase or punctuation reconstruction is needed.

Bias considerations

• Speaker bias: Most training speakers are likely adult speakers from urban areas of Benin. Performance on speakers from other demographics (children, elderly, rural speakers) has not been formally evaluated.
• Gender balance: The gender distribution of speakers in the corpus has not been formally analyzed and may introduce performance disparities.

Environmental Impact

• Hardware Type: NVIDIA T4 GPU
• Hours used: ~2 hours
• Cloud Provider: Google Colab
• Carbon Emitted: Not formally measured. Approximate estimate based on the ML CO2 Impact calculator: ~0.15-0.25 kg CO2eq.

Citation

If you use this model, please cite:

@mastersthesis{hounkponou2026fonbench,
  title={Evaluation and adaptation of automatic speech recognition models for the Fon language},
  author={Hounkponou, Inès Assia},
  school={ESGIS Bénin},
  year={2026}
}

Resources

• 🎤 FonBench Leaderboard (interactive)
• 🤗 LoRA variant (r=32, more efficient)
• 📊 W&B project (all experiments)
• 📄 Dataset on HF

Acknowledgements

This work was supervised by Prof. Fréjus LALEYE at ESGIS Bénin. The author thanks the alaleye/fon dataset providers for their contribution to Fon language resources, and the MMS team at Meta AI for releasing the multilingual pre-trained model that served as the foundation for this work.