andrewbawitlung/wav2vec2-bert-mizonal2-lus-v25.03.2

wav2vec2-bert-Mizo-Lus-v25.03.2

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the generator dataset. It achieves the following results on the evaluation set:

• Loss: 0.4014
• Wer: 0.1790

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 16
• eval_batch_size: 8
• seed: 49
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 1200
• num_epochs: 70
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
0.7975	2.62	1000	0.5053	0.4364
0.8518	5.24	2000	0.4891	0.4265
0.7219	7.86	3000	0.4057	0.3543
0.6005	10.48	4000	0.3664	0.3165
0.5312	13.11	5000	0.3695	0.2991
0.4785	15.73	6000	0.3264	0.2761
0.4022	18.35	7000	0.2972	0.2724
0.3761	20.97	8000	0.3167	0.2473
0.3299	23.59	9000	0.3106	0.2510
0.2788	26.21	10000	0.3112	0.2315
0.282	28.83	11000	0.2876	0.2277
0.2395	31.45	12000	0.3009	0.2179
0.1922	34.08	13000	0.3159	0.2093
0.1741	36.7	14000	0.2760	0.2135
0.1719	39.32	15000	0.3127	0.2221
0.1499	41.94	16000	0.3021	0.2098
0.1367	44.56	17000	0.3238	0.2032
0.1101	47.18	18000	0.3079	0.1894
0.1114	49.8	19000	0.3370	0.1851
0.0957	52.42	20000	0.3311	0.1936
0.084	55.05	21000	0.3571	0.1875
0.0668	57.67	22000	0.3858	0.1854
0.0665	60.29	23000	0.3940	0.1883
0.0553	62.91	24000	0.3668	0.1917
0.0558	65.53	25000	0.3928	0.1783
0.049	68.15	26000	0.4014	0.1790

Framework versions

• Transformers 4.37.2
• Pytorch 2.3.1+cu121
• Datasets 2.16.1
• Tokenizers 0.15.1