bert-base-cased-azerbaijani

This model is a fine-tuned version of google-bert/bert-base-multilingual-cased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.7046

We thank Microsoft Accelerating Foundation Models Research Program for supporting our research. Authors: Mammad Hajili, Duygu Ataman

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.1035	0.2500	15300	0.9753
0.988	0.5000	30600	0.8985
0.9276	0.7500	45900	0.8464
0.8903	1.0000	61200	0.7815
0.8631	1.2500	76500	0.7778
0.8435	1.5000	91800	0.7642
0.8246	1.7500	107100	0.7496
0.8132	2.0000	122400	0.7372
0.7999	2.2500	137700	0.7270
0.7924	2.5000	153000	0.7270
0.7876	2.7500	168300	0.7178

Framework versions

• Transformers 4.40.2
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1