Add Turkish MMLU benchmark results
Browse files
README.md
CHANGED
|
@@ -568,3 +568,33 @@ Open the notebook in a Tesla T4 Google Colab runtime and run the cells in order.
|
|
| 568 |
Import Unsloth before TRL/Transformers/PEFT, configure Hugging Face credentials
|
| 569 |
through Colab Secrets or environment variables, and add target-domain inference
|
| 570 |
and evaluation cells before treating the model as production-ready.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 568 |
Import Unsloth before TRL/Transformers/PEFT, configure Hugging Face credentials
|
| 569 |
through Colab Secrets or environment variables, and add target-domain inference
|
| 570 |
and evaluation cells before treating the model as production-ready.
|
| 571 |
+
|
| 572 |
+
## Turkish MMLU benchmark
|
| 573 |
+
|
| 574 |
+
The fine-tuned model was evaluated with the existing Turkish MMLU benchmark
|
| 575 |
+
algorithm from
|
| 576 |
+
[`alibayram/yapay_zeka_turkce_mmlu_bolum_sonuclari`](https://huggingface.co/datasets/alibayram/yapay_zeka_turkce_mmlu_bolum_sonuclari/blob/main/olcum.py).
|
| 577 |
+
All models received the same Turkish multiple-choice prompt, random seed (`42`),
|
| 578 |
+
generation limit (`42` new tokens), answer parsing, semantic-similarity fallback,
|
| 579 |
+
and scoring procedure. The run used a Hugging Face Job with one NVIDIA T4 GPU.
|
| 580 |
+
|
| 581 |
+
| Model | Parameters | Correct answers | Accuracy | Test duration |
|
| 582 |
+
|---|---:|---:|---:|---:|
|
| 583 |
+
| `unsloth/gemma-3-1b-it` (base) | 1B | 2,680 | **43.23%** | 1,072.517 s |
|
| 584 |
+
| `Qwen/Qwen2.5-1.5B-Instruct` | 1.5B | 2,485 | **40.08%** | 1,323.730 s |
|
| 585 |
+
| `berkbirkan/gemma-3-lora-finetune-x-replies` | 1B | 2,411 | **38.89%** | 1,855.230 s |
|
| 586 |
+
|
| 587 |
+
The reply-focused fine-tuned model scored 4.34 percentage points below its base
|
| 588 |
+
model and 1.19 points below Qwen 2.5 1.5B. This result does not show that the
|
| 589 |
+
fine-tuning failed at its intended social-media reply task: the training data
|
| 590 |
+
targeted short Turkish X replies, whereas MMLU measures multiple-choice academic
|
| 591 |
+
knowledge and reasoning. It does show that this short LoRA run did not improve
|
| 592 |
+
general Turkish MMLU performance and may have traded some general benchmark
|
| 593 |
+
ability for target-domain behavior.
|
| 594 |
+
|
| 595 |
+
Detailed outputs:
|
| 596 |
+
|
| 597 |
+
- [Overall leaderboard](https://huggingface.co/datasets/berkbirkan/yapay_zeka_turkce_mmlu_liderlik_tablosu)
|
| 598 |
+
- [Per-section results](https://huggingface.co/datasets/berkbirkan/yapay_zeka_turkce_mmlu_bolum_sonuclari)
|
| 599 |
+
- [Model answers](https://huggingface.co/datasets/berkbirkan/yapay_zeka_turkce_mmlu_model_cevaplari)
|
| 600 |
+
- [Hugging Face Job](https://huggingface.co/jobs/berkbirkan/6a6124e113e6ef894d54c3ac)
|