Shumatsurontek commited on
Commit
6286672
Β·
verified Β·
1 Parent(s): 4bc4f68

Add benchmark results (mmlu,gsm8k,arc_challenge,hellaswag)

Browse files
Files changed (1) hide show
  1. README.md +13 -130
README.md CHANGED
@@ -160,136 +160,19 @@ or production use without human review of generated SQL.
160
 
161
  ## Benchmark Results
162
 
163
- Compared against baseline [`Qwen/Qwen3.5-4B`](https://huggingface.co/Qwen/Qwen3.5-4B) using [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness) with vLLM backend on NVIDIA L40S.
164
-
165
- > Note: evaluated on 50 samples per task (quick test).
166
-
167
- | Task | Metric | Baseline | Finetuned | Delta |
168
- |---|---|---|---|---|
169
- | **mmlu_human_aging** | acc | 80.0% | **78.0%** | πŸ”΄ -2.0% |
170
- | **mmlu_human_aging** | acc_stderr | 5.7% | **5.9%** | 🟒 +0.2% |
171
- | **mmlu_nutrition** | acc | 88.0% | **88.0%** | βšͺ 0.0% |
172
- | **mmlu_nutrition** | acc_stderr | 4.6% | **4.6%** | βšͺ 0.0% |
173
- | **mmlu_anatomy** | acc | 68.0% | **66.0%** | πŸ”΄ -2.0% |
174
- | **mmlu_anatomy** | acc_stderr | 6.7% | **6.8%** | 🟒 +0.1% |
175
- | **mmlu_high_school_chemistry** | acc | 84.0% | **80.0%** | πŸ”΄ -4.0% |
176
- | **mmlu_high_school_chemistry** | acc_stderr | 5.2% | **5.7%** | 🟒 +0.5% |
177
- | **mmlu_professional_medicine** | acc | 86.0% | **78.0%** | πŸ”΄ -8.0% |
178
- | **mmlu_professional_medicine** | acc_stderr | 5.0% | **5.9%** | 🟒 +1.0% |
179
- | **mmlu_moral_scenarios** | acc | 46.0% | **42.0%** | πŸ”΄ -4.0% |
180
- | **mmlu_moral_scenarios** | acc_stderr | 7.1% | **7.0%** | πŸ”΄ -0.1% |
181
- | **mmlu_high_school_government_and_politics** | acc | 96.0% | **94.0%** | πŸ”΄ -2.0% |
182
- | **mmlu_high_school_government_and_politics** | acc_stderr | 2.8% | **3.4%** | 🟒 +0.6% |
183
- | **mmlu_high_school_psychology** | acc | 96.0% | **94.0%** | πŸ”΄ -2.0% |
184
- | **mmlu_high_school_psychology** | acc_stderr | 2.8% | **3.4%** | 🟒 +0.6% |
185
- | **mmlu_high_school_statistics** | acc | 62.0% | **60.0%** | πŸ”΄ -2.0% |
186
- | **mmlu_high_school_statistics** | acc_stderr | 6.9% | **7.0%** | 🟒 +0.1% |
187
- | **mmlu_high_school_mathematics** | acc | 48.0% | **58.0%** | 🟒 +10.0% |
188
- | **mmlu_high_school_mathematics** | acc_stderr | 7.1% | **7.0%** | πŸ”΄ -0.1% |
189
- | **mmlu_abstract_algebra** | acc | 58.0% | **60.0%** | 🟒 +2.0% |
190
- | **mmlu_abstract_algebra** | acc_stderr | 7.0% | **7.0%** | πŸ”΄ -0.1% |
191
- | **mmlu_computer_security** | acc | 88.0% | **82.0%** | πŸ”΄ -6.0% |
192
- | **mmlu_computer_security** | acc_stderr | 4.6% | **5.5%** | 🟒 +0.9% |
193
- | **mmlu_miscellaneous** | acc | 82.0% | **78.0%** | πŸ”΄ -4.0% |
194
- | **mmlu_miscellaneous** | acc_stderr | 5.5% | **5.9%** | 🟒 +0.4% |
195
- | **mmlu_high_school_european_history** | acc | 82.0% | **76.0%** | πŸ”΄ -6.0% |
196
- | **mmlu_high_school_european_history** | acc_stderr | 5.5% | **6.1%** | 🟒 +0.6% |
197
- | **mmlu_high_school_computer_science** | acc | 88.0% | **84.0%** | πŸ”΄ -4.0% |
198
- | **mmlu_high_school_computer_science** | acc_stderr | 4.6% | **5.2%** | 🟒 +0.6% |
199
- | **mmlu_high_school_physics** | acc | 68.0% | **64.0%** | πŸ”΄ -4.0% |
200
- | **mmlu_high_school_physics** | acc_stderr | 6.7% | **6.9%** | 🟒 +0.2% |
201
- | **mmlu_human_sexuality** | acc | 84.0% | **86.0%** | 🟒 +2.0% |
202
- | **mmlu_human_sexuality** | acc_stderr | 5.2% | **5.0%** | πŸ”΄ -0.3% |
203
- | **mmlu_machine_learning** | acc | 56.0% | **58.0%** | 🟒 +2.0% |
204
- | **mmlu_machine_learning** | acc_stderr | 7.1% | **7.0%** | πŸ”΄ -0.0% |
205
- | **mmlu_astronomy** | acc | 94.0% | **94.0%** | βšͺ 0.0% |
206
- | **mmlu_astronomy** | acc_stderr | 3.4% | **3.4%** | βšͺ 0.0% |
207
- | **mmlu_professional_law** | acc | 52.0% | **52.0%** | βšͺ 0.0% |
208
- | **mmlu_professional_law** | acc_stderr | 7.1% | **7.1%** | βšͺ 0.0% |
209
- | **mmlu_high_school_us_history** | acc | 90.0% | **84.0%** | πŸ”΄ -6.0% |
210
- | **mmlu_high_school_us_history** | acc_stderr | 4.3% | **5.2%** | 🟒 +0.9% |
211
- | **mmlu_management** | acc | 94.0% | **94.0%** | βšͺ 0.0% |
212
- | **mmlu_management** | acc_stderr | 3.4% | **3.4%** | βšͺ 0.0% |
213
- | **mmlu_college_computer_science** | acc | 72.0% | **74.0%** | 🟒 +2.0% |
214
- | **mmlu_college_computer_science** | acc_stderr | 6.4% | **6.3%** | πŸ”΄ -0.1% |
215
- | **mmlu_virology** | acc | 60.0% | **60.0%** | βšͺ 0.0% |
216
- | **mmlu_virology** | acc_stderr | 7.0% | **7.0%** | βšͺ 0.0% |
217
- | **mmlu_jurisprudence** | acc | 82.0% | **86.0%** | 🟒 +4.0% |
218
- | **mmlu_jurisprudence** | acc_stderr | 5.5% | **5.0%** | πŸ”΄ -0.5% |
219
- | **mmlu_college_medicine** | acc | 82.0% | **80.0%** | πŸ”΄ -2.0% |
220
- | **mmlu_college_medicine** | acc_stderr | 5.5% | **5.7%** | 🟒 +0.2% |
221
- | **mmlu_professional_psychology** | acc | 86.0% | **80.0%** | πŸ”΄ -6.0% |
222
- | **mmlu_professional_psychology** | acc_stderr | 5.0% | **5.7%** | 🟒 +0.8% |
223
- | **mmlu_conceptual_physics** | acc | 86.0% | **88.0%** | 🟒 +2.0% |
224
- | **mmlu_conceptual_physics** | acc_stderr | 5.0% | **4.6%** | πŸ”΄ -0.3% |
225
- | **mmlu_social_sciences** | acc | 83.0% | **79.3%** | πŸ”΄ -3.7% |
226
- | **mmlu_social_sciences** | acc_stderr | 1.5% | **1.6%** | 🟒 +0.1% |
227
- | **mmlu_electrical_engineering** | acc | 82.0% | **84.0%** | 🟒 +2.0% |
228
- | **mmlu_electrical_engineering** | acc_stderr | 5.5% | **5.2%** | πŸ”΄ -0.2% |
229
- | **mmlu_international_law** | acc | 90.0% | **88.0%** | πŸ”΄ -2.0% |
230
- | **mmlu_international_law** | acc_stderr | 4.3% | **4.6%** | 🟒 +0.4% |
231
- | **mmlu_global_facts** | acc | 42.0% | **36.0%** | πŸ”΄ -6.0% |
232
- | **mmlu_global_facts** | acc_stderr | 7.0% | **6.9%** | πŸ”΄ -0.2% |
233
- | **mmlu_high_school_geography** | acc | 90.0% | **80.0%** | πŸ”΄ -10.0% |
234
- | **mmlu_high_school_geography** | acc_stderr | 4.3% | **5.7%** | 🟒 +1.4% |
235
- | **mmlu_philosophy** | acc | 82.0% | **82.0%** | βšͺ 0.0% |
236
- | **mmlu_philosophy** | acc_stderr | 5.5% | **5.5%** | βšͺ 0.0% |
237
- | **mmlu_world_religions** | acc | 86.0% | **84.0%** | πŸ”΄ -2.0% |
238
- | **mmlu_world_religions** | acc_stderr | 5.0% | **5.2%** | 🟒 +0.3% |
239
- | **mmlu_sociology** | acc | 88.0% | **84.0%** | πŸ”΄ -4.0% |
240
- | **mmlu_sociology** | acc_stderr | 4.6% | **5.2%** | 🟒 +0.6% |
241
- | **mmlu_stem** | acc | 73.5% | **72.3%** | πŸ”΄ -1.1% |
242
- | **mmlu_stem** | acc_stderr | 1.4% | **1.4%** | 🟒 +0.0% |
243
- | **mmlu_marketing** | acc | 92.0% | **92.0%** | βšͺ 0.0% |
244
- | **mmlu_marketing** | acc_stderr | 3.9% | **3.9%** | βšͺ 0.0% |
245
- | **mmlu_public_relations** | acc | 74.0% | **64.0%** | πŸ”΄ -10.0% |
246
- | **mmlu_public_relations** | acc_stderr | 6.3% | **6.9%** | 🟒 +0.6% |
247
- | **mmlu_college_mathematics** | acc | 60.0% | **52.0%** | πŸ”΄ -8.0% |
248
- | **mmlu_college_mathematics** | acc_stderr | 7.0% | **7.1%** | 🟒 +0.1% |
249
- | **mmlu_elementary_mathematics** | acc | 80.0% | **74.0%** | πŸ”΄ -6.0% |
250
- | **mmlu_elementary_mathematics** | acc_stderr | 5.7% | **6.3%** | 🟒 +0.6% |
251
- | **mmlu_logical_fallacies** | acc | 82.0% | **74.0%** | πŸ”΄ -8.0% |
252
- | **mmlu_logical_fallacies** | acc_stderr | 5.5% | **6.3%** | 🟒 +0.8% |
253
- | **mmlu_medical_genetics** | acc | 90.0% | **92.0%** | 🟒 +2.0% |
254
- | **mmlu_medical_genetics** | acc_stderr | 4.3% | **3.9%** | πŸ”΄ -0.4% |
255
- | **mmlu_econometrics** | acc | 66.0% | **64.0%** | πŸ”΄ -2.0% |
256
- | **mmlu_econometrics** | acc_stderr | 6.8% | **6.9%** | 🟒 +0.1% |
257
- | **mmlu_us_foreign_policy** | acc | 80.0% | **80.0%** | βšͺ 0.0% |
258
- | **mmlu_us_foreign_policy** | acc_stderr | 5.7% | **5.7%** | βšͺ 0.0% |
259
- | **mmlu** | acc | 77.1% | **75.0%** | πŸ”΄ -2.1% |
260
- | **mmlu** | acc_stderr | 0.8% | **0.8%** | 🟒 +0.0% |
261
- | **mmlu_college_biology** | acc | 90.0% | **88.0%** | πŸ”΄ -2.0% |
262
- | **mmlu_college_biology** | acc_stderr | 4.3% | **4.6%** | 🟒 +0.4% |
263
- | **mmlu_moral_disputes** | acc | 64.0% | **66.0%** | 🟒 +2.0% |
264
- | **mmlu_moral_disputes** | acc_stderr | 6.9% | **6.8%** | πŸ”΄ -0.1% |
265
- | **mmlu_high_school_biology** | acc | 90.0% | **88.0%** | πŸ”΄ -2.0% |
266
- | **mmlu_high_school_biology** | acc_stderr | 4.3% | **4.6%** | 🟒 +0.4% |
267
- | **mmlu_professional_accounting** | acc | 52.0% | **54.0%** | 🟒 +2.0% |
268
- | **mmlu_professional_accounting** | acc_stderr | 7.1% | **7.1%** | πŸ”΄ -0.0% |
269
- | **mmlu_security_studies** | acc | 74.0% | **70.0%** | πŸ”΄ -4.0% |
270
- | **mmlu_security_studies** | acc_stderr | 6.3% | **6.6%** | 🟒 +0.3% |
271
- | **mmlu_high_school_macroeconomics** | acc | 74.0% | **68.0%** | πŸ”΄ -6.0% |
272
- | **mmlu_high_school_macroeconomics** | acc_stderr | 6.3% | **6.7%** | 🟒 +0.4% |
273
- | **mmlu_high_school_microeconomics** | acc | 88.0% | **88.0%** | βšͺ 0.0% |
274
- | **mmlu_high_school_microeconomics** | acc_stderr | 4.6% | **4.6%** | βšͺ 0.0% |
275
- | **mmlu_clinical_knowledge** | acc | 78.0% | **78.0%** | βšͺ 0.0% |
276
- | **mmlu_clinical_knowledge** | acc_stderr | 5.9% | **5.9%** | βšͺ 0.0% |
277
- | **mmlu_business_ethics** | acc | 84.0% | **84.0%** | βšͺ 0.0% |
278
- | **mmlu_business_ethics** | acc_stderr | 5.2% | **5.2%** | βšͺ 0.0% |
279
- | **mmlu_college_chemistry** | acc | 54.0% | **54.0%** | βšͺ 0.0% |
280
- | **mmlu_college_chemistry** | acc_stderr | 7.1% | **7.1%** | βšͺ 0.0% |
281
- | **mmlu_other** | acc | 77.7% | **76.3%** | πŸ”΄ -1.4% |
282
- | **mmlu_other** | acc_stderr | 1.5% | **1.6%** | 🟒 +0.0% |
283
- | **mmlu_prehistory** | acc | 80.0% | **76.0%** | πŸ”΄ -4.0% |
284
- | **mmlu_prehistory** | acc_stderr | 5.7% | **6.1%** | 🟒 +0.4% |
285
- | **mmlu_humanities** | acc | 76.5% | **73.9%** | πŸ”΄ -2.6% |
286
- | **mmlu_humanities** | acc_stderr | 1.6% | **1.7%** | 🟒 +0.1% |
287
- | **mmlu_high_school_world_history** | acc | 90.0% | **86.0%** | πŸ”΄ -4.0% |
288
- | **mmlu_high_school_world_history** | acc_stderr | 4.3% | **5.0%** | 🟒 +0.7% |
289
- | **mmlu_college_physics** | acc | 68.0% | **66.0%** | πŸ”΄ -2.0% |
290
- | **mmlu_college_physics** | acc_stderr | 6.7% | **6.8%** | 🟒 +0.1% |
291
- | **mmlu_formal_logic** | acc | 68.0% | **64.0%** | πŸ”΄ -4.0% |
292
- | **mmlu_formal_logic** | acc_stderr | 6.7% | **6.9%** | 🟒 +0.2% |
293
 
294
  ## Citation
295
 
 
160
 
161
  ## Benchmark Results
162
 
163
+ Evaluated against baseline [`unsloth/Qwen3.5-4B`](https://huggingface.co/unsloth/Qwen3.5-4B) using [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness) on NVIDIA L40S.
164
+
165
+ > Evaluated on 50 samples per task.
166
+
167
+ | Benchmark | Baseline | Finetuned | Delta |
168
+ |---|:---:|:---:|:---:|
169
+ | **MMLU: STEM** | 73.5 | 71.8 | πŸ”΄ -1.7 |
170
+ | **MMLU: HUMANITIES** | 76.5 | 73.4 | πŸ”΄ -3.1 |
171
+ | **HELLASWAG** | 48.0 | 52.0 | 🟒 +4.0 |
172
+ | **MMLU** | 77.1 | 74.7 | πŸ”΄ -2.4 |
173
+ | **MMLU: OTHER** | 77.7 | 75.8 | πŸ”΄ -1.8 |
174
+ | **ARC_CHALLENGE** | 60.0 | 60.0 | βšͺ 0.0 |
175
+ | **MMLU: SOCIAL SCIENCES** | 83.0 | 79.7 | πŸ”΄ -3.3 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
176
 
177
  ## Citation
178