Update README.md
Browse files
README.md
CHANGED
|
@@ -4,6 +4,9 @@ tags:
|
|
| 4 |
- mlx
|
| 5 |
- oq
|
| 6 |
- quantized
|
|
|
|
|
|
|
|
|
|
| 7 |
---
|
| 8 |
|
| 9 |
# Darwin-35B-A3B-Opus-oQ4
|
|
@@ -16,3 +19,33 @@ This model was quantized using [oQ](https://github.com/jundot/omlx) (oMLX v0.3.1
|
|
| 16 |
- **Bits**: 4
|
| 17 |
- **Group size**: 64
|
| 18 |
- **Format**: MLX safetensors
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4 |
- mlx
|
| 5 |
- oq
|
| 6 |
- quantized
|
| 7 |
+
license: apache-2.0
|
| 8 |
+
base_model:
|
| 9 |
+
- FINAL-Bench/Darwin-35B-A3B-Opus
|
| 10 |
---
|
| 11 |
|
| 12 |
# Darwin-35B-A3B-Opus-oQ4
|
|
|
|
| 19 |
- **Bits**: 4
|
| 20 |
- **Group size**: 64
|
| 21 |
- **Format**: MLX safetensors
|
| 22 |
+
|
| 23 |
+
# Benchmark
|
| 24 |
+
|
| 25 |
+
| Model | File size | MMLU | JMMLU | HELLASWAG | ARC_CHALLENGE | GSM8K |
|
| 26 |
+
| -------------------------------------------------------------- | --------- | ----- | ----- | --------- | ------------- | ----- |
|
| 27 |
+
| Darwin-35B-A3B-Opus-oQ3 | 16.8 GB | 77.7% | 77.0% | 87.3% | 94.7% | 92.3% |
|
| 28 |
+
| Darwin-35B-A3B-Opus-oQ3.5 | 18.1 GB | 79.0% | 77.0% | 88.3% | 94.7% | 91.7% |
|
| 29 |
+
| Darwin-35B-A3B-Opus-oQ4 | 21.1 GB | 83.7% | 77.3% | 94.3% | 94.7% | 92.0% |
|
| 30 |
+
|
| 31 |
+
## Detail
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
| Model | Benchmark | Accuracy | Correct | Total | Time(s) |
|
| 36 |
+
| -------------------------------------------------------------- | ------------- | -------- | ------- | ----- | ------- |
|
| 37 |
+
| Darwin-35B-A3B-Opus-oQ3 | MMLU | 77.7% | 233 | 300 | 579.3 |
|
| 38 |
+
| Darwin-35B-A3B-Opus-oQ3 | JMMLU | 77.0% | 231 | 300 | 215.6 |
|
| 39 |
+
| Darwin-35B-A3B-Opus-oQ3 | HELLASWAG | 87.3% | 262 | 300 | 273.5 |
|
| 40 |
+
| Darwin-35B-A3B-Opus-oQ3 | ARC_CHALLENGE | 94.7% | 284 | 300 | 186.5 |
|
| 41 |
+
| Darwin-35B-A3B-Opus-oQ3 | GSM8K | 92.3% | 277 | 300 | 1064.6 |
|
| 42 |
+
| Darwin-35B-A3B-Opus-oQ3.5 | MMLU | 79.0% | 237 | 300 | 542.6 |
|
| 43 |
+
| Darwin-35B-A3B-Opus-oQ3.5 | JMMLU | 77.0% | 231 | 300 | 216.8 |
|
| 44 |
+
| Darwin-35B-A3B-Opus-oQ3.5 | HELLASWAG | 88.3% | 265 | 300 | 274.9 |
|
| 45 |
+
| Darwin-35B-A3B-Opus-oQ3.5 | ARC_CHALLENGE | 94.7% | 284 | 300 | 189.7 |
|
| 46 |
+
| Darwin-35B-A3B-Opus-oQ3.5 | GSM8K | 91.7% | 275 | 300 | 1130.3 |
|
| 47 |
+
| Darwin-35B-A3B-Opus-oQ4 | MMLU | 83.7% | 251 | 300 | 601.9 |
|
| 48 |
+
| Darwin-35B-A3B-Opus-oQ4 | JMMLU | 77.3% | 232 | 300 | 220 |
|
| 49 |
+
| Darwin-35B-A3B-Opus-oQ4 | HELLASWAG | 94.3% | 283 | 300 | 297.4 |
|
| 50 |
+
| Darwin-35B-A3B-Opus-oQ4 | ARC_CHALLENGE | 94.7% | 284 | 300 | 188.8 |
|
| 51 |
+
| Darwin-35B-A3B-Opus-oQ4 | GSM8K | 92.0% | 276 | 300 | 1230.5 |
|