Add benchmark screenshots and results
Browse files
README.md
CHANGED
|
@@ -40,9 +40,48 @@ model remains in the upstream Hugging Face repository.
|
|
| 40 |
Initial local testing was performed on a Mac Studio with an M4 Max and 64 GB
|
| 41 |
unified memory using oMLX.
|
| 42 |
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
|
| 47 |
The 8-bit variant should be retested after raising the Apple GPU wired-memory
|
| 48 |
cap and restarting the local serving process, for example:
|
|
|
|
| 40 |
Initial local testing was performed on a Mac Studio with an M4 Max and 64 GB
|
| 41 |
unified memory using oMLX.
|
| 42 |
|
| 43 |
+
### 4-bit
|
| 44 |
+
|
| 45 |
+
Screenshot: [`benchmarks/grug-35b-a3b-4bit-omlx.png`](benchmarks/grug-35b-a3b-4bit-omlx.png)
|
| 46 |
+
|
| 47 |
+
| Test | TTFT | Prompt TPS | Generation TPS | End-to-end | Throughput | Peak memory |
|
| 48 |
+
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 49 |
+
| pp1024 / tg128 | 566.6 ms | 1807.2 tok/s | 103.9 tok/s | 1.805 s | 638.2 tok/s | 19.35 GB |
|
| 50 |
+
| pp4096 / tg128 | 2258.3 ms | 1813.7 tok/s | 100.2 tok/s | 3.545 s | 1191.5 tok/s | 20.08 GB |
|
| 51 |
+
| pp8192 / tg128 | 4690.0 ms | 1746.7 tok/s | 97.1 tok/s | 6.019 s | 1382.2 tok/s | 20.50 GB |
|
| 52 |
+
| pp16384 / tg128 | 12050.5 ms | 1359.6 tok/s | 90.3 tok/s | 13.478 s | 1225.1 tok/s | 20.38 GB |
|
| 53 |
+
|
| 54 |
+
Continuous batching, pp1024 / tg128:
|
| 55 |
+
|
| 56 |
+
| Batch | Generation TPS | Speedup | Prompt TPS | Prompt TPS/request | TTFT | End-to-end |
|
| 57 |
+
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 58 |
+
| 1x | 103.9 tok/s | 1.00x | 1807.2 tok/s | 1807.2 tok/s | 566.6 ms | 1.805 s |
|
| 59 |
+
| 2x | 116.1 tok/s | 1.12x | 1468.4 tok/s | 734.2 tok/s | 1394.6 ms | 3.600 s |
|
| 60 |
+
| 4x | 180.0 tok/s | 1.73x | 1549.7 tok/s | 387.4 tok/s | 2530.3 ms | 5.488 s |
|
| 61 |
+
|
| 62 |
+
### 6-bit
|
| 63 |
+
|
| 64 |
+
Screenshot: [`benchmarks/grug-35b-a3b-6bit-omlx.png`](benchmarks/grug-35b-a3b-6bit-omlx.png)
|
| 65 |
+
|
| 66 |
+
| Test | TTFT | Prompt TPS | Generation TPS | End-to-end | Throughput | Peak memory |
|
| 67 |
+
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 68 |
+
| pp1024 / tg128 | 645.4 ms | 1586.6 tok/s | 87.1 tok/s | 2.119 s | 543.5 tok/s | 45.58 GB |
|
| 69 |
+
| pp4096 / tg128 | 3175.8 ms | 1289.8 tok/s | 82.9 tok/s | 4.732 s | 892.6 tok/s | 45.16 GB |
|
| 70 |
+
| pp8192 / tg128 | 6701.5 ms | 1222.4 tok/s | 80.9 tok/s | 8.294 s | 1003.1 tok/s | 45.30 GB |
|
| 71 |
+
| pp16384 / tg128 | 22365.0 ms | 732.6 tok/s | 76.7 tok/s | 24.045 s | 686.7 tok/s | 45.68 GB |
|
| 72 |
+
|
| 73 |
+
The 6-bit batching run shown in
|
| 74 |
+
[`benchmarks/grug-35b-a3b-6bit-prefill-cap.png`](benchmarks/grug-35b-a3b-6bit-prefill-cap.png)
|
| 75 |
+
hit oMLX's prefill safety cap at the tested settings: predicted peak exceeded
|
| 76 |
+
45.5 GB, which was 90% of the effective 50.5 GB ceiling.
|
| 77 |
+
|
| 78 |
+
### 8-bit
|
| 79 |
+
|
| 80 |
+
The 8-bit variant did not load under the default local memory cap. oMLX
|
| 81 |
+
projected 54.62 GB total memory use against a 51.84 GB effective ceiling. The
|
| 82 |
+
model files themselves are 36.85 GB; the higher runtime estimate includes the
|
| 83 |
+
current oMLX process footprint, MLX runtime/allocator overhead, buffers, and
|
| 84 |
+
KV/cache planning.
|
| 85 |
|
| 86 |
The 8-bit variant should be retested after raising the Apple GPU wired-memory
|
| 87 |
cap and restarting the local serving process, for example:
|