Update benchmark notes
Browse files
README.md
CHANGED
|
@@ -31,11 +31,28 @@ model remains in the upstream Hugging Face repository.
|
|
| 31 |
|
| 32 |
| Variant | Folder | Quantization | Size | Best fit |
|
| 33 |
| --- | --- | --- | ---: | --- |
|
| 34 |
-
| MLX 8-bit | `mlx-8bit/` | affine, group size 64 |
|
| 35 |
-
| MLX 6-bit | `mlx-6bit/` | affine, group size 64 |
|
| 36 |
-
| MLX 4-bit | `mlx-4bit/` | affine, group size 64 |
|
| 37 |
|
| 38 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
|
| 40 |
## Usage
|
| 41 |
|
|
|
|
| 31 |
|
| 32 |
| Variant | Folder | Quantization | Size | Best fit |
|
| 33 |
| --- | --- | --- | ---: | --- |
|
| 34 |
+
| MLX 8-bit | `mlx-8bit/` | affine, group size 64 | 36.85 GB | Highest-quality local MLX run. |
|
| 35 |
+
| MLX 6-bit | `mlx-6bit/` | affine, group size 64 | 28.19 GB | Balanced memory and quality. |
|
| 36 |
+
| MLX 4-bit | `mlx-4bit/` | affine, group size 64 | 19.53 GB | Smallest footprint and easiest local run. |
|
| 37 |
|
| 38 |
+
## Local benchmark notes
|
| 39 |
+
|
| 40 |
+
Initial local testing was performed on a Mac Studio with an M4 Max and 64 GB
|
| 41 |
+
unified memory using oMLX.
|
| 42 |
+
|
| 43 |
+
| Variant | Result | Notes |
|
| 44 |
+
| --- | --- | --- |
|
| 45 |
+
| MLX 8-bit | Not loaded under default memory cap | oMLX projected 54.62 GB total memory use against a 51.84 GB effective ceiling. The model files themselves are 36.85 GB; the higher runtime estimate includes the current oMLX process footprint, MLX runtime/allocator overhead, buffers, and KV/cache planning. |
|
| 46 |
+
|
| 47 |
+
The 8-bit variant should be retested after raising the Apple GPU wired-memory
|
| 48 |
+
cap and restarting the local serving process, for example:
|
| 49 |
+
|
| 50 |
+
```bash
|
| 51 |
+
sudo sysctl iogpu.wired_limit_mb=59392
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
Throughput numbers are not published yet. This section will be updated after a
|
| 55 |
+
successful full benchmark run.
|
| 56 |
|
| 57 |
## Usage
|
| 58 |
|