Grug-35B-A3B MLX
Apple Silicon MLX quantizations of
kai-os/Grug-35B-A3B, packaged
as a single Hugging Face repo with one folder per quantization level.
This repository contains MLX safetensors folders, not GGUF files. The source model remains in the upstream Hugging Face repository.
Available variants
| Variant | Folder | Quantization | Size | Best fit |
|---|---|---|---|---|
| MLX 8-bit | mlx-8bit/ |
affine, group size 64 | 36.85 GB | Highest-quality local MLX run. |
| MLX 6-bit | mlx-6bit/ |
affine, group size 64 | 28.19 GB | Balanced memory and quality. |
| MLX 4-bit | mlx-4bit/ |
affine, group size 64 | 19.53 GB | Smallest footprint and easiest local run. |
Local benchmark notes
Initial local testing was performed on a Mac Studio with an M4 Max and 64 GB unified memory using oMLX.
4-bit
Screenshot: benchmarks/grug-35b-a3b-4bit-omlx.png
| Test | TTFT | Prompt TPS | Generation TPS | End-to-end | Throughput | Peak memory |
|---|---|---|---|---|---|---|
| pp1024 / tg128 | 566.6 ms | 1807.2 tok/s | 103.9 tok/s | 1.805 s | 638.2 tok/s | 19.35 GB |
| pp4096 / tg128 | 2258.3 ms | 1813.7 tok/s | 100.2 tok/s | 3.545 s | 1191.5 tok/s | 20.08 GB |
| pp8192 / tg128 | 4690.0 ms | 1746.7 tok/s | 97.1 tok/s | 6.019 s | 1382.2 tok/s | 20.50 GB |
| pp16384 / tg128 | 12050.5 ms | 1359.6 tok/s | 90.3 tok/s | 13.478 s | 1225.1 tok/s | 20.38 GB |
Continuous batching, pp1024 / tg128:
| Batch | Generation TPS | Speedup | Prompt TPS | Prompt TPS/request | TTFT | End-to-end |
|---|---|---|---|---|---|---|
| 1x | 103.9 tok/s | 1.00x | 1807.2 tok/s | 1807.2 tok/s | 566.6 ms | 1.805 s |
| 2x | 116.1 tok/s | 1.12x | 1468.4 tok/s | 734.2 tok/s | 1394.6 ms | 3.600 s |
| 4x | 180.0 tok/s | 1.73x | 1549.7 tok/s | 387.4 tok/s | 2530.3 ms | 5.488 s |
6-bit
Screenshot: benchmarks/grug-35b-a3b-6bit-omlx.png
| Test | TTFT | Prompt TPS | Generation TPS | End-to-end | Throughput | Peak memory |
|---|---|---|---|---|---|---|
| pp1024 / tg128 | 645.4 ms | 1586.6 tok/s | 87.1 tok/s | 2.119 s | 543.5 tok/s | 45.58 GB |
| pp4096 / tg128 | 3175.8 ms | 1289.8 tok/s | 82.9 tok/s | 4.732 s | 892.6 tok/s | 45.16 GB |
| pp8192 / tg128 | 6701.5 ms | 1222.4 tok/s | 80.9 tok/s | 8.294 s | 1003.1 tok/s | 45.30 GB |
| pp16384 / tg128 | 22365.0 ms | 732.6 tok/s | 76.7 tok/s | 24.045 s | 686.7 tok/s | 45.68 GB |
The 6-bit batching run shown in
benchmarks/grug-35b-a3b-6bit-prefill-cap.png
hit oMLX's prefill safety cap at the tested settings: predicted peak exceeded
45.5 GB, which was 90% of the effective 50.5 GB ceiling.
8-bit
The 8-bit variant could not be thoroughly tested under the default local memory cap. In the initial oMLX test environment, oMLX projected 54.62 GB total memory use against a 51.84 GB effective ceiling. The model files themselves are 36.85 GB; the higher runtime estimate includes the current oMLX process footprint, MLX runtime/allocator overhead, buffers, and KV/cache planning.
The 8-bit variant should be retested after raising the Apple GPU wired-memory cap and restarting the local serving process, for example:
sudo sysctl iogpu.wired_limit_mb=59392
Throughput numbers are not published yet. This section will be updated after a successful full benchmark run.
Usage
Download only the variant you want:
from pathlib import Path
from huggingface_hub import snapshot_download
repo_id = "chanderbalaji/Grug-35B-A3B-MLX"
variant = "mlx-4bit"
snapshot = snapshot_download(
repo_id,
allow_patterns=[f"{variant}/*"],
)
model_path = Path(snapshot) / variant
print(model_path)
Run with mlx-lm:
python -m mlx_lm.generate \
--model /path/to/downloaded/snapshot/mlx-4bit \
--prompt "Reply with a short explanation of what this model is." \
--max-tokens 256
Local compatibility note
The source config uses model_type: qwen3_5_moe_text. At conversion time,
mlx-lm 0.31.3 did not include a native loader for that exact model type, so
the conversion used a local compatibility shim mapping the model to the
existing Qwen3-Next style MLX implementation and adapting the published weight
names.
Consumers may need an mlx-lm build that supports qwen3_5_moe_text, or an
equivalent local compatibility shim, until upstream support is available.
Provenance and attribution
- Source model:
kai-os/Grug-35B-A3B - Relationship: MLX quantized derivatives of the source model
- Source revision used locally:
f6349f9c7beba10ad44ce0210b0a0f6fba414a05 - Conversion tool:
mlx-lm 0.31.3 - Quantization mode: affine, group size 64
Please refer to the source model card for upstream training details, intended use, limitations, acknowledgements, and license context.
Limitations
Quantization can change output quality, numerical behavior, and edge-case performance. These files are intended for local MLX inference on Apple Silicon. Use the source model repo for the original Transformers/safetensors weights.
4-bit