---
pipeline_tag: text-generation
library_name: mlx
license: other
base_model: kai-os/Grug-35B-A3B
base_model_relation: quantized
tags:
- mlx
- mlx-lm
- qwen3
- qwen3-next
- mixture-of-experts
- text-generation
- quantized
- 4-bit
- 6-bit
- 8-bit
- apple-silicon
---

# Grug-35B-A3B MLX

Apple Silicon MLX quantizations of
[`kai-os/Grug-35B-A3B`](https://huggingface.co/kai-os/Grug-35B-A3B), packaged
as a single Hugging Face repo with one folder per quantization level.

This repository contains MLX safetensors folders, not GGUF files. The source
model remains in the upstream Hugging Face repository.

## Available variants

| Variant | Folder | Quantization | Size | Best fit |
| --- | --- | --- | ---: | --- |
| MLX 8-bit | `mlx-8bit/` | affine, group size 64 | 36.85 GB | Highest-quality local MLX run. |
| MLX 6-bit | `mlx-6bit/` | affine, group size 64 | 28.19 GB | Balanced memory and quality. |
| MLX 4-bit | `mlx-4bit/` | affine, group size 64 | 19.53 GB | Smallest footprint and easiest local run. |

## Local benchmark notes

Initial local testing was performed on a Mac Studio with an M4 Max and 64 GB
unified memory using oMLX.

### 4-bit

Screenshot: [`benchmarks/grug-35b-a3b-4bit-omlx.png`](benchmarks/grug-35b-a3b-4bit-omlx.png)

| Test | TTFT | Prompt TPS | Generation TPS | End-to-end | Throughput | Peak memory |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| pp1024 / tg128 | 566.6 ms | 1807.2 tok/s | 103.9 tok/s | 1.805 s | 638.2 tok/s | 19.35 GB |
| pp4096 / tg128 | 2258.3 ms | 1813.7 tok/s | 100.2 tok/s | 3.545 s | 1191.5 tok/s | 20.08 GB |
| pp8192 / tg128 | 4690.0 ms | 1746.7 tok/s | 97.1 tok/s | 6.019 s | 1382.2 tok/s | 20.50 GB |
| pp16384 / tg128 | 12050.5 ms | 1359.6 tok/s | 90.3 tok/s | 13.478 s | 1225.1 tok/s | 20.38 GB |

Continuous batching, pp1024 / tg128:

| Batch | Generation TPS | Speedup | Prompt TPS | Prompt TPS/request | TTFT | End-to-end |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| 1x | 103.9 tok/s | 1.00x | 1807.2 tok/s | 1807.2 tok/s | 566.6 ms | 1.805 s |
| 2x | 116.1 tok/s | 1.12x | 1468.4 tok/s | 734.2 tok/s | 1394.6 ms | 3.600 s |
| 4x | 180.0 tok/s | 1.73x | 1549.7 tok/s | 387.4 tok/s | 2530.3 ms | 5.488 s |

### 6-bit

Screenshot: [`benchmarks/grug-35b-a3b-6bit-omlx.png`](benchmarks/grug-35b-a3b-6bit-omlx.png)

| Test | TTFT | Prompt TPS | Generation TPS | End-to-end | Throughput | Peak memory |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| pp1024 / tg128 | 645.4 ms | 1586.6 tok/s | 87.1 tok/s | 2.119 s | 543.5 tok/s | 45.58 GB |
| pp4096 / tg128 | 3175.8 ms | 1289.8 tok/s | 82.9 tok/s | 4.732 s | 892.6 tok/s | 45.16 GB |
| pp8192 / tg128 | 6701.5 ms | 1222.4 tok/s | 80.9 tok/s | 8.294 s | 1003.1 tok/s | 45.30 GB |
| pp16384 / tg128 | 22365.0 ms | 732.6 tok/s | 76.7 tok/s | 24.045 s | 686.7 tok/s | 45.68 GB |

The 6-bit batching run shown in
[`benchmarks/grug-35b-a3b-6bit-prefill-cap.png`](benchmarks/grug-35b-a3b-6bit-prefill-cap.png)
hit oMLX's prefill safety cap at the tested settings: predicted peak exceeded
45.5 GB, which was 90% of the effective 50.5 GB ceiling.

### 8-bit

The 8-bit variant did not load under the default local memory cap. oMLX
projected 54.62 GB total memory use against a 51.84 GB effective ceiling. The
model files themselves are 36.85 GB; the higher runtime estimate includes the
current oMLX process footprint, MLX runtime/allocator overhead, buffers, and
KV/cache planning.

The 8-bit variant should be retested after raising the Apple GPU wired-memory
cap and restarting the local serving process, for example:

```bash
sudo sysctl iogpu.wired_limit_mb=59392
```

Throughput numbers are not published yet. This section will be updated after a
successful full benchmark run.

## Usage

Download only the variant you want:

```python
from pathlib import Path
from huggingface_hub import snapshot_download

repo_id = "chanderbalaji/Grug-35B-A3B-MLX"
variant = "mlx-4bit"

snapshot = snapshot_download(
    repo_id,
    allow_patterns=[f"{variant}/*"],
)
model_path = Path(snapshot) / variant
print(model_path)
```

Run with `mlx-lm`:

```bash
python -m mlx_lm.generate \
  --model /path/to/downloaded/snapshot/mlx-4bit \
  --prompt "Reply with a short explanation of what this model is." \
  --max-tokens 256
```

## Local compatibility note

The source config uses `model_type: qwen3_5_moe_text`. At conversion time,
`mlx-lm 0.31.3` did not include a native loader for that exact model type, so
the conversion used a local compatibility shim mapping the model to the
existing Qwen3-Next style MLX implementation and adapting the published weight
names.

Consumers may need an `mlx-lm` build that supports `qwen3_5_moe_text`, or an
equivalent local compatibility shim, until upstream support is available.

## Provenance and attribution

- Source model: [`kai-os/Grug-35B-A3B`](https://huggingface.co/kai-os/Grug-35B-A3B)
- Relationship: MLX quantized derivatives of the source model
- Source revision used locally: `f6349f9c7beba10ad44ce0210b0a0f6fba414a05`
- Conversion tool: `mlx-lm 0.31.3`
- Quantization mode: affine, group size 64

Please refer to the source model card for upstream training details, intended
use, limitations, acknowledgements, and license context.

## Limitations

Quantization can change output quality, numerical behavior, and edge-case
performance. These files are intended for local MLX inference on Apple Silicon.
Use the source model repo for the original Transformers/safetensors weights.