README.md

---
pipeline_tag: text-generation
library_name: mlx
license: other
base_model: kai-os/Grug-35B-A3B
base_model_relation: quantized
tags:
- mlx
- mlx-lm
- qwen3
- qwen3-next
- mixture-of-experts
- text-generation
- quantized
- 4-bit
- 6-bit
- 8-bit
- apple-silicon
---

# Grug-35B-A3B MLX

Apple Silicon MLX quantizations of
[`kai-os/Grug-35B-A3B`](https://huggingface.co/kai-os/Grug-35B-A3B), packaged
as a single Hugging Face repo with one folder per quantization level.

This repository contains MLX safetensors folders, not GGUF files. The source
model remains in the upstream Hugging Face repository.

## Available variants

| Variant | Folder | Quantization | Size | Best fit |
| --- | --- | --- | ---: | --- |
| MLX 8-bit | `mlx-8bit/` | affine, group size 64 | 36.85 GB | Highest-quality local MLX run. |
| MLX 6-bit | `mlx-6bit/` | affine, group size 64 | 28.19 GB | Balanced memory and quality. |
| MLX 4-bit | `mlx-4bit/` | affine, group size 64 | 19.53 GB | Smallest footprint and easiest local run. |

## Local benchmark notes

Initial local testing was performed on a Mac Studio with an M4 Max and 64 GB
unified memory using oMLX.

| Variant | Result | Notes |
| --- | --- | --- |
| MLX 8-bit | Not loaded under default memory cap | oMLX projected 54.62 GB total memory use against a 51.84 GB effective ceiling. The model files themselves are 36.85 GB; the higher runtime estimate includes the current oMLX process footprint, MLX runtime/allocator overhead, buffers, and KV/cache planning. |

The 8-bit variant should be retested after raising the Apple GPU wired-memory
cap and restarting the local serving process, for example:

```bash
sudo sysctl iogpu.wired_limit_mb=59392
```

Throughput numbers are not published yet. This section will be updated after a
successful full benchmark run.

## Usage

Download only the variant you want:

```python
from pathlib import Path
from huggingface_hub import snapshot_download

repo_id = "chanderbalaji/Grug-35B-A3B-MLX"
variant = "mlx-4bit"

snapshot = snapshot_download(
    repo_id,
    allow_patterns=[f"{variant}/*"],
)
model_path = Path(snapshot) / variant
print(model_path)
```

Run with `mlx-lm`:

```bash
python -m mlx_lm.generate \
  --model /path/to/downloaded/snapshot/mlx-4bit \
  --prompt "Reply with a short explanation of what this model is." \
  --max-tokens 256
```

## Local compatibility note

The source config uses `model_type: qwen3_5_moe_text`. At conversion time,
`mlx-lm 0.31.3` did not include a native loader for that exact model type, so
the conversion used a local compatibility shim mapping the model to the
existing Qwen3-Next style MLX implementation and adapting the published weight
names.

Consumers may need an `mlx-lm` build that supports `qwen3_5_moe_text`, or an
equivalent local compatibility shim, until upstream support is available.

## Provenance and attribution

- Source model: [`kai-os/Grug-35B-A3B`](https://huggingface.co/kai-os/Grug-35B-A3B)
- Relationship: MLX quantized derivatives of the source model
- Source revision used locally: `f6349f9c7beba10ad44ce0210b0a0f6fba414a05`
- Conversion tool: `mlx-lm 0.31.3`
- Quantization mode: affine, group size 64

Please refer to the source model card for upstream training details, intended
use, limitations, acknowledgements, and license context.

## Limitations

Quantization can change output quality, numerical behavior, and edge-case
performance. These files are intended for local MLX inference on Apple Silicon.
Use the source model repo for the original Transformers/safetensors weights.