---
pipeline_tag: text-generation
library_name: mlx
license: other
base_model: kai-os/Grug-35B-A3B
base_model_relation: quantized
tags:
- mlx
- mlx-lm
- qwen3
- qwen3-next
- mixture-of-experts
- text-generation
- quantized
- 4-bit
- 6-bit
- 8-bit
- apple-silicon
---

# Grug-35B-A3B MLX

Apple Silicon MLX quantizations of
[`kai-os/Grug-35B-A3B`](https://huggingface.co/kai-os/Grug-35B-A3B), packaged
as a single Hugging Face repo with one folder per quantization level.

This repository contains MLX safetensors folders, not GGUF files. The source
model remains in the upstream Hugging Face repository.

## Available variants

| Variant | Folder | Quantization | Size | Best fit |
| --- | --- | --- | ---: | --- |
| MLX 8-bit | `mlx-8bit/` | affine, group size 64 | 34 GB | Highest-quality local MLX run. |
| MLX 6-bit | `mlx-6bit/` | affine, group size 64 | 26 GB | Balanced memory and quality. |
| MLX 4-bit | `mlx-4bit/` | affine, group size 64 | 18 GB | Smallest footprint and easiest local run. |

Benchmarks will be added later.

## Usage

Download only the variant you want:

```python
from pathlib import Path
from huggingface_hub import snapshot_download

repo_id = "chanderbalaji/Grug-35B-A3B-MLX"
variant = "mlx-4bit"

snapshot = snapshot_download(
    repo_id,
    allow_patterns=[f"{variant}/*"],
)
model_path = Path(snapshot) / variant
print(model_path)
```

Run with `mlx-lm`:

```bash
python -m mlx_lm.generate \
  --model /path/to/downloaded/snapshot/mlx-4bit \
  --prompt "Reply with a short explanation of what this model is." \
  --max-tokens 256
```

## Local compatibility note

The source config uses `model_type: qwen3_5_moe_text`. At conversion time,
`mlx-lm 0.31.3` did not include a native loader for that exact model type, so
the conversion used a local compatibility shim mapping the model to the
existing Qwen3-Next style MLX implementation and adapting the published weight
names.

Consumers may need an `mlx-lm` build that supports `qwen3_5_moe_text`, or an
equivalent local compatibility shim, until upstream support is available.

## Provenance and attribution

- Source model: [`kai-os/Grug-35B-A3B`](https://huggingface.co/kai-os/Grug-35B-A3B)
- Relationship: MLX quantized derivatives of the source model
- Source revision used locally: `f6349f9c7beba10ad44ce0210b0a0f6fba414a05`
- Conversion tool: `mlx-lm 0.31.3`
- Quantization mode: affine, group size 64

Please refer to the source model card for upstream training details, intended
use, limitations, acknowledgements, and license context.

## Limitations

Quantization can change output quality, numerical behavior, and edge-case
performance. These files are intended for local MLX inference on Apple Silicon.
Use the source model repo for the original Transformers/safetensors weights.