File size: 3,549 Bytes
7f3964d 3506f28 7f3964d 3506f28 7f3964d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 | ---
pipeline_tag: text-generation
library_name: mlx
license: other
base_model: kai-os/Grug-35B-A3B
base_model_relation: quantized
tags:
- mlx
- mlx-lm
- qwen3
- qwen3-next
- mixture-of-experts
- text-generation
- quantized
- 4-bit
- 6-bit
- 8-bit
- apple-silicon
---
# Grug-35B-A3B MLX
Apple Silicon MLX quantizations of
[`kai-os/Grug-35B-A3B`](https://huggingface.co/kai-os/Grug-35B-A3B), packaged
as a single Hugging Face repo with one folder per quantization level.
This repository contains MLX safetensors folders, not GGUF files. The source
model remains in the upstream Hugging Face repository.
## Available variants
| Variant | Folder | Quantization | Size | Best fit |
| --- | --- | --- | ---: | --- |
| MLX 8-bit | `mlx-8bit/` | affine, group size 64 | 36.85 GB | Highest-quality local MLX run. |
| MLX 6-bit | `mlx-6bit/` | affine, group size 64 | 28.19 GB | Balanced memory and quality. |
| MLX 4-bit | `mlx-4bit/` | affine, group size 64 | 19.53 GB | Smallest footprint and easiest local run. |
## Local benchmark notes
Initial local testing was performed on a Mac Studio with an M4 Max and 64 GB
unified memory using oMLX.
| Variant | Result | Notes |
| --- | --- | --- |
| MLX 8-bit | Not loaded under default memory cap | oMLX projected 54.62 GB total memory use against a 51.84 GB effective ceiling. The model files themselves are 36.85 GB; the higher runtime estimate includes the current oMLX process footprint, MLX runtime/allocator overhead, buffers, and KV/cache planning. |
The 8-bit variant should be retested after raising the Apple GPU wired-memory
cap and restarting the local serving process, for example:
```bash
sudo sysctl iogpu.wired_limit_mb=59392
```
Throughput numbers are not published yet. This section will be updated after a
successful full benchmark run.
## Usage
Download only the variant you want:
```python
from pathlib import Path
from huggingface_hub import snapshot_download
repo_id = "chanderbalaji/Grug-35B-A3B-MLX"
variant = "mlx-4bit"
snapshot = snapshot_download(
repo_id,
allow_patterns=[f"{variant}/*"],
)
model_path = Path(snapshot) / variant
print(model_path)
```
Run with `mlx-lm`:
```bash
python -m mlx_lm.generate \
--model /path/to/downloaded/snapshot/mlx-4bit \
--prompt "Reply with a short explanation of what this model is." \
--max-tokens 256
```
## Local compatibility note
The source config uses `model_type: qwen3_5_moe_text`. At conversion time,
`mlx-lm 0.31.3` did not include a native loader for that exact model type, so
the conversion used a local compatibility shim mapping the model to the
existing Qwen3-Next style MLX implementation and adapting the published weight
names.
Consumers may need an `mlx-lm` build that supports `qwen3_5_moe_text`, or an
equivalent local compatibility shim, until upstream support is available.
## Provenance and attribution
- Source model: [`kai-os/Grug-35B-A3B`](https://huggingface.co/kai-os/Grug-35B-A3B)
- Relationship: MLX quantized derivatives of the source model
- Source revision used locally: `f6349f9c7beba10ad44ce0210b0a0f6fba414a05`
- Conversion tool: `mlx-lm 0.31.3`
- Quantization mode: affine, group size 64
Please refer to the source model card for upstream training details, intended
use, limitations, acknowledgements, and license context.
## Limitations
Quantization can change output quality, numerical behavior, and edge-case
performance. These files are intended for local MLX inference on Apple Silicon.
Use the source model repo for the original Transformers/safetensors weights.
|