README.md

---
license: gemma
language:
  - en
base_model: google/gemma-4-26b-a4b-it
tags:
  - mlx
  - mlx-node
  - quantized
  - awq
  - gemma4
  - moe
  - sliding-window-attention
  - vision-language
  - apple-silicon
  - unsloth-dynamic
library_name: mlx-node
quantized_by: mlx-node
pipeline_tag: text-generation
model_type: gemma4
---

# Gemma-4-26B-A4B-IT — UD-Q5_K_XL (mlx-node)

5-bit affine quantization of [google/gemma-4-26b-a4b-it](https://huggingface.co/google/gemma-4-26b-a4b-it) for Apple Silicon, using the [**Unsloth Dynamic** quantization strategy](https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks) via [mlx-node](https://github.com/mlx-node/mlx-node).

| | Original (BF16) | UD-Q5_K_XL (this model) |
|---|---|---|
| **Size** | ~49 GB | **20 GB** |
| **Format** | SafeTensors | SafeTensors |
| **Precision** | BF16 uniform | 5-bit affine + mixed bits + BF16 |
| **FFN group size** | — | **64** |
| **Biases** | — | yes |

## All Variants

| Repo | Bit budget | Size | Decode (tok/s) |
|---|---|---|---|
| [Brooooooklyn/Gemma-4-26B-A4B-IT-UD-Q3_K_XL-mlx](https://huggingface.co/Brooooooklyn/Gemma-4-26B-A4B-IT-UD-Q3_K_XL-mlx) | 3-bit base | 14 GB | 60.6 |
| [Brooooooklyn/Gemma-4-26B-A4B-IT-UD-MXFP4_K_XL-mlx](https://huggingface.co/Brooooooklyn/Gemma-4-26B-A4B-IT-UD-MXFP4_K_XL-mlx) | mxfp4 | 16 GB | 58.4 |
| [Brooooooklyn/Gemma-4-26B-A4B-IT-UD-Q4_K_XL-mlx](https://huggingface.co/Brooooooklyn/Gemma-4-26B-A4B-IT-UD-Q4_K_XL-mlx) | 4-bit base | 17 GB | 58.6 |
| [Brooooooklyn/Gemma-4-26B-A4B-IT-UD-NVFP4_K_XL-mlx](https://huggingface.co/Brooooooklyn/Gemma-4-26B-A4B-IT-UD-NVFP4_K_XL-mlx) | nvfp4 | 17 GB | 57.9 |
| **[Brooooooklyn/Gemma-4-26B-A4B-IT-UD-Q5_K_XL-mlx](https://huggingface.co/Brooooooklyn/Gemma-4-26B-A4B-IT-UD-Q5_K_XL-mlx) (this model)** | **5-bit base** | **20 GB** | **50.3** |
| [Brooooooklyn/Gemma-4-26B-A4B-IT-UD-Q6_K_XL-mlx](https://huggingface.co/Brooooooklyn/Gemma-4-26B-A4B-IT-UD-Q6_K_XL-mlx) | 6-bit base | 23 GB | 51.9 |
| [Brooooooklyn/Gemma-4-26B-A4B-IT-UD-MXFP8_K_XL-mlx](https://huggingface.co/Brooooooklyn/Gemma-4-26B-A4B-IT-UD-MXFP8_K_XL-mlx) | mxfp8 | 26 GB | 49.8 |
| [Brooooooklyn/Gemma-4-26B-A4B-IT-UD-Q8_K_XL-mlx](https://huggingface.co/Brooooooklyn/Gemma-4-26B-A4B-IT-UD-Q8_K_XL-mlx) | 8-bit base | 27 GB | 49.8 |

Benchmarked on Apple M3 Max 128GB via [`examples/lm.ts`](https://github.com/mlx-node/mlx-node/blob/main/examples/lm.ts) (best decode tok/s across turns 2–4, steady-state, capitals chat with `reasoningEffort: 'low'`).

**Note:** No Q2 variant is published — Gemma-4-26B-A4B-IT has only ~4B active parameters per token, which is below the architectural redundancy needed for 2-bit quantization to remain coherent. Both `unsloth` and `mixed_2_6` recipes produced gibberish at Q2 on this model.

## Performance

Steady-state decode: **50.3 tok/s** on Apple M3 Max 128GB (best of turns 2–4, `examples/lm.ts` capitals chat with `reasoningEffort: 'low'`). Decode is memory-bandwidth bound on Apple Silicon — fewer bytes per token directly translates to higher throughput. The MoE architecture activates only top-K of 128 experts per token (~4B active out of ~26B total), and the compiled C++ forward graph fuses the per-layer dispatch.

## Per-Tensor Bit Assignments (N=5)

| Weight | Mode | Bits | Group | Rationale |
|---|---|---|---|---|
| `embed_tokens` | 8-bit affine | 8 | 64 | Tied with lm_head (Gemma4 shares weights); affine-only loader |
| `self_attn.q_proj` | 8-bit affine | 8 | 64 | AWQ-corrected via input_layernorm |
| `self_attn.k_proj` | 8-bit affine | 8 | 64 | AWQ-corrected via input_layernorm |
| `self_attn.v_proj` | 8-bit affine | 8 | 64 | AWQ-corrected via input_layernorm (only on full-attention layers) |
| `mlp.gate_proj` | 5-bit affine | 5 | 64 | Shared dense MLP (top-level default) |
| `mlp.up_proj` | 5-bit affine | 5 | 64 | Shared dense MLP (top-level default) |
| `mlp.down_proj` | 6-bit affine | 6 | 64 | Shared dense MLP; "slightly more sensitive" (unsloth `base+1`) |
| `experts.switch_glu.gate_proj` | 5-bit affine | 5 | 64 | MoE expert gate (per-expert across all 128); base bits (top-level default) |
| `experts.switch_glu.up_proj` | 5-bit affine | 5 | 64 | MoE expert up (per-expert across all 128); base bits (top-level default) |
| `experts.switch_glu.down_proj` | 6-bit affine | 6 | 64 | MoE expert down (per-expert across all 128 + routing); unsloth `base+1` |
| `router.proj` | 8-bit affine | 8 | 64 | MoE routing — low-bit noise breaks top-K expert selection |
| `self_attn.o_proj` | **bf16** | — | — | NOT AWQ-correctable; kept full-precision |

## Quantization Strategy

Built on Unsloth Dynamic 2.0 per-tensor KLD analysis. At `--q-bits 5` the unsloth recipe assigns the base bits to MLP gate/up projections (the bulk of the parameter budget), `base+1` to down_proj (slightly more sensitive), `base+2` (snapped to a valid bit width) + AWQ pre-scaling to attention q/k/v projections, `base+2` to `embed_tokens`, `base+3` (capped/snapped) to the routing-critical paths, and keeps `self_attn.o_proj` as bf16 (AWQ-uncorrectable — its inputs come from the attention compute, not from a norm layer). The MoE router (`router.proj`) is forced to 8-bit affine to preserve top-K expert selection accuracy.

imatrix AWQ pre-scaling amplifies important weight channels and fuses inverse scales into preceding layer norms (zero inference overhead).

## Architecture

| Parameter | Value |
|---|---|
| Total parameters | ~26B (~4B active per token) |
| Hidden size | 2,816 |
| Layers | 30 (sliding-window attention) |
| Attention heads | 16 (8 KV heads, GQA 2:1) |
| Head dimension | 256 |
| Experts | 128 per MoE layer |
| MoE intermediate size | 704 |
| Vocab size | 262,144 |
| Max context | 262,144 tokens |
| Vision | yes (Gemma4ForConditionalGeneration) |

## Usage

```typescript
import { loadSession } from '@mlx-node/lm';

const session = await loadSession('./Gemma-4-26B-A4B-IT-UD-Q5_K_XL-mlx');

for await (const event of session.sendStream('Explain the MoE architecture in Gemma-4.', {
  config: { maxNewTokens: 2048, temperature: 0.6, reasoningEffort: 'low' },
})) {
  if (!event.done) process.stdout.write(event.text);
}
```

## How It Was Made

```bash
mlx convert \
  -i gemma-4-26b-a4b-it \
  -o Gemma-4-26B-A4B-IT-UD-Q5_K_XL-mlx \
  -q --q-bits 5 --q-recipe unsloth \
  --imatrix-path imatrix_unsloth.gguf
```

## Acknowledgments

- **[Unsloth](https://unsloth.ai)** — Quantization strategy based on their [per-layer KLD benchmarks](https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks) and Dynamic 2.0 methodology
- **[Google DeepMind](https://deepmind.google/)** — For the Gemma-4 model family
- **[Apple MLX](https://github.com/ml-explore/mlx)** — For the Metal-accelerated ML framework

## License

[Gemma Terms of Use](https://ai.google.dev/gemma/terms) (inherited from base model).