---
license: apache-2.0
base_model:
  - google/gemma-4-E2B-it-qat-q4_0-unquantized-assistant
  - google/gemma-4-E4B-it-qat-q4_0-unquantized-assistant
  - google/gemma-4-12B-it-qat-q4_0-unquantized-assistant
  - google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant
  - google/gemma-4-31B-it-qat-q4_0-unquantized-assistant
tags:
  - gguf
  - gemma4
  - qat
  - q4_0
  - speculative-decoding
  - mtp
  - ik_llama
---

# Gemma 4 QAT Q4_0 MTP Assistants for ik_llama

This repo contains `ik_llama`-compatible GGUF assistant/draft models converted
from Google's Gemma 4 QAT Q4_0 assistant checkpoints.

These are not standalone chat models. Use them as `--model-draft` files next to
the matching Google Gemma 4 QAT Q4_0 target GGUF.

## Important E2B/E4B Caveat

The E2B and E4B files are experimental edge-model assistants. They converted
and smoke-tested successfully with matched QAT targets only after an `ik_llama`
shared-KV target loader fix.

Current `ik_llama` builds without that fix may fail before draft loading with
errors such as:

```text
E2B: missing blk.15.attn_k.weight
E4B: missing blk.24.attn_k.weight
```

Those target tensors appear intentionally absent in Google's QAT edge GGUFs:
the E2B/E4B targets advertise shared-KV tail layers. A test branch with the
loader fix is here:

```text
https://github.com/joelfarthing/ik_llama.cpp/tree/filament/gemma4-edge-shared-kv-20260606
```

Use matched pairs only. The E2B assistant is not expected to work with the E4B
target, and the E4B assistant is not expected to work with the E2B target.

## Files

| Model | Q4_0 assistant | Q8_0 assistant |
| --- | --- | --- |
| Gemma 4 E2B IT QAT Q4_0 | `gemma-4-E2B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf` | `gemma-4-E2B-it-qat-q4_0-MTP-ik_llama-Q8_0.gguf` |
| Gemma 4 E4B IT QAT Q4_0 | `gemma-4-E4B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf` | `gemma-4-E4B-it-qat-q4_0-MTP-ik_llama-Q8_0.gguf` |
| Gemma 4 12B IT QAT Q4_0 | `gemma-4-12B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf` | `gemma-4-12B-it-qat-q4_0-MTP-ik_llama-Q8_0.gguf` |
| Gemma 4 26B-A4B IT QAT Q4_0 | `gemma-4-26B-A4B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf` | `gemma-4-26B-A4B-it-qat-q4_0-MTP-ik_llama-Q8_0.gguf` |
| Gemma 4 31B IT QAT Q4_0 | `gemma-4-31B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf` | `gemma-4-31B-it-qat-q4_0-MTP-ik_llama-Q8_0.gguf` |

BF16 conversion intermediates are not published here.

## Matching Target Repos

Use the assistants with Google's official target GGUFs:

- `google/gemma-4-E2B-it-qat-q4_0-gguf`
- `google/gemma-4-E4B-it-qat-q4_0-gguf`
- `google/gemma-4-12B-it-qat-q4_0-gguf`
- `google/gemma-4-26B-A4B-it-qat-q4_0-gguf`
- `google/gemma-4-31B-it-qat-q4_0-gguf`

## Example

```bash
llama-server \
  -m /path/to/gemma-4-12b-it-qat-q4_0.gguf \
  --model-draft /path/to/gemma-4-12B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf \
  --spec-type mtp:n_max=4,p_min=0.0 \
  --jinja
```

Use a current `ik_llama` build with Gemma 4 MTP support. Reproducing these
conversions from Google's safetensors currently requires `convert_hf_to_gguf.py`
to recognize `Gemma4UnifiedAssistantForCausalLM` as the existing Gemma 4 MTP
assistant converter path.

## Validation

Local conversion metadata checks:

| Model | Tensors | Backbone | Centroid tensors |
| --- | ---: | ---: | --- |
| E2B | 50 | 1536 | yes |
| E4B | 50 | 2560 | yes |
| 12B | 48 | 3840 | no, metadata only |
| 26B-A4B | 48 | 2816 | no, metadata only |
| 31B | 48 | 5376 | no, metadata only |

All published files report `general.architecture=gemma4_mtp`.

Runtime smoke on an RTX 4070 with `ik_llama` build `4561 (6b9de3dba)`:

| Target + Q4_0 draft | Status | Notes |
| --- | --- | --- |
| E2B | passed on shared-KV branch | MTP context ready; raw completion generated; 37/92 draft tokens accepted |
| E4B | passed on shared-KV branch | MTP context ready; raw completion generated; 33/116 draft tokens accepted |
| 12B | passed | MTP context ready; raw completion generated |
| 26B-A4B | passed | MTP context ready; raw completion generated with CPU/system-RAM offload |
| 31B | passed | MTP context ready; raw completion generated with CPU/system-RAM offload |

## Conversion Notes

Source assistant repos:

- `google/gemma-4-E2B-it-qat-q4_0-unquantized-assistant`
- `google/gemma-4-E4B-it-qat-q4_0-unquantized-assistant`
- `google/gemma-4-12B-it-qat-q4_0-unquantized-assistant`
- `google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant`
- `google/gemma-4-31B-it-qat-q4_0-unquantized-assistant`

The assistants were converted through `ik_llama`'s Gemma 4 MTP assistant
converter, then quantized with `llama-quantize` to `Q4_0` and `Q8_0`.

The `Q4_0` files are the intended pairing for the QAT Q4_0 targets. The `Q8_0`
files are provided as higher-fidelity draft/reference variants.