---
license: apache-2.0
base_model:
  - google/gemma-4-E2B-it-qat-q4_0-unquantized-assistant
  - google/gemma-4-E4B-it-qat-q4_0-unquantized-assistant
  - google/gemma-4-12B-it-qat-q4_0-unquantized-assistant
  - google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant
  - google/gemma-4-31B-it-qat-q4_0-unquantized-assistant
tags:
  - gguf
  - gemma4
  - qat
  - q4_0
  - speculative-decoding
  - mtp
  - ik_llama
---

# Gemma 4 QAT Q4_0 MTP Assistants for ik_llama

This repo contains `ik_llama`-compatible GGUF assistant/draft models converted
from Google's Gemma 4 QAT Q4_0 assistant checkpoints.

These are not standalone chat models. Use them as `--model-draft` files next to
the matching Google Gemma 4 QAT Q4_0 target GGUF.

Do not load this repository as the primary model. Hugging Face may show generic
GGUF usage snippets for this repo, but these files are assistant/draft GGUFs
only.

## Important E2B/E4B Caveat

The E2B and E4B files are experimental edge-model assistants. They converted
and smoke-tested successfully with matched QAT targets after an `ik_llama`
shared-KV target loader fix.

Current `ik_llama` main includes that loader fix via PR #1927. Older builds may
fail before draft loading with errors such as:

```text
E2B: missing blk.15.attn_k.weight
E4B: missing blk.24.attn_k.weight
```

Those target tensors appear intentionally absent in Google's QAT edge GGUFs:
the E2B/E4B targets advertise shared-KV tail layers.

E4B has an additional runtime caveat. On current `ik_llama` main, local tests
could load E4B QAT + matched E4B assistant and generate in a small CPU/offload
smoke path. The longer full-GPU flash-attention path was still unstable before
acceptance counters in our RTX 4070 test. Treat E4B as experimental until the
remaining runtime/FA path is resolved upstream.

Use matched pairs only. The E2B assistant is not expected to work with the E4B
target, and the E4B assistant is not expected to work with the E2B target.

## Files

| Model | Q4_0 assistant |
| --- | --- |
| Gemma 4 E2B IT QAT Q4_0 | `gemma-4-E2B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf` |
| Gemma 4 E4B IT QAT Q4_0 | `gemma-4-E4B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf` |
| Gemma 4 12B IT QAT Q4_0 | `gemma-4-12B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf` |
| Gemma 4 26B-A4B IT QAT Q4_0 | `gemma-4-26B-A4B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf` |
| Gemma 4 31B IT QAT Q4_0 | `gemma-4-31B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf` |

BF16 conversion intermediates and Q8_0 reference variants are intentionally not
published here. The Q4_0 files are the intended pairing for Google's QAT Q4_0
target GGUFs.

## Matching Target Repos

Use the assistants with Google's official target GGUFs:

- `google/gemma-4-E2B-it-qat-q4_0-gguf`
- `google/gemma-4-E4B-it-qat-q4_0-gguf`
- `google/gemma-4-12B-it-qat-q4_0-gguf`
- `google/gemma-4-26B-A4B-it-qat-q4_0-gguf`
- `google/gemma-4-31B-it-qat-q4_0-gguf`

## Example

```bash
llama-server \
  -m /path/to/gemma-4-12b-it-qat-q4_0.gguf \
  --model-draft /path/to/gemma-4-12B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf \
  --spec-type mtp:n_max=4,p_min=0.0 \
  --jinja
```

Use a current `ik_llama` build with Gemma 4 MTP support. Reproducing these
conversions from Google's safetensors currently requires `convert_hf_to_gguf.py`
to recognize `Gemma4UnifiedAssistantForCausalLM` as the existing Gemma 4 MTP
assistant converter path.

## Validation

Local conversion metadata checks:

| Model | Tensors | Backbone | Centroid tensors |
| --- | ---: | ---: | --- |
| E2B | 50 | 1536 | yes |
| E4B | 50 | 2560 | yes |
| 12B | 48 | 3840 | no, metadata only |
| 26B-A4B | 48 | 2816 | no, metadata only |
| 31B | 48 | 5376 | no, metadata only |

All published files report `general.architecture=gemma4_mtp`.

Historical runtime smoke on an RTX 4070 with `ik_llama` build
`4561 (6b9de3dba)`:

| Target + Q4_0 draft | Status | Notes |
| --- | --- | --- |
| E2B | passed on shared-KV branch | MTP context ready; raw completion generated; 37/92 draft tokens accepted |
| E4B | passed on shared-KV branch | MTP context ready; raw completion generated; 33/116 draft tokens accepted |
| 12B | passed | MTP context ready; raw completion generated |
| 26B-A4B | passed | MTP context ready; raw completion generated with CPU/system-RAM offload |
| 31B | passed | MTP context ready; raw completion generated with CPU/system-RAM offload |

Additional current-main check with `ik_llama` `bbe1a511e`:

| Target + Q4_0 draft | Status | Notes |
| --- | --- | --- |
| E2B | passed | full-GPU flash-attention direct-server request completed |
| E4B | partial | small CPU/offload `-c 512` raw completion smoke passed; local full-GPU flash-attention path failed before acceptance counters |

## Conversion Notes

Source assistant repos:

- `google/gemma-4-E2B-it-qat-q4_0-unquantized-assistant`
- `google/gemma-4-E4B-it-qat-q4_0-unquantized-assistant`
- `google/gemma-4-12B-it-qat-q4_0-unquantized-assistant`
- `google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant`
- `google/gemma-4-31B-it-qat-q4_0-unquantized-assistant`

The assistants were converted through `ik_llama`'s Gemma 4 MTP assistant
converter, then quantized with `llama-quantize` to `Q4_0`.