---
license: apache-2.0
base_model:
  - google/gemma-4-12B-it-qat-q4_0-unquantized-assistant
  - google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant
  - google/gemma-4-31B-it-qat-q4_0-unquantized-assistant
tags:
  - gguf
  - gemma4
  - qat
  - q4_0
  - speculative-decoding
  - mtp
  - ik_llama
---

# Gemma 4 QAT Q4_0 MTP Assistants for ik_llama

This repo contains `ik_llama`-compatible GGUF assistant/draft models converted
from Google's Gemma 4 QAT Q4_0 assistant checkpoints.

These are not standalone chat models. Use them as `--model-draft` files next to
the matching Google Gemma 4 QAT Q4_0 target GGUF.

## Files

| Model | Q4_0 assistant | Q8_0 assistant |
| --- | --- | --- |
| Gemma 4 12B IT QAT Q4_0 | `gemma-4-12B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf` | `gemma-4-12B-it-qat-q4_0-MTP-ik_llama-Q8_0.gguf` |
| Gemma 4 26B-A4B IT QAT Q4_0 | `gemma-4-26B-A4B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf` | `gemma-4-26B-A4B-it-qat-q4_0-MTP-ik_llama-Q8_0.gguf` |
| Gemma 4 31B IT QAT Q4_0 | `gemma-4-31B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf` | `gemma-4-31B-it-qat-q4_0-MTP-ik_llama-Q8_0.gguf` |

BF16 conversion intermediates are not published here.

## Matching Target Repos

Use the assistants with Google's official target GGUFs:

- `google/gemma-4-12B-it-qat-q4_0-gguf`
- `google/gemma-4-26B-A4B-it-qat-q4_0-gguf`
- `google/gemma-4-31B-it-qat-q4_0-gguf`

## Example

```bash
llama-server \
  -m /path/to/gemma-4-12b-it-qat-q4_0.gguf \
  --model-draft /path/to/gemma-4-12B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf \
  --spec-type mtp:n_max=4,p_min=0.0 \
  --jinja
```

Use a current `ik_llama` build with Gemma 4 MTP support. Reproducing these
conversions from Google's safetensors currently requires `convert_hf_to_gguf.py`
to recognize `Gemma4UnifiedAssistantForCausalLM` as the existing Gemma 4 MTP
assistant converter path.

## Validation

Local conversion metadata checks:

| Model | Tensors | Backbone | Centroid tensors |
| --- | ---: | ---: | --- |
| 12B | 48 | 3840 | no, metadata only |
| 26B-A4B | 48 | 2816 | no, metadata only |
| 31B | 48 | 5376 | no, metadata only |

All published files report `general.architecture=gemma4_mtp`.

Runtime smoke on an RTX 4070 with `ik_llama` build `4561 (6b9de3dba)`:

| Target + Q4_0 draft | Status | Notes |
| --- | --- | --- |
| 12B | passed | MTP context ready; raw completion generated |
| 26B-A4B | passed | MTP context ready; raw completion generated with CPU/system-RAM offload |
| 31B | passed | MTP context ready; raw completion generated with CPU/system-RAM offload |

## Conversion Notes

Source assistant repos:

- `google/gemma-4-12B-it-qat-q4_0-unquantized-assistant`
- `google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant`
- `google/gemma-4-31B-it-qat-q4_0-unquantized-assistant`

The assistants were converted through `ik_llama`'s Gemma 4 MTP assistant
converter, then quantized with `llama-quantize` to `Q4_0` and `Q8_0`.

The `Q4_0` files are the intended pairing for the QAT Q4_0 targets. The `Q8_0`
files are provided as higher-fidelity draft/reference variants.