matrixportalx/gemma-4-E4B-it-assistant-MTP-GGUF

Gemma 4 E4B — MTP (assistant) head · GGUF

GGUF conversions of google/gemma-4-E4B-it-assistant, the Gemma 4 Multi-Token-Prediction (MTP) assistant head — architecture gemma4-assistant in llama.cpp.

These are not a standalone chat model. This is the small speculative / draft-mtp head (~76M parameters) that runs alongside a Gemma 4 E4B main model to accelerate generation via speculative decoding. Load it as the draft model, not as the primary model.

Files

File	Type
gemma-4-E4B-it-assistant-f16.gguf	F16
gemma-4-E4B-it-assistant-Q4_0.gguf	Q4_0
gemma-4-E4B-it-assistant-Q6_K.gguf	Q6_K
gemma-4-E4B-it-assistant-Q8_0.gguf	Q8_0

All variants are converted from the original bf16 checkpoint with llama.cpp (commit f955e39).

Choosing a quantization

Most of this head's weights live in the token-embedding tensor, so quantization mainly shrinks the embedding:

• F16 — highest draft acceptance rate; recommended when disk isn't a concern.
• Q8_0 — near-F16 quality, ~half the size.
• Q6_K — smaller still, minor acceptance loss.
• Q4_0 — smallest; acceptance drops the most (the embedding is quantized to 4-bit).

Because the head is tiny to begin with, the size savings from lower bit-widths are modest while acceptance decreases — F16 or Q8_0 are the practical picks.

Usage (llama.cpp speculative decoding)

Use this file as the draft/MTP head together with a Gemma 4 E4B main model in a llama.cpp build that supports gemma4 MTP (draft-mtp) speculative decoding. The head shares the KV cache with the target model. Typical speculative-decoding settings: n_max (draft length) of 3 tends to give the best throughput; higher values reduce acceptance and can end up slower than no speculation.

Base model & license

Derived from the gated model google/gemma-4-E4B-it-assistant. Distributed under the Gemma license; you are responsible for complying with Google's Gemma Terms of Use.