steampunque/gemma-4-12B-it-MP-GGUF

Mixed Precision GGUF layer quantization of gemma-4-12B-it by Google

Original model: https://huggingface.co/google/gemma-4-12B-it

The hybrid quant employs different quantization levels on a per layer basis to enable both high performance and small file size at the same time. The quants employed are all K to avoid slow CPU or older GPU processing of IQ quants. For this file the Q4_E_H layer quants using new extended K layer definitions are as follows:

  LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOD","0","QN_K","2","Q2_K","3","Q3_K","4","Q4_K","5","Q5_K","6","Q6_K","8","Q8_0","h","F16","f","F32"],
   [0 ,"Q5_K_666"],[1 ,"Q4_K_656"],[2 ,"Q4_K_545"],[3 ,"Q4_K_554"],[4 ,"Q4_K_554"],[5 ,"Q4_K_555"],[6 ,"Q4_K_544"],[7 ,"Q4_K_554"],
   [8 ,"Q4_K_544"],[9 ,"Q4_K_544"],[10,"Q4_K_544"],[11,"Q4_K_555"],[12,"Q4_K_544"],[13,"Q4_K_544"],[14,"Q4_K_544"],[15,"Q4_K_544"],
   [16,"Q4_K_544"],[17,"Q4_K_555"],[18,"Q4_K_544"],[19,"Q4_K_544"],[20,"Q4_K_544"],[21,"Q4_K_544"],[22,"Q4_K_544"],[23,"Q4_K_555"],
   [24,"Q4_K_644"],[25,"Q4_K_644"],[26,"Q4_K_644"],[27,"Q4_K_644"],[28,"Q4_K_644"],[29,"Q4_K_655"],[30,"Q4_K_644"],[31,"Q4_K_644"],
   [32,"Q4_K_654"],[33,"Q4_K_654"],[34,"Q4_K_654"],[35,"Q4_K_655"],[36,"Q4_K_654"],[37,"Q4_K_654"],[38,"Q4_K_654"],[39,"Q4_K_654"],
   [40,"Q4_K_665"],[41,"Q4_K_666"],[42,"Q4_K_665"],[43,"Q4_K_665"],[44,"Q5_K_665"],[45,"Q5_K_666"],[46,"Q5_K_668"],[47,"Q6_K_886"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

The layer quants were optimized for very strong performance over a small set of curated reasoning prompts and sized to slightly larger than Q4_K_M bit efficiency. It solves the majority of the set of eval prompts efficiently and correctly without using think block.

Comparison:

Quant	size	PPL	Comment
Q4_K_M	7.38e9	14.4	modified PPL, see discussion below.
Q4_E_H	7.43e9	13.6	modified PPL, 0.05B larger than Q4_K_M, Q6_K embed/out

Usage:

gemma 4 12B it is a vision capable dense RL model. It can be used together with its multimedia projector layers to process audio and images and text inputs and generate text outputs. The mmproj file is made available in this repository.

Thinking:

By default the model will not create a RL reasoning block and just outputs

<|channel>thought
<channel|>

at the start of gen. To get it to fill in the think block use a system prompt with:

<|think|>

as the first token. This is a special token in the model vocab and must be tokenized as such to work. No other text in the system prompt besides the think token is needed to get it to fill in the RL block though other text can be added if desired.

The model was found to be highly capable on reasoning tasks when skipping think block.

Running:

The model can be speculated with gemma-3-270m-it-256k. To use this speculator the inference platform must support dynamic vocab translation between draft and target. Google is creating optimized speculators for gemma 4 series which might work also but have not been tested.

The model can be run fully offloaded into a 12G VRAM GPU. Due to its SWA attention scheme very large context is available.

On a 9900k + 4070 setup approx performance for the Q4_E_H quant using fixed speculation block size ND with a downstream llama.cpp speculator:

CONFIG (no multimodal tower)	QKV	NKV	gen tps	pp tps (batch 128)
ND=0 (no draft)	F16	~204k	48	~1400
ND=2	""	~150k	69	""
ND=3	""	""	63	""
ND=0 (no draft)	Q8_0	256k	47	~1100
ND=2		~200k	64	""

The Q4_E_H model passed two long context tests showing very fast prompt processing speed for 100k+ token prompts. This fast speed is due to combination of SWA + decimated full attention layers (only 1 of 6 layers is full attention). The attention scheme seems to work well. The quant handles a 106k token prompt https://thireus.com/REDDIT/Qwen3_Runescape_Massive_Prompt.txt (greedy sampling used for gen) with a self correction:

lm Qwen3_Runescape_Massive_Prompt.txt 
<|channel>thought
<channel|>Based on the "Skills/Experience Table" provided in the text, you reach the halfway point of the experience required for the maximum level (level 99) at **level 70**.

**Calculation:**
*   The experience required to reach the maximum level (99) is **13,034,431**.
*   Half of that amount is **6,517,215.5**.
*   According to the table, the experience required to reach **level 70** is **737,627**, and the experience required to reach **level 71** is **814,445**. 

Wait, looking closer at the table provided, the "Experience needed to reach level" column represents the total cumulative experience for that level. To find the level where you have half of the 13,034,431 required for level 99:
*   Level 70: 737,627
*   Level 80: 1,986,068
*   Level 90: 5,346,332
*   Level 91: 5,902,831
*   Level 92: 6,517,253
*   Level 93: 7,195,629

The value **6,517,253** (which is the closest whole number to the 6,517,215.5 midpoint) is the experience required to reach **level 92**.

This prompt and answer are proven baked into gemma 4 training set by prior experiments with gemma 26/31. What is notable on this 12B response is the self correction showing the RL/introspection training kicking in to direct the model through differnt parts of its latent space to increase the accuracy of generating the (pretrained) correct answer.

Vision:

As of llama.cpp b9518 vision mode will not work without applying the patch https://github.com/ggml-org/llama.cpp/compare/master...chippydip:llama.cpp:gemma4uv-vision-fix.diff from issue https://github.com/ggml-org/llama.cpp/issues/24146 . With the patch vision works but still exhibits the same poor performance found in 26B/31B, failing to identify tough bird ID images which entire Qwen3.5 series aces. The root cause of the poor performance can now more confidently be inferred to be low quality vision training since this model using a completely different mechanism to handle multimodal compared to 26B/31B and still shows similar poor performance.

Audio:

A quick test on audio transcription showed the model to be fairly accurate on Oppenheimer bhagavad gita:

The new world would not be the same. A few people laughed, a few people cried, most people were silent. I remembered the line from the Hindu scripture, the Bhagavad Gita. Vishnu is trying to persuade the prince that he should do his duty and to impress him takes on his multi-armed form and says now i am become death the destroyer of worlds. I suppose we all thought that one way or another.

Code:

The model was tested across a small set of code gen promtps and did OK, nothing spectacular but was able to generate working programs from short functional descriptions.

Llama.cpp inference/isssues:

multimodal vision functionality is borked as of llama.cpp b9518 but vision can be patched as described above.

The model cannot compute valid perplexity due to the instruct tune forcing it to generate

<|channel>thought

as assitant gen independent of previous prompt contents. To work around this problem a modifed perplexity is computed by overwriting the beginning of the perplexity chunk contents with the forced assistent gen as follows:

      # chunk is a string of text to eval perplexity on
      injects='model\n<|channel>thought\n<channel|>'
      chunk="${injects}${chunk:${#injects}}"

logprobs are skipped over the beginning part of the perplexity prompt using a modified llama.cpp downstream server to compute perplexity. Discussion at: https://github.com/ggml-org/llama.cpp/issues/21388#issuecomment-4185889930

Benchmarks:

Benchmarks for the model (audio, vision, and math) are given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

Link	Type	Size/e9 B	Notes
gemma-4-12B-it.Q4_E_H.gguf	Q4_E_H	7.43e9 B	0.05B bigger than Q4_K_M
gemma-4-12B-it.mmproj.gguf	F16	0.12e9 B	multimedia projector

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040