mradermacher

/

gemma-4-26B-A4B-it-abliterix-i1-GGUF

like 5

Every other uncensored model of Gemma 4-26B-A4B I've tried so far has a problem where it will sometimes output a single token and then hang. I end up needing to stop the hung response, and edit my prompt slightly, and sometimes even then it continues to hang. The only surefire workaround is to attach an image (I created a little black PNG just for the purpose) which seems to dislodge the log jam. I could not find anyone else talking about this, so I have been left to my own devices to figure out the nature of the problem.

Then earlier tonight, I was trying to get Gemma 4-31B to perform to a reasonable speed. I completely failed, but along the way that model said something about MoE models sometimes activating the wrong neurons and outputting a single token and then seizing up—exactly the behavior I've been getting from every single uncensored 26B model. I abandoned the attempt to make 31B performant enough to be tolerable, and decided to make one more check of Hugging Face to see if just maybe someone else was talking about my problem. That's when I found this model, last updated ONE MINUTE (!) prior to my check. I looked at the model card describing how this one is different and decided to give it a shot.

So far, it's actually working! I've had a couple of glitches in the output that I can attribute to using the IQ3_M quantization, but it hasn't hung on me even once! This one just might have the magic sauce. And the model card explaining how the abliteration process was done differently is a very enlightening read, I highly recommend giving it a look.

I also grabbed the i1-Q5_K_M quantization to hopefully avoid those occasional moments of derp from IQ3_M, figuring I'll have a "fast model" and a "smart model" and switch between them as necessary. The speed penalty of the larger version is only about 25% too, running on my potato of an office PC with an RTX 3060 (12 GB).

These are the parameters I'm using with IQ3_M:

[Path]\llama-server.exe ^
--model "[Path]\gemma-4-26B-A4B-it-abliterix.i1-IQ3_M.gguf" ^
--mmproj "[Path]\gemma-4-26B-A4B-it.mmproj-Q8_0.gguf" ^
--image-max-tokens 1120 ^
--chat-template-file "[Path]\chat_template.jinja" ^
--ctx-size 131072 ^
--n-cpu-moe 9 ^
--cache-type-k iq4_nl ^
--cache-type-v iq4_nl ^
--reasoning-budget 2048 ^
--reasoning-budget-message "Reasoning budget exceeded (looping?)" ^
--log-timestamps

And with i1-Q5_K_M:

[Path]\llama-server.exe ^
--model "[Path]\gemma-4-26B-A4B-it-abliterix.i1-Q5_K_M.gguf" ^
--mmproj "[Path]\gemma-4-26B-A4B-it.mmproj-Q8_0.gguf" ^
--image-max-tokens 1120 ^
--chat-template-file "[Path]\chat_template.jinja" ^
--ctx-size 131072 ^
--n-cpu-moe 18 ^
--cache-type-k q5_1 ^
--cache-type-v q5_1 ^
--reasoning-budget 2048 ^
--reasoning-budget-message "Reasoning budget exceeded (looping?)" ^
--log-timestamps

You may be able to figure out that I'm running Windows, which uses ^ as the "don't break the line" character. Obviously, substitute whatever your OS uses if it's different.

EDIT: Apparently the model's structure has changed sufficiently that the .mmproj file for the "normal" Gemma 4 26B-A4B no longer works. The model now crashes when presented with vision tasks. No replacement .mmproj file has been provided. So much for my attempt to Agentify this model!