google

/

gemma-4-26B-A4B-it

like 1.37k

Follow

Google 64k

Summary

Improves Gemma4 chat template:

Bug fixes

• None values now render as null instead of Python's None
• String-typed tool_calls[].function.arguments now raises a clear error instead of silently producing malformed DSL
• Prior-turn reasoning/thinking is preserved across multi-turn tool-call chains (preserve_thinking flag, default=true)
• Consecutive assistant messages now produce balanced <|turn>model/<turn|> tags via forward-scan continuation detection

Improvements

• enable_thinking normalized once with | default(false), eliminating repetitive is defined and checks
• image_url and input_audio content types now map to <|image|> and <|audio|> (OpenAI compatibility)
• Empty messages=[] handled gracefully instead of crashing
• Unmatched tool_call_id in tool responses falls back to 'unknown' instead of crashing
• Consistent .get() access prevents StrictUndefined errors for optional message keys
• O(1) backward scan for model-turn continuation (was O(n) per message)

I'm not particularly sure if it helps with Gemma 4 26B looping that happens when it's trying to list things (I'm no engineer), but if it does is implementing https://huggingface.co/google/gemma-4-26B-A4B-it/discussions/48 into this set of improvements possible as well? I tried that template out for a bit and it did seem to get better, even when it did get stuck for a moment it was no longer a complete breakdown. For reference there's some more information about looping on DeepMind GitHub too - https://github.com/google-deepmind/gemma/issues/610

Hi! I tested this with gemma-4-12B (QAT Q4 version by Unsloth) and llama.cpp server with the --chat-template-file override. I used llama-ui to verify the chat template used was the one in this PR, that I downloaded previously. The were still instances where the model were unable to apply changes using the edit tool:

Happened again in the same session:

For this particular coding session, it started making edits properly, but as iterations were happening, it tried edit twice, failed, and then opted to use write instead each time. Small Go files. The agent ended up seceding in the task, but it failed to call the edit tool consistently. Sometimes it leads to several minutes of trial and error to make changes, and it slows down significantly the process.

I am still testing and the use case is: use Open Code (or Pi Coding Agent) as the coding agent, and use Gemma 4 (tested 12B, 26B-A4B, all quantized to run on my local PC), and it almost consistently fails to make edit tool calls. Write tool calls work. I'm not sure if the issue is with the tool call itself, with how these open agents describe the tools or if this is related to the quantization itself. I don't have a larger VRAM machine to run unquantized versions for testing.

Honestly, I'm assuming that the issue is solvable by using a LoRA / fine tuning for this case of coding agents. Just need to figure out how to prepare a data-set for this training.

Thanks for the help so far Luciano!

Ok, so, I tested another coding session and it succeeded properly at around 17k tokens of context and made several import fixes with the edit tool. This was a first so I do think the chat template appears to have helped here. I'll prepare some test cases for this use case and come up with a reproducible repo + prompt so we can compare the before/after more reliably.

OK, it feels like the edit tool call success rate is now significantly higher ... it barely failed now and it is working much better than previous attempts for a long session of 41k tokens of context and several agent turns. Progress!

I'm not particularly sure if it helps with Gemma 4 26B looping that happens when it's trying to list things (I'm no engineer), but if it does is implementing https://huggingface.co/google/gemma-4-26B-A4B-it/discussions/48 into this set of improvements possible as well? I tried that template out for a bit and it did seem to get better, even when it did get stuck for a moment it was no longer a complete breakdown. For reference there's some more information about looping on DeepMind GitHub too - https://github.com/google-deepmind/gemma/issues/610

After some more testing the fix I linked doesn't actually help in any major way, so yeah it wouldn't be useful here. I'm not sure if the looping is fixable from the template-level at all, it seems to happen when Gemma lists things; then in the loop, it does something wrong but then interrupts itself with "Wait, but" and then enters a loop again.

A good way to reproduce if someone is interested, is to prompt Gemma4 26B with something along the lines of "List all the episodes of the TV series Firefly" when it doesn't have an internet search tool - it's not actually 100%, but it's trivial to trigger degenerate looping over a few tries. I had tool context in each case, not sure if it matters (just generic tools for file operations etc, the backend was llama.cpp).
This happens both with BF16 and quants.

Hi @boneshr , I'm sorry for the experience you are facing!

can you share more details about the env where you ask for the "List all the episodes of the TV series Firefly" prompt? ie.:

• what is the temperature you are using?
• are you using any particular framework (Transformers, vLLM, SGLang, etc)?
• can you see this looping consistently with the 26B-A4B MoE model?

Thanks!

I'm using the recommended 1.0 temperature.
I can see this looping more than 50% of the time when asking Gemma 4 26B this question, but even if it does not indefinitely loop the model clearly has very ugly behaviour when trying to formulate a response. I'm including a screenshot of a fragment of how the response typically looks, even when grounded with a web search (via Brave's Search API)

I'm using a .gguf file in BF16 format (for testing, though typically I'd use Q8_0 from Bartowski) with llama.cpp's llama-server. I'm getting completions via Python using OpenAI API's (OpenAI).chat.completions() method. This is the minimal system prompt that I can use to reproduce this issue:
"You are an personal AI assistant based on the Gemma 4 architecture. Base your responses on facts and available sources, especially if you are not sure about something. You have several tools that you should use to help you with the User's requests."

My llama-server run command is as follows: (the chat template file was taken from this PR, from before today's changes)
./llama-server -m ~/Models/google_gemma-4-26B-A4B-it-bf16-00001-of-00002.gguf --temp 1.0 --top-p 0.95 --top-k 64 -np 4 --kv-unified --spec-type ngram-mod,ngram-map-k4v,draft-mtp --ctx-checkpoints 3 --model-draft ~/Models/mtp/mtp-google_gemma-4-26B-A4B-it-Q8_0.gguf --chat-template-file ~/Models/templates/g4_template_pr118_new.jinja --spec-draft-n-max 4 --mmproj ~/Models/mmproj/mmproj_gemma_4_26B_A4B_mmproj-BF16.gguf --image-min-tokens 1120 --image-max-tokens 1120 --batch-size 2048 --ubatch-size 2048

I would prefer not to share all tool code since I'm just making it for myself, but to me it seems like it's trivial to reproduce this issue just off what I've given (and I can reliably reproduce it on my end). If that is not the case, I'll just try to upload a minimal example to a GitHub repo, it's just that I'm a bit too tired to do it now, I'm really sorry.

I really like Gemma 4 MoE and I think it's super useful for a lot of 'semi'-automation tasks (running tests based on descriptions, with appropriate tools), but this one issue with lists is really annoying if you run into it, and the higher you go in context the easier it feels to hit in my experience.

Can someone test this for E4B and E2B as well for how stable it is?