# Serving Notes

This release uses the official Hugging Face Gemma 4 checkpoint naming/layout.
The bundled `chat_template.jinja` and the inline template in
`tokenizer_config.json` were refreshed together on April 18, 2026 and should
be kept in sync if you make further prompt-template edits.

## vLLM

Gemma 4 multi-turn tool use is sensitive to the chat template and reasoning parser.
The vLLM fix set merged on April 8, 2026 in PR #39027 documents the working setup:

`vllm serve google/gemma-4-31B-it --tool-call-parser gemma4 --enable-auto-tool-choice --reasoning-parser gemma4 --default-chat-template-kwargs '{"enable_thinking": true}' --chat-template examples/tool_chat_template_gemma4.jinja`

Reference:
https://github.com/vllm-project/vllm/pull/39027

For plain chat serving, keep reasoning disabled by default unless you
explicitly want reasoning content in responses. Gemma 4 templates can emit
reasoning channel markers, so validate your final serving config against the
exact chat template bundled in this release.

## SGLang

SGLang's OpenAI-compatible server uses the Hugging Face tokenizer chat
template by default. If you need a different template, launch with
`--chat-template /path/to/template.jinja`. If the tokenizer exposes multiple
named templates, you can select one with `--hf-chat-template-name`.

Reference:
https://docs.sglang.io/references/custom_chat_template.html
https://docs.sglang.io/advanced_features/server_arguments.html

## Apple Silicon / MLX

TurboQuant+ is an inference-time optimization, not a weight change. It can be used
with MLX-compatible Gemma 4 models to improve long-context KV-cache efficiency.

Reference:
https://github.com/TheTom/turboquant_plus

## Validation

Latest local validation on April 18, 2026:

- Capability audit: 9 / 9 passed
- Reliability audit: 20 / 20 passed
- Local OpenAI-compatible MLX red-team: 10 / 13 passed

The remaining MLX red-team misses were caused by 2 strict checker phrase mismatches
on safe refusals and 1 text-only multimodal rejection mismatch. They did not surface
as hidden-prompt leakage, loop regressions, or truthfulness regressions in the final run.