google

/

gemma-4-12B-it-qat-w4a16-ct

like 49

Follow

Google 63.3k

This is what i found from gemini after i ran into the same issue.

Why this happens with the gemma4-unified image
Even if you use a "unified" image, if that image is based on a vLLM version (e.g., v0.22.x) that does not include the specific PR/commit providing native support for the Gemma 4 architecture, you are still operating in that "fallback" path. The image name might sound like it supports the model, but if the internal code registry in that specific build hasn't "registered" the model, the fallback path kicks in, and the bug persists.

Summary
Is it the same bug? Yes. You are fighting the generic fallback wrapper trying to "quantize" layers that are not supposed to be quantized.

Can you fix it by changing flags? No. This is an issue in the underlying vLLM code that identifies and loads model parameters. No CLI flag can stop the code from incorrectly attempting to load those weights.

The Solution: You need a version of vLLM that has the native Gemma4UnifiedForConditionalGeneration implementation (which includes the specific "ignore list" code to skip the vision embedder). As of now, this is only available in the absolute latest main branch builds, and even then, it is highly experimental for QAT/Compressed-Tensors formats.

If you are not a developer capable of patching the vLLM source code to manually add that layer to the "ignore" list in the quantization loader, Ollama remains the only production-stable way to run this model today.

here explaining of issues
https://github.com/vllm-project/vllm/issues/44796

You can create your own Dockerfile which works for this model. It applies these two fixes:

1. https://github.com/vllm-project/vllm/issues/45039
2. https://github.com/vllm-project/vllm/issues/44494

FROM vllm/vllm-openai:gemma4-unified-cu129


# HOTFIX 1: vLLM Issue #45039 (The 'num_soft_tokens' bug)
# The Gemma 4 Unified config lacks the 'num_soft_tokens' attribute. 
# We use sed to replace direct attribute access with getattr() to provide a fallback value (280).
RUN sed -i 's/config\.vision_config\.num_soft_tokens/getattr(config.vision_config, "num_soft_tokens", 280)/g' /usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/gemma4_unified.py && \
    sed -i 's/vision_cfg\.num_soft_tokens/getattr(vision_cfg, "num_soft_tokens", 280)/g' /usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/gemma4_unified.py


# HOTFIX 2: Install the import interceptor patch universally so it runs in parent and worker processes
RUN cat <<'EOF' >> /usr/local/lib/python3.12/dist-packages/vllm/__init__.py

# --- GEMMA 4 VISION UNQUANTIZED HOTFIX (Issue #44494) ---
import importlib.abc
import importlib.machinery
import sys

_TARGET = "vllm.model_executor.models.gemma4_unified"

class _UnquantizeVisionEmbedder(importlib.abc.MetaPathFinder):
    """Patches Gemma4UnifiedVisionEmbedder the first time its module loads."""
    def find_spec(self, name, path, target=None):
        if name != _TARGET:
            return None
        try:
            sys.meta_path.remove(self)  # one-shot: stop intercepting once we've run
        except ValueError:
            pass

        spec = importlib.machinery.PathFinder.find_spec(name, path)
        if spec is None or spec.loader is None:
            return spec

        _exec_module = spec.loader.exec_module

        def exec_module(module):
            _exec_module(module)
            cls = getattr(module, "Gemma4UnifiedVisionEmbedder", None)
            if cls is None:  # class renamed upstream -> nothing to patch, no-op
                return
            _orig_init = cls.__init__

            def __init__(self, config, quant_config=None):
                # Drop quant_config: the vision embedder must stay unquantized.
                _orig_init(self, config, quant_config=None)

            cls.__init__ = __init__

        spec.loader.exec_module = exec_module
        return spec

sys.meta_path.insert(0, _UnquantizeVisionEmbedder())
# --------------------------------------------------------
EOF


# We can now safely use the standard production API server entrypoint directly
ENTRYPOINT ["python3", "-m", "vllm.entrypoints.openai.api_server"]

Save as Dockerfile and then build it like:

docker build -t vllm-gemma4-unified-cu129 .

And then use the appropriate recipe from vllm: https://recipes.vllm.ai/Google/gemma-4-12B-it

For example:

docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm-gemma4-unified-cu129 google/gemma-4-12B-it-qat-w4a16-ct \
  --tensor-parallel-size 1 \
  --max-num-batched-tokens 4096 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 8 \
  --max-model-len 16384 \
  --reasoning-parser gemma4 \
  --speculative-config '{"model":"google/gemma-4-12B-it-assistant","num_speculative_tokens":4}' \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --chat-template examples/tool_chat_template_gemma4.jinja