google

/

gemma-4-12B-it-qat-w4a16-ct

like 54

Follow

Google 64.2k

Summary

The QAT checkpoint google/gemma-4-12B-it-qat-w4a16-ct is missing vision_config.num_soft_tokens in config.json, while the BF16 sibling google/gemma-4-12B-it includes it (280).

This causes multimodal inference failures with vLLM (e.g. AttributeError: 'Gemma4UnifiedVisionConfig' object has no attribute 'num_soft_tokens' during image requests). See also: https://github.com/vllm-project/vllm/issues/45039

Comparison

Per Gemma4 Unified docs, the default soft-token budget for images is 280 (not mm_posemb_size / 1120, which is the positional-embedding table size).

Suggested fix

Add to config.json:

"vision_config": {
  ...
  "num_soft_tokens": 280,
  ...
}

Workaround (confirmed on SageMaker + vLLM gemma4-unified)

Manually patch config.json with "num_soft_tokens": 280 before deployment. Image + text inference works after this change.

Environment: vLLM gemma4-unified-cu129, google/gemma-4-12B-it-qat-w4a16-ct, ml.g5.2xlarge, compressed-tensors (w4a16-ct).