README.md

metadata

license: gemma
base_model: google/gemma-4-31B-it-qat-q4_0-unquantized
tags:
  - tensorizer
  - modelopt
  - nvidia
  - fp4
  - nvfp4
  - blackwell
  - vllm
  - qat
pipeline_tag: text-generation

Gemma 4 31B IT QAT NVFP4 Blackwell

This repository contains the Gemma 4 31B Instruction-Tuned model quantized to native FP4 (NVFP4) precision for high-efficiency inference on NVIDIA Blackwell GPU architectures (e.g., B200, B100).

Quantized from the official unquantized QAT weights (google/gemma-4-31B-it-qat-q4_0-unquantized) using NVIDIA Model Optimizer (modelopt), this model delivers exceptional throughput and memory savings while preserving high quality by leveraging the Quantization-Aware Training (QAT) properties of the base checkpoint.

Model Details

• Base Model: google/gemma-4-31B-it-qat-q4_0-unquantized
• Quantization Format: NVFP4 (native FP4 on Blackwell)
• Group Size: 16 (per-group scaling)
• Excluded Modules: lm_head, model.embed_vision*, model.vision_tower* (runs in high precision)
• Producer: NVIDIA Model Optimizer (modelopt)

Why does Hugging Face show "17B params" instead of "31B"? Hugging Face calculates the parameter count dynamically from the tensor shapes in the safetensors files. Because this model's weights are quantized to native 4-bit (FP4) and packed (two 4-bit weights per single int8 byte), the physical tensor dimensions are halved (e.g., a shape of [5376, 21504] becomes [5376, 10752]). This causes the Hugging Face auto-estimator to count only half of the quantized parameters, reporting around 17.7B total active parameters. The model architecture is identical to the full 31B dense model.

Quickstart with vLLM

To run this model on an NVIDIA Blackwell GPU using vLLM (version >= 0.6.0 recommended):

vllm serve melcheikh/gemma-4-31B-it-qat-NVFP4-Blackwell --quantization modelopt

Speculative Decoding (Recommended)

To achieve maximum tokens/sec, combine this model with its companion quantized assistant model:

vllm serve melcheikh/gemma-4-31B-it-qat-NVFP4-Blackwell \
    --quantization modelopt \
    --speculative-model melcheikh/gemma-4-31B-it-qat-assistant-NVFP4-Blackwell \
    --num-speculative-tokens 3 \
    --speculative-draft-limit 4

License

Subject to the Gemma Terms of Use.