---
license: gemma
base_model: google/gemma-4-31B-it-qat-q4_0-unquantized
tags:
- tensorizer
- modelopt
- nvidia
- fp4
- nvfp4
- blackwell
- vllm
- qat
pipeline_tag: text-generation
---

# Gemma 4 31B IT QAT NVFP4 Blackwell

This repository contains the **Gemma 4 31B Instruction-Tuned** model quantized to **native FP4 (NVFP4)** precision for high-efficiency inference on **NVIDIA Blackwell** GPU architectures (e.g., B200, B100).

Quantized from the official unquantized QAT weights (`google/gemma-4-31B-it-qat-q4_0-unquantized`) using **NVIDIA Model Optimizer (`modelopt`)**, this model delivers exceptional throughput and memory savings while preserving high quality by leveraging the Quantization-Aware Training (QAT) properties of the base checkpoint.

## Model Details
- **Base Model:** [google/gemma-4-31B-it-qat-q4_0-unquantized](https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-unquantized)
- **Quantization Format:** `NVFP4` (native FP4 on Blackwell)
- **Group Size:** 16 (per-group scaling)
- **Excluded Modules:** `lm_head`, `model.embed_vision*`, `model.vision_tower*` (runs in high precision)
- **Producer:** NVIDIA Model Optimizer (`modelopt`)

> [!NOTE]
> **Why does Hugging Face show "17B params" instead of "31B"?**
> Hugging Face calculates the parameter count dynamically from the tensor shapes in the `safetensors` files. Because this model's weights are quantized to native 4-bit (FP4) and packed (two 4-bit weights per single `int8` byte), the physical tensor dimensions are halved (e.g., a shape of `[5376, 21504]` becomes `[5376, 10752]`). This causes the Hugging Face auto-estimator to count only half of the quantized parameters, reporting around 17.7B total active parameters. The model architecture is identical to the full 31B dense model.

## Quickstart with vLLM

To run this model on an NVIDIA Blackwell GPU using vLLM (version >= 0.6.0 recommended):

```bash
vllm serve melcheikh/gemma-4-31B-it-qat-NVFP4-Blackwell --quantization modelopt
```

## Speculative Decoding (Recommended)

To achieve maximum tokens/sec, combine this model with its companion quantized assistant model:

```bash
vllm serve melcheikh/gemma-4-31B-it-qat-NVFP4-Blackwell \
    --quantization modelopt \
    --speculative-model melcheikh/gemma-4-31B-it-qat-assistant-NVFP4-Blackwell \
    --num-speculative-tokens 3 \
    --speculative-draft-limit 4
```

## License
Subject to the Gemma Terms of Use.