README.md

---
license: apache-2.0
base_model: google/gemma-4-12B-it
tags:
- multimodal
- vision
- audio
- quantization
- autoround
- vllm
- tensor_type:q4
---

# Gemma 4 12B IT - W4A16 AutoRound Quantized Variants

This repository hosts 4-bit weight, 16-bit activation (W4A16) quantized variants of **google/gemma-4-12B-it**. The models were quantized using Intel's AutoRound framework tailored specifically for the architectural requirements of the Gemma 4 family.

Available formats in the series:
* `Vishva007/gemma-4-12B-it-W4A16-AutoRound` (Standard AutoRound format)
* `Vishva007/gemma-4-12B-it-W4A16-AutoRound-AWQ` (AWQ format conversion)
* `Vishva007/gemma-4-12B-it-W4A16-AutoRound-GPTQ` (GPTQ format conversion)

## Architectural Advantage: Gemma 4 12B Unified

The **Gemma 4 12B Unified** model features a ground-breaking **encoder-free multimodal architecture**. Unlike traditional vision-language models that rely on separate heavy visual/audio encoders (like ViT or Whisper), the 12B Unified model projects raw image patches and audio waveforms directly into the main LLM's embedding space via lightweight linear layers. 

Because text, image, and audio flow natively into a single decoder-only transformer, this model benefits dramatically from weight-only quantization, offering minimal multimodal latency and a highly streamlined memory footprint.

---

## Quantization Recipe & Environment

The models were generated using the following computational setup and tuning parameters. Notably, it leverages **RTN mode (`iters: 0`) with non-text module protection**, which is mandatory for maintaining performance stability on the Gemma 4 native multimodal layers.

### Environment Setup
* **PyTorch Version:** `2.10.0+cu128`
* **CUDA Version:** `12.8` (CUDA Available: `True`)
* **Hardware:** NVIDIA L40 (48 GB VRAM)

### Tuning Configuration
```python
TUNING_CONFIG = {
    "group_size": 128,
    "sym": True,
    "iters": 0,                      # RTN mode — required for Gemma 4 stability
    "disable_opt_rtn": True,
    "nsamples": 256,
    "seqlen": 2048,
    "low_gpu_mem_usage": False,
    "quant_nontext_module": False,   # Keeps linear projection layers intact for vision/audio inputs
    "layer_config": layer_config,
}

```

---

## Deployment & Inference Guide

Due to the unique unified nature of Gemma 4's multimodal inputs, deployment requires the specialized vLLM unified image containing updated parser configurations.

### Running with Docker (vLLM)

Deploy a production-ready OpenAI-compatible API server using the custom `vllm-openai:gemma4-unified` engine:

```bash
docker run --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:gemma4-unified \
    --model Vishva007/gemma-4-12B-it-W4A16-AutoRound \
    --quantization autoround \
    --port 8000 \
    --max-model-len 4096 \
    --trust-remote-code

```

*(Note: Change `--quantization` flag to `awq` or `gptq` depending on the specific variant file path you mount).*

---

## 🛠️ Reproduction & Source Code

The complete script used to perform this quantization, manage the layer configurations, and optimize the memory footprint is fully open-sourced. 

You can find the step-by-step Jupyter Notebook containing the exact execution environment, dependency setups, and layer configurations in the GitHub repository:

👉 **[AutoRound-Quantization: Gemma 4 12B RTN Notebook](https://github.com/vishvaRam/AutoRound-Quantaization/blob/main/Gemma4/auto_round_Gemma4-12B-RTN.ipynb)**

### How to Reproduce
To replicate this quantization run on your own hardware (e.g., NVIDIA L40 or similar 40GB+ VRAM instances), clone the setup and run the notebook:

```bash
git clone [https://github.com/vishvaRam/AutoRound-Quantaization.git](https://github.com/vishvaRam/AutoRound-Quantaization.git)
cd AutoRound-Quantaization/Gemma4