Create README.md

README.md

@@ -0,0 +1,98 @@
+---
+license: apache-2.0
+base_model: google/gemma-4-12B-it
+tags:
+- multimodal
+- vision
+- audio
+- quantization
+- autoround
+- vllm
+- tensor_type:q4
+---
+
+# Gemma 4 12B IT - W4A16 AutoRound Quantized Variants
+
+This repository hosts 4-bit weight, 16-bit activation (W4A16) quantized variants of **google/gemma-4-12B-it**. The models were quantized using Intel's AutoRound framework tailored specifically for the architectural requirements of the Gemma 4 family.
+
+Available formats in the series:
+* `Vishva007/gemma-4-12B-it-W4A16-AutoRound` (Standard AutoRound format)
+* `Vishva007/gemma-4-12B-it-W4A16-AutoRound-AWQ` (AWQ format conversion)
+* `Vishva007/gemma-4-12B-it-W4A16-AutoRound-GPTQ` (GPTQ format conversion)
+
+## Architectural Advantage: Gemma 4 12B Unified
+
+The **Gemma 4 12B Unified** model features a ground-breaking **encoder-free multimodal architecture**. Unlike traditional vision-language models that rely on separate heavy visual/audio encoders (like ViT or Whisper), the 12B Unified model projects raw image patches and audio waveforms directly into the main LLM's embedding space via lightweight linear layers. 
+
+Because text, image, and audio flow natively into a single decoder-only transformer, this model benefits dramatically from weight-only quantization, offering minimal multimodal latency and a highly streamlined memory footprint.
+
+---
+
+## Quantization Recipe & Environment
+
+The models were generated using the following computational setup and tuning parameters. Notably, it leverages **RTN mode (`iters: 0`) with non-text module protection**, which is mandatory for maintaining performance stability on the Gemma 4 native multimodal layers.
+
+### Environment Setup
+* **PyTorch Version:** `2.10.0+cu128`
+* **CUDA Version:** `12.8` (CUDA Available: `True`)
+* **Hardware:** NVIDIA L40 (48 GB VRAM)
+
+### Tuning Configuration
+```python
+TUNING_CONFIG = {
+    "group_size": 128,
+    "sym": True,
+    "iters": 0,                      # RTN mode — required for Gemma 4 stability
+    "disable_opt_rtn": True,
+    "nsamples": 256,
+    "seqlen": 2048,
+    "low_gpu_mem_usage": False,
+    "quant_nontext_module": False,   # Keeps linear projection layers intact for vision/audio inputs
+    "layer_config": layer_config,
+}
+
+```
+
+---
+
+## Deployment & Inference Guide
+
+Due to the unique unified nature of Gemma 4's multimodal inputs, deployment requires the specialized vLLM unified image containing updated parser configurations.
+
+### Running with Docker (vLLM)
+
+Deploy a production-ready OpenAI-compatible API server using the custom `vllm-openai:gemma4-unified` engine:
+
+```bash
+docker run --gpus all \
+    -v ~/.cache/huggingface:/root/.cache/huggingface \
+    -p 8000:8000 \
+    --ipc=host \
+    vllm/vllm-openai:gemma4-unified \
+    --model Vishva007/gemma-4-12B-it-W4A16-AutoRound \
+    --quantization autoround \
+    --port 8000 \
+    --max-model-len 4096 \
+    --trust-remote-code
+
+```
+
+*(Note: Change `--quantization` flag to `awq` or `gptq` depending on the specific variant file path you mount).*
+
+---
+
+## 🛠️ Reproduction & Source Code
+
+The complete script used to perform this quantization, manage the layer configurations, and optimize the memory footprint is fully open-sourced. 
+
+You can find the step-by-step Jupyter Notebook containing the exact execution environment, dependency setups, and layer configurations in the GitHub repository:
+
+👉 **[AutoRound-Quantization: Gemma 4 12B RTN Notebook](https://github.com/vishvaRam/AutoRound-Quantaization/blob/main/Gemma4/auto_round_Gemma4-12B-RTN.ipynb)**
+
+### How to Reproduce
+To replicate this quantization run on your own hardware (e.g., NVIDIA L40 or similar 40GB+ VRAM instances), clone the setup and run the notebook:
+
+```bash
+git clone [https://github.com/vishvaRam/AutoRound-Quantaization.git](https://github.com/vishvaRam/AutoRound-Quantaization.git)
+cd AutoRound-Quantaization/Gemma4
+