---
language:
  - en
license: apache-2.0
base_model: google/gemma-4-12B-it
tags:
  - gguf
  - gemma
  - gemma4
  - multimodal
  - vision
  - audio
  - text-generation
  - image-text-to-text
  - llama-cpp
  - agent
library_name: gguf
pipeline_tag: image-text-to-text
inference: true
model_name: Gemma 4 12B IT (GGUF — full quant collection)
---

# Gemma 4 12B Instruction-Tuned — GGUF (multimodal)

**Community GGUF mirror** of [google/gemma-4-12B-it](https://huggingface.co/google/gemma-4-12B-it) for **local, encoder-free multimodal AI** on consumer hardware (~16 GB VRAM).

Announced June 2026: [Google blog](https://blog.google/innovation-and-ai/technology/developers-tools/introducing-gemma-4-12B/) · [Developer guide](https://developers.googleblog.com/gemma-4-12b-the-developer-guide/)

| | |
|---|---|
| **Parameters** | ~12B dense |
| **Modalities** | Text, **vision**, **audio** (native in backbone) |
| **License** | Apache 2.0 |
| **Architecture** | Encoder-free (no separate vision/audio towers) |
| **Context** | See upstream config |
| **Vision in GGUF** | Requires **`mmproj-*.gguf`** alongside main weights |

## Why this repo exists

- **One download hub** for all major quants (K-quants, IQ, Q8, mmproj).
- **Fast Hub-side sync** from [bartowski/gemma-4-12B-it-GGUF](https://huggingface.co/bartowski/gemma-4-12B-it-GGUF) — no re-upload from your laptop.
- **Documented use cases** for contributors: [gemma-4-12b-local](https://github.com/Edmon02/audio_set/tree/main/projects/gemma-4-12b-local) (agents, LiteRT, llama.cpp, MLX).
- **Apple Silicon MLX quants:** [Edmon02/gemma-4-12B-it-MLX](https://huggingface.co/Edmon02/gemma-4-12B-it-MLX)

## Available files

See [`gguf-manifest.json`](gguf-manifest.json) for the live file list.

### Essential tier (recommended)

| File | Use |
|------|-----|
| `gemma-4-12B-it-Q4_K_M.gguf` | **Best balance** — 16 GB laptops |
| `gemma-4-12B-it-Q5_K_M.gguf` | Higher quality |
| `gemma-4-12B-it-Q6_K.gguf` / `Q8_0` | Max quality |
| `gemma-4-12B-it-Q3_K_M.gguf` | Tighter VRAM |
| `gemma-4-12B-it-Q2_K.gguf` | Minimum size |
| `gemma-4-12B-it-IQ4_XS.gguf` / `IQ4_NL.gguf` | IQ variants |
| `mmproj-gemma-4-12B-it-f16.gguf` | **Required for images** in llama.cpp |

### Full tier

All bartowski quants (Q2_K_L, Q3_K_XL, Q4_0, Q4_1, bf16, imatrix, etc.) — run `make sync-gemma4-gguf-full`.

## Download

```bash
pip install -U huggingface_hub

# Text + vision (recommended)
huggingface-cli download Edmon02/gemma-4-12B-it-GGUF \
  gemma-4-12B-it-Q4_K_M.gguf \
  mmproj-gemma-4-12B-it-f16.gguf \
  --local-dir ./models/gemma-4-12b
```

Accept the license on [google/gemma-4-12B-it](https://huggingface.co/google/gemma-4-12B-it) before using weights.

## Quick start

### llama.cpp (text)

```bash
llama-cli -m gemma-4-12B-it-Q4_K_M.gguf -p "Explain encoder-free multimodal models in 3 bullets." -n 256
```

### llama.cpp (image + text)

```bash
llama-mtmd-cli \
  -m gemma-4-12B-it-Q4_K_M.gguf \
  --mmproj mmproj-gemma-4-12B-it-f16.gguf \
  --image photo.jpg \
  -p "Describe this image."
```

### LiteRT-LM (OpenAI-compatible local server)

```bash
litert-lm import --from-huggingface-repo=litert-community/gemma-4-12B-it-litert-lm gemma-4-12B-it.litertlm gemma4-12b
litert-lm serve
```

### LM Studio / Ollama

Import `Edmon02/gemma-4-12B-it-GGUF` and select **Q4_K_M** + mmproj.

### Apple Silicon (MLX)

Native inference on M-series Macs via **mlx-vlm** (not mlx-lm — Gemma 4 is multimodal):

```bash
pip install -U mlx-vlm
huggingface-cli download Edmon02/gemma-4-12B-it-MLX 4bit/ --local-dir ./models/gemma-4-12b-mlx
python -m mlx_vlm.generate --model ./models/gemma-4-12b-mlx/4bit --prompt "Hello" --max-tokens 128
```

See [Edmon02/gemma-4-12B-it-MLX](https://huggingface.co/Edmon02/gemma-4-12B-it-MLX) and `make convert-gemma4-mlx` for maintainer conversion.

## Use cases

| Use case | Quant | Tool |
|----------|-------|------|
| Local coding agent | Q4_K_M | OpenCode, Continue, Aider |
| Voice + vision assistant | Q5_K_M + mmproj | Google AI Edge Gallery / Eloquent (Mac) |
| Armenian + English research | Q4_K_M | Pair with [HyVoxPopuli](https://huggingface.co/datasets/Edmon02/hyvoxpopuli) ASR/TTS |
| Low-VRAM laptop | Q3_K_M or IQ4_XS | llama.cpp |
| Fast inference | MTP drafter (upstream) | Google checkpoint + compatible runtime |

## Hardware guide

| VRAM | Suggested files |
|------|-----------------|
| 8 GB | IQ4_XS or Q3_K_M (text only, short context) |
| 16 GB | **Q4_K_M + mmproj** |
| 24 GB+ | Q6_K or Q8_0 + mmproj |

## Provenance

| Item | Source |
|------|--------|
| Base model | `google/gemma-4-12B-it` |
| GGUF quants | Mirrored from `bartowski/gemma-4-12B-it-GGUF` |
| Maintainer scripts | [Edmon02/audio_set](https://github.com/Edmon02/audio_set) `scripts/sync_gemma4_gguf_quants.py` |

## Limitations

- Community quants — validate quality on your tasks vs official BF16.
- **Audio** in GGUF may require latest llama.cpp / LM Studio builds.
- Gated upstream — HF token + license acceptance required for `google/*` repos.

## Contributing

Add recipes under `projects/gemma-4-12b-local/examples/`. See CONTRIBUTING.md in that folder.

## Citation

```bibtex
@article{gemma_2026,
  title={Gemma 4},
  author={Google DeepMind},
  year={2026},
  url={https://blog.google/innovation-and-ai/technology/developers-tools/introducing-gemma-4-12B/}
}
```