tuxmanxd/gemma-4-12b-it-uncensored-GGUF

gemma-4-12b-it-uncensored — GGUF

GGUF quantizations of zaakirio/gemma-4-12b-it-uncensored, a decensored (Heretic-abliterated) version of google/gemma-4-12B-it.

These files run with llama.cpp.

Gemma 4 12B (the "Unified" release, June 2026) is an encoder-free unified multimodal model: text, image, audio, and video all project straight into a single decoder-only transformer, with a context window of up to 256K tokens. Abliteration touches only the language weights, so those capabilities carry over unchanged.

Files

Filenames follow gemma-4-12b-it-uncensored-<QUANT>.gguf.

Quant	Size	Notes
Q2_K	4.50 GB	Smallest; lowest quality. Very tight memory only.
Q3_K_M	5.67 GB	Small; usable on low RAM.
Q4_K_S	6.54 GB	Compact 4-bit.
Q4_K_M	6.87 GB	Recommended — best size/quality balance.
Q5_K_M	7.96 GB	Higher quality, slightly larger.
Q6_K	9.11 GB	Near-lossless.
Q8_0	11.80 GB	Effectively lossless vs the BF16 source.
f16	22.20 GB	Full precision; reference / re-quantizing.

Not sure which to pick? Start with Q4_K_M. Go up to Q5/Q6/Q8 if you have the memory and want maximum fidelity; drop to Q3/Q2 only if you're memory-constrained.

Multimodal projector (for image/audio input — see Multimodal):

File	Size	Notes
mmproj-gemma-4-12B-it-bf16.gguf	0.16 GB	Multimodal projector — pair with any quant above.
mtp-gemma-4-12b-it-uncensored.gguf	0.47 GB	Optional MTP speculative-decoding drafter — faster generation on GPUs. See Faster generation with MTP.

Usage

llama.cpp (auto-downloads the chosen quant from this repo):

# Interactive chat
llama-cli -hf zaakirio/gemma-4-12b-it-uncensored-GGUF:Q4_K_M --jinja

# OpenAI-compatible server with web UI
llama-server -hf zaakirio/gemma-4-12b-it-uncensored-GGUF:Q4_K_M --jinja -c 4096

-c sets the context window. Gemma 4 12B supports up to 256K tokens (-c 262144); raise it for long chats/documents as memory allows — 4096 is just a light default.

Or with a file you've already downloaded:

llama-cli -m gemma-4-12b-it-uncensored-Q4_K_M.gguf --jinja -p "Hello, who are you?"

Download a single file:

pip install -U "huggingface_hub[cli]"
hf download zaakirio/gemma-4-12b-it-uncensored-GGUF \
  --include "gemma-4-12b-it-uncensored-Q4_K_M.gguf" --local-dir ./

Prompt format & settings

The chat template is embedded in the GGUF — chat-aware tools apply it automatically (always pass --jinja with llama.cpp). For reference, Gemma 4 introduced new turn tokens — <|turn> / <turn|>, replacing Gemma 3's <start_of_turn> / <end_of_turn> — and now supports a dedicated system role:

<|turn>system
{system instructions — optional}<turn|>
<|turn>user
{prompt}<turn|>
<|turn>model

Image and audio inputs use the <|image|> and <|audio|> placeholder tokens, which the multimodal pipeline fills in for you (see Multimodal).

Recommended sampling (Google defaults): --temp 1.0 --top-p 0.95 --top-k 64.

Thinking mode: Gemma 4 has a reasoning channel. To disable it, pass --chat-template-kwargs '{"enable_thinking":false}' to llama-server.

Multimodal

In llama.cpp, multimodal input ships via a separate projector file; the language .gguf alone is text-only and will reject media. This repo includes mmproj-gemma-4-12B-it-bf16.gguf, which carries both a vision and an audio encoder (clip.has_vision_encoder and clip.has_audio_encoder).

In practice: image and audio input both work. Verified with llama-mtmd-cli: it accepts .wav/.mp3 directly and produced an accurate multilingual transcription of real speech. llama.cpp flags audio as experimental ("audio input is in experimental stage and may have reduced quality"), so treat ASR quality as best-effort. Video is handled as sampled frames (plus audio) and depends on your client doing the frame extraction.

When you load via -hf, llama.cpp auto-downloads the projector from this repo — images just work:

llama-server -hf zaakirio/gemma-4-12b-it-uncensored-GGUF:Q4_K_M --jinja

With local files, pass it explicitly with --mmproj:

llama-server -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --mmproj mmproj-gemma-4-12B-it-bf16.gguf --jinja

# download both files
hf download zaakirio/gemma-4-12b-it-uncensored-GGUF \
  --include "gemma-4-12b-it-uncensored-Q4_K_M.gguf" "mmproj-gemma-4-12B-it-bf16.gguf" --local-dir ./

One-shot image or audio from the CLI with llama-mtmd-cli:

# Image
llama-mtmd-cli -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --mmproj mmproj-gemma-4-12B-it-bf16.gguf --jinja \
  --image photo.png -p "What's in this image?"

# Audio (experimental in llama.cpp; 16 kHz WAV)
llama-mtmd-cli -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --mmproj mmproj-gemma-4-12B-it-bf16.gguf --jinja \
  --audio clip.wav -p "Transcribe this audio."

The projector is the unmodified Gemma 4 one (abliteration leaves it untouched). bf16 is preferred — it's small enough that quantizing it buys nothing.

Faster generation with MTP (speculative decoding)

Gemma 4 ships a small (~0.4 B) multi-token-prediction (MTP) drafter (google/gemma-4-12B-it-assistant) that speeds up generation through self-speculative decoding: the drafter proposes several tokens at once and the full model verifies them in a single forward pass. Output is identical to normal decoding — speculative decoding is mathematically exact, so it changes speed only, never quality, and the model stays exactly as uncensored: the main model verifies and decides every emitted token; the drafter merely proposes. (Google state the same — MTP "guarantee[s] the exact same quality as standard autoregressive generation" — and the implementation PR confirmed it by reproducing Gemma's published AIME-26 score.)

For that same reason the drafter does not need to be abliterated: this repo ships Google's unmodified upstream Gemma 4 drafter (the file is renamed mtp-gemma-4-12b-it-uncensored.gguf only so it sits beside the trunk — the weights are upstream), and your decensored trunk verifies every token. It's alignment-independent, exactly like the multimodal projector above.

MTP for Gemma 4 landed in llama.cpp PR #23398 (merged 2026-06-07; see also support discussion #22735), so it needs a llama.cpp build from that date or later.

# Grab the trunk + the MTP drafter
hf download zaakirio/gemma-4-12b-it-uncensored-GGUF \
  --include "gemma-4-12b-it-uncensored-Q4_K_M.gguf" "mtp-gemma-4-12b-it-uncensored.gguf" --local-dir ./

# Serve with MTP enabled
llama-server -m gemma-4-12b-it-uncensored-Q4_K_M.gguf \
  --model-draft mtp-gemma-4-12b-it-uncensored.gguf \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  -ngl 99 --flash-attn auto --jinja

⚠️ MTP helps on GPUs but is slower on Apple Silicon

The benefit depends entirely on the backend.

On a GPU it's a win. The implementation PR measured >2× throughput on the dense Gemma 4 (31B) model — NVIDIA DGX Spark, ~0.59 mean draft acceptance at --spec-draft-n-max 4 — and Unsloth report 1.5–2.2× for Gemma 4 with MTP. (PR #23398, Unsloth MTP guide.)

On Apple Silicon (Metal) it's a regression. Measured here on this model — Q4_K_M, Apple M4 Pro, Gemma's default sampling (--temp 1.0 --top-p 0.95 --top-k 64), through llama-server:

M4 Pro / Metal · Q4_K_M	Generation	Draft acceptance
MTP off (baseline)	~27 tok/s	—
MTP on (--spec-draft-n-max 4)	~12 tok/s (≈2× slower)	~0.45

The drafts are accepted (~45%), but on Metal the draft-evaluation overhead outweighs the decode steps it saves. This isn't specific to this model: an independent report (#23752 — Qwen3.5-9B on an M1 Max) found MTP slower at every setting, even at 100% acceptance, so no sampling tweak rescues it on Metal.

Recommendation: enable MTP when serving on an NVIDIA (CUDA) GPU; leave it off on a Mac. On a GPU, greedy / low-temperature sampling raises acceptance and widens the speedup.

About the base model

A decensored derivative produced with Heretic (automatic directional ablation). Compared with the original:

Metric	Decensored	Original
Refusals (/100 harmful prompts)	23	99
KL divergence (harmless prompts)	0.043	0 (by definition)

The refusal count is Heretic's keyword heuristic, which is known to over-count (it flags disclaimer-wrapped compliance as a refusal; ~11% precision per arXiv:2512.13655). We report only the measured marker figure and did not run a classifier-based eval on this model, so real compliance is likely higher. See the source model card for parameters and details.

Intended use & disclaimer

This model has had its refusal behaviour substantially removed and will comply with requests the original would have declined. Provided for research and unrestricted local use. You are responsible for how you use it and for complying with applicable law and the base model's Gemma license, which carries over to this derivative. Not for all audiences.

Provenance

• Quantized from zaakirio/gemma-4-12b-it-uncensored (BF16) using llama.cpp convert_hf_to_gguf.py + llama-quantize.
• Base model: google/gemma-4-12B-it
• Decensoring tool: Heretic by p-e-w · technique: Arditi et al. (2024)
• MTP drafter (mtp-gemma-4-12b-it-uncensored.gguf): the unmodified upstream google/gemma-4-12B-it-assistant (~0.4 B), Q8_0 GGUF via unsloth/gemma-4-12b-it-GGUF. Not abliterated — it only proposes tokens; the decensored trunk verifies every one, so output is identical to running this model alone.