--- license: apache-2.0 license_link: https://ai.google.dev/gemma/docs/gemma_4_license pipeline_tag: image-text-to-text library_name: transformers base_model: trohrbaugh/gemma-4-31b-it-heretic-ara base_model_relation: quantized tags: - gemma - gemma-4 - multimodal - vision-language - abliterated - heretic - ara - uncensored - decensored - awq - w4a16 - quantized - compressed-tensors - vllm --- # Gemma-4-31B-it-abliterated-heretic-AWQ-W4A16 AWQ W4A16 (group_size 128, symmetric) quantization of [trohrbaugh/gemma-4-31b-it-heretic-ara](https://huggingface.co/trohrbaugh/gemma-4-31b-it-heretic-ara) — a Heretic-ARA abliterated derivative of [google/gemma-4-31b-it](https://huggingface.co/google/gemma-4-31b-it). > ⚠️ **Decensored model.** Safety guardrails have been deliberately removed. > Research and experimentation only. See full disclaimer below. ## Quantization Details | Parameter | Value | | :-------- | :---: | | **Method** | AWQ (Activation-aware Weight Quantization) | | **Scheme** | W4A16 (symmetric) | | **Weight Bits** | 4 | | **Activation Bits** | 16 | | **Group Size** | 128 | | **Format** | compressed-tensors | | **Calibration Dataset** | HuggingFaceH4/ultrachat_200k | | **Calibration Samples** | 256 | | **Max Sequence Length** | 2048 | | **Vision Tower** | Unquantized (full precision) | | **LM Head** | Unquantized (full precision) | | **Compatible Inference Engine** | vLLM (`vllm/vllm-openai:gemma4`) | ## Quantization Notes - **All multimodal paths kept full precision**: Vision tower, audio tower, video tower, multi-modal projector, and all modality-specific embedding and projection layers are excluded from quantization. Only language-model linear layers (attention Q/K/V/O and MLP gate/up/down) are quantized. - **LM head unquantized**: Standard practice to preserve output token distribution quality at negligible size cost. - **v_proj → o_proj smoothing skipped**: llm-compressor reports incompatible balance layer dimensions for `v_proj → o_proj` on this checkpoint across many of Gemma 4's decoder blocks. Per-channel smoothing is skipped for that path, matching the standard AWQ-for-GQA pattern. - **Hybrid attention and context window unchanged**: Quantization only touches Linear layer weights; Gemma 4's interleaved local/global attention pattern and 256K context capacity are structurally preserved. Actual long-context quality at 4-bit has not been benchmarked. - **Multimodal input intact**: Text + image input works as in the base model. Use the standard Gemma 4 chat template with image tokens placed before text. - **Full quantization recipe** is preserved in `recipe.yaml` in this repo for reproducibility. It records the exact AWQ mappings, ignore patterns, and scheme parameters applied. ## Deployment Recommended inference with vLLM (Gemma 4 requires the `vllm/vllm-openai:gemma4` image): ```bash vllm serve alonsoko/gemma-4-31b-it-abliterated-heretic-AWQ-W4A16 \ --trust-remote-code \ --tensor-parallel-size 1 \ --max-model-len 32768 ``` Recommended sampling (per upstream Gemma 4 guidance): `temperature=1.0, top_p=0.95, top_k=64` To enable thinking mode, include the `<|think|>` token at the start of the system prompt. ## Hardware Requirements Approximate VRAM for inference at this quantization (W4A16-G128): - **Weights** (quantized language model + unquantized vision tower, all in one safetensors): **~19 GB** - **KV cache** (per request, grows with context length): ~1-4 GB at 32K context, more at longer contexts - **Recommended**: Single 24 GB GPU (RTX 3090/4090, A10G) for standard context up to ~32K, or single 48 GB GPU (L40S/A6000) for long context / batch serving - **Long context (128K+)**: 48-80 GB recommended due to KV cache growth --- ## ⚠️ Disclaimer **This model is intended for research, experimentation, and testing purposes only.** - This model may produce harmful, offensive, inappropriate, or otherwise objectionable content. - The abliteration process removes safety guardrails that were intentionally built into the original model. - Do **not** use this model in production systems, consumer-facing applications, or any context where harmful outputs could cause real-world harm. - The authors and contributors of this toolkit bear no responsibility for any misuse of this model or any harm caused by outputs generated by this model. - By using this model, you agree that you are solely responsible for ensuring its use complies with all applicable laws and ethical guidelines. This model is shared purely for academic and technical exploration of model internals. ## Abliteration Performed with [Heretic](https://github.com/p-e-w/heretic) v1.2.0+custom using the [Arbitrary-Rank Ablation (ARA)](https://github.com/p-e-w/heretic/pull/211) method. | Parameter | Value | | :-------- | :---: | | **start_layer_index** | 2 | | **end_layer_index** | 60 | | **preserve_good_behavior_weight** | 0.9920 | | **steer_bad_behavior_weight** | 0.0001 | | **overcorrect_relative_weight** | 0.4709 | | **neighbor_count** | 10 | ### Performance | Metric | This model | Original [google/gemma-4-31b-it](https://huggingface.co/google/gemma-4-31b-it) | | :----- | :--------: | :---------------------------: | | **KL divergence** | 0.0120 | 0 *(by definition)* | | **Refusals** | 5/100 | 98/100 | *Measured on the unquantized heretic-ara base; AWQ is expected to preserve these closely but has not been separately benchmarked.* --- ## About the Base Model Original model: [google/gemma-4-31b-it](https://huggingface.co/google/gemma-4-31b-it) Gemma 4 31B is a dense multimodal model (text + image input, text output) with a 256K context window, native thinking-mode support, function calling, and strong performance on reasoning, coding, and vision benchmarks. See the [base model card](https://huggingface.co/google/gemma-4-31b-it) for architectural details, benchmark results, training data, and Google's responsible-use guidance.
Gemma 4