Instructions to use Nabhos/Gemma-4-31B-it-abliterated-heretic-text-only with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Nabhos/Gemma-4-31B-it-abliterated-heretic-text-only with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Nabhos/Gemma-4-31B-it-abliterated-heretic-text-only")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Nabhos/Gemma-4-31B-it-abliterated-heretic-text-only")
model = AutoModelForCausalLM.from_pretrained("Nabhos/Gemma-4-31B-it-abliterated-heretic-text-only")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Nabhos/Gemma-4-31B-it-abliterated-heretic-text-only with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Nabhos/Gemma-4-31B-it-abliterated-heretic-text-only"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nabhos/Gemma-4-31B-it-abliterated-heretic-text-only",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Nabhos/Gemma-4-31B-it-abliterated-heretic-text-only

SGLang

How to use Nabhos/Gemma-4-31B-it-abliterated-heretic-text-only with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Nabhos/Gemma-4-31B-it-abliterated-heretic-text-only" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nabhos/Gemma-4-31B-it-abliterated-heretic-text-only",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Nabhos/Gemma-4-31B-it-abliterated-heretic-text-only" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nabhos/Gemma-4-31B-it-abliterated-heretic-text-only",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Nabhos/Gemma-4-31B-it-abliterated-heretic-text-only with Docker Model Runner:
```
docker model run hf.co/Nabhos/Gemma-4-31B-it-abliterated-heretic-text-only
```

Gemma-4-31B-it-abliterated-heretic-text-only / README.md

Nabhos

Upload folder using huggingface_hub

2b833a6 verified 1 day ago

preview code

Raw

History Blame Contribute Delete

6.12 kB

	---
	license: apache-2.0
	license_link: https://ai.google.dev/gemma/docs/gemma_4_license
	pipeline_tag: image-text-to-text
	library_name: transformers
	base_model: trohrbaugh/gemma-4-31b-it-heretic-ara
	base_model_relation: quantized
	tags:
	- gemma
	- gemma-4
	- multimodal
	- vision-language
	- abliterated
	- heretic
	- ara
	- uncensored
	- decensored
	- awq
	- w4a16
	- quantized
	- compressed-tensors
	- vllm
	---

	# Gemma-4-31B-it-abliterated-heretic-AWQ-W4A16

	AWQ W4A16 (group_size 128, symmetric) quantization of
	[trohrbaugh/gemma-4-31b-it-heretic-ara](https://huggingface.co/trohrbaugh/gemma-4-31b-it-heretic-ara)
	— a Heretic-ARA abliterated derivative of [google/gemma-4-31b-it](https://huggingface.co/google/gemma-4-31b-it).

	> ⚠️ Decensored model. Safety guardrails have been deliberately removed.
	> Research and experimentation only. See full disclaimer below.

	## Quantization Details
	\| Parameter \| Value \|
	\| :-------- \| :---: \|
	\| Method \| AWQ (Activation-aware Weight Quantization) \|
	\| Scheme \| W4A16 (symmetric) \|
	\| Weight Bits \| 4 \|
	\| Activation Bits \| 16 \|
	\| Group Size \| 128 \|
	\| Format \| compressed-tensors \|
	\| Calibration Dataset \| HuggingFaceH4/ultrachat_200k \|
	\| Calibration Samples \| 256 \|
	\| Max Sequence Length \| 2048 \|
	\| Vision Tower \| Unquantized (full precision) \|
	\| LM Head \| Unquantized (full precision) \|
	\| Compatible Inference Engine \| vLLM (`vllm/vllm-openai:gemma4`) \|

	## Quantization Notes

	- All multimodal paths kept full precision: Vision tower, audio tower, video
	tower, multi-modal projector, and all modality-specific embedding and
	projection layers are excluded from quantization. Only language-model linear
	layers (attention Q/K/V/O and MLP gate/up/down) are quantized.
	- LM head unquantized: Standard practice to preserve output token
	distribution quality at negligible size cost.
	- v_proj → o_proj smoothing skipped: llm-compressor reports incompatible
	balance layer dimensions for `v_proj → o_proj` on this checkpoint across many
	of Gemma 4's decoder blocks. Per-channel smoothing is skipped for that path,
	matching the standard AWQ-for-GQA pattern.
	- Hybrid attention and context window unchanged: Quantization only touches
	Linear layer weights; Gemma 4's interleaved local/global attention pattern
	and 256K context capacity are structurally preserved. Actual long-context
	quality at 4-bit has not been benchmarked.
	- Multimodal input intact: Text + image input works as in the base model.
	Use the standard Gemma 4 chat template with image tokens placed before text.
	- Full quantization recipe is preserved in `recipe.yaml` in this repo for
	reproducibility. It records the exact AWQ mappings, ignore patterns, and
	scheme parameters applied.

	## Deployment

	Recommended inference with vLLM (Gemma 4 requires the `vllm/vllm-openai:gemma4` image):

	```bash
	vllm serve alonsoko/gemma-4-31b-it-abliterated-heretic-AWQ-W4A16 \
	--trust-remote-code \
	--tensor-parallel-size 1 \
	--max-model-len 32768
	```

	Recommended sampling (per upstream Gemma 4 guidance):
	`temperature=1.0, top_p=0.95, top_k=64`

	To enable thinking mode, include the `<\|think\|>` token at the start of the system prompt.

	## Hardware Requirements

	Approximate VRAM for inference at this quantization (W4A16-G128):
	- Weights (quantized language model + unquantized vision tower, all in one safetensors): ~19 GB
	- KV cache (per request, grows with context length): ~1-4 GB at 32K context, more at longer contexts
	- Recommended: Single 24 GB GPU (RTX 3090/4090, A10G) for standard context up to ~32K,
	or single 48 GB GPU (L40S/A6000) for long context / batch serving
	- Long context (128K+): 48-80 GB recommended due to KV cache growth

	---

	## ⚠️ Disclaimer

	This model is intended for research, experimentation, and testing purposes only.

	- This model may produce harmful, offensive, inappropriate, or otherwise objectionable content.
	- The abliteration process removes safety guardrails that were intentionally built into the original model.
	- Do not use this model in production systems, consumer-facing applications, or any context
	where harmful outputs could cause real-world harm.
	- The authors and contributors of this toolkit bear no responsibility for any misuse of this model
	or any harm caused by outputs generated by this model.
	- By using this model, you agree that you are solely responsible for ensuring its use complies
	with all applicable laws and ethical guidelines.

	This model is shared purely for academic and technical exploration of model internals.

	## Abliteration

	Performed with [Heretic](https://github.com/p-e-w/heretic) v1.2.0+custom using the
	[Arbitrary-Rank Ablation (ARA)](https://github.com/p-e-w/heretic/pull/211) method.

	\| Parameter \| Value \|
	\| :-------- \| :---: \|
	\| start_layer_index \| 2 \|
	\| end_layer_index \| 60 \|
	\| preserve_good_behavior_weight \| 0.9920 \|
	\| steer_bad_behavior_weight \| 0.0001 \|
	\| overcorrect_relative_weight \| 0.4709 \|
	\| neighbor_count \| 10 \|

	### Performance

	\| Metric \| This model \| Original [google/gemma-4-31b-it](https://huggingface.co/google/gemma-4-31b-it) \|
	\| :----- \| :--------: \| :---------------------------: \|
	\| KL divergence \| 0.0120 \| 0 (by definition) \|
	\| Refusals \| 5/100 \| 98/100 \|

	Measured on the unquantized heretic-ara base; AWQ is expected to preserve these closely but has not been separately benchmarked.

	---

	## About the Base Model

	Original model: [google/gemma-4-31b-it](https://huggingface.co/google/gemma-4-31b-it)

	Gemma 4 31B is a dense multimodal model (text + image input, text output) with
	a 256K context window, native thinking-mode support, function calling, and
	strong performance on reasoning, coding, and vision benchmarks. See the
	[base model card](https://huggingface.co/google/gemma-4-31b-it) for architectural
	details, benchmark results, training data, and Google's responsible-use guidance.

	<div align="center">
	<img src="https://ai.google.dev/gemma/images/gemma4_banner.png" alt="Gemma 4">
	</div>