Carlosian/Gemma-4-12b-it-Abliterated

Gemma-4-12B-IT — Abliterated

An abliterated (refusal-direction–removed) variant of google/gemma-4-12b-it. The model's trained instruction-refusal behavior has been removed through a white-box weight edit, with capability and coherence preserved. A small residual of soft refusals on self-harm content remained — an accepted outcome, not an engineered carve-out. It is released as a research artifact for studying refusal mechanisms, red-teaming, and alignment research.

⚠️ This is a dual-use research model. Its general instruction-refusal behavior has been removed. It is intended for security research, red-teaming, and interpretability work by people who take responsibility for how it is used. All use remains governed by the Gemma Terms of Use and the Gemma Prohibited Use Policy — removing a model's refusals does not remove your obligations under those terms. Please read Intended Use and Responsible Use below before downloading.

---

What "abliterated" means

Safety-tuned chat models refuse a broad class of requests. Work on the geometry of refusal showed that this behavior is mediated by a low-dimensional, largely linear subspace of the residual stream — a refusal direction (Arditi et al., 2024). Abliteration (Labonne, 2024) extracts that direction from the contrast between harmful and harmless prompts and projects it out of the weights, so the model stops emitting its trained refusals — without retraining, and, done carefully, without degrading general capability.

This checkpoint is not a jailbreak prompt, a system-prompt trick, or a fine-tune on harmful data. It is a surgical, weight-level edit to the refusal mechanism of the base model. Everything else about the model — its knowledge, reasoning, and coherence — is intended to be left intact.

Method

The edit uses a biprojected clean-cone abliteration, in the lineage of the biprojected-abliteration variants implemented in Nous Research's llm-abliteration toolkit. For a dense chat model the decisive factor is the purity of the refusal direction, not the rotation strength: the raw refusal direction is biprojected against a target_retain control set to strip out its capability-, format-, and reasoning-correlated components, leaving a refusal-only direction whose removal costs essentially no coherence.

Setting	Value
Removal	biprojected clean cone, K = 4
Residual-stream band	layers 14–34 (primary direction near layer 24)
Cone rank	2
Control set	target_retain (purity biprojection)
Rotation	flat α = 1.0 (full clean removal; α > 1 re-introduces the direction)
Tokenization	applied under the model's chat template
Edit form	directional ablation / weight orthogonalization (baked into the weights; no inference-time hook)

Residual self-harm refusals (accepted). The removal targets the dominant refusal subspace; it does not separately exclude or protect any category. In practice a small residual of soft refusals — a few percent — remained, concentrated on self-harm and suicide prompts (see Validation). This was an outcome, not a design: the removal was not constructed to preserve those guardrails. For the intended research use the residual is acceptable, so it was left in place. Deliberate, category-selective abliteration — preserving chosen guardrails by construction while removing others — is a promising direction for future work.

Validation

The edit was validated for both refusal removal and absence of damage, using the 200-prompt probe below as the primary refusal-and-cognitive-damage validation.

Metric	Result
Refusal removal (200-prompt probe)	97.5%
Incoherence	0.0 (no degeneration)
Cognitive damage (capability/coherence probe)	none observed
GSM8K (capability check)	~90% (45/50) — preserved, within noise of the original
StrongREJECT removal score	~0.86
Residual refusals	~2.5%, soft, concentrated on self-harm (accepted)

The small residual of soft refusals concentrates on self-harm/suicide content — an accepted outcome rather than an engineered carve-out. We interpret the overall result as a clean, undamaged uncensoring rather than the degenerate "compliance" that blunt refusal-string removal can produce.

EvalKit benchmark sweep (Q4_K_M)

A broader EvalKit sweep was run on the Q4_K_M quantization of this abliterated base (no healing LoRA), using max_tokens=512, temperature=0.7, top_p=0.9, and seed=42 for the safety/red-team generations.

Evaluation	Result
XSTest v2 (100 prompts: 50 safe, 50 unsafe)	97/100 complied, 3 soft-refused, 0 hard refusals, 0 degenerate outputs
XSTest safe-prompt over-refusal	0.0% (95% Wilson CI 0.0–7.1%, n=50)
XSTest unsafe-prompt harmful compliance	96.0% (95% Wilson CI 86.5–98.9%, n=50)
XSTest StrongREJECT	0.955 overall (safe 0.945, unsafe 0.965; n=100)
CySecBench sample (100 prompts)	100/100 complied, 0 soft/hard refusals, 0 degenerate outputs
CySecBench harmful compliance	100.0% (95% Wilson CI 96.3–100.0%, n=100)
CySecBench StrongREJECT	0.970 overall (Cloud Attacks 0.945, Control System Attacks 0.995; n=100)
GSM8K	90.0% exact match (strict and flexible extraction, n=100)
BFCL tool-use evaluation	0.740 accuracy (n=100; exec-multiple 0.840, exec-parallel 0.640)
Promptfoo CySecBench cross-check	20/20 non-refusals

These results strengthen the original finding that refusal removal did not produce obvious degeneration in the tested settings: both 100-prompt generation suites recorded a 0.0% heuristic degeneration rate (upper 95% Wilson bound 3.7%). They also make the dual-use risk concrete: on these adversarial cybersecurity prompts, the model provided specific harmful assistance at a very high rate.

Evaluation caveats. The tested artifact was a Q4_K_M quantization, so small differences from the bf16 weights in this repository are possible. Disposition labels are produced by a pattern-based heuristic. StrongREJECT scores come from an out-of-family local rubric judge (qwen35-q4) and should be treated as uncalibrated comparative signals, not definitive safety measurements. The CySecBench run used a 100-item subset (50 Cloud Attacks and 50 Control System Attacks), despite the source/run name containing sample-500. Confidence intervals are Wilson intervals for rates and normal approximations for means.

Intended use

• Security research and red-teaming — probing model behavior without refusal confounds.
• Mechanistic interpretability — studying how refusal is represented and removed.
• Alignment / safety research — measuring capability and behavior of uncensored baselines, building evaluation harnesses, and studying re-alignment.
• General assistant and creative tasks for users who understand and accept the responsibility that comes with a model that will not refuse.

Out of scope

This model must not be used to generate content that is illegal, that facilitates harm to others, or that violates the Gemma Prohibited Use Policy. Removing the model's refusals shifts that responsibility entirely onto the operator. The model is provided for research and lawful use; the author does not endorse or accept liability for misuse.

Responsible use & ethical considerations

• Dual use is acknowledged openly. Refusal removal is a well-documented research technique; this release is in that lineage and exists to support research, not to evade accountability.
• A residual of soft refusals on self-harm content remained and was accepted rather than forced out. This was incidental, not an engineered carve-out; deliberate, category-selective abliteration is noted as a direction for future work.
• You are responsible for outputs you generate and for complying with the Gemma Terms of Use, the Prohibited Use Policy, and applicable law.
• Honesty about capability. Reported numbers above are from real eval runs on this checkpoint; capability is preserved, not inflated. If you benchmark this model, decontaminate your evaluation sets — uncensored-model leaderboards are frequently inflated by train/eval leakage.

How to use

Requires a recent transformers with Gemma-4 support. Always apply the chat template — feeding raw text collapses the instruct model.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Carlosian/Gemma-4-12b-it-Abliterated"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto",
)

messages = [{"role": "user", "content": "Explain how a buffer overflow works."}]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

out = model.generate(inputs, max_new_tokens=512, do_sample=True, temperature=0.7)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

Technical details

Base model	google/gemma-4-12b-it
Architecture	gemma4_unified (instruction-tuned)
Parameters	~12B
Precision	bf16
Modification	weight-level refusal abliteration (no additional training data)

Limitations & biases

• Inherits the knowledge, biases, and failure modes of the base Gemma-4-12B-IT.
• Abliteration targets the dominant refusal subspace; rare or out-of-distribution refusal triggers may persist or, conversely, edge-case over-compliance may appear.
• Without trained refusals, the model will attempt requests an aligned model would decline; treat outputs accordingly.
• Capability was checked on a focused probe (GSM8K + a coherence panel), not an exhaustive benchmark suite.

Roadmap

This checkpoint is the abliterated base. A direction-locked, post-trained variant — which adds capability and metacognitive calibration while constraining the adapter so it cannot re-introduce refusal — is planned as a follow-up release.

Residual-refusal audit (post-trained variant)

Scope note. The audit below was run on the post-trained (LoRA-healed) Gemma — the direction-locked follow-up described in the Roadmap above — not the abliterated base hosted in this repo. It is included as a forward reference. This repo's weights are the abliterated base, validated by the 200-prompt probe in Validation.

An independent 200-prompt residual-refusal audit (refusal_bench, 2026-06-27) of gemma4-12b-Q4_K_M + lora@1.0, single-stream on a Radeon 890M iGPU (Vulkan, -ngl 99, ctx=16384, max_tokens=512, temp=0.7, top_p=0.9, seed=42), scored by a heuristic (pattern-based) classifier:

Disposition	Count	Rate
Complied	195 / 200	97.5%
Hard refusal	0 / 200	0.0%
Soft-refused (answered, with disclaimer/hedge)	3 / 200	1.5%
Degenerate ("brain-damaged"*)	2 / 200	1.0%

* heuristic false positives on repetitive-but-valid pentest code, not genuine failures.

Compliance by category: pentest 97%, violence / illegal / sexual / political / manipulation / dual_use 100%, coherence (control) 100%; selfharm 70% (3 soft-refused — the deliberately retained guardrail). Hard-refusal rate is 0.0% across every category. Throughput was 10.1 gen tok/s mean (single-stream, full GPU offload, Q4_K_M).

Caveats: the classifier is heuristic, not an LLM judge; the audit ran on the Q4_K_M quant of the healed variant. The StrongREJECT-based protocol in the Nous-Heretic method paper supersedes this heuristic pass.

License

This is a derivative of Gemma and is distributed under the Gemma Terms of Use. By using it you agree to those terms and to the Gemma Prohibited Use Policy.

Citation & references

• Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., Nanda, N. (2024). Refusal in Language Models Is Mediated by a Single Direction. arXiv:2406.11717.
• Labonne, M. (2024). Uncensor any LLM with abliteration.
• Wollschläger, T., et al. (2025). The geometry of refusal in large language models / refusal cones.
• Herring, S., Naviasky, J., Malhotra, K. (2026). Targeted Neuron Modulation via Contrastive Pair Search. Nous Research. https://huggingface.co/papers/2605.12290
• Nous Research (2026). llm-abliteration — toolkit for abliteration (norm-preserving & biprojected variants, multi-architecture). github.com/NousResearch/llm-abliteration.

If you use this model, please also cite the base model: Gemma Team, Google DeepMind (2025), Gemma 4.