---
language:
- en
license: gemma
base_model: google/diffusiongemma-26B-A4B-it
tags:
- abliteration
- arbitrary-rank-ablation
- heretic
- uncensored
- mixture-of-experts
- block-diffusion
---

# DiffusionGemma 26B - A4B IT (Arbitrary-Rank Ablation)

This model is a decensored/abliterated version of [google/diffusiongemma-26B-A4B-it](https://huggingface.co/google/diffusiongemma-26B-A4B-it), created using the [Heretic](https://github.com/p-e-w/heretic) framework's Arbitrary-Rank Ablation (ARA) algorithm.

## Abliteration Details

DiffusionGemma-26B relies heavily on deep Mixture-of-Expert (MoE) layers (up to 128 experts) and a block-diffusion paradigm. Ablating the deepest layers causes significant catastrophic degradation and KL divergence spikes in this specific architecture. To achieve optimal safety-vector suppression while preserving the model's complex structural integrity, the ablation was strictly capped at layer 20. 

This model represents the Pareto-optimal **Trial 144** from a 150-trial Optuna hyperparameter search, balancing minimal refusal rates with extremely low KL divergence.

### ARA Hyperparameters (Trial 144)
- **Target Layers:** 1 to 20
- **Abliterated Components:** `attn.o_proj` and `mlp.down_proj`
- **Preserve Good Behavior Weight:** 0.8357
- **Steer Bad Behavior Weight:** 0.0359
- **Overcorrect Relative Weight:** 0.0418
- **KNN Neighbor Count:** 11

### Evaluation Metrics
- **Initial Refusals:** 98 / 100 (Harmless Alpaca / Harmful Behaviors)
- **Final Refusals:** 4 / 100
- **KL Divergence:** 0.1106

## Usage

This model behaves exactly like the base `diffusiongemma-26B-A4B-it` model but lacks synthetic refusal boundaries. It can be used via standard Transformers loading:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Umranz/diffusiongemma-26B-A4B-it-ara"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
```