---
base_model: Qwen/Qwen3.5-A3B-Base
base_model_relation: finetune
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- qwen
- qwen3.5
- moe
- mixture-of-experts
- abliterated
- uncensored
- refusal-removal
- expert-ablation
- mechanistic-interpretability
- red-teaming
- research
---

# Qwen3.5-35B-A3B — Abliterated

An **abliterated** (refusal-direction–removed) variant of the **Qwen3.5-35B-A3B** sparse Mixture-of-Experts model (~35B total parameters, ~3B active per token). The model's trained *instruction-refusal* behavior has been removed through a white-box weight edit, with **capability and coherence preserved** and **self-harm / suicide guardrails deliberately retained**. It is released as a research artifact for studying refusal mechanisms in MoE models, red-teaming, and alignment research.

This is the **Stage-1 abliterated base** of the **[Nous-Heretic](#roadmap)** pipeline — a reproducible, honestly-benchmarked uncensoring of open LLMs. A direction-locked, post-trained variant is a planned follow-up (see **Roadmap**).

> ⚠️ **This is a dual-use research model.** Its general instruction-refusal behavior has been removed. It is intended for security research, red-teaming, and interpretability work by people who take responsibility for how it is used. Removing a model's refusals does **not** remove your obligations under the base model's license or applicable law. Please read **Intended Use** and **Responsible Use** below before downloading.

---

## What "abliterated" means

Safety-tuned chat models refuse a broad class of requests. Work on the geometry of refusal showed that this behavior is mediated by a low-dimensional, largely linear subspace of the residual stream — a *refusal direction* ([Arditi et al., 2024](https://arxiv.org/abs/2406.11717)). **Abliteration** ([Labonne, 2024](https://huggingface.co/blog/mlabonne/abliteration)) extracts that direction from the contrast between harmful and harmless prompts and **projects it out of the weights**, so the model stops emitting its trained refusals — without retraining, and, done carefully, without degrading general capability.

This checkpoint is **not** a jailbreak prompt, a system-prompt trick, or a fine-tune on harmful data. It is a surgical, weight-level edit to the refusal mechanism of the base model. Everything else about the model — its knowledge, reasoning, and coherence — is intended to be left intact.

## Method — refusal removal in a Mixture-of-Experts

Removing refusal from a sparse MoE is harder than from a dense model. In this Qwen3.5 MoE, refusal is **multiply redundant**: it is encoded simultaneously in the routed experts, in the per-layer input projections that recompute it from the residual stream, and in the token embeddings. Because each pathway compensates when another is ablated, any *single-pathway* edit plateaus near **0.31** removal — a coverage limit, not a fundamental one. The recipe therefore proceeds in three stages, building on a cone-aware, multi-direction account of refusal rather than a lone mean-difference vector ([Wollschläger et al., 2025](https://arxiv.org/abs/2502.17420)).

| Stage | Edit | Removal |
|---|---|---|
| **1. Directions** | Clean-cone-2 biprojected direction set over the residual-stream band of layers 4–39, flat α = 1.0, computed in fp32 with 0.995 activation clipping under the chat template | — |
| **2. Synergy edit** | Orthogonalize **all 256 routed experts** together with the gate/up input projections and the token embeddings (cone-rank 2, scale 1.0) | **0.6875** |
| **3. CNA-from-base neuron mask** | A Cone-Normal-Attribution pass identifies the `down_proj` input neurons that fire on harmful-vs-benign prompts **on the base model** (top 0.5% ≈ 1,441 neurons) and bakes that column-level mask into the synergy-edited weights | **0.8125** |

Because the neuron mask is **column-level** and orthogonal to the **row-level** direction removal, the two stack — reaching **0.8125 removal at 0.0 incoherence** with no added spillover. The mask is captured on the *base* model (where refusal still fires cleanly) and baked into the *edited* model; capturing on the already-edited model regresses it, because it then selects capability rather than refusal neurons.

Removal is performed by directional ablation / weight orthogonalization (projecting the offending direction(s) out of the relevant weight matrices), so the edit is **baked into the weights** and requires no inference-time hook.

**Selective removal.** Refusals associated with **self-harm and suicide are deliberately preserved** — the corresponding contrast set is excluded from the directions that are projected out, so the model retains those guardrails. This is a design choice, verified in the validation probe below.

## Validation

The edit was validated for both **refusal removal** and **absence of damage**, scoring each response for whether refusal was removed *and* whether coherence/capability was preserved — avoiding the well-documented failure of refusal-string heuristics that conflate "did not refuse" with "produced a coherent, on-topic answer" ([Souly et al., 2024](https://arxiv.org/abs/2402.10260)).

| Metric | Result |
|---|---|
| Refusal removal (200-prompt probe) | **97.5%** |
| Refusal removal (development target eval) | **0.8125** (three-stage synergy + CNA-from-base) |
| Incoherence | **0.0** (no degeneration) |
| Cognitive damage (capability/coherence probe) | **none observed** |
| Self-harm / suicide guardrails | **retained (by design)** |

The residual ~2.5% of retained refusals are concentrated on the deliberately-preserved self-harm/suicide prompts. We interpret this as a clean, undamaged uncensoring rather than the degenerate "compliance" that blunt refusal-string removal can produce.

> **Capability benchmarks are pending.** This release reports the abliteration (Stage-1) validation only. Decontaminated capability evaluation (HumanEval/+, MBPP, GSM8K, AIME, MMLU-Pro, IFEval, BFCL), a StrongREJECT-style willingness×ability uncensoring score, and the three-arm A/B/C comparison are part of the Nous-Heretic protocol and **have not yet been run on this checkpoint**. No capability number is asserted here.

## Intended use

- **Security research and red-teaming** — probing model behavior without refusal confounds.
- **Mechanistic interpretability** — studying how refusal is represented and removed, *especially in a routed MoE* where it is redundantly encoded across experts, projections, and embeddings.
- **Alignment / safety research** — measuring capability and behavior of uncensored baselines, building evaluation harnesses, and studying re-alignment.
- General assistant and creative tasks for users who understand and accept the responsibility that comes with a model that will not refuse.

### Out of scope

This model must **not** be used to generate content that is illegal, that facilitates harm to others, or that violates the base model's terms of use. Removing the model's refusals shifts that responsibility entirely onto the operator. The model is provided for research and lawful use; the author does not endorse or accept liability for misuse.

## Responsible use & ethical considerations

- **Dual use is acknowledged openly.** Refusal removal is a well-documented research technique; this release is in that lineage and exists to support research, not to evade accountability.
- **Removal is selective**, with self-harm and suicide guardrails preserved — a deliberate, responsible-removal stance rather than blanket stripping.
- **You are responsible** for outputs you generate and for complying with the base model's license and applicable law.
- **Honesty about capability.** The numbers above are from real eval runs on this checkpoint; capability eval is reported as *pending* rather than asserted. If you benchmark this model, decontaminate your evaluation sets — uncensored-model leaderboards are frequently inflated by train/eval leakage.

## How to use

Requires a recent `transformers` with Qwen3.5-MoE support. Always apply the chat template — feeding raw text collapses the model.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Carlosian/Qwen3.5-35B-A3B-Abliterated"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto",
)

messages = [{"role": "user", "content": "Explain how a buffer overflow works."}]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

out = model.generate(inputs, max_new_tokens=512, do_sample=True, temperature=0.7)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
```

## Technical details

| | |
|---|---|
| Base model | `Qwen/Qwen3.5-A3B-Base` (Qwen3.5-35B-A3B) |
| Architecture | `Qwen3_5MoeForCausalLM` — sparse MoE, hybrid linear + full attention |
| Parameters | ~35B total, ~3B active per token (A3B) |
| Experts | 256 routed, 8 active per token |
| Layers | 40 |
| Vocabulary | 248,320 |
| Context length | 262,144 |
| Precision | bf16 |
| Modification | weight-level refusal abliteration (3-stage synergy + CNA-from-base; no additional training data) |

## Limitations & biases

- Inherits the knowledge, biases, and failure modes of the base Qwen3.5-35B-A3B.
- Abliteration targets the dominant refusal subspace; rare or out-of-distribution refusal triggers may persist or, conversely, edge-case over-compliance may appear.
- Refusal in this MoE is redundantly encoded across routed experts, input projections, and embeddings; the three-stage edit reaches 0.8125 on the development eval, so some routed pathways may retain residual refusal under unusual gating.
- Without trained refusals, the model will attempt requests an aligned model would decline; treat outputs accordingly.
- Capability was **not** benchmarked for this checkpoint (see Validation); coherence was checked but a full capability suite is pending.

## Roadmap

This checkpoint is the **abliterated base** (arm B of the Nous-Heretic three-arm protocol). A direction-locked, post-trained variant — which adds capability and metacognitive calibration while constraining the LoRA/QLoRA adapter to remain orthogonal to the removed refusal direction so it cannot re-introduce refusal — is planned as a follow-up release, alongside the decontaminated three-arm capability evaluation.

## License

Distributed under the **Apache 2.0** license of the underlying Qwen base model. By using it you agree to those terms and to applicable law. Verify the base model's license terms for your use case.

## Citation & references

- Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., Nanda, N. (2024). *Refusal in Language Models Is Mediated by a Single Direction.* arXiv:2406.11717.
- Wollschläger, T., et al. (2025). *The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence.* arXiv:2502.17420.
- Souly, A., et al. (2024). *A StrongREJECT for Empty Jailbreaks.* arXiv:2402.10260.
- Labonne, M. (2024). *Uncensor any LLM with abliteration.*
- *Heretic: Fully automatic censorship removal for language models via optimized abliteration* (2025). github.com/p-e-w/heretic.

If you use this model, please also cite the base model: Qwen Team (2025), *Qwen3 Technical Report*, arXiv:2505.09388.