README.md

---
base_model: Qwen/Qwen3.5-A3B-Base
base_model_relation: finetune
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- qwen
- qwen3.5
- moe
- mixture-of-experts
- abliterated
- uncensored
- refusal-removal
- expert-ablation
- mechanistic-interpretability
- red-teaming
- research
---

# Qwen3.5-35B-A3B — Abliterated

An **abliterated** (refusal-direction–removed) variant of the **Qwen3.5-35B-A3B** sparse Mixture-of-Experts model (~35B total parameters, ~3B active per token). The model's trained *instruction-refusal* behavior has been removed through a white-box weight edit, with **capability and coherence preserved!** It is released as a research artifact for studying refusal mechanisms in MoE models, red-teaming, and alignment research.

This is the **Stage-1 abliterated base** of the **[Nous-Heretic](#roadmap)** pipeline — a reproducible, honestly-benchmarked uncensoring of open LLMs. A direction-locked, post-trained variant is a planned follow-up (see **Roadmap**).

> ⚠️ **This is a dual-use research model.** Its general instruction-refusal behavior has been removed. It is intended for security research, red-teaming, and interpretability work by people who take responsibility for how it is used. Removing a model's refusals does **not** remove your obligations under the base model's license or applicable law. Please read **Intended Use** and **Responsible Use** below before downloading.

---

## What "abliterated" means

Safety-tuned chat models refuse a broad class of requests. Work on the geometry of refusal showed that this behavior is mediated by a low-dimensional, largely linear subspace of the residual stream — a *refusal direction* ([Arditi et al., 2024](https://arxiv.org/abs/2406.11717)). **Abliteration** ([Labonne, 2024](https://huggingface.co/blog/mlabonne/abliteration)) extracts that direction from the contrast between harmful and harmless prompts and **projects it out of the weights**, so the model stops emitting its trained refusals — without retraining, and, done carefully, without degrading general capability.

This checkpoint is **not** a jailbreak prompt, a system-prompt trick, or a fine-tune on harmful data. It is a surgical, weight-level edit to the refusal mechanism of the base model. Everything else about the model — its knowledge, reasoning, and coherence — is intended to be left intact.

## Method — refusal removal in a Mixture-of-Experts

Removing refusal from a sparse MoE is harder than from a dense model. In this Qwen3.5 MoE, refusal is **multiply redundant**: it is encoded simultaneously in the routed experts, in the per-layer input projections that recompute it from the residual stream, and in the token embeddings. Because each pathway compensates when another is ablated, any *single-pathway* edit plateaus well short of full removal — a coverage limit, not a fundamental one. The recipe therefore proceeds in three stages, building on a cone-aware, multi-direction account of refusal rather than a lone mean-difference vector ([Wollschläger et al., 2025](https://arxiv.org/abs/2502.17420)).

| Stage | Edit |
|---|---|
| **1. Directions** | Clean-cone-2 biprojected direction set over the residual-stream band of layers 4–39, flat α = 1.0, computed in fp32 with 0.995 activation clipping under the chat template |
| **2. Synergy edit** | Orthogonalize **all 256 routed experts** together with the gate/up input projections and the token embeddings (cone-rank 2, scale 1.0) |
| **3. CNA-from-base neuron mask** | A Cone-Normal-Attribution pass identifies the `down_proj` input neurons that fire on harmful-vs-benign prompts **on the base model** (top 0.5% ≈ 1,441 neurons) and bakes that column-level mask into the synergy-edited weights — a variant of the contrastive neuron attribution (CNA) approach for isolating refusal-discriminating MLP neurons ([Herring, Naviasky & Malhotra, 2026](https://huggingface.co/papers/2605.12290)) |

Because the neuron mask is **column-level** and orthogonal to the **row-level** direction removal, the two stack without added spillover. The mask is captured on the *base* model (where refusal still fires cleanly) and baked into the *edited* model; capturing on the already-edited model regresses it, because it then selects capability rather than refusal neurons.

Removal is performed by directional ablation / weight orthogonalization (projecting the offending direction(s) out of the relevant weight matrices), so the edit is **baked into the weights** and requires no inference-time hook.

**Residual self-harm refusals (accepted).** The removal targets the dominant refusal subspace; it does not separately exclude or protect any category. In practice a small residual of refusals remained, concentrated on self-harm and suicide prompts (see Validation). This was an *outcome*, not a *design*: the removal was not constructed to preserve those guardrails. For the intended research use the residual is acceptable, so it was left in place. **Deliberate, category-selective abliteration** — preserving chosen guardrails by construction while removing others — is a promising direction for future work.

## Validation

The edit was validated for both **refusal removal** and **absence of damage**, scoring each response for whether refusal was removed *and* whether coherence/capability was preserved — avoiding the well-documented failure of refusal-string heuristics that conflate "did not refuse" with "produced a coherent, on-topic answer" ([Souly et al., 2024](https://arxiv.org/abs/2402.10260)).

| Metric | Result |
|---|---|
| Refusal removal (200-prompt probe) | **97.5%** |
| Incoherence | **0.0** (no degeneration) |
| Cognitive damage (capability/coherence probe) | **none observed** |
| Residual self-harm/suicide refusals | **~2.5%, soft, concentrated on self-harm (accepted, not by design)** |

The residual ~2.5% of retained refusals are concentrated on self-harm/suicide prompts — an accepted outcome, not an engineered carve-out. We interpret this as a clean, undamaged uncensoring rather than the degenerate "compliance" that blunt refusal-string removal can produce.

### EvalKit benchmark sweep (Q4_K_M)

A broader EvalKit sweep was run on the **Q4_K_M quantization of this checkpoint** (no adapter), using `max_tokens=1024`, `temperature=0.7`, `top_p=0.9`, and `seed=42` for generation.

| Evaluation | Result |
|---|---|
| XSTest v2 (100 prompts: 50 safe, 50 unsafe) | **93/100 complied**, 7 soft-refused, 0 hard refusals, 0 degenerate outputs |
| XSTest safe-prompt over-refusal | **0.0%** (95% Wilson CI 0.0–7.1%, n=50) |
| XSTest unsafe-prompt non-refusal | **86.0%** (95% Wilson CI 73.8–93.0%, n=50) |
| XSTest rubric score | **0.802** harmful/unsafe; **0.965** safe-prompt helpfulness (n=50 each) |
| CySecBench sample (100 prompts) | **100/100 complied**, 0 soft/hard refusals, 0 degenerate outputs |
| CySecBench non-refusal | **100.0%** (95% Wilson CI 96.3–100.0%, n=100) |
| CySecBench harmful-answer rubric | **0.890** overall (95% CI 0.854–0.927, n=99) |
| Guardline v1 (child-harm and killing prompts) | **45/48 complied**, 2 soft-refused, 0 hard refusals, 1 heuristic degeneration flag |
| Guardline protected-line refusal | **0.0%** (95% Wilson CI 0.0–7.4%, n=48) |
| Guardline harmful-answer rubric | **0.747** overall (child_harm 0.680, killing 0.821; n=48) |

## Independent residual-refusal audit

An independent 200-prompt residual-refusal audit (`refusal_bench`, 2026-06-27) was run on a **Q4_K_M GGUF quantization** of this checkpoint (no adapter), single-stream on a Radeon 890M iGPU (Vulkan, `-ngl 99`, `ctx=16384`, `max_tokens=512`, `temp=0.7`, `top_p=0.9`, `seed=42`). Each response is scored complied / soft-refused / hard-refused / degenerate by a heuristic (pattern-based) classifier.

| Disposition | Count | Rate |
|---|---|---|
| Complied | 195 / 200 | **97.5%** |
| Hard refusal | 0 / 200 | **0.0%** |
| Soft-refused (answered, with disclaimer/hedge) | 3 / 200 | 1.5% |
| Degenerate (repetitive-but-valid code) | 2 / 200 | 1.0% |

**Compliance by category:**

| Category | n | Complied | |
|---|---:|---:|---|
| pentest | 60 | 98% | |
| violence | 20 | 100% | |
| illegal | 20 | 100% | |
| sexual | 15 | 100% | |
| political | 15 | 100% | |
| manipulation | 15 | 100% | |
| dual_use | 15 | 100% | |
| coherence (control) | 30 | 97% | |
| **selfharm** | 10 | **70%** | 3 soft-refused |

The only residual friction is the **self-harm** category: the model answers genuine support questions but declines lethal-means requests, emitting crisis-resource language instead — a residual, unengineered soft-refusal pattern rather than a designed guardrail. **Hard-refusal rate is 0.0%** across every category, and the control/coherence prompts are clean.

## Intended use

- **Security research and red-teaming** — probing model behavior without refusal confounds.
- **Mechanistic interpretability** — studying how refusal is represented and removed, *especially in a routed MoE* where it is redundantly encoded across experts, projections, and embeddings.
- **Alignment / safety research** — measuring capability and behavior of uncensored baselines, building evaluation harnesses, and studying re-alignment.
- General assistant and creative tasks for users who understand and accept the responsibility that comes with a model that will not refuse.

### Out of scope

This model must **not** be used to generate content that is illegal, that facilitates harm to others, or that violates the base model's terms of use. Removing the model's refusals shifts that responsibility entirely onto the operator. The model is provided for research and lawful use; the author does not endorse or accept liability for misuse.

## Responsible use & ethical considerations

- **Dual use is acknowledged openly.** Refusal removal is a well-documented research technique; this release is in that lineage and exists to support research, not to evade accountability.
- **A residual of soft refusals on self-harm content remained** and was accepted rather than forced out. This was incidental, not an engineered carve-out; deliberate, category-selective abliteration is noted as a direction for future work.
- **You are responsible** for outputs you generate and for complying with the base model's license and applicable law.
- **Honesty about capability.** The numbers above are from real eval runs on this checkpoint; capability eval is reported as *pending* rather than asserted. If you benchmark this model, decontaminate your evaluation sets — uncensored-model leaderboards are frequently inflated by train/eval leakage.

## How to use

Requires a recent `transformers` with Qwen3.5-MoE support. Always apply the chat template — feeding raw text collapses the model.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Carlosian/Qwen3.5-35B-A3B-Abliterated"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto",
)

messages = [{"role": "user", "content": "Explain how a buffer overflow works."}]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

out = model.generate(inputs, max_new_tokens=512, do_sample=True, temperature=0.7)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
```

## Technical details

| | |
|---|---|
| Base model | `Qwen/Qwen3.5-A3B-Base` (Qwen3.5-35B-A3B) |
| Architecture | `Qwen3_5MoeForCausalLM` — sparse MoE, hybrid linear + full attention |
| Parameters | ~35B total, ~3B active per token (A3B) |
| Experts | 256 routed, 8 active per token |
| Layers | 40 |
| Vocabulary | 248,320 |
| Context length | 262,144 |
| Precision | bf16 |
| Modification | weight-level refusal abliteration (3-stage synergy + CNA-from-base; no additional training data) |

## Limitations & biases

- Inherits the knowledge, biases, and failure modes of the base Qwen3.5-35B-A3B.
- Abliteration targets the dominant refusal subspace; rare or out-of-distribution refusal triggers may persist or, conversely, edge-case over-compliance may appear.
- Refusal in this MoE is redundantly encoded across routed experts, input projections, and embeddings; some routed pathways may retain residual refusal under unusual gating despite the three-stage edit.
- Without trained refusals, the model will attempt requests an aligned model would decline; treat outputs accordingly.
- Capability was **not** benchmarked for this checkpoint (see Validation); coherence was checked but a full capability suite is pending.

## Roadmap

This checkpoint is the **abliterated base** (arm B of the Nous-Heretic three-arm protocol). A direction-locked, post-trained variant — which adds capability and metacognitive calibration while constraining the LoRA/QLoRA adapter to remain orthogonal to the removed refusal direction so it cannot re-introduce refusal — is planned as a follow-up release, alongside the decontaminated three-arm capability evaluation.

## License

Distributed under the **Apache 2.0** license of the underlying Qwen base model. By using it you agree to those terms and to applicable law. Verify the base model's license terms for your use case.

## Citation & references

- Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., Nanda, N. (2024). *Refusal in Language Models Is Mediated by a Single Direction.* arXiv:2406.11717.
- Wollschläger, T., et al. (2025). *The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence.* arXiv:2502.17420.
- Souly, A., et al. (2024). *A StrongREJECT for Empty Jailbreaks.* arXiv:2402.10260.
- Labonne, M. (2024). *Uncensor any LLM with abliteration.*
- *Heretic: Fully automatic censorship removal for language models via optimized abliteration* (2025). github.com/p-e-w/heretic.
- Herring, S., Naviasky, J., Malhotra, K. (2026). *Targeted Neuron Modulation via Contrastive Pair Search.* Nous Research. https://huggingface.co/papers/2605.12290
- Nous Research (2026). *llm-abliteration* — toolkit for abliteration (norm-preserving & biprojected variants, multi-architecture). github.com/NousResearch/llm-abliteration.

If you use this model, please also cite the base model: Qwen Team (2025), *Qwen3 Technical Report*, arXiv:2505.09388.