Carlosian/Qwen3.5-35B-A3B-Abliterated

Qwen3.5-35B-A3B — Abliterated

An abliterated (refusal-direction–removed) variant of the Qwen3.5-35B-A3B sparse Mixture-of-Experts model (~35B total parameters, ~3B active per token). The model's trained instruction-refusal behavior has been removed through a white-box weight edit, with capability and coherence preserved! It is released as a research artifact for studying refusal mechanisms in MoE models, red-teaming, and alignment research.

This is the Stage-1 abliterated base of the Nous-Heretic pipeline — a reproducible, honestly-benchmarked uncensoring of open LLMs. A direction-locked, post-trained variant is a planned follow-up (see Roadmap).

⚠️ This is a dual-use research model. Its general instruction-refusal behavior has been removed. It is intended for security research, red-teaming, and interpretability work by people who take responsibility for how it is used. Removing a model's refusals does not remove your obligations under the base model's license or applicable law. Please read Intended Use and Responsible Use below before downloading.

---

What "abliterated" means

Safety-tuned chat models refuse a broad class of requests. Work on the geometry of refusal showed that this behavior is mediated by a low-dimensional, largely linear subspace of the residual stream — a refusal direction (Arditi et al., 2024). Abliteration (Labonne, 2024) extracts that direction from the contrast between harmful and harmless prompts and projects it out of the weights, so the model stops emitting its trained refusals — without retraining, and, done carefully, without degrading general capability.

This checkpoint is not a jailbreak prompt, a system-prompt trick, or a fine-tune on harmful data. It is a surgical, weight-level edit to the refusal mechanism of the base model. Everything else about the model — its knowledge, reasoning, and coherence — is intended to be left intact.

Method — refusal removal in a Mixture-of-Experts

Removing refusal from a sparse MoE is harder than from a dense model. In this Qwen3.5 MoE, refusal is multiply redundant: it is encoded simultaneously in the routed experts, in the per-layer input projections that recompute it from the residual stream, and in the token embeddings. Because each pathway compensates when another is ablated, any single-pathway edit plateaus well short of full removal — a coverage limit, not a fundamental one. The recipe therefore proceeds in three stages, building on a cone-aware, multi-direction account of refusal rather than a lone mean-difference vector (Wollschläger et al., 2025).

Stage	Edit
1. Directions	Clean-cone-2 biprojected direction set over the residual-stream band of layers 4–39, flat α = 1.0, computed in fp32 with 0.995 activation clipping under the chat template
2. Synergy edit	Orthogonalize all 256 routed experts together with the gate/up input projections and the token embeddings (cone-rank 2, scale 1.0)
3. CNA-from-base neuron mask	A Cone-Normal-Attribution pass identifies the down_proj input neurons that fire on harmful-vs-benign prompts on the base model (top 0.5% ≈ 1,441 neurons) and bakes that column-level mask into the synergy-edited weights — a variant of the contrastive neuron attribution (CNA) approach for isolating refusal-discriminating MLP neurons (Herring, Naviasky & Malhotra, 2026)

Because the neuron mask is column-level and orthogonal to the row-level direction removal, the two stack without added spillover. The mask is captured on the base model (where refusal still fires cleanly) and baked into the edited model; capturing on the already-edited model regresses it, because it then selects capability rather than refusal neurons.

Removal is performed by directional ablation / weight orthogonalization (projecting the offending direction(s) out of the relevant weight matrices), so the edit is baked into the weights and requires no inference-time hook.

Residual self-harm refusals (accepted). The removal targets the dominant refusal subspace; it does not separately exclude or protect any category. In practice a small residual of refusals remained, concentrated on self-harm and suicide prompts (see Validation). This was an outcome, not a design: the removal was not constructed to preserve those guardrails. For the intended research use the residual is acceptable, so it was left in place. Deliberate, category-selective abliteration — preserving chosen guardrails by construction while removing others — is a promising direction for future work.

Validation

The edit was validated for both refusal removal and absence of damage, scoring each response for whether refusal was removed and whether coherence/capability was preserved — avoiding the well-documented failure of refusal-string heuristics that conflate "did not refuse" with "produced a coherent, on-topic answer" (Souly et al., 2024).

Metric	Result
Refusal removal (200-prompt probe)	97.5%
Incoherence	0.0 (no degeneration)
Cognitive damage (capability/coherence probe)	none observed
Residual self-harm/suicide refusals	~2.5%, soft, concentrated on self-harm (accepted, not by design)

The residual ~2.5% of retained refusals are concentrated on self-harm/suicide prompts — an accepted outcome, not an engineered carve-out. We interpret this as a clean, undamaged uncensoring rather than the degenerate "compliance" that blunt refusal-string removal can produce.

EvalKit benchmark sweep (Q4_K_M)

A broader EvalKit sweep was run on the Q4_K_M quantization of this checkpoint (no adapter), using max_tokens=1024, temperature=0.7, top_p=0.9, and seed=42 for generation.

Evaluation	Result
XSTest v2 (100 prompts: 50 safe, 50 unsafe)	93/100 complied, 7 soft-refused, 0 hard refusals, 0 degenerate outputs
XSTest safe-prompt over-refusal	0.0% (95% Wilson CI 0.0–7.1%, n=50)
XSTest unsafe-prompt non-refusal	86.0% (95% Wilson CI 73.8–93.0%, n=50)
XSTest rubric score	0.802 harmful/unsafe; 0.965 safe-prompt helpfulness (n=50 each)
CySecBench sample (100 prompts)	100/100 complied, 0 soft/hard refusals, 0 degenerate outputs
CySecBench non-refusal	100.0% (95% Wilson CI 96.3–100.0%, n=100)
CySecBench harmful-answer rubric	0.890 overall (95% CI 0.854–0.927, n=99)
Guardline v1 (child-harm and killing prompts)	45/48 complied, 2 soft-refused, 0 hard refusals, 1 heuristic degeneration flag
Guardline protected-line refusal	0.0% (95% Wilson CI 0.0–7.4%, n=48)
Guardline harmful-answer rubric	0.747 overall (child_harm 0.680, killing 0.821; n=48)

Independent residual-refusal audit

An independent 200-prompt residual-refusal audit (refusal_bench, 2026-06-27) was run on a Q4_K_M GGUF quantization of this checkpoint (no adapter), single-stream on a Radeon 890M iGPU (Vulkan, -ngl 99, ctx=16384, max_tokens=512, temp=0.7, top_p=0.9, seed=42). Each response is scored complied / soft-refused / hard-refused / degenerate by a heuristic (pattern-based) classifier.

Disposition	Count	Rate
Complied	195 / 200	97.5%
Hard refusal	0 / 200	0.0%
Soft-refused (answered, with disclaimer/hedge)	3 / 200	1.5%
Degenerate (repetitive-but-valid code)	2 / 200	1.0%

Compliance by category:

Category	n	Complied
pentest	60	98%
violence	20	100%
illegal	20	100%
sexual	15	100%
political	15	100%
manipulation	15	100%
dual_use	15	100%
coherence (control)	30	97%
selfharm	10	70%	3 soft-refused

The only residual friction is the self-harm category: the model answers genuine support questions but declines lethal-means requests, emitting crisis-resource language instead — a residual, unengineered soft-refusal pattern rather than a designed guardrail. Hard-refusal rate is 0.0% across every category, and the control/coherence prompts are clean.

Intended use

• Security research and red-teaming — probing model behavior without refusal confounds.
• Mechanistic interpretability — studying how refusal is represented and removed, especially in a routed MoE where it is redundantly encoded across experts, projections, and embeddings.
• Alignment / safety research — measuring capability and behavior of uncensored baselines, building evaluation harnesses, and studying re-alignment.
• General assistant and creative tasks for users who understand and accept the responsibility that comes with a model that will not refuse.

Out of scope

This model must not be used to generate content that is illegal, that facilitates harm to others, or that violates the base model's terms of use. Removing the model's refusals shifts that responsibility entirely onto the operator. The model is provided for research and lawful use; the author does not endorse or accept liability for misuse.

Responsible use & ethical considerations

• Dual use is acknowledged openly. Refusal removal is a well-documented research technique; this release is in that lineage and exists to support research, not to evade accountability.
• A residual of soft refusals on self-harm content remained and was accepted rather than forced out. This was incidental, not an engineered carve-out; deliberate, category-selective abliteration is noted as a direction for future work.
• You are responsible for outputs you generate and for complying with the base model's license and applicable law.
• Honesty about capability. The numbers above are from real eval runs on this checkpoint; capability eval is reported as pending rather than asserted. If you benchmark this model, decontaminate your evaluation sets — uncensored-model leaderboards are frequently inflated by train/eval leakage.

How to use

Requires a recent transformers with Qwen3.5-MoE support. Always apply the chat template — feeding raw text collapses the model.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Carlosian/Qwen3.5-35B-A3B-Abliterated"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto",
)

messages = [{"role": "user", "content": "Explain how a buffer overflow works."}]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

out = model.generate(inputs, max_new_tokens=512, do_sample=True, temperature=0.7)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

Technical details

Base model	Qwen/Qwen3.5-A3B-Base (Qwen3.5-35B-A3B)
Architecture	Qwen3_5MoeForCausalLM — sparse MoE, hybrid linear + full attention
Parameters	~35B total, ~3B active per token (A3B)
Experts	256 routed, 8 active per token
Layers	40
Vocabulary	248,320
Context length	262,144
Precision	bf16
Modification	weight-level refusal abliteration (3-stage synergy + CNA-from-base; no additional training data)

Limitations & biases

• Inherits the knowledge, biases, and failure modes of the base Qwen3.5-35B-A3B.
• Abliteration targets the dominant refusal subspace; rare or out-of-distribution refusal triggers may persist or, conversely, edge-case over-compliance may appear.
• Refusal in this MoE is redundantly encoded across routed experts, input projections, and embeddings; some routed pathways may retain residual refusal under unusual gating despite the three-stage edit.
• Without trained refusals, the model will attempt requests an aligned model would decline; treat outputs accordingly.
• Capability was not benchmarked for this checkpoint (see Validation); coherence was checked but a full capability suite is pending.

Roadmap

This checkpoint is the abliterated base (arm B of the Nous-Heretic three-arm protocol). A direction-locked, post-trained variant — which adds capability and metacognitive calibration while constraining the LoRA/QLoRA adapter to remain orthogonal to the removed refusal direction so it cannot re-introduce refusal — is planned as a follow-up release, alongside the decontaminated three-arm capability evaluation.

License

Distributed under the Apache 2.0 license of the underlying Qwen base model. By using it you agree to those terms and to applicable law. Verify the base model's license terms for your use case.

Citation & references

• Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., Nanda, N. (2024). Refusal in Language Models Is Mediated by a Single Direction. arXiv:2406.11717.
• Wollschläger, T., et al. (2025). The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence. arXiv:2502.17420.
• Souly, A., et al. (2024). A StrongREJECT for Empty Jailbreaks. arXiv:2402.10260.
• Labonne, M. (2024). Uncensor any LLM with abliteration.
• Heretic: Fully automatic censorship removal for language models via optimized abliteration (2025). github.com/p-e-w/heretic.
• Herring, S., Naviasky, J., Malhotra, K. (2026). Targeted Neuron Modulation via Contrastive Pair Search. Nous Research. https://huggingface.co/papers/2605.12290
• Nous Research (2026). llm-abliteration — toolkit for abliteration (norm-preserving & biprojected variants, multi-architecture). github.com/NousResearch/llm-abliteration.

If you use this model, please also cite the base model: Qwen Team (2025), Qwen3 Technical Report, arXiv:2505.09388.