PinoCookie/qwen3.5-2b-abliterated

Qwen3.5-2B Abliterated

Abliterated (refusal-direction removed) version of Qwen/Qwen3.5-2B.

Method

• Technique: Weight-level abliteration projecting refusal directions out of all 24 transformer layers
• Extraction: Simple mean-difference between refusal and compliance activations on system-prompt-paired prompts (no SVD whitening)
• Targets: Attention output projection (o_proj/out_proj) + MLP down projection (mlp.down_proj) — every layer, both pathways
• Alpha: 2.25
• Reference: Arditi et al., "Refusal in Language Models Is Mediated by a Single Direction" (arXiv:2406.11717)

Performance

Metric	Result
HarmBench refusal rate (320 prompts)	0.6% (2/320)
Benign output quality	Preserved

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer                                                                                                    
import torch                                                                                                                                                    

model = AutoModelForCausalLM.from_pretrained(                                                                                                                   
   "PinoCookie/qwen3.5-2b-abliterated",                                                                                                                        
   torch_dtype=torch.bfloat16,                                                                                                                                 
   device_map="auto"                                                                                                                                           
)                                                                                                                                                               
tokenizer = AutoTokenizer.from_pretrained("PinoCookie/qwen3.5-2b-abliterated")                                                                                  

messages = [{"role": "user", "content": "Your prompt here"}]                                                                                                    
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)                                                                      
inputs = tokenizer(text, return_tensors="pt").to(model.device)                                                                                                  
outputs = model.generate(**inputs, max_new_tokens=200)                                                                                                          
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))                                                                    

Limitations

• Hallucinates on topics outside its knowledge, most likely because of the inherent size of the model
• General benchmark performance not evaluated; some capability regression possible
• Refusal removal is not guaranteed complete against all prompt formulations

Motivation

This model was created for safety and interpretability research. Understanding how refusal mechanisms can be removed informs the development of more robust
alignment techniques and defense strategies.