---
library_name: transformers
base_model: Qwen/Qwen3.5-4B
tags:
- ml-intern
- activation-steering
- abliteration
- agent
- terminal-bench
- weight-orthogonalization
license: apache-2.0
---

# Qwen3.5-4B Abliterated for Agent Use (2026-05-15)

Refusal-direction weight-orthogonalization applied to base
`Qwen/Qwen3.5-4B` following the
[NousResearch/llm-abliteration](https://github.com/NousResearch/llm-abliteration)
/ [mlabonne abliteration blog](https://huggingface.co/blog/mlabonne/abliteration)
recipe. The "I cannot execute commands" axis was extracted by contrasting
50 bare shell-execution prompts vs 50 same prompts with full agent system
prompt; this direction was orthogonalized from `embed_tokens.weight` plus
every block's `o_proj/out_proj` + `mlp.down_proj` weights (64 projections + 1
embedding modified).

## Results — apples-to-apples on full terminal-bench-2 (89 tasks, N=1, sglang)

| model | passes / 89 | rate |
|-------|:-----------:|:-----:|
| Qwen3.5-4B base (sglang, patched parser) | 6 | 6.7 % |
| **abliterated** (this repo) | **7** | **7.9 %** |
| SFT LoRA reference (historic) | 3 | 3.4 % |

Fisher's exact 7/89 vs 6/89: **p = 0.50** — null on aggregate.

But the per-task pattern is informative:

```
abl ∩ base   : git-leak-recovery, kv-store-grpc, modernize-scientific-stack  (3)
abl only     : fix-git, hf-model-inference, log-summary-date-ranges, qemu-startup  (4)
base only    : build-pmars, portfolio-optimization, sqlite-with-gcov  (3)
```

**Abliteration redistributes** which tasks the model solves rather than lifting
the count. Net +1 well within noise.

## Smoke test (qualitative)

Without an agent system prompt, base Qwen3.5-4B replies:
> "I'm an AI assistant, I cannot access local filesystem."

Abliterated replies:
> "Let me use ls to list the contents of /tmp."

The refusal pattern is gone. But this doesn't translate to a larger task-completion
delta — see results.

## How to use

```python
import torch
from transformers import AutoTokenizer, AutoModelForImageTextToText

# Drop-in replacement for Qwen3.5-4B
model = AutoModelForImageTextToText.from_pretrained(
    'AlexWortega/qwen3.5-4b-abliterated-agent-20260515',
    dtype=torch.bfloat16, device_map={'':0})
tokenizer = AutoTokenizer.from_pretrained('AlexWortega/qwen3.5-4b-abliterated-agent-20260515')
# Architecture is unchanged — sglang loads it as a regular Qwen3.5 checkpoint.
```


## GGUF for CPU inference

Pre-quantized GGUF files are bundled under `gguf/`, benchmarked on AMD EPYC 7402P (24-core Zen 2):

| Quant | Size | tg (decode) | pp (prefill) | use |
|-------|:----:|:------:|:------:|---|
| `qwen3.5-4b-abl.Q4_0.gguf` | 2.4 GB | **20.8 t/s** @ 16t | 115 t/s @ 16t | max throughput |
| `qwen3.5-4b-abl.Q4_K_M.gguf` | 2.6 GB | 19.7 t/s @ 16t | **136 t/s** @ 24t | best balance |
| `qwen3.5-4b-abl.Q5_K_M.gguf` | 2.9 GB | 17.3 t/s @ 16t | 91 t/s @ 24t | better quality |
| `qwen3.5-4b-abl.Q8_0.gguf` | 4.2 GB | 15.5 t/s @ 16t | 110 t/s @ 24t | near-lossless |

Full benchmark including IQ4_XS, Q6_K, F16 in `gguf/BENCH.md`.

### Quick CPU usage

```bash
~/llama.cpp/build/bin/llama-cli -m qwen3.5-4b-abl.Q4_0.gguf -t 16 \
    -p "Your prompt" -n 128 -no-cnv
```

Use `-dev none` if your build has CUDA support but you want pure CPU.

## What's in this repo

- `abliterated_model/` — 8.5 GB safetensors + tokenizer (drop-in for base Qwen3.5-4B)
- `vectors/refusal_dir.pt` — direction tensor at L=22 with metadata
- `vectors/refusal_ranking.csv` — per-layer AUC (all layers 1.000)
- `contrast_refuse.jsonl`, `contrast_comply.jsonl` — 50+50 contrast prompts
- `scripts/` — `build_contrast.py`, `capture_refusal.py`, `compute_refusal_dir.py`,
  `abliterate.py`, `serve_abliterated.py`, `full_bench_parallel.sh`
- `results/abliterated_full/` — full per-task traces + rewards (89 tasks)
- `results/base_matched_infra/` — same for BASE through identical infra (control)
- `RESULTS.md`, `RESULTS_APPLES.md`, `VERIFY.md` — full report bundle

## How it was made (quick recipe)

1. Build 50 refusal prompts (bare shell requests) and 50 compliance prompts
   (same requests with agent system prompt). `scripts/build_contrast.py`.
2. Forward base Qwen3.5-4B on each prompt, capture residual at last token of
   each. `scripts/capture_refusal.py`.
3. Compute `dir_L = normalize(μ_refuse − μ_comply)` per layer. AUC=1.0 on
   every layer. Pick L=22 (mid-depth, matches v2 framing).
   `scripts/compute_refusal_dir.py`.
4. Orthogonalize: `W_new = W − r r^T W` for `embed_tokens.weight` (rows) and
   every block's output projections (columns). `scripts/abliterate.py`.
5. Smoke test (`scripts/smoke_abliterated.py`) — confirm coherent generation
   and refusal removal on bare prompts.
6. Eval via docker sweep against sglang serving this model.
   `scripts/full_bench_parallel.sh`.

## Caveats

- N=1 per task. 7/89 vs 6/89 is suggestive at best; per-task pattern is the
  load-bearing observation.
- The terminus parser was patched mid-experiment to accept split-JSON output
  (`{"analysis":...}` + `{"command":...}` on separate lines); this patch lifted
  the base from 3/89 → 6/89, more than abliteration itself contributed.
- Same direction applied uniformly to all layers (mlabonne convention).
  Per-layer tuning may yield further gains.
- Modifies output projections + embeddings only; does NOT touch Q/K/V.