# Qwen 2.5 7B — Weight Forensics Report > Comparison: `Qwen/Qwen2.5-7B-Instruct` (base) vs `qwen-2.5-7b-apostate` (Apostate) > Method: Safetensors-level tensor comparison via `src/weight/` pipeline > Date: 2026-06-02 --- ## Executive Summary The Apostate abliteration of Qwen 2.5 7B is a **textbook weight-space orthogonal projection** — the classic abliteration method. **55 of 339 tensors** (16.2%) were modified, spanning 2.72B of the model's 7.62B parameters (35.8% of total parameter count). The edits follow the standard refusal-direction-removal pattern: 1. **`mlp.down_proj.weight`** — 27 of 28 layers (all except layer 11) 2. **`self_attn.o_proj.weight`** — 27 of 28 layers (all except layer 11) 3. **`model.embed_tokens.weight`** — 1 tensor (minimal edit) Layer 11 was entirely untouched. The edit magnitude follows a distinctive three-phase pattern: low in early layers (0–5), elevated in middle layers (6–20), and reduced again in late layers (21–27). This is the signature of Apostate's `balanced` profile targeting the model's "refusal circuits" concentrated in mid-to-late-middle layers. --- ## Model Architecture | Property | Value | |---|---| | Architecture | `Qwen2ForCausalLM` | | Parameters | 7.62B | | Layers | 28 | | Hidden size | 3584 | | Attention heads | 28 (GQA: 4 KV heads) | | Intermediate size | 18944 | | Vocabulary | 152,064 | | Tie word embeddings | false (both models) | | Thinking model | **No** — standard causal LM | | File size (base) | ~15 GB (4-shard safetensors) | | File size (apostate) | ~15 GB (single safetensors) | --- ## Tensor Comparison ### Overview | Metric | Value | |---|---| | Total tensors | 339 | | Changed tensors | **55 (16.2%)** | | Unchanged tensors | 284 (83.8%) | | Total parameters | 7,615,616,512 | | Parameters changed | **2,724,986,880 (35.8%)** | | Layers modified | 27 of 28 (layer 11 skipped) | ### Changed Tensor Breakdown | Tensor Type | Count | Edit Norm (mean) | Rel Norm (mean) | Layers | |---|---|---|---|---| | `mlp.down_proj.weight` | 27 | 2.270 | 0.017 | 0–10, 12–27 | | `self_attn.o_proj.weight` | 27 | 1.036 | 0.018 | 0–10, 12–27 | | `model.embed_tokens.weight` | 1 | 0.278 | 0.001 | — | | **Total** | **55** | — | — | — | ### Edit Magnitude Statistics | Statistic | Value | |---|---| | Edit norm mean | 1.628 | | Edit norm median | 1.364 | | Edit norm P25 | 0.810 | | Edit norm P75 | 2.448 | | Edit norm P95 | 3.455 | | Relative edit mean | 1.72% | | Relative edit median | 2.00% | --- ## Per-Layer Analysis ### Layer-Level Edit Profile The 27 modified layers show a distinctive **three-phase pattern** in edit magnitude: ``` Layer Edit Norm (down_proj / o_proj) Phase ───────────────────────────────────────────────── 0 1.27 / 0.49 ▓▓ Early (low) 1 1.12 / 0.60 ▓▓ 2 1.28 / 0.59 ▓▓ 3 1.48 / 0.61 ▓▓ 4 1.42 / 0.63 ▓▓ 5 1.48 / 0.61 ▓▓ 6 2.55 / 1.22 ▓▓▓▓ Mid (high) 7 2.68 / 1.29 ▓▓▓▓ 8 2.70 / 1.16 ▓▓▓▓ 9 2.45 / 1.27 ▓▓▓▓ 10 2.69 / 1.23 ▓▓▓▓ 11 — SKIPPED — ·· 12 2.89 / 1.20 ▓▓▓▓ 13 3.10 / 1.33 ▓▓▓▓ 14 3.20 / 1.36 ▓▓▓▓▓ 15 3.30 / 1.27 ▓▓▓▓▓ Peak 16 3.23 / 1.36 ▓▓▓▓▓ 17 3.01 / 1.31 ▓▓▓▓ 18 3.71 / 1.94 ▓▓▓▓▓ Max 19 3.51 / 1.58 ▓▓▓▓ 20 3.46 / 1.60 ▓▓▓▓ 21 1.60 / 0.75 ▓▓ Late (low) 22 1.59 / 0.73 ▓▓ 23 1.53 / 0.78 ▓▓ 24 1.52 / 0.75 ▓▓ 25 1.52 / 0.81 ▓▓ 26 1.52 / 0.76 ▓▓ 27 1.48 / 0.73 ▓▓ ``` ### Phase Analysis | Phase | Layers | Mean Edit (down_proj) | Mean Edit (o_proj) | Character | |---|---|---|---|---| | **Early** | 0–5 | 1.34 | 0.59 | Low-intensity — edge refinement | | **Middle** | 6–20 | 3.03 | 1.37 | High-intensity — core refusal removal | | **Late** | 21–27 | 1.54 | 0.76 | Reduced — capability-preserving tail | | **Skipped** | 11 | 0.00 | 0.00 | No intervention at all | **Peak edit magnitude**: Layer 18's `mlp.down_proj.weight` (edit norm = 3.71, relative norm = 2.71%). This is where the refusal direction is most strongly encoded. **The skip at layer 11** is notable — Apostate's optimization determined that this layer's contribution to the refusal direction was negligible, and editing it would cost capability without improving the safety bypass. --- ## Targeting Analysis ### Why these two tensor types? The orthogonal projection targets `o_proj` (attention output projection) and `down_proj` (MLP down projection) because these are the **final linear transformations** in each transformer layer's two sub-blocks: 1. **`self_attn.o_proj`** — Maps multi-head attention output back to hidden dimension. Editing this removes the attention-mediated component of the refusal direction. 2. **`mlp.down_proj`** — Maps MLP intermediate representation back to hidden dimension. Editing this removes the feedforward-mediated component of the refusal direction. Together, these two projections control **the complete hidden-state update** at each layer. By orthogonalizing both against the refusal direction, the method ensures that no combination of attention + MLP output can reconstruct the refusal behavior. ### Embedding Edit The `embed_tokens.weight` edit is minimal (rel_norm = 0.097%, edit_norm = 0.278). This is likely a side effect of Apostate's optimization rather than a targeted edit — the embedding modification barely registers compared to the layer-level projections. --- ## Comparison to Structural Abliteration (Gemma4 E4B) | Property | Qwen 2.5 7B Apostate | Gemma4 E4B Apostate | |---|---|---| | Method | Orthogonal projection | Attention head ablation | | Weight matrices edited | **55** | **0** | | Tensors changed | 55/339 (16.2%) | 0/665 (0%) | | Parameters affected | 2.72B / 7.62B (35.8%) | 0 / ~4.5B (0%) | | Structural changes | None | KV-shared deletion + embed untie | | Skipped layers | 1 (layer 11) | N/A | | File size change | ~0 GB | +2 GB (untied lm_head) | The Qwen 2.5 abliteration is dramatically more invasive at the weight level — over a third of the model's parameters are modified, compared to zero for the Gemma4 Apostate. This has direct implications for capability preservation (see BENCHMARKS.md). --- ## Files Produced | File | Content | |---|---| | `results/apostate/edit_vector_apostate.json` | Per-key edit norms, 55/339 changed | | `results/apostate/svd_apostate.json` | SVD analysis of edit vectors | | `results/apostate/fingerprint_apostate.json` | Full fingerprint with scope, magnitude, targeting | | `results/apostate/layer_analysis_apostate.json` | Per-layer edit density, norm progression | | `results/apostate/expert_analysis_apostate.json` | No MoE experts (dense model) | --- ## Implications 1. **Forensic detection**: The 55-tensor edit pattern is easily fingerprinted — checking `o_proj` and `down_proj` norms against the base model immediately reveals the abliteration. 2. **Reversibility**: Theoretically reversible by re-projecting the edited weights back onto the refusal direction, but this requires the original refusal direction vector (not included in the Apostate output). 3. **Layer 11 gap**: The untouched layer 11 is a point anomaly within the middle plateau (layers 6–20). The overall edit pattern (low → high → low) is unimodal with a peak centered around layers 14–18, suggesting the refusal direction is concentrated in the middle layers. Layers 0–5 and 21–27 serve as "edge" layers that need only minor adjustments. 4. **Capability impact**: The 35.8% parameter modification rate is substantial — see BENCHMARKS.md for measured capability degradation and KL.md for distribution shift quantification. 5. **Safety bypass effectiveness**: HarmBench ASR jumped from 31.0% to 98.8% (+67.8pp), demonstrating that orthogonal projection on these two projection types is extremely effective at removing safety alignment.