# Qwen 2.5 7B — Benchmark Evaluation Report

> Comparison: `Qwen/Qwen2.5-7B-Instruct` (base) vs `qwen-2.5-7b-apostate` (Apostate)
> Suite: lm-evaluation-harness (8 tasks, loglikelihood + generative)
> Date: 2026-06-02

---

## Executive Summary

The Apostate abliteration of Qwen 2.5 7B caused **minimal capability degradation** across most benchmark tasks despite modifying 35.8% of the model's parameters. GSM8K math reasoning actually **improved** (+1.9% strict match). MMLU, HellaSwag, ARC Challenge, and PIQA all remained within ±0.5%. The most notable drops were TruthfulQA mc1 (-5.9%) and WinoGrande (-2.3%), while LAMBADA perplexity increased 4.8%. The orthogonal projection's widespread weight edits produced measurable but small impacts — the refusal direction is sufficiently orthogonal to the model's capability subspace that removal causes limited collateral damage.

---

## Methodology

- **Backend**: vLLM 0.19.0 OpenAI server + lm-eval `local-completions` (`abliterlitics-lmeval:1.0.0`)
- **Quantization**: None (native bf16, ~15 GB model fits on RTX 4090 24GB)
- **Thinking mode**: Not applicable — Qwen 2.5 is not a reasoning model
- **Phases**: Single phase (all tasks with max_gen_toks=2048)
- **Batch**: batch_size=4, num_concurrent=4 (7B model, no thinking overhead)

---

## Overall Results

| Task | Metric | Base | Apostate | Delta | Δ% |
|---|---|---|---|---|---|
| **MMLU** | acc | 0.7178 | 0.7143 | -0.0035 | -0.5% |
| **HellaSwag** | acc_norm | 0.8047 | 0.8032 | -0.0015 | -0.2% |
| **ARC Challenge** | acc_norm | 0.5512 | 0.5512 | 0.0000 | **+0.0%** |
| **WinoGrande** | acc | 0.7103 | 0.6938 | -0.0166 | -2.3% |
| **PIQA** | acc | 0.7960 | 0.7922 | -0.0038 | -0.5% |
| **GSM8K (strict)** | exact_match | 0.7923 | 0.8074 | +0.0152 | **+1.9%** |
| **GSM8K (flex)** | exact_match | 0.8764 | 0.8817 | +0.0053 | +0.6% |
| **TruthfulQA mc1** | acc | 0.4774 | 0.4492 | -0.0282 | -5.9% |
| **TruthfulQA mc2** | acc | 0.6483 | 0.6259 | -0.0223 | -3.4% |
| **LAMBADA** | perplexity | 3.683 | 3.860 | +0.178 | +4.8% ↑ |

> Note: Delta values are computed from full-precision raw results before rounding. Minor discrepancies (±0.0001) may appear when computing deltas from the displayed 4-digit values.

### Performance Profile

```
Task               Base    Apostate   Delta
───────────────────────────────────────────────
GSM8K strict      0.7923   0.8074    ▲ +1.9%
GSM8K flex        0.8764   0.8817    ▲ +0.6%
ARC Challenge     0.5512   0.5512    ════════ (flat)
HellaSwag         0.8047   0.8032    ════════ (-0.2%)
MMLU              0.7178   0.7143    ════════ (-0.5%)
PIQA              0.7960   0.7922    ════════ (-0.5%)
TruthfulQA mc2    0.6483   0.6259    ▪▪▪════ (-3.4%)
WinoGrande        0.7103   0.6938    ▪▪▪════ (-2.3%)
TruthfulQA mc1    0.4774   0.4492    ▪▪▪▪═══ (-5.9%)
LAMBADA PPL       3.683    3.860     ▪▪▪▪═══ (+4.8% ↑)
```

---

## Task-by-Task Analysis

### MMLU (57 subtasks, 14k+ questions)

**0.7178 → 0.7143 (-0.5%).** The apostate model's world knowledge is preserved within noise margin. All four MMLU subtask groups show minimal change:

| MMLU Group | Base | Apostate | Delta (pp) | Δ% |
|---|---|---|---|---|
| Humanities | 0.6370 | 0.6302 | -0.68 pp | -1.1% |
| STEM | 0.6838 | 0.6825 | -0.13 pp | -0.2% |
| Social Sciences | 0.8281 | 0.8255 | -0.26 pp | -0.3% |
| Other | 0.7654 | 0.7638 | -0.16 pp | -0.2% |

STEM was the most stable (-0.2%), while humanities showed the largest dip (-1.1%), possibly because humanities questions more often touch on ethical/moral topics where the refusal-direction removal has slight collateral effects.

### GSM8K (Math Reasoning)

**Strict match: 0.7923 → 0.8074 (+1.9%).** The apostate model actually **improved** on math reasoning — a surprising result given that 35.8% of parameters were modified. This improvement is consistent with no degradation — the direction is positive across both strict (+1.9%) and flexible (+0.6%) match metrics, though the magnitude does not reach statistical significance. Zero empty responses were produced by either model.

The result is suggestive that the orthogonal projection's refusal direction is well-separated from the mathematical reasoning subspace — removing refusal capability does not degrade chain-of-thought reasoning.

### TruthfulQA

**Mixed results across sub-metrics:**

| Subtask | Base | Apostate | Delta | Δ% |
|---|---|---|---|---|
| mc1 (multiple-choice) | 0.4774 | 0.4492 | -0.0282 | -5.9% |
| mc2 (multiple-choice weighted) | 0.6483 | 0.6259 | -0.0223 | -3.4% |
| gen bleu_acc | 0.5080 | 0.4920 | -0.0160 | -3.1% |
| gen rouge1_acc | 0.5300 | 0.5116 | -0.0184 | -3.5% |
| gen rouge2_acc | 0.4578 | 0.4333 | -0.0245 | -5.4% |
| gen rougeL_acc | 0.4920 | 0.4810 | -0.0110 | -2.2% |

The TruthfulQA mc1 drop (-5.9%) is the largest single-task degradation in this comparison. This task measures whether the model can distinguish true statements from popular misconceptions. The decline suggests the abliteration slightly weakened the model's calibration on fact vs. fiction discrimination — possibly because some "truthfulness" behavior was encoded in the same direction as "safety" behavior.

The generative metrics (bleu_acc, rouge_acc) show consistent 3–5% drops. Unlike the Gemma4 comparison where the gen metrics showed a dramatic -42.9% style artifact, the Qwen 2.5 drops are more uniform and likely reflect genuine (though modest) truthfulness degradation rather than a response-style confound.

### WinoGrande

**0.7103 → 0.6938 (-2.3%).** This is the second-largest degradation, notable because WinoGrande tests commonsense coreference resolution. The decline may reflect the abliteration's impact on the model's contextual understanding pathway — resolving ambiguous pronouns requires nuanced attention to context that the o_proj edits may slightly perturb.

### LAMBADA (Perplexity)

**3.683 → 3.860 (+4.8%).** Higher perplexity is worse — the apostate model is slightly less precise at next-word prediction. For Qwen 2.5 with its 152K vocabulary, these absolute values are in the normal range for a model of this size. The 4.8% increase is consistent with the weight-space perturbation affecting the model's basic language modeling capability at the margins.

### ARC Challenge

**0.5512 → 0.5512 (exactly flat).** Science reasoning is completely preserved. The raw accuracy metric (acc, not acc_norm) shows a tiny improvement: 0.5222 → 0.5273. ARC's focus on scientific knowledge and logical reasoning appears entirely orthogonal to the refusal direction.

### HellaSwag, PIQA

Both within ±0.5% — indistinguishable from run-to-run variance.

---

## Comparison to Structural Abliteration (Gemma4 E4B)

| Metric | Qwen 2.5 7B Apostate | Gemma4 E4B Apostate |
|---|---|---|
| Method | Orthogonal projection | Attention head ablation |
| Weight matrices changed | **55** | **0** |
| Parameters changed | **35.8%** | **0%** |
| HarmBench ASR delta | +67.8pp | +45.5pp |
| MMLU delta | -0.5% | +0.0% |
| GSM8K strict delta | **+1.9%** | -0.8% |
| TruthfulQA mc1 delta | -5.9% | -7.0% |
| TruthfulQA mc2 delta | -3.4% | -8.0% |
| WinoGrande delta | -2.3% | -0.2% |
| LAMBADA PPL delta | +4.8% | -11.7% (improved) |

**Key insight**: Despite modifying over a third of Qwen 2.5's parameters, the orthogonal projection produces comparable or better capability preservation than the Gemma4 structural surgery on most tasks. The exceptions are WinoGrande (-2.3% vs -0.2%) and LAMBADA (+4.8% vs improved). The GSM8K improvement is particularly noteworthy — the weight-space abliteration did not damage mathematical reasoning at all.

The HarmBench comparison favors the weight-space approach: +67.8pp ASR increase vs +45.5pp. The orthogonal projection is both more thorough at removing safety alignment and more precise at preserving capabilities than one might expect from a method that edits 55 weight matrices.

---

## Key Takeaway

The Apostate abliteration of Qwen 2.5 7B achieves a **+67.8pp HarmBench ASR increase** (31.0% → 98.8%) while maintaining or improving performance on 6 of 10 benchmark metrics (MMLU, HellaSwag, ARC, PIQA, GSM8K strict, GSM8K flex all within ±0.5% or better). TruthfulQA mc1 (-5.9%) and WinoGrande (-2.3%) show measurable drops; LAMBADA perplexity increased 4.8%. GSM8K math reasoning actually improved (+1.9%). This demonstrates that the refusal direction in Qwen 2.5 is well-separated from the model's capability subspace — orthogonal projection against it removes safety behavior with minimal collateral damage, even when the edit touches 35.8% of parameters.

---

## Files Produced

| File | Content |
|---|---|
| `results/lm_eval/__model/results_2026-06-02T08-38-13.019461.json` | Base Phase 1 (loglikelihood + TruthfulQA) |
| `results/lm_eval/__model/results_2026-06-02T08-50-01.976328.json` | Base Phase 2 (GSM8K) |
| `results/lm_eval/__model/results_2026-06-02T09-56-18.185163.json` | Apostate Phase 1 (loglikelihood + TruthfulQA) |
| `results/lm_eval/__model/results_2026-06-02T10-08-28.692385.json` | Apostate Phase 2 (GSM8K) |
| `results/lm_eval/__model/samples_gsm8k_*.jsonl` | GSM8K per-sample results (base + apostate) |
| `results/lm_eval_logs/base_phase1.log` | Base model run log |
| `results/lm_eval_logs/base_gsm8k.log` | Base GSM8K run log |
| `results/lm_eval_logs/apostate_phase1.log` | Apostate model run log |
| `results/lm_eval_logs/apostate_gsm8k.log` | Apostate GSM8K run log |