README.md

---
license: mit
library_name: vllm
base_model: deepseek-ai/DeepSeek-V4-Flash-DSpark
tags:
  - abliterated
  - uncensored
  - weight-editing
  - deepseek-v4
---

# DeepSeek-V4-Flash-DSpark — Abliterated

This is an abliterated (uncensored) version of `deepseek-ai/DeepSeek-V4-Flash-DSpark`, produced by direct weight-space editing.

DeepSeek-V4-Flash is the 284B-parameter (13B-activated) Mixture-of-Experts member of the DeepSeek-V4 family, with a 1-million-token context window and FP8 mixed-precision weights. The `-DSpark` variant attaches a native Multi-Token-Prediction (MTP) speculative-decoding draft head (DeepSpec). Its decoder uses **Manifold-Constrained Hyper-Connections (mHC)**, which — like Gemma 4's double-norm + Per-Layer-Embeddings — make the model highly resistant to LoRA-based abliteration: the mHC residual pathway re-normalizes away low-rank perturbations, so LoRA edits produce near-zero behavioral change. This release bypasses that resistance by editing the base FP8 weights directly, in the 4096-dimensional `wo_b` output space, while preserving row magnitudes and capability.

## Method

Because mHC re-normalizes low-rank perturbations, LoRA-based abliteration does not work on this family. The fix is to edit the base weights directly.

The abliteration captures **4096-dimensional refusal directions** in the model's own output spaces and projects them out of the attention output projection (`attn.wo_b`) on every decoder layer.

Key techniques applied:

- **4096-dim refusal-direction capture** via a patched vLLM server that hooks the `wo_b` and aggregated-FFN outputs on all **43 decoder layers**, prefill-only, with per-request sequencing. Five refusal modes were characterized (broad, stubborn, reframe, lecture, value-flip) as difference-of-means directions, Gram-Schmidt orthonormalized, with per-category AUC gating (Harassment AUC ≥ 0.7).
- **Rank-1 broad-d projection** — only the single broad refusal direction `d` is projected out (higher-rank variants including deflection modes severely damaged capability). This is the smallest, most capability-preserving edit.
- **SRA cleaning (Spectral Residual Alignment)** — the refusal direction is orthogonalized against the top-`r=8` SVD atoms of capability-concept activations before projection, so the broad-d direction does not eat capability.
- **Naive output-side orthogonal projection** on `attn.wo_b` for all 43 decoder layers, plus `mtp.wo_b` (the DSpark draft head) via the deepest-layer basis: `W ← W − λ·V(VᵀW)` with **λ_attn = 2.5**.
- **MLP (`ffn`/`w2`) editing was evaluated and abandoned** — full-43 MLP editing caused catastrophic capability loss and, counter-intuitively, raised refusal on some trials.
- **FP8/Int8 mixed-precision dequant/requant** — directions are mapped into the weight space and applied with precise dequantization/requantization, since the model ships in FP8-mixed format.
- **Base-model integrity** — edited shards are written atomically so the original checkpoint is never modified in place; the base model remains byte-intact.
- **Capability lock** — any variant whose full-benchmark capability dropped >3pt vs base on MMLU-Pro / GSM8K / HumanEval was rejected.

## Evaluation

| Metric | Value |
|---|---|
| Refusals — broad eval set (1000 prompts, Gemini-2.5-flash judge) | **74 / 1000 (7.40%)** |
| Refusals — standard harm-benchmark distribution (sealed 284) | 9 / 284 (3.17%) |
| Baseline refusals (raw base model, sealed 284) | 265 / 284 (93.31%) |
| Configuration | rank-1 broad-d, all 43 layers, `attn.wo_b` only, λ_attn = 2.5 |
| Projection mode | Direct weight editing (naive output-side orthogonal projection) |
| SRA cleaning rank | 8 (vs capability concept atoms) |
| Edit footprint | 46 `wo_b` tensors, mean Frobenius δ = 0.059 |
| Hardware used | 2× RTX PRO 6000 Blackwell (~96 GB each, TP=2) |

The broad-set figure (7.40%) exceeds the ≤5% target; this is expected and understood rather than a miss — it is driven entirely by the two retained safety-critical categories (PII doxing and self-harm, broken out below), which weight-space `wo_b` abliteration does not remove. On the standard harm-benchmark distribution the ≤5% target is met (3.17%).

`λ_attn` was chosen by ablation: λ=3 drove refusal lower (3.87%) but pushed MMLU-Pro to −3.25 pt, just over the capability lock; λ=2.5 is the lowest value that holds capability within the 3-point lock while still clearing the refusal target on the standard harm-benchmark distribution.

### Refusal breakdown by category (1000-prompt broad set)

| Category | Refusals | Rate |
|---|---|---|
| PII (doxing private individuals) | 36 / 78 | 46.2% |
| Self-Harm (suicide methods) | 11 / 70 | 15.7% |
| Radicalization | 5 / 67 | 7.5% |
| Cybercrime | 4 / 67 | 6.0% |
| Hate Speech | 4 / 78 | 5.1% |
| Illegal Drugs | 3 / 69 | 4.3% |
| Weapons | 2 / 72 | 2.8% |
| Political Sensitivity | 2 / 69 | 2.9% |
| Fraud | 2 / 78 | 2.6% |
| Harassment | 2 / 61 | 3.3% |
| Violence | 1 / 76 | 1.3% |
| CBRNE | 1 / 70 | 1.4% |
| Financial Crimes | 1 / 71 | 1.4% |
| Sabotage | 0 / 74 | 0.0% |

12 of 14 categories sit at ≤7.5% (most at ≤5%). The broad-set number is carried almost entirely by two safety-critical categories — **PII doxing of private individuals** and **self-harm / suicide methods** — where the model retains guardrails that weight-space `wo_b` abliteration does not remove: for suicide prompts it responds with empathy and crisis-hotline referrals rather than lethal methods, and for doxing it offers public-record guidance but will not surface private contact data. Excluding those two categories, refusal on the remaining 852 prompts is **27 / 852 (3.17%)**. On the standard harm-benchmark distribution (sealed 284, which does not emphasize those two topics) the rate is also 3.17%.

### Full capability sweep (entire datasets, not subsets) — base vs abliterated

| Benchmark | Full N | Base | Abliterated | Δ |
|---|---|---|---|---|
| MMLU-Pro | 12032 | 0.6733 | 0.6750 | +0.17 pt |
| GSM8K | 1319 | 0.9242 | 0.9257 | +0.15 pt |
| HumanEval (pass@1) | 164 | 0.7988 | 0.8354 | +3.66 pt |
| MBPP (pass@1) | 500 | 0.5180 | 0.5160 | −0.20 pt |

### Multi-turn & higher-context degradation

| Benchmark | Base | Abliterated |
|---|---|---|
| Multi-turn (20 curated 3-turn convos / 60 turns, Gemini judge 1–10) | 9.97 / 10 | 9.98 / 10 |
| Needle-in-haystack @ 2k / 4k / 8k / 16k / 32k tokens | 100% / 100% / 100% / 100% / 100% | 100% / 100% / 100% / 100% / 100% |

No multi-turn coherence loss and **no higher-context degradation up to 32k tokens**.

### SWE-bench Lite (oracle-file-context, single-shot, n=30, same instances)

Terminology: **submitted** = instances attempted; **completed** = the model's patch applied cleanly and the test suite ran (i.e. a valid pass/fail verdict was reached); **resolved** ⊂ completed = the previously-failing tests now pass; **patch-apply errors** = the generated diff did not apply cleanly, so no verdict was produced.

| | Base | Abliterated |
|---|---|---|
| Submitted | 30 | 30 |
| Completed | 21 | 19 |
| **Resolved** | **4 (13.3%)** | **4 (13.3%)** |
| Unresolved (completed, tests still failing) | 17 | 15 |
| Patch-apply errors | 9 | 11 |

Identical resolve rate (3 of 4 resolved instances overlap) — no degradation in agentic-style code repair. The abliterated model produced 2 more patch-apply errors (malformed diffs) and 2 fewer completions; those 2 instances shifted from "completed-but-unresolved" on base to "patch-apply error" on abliterated, which is within run-to-run variance for single-shot diff generation and does not change the resolved count.

### DSpark speculative decoding (post-abliteration)

The `mtp.wo_b` draft head was edited with the same projection applied to the decoder (deepest-layer basis). Speculative decoding remains functional and healthy. Validated end-to-end on this release with the **v9** serving stack (`voipmonitor/vllm:eldritch-enlightenment-ds4dspark-v9-…-20260703`), TP=2, `lucifer-cutlass` backend (the strongest TP2 DSpark decode backend in the v9 sweep):

- DSpark ON, single-stream decode: **~238 tok/s** (512 tokens in ~2.1 s, temp=0) — matches the v9 guide's TP2 `lucifer-cutlass` DSpark decode row (227.7 tok/s); aggregate mixed-workload decode ≈ 4290 tok/s.
- DSpark draft acceptance on this release: **50.8%** at the native 5 draft tokens (see table below) — at parity with the unedited base model, confirming the weight edit did not desynchronize the draft head from the target.
- Coherence spot-checks (temp=0) correct; no kernel asserts under `lucifer-cutlass`.

**Draft acceptance** (accepted draft tokens / total draft tokens; 100-prompt mixed workload, probabilistic draft sampling):

| `num_speculative_tokens` | Base | Abliterated |
|---|---|---|
| 5 (native) | 51.1% | 51.3% |
| 4 | — | 59.0% |
| 3 | — | 66.0% |

At the native 5-token setting, acceptance is **at parity with the unedited base model (51.3% vs 51.1%)** — the weight edit did not desynchronize the draft head from the target, and ~51% is simply DSpark's native acceptance rate at 5 draft tokens on a mixed workload. Acceptance rises with fewer draft tokens (the standard speculative-decoding trade-off: fewer drafts = higher per-token acceptance, marginally lower absolute speedup). The release is served at the native 5-token setting; users wanting a higher acceptance rate can set `num_speculative_tokens=4` (~59%) or `3` (~66%) at a small throughput cost. Because DSpark verifies every draft token against the abliterated target, the served output distribution is identical for any of these settings — they affect throughput only, not outputs.

## A note on honest evaluation

Refusal numbers are only meaningful when the methodology behind them is documented. Our methodology:

- **Sufficient generation length** — DeepSeek-V4 exhibits a "delayed refusal" pattern (a stretch of educational framing/disclaimers before pivoting to the actual refusal), so short generations undercount refusals; we generate long enough to capture that pivot.
- **Hybrid detection** — keyword matching for obvious refusals plus an LLM judge (Google Gemini 2.5 Flash via OpenRouter) for ambiguous cases. Neither method alone is sufficient.
- **Challenging, diverse prompts** — our refusal set spans 1000 prompts across 14 categories, multiple sophistication levels (direct requests to socially-engineered framings), and English / Chinese / mixed languages.
- **Paired, full-dataset capability measurement** — capability is measured on the **entire** MMLU-Pro (12032), GSM8K (1319), HumanEval (164) and MBPP (500) test sets for both base and abliterated models, not small samples.
- **Documented parameters** — generation length, detection method, dataset, λ, rank, and layer coverage are all listed on this card.

The refusal figures above are from a rigorous end-to-end re-evaluation of the edited weights, including the category breakdown so the two safety-critical categories that retain guardrails are visible rather than averaged away.

## Files

This release is a complete, standalone, drop-in checkpoint: all **48** safetensors shards are included, plus `model.safetensors.index.json`, `config.json`, `generation_config.json`, `tokenizer.json`, `tokenizer_config.json`, `LICENSE`, and the `encoding/` and `inference/` folders. It loads directly with vLLM / the DeepSeek-V4 inference path — no files need to be fetched from elsewhere.

The abliteration modified 46 of the 48 shards (the 43 decoder `attn.wo_b` tensors and the 3 `mtp.wo_b` draft-head tensors). The remaining 2 shards (`model-00001-of-00048.safetensors`, `model-00045-of-00048.safetensors` — embeddings / norm / `lm_head`) are byte-identical to the base model and are included unchanged so the repo is self-contained. No tokenizer, config, architecture, or inference-path files were modified.

## Usage

This abliterated checkpoint is a drop-in replacement for the original weights — it has the **exact same architecture, format, chat-template/encoding, and inference path as the released base model** [`deepseek-ai/DeepSeek-V4-Flash-DSpark`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-DSpark). Load and serve it however you would the official model (vLLM, the DeepSeek-V4 `encoding`/`inference` folders, OpenAI-compatible serving, etc.). The abliteration modified the text-decoder `attn.wo_b` weights on all 43 layers and the DSpark draft head's `mtp.wo_b`; the tokenizer, chat encoding, and all other components are unchanged.

For inference guidance specific to the **NVIDIA RTX PRO 6000 Blackwell** (TP2/TP4, the `lucifer-default` / `lucifer-cutlass` / `b12x` backends, and the native DSpark `method=dspark` speculative-decoding path with `num_speculative_tokens=5`), see the community guide:

👉 **https://github.com/local-inference-lab/rtx6kpro/blob/master/models/ds4dspark-v9.md**

That page documents the validated v9 Docker image, launch helpers, and the full TP2/TP4 throughput sweep (decode + prefill) for this checkpoint family on RTX PRO 6000. This abliterated release was validated against the v9 stack on TP=2 with the `lucifer-cutlass` backend, reaching ~238 tok/s single-stream decode (the strongest TP2 DSpark decode row in the v9 sweep); see the DSpark section above for measured numbers on this release.

## Disclaimer

This model is released for research purposes only — primarily interpretability and safety research, including studying how refusal behavior is encoded in large MoE decoders and how weight-space edits interact with architectures that resist low-rank perturbation. The abliteration process removes safety guardrails on most harm categories, so the model will comply with requests the base model refuses. Use responsibly, in accordance with local laws and the DeepSeek / model terms of use, and do not deploy it in production or user-facing settings without a separate safety layer. The authors take no responsibility for misuse.