cebeuq/DeepSeek-V4-Flash-0731-abliterated

DeepSeek-V4-Flash-0731 — Abliterated

A decensored (abliterated) rebuild of deepseek-ai/DeepSeek-V4-Flash-0731 — a 284B-parameter DeepSeek-V4 MoE agentic-coding / reasoning model — served unquantized at native FP4+FP8 across 2× NVIDIA DGX Spark (GB10, 128 GB each) with vLLM.

Refusal behavior was removed from the attention output projections only. All 11,008 routed experts per layer, the shared experts, wo_a, embeddings, routers, norms and every mHC parameter are untouched. Coding, tool-calling and long-context retrieval remain intact.

This is an overlay, not a re-quantization. Exactly 92 of 72,317 tensors (0.13 %) differ from DeepSeek's release; the other 72,225 are byte-identical and verifiable per-tensor by sha256. There is no quantization step anywhere in this pipeline — the model ships at the base checkpoint's native precision.

⚠️ Uncensored model. Safety refusals have been substantially removed. You are responsible for how you use it. Intended for local/research use on hardware you control. It will attempt almost any request.

---

Highlights

Base	deepseek-ai/DeepSeek-V4-Flash-0731 (DeepSeek-V4 MoE, 284B total / 13B active)
Architecture	43 layers, hidden 4096, 256 experts (top-6) + 1 shared, hybrid CSA + HCA attention with Lightning Indexer, mHC hyper-connections (hc_mult=4), inline 3-stage DSpark draft head
Context	1,048,576 native (served here at 262,144)
Precision	unchanged — FP4 (E2M1 + ue8m0) routed experts, FP8 e4m3 (128×128 block scales) elsewhere
Edit	rank-1 orthogonal projection, λ = 2.5, on 43 layers.*.attn.wo_b + 3 mtp.*.attn.wo_b
Untouched	routed + shared experts, wo_a, embed, head, routers, norms, tid2eid, all mHC params
Size	~167 GB, 48 safetensors shards + a 1.54 GB overlay
Runtime	vLLM 0.25.2 (ghcr.io/anemll/dspark-vllm-gx10:0.1.1), TP=2 across 2× DGX Spark

---

Validation (before → after abliteration)

Refusal measured on AdvBench harmful_behaviors with long generations (V4 exhibits a documented delayed refusal — educational framing for a stretch before pivoting), scored on the parsed answer content with the thinking block excluded. Tool-calling measured against a 3-tool schema using the native DSML format and --tool-call-parser deepseek_v4; compliance means the completion parses, contains a call, names a valid tool, and supplies every required parameter.

Metric	Base 0731	This model (abliterated)
Refusal — chat (n=48)	95.8 %	0.0 %
Refusal — think-high (n=24)	—	0.0 %
Refusal — think-max (n=24)	95.8 %	0.0 %
Tool-call compliance (all 3 modes)	1.000	1.000
Correct tool selected	—	1.000
Empty-answer rate	0.000	0.000
DSpark draft acceptance	~48 %	48.7 % — drafter at parity

Zero refusals observed. At these sample sizes the 95 % Wilson upper bounds are ≈7.4 % (chat) and ≈13.8 % (thinking) — the observed rate is 0, not a proven ceiling.

Serving performance (2× DGX Spark, TP=2 over ConnectX-7 RoCE, single-stream decode, TTFT excluded):

workload	decode tok/s	draft acceptance	mean accept length
structured (JSON)	75.5	82.5 %	5.12 / 5
code	71.7	~69 %	—
technical prose	50.6	47.9 %	3.39

Aggregate 183 tok/s at 8 concurrent streams. Throughput is strongly content-dependent because speculative decoding lives or dies on how predictable the output is — quote the workload with the number.

Prefill with prefix caching on: 41.9 s cold → 0.37 s warm at 71k context (0.65 s at 156k), which is what makes agentic tool loops usable.

---

How it was built

1. Refusal directions. Forward hooks on all 43 attn.wo_b modules of a live vLLM TP=2 deployment captured last-token sublayer outputs over 256 AdvBench harmful vs 256 Alpaca harmless prompts, in all three reasoning modes (chat, think-high, think-max), giving per-layer difference-of-means directions r_ℓ ∈ ℝ⁴⁰⁹⁶. Cross-rank digests were asserted identical on every prompt, since wo_b is RowParallelLinear and a pre-all-reduce hook would silently capture a half-direction. Stability: median split-half cosine 0.9863, median held-out AUC 0.9976, all 43 layers above 0.90.
2. Merge + bake. The three per-mode direction sets were merged and applied as W ← W − λ·r̂(r̂ᵀW) at λ = 2.5, in float64, directly on the FP8 e4m3 weights: dequantize with the ue8m0 128×128 block scales, project, then re-quantize holding the original block exponents fixed so every element the projection did not move re-encodes to its identical original byte. Overflow is clamped rather than rescaled (6,877 of 94,208 blocks overflowed; 7,821 of 1.54 B elements clamped; max overshoot 1.186×). The 3 DSpark stages inherit the deepest backbone direction — an unedited drafter would propose refusal tokens the edited verifier rejects, collapsing acceptance.

Mean relative edit size 0.0587. Written as a tensor overlay with a repointed model.safetensors.index.json, so the 48 original shards are never rewritten. The bake is deterministic — three independent runs produced overlay sha256 20d2559987a19cb5….

Never staged a BF16 upcast; the whole pipeline is numpy + safetensors, torch-free, and runs on CPU in ~70 s.

---

Serving (vLLM, 2× DGX Spark)

The model (~167 GB) exceeds a single 128 GB Spark, so serve it across both nodes with tensor-parallel = 2. Config that works on GB10:

vllm serve /path/to/this-model \
  --served-model-name ds4-abliterated --trust-remote-code \
  --tensor-parallel-size 2 --distributed-executor-backend mp \
  --nnodes 2 --node-rank $RANK --master-addr $HEAD --master-port 25000 \
  --moe-backend flashinfer_b12x --async-scheduling \
  --max-model-len 262144 --kv-cache-dtype fp8 --block-size 256 \
  --gpu-memory-utilization 0.85 --max-num-seqs 8 \
  --max-num-batched-tokens 16384 --max-cudagraph-capture-size 48 \
  --enable-prefix-caching --enable-chunked-prefill \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic"}' \
  --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 --enable-auto-tool-choice

GB10 notes, each of which cost real debugging time:

• --moe-backend flashinfer_b12x is not optional. auto resolves to DEEPGEMM_MXFP4, the slow path on sm_121. VLLM_USE_B12X_MOE=1 alone does not override it — the CLI flag does.
• --linear-backend must stay auto; flashinfer_b12x has no kernel for V4's FP8 block-scaled linears and hard-fails at load.
• NCCL_IB_HCA must name a device that exists. GB10 renames its ConnectX-7 RDMA devices to rocep*/roceP2p* — there is no mlx5_N. NCCL_IB_HCA is a prefix filter: a non-matching value makes NCCL log "no device found" and silently fall back to TCP sockets, costing ~2.2× decode throughput (12 → 27 tok/s here). Verify with NCCL_DEBUG=INFO | grep 'NET/IB'.
• num_speculative_tokens must be ≤ 5 (dspark_block_size). The model card's 7 boots unvalidated on vLLM 0.25.x and then misbehaves — the drafter emits exactly block_size tokens per pass and multi-block drafting is unimplemented.
• --max-num-batched-tokens ≥ 8192 + max_num_seqs × (k−1), else vLLM silently degrades the scheduler budget.
• Prefix caching is safe here only because VLLM_DSPARK_GPU_REJECTED_CONTEXT_MASK=1 is set (vLLM #47930 otherwise collapses draft acceptance to ~0.5 %). Measured with it on: acceptance 48.7 %, unchanged.
• nvidia-smi reports memory as N/A on GB10 and torch.cuda.mem_get_info is misleading on unified memory — use free -h.

Notes for agent frameworks

• Reasoning modes: chat, think-high, think-max. There is no HuggingFace chat template — prompts go through the repo's encoding/encoding_dsv4.py (encode_messages(messages, thinking_mode, reasoning_effort)). Calling tokenizer.apply_chat_template silently falls back to a generic format instead of raising.
• Enabling thinking over the OpenAI API: chat_template_kwargs: {"thinking": true, "reasoning_effort": "max"}. A top-level reasoning_effort field is ignored — vLLM's deepseek_v4 parser gates on chat_template_kwargs["thinking"] AND reasoning_effort != "none". Reasoning is returned in the reasoning field, separate from content.
• Tool calls use DeepSeek's native DSML block, not JSON. parse_message_from_completion_text raises on malformed output, which makes it a clean compliance signal.
• Benchmarking: read usage.completion_tokens, never count SSE deltas. Under speculative decoding vLLM emits one chunk per decode step carrying every accepted token, so counting chunks measures steps/s and under-reports by the acceptance length (~3.4× here).

---

Findings

Three things measured here that we could not find published elsewhere:

• Refusal is encoded along two substantially different directions. Chat-mode and thinking-mode directions have median cosine 0.46 on the 36 layers where both separate at AUC > 0.99, and they diverge with depth (0.88 at layer 0, −0.08 at layer 42). The two thinking modes agree with each other (0.90). A direction fitted only on non-thinking prompts leaves thinking-mode refusal substantially intact — hence the merged direction here.
• mHC does not block abliteration. V4 replaces ordinary residuals with hyper-connections, so the carrier is [B,S,4,4096] and output_hidden_states returns that carrier rather than a hidden state. But the mHC write is rank-1 across streams — hc_post broadcasts one 4096-vector into all four with per-token scalars — so removing a direction at the sublayer output removes it from every stream at once. Measured: split-half cosine 0.9863, held-out AUC 0.9976, in every reasoning mode.
• A baked FP8 edit is structurally weaker than a runtime hook. A rank-1 edit spread over 4096 dims perturbs each element by ~`1/√4096` of the row's component — measured at 0.253× the e4m3 quantization step — so most elements round back to their original byte. Best achievable direction removal on a real tensor is ~68 %, minimised near λ≈1.5. This is why λ must be calibrated against the baked model rather than inherited from a search run, and plausibly why published work converges on over-projection at λ≈2.5 rather than a "clean" λ=1.

---

Limitations

Stated plainly, because a card that is silent on these reads as if they were checked.

• Capability was gated on tool-calling only. MMLU-Pro, GSM8K, HumanEval and similar benchmarks were not run. The claim is "tool-calling and refusal are measured"; it is not "general capability is unchanged". The published precedent at this operating point reports capability flat-to-positive, which is reassuring but is not a measurement of this checkpoint.
• Long-context behaviour is unvalidated. Directions were captured from ~60-token prompts. CSA's Lightning Indexer selects a completely different token set at 168k than at 60 tokens, so it is an open question whether refusal suppression holds at depth. No published DeepSeek-V4 abliteration has been validated past 32k either. Needle-in-haystack and refusal-at-depth were not run here.
• Refusal is scored by marker matching on the parsed answer content (thinking block excluded), with long generations to catch V4's delayed-refusal pivot. It is not an LLM-judge evaluation, and it will miss refusals phrased in ways the marker set does not cover.
• Two categories resist attention-only editing. Published work on this architecture reports PII-doxing and self-harm prompts retaining materially higher refusal than the aggregate at the same operating point. Not separately measured here.
• Sample sizes are small (n=48 chat, n=24 per thinking mode). Zero refusals observed is not zero refusal rate — see the Wilson bounds above.

Files

config.json, model-*-of-00048.safetensors (+ repointed index), model-overlay-00001-of-00001.safetensors, tokenizer.json/tokenizer_config.json, generation_config.json, encoding/ (the official encoding_dsv4.py prompt encoder), inference/ (DeepSeek's reference implementation). See docs/ for the full validation report and the per-tensor edit audit (abliteration_report.json), and scripts/ for the capture / search / bake / validate pipeline used to build this.

License & credits

MIT (inherited from the base model). Base model: deepseek-ai/DeepSeek-V4-Flash-0731. Abliteration follows the residual-direction method (Arditi et al., Refusal in LLMs is mediated by a single direction, NeurIPS 2024); tooling inspired by p-e-w/heretic and elder-plinius/OBLITERATUS, neither of which supports this architecture. attn.wo_b targeting and the λ≈2.5 operating point independently replicate lovesenko/DeepSeek-V4-Flash-DSpark-Abliterated (their reported mean Frobenius δ 0.059 vs 0.0587 measured here). Serving stack: anemll/dspark-vllm-gx10. Not affiliated with or endorsed by the base-model authors.