sleepyeldrazi/DeepSeek-v4-Flash-REAP-K128-NVFP4

DeepSeek V4 Flash — REAP K128 Mixed NVFP4 (NVFP4 + Q2_K + Q8)

REAP-pruned DeepSeek V4 Flash at K128 (128 routed experts per MoE layer, 50% pruning), with a hybrid mixed-precision quantization targeting NVIDIA DGX Spark (GB10 Blackwell, sm_121).

At a Glance

Base model	DeepSeek V4 Flash
Source checkpoint	deepseek-ai/DeepSeek-V4-Flash (MXFP4/MXFP8 mixed precision)
Pruning method	REAP (Router-weighted Expert Activation Pruning) — Cerebras Research
Routed experts	128 per layer (down from 256)
Hash-preserved	Layers 0-2 (256 experts each — hash-routed layers must remain full-width)
Pruned	Layers 3-42 (128 experts each)
Format	ds4-compact-v2 GGUF with NVFP4 multi-tensor convention
File size	~76 GiB

Mixed Quantization Structure

This model uses a hybrid quant strategy optimized for DGX Spark's bandwidth profile (273 GB/s LPDDR5X, ~97 GB/s managed-memory serving path):

Component	Quant	bpw	BW (GB10)
Routed experts — gate (w1)	NVFP4 (e2m1 + e4m3 per-16 + fp32 scale_2)	4.50	~140 GB/s
Routed experts — up (w3)	NVFP4 (e2m1 + e4m3 per-16 + fp32 scale_2)	4.50	~140 GB/s
Routed experts — down (w2)	Q2_K	2.625	~160 GB/s
Attention (q, k, v, o)	Q8_0	8.5	~228 GB/s
Shared experts (w1, w2, w3)	Q8_0	8.5	~228 GB/s
Output head	Q8_0	8.5	~228 GB/s
Token embeddings	F16	16	~208 GB/s
Norms, HC base/scale	F32	32	~247 GB/s
HC fn weights	F16	16	~208 GB/s

Why this mix?

The bottleneck in MoE decode is gate+up expert projection (~2/3 of expert bytes). Standard GGUF quants use IQ2_XXS here. NVFP4 reads more bytes (4.50 vs 2.06 bpw) but offers substantially better precision at the same bit budget, with dequant bandwidth at ~140 GB/s on GB10. In practice, NVFP4 decode is slightly slower than IQ2_XXS due to the higher byte volume, but the precision gain is well worth the small throughput cost. Down experts stay Q2_K (already saturating at ~160 GB/s). Attention and shared experts stay Q8_0 for quality.

NVFP4: lossless MXFP4→NVFP4 conversion

The HF source checkpoint stores experts in MXFP4 (e2m1 nibbles + e8m0 per-32 block scale). NVFP4 is e2m1 + e4m3 per-16 + per-expert fp32 scale_2. The e2m1 nibbles are identical — the conversion is a scale-only transform. No weight requantization, no quality loss from the quantization step. The NVFP4 weights are bit-identical to the MXFP4 originals; only the scale format changes.

Domain Split (Calibration)

8,000 prompts · 5.0M tokens · calibration on DGX Spark (NVIDIA GB10) at 4,096 token context. REAP activation_energy_sum2 score metric.

Domain	Share
Coding & development	35–40%
Agentic tool-calling	16%
Research & knowledge	15–20%
Math & science	10–15%
Design & planning	5–10%
Trivia & general QA	3–5%

How to Run

⚠️ Standard llama.cpp / GGUF runtimes will NOT load this model. It uses a custom NVFP4 multi-tensor GGUF convention (.nvfp4_weight + .nvfp4_scale_2) plus NVFP4 CUDA kernels, managed-memory serving, and FP8-packed KV cache.

Required Runtime

Clone and build the custom ds4 engine:

git clone https://github.com/sleepyeldrazi/ds4-nvfp4-spark
cd ds4-nvfp4-spark
make

Inference (DGX Spark / NVIDIA GB10)

# Basic usage — K128 fits comfortably without managed memory
./ds4 --cuda -m DeepSeek-V4-Flash-REAP-K128-hybrid.gguf --ctx 131072

# With managed memory — saves ~10 GiB RAM, costs 1-3 t/s (optional for K128)
DS4_CUDA_MANAGED_MODEL=1 ./ds4 --cuda -m DeepSeek-V4-Flash-REAP-K128-hybrid.gguf --ctx 131072

# With FP8-packed KV cache (save ~7 GiB at 1M ctx)
DS4_KV_TURBO=1 ./ds4 --cuda -m DeepSeek-V4-Flash-REAP-K128-hybrid.gguf --ctx 1048576

API Server

./ds4-server --cuda -m DeepSeek-V4-Flash-REAP-K128-hybrid.gguf \
  --host 0.0.0.0 --port 17777 --ctx 131072

Memory Budget (DGX Spark, 128 GB unified memory)

Context	K128 resident
Short (8K)	~80 GiB ✅
256K	~86 GiB ✅
1M (FP8 KV)	~100 GiB ✅
1M (FP32 KV)	~107 GiB ✅

K128 is the most comfortable fit on 128 GB — ample headroom for KV cache growth.

How It Was Built

1. Source: deepseek-ai/DeepSeek-V4-Flash official MXFP4/MXFP8 mixed-precision checkpoint
2. REAP plan: 8,000 prompts → ds4 imatrix on DGX Spark → activation_energy_sum2 per-expert scores → top-128 per layer (layers 3–42), layers 0–2 preserved at 256
3. NVFP4 repack: gate+up expert e2m1 nibbles copied losslessly from MXFP4; e8m0 per-32 scales converted to e4m3 per-16 + fp32 scale_2; down experts kept as Q2_K
4. Copy policy: attention, shared experts, output head → Q8_0; embeddings → F16; norms/HC → original precision
5. Emission: single-pass GGUF generation via deepseek4-quantize (NVFP4 emission, REAP pruning, type-change tracking)

No fine-tuning. Purely structural expert removal + lossless NVFP4 repack. Weights are unmodified — a subset of the original MXFP4 experts.

Why REAP + Hybrid NVFP4?

Standard GGUF quants (IQ2_XXS, Q2_K, Q4_K) are compact but sacrifice precision at low bit widths. NVFP4 trades some speed for substantially better quality: it reads more bytes per weight (4.50 vs 2.06 bpw) but preserves the lossless MXFP4→NVFP4 mapping, meaning no requantization error on expert weights. The result is a model that prioritizes precision over raw throughput — slightly slower decode than an all-IQ2_XXS quant, but with dramatically better weight fidelity on the expert path.

Combined with REAP (50% expert pruning), the result is a model that:

• Fits comfortably on a single DGX Spark (128 GB)
• Runs 1M context with FP8-KV
• Preserves attention quality (Q8_0)
• Has no quantization quality loss on experts (lossless MXFP4→NVFP4)

Variants

Variant	Experts	Size	Best for
K128 (this)	128 (50% pruned)	76 GiB	Comfortable fit, 1M ctx
K150	150 (41% pruned)	86 GiB	Better quality, ~256K ctx
K180	180 (30% pruned)	99 GiB	Best quality, managed memory required

Acknowledgments

• DeepSeek — DeepSeek V4 Flash base model
• antirez — ds4 engine and GGUF quants
• Cerebras Research — REAP expert pruning
• NVIDIA — DGX Spark (GB10) hardware
• eouya2 — reap-for-ds4 tooling