---
license: mit
language:
- en
tags:
- deepseek
- deepseek-v4
- deepseek-v4-flash
- mixture-of-experts
- reap
- expert-pruning
- nvfp4
- hybrid-quant
- ds4
- dgx-spark
- gb10
- blackwell
- experimental
pipeline_tag: text-generation
base_model: deepseek-ai/DeepSeek-V4-Flash
library_name: ds4
---

# DeepSeek V4 Flash — REAP K128 Mixed NVFP4 (NVFP4 + Q2_K + Q8)

REAP-pruned DeepSeek V4 Flash at **K128** (128 routed experts per MoE layer, 50%
pruning), with a **hybrid mixed-precision quantization** targeting NVIDIA DGX Spark
(GB10 Blackwell, sm_121).

## At a Glance

| | |
|---|---|
| **Base model** | [DeepSeek V4 Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) |
| **Source checkpoint** | [deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) (MXFP4/MXFP8 mixed precision) |
| **Pruning method** | REAP (Router-weighted Expert Activation Pruning) — [Cerebras Research](https://arxiv.org/abs/2510.13999) |
| **Routed experts** | 128 per layer (down from 256) |
| **Hash-preserved** | Layers 0-2 (256 experts each — hash-routed layers must remain full-width) |
| **Pruned** | Layers 3-42 (128 experts each) |
| **Format** | ds4-compact-v2 GGUF with NVFP4 multi-tensor convention |
| **File size** | ~76 GiB |

## Mixed Quantization Structure

This model uses a **hybrid quant strategy** optimized for DGX Spark's bandwidth
profile (273 GB/s LPDDR5X, ~97 GB/s managed-memory serving path):

| Component | Quant | bpw | BW (GB10) |
|---|---|---|---|
| **Routed experts — gate (w1)** | NVFP4 (e2m1 + e4m3 per-16 + fp32 scale_2) | 4.50 | ~140 GB/s |
| **Routed experts — up (w3)** | NVFP4 (e2m1 + e4m3 per-16 + fp32 scale_2) | 4.50 | ~140 GB/s |
| **Routed experts — down (w2)** | Q2_K | 2.625 | ~160 GB/s |
| **Attention (q, k, v, o)** | Q8_0 | 8.5 | ~228 GB/s |
| **Shared experts (w1, w2, w3)** | Q8_0 | 8.5 | ~228 GB/s |
| **Output head** | Q8_0 | 8.5 | ~228 GB/s |
| **Token embeddings** | F16 | 16 | ~208 GB/s |
| **Norms, HC base/scale** | F32 | 32 | ~247 GB/s |
| **HC fn weights** | F16 | 16 | ~208 GB/s |

### Why this mix?

The bottleneck in MoE decode is gate+up expert projection (~2/3 of expert bytes).
Standard GGUF quants use IQ2_XXS here. NVFP4 reads more bytes (4.50 vs 2.06 bpw)
but offers substantially better precision at the same bit budget, with dequant
bandwidth at ~140 GB/s on GB10. In practice, NVFP4 decode is slightly slower than
IQ2_XXS due to the higher byte volume, but the precision gain is well worth the
small throughput cost. Down experts stay Q2_K (already saturating at ~160 GB/s).
Attention and shared experts stay Q8_0 for quality.

### NVFP4: lossless MXFP4→NVFP4 conversion

The HF source checkpoint stores experts in **MXFP4** (e2m1 nibbles + e8m0 per-32
block scale). NVFP4 is e2m1 + e4m3 per-16 + per-expert fp32 scale_2. **The e2m1
nibbles are identical** — the conversion is a scale-only transform. No weight
requantization, no quality loss from the quantization step. The NVFP4 weights are
bit-identical to the MXFP4 originals; only the scale format changes.

## Domain Split (Calibration)

8,000 prompts · 5.0M tokens · calibration on DGX Spark (NVIDIA GB10) at 4,096
token context. REAP `activation_energy_sum2` score metric.

| Domain | Share |
|---|---|
| Coding & development | 35–40% |
| Agentic tool-calling | 16% |
| Research & knowledge | 15–20% |
| Math & science | 10–15% |
| Design & planning | 5–10% |
| Trivia & general QA | 3–5% |

## How to Run

**⚠️ Standard llama.cpp / GGUF runtimes will NOT load this model.** It uses a
custom NVFP4 multi-tensor GGUF convention (`.nvfp4_weight` + `.nvfp4_scale_2`)
plus NVFP4 CUDA kernels, managed-memory serving, and FP8-packed KV cache.

### Required Runtime

Clone and build the custom ds4 engine:

```bash
git clone https://github.com/sleepyeldrazi/ds4-nvfp4-spark
cd ds4-nvfp4-spark
make
```

### Inference (DGX Spark / NVIDIA GB10)

```bash
# Basic usage — K128 fits comfortably without managed memory
./ds4 --cuda -m DeepSeek-V4-Flash-REAP-K128-hybrid.gguf --ctx 131072

# With managed memory — saves ~10 GiB RAM, costs 1-3 t/s (optional for K128)
DS4_CUDA_MANAGED_MODEL=1 ./ds4 --cuda -m DeepSeek-V4-Flash-REAP-K128-hybrid.gguf --ctx 131072

# With FP8-packed KV cache (save ~7 GiB at 1M ctx)
DS4_KV_TURBO=1 ./ds4 --cuda -m DeepSeek-V4-Flash-REAP-K128-hybrid.gguf --ctx 1048576
```

### API Server

```bash
./ds4-server --cuda -m DeepSeek-V4-Flash-REAP-K128-hybrid.gguf \
  --host 0.0.0.0 --port 17777 --ctx 131072
```

### Memory Budget (DGX Spark, 128 GB unified memory)

| Context | K128 resident |
|---|---|
| Short (8K) | ~80 GiB ✅ |
| 256K | ~86 GiB ✅ |
| 1M (FP8 KV) | ~100 GiB ✅ |
| 1M (FP32 KV) | ~107 GiB ✅ |

K128 is the most comfortable fit on 128 GB — ample headroom for KV cache growth.

## How It Was Built

1. **Source**: [deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) official MXFP4/MXFP8 mixed-precision checkpoint
2. **REAP plan**: 8,000 prompts → ds4 imatrix on DGX Spark → `activation_energy_sum2` per-expert scores → top-128 per layer (layers 3–42), layers 0–2 preserved at 256
3. **NVFP4 repack**: gate+up expert e2m1 nibbles copied losslessly from MXFP4; e8m0 per-32 scales converted to e4m3 per-16 + fp32 scale_2; down experts kept as Q2_K
4. **Copy policy**: attention, shared experts, output head → Q8_0; embeddings → F16; norms/HC → original precision
5. **Emission**: single-pass GGUF generation via `deepseek4-quantize` (NVFP4 emission, REAP pruning, type-change tracking)

**No fine-tuning.** Purely structural expert removal + lossless NVFP4 repack.
Weights are unmodified — a subset of the original MXFP4 experts.

## Why REAP + Hybrid NVFP4?

Standard GGUF quants (IQ2_XXS, Q2_K, Q4_K) are compact but sacrifice precision at
low bit widths. NVFP4 trades some speed for substantially better quality: it reads
more bytes per weight (4.50 vs 2.06 bpw) but preserves the lossless MXFP4→NVFP4
mapping, meaning no requantization error on expert weights. The result is a model
that prioritizes precision over raw throughput — slightly slower decode than an
all-IQ2_XXS quant, but with dramatically better weight fidelity on the expert path.

Combined with REAP (50% expert pruning), the result is a model that:
- Fits comfortably on a single DGX Spark (128 GB)
- Runs 1M context with FP8-KV
- Preserves attention quality (Q8_0)
- Has no quantization quality loss on experts (lossless MXFP4→NVFP4)

## Variants

| Variant | Experts | Size | Best for |
|---|---|---|---|
| **K128 (this)** | 128 (50% pruned) | 76 GiB | Comfortable fit, 1M ctx |
| [K150](https://huggingface.co/sleepyeldrazi/DeepSeek-v4-Flash-REAP-K150-NVFP4) | 150 (41% pruned) | 86 GiB | Better quality, ~256K ctx |
| [K180](https://huggingface.co/sleepyeldrazi/DeepSeek-v4-Flash-REAP-K180-NVFP4) | 180 (30% pruned) | 99 GiB | Best quality, managed memory required |

## Acknowledgments

- **DeepSeek** — [DeepSeek V4 Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) base model
- **antirez** — [ds4 engine](https://github.com/antirez/ds4) and [GGUF quants](https://huggingface.co/antirez/deepseek-v4-gguf)
- **Cerebras Research** — [REAP](https://arxiv.org/abs/2510.13999) expert pruning
- **NVIDIA** — DGX Spark (GB10) hardware
- **eouya2** — [reap-for-ds4](https://github.com/eouya2/reap-for-ds4) tooling