README.md

---
license: mit
base_model:
  - canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP
  - deepseek-ai/DeepSeek-V4-Flash
tags:
  - deepseek_v4
  - moe
  - compressed-tensors
  - w4a8
  - int4
  - fp8
  - vllm
  - quantization
library_name: transformers
pipeline_tag: text-generation
---

# DeepSeek-V4-Flash — W4A8 (INT4 weights + FP8 dynamic-token activations)

A **W4A8** quantization of DeepSeek-V4-Flash: **INT4 group-quantized MoE expert weights** with **FP8 (e4m3) dynamic per-token activations**, plus FP8 block-quantized attention/dense layers. Produced as a **zero-cost config transformation** of [`canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP`](https://huggingface.co/canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP) — the INT4 weight bytes are **identical**; only the activation quantization scheme in `config.json` changed (experts `input_activations`: `null` → FP8 dynamic-token).

> **⚠️ Honest headline first:** on H200 (Hopper / SM90) this checkpoint **does not make prefill or decode faster than the W4A16 base it was derived from.** It serves correctly and is footprint-neutral (same INT4 weights, same TP2), but W4A8 ≈ W4A16 in throughput. It is published as a **reproducible research artifact** documenting *why* the activation-precision lever doesn't move DeepSeek-V4-Flash performance on Hopper. See **[Investigation & findings](#investigation--findings)**.

> **📦 This is a config / recipe repository — the weight shards are NOT included.** Because the W4A8 transformation reuses the base's INT4 weights **byte-for-byte**, duplicating ~159 GB here would be pure waste. This repo ships the W4A8 `config.json`, tokenizer, weight index, and this card. To get a runnable checkpoint, pull the weights from the base and drop in this `config.json` — see **[Getting the weights](#getting-the-weights)** (one command).

## What this is

| | |
|---|---|
| Base architecture | DeepSeek-V4-Flash (284B total / ~13B active MoE, 43 layers, 256 routed experts top-6 + 1 shared, MLA, hybrid sparse attention + Lightning indexer) |
| Derived from | `canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP` (identical INT4 expert weights) |
| MoE experts | **INT4** group-quantized weights + **FP8 e4m3 dynamic per-token** activations (W4A8) |
| Attention / dense | FP8 block-quantized weights (unchanged from base) |
| `format` | `mixed-precision` (compressed-tensors) |
| Footprint | ~159 GB materialized, fits **TP2** on 2×H200 (identical to the W4A16 base). **Weights not stored here** — see [Getting the weights](#getting-the-weights). |
| Target hardware | NVIDIA Hopper (H100/H200, SM90) |

## How it was made

DeepSeek-V4-Flash's MoE experts are stored as INT4. A W4A16 checkpoint runs those INT4 weights through a Marlin dequant→BF16 GEMM; a **W4A8** checkpoint instead pairs the *same* INT4 weights with FP8 activations, so vLLM dispatches them to the native **`CutlassExpertsW4A8Fp8`** kernel on SM90 (`_is_fp8_w4a8_sm90`).

Because the weights are unchanged, the conversion is a **pure `config.json` edit** — no re-quantization, no calibration:

```jsonc
// experts config group, input_activations: null  ->
"input_activations": {
  "num_bits": 8, "type": "float", "strategy": "token",
  "dynamic": true, "symmetric": true
}
```

The `_w4a8_conversion` key in `config.json` records this provenance.

## Getting the weights

The INT4 weight shards are identical to the base. Materialize a full checkpoint by downloading the base weights and overwriting `config.json` with this repo's W4A8 config:

```bash
# 1. base weights (INT4 shards, tokenizer) — the actual ~159 GB
hf download canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP --local-dir dsv4-w4a8

# 2. this repo's W4A8 config + card (the only real diff)
hf download endnai/DeepSeek-V4-Flash-W4A8-FP8 config.json README.md --local-dir dsv4-w4a8

# dsv4-w4a8/ is now a complete W4A8 checkpoint (INT4 weights + FP8-activation config)
```

The `.safetensors` bytes are unchanged; only `config.json`'s expert `input_activations` differ (see below).

## Serving (vLLM)

Requires a recent **vLLM nightly** and, at the time of writing, four small patches to load the DeepSeek-V4-Flash compressed-tensors checkpoint (these are model-loading fixes, not W4A8-specific — the same patches are needed for the W4A16 base on nightly):

1. `packed_modules_mapping` for the model **and** MTP module (`fused_wqa_wkv`, `fused_wkv_wgate`, `gate_up_proj`).
2. `hash_moe` added to the transformers `ALLOWED_LAYER_TYPES` global allowlist.
3. `o_proj` weight-scale name alias (`weight_scale_inv` → `weight_scale`).

Launch (2×H200, TP2) from the materialized directory (see [Getting the weights](#getting-the-weights)):

```bash
vllm serve ./dsv4-w4a8 \
  --tensor-parallel-size 2 \
  --disable-custom-all-reduce \
  --trust-remote-code
```

`--disable-custom-all-reduce` avoids a TP2 init hang under confidential-compute (custom all-reduce needs CUDA-IPC/symmetric memory, which is unavailable inside TDX CVMs).

**Correctness:** verified matching the W4A16 base on a temp=0 quality probe (GSM8K 3/3 identical).

## Investigation & findings

This checkpoint was built to test a hypothesis: *the DeepSeek-V4-Flash prefill bottleneck is the INT4→BF16 Marlin MoE GEMM, so a W4A8 path (native FP8 activation GEMM) should be ~1.5–2× faster.* **The hypothesis was refuted.** Full sweep on 2–8×H200 (TP2 unless noted), single-request prefill ladder (c=1), long-context (ISL up to 24k):

### Headline: W4A8 gives no throughput advantage over W4A16

| Config | Engine | TP | Prefill TTFT @24k | Prefill tok/s/GPU @24k |
|---|---|---|---|---|
| **W4A8** (this model) | vLLM | 2 | **1658 ms** | **7410** |
| W4A16 (base) | vLLM | 2 | 1691 ms | 7267 |
| FP4 (marlin) | vLLM | 2 | 1824 ms | 7090 |
| FP4 (marlin) | sglang | 2 | 1894 ms | 6832 |
| FP8 (native) | sglang | 4 | 892 ms | 6888 |

**Per-GPU prefill throughput is flat at ~6.8–7.4k tok/s/GPU across every engine and every quantization.** W4A8 and W4A16 are a **wash** (1658 vs 1691 ms — within noise). The FP8-TP4 config's lower absolute TTFT (892 ms) is **pure tensor-parallel scaling** (2× the GPUs); per-GPU it is *also* a wash.

### Why the activation-precision lever doesn't help

At prefill batch sizes, the DeepSeek-V4-Flash MoE (top-6 of 256 small experts) is **weight-bandwidth-bound**, not compute-bound on the expert GEMM. INT4 weights are already the bandwidth-optimal format, and Marlin's INT4→BF16 path already matches the Cutlass W4A8 kernel in practice. Switching activations from BF16/FP8-implicit to FP8 changes the *activation* precision but not the dominant cost. The compute-bound portion of prefill is dominated by **format-shared work** — FP8-block MLA attention and the sparse / Lightning-indexer passes over long context — which is identical across all three checkpoints.

### The prefill ceiling is architectural on Hopper

- Prefill scales **linearly** above ~8k tokens (~+547 ms per +8k) with GPUs at ~100% util and ~690 W (near TDP) → tensor-core-bound, not launch- or attention-quadratic-bound.
- The two kernel improvements that *would* help — **native NVFP4 MoE GEMM** and the **FP4 Lightning-indexer cache** — are **Blackwell-only** (SM100). On Hopper, sglang/vLLM fall back to Marlin.
- A **W4A8 SM90 grouped-GEMM** tuned for the DeepSeek-V4 MoE path is unimplemented upstream (relevant issues closed inactive). Even so, the wash above suggests it would offer little at prefill batch-M.

### What *does* move the needle (deployment)

- **Prefix caching** is the dominant lever: in production, DeepSeek-V4-Flash realizes **~55% radix prefix-cache hit** on real agent/RAG traffic (measured over 24h), i.e. more than half of all prefill is skipped. This is already captured by sglang RadixAttention in production.
- Larger **chunked-prefill** (8192 → 16384) gives ~7% faster long-context prefill TTFT on sglang, at the cost of KV-concurrency — a free win when the server isn't KV-bound.

### Bottom line

Use W4A8 for **GPU-footprint efficiency** (TP2, ~159 GB, native FP8 activation compute where a downstream kernel benefits) — but **not** expecting it to beat W4A16 on DeepSeek-V4-Flash prefill/decode on Hopper. On this architecture the two are equivalent; the real gains come from prefix caching and, eventually, Blackwell hardware.

## Reproducibility

- Weights: byte-identical to `canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP`.
- Transformation: the single `config.json` `input_activations` edit shown above (see the `_w4a8_conversion` provenance key).
- To rebuild: take the W4A16 base, apply the config edit, serve with the vLLM nightly + patches above.

## Acknowledgements

Built and benchmarked by Evrard Nil with Claude (2026-06). Base quantization by `canada-quant`; original model by DeepSeek-AI.