jarrelscy/DeepSeek-V4-Pro-0813-NVFP4-AQLM-hybrid

DeepSeek-V4-Pro-0813 — NVFP4 + AQLM two-tier hybrid (routed experts)

A per-expert two-tier quantization of the routed experts of deepseek-ai/DeepSeek-V4-Pro-0813, built to fit the full 61-layer model in ~384 GB of VRAM with headroom for a long context. Attention, dense projections, and the shared expert are served in FP8; only the 384 routed experts per layer are split across the two tiers.

Each routed expert is placed in one of two tiers by saliency:

Tier	Format	Share	Bits	Notes
Hot	NVFP4	14.00% (3279 / 23424 experts)	~4 bpw	group-16 e4m3 block scale + per-tensor f32 global scale
Cold	AQLM	86.00% (20145 / 23424 experts)	1.0 bpw	group_size=16, 1 codebook, 65536 entries

The hot/cold assignment is per (layer, expert): the most salient experts in each layer are kept in NVFP4, the rest go to 1.0-bpw AQLM. The per-layer hot count ranges from 24 to 93 of 384; the full assignment is in config.json → quantization_config.aqlm_layer_books.

• Total on disk: ~335 GiB, 99 safetensors shards (10031 tensors).
• Model: model_type: deepseek_v4, DeepseekV4ForCausalLM, 61 all-MoE layers, 384 routed experts (top-6) + 1 shared, H=7168, moe_intermediate=3072, MLA with sparse-attention indexer.

Serving

This checkpoint uses the nvfp4_aqlm_hybrid quant method, which is not in upstream vLLM. It requires the companion fork branch, which ships the fused per-expert NVFP4+AQLM MoE kernel (templated for group_size 8 and 16), the hybrid loader/dispatch, the DeepSeek-V4 expert name remap, and a tensor-parallel path for the hybrid experts (column-parallel gate_up / row-parallel down sharding with replicated AQLM codebooks):

• Fork: https://github.com/jarrelscy/vllm-glm52-sm120
• Branch: dsv4pro0813 (commit 620e4276c or later — earlier commits boot-fail against cutlass-dsl 4.7.0, see Known Issues)

The model requires the DeepSeek MLA FP8 KV cache (--kv-cache-dtype fp8_ds_mla).

Verified configuration (4x RTX PRO 6000 Blackwell, SM120, 96 GB each)

TP4, full native 1M context (1,048,576), CUDA graphs on. Measured:

• coherent output, ~52 tok/s decode at short context;
• prefill ~2000 tok/s at 31K, ~1430 tok/s at 100K, ~880 tok/s averaged over a full-window prefill (per-token cost grows with depth);
• 1,048,576 tokens per request fits at 1.27x concurrency (util 0.97, --max-num-batched-tokens 4096);
• needle-in-haystack at 1,039,538 prompt tokens (needle at 50% depth) recalled exactly.

--max-num-batched-tokens 4096 is the sweet spot: +30% prefill over 2048; 8192 does not co-fit with the 1M KV (activation scratch grows ~1.5 GiB per 1024 chunk tokens). Speculative decoding via the model's MTP layer does not work yet (the drafter weights use mtp.* naming the deepseek_mtp loader doesn't map).

git clone --branch dsv4pro0813 https://github.com/jarrelscy/vllm-glm52-sm120 vllm
docker build -f vllm/Dockerfile.glm52-sm120 -t dsv4pro-sm120 vllm

docker run -d --name dsv4pro --gpus all --ipc host --shm-size 16g \
  -p 8001:8001 \
  -v /path/to/DeepSeek-V4-Pro-0813-NVFP4-AQLM-hybrid:/models/dsv4pro:ro \
  --entrypoint bash dsv4pro-sm120 -c '
    cd /opt/vllm && source .venv/bin/activate && \
    # JIT toolchain for the TileLang/AQLM kernel builds: Python.h + CUDA home
    export CPATH=$(find /root/.local/share/uv/python -maxdepth 4 -type d -path "*/include/python3.12" | head -1) && \
    export C_INCLUDE_PATH="$CPATH" && \
    export CUDA_HOME=$(python -c "import nvidia,glob; print(sorted(glob.glob(nvidia.__path__[0]+\"/cu1*\"))[-1])") && \
    vllm serve /models/dsv4pro \
      --tensor-parallel-size 4 \
      --trust-remote-code \
      --kv-cache-dtype fp8_ds_mla \
      --gpu-memory-utilization 0.97 \
      --max-model-len 1048576 \
      --max-num-seqs 2 \
      --max-num-batched-tokens 4096 \
      --no-enable-flashinfer-autotune \
      --compilation-config "{\"mode\": 3, \"cudagraph_mode\": \"FULL_AND_PIECEWISE\"}" \
      --served-model-name deepseek/v4pro \
      --port 8001'

Weight load + TileLang/AQLM JIT warmup takes several minutes on first boot.

Parallelism notes

• TP4 is the recommended topology. The branch shards each expert's intermediate dim across ranks (3072/4 = 768, a multiple of the group-16 quant blocks) and all-reduces the MoE output.
• Decode-context-parallel (DCP) does not work with this checkpoint: the sparse-attention indexer stores compressed KV (compress_ratio=4) and the fork rejects DCP in that combination. MLA latent KV is small enough that the full 1M window fits per-GPU without it.
• PP still works (--pipeline-parallel-size N with tp=1) as a fallback if the TP expert sharding misbehaves, at a large decode-speed cost (pipeline bubble, no useful CUDA-graph gain).

Known issues (fixed in commit 620e4276c)

flashinfer main pins nvidia-cutlass-dsl==4.7.0, which post-dates the cute kernels vendored by the precompiled vLLM wheel and quack 0.5.0. Three boot failures result, all baked-fixed in the Dockerfile at 620e4276c:

1. 4.7.0 moved ThrMma/ThrCopy/TiledMma from cutlass.cute.core to cutlass.cute; stale annotations evaluate at import and kill the DSV4 sparse indexer (AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma').
2. vllm_flash_attn/cute/utils.py::fmax assumes CUDA 12.9 implies the old 3-positional nvvm.fmax; 4.7.0 uses the new 2-positional API even on cu129 (TypeError: fmax() takes 2 positional arguments).
3. 4.7.0's tvm_ffi_provider needs apache-tvm-ffi>=0.1.11 for map_dataclass_to_tuple, but 0.1.12+ breaks import tilelang — 0.1.11 is the only version both accept.

How it was made

• Cold tier (AQLM 1.0 bpw): additive quantization at group_size=16 with a single 65536-entry codebook per expert, fit per layer.
• Hot tier (NVFP4): the 0813 release ships routed experts as MXFP4 (e2m1 + UE8M0 group-32 scales); there is no BF16 master on the Hub. The hot experts are transcoded MXFP4 → NVFP4 (dequant to bf16, requantize to group-16 e4m3 + f32 global scale). Weight-only, data-free.
• Everything else (*.attn.*, *.ffn.shared_experts.*, dense layers, head, mtp.*) is served in FP8 and left byte-identical to the base FP8 path.

Provenance

• Base: deepseek-ai/DeepSeek-V4-Pro-0813 (MIT).
• This is built on the 0813 base; other public DeepSeek-V4-Pro-NVFP4 checkpoints target the older deepseek-ai/DeepSeek-V4-Pro base.