prometheusAIR/DeepSeek-V4-Flash-0731-GGUF

DeepSeek-V4-Flash-0731 GGUF — fits one RTX PRO 6000 96GB

Two imatrix-guided GGUF variants of deepseek-ai/DeepSeek-V4-Flash-0731, both sized so that every tensor lives on a single NVIDIA RTX PRO 6000 Blackwell 96GB GPU — no CPU expert offload, no second card, no layer spill.

• IQ2_XS quality-first: 89.9 GiB, approximately 75 tok/s short decode, up to 384K context with a practical margin.
• IQ2_XXS full-context: 83.15 GiB, the complete 1,048,576-token context, validated with exact retrieval from a 989,987-token prompt.

At a glance

	IQ2_XS quality-first	IQ2_XXS full-context
Size	96,556,502,624 bytes / 89.925 GiB	89,283,036,320 bytes / 83.151 GiB
Expert gate/up	IQ2_XS	IQ2_XXS
Expert down	IQ3_XXS	IQ3_XXS
Dense path	Q8_0	Q6_K
Embedding/output	Q8_0	Q8_0
Calibration processed	327,680 tokens at 2K context	606,208 tokens at 8K context; 100% final expert-slice coverage
Recommended context	64K default; 384K high-context	Full 1,048,576 tokens
CPU/second-GPU offload	none	none
DSpark/MTP draft stages	excluded	excluded
License	MIT	MIT

Will this run on my GPU?

Your setup	Verdict
RTX PRO 6000 Blackwell 96GB (Workstation Edition)	Yes — this is the tested configuration
Two smaller cards totalling ≥ 96 GB	Should work with --split-mode layer. Untested here
One card under 96 GB	Needs -ncmoe N to push expert layers to CPU, which costs a lot of speed. Untested here
A 96 GB card of another architecture	Requires working IQ2_XS / IQ2_XXS / IQ3_XXS / Q6_K kernels as applicable. Untested here

Leave roughly 2–3 GB of the card free for your display/compositor if the GPU is also driving a monitor. The full-context variant left 1,752 MiB in the tested headless configuration, so a display workload can make 1M impractical even on the same card.

Which variant should I use?

• Choose IQ2_XS for the stronger weight recipe, approximately 75 tok/s short decode, and contexts through the recommended 384K profile.
• Choose IQ2_XXS only when you need more than the IQ2_XS artifact can allocate. It reaches the full 1M context by lowering gate/up experts and dense matrices, and long prompts become much slower to prefill.

Quick start

Point llama.cpp at the first shard; it finds the other three automatically.

CUDA_VISIBLE_DEVICES=0 llama-server \
  --model DeepSeek-V4-Flash-0731-IQ2_XS-00001-of-00004.gguf \
  --host 127.0.0.1 --port 8080 \
  --device CUDA0 --split-mode none -ngl 999 --fit off \
  --ctx-size 65536 \
  -b 2048 -ub 2048 \
  -np 1 -fa on --jinja

--fit off and --split-mode none are deliberate: they make a bad configuration fail loudly instead of silently spilling weights to CPU or to a second GPU and quietly costing you 3× the speed.

For long context, drop ubatch to 1,024 and raise the context to 384K:

CUDA_VISIBLE_DEVICES=0 llama-server \
  --model DeepSeek-V4-Flash-0731-IQ2_XS-00001-of-00004.gguf \
  --host 127.0.0.1 --port 8080 \
  --device CUDA0 --split-mode none -ngl 999 --fit off \
  --ctx-size 393216 \
  -b 2048 -ub 1024 \
  -np 1 -fa on --jinja

Do not use ubatch 2,048 at 384K on a 96 GB card — see the table below.

Full 1M profile

CUDA_VISIBLE_DEVICES=0 llama-server \
  --model DeepSeek-V4-Flash-0731-IQ2_XXS-00001-of-00004.gguf \
  --host 127.0.0.1 --port 8080 \
  --device CUDA0 --split-mode none -ngl 999 --fit off \
  --ctx-size 1048576 \
  -b 2048 -ub 1024 \
  -np 1 -fa on --jinja \
  --reasoning-budget 4096 \
  --reasoning-budget-message "I am out of thinking time. I will now stop deliberating and give my best final answer directly."

This profile uses the IQ2_XXS shards, not the larger IQ2_XS files. Keep ubatch at 1,024 and one parallel slot unless you remeasure the entire allocation.

Measured VRAM

Measured on a card reporting 97,887 MiB total, CUDA device 0 only, one slot, flash attention on, --fit off. "Free" is what remained after startup, CUDA warmup and a real generation.

Context	ubatch	Free VRAM after load	Verdict
64K	2,048	3,302 MiB	Default. Most headroom
256K	2,048	1,234 MiB	Recommended if you want ubatch 2,048
320K	2,048	514 MiB	Tight
352K	2,048	138 MiB	Boundary only
368K	2,048	—	OOM during warmup
384K	1,024	998 MiB	Recommended high-context profile
448K	1,024	390 MiB	Tight
480K	1,024	102 MiB	Hard edge — not usable
512K	1,024	—	OOM allocating compute buffers

The 480K row fell to 12 MiB free after a 6,012-token prefill. It documents where the wall is; it is not a deployment recommendation. Driver version, display usage, CUDA version and llama.cpp revision all shift these numbers, so treat them as a starting point and verify on your own card.

These are allocation and generation checks for the IQ2_XS variant. They do not establish retrieval quality at every listed depth.

IQ2_XXS full-context measurements

The full-context artifact loaded at --ctx-size 1048576, -b 2048, -ub 1024, and -np 1, with every model tensor on CUDA0:

Measurement	Result
Total CUDA0 use after startup/warmup	95,409 MiB
Highest CUDA0 use during tested prefills	95,537 MiB
Minimum free CUDA0 VRAM	1,752 MiB
CUDA1 model allocation	zero
6,010-token retrieval	exact; 1,746 tok/s prefill
130,999-token retrieval	exact; 1,101 tok/s prefill
989,987-token retrieval	exact at 97% document depth; 308.56 tok/s average prefill
Near-1M decode	25.44 tok/s
Near-1M prefill wall time	3,208.36 seconds / 53.47 minutes

The final test processed 989,987 prompt tokens and 990,079 total tokens, then returned OBSIDIAN-LANTERN-ONE-MILLION exactly from record 40,000 of 41,248. The context allocation did not grow beyond the measured 95,537 MiB peak. This proves startup, warmup, meaningful prefill, generation, and retrieval near the model's configured limit on the tested runtime; it is not a broad long-context benchmark.

Reasoning, tools, and the empty-response trap

DeepSeek-V4-Flash-0731 thinks by default, and llama.cpp returns that thinking in a separate reasoning_content field. If your client caps max_tokens too low, the budget is consumed by reasoning and content comes back empty — the model is fine, your token budget is not.

This is not hypothetical. On the same prompt during validation:

max_tokens	Tokens used	Result
768	768	finish_reason: length, empty content
2,048	1,117	finish_reason: stop, correct answer

Allow at least ~1,500–2,000 completion tokens for anything non-trivial. If your client won't send a large enough max_tokens (some agent harnesses send none at all, or send a small default), cap thinking server-side instead:

--reasoning-budget 4096
--reasoning-budget-message "I am out of thinking time. I will now stop deliberating and give my best final answer directly."

The message matters — without it, truncated reasoning can spill into content.

Reasoning effort

0731 supports low, high and max. llama.cpp's top-level reasoning_effort field only reliably forwards none; for the others, pass the model-specific values through chat_template_kwargs:

{
  "messages": [{"role": "user", "content": "Solve this carefully."}],
  "temperature": 1.0,
  "top_p": 1.0,
  "max_tokens": 4096,
  "chat_template_kwargs": {"thinking": true, "reasoning_effort": "high"}
}

Tools

Standard OpenAI-compatible function schemas in the request's tools field work; the embedded template renders and parses DeepSeek's DSML tool protocol. The base-model authors recommend temperature = 1.0, top_p = 0.95 for agentic use. Tool calls and tool-result continuation were both validated here.

Requirements

• A CUDA llama.cpp build with DeepSeek-V4 support and working SM 12.0 kernels for IQ2_XS, IQ2_XXS, IQ3_XXS, Q6_K, and Q8_0. Validated on build b10148 (1476ef0e5).
• --jinja, so the embedded chat template is used.

Older builds may not know the architecture, its metadata, or the template.

What's in the file

IQ2_XS quality-first tensor census

Tensor group	Count	Type
Routed expert gate/up	86	IQ2_XS
Routed expert down	43	IQ3_XXS
Dense matrices, embedding, output	661	Q8_0
Routers	43	BF16
Norms, biases, tables	492	F32
Routing/index tables	3	I32
Total	1,328

IQ2_XXS full-context tensor census

Tensor group	Count	Type
Routed expert gate/up	86	IQ2_XXS
Routed expert down	43	IQ3_XXS
Eligible dense matrices	659	Q6_K
Token embedding and output	2	Q8_0
Routers	43	BF16
Norms, biases, tables	492	F32
Routing/index tables	3	I32
Total	1,328

Routed experts dominate this checkpoint's storage, so quantizing only the dense layers cannot reach 96 GB. Both recipes spend more bits on expert down projections than gate/up. The full-context variant additionally moves eligible dense matrices from Q8_0 to Q6_K while retaining embedding and output at Q8_0.

Note on the displayed quant type. These are per-tensor mixed recipes. The GGUF general.file_type records only the base type passed to llama-quantize, so Hub viewers may show Q8_0 for the IQ2_XS files or Q6_K for the IQ2_XXS files. The tensor censuses above are authoritative.

No IQ1_S, IQ2_S, or IQ3_S tensors are present. Those three types have broken CUDA kernels on SM 12.0, which silently poisons many otherwise-plausible sub-4-bit quants on Blackwell. All 43 layers, the release compression-ratio metadata, and the validated 0731 chat template are intact.

The three DSpark/MTP speculative draft stages from the source repository are excluded, because this llama.cpp target does not implement that draft path.

Provenance

Source revision 9e165c30e2704aec5d9d593cce3eebd58bbef1cb.

A high-fidelity master GGUF was built first, preserving the released checkpoint rather than re-quantizing it: packed FP4 experts losslessly repacked to MXFP4, block-scaled FP8 dense products expanded exactly to BF16, native BF16/F32/ I32 preserved. The mixed quant was then produced from that master with llama.cpp quantizer f8a9b8ab (based on b10173) using --allow-requantize, --keep-split and a model-specific importance matrix.

Calibration corpus: 567,086 tokenizer tokens spanning scientific and general prose, code, multilingual text, agent/tool traces, release documentation and the official 0731 encoding fixtures. Processed as 160 chunks × 2,048 tokens into 769 imatrix entries; lowest final sparse routed-expert coverage 99.61%. Corpus SHA-256 401ea959421622e148d062d4c9719d4c31027f4af1ee10ebff0525e4bb0a13ea.

The IQ2_XXS variant used a separate expanded corpus with 613,766 tokenizer tokens across 23 sections. llama-imatrix processed 74 chunks × 8,192 tokens, or 606,208 tokens, into 769 entries. Direct inspection of the final imatrix found all 256 slices populated for every routed-expert matrix: 100.00% final slice coverage. Corpus SHA-256: 0b0503b27680df69a8f40507cb329bc49a0bb3cd8c7f30fa4da1fde9ce8cc3bc. Quantization took 5,323.19 seconds (88.72 minutes) with 12 threads.

The embedded Jinja template was validated byte-for-byte against the official 0731 encoder fixtures, including tool calls and tool-result continuation.

Validation

Focused smoke tests, run against both this quant and the fidelity master:

• exact-instruction following
• 7^222 mod 1000 = 49 at high reasoning effort
• coherent Chinese output
• DSML tool call plus tool-result continuation
• a generated Python merge_intervals, passing 4/4 executable cases

Decode for the IQ2_XS variant clustered at approximately 75 tok/s (63–77 across individual tests; the low outlier was a 21-token generation where fixed overhead dominates), against 21–26 tok/s for the partially-offloaded fidelity master on identical prompts — about 3×.

The IQ2_XXS variant passed the same exact-output, arithmetic, executable coding, Chinese, DSML tool-call, and tool-continuation checks. High-reasoning arithmetic and coding hit an old 768-token smoke cap before emitting content, but both completed correctly with the documented 4,096-token budget. Its generated merge_intervals passed 4/4 executable cases. Retrieval was exact at 6K, 131K, and 989,987 prompt tokens; the near-1M needle was at 97% document depth.

These are implementation smoke tests, not a benchmark suite, and not evidence of quality parity with the released checkpoint.

Limitations

• Low-bit expert quantization can change reasoning, coding, factuality, routing and tool behaviour in ways short smoke tests will not surface. Evaluate on your own workload before relying on it.
• The near-1M check is a single synthetic needle test, not evidence of broad million-token reasoning, multi-hop retrieval, or benchmark quality.
• A near-1M prefill took approximately 53.5 minutes on the tested host.
• The full-context configuration retained only 1,752 MiB at its measured peak; display use and runtime/driver differences can eliminate that margin.
• High-reasoning requests may consume more than 768 completion tokens before emitting final content; provide a realistic max_tokens budget.
• No DSpark speculative draft stages.
• No guarantees for other GPUs or other llama.cpp revisions.

This quant inherits the capabilities, limitations, risks and MIT license of the base model. You are responsible for evaluating outputs and applying safeguards appropriate to your application.

Files

IQ2_XS quality-first

File	Size	SHA-256
DeepSeek-V4-Flash-0731-IQ2_XS-00001-of-00004.gguf	27.50 GiB	b09025603eb0192bb5dae42084c096427813658f5be0b7ec19341a96b3ce7e20
DeepSeek-V4-Flash-0731-IQ2_XS-00002-of-00004.gguf	27.67 GiB	faf2467279b673bbcdfea7c8413f762f39973b3a758808ec2d6bbdd008710f5f
DeepSeek-V4-Flash-0731-IQ2_XS-00003-of-00004.gguf	27.48 GiB	8c7dd99da2d54f39660659d8c401811149bf341c307fedb30a19f510299e2495
DeepSeek-V4-Flash-0731-IQ2_XS-00004-of-00004.gguf	7.27 GiB	eb0cccbdf4b94ecf42d408b2d03b5a0bb3d706a6f0ca0f63ac0256f947a08f94

IQ2_XXS full-context

File	Size	SHA-256
DeepSeek-V4-Flash-0731-IQ2_XXS-00001-of-00004.gguf	25.685443 GiB	0b220f41ed3260c709adaab5bd6c9d60f65055d9dc2a5d9522abff081f68163f
DeepSeek-V4-Flash-0731-IQ2_XXS-00002-of-00004.gguf	25.921878 GiB	4bba1123d7eabeff65386efed047f8dff6c3c3a0ce223aae2fef4b3739070d2e
DeepSeek-V4-Flash-0731-IQ2_XXS-00003-of-00004.gguf	25.671878 GiB	fefe6269b82105de55d2b2f16c2988f1501912c9a2b2c7f64ff86360989e5884
DeepSeek-V4-Flash-0731-IQ2_XXS-00004-of-00004.gguf	5.872108 GiB	5b1d48df2dcc3944394424c89e531179375960bcdbbdea882711421e57df9ed0

sha256sum -c SHA256SUMS

Attribution

Base model and architecture by DeepSeek-AI. Quantization, calibration and single-GPU validation by prometheusAIR.

Please read the original model card for architecture details, recommended sampling and the base model's own limitations.

@misc{deepseekai2026deepseekv4,
  title  = {DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence},
  author = {DeepSeek-AI},
  year   = {2026}
}