---
license: mit
base_model: deepseek-ai/DeepSeek-V4-Flash-0731
base_model_relation: quantized
pipeline_tag: text-generation
library_name: gguf
tags:
  - gguf
  - llama.cpp
  - deepseek
  - deepseek-v4
  - mixture-of-experts
  - imatrix
  - single-gpu
  - rtx-pro-6000
  - blackwell
  - 96gb
  - tool-use
  - reasoning
quantized_by: prometheusAIR
---

# DeepSeek-V4-Flash-0731 GGUF — fits one RTX PRO 6000 96GB

Two imatrix-guided GGUF variants of
[deepseek-ai/DeepSeek-V4-Flash-0731](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731),
both sized so that **every tensor lives on a single NVIDIA RTX PRO 6000
Blackwell 96GB GPU** — no CPU expert offload, no second card, no layer spill.

- **IQ2_XS quality-first:** 89.9 GiB, approximately 75 tok/s short decode, up to
  384K context with a practical margin.
- **IQ2_XXS full-context:** 83.15 GiB, the complete 1,048,576-token context,
  validated with exact retrieval from a 989,987-token prompt.

## At a glance

| | IQ2_XS quality-first | IQ2_XXS full-context |
|---|---|---|
| Size | 96,556,502,624 bytes / 89.925 GiB | 89,283,036,320 bytes / 83.151 GiB |
| Expert gate/up | `IQ2_XS` | `IQ2_XXS` |
| Expert down | `IQ3_XXS` | `IQ3_XXS` |
| Dense path | `Q8_0` | `Q6_K` |
| Embedding/output | `Q8_0` | `Q8_0` |
| Calibration processed | 327,680 tokens at 2K context | 606,208 tokens at 8K context; 100% final expert-slice coverage |
| Recommended context | 64K default; 384K high-context | Full 1,048,576 tokens |
| CPU/second-GPU offload | none | none |
| DSpark/MTP draft stages | excluded | excluded |
| License | MIT | MIT |

## Will this run on my GPU?

| Your setup | Verdict |
|---|---|
| RTX PRO 6000 Blackwell 96GB (Workstation Edition) | Yes — this is the tested configuration |
| Two smaller cards totalling ≥ 96 GB | Should work with `--split-mode layer`. Untested here |
| One card under 96 GB | Needs `-ncmoe N` to push expert layers to CPU, which costs a lot of speed. Untested here |
| A 96 GB card of another architecture | Requires working `IQ2_XS` / `IQ2_XXS` / `IQ3_XXS` / `Q6_K` kernels as applicable. Untested here |

Leave roughly 2–3 GB of the card free for your display/compositor if the GPU
is also driving a monitor. The full-context variant left 1,752 MiB in the tested
headless configuration, so a display workload can make 1M impractical even on
the same card.

### Which variant should I use?

- Choose `IQ2_XS` for the stronger weight recipe, approximately 75 tok/s short
  decode, and contexts through the recommended 384K profile.
- Choose `IQ2_XXS` only when you need more than the `IQ2_XS` artifact can
  allocate. It reaches the full 1M context by lowering gate/up experts and dense
  matrices, and long prompts become much slower to prefill.

## Quick start

Point llama.cpp at the first shard; it finds the other three automatically.

```bash
CUDA_VISIBLE_DEVICES=0 llama-server \
  --model DeepSeek-V4-Flash-0731-IQ2_XS-00001-of-00004.gguf \
  --host 127.0.0.1 --port 8080 \
  --device CUDA0 --split-mode none -ngl 999 --fit off \
  --ctx-size 65536 \
  -b 2048 -ub 2048 \
  -np 1 -fa on --jinja
```

`--fit off` and `--split-mode none` are deliberate: they make a bad
configuration fail loudly instead of silently spilling weights to CPU or to a
second GPU and quietly costing you 3× the speed.

For long context, drop ubatch to 1,024 and raise the context to 384K:

```bash
CUDA_VISIBLE_DEVICES=0 llama-server \
  --model DeepSeek-V4-Flash-0731-IQ2_XS-00001-of-00004.gguf \
  --host 127.0.0.1 --port 8080 \
  --device CUDA0 --split-mode none -ngl 999 --fit off \
  --ctx-size 393216 \
  -b 2048 -ub 1024 \
  -np 1 -fa on --jinja
```

Do not use ubatch 2,048 at 384K on a 96 GB card — see the table below.

### Full 1M profile

```bash
CUDA_VISIBLE_DEVICES=0 llama-server \
  --model DeepSeek-V4-Flash-0731-IQ2_XXS-00001-of-00004.gguf \
  --host 127.0.0.1 --port 8080 \
  --device CUDA0 --split-mode none -ngl 999 --fit off \
  --ctx-size 1048576 \
  -b 2048 -ub 1024 \
  -np 1 -fa on --jinja \
  --reasoning-budget 4096 \
  --reasoning-budget-message "I am out of thinking time. I will now stop deliberating and give my best final answer directly."
```

This profile uses the `IQ2_XXS` shards, not the larger `IQ2_XS` files. Keep
ubatch at 1,024 and one parallel slot unless you remeasure the entire allocation.

## Measured VRAM

Measured on a card reporting 97,887 MiB total, CUDA device 0 only, one slot,
flash attention on, `--fit off`. "Free" is what remained after startup, CUDA
warmup and a real generation.

| Context | ubatch | Free VRAM after load | Verdict |
|---:|---:|---:|---|
| 64K | 2,048 | 3,302 MiB | Default. Most headroom |
| 256K | 2,048 | 1,234 MiB | **Recommended if you want ubatch 2,048** |
| 320K | 2,048 | 514 MiB | Tight |
| 352K | 2,048 | 138 MiB | Boundary only |
| 368K | 2,048 | — | OOM during warmup |
| 384K | 1,024 | 998 MiB | **Recommended high-context profile** |
| 448K | 1,024 | 390 MiB | Tight |
| 480K | 1,024 | 102 MiB | Hard edge — not usable |
| 512K | 1,024 | — | OOM allocating compute buffers |

The 480K row fell to **12 MiB free** after a 6,012-token prefill. It documents
where the wall is; it is not a deployment recommendation. Driver version,
display usage, CUDA version and llama.cpp revision all shift these numbers, so
treat them as a starting point and verify on your own card.

These are allocation and generation checks for the `IQ2_XS` variant. They do
**not** establish retrieval quality at every listed depth.

### IQ2_XXS full-context measurements

The full-context artifact loaded at `--ctx-size 1048576`, `-b 2048`,
`-ub 1024`, and `-np 1`, with every model tensor on CUDA0:

| Measurement | Result |
|---|---:|
| Total CUDA0 use after startup/warmup | 95,409 MiB |
| Highest CUDA0 use during tested prefills | 95,537 MiB |
| Minimum free CUDA0 VRAM | 1,752 MiB |
| CUDA1 model allocation | zero |
| 6,010-token retrieval | exact; 1,746 tok/s prefill |
| 130,999-token retrieval | exact; 1,101 tok/s prefill |
| 989,987-token retrieval | exact at 97% document depth; 308.56 tok/s average prefill |
| Near-1M decode | 25.44 tok/s |
| Near-1M prefill wall time | 3,208.36 seconds / 53.47 minutes |

The final test processed 989,987 prompt tokens and 990,079 total tokens, then
returned `OBSIDIAN-LANTERN-ONE-MILLION` exactly from record 40,000 of 41,248.
The context allocation did not grow beyond the measured 95,537 MiB peak. This
proves startup, warmup, meaningful prefill, generation, and retrieval near the
model's configured limit on the tested runtime; it is not a broad long-context
benchmark.

## Reasoning, tools, and the empty-response trap

DeepSeek-V4-Flash-0731 **thinks by default**, and llama.cpp returns that
thinking in a separate `reasoning_content` field. If your client caps
`max_tokens` too low, the budget is consumed by reasoning and `content` comes
back **empty** — the model is fine, your token budget is not.

This is not hypothetical. On the same prompt during validation:

| `max_tokens` | Tokens used | Result |
|---:|---:|---|
| 768 | 768 | `finish_reason: length`, **empty content** |
| 2,048 | 1,117 | `finish_reason: stop`, correct answer |

Allow at least ~1,500–2,000 completion tokens for anything non-trivial. If your
client won't send a large enough `max_tokens` (some agent harnesses send none
at all, or send a small default), cap thinking server-side instead:

```
--reasoning-budget 4096
--reasoning-budget-message "I am out of thinking time. I will now stop deliberating and give my best final answer directly."
```

The message matters — without it, truncated reasoning can spill into `content`.

### Reasoning effort

0731 supports `low`, `high` and `max`. llama.cpp's top-level
`reasoning_effort` field only reliably forwards `none`; for the others, pass
the model-specific values through `chat_template_kwargs`:

```json
{
  "messages": [{"role": "user", "content": "Solve this carefully."}],
  "temperature": 1.0,
  "top_p": 1.0,
  "max_tokens": 4096,
  "chat_template_kwargs": {"thinking": true, "reasoning_effort": "high"}
}
```

### Tools

Standard OpenAI-compatible function schemas in the request's `tools` field
work; the embedded template renders and parses DeepSeek's DSML tool protocol.
The base-model authors recommend `temperature = 1.0`, `top_p = 0.95` for
agentic use. Tool calls and tool-result continuation were both validated here.

## Requirements

- A CUDA llama.cpp build with DeepSeek-V4 support and working SM 12.0 kernels
  for `IQ2_XS`, `IQ2_XXS`, `IQ3_XXS`, `Q6_K`, and `Q8_0`. Validated on
  build `b10148` (`1476ef0e5`).
- `--jinja`, so the embedded chat template is used.

Older builds may not know the architecture, its metadata, or the template.

## What's in the file

### IQ2_XS quality-first tensor census

| Tensor group | Count | Type |
|---|---:|---|
| Routed expert gate/up | 86 | `IQ2_XS` |
| Routed expert down | 43 | `IQ3_XXS` |
| Dense matrices, embedding, output | 661 | `Q8_0` |
| Routers | 43 | `BF16` |
| Norms, biases, tables | 492 | `F32` |
| Routing/index tables | 3 | `I32` |
| **Total** | **1,328** | |

### IQ2_XXS full-context tensor census

| Tensor group | Count | Type |
|---|---:|---|
| Routed expert gate/up | 86 | `IQ2_XXS` |
| Routed expert down | 43 | `IQ3_XXS` |
| Eligible dense matrices | 659 | `Q6_K` |
| Token embedding and output | 2 | `Q8_0` |
| Routers | 43 | `BF16` |
| Norms, biases, tables | 492 | `F32` |
| Routing/index tables | 3 | `I32` |
| **Total** | **1,328** | |

Routed experts dominate this checkpoint's storage, so quantizing only the dense
layers cannot reach 96 GB. Both recipes spend more bits on expert down
projections than gate/up. The full-context variant additionally moves eligible
dense matrices from Q8_0 to Q6_K while retaining embedding and output at Q8_0.

> **Note on the displayed quant type.** These are per-tensor mixed recipes. The
> GGUF `general.file_type` records only the base type passed to `llama-quantize`,
> so Hub viewers may show `Q8_0` for the IQ2_XS files or `Q6_K` for the IQ2_XXS
> files. The tensor censuses above are authoritative.

**No `IQ1_S`, `IQ2_S`, or `IQ3_S` tensors are present.** Those three types have
broken CUDA kernels on SM 12.0, which silently poisons many otherwise-plausible
sub-4-bit quants on Blackwell. All 43 layers, the release compression-ratio
metadata, and the validated 0731 chat template are intact.

The three DSpark/MTP speculative draft stages from the source repository are
excluded, because this llama.cpp target does not implement that draft path.

## Provenance

Source revision `9e165c30e2704aec5d9d593cce3eebd58bbef1cb`.

A high-fidelity master GGUF was built first, preserving the released checkpoint
rather than re-quantizing it: packed FP4 experts losslessly repacked to `MXFP4`,
block-scaled FP8 dense products expanded exactly to `BF16`, native `BF16`/`F32`/
`I32` preserved. The mixed quant was then produced from that master with
llama.cpp quantizer `f8a9b8ab` (based on `b10173`) using `--allow-requantize`,
`--keep-split` and a model-specific importance matrix.

Calibration corpus: 567,086 tokenizer tokens spanning scientific and general
prose, code, multilingual text, agent/tool traces, release documentation and the
official 0731 encoding fixtures. Processed as 160 chunks × 2,048 tokens into 769
imatrix entries; lowest final sparse routed-expert coverage 99.61%. Corpus
SHA-256 `401ea959421622e148d062d4c9719d4c31027f4af1ee10ebff0525e4bb0a13ea`.

The `IQ2_XXS` variant used a separate expanded corpus with 613,766 tokenizer
tokens across 23 sections. `llama-imatrix` processed 74 chunks × 8,192 tokens,
or 606,208 tokens, into 769 entries. Direct inspection of the final imatrix
found all 256 slices populated for every routed-expert matrix: 100.00% final
slice coverage. Corpus SHA-256:
`0b0503b27680df69a8f40507cb329bc49a0bb3cd8c7f30fa4da1fde9ce8cc3bc`.
Quantization took 5,323.19 seconds (88.72 minutes) with 12 threads.

The embedded Jinja template was validated byte-for-byte against the official
0731 encoder fixtures, including tool calls and tool-result continuation.

## Validation

Focused smoke tests, run against both this quant and the fidelity master:

- exact-instruction following
- `7^222 mod 1000 = 49` at high reasoning effort
- coherent Chinese output
- DSML tool call plus tool-result continuation
- a generated Python `merge_intervals`, passing 4/4 executable cases

Decode for the IQ2_XS variant clustered at approximately 75 tok/s (63–77
across individual tests; the low outlier was a 21-token generation where fixed
overhead dominates), against 21–26 tok/s for the partially-offloaded fidelity
master on identical prompts — about 3×.

The IQ2_XXS variant passed the same exact-output, arithmetic, executable coding,
Chinese, DSML tool-call, and tool-continuation checks. High-reasoning arithmetic
and coding hit an old 768-token smoke cap before emitting content, but both
completed correctly with the documented 4,096-token budget. Its generated
`merge_intervals` passed 4/4 executable cases. Retrieval was exact at 6K, 131K,
and 989,987 prompt tokens; the near-1M needle was at 97% document depth.

**These are implementation smoke tests, not a benchmark suite, and not evidence
of quality parity with the released checkpoint.**

## Limitations

- Low-bit expert quantization can change reasoning, coding, factuality, routing
  and tool behaviour in ways short smoke tests will not surface. Evaluate on
  your own workload before relying on it.
- The near-1M check is a single synthetic needle test, not evidence of broad
  million-token reasoning, multi-hop retrieval, or benchmark quality.
- A near-1M prefill took approximately 53.5 minutes on the tested host.
- The full-context configuration retained only 1,752 MiB at its measured peak;
  display use and runtime/driver differences can eliminate that margin.
- High-reasoning requests may consume more than 768 completion tokens before
  emitting final content; provide a realistic `max_tokens` budget.
- No DSpark speculative draft stages.
- No guarantees for other GPUs or other llama.cpp revisions.

This quant inherits the capabilities, limitations, risks and MIT license of the
base model. You are responsible for evaluating outputs and applying safeguards
appropriate to your application.

## Files

### IQ2_XS quality-first

| File | Size | SHA-256 |
|---|---:|---|
| `DeepSeek-V4-Flash-0731-IQ2_XS-00001-of-00004.gguf` | 27.50 GiB | `b09025603eb0192bb5dae42084c096427813658f5be0b7ec19341a96b3ce7e20` |
| `DeepSeek-V4-Flash-0731-IQ2_XS-00002-of-00004.gguf` | 27.67 GiB | `faf2467279b673bbcdfea7c8413f762f39973b3a758808ec2d6bbdd008710f5f` |
| `DeepSeek-V4-Flash-0731-IQ2_XS-00003-of-00004.gguf` | 27.48 GiB | `8c7dd99da2d54f39660659d8c401811149bf341c307fedb30a19f510299e2495` |
| `DeepSeek-V4-Flash-0731-IQ2_XS-00004-of-00004.gguf` | 7.27 GiB | `eb0cccbdf4b94ecf42d408b2d03b5a0bb3d706a6f0ca0f63ac0256f947a08f94` |

### IQ2_XXS full-context

| File | Size | SHA-256 |
|---|---:|---|
| `DeepSeek-V4-Flash-0731-IQ2_XXS-00001-of-00004.gguf` | 25.685443 GiB | `0b220f41ed3260c709adaab5bd6c9d60f65055d9dc2a5d9522abff081f68163f` |
| `DeepSeek-V4-Flash-0731-IQ2_XXS-00002-of-00004.gguf` | 25.921878 GiB | `4bba1123d7eabeff65386efed047f8dff6c3c3a0ce223aae2fef4b3739070d2e` |
| `DeepSeek-V4-Flash-0731-IQ2_XXS-00003-of-00004.gguf` | 25.671878 GiB | `fefe6269b82105de55d2b2f16c2988f1501912c9a2b2c7f64ff86360989e5884` |
| `DeepSeek-V4-Flash-0731-IQ2_XXS-00004-of-00004.gguf` | 5.872108 GiB | `5b1d48df2dcc3944394424c89e531179375960bcdbbdea882711421e57df9ed0` |

```bash
sha256sum -c SHA256SUMS
```

## Attribution

Base model and architecture by DeepSeek-AI. Quantization, calibration and
single-GPU validation by `prometheusAIR`.

Please read the [original model card](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731)
for architecture details, recommended sampling and the base model's own
limitations.

```bibtex
@misc{deepseekai2026deepseekv4,
  title  = {DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence},
  author = {DeepSeek-AI},
  year   = {2026}
}
```