bullerwins/DeepSeek-V4-Flash-0731-GGUF

DeepSeek-V4-Flash-0731 GGUF

GGUF conversions and expert-focused quantizations of deepseek-ai/DeepSeek-V4-Flash-0731 for llama.cpp.

The routed MoE experts in the lossless reference remain in their native MXFP4 representation. FP8-origin non-expert matrices are dequantized into BF16, which represents their E4M3 values and power-of-two E8M0 scales exactly. Original BF16 and F32 tensors are also preserved. The resulting MXFP4_MOE-BF16 file is lossless relative to the model tensors used by llama.cpp.

The MXFP4_MOE-Q8_0 file is not lossless. It preserves the routed experts in native MXFP4 but converts FP8-origin matrices to Q8_0. Equal nominal bit width does not make FP8 to Q8_0 conversion lossless because the formats use different value encodings and block scales.

Files

File	Routed experts	Other tensors	Size
DeepSeek-V4-Flash-0731-MXFP4_MOE-BF16.gguf	Native MXFP4	Lossless BF16/F32	161.87 GB / 150.75 GiB
DeepSeek-V4-Flash-0731-MXFP4_MOE-Q8_0.gguf	Native MXFP4	FP8-origin matrices in Q8_0; source BF16/F32 preserved	156.38 GB / 145.64 GiB
DeepSeek-V4-Flash-0731-IQ3_S-Experts-Q8_0.gguf	IQ3_S	Eligible non-expert matrices in Q8_0	126.86 GB / 118.14 GiB
DeepSeek-V4-Flash-0731-IQ3_XXS-Experts-Q8_0.gguf	IQ3_XXS	Eligible non-expert matrices in Q8_0	113.87 GB / 106.05 GiB
DeepSeek-V4-Flash-0731-IQ2_XS-Experts-Q8_0.gguf	IQ2_XS	Eligible non-expert matrices in Q8_0	87.90 GB / 81.86 GiB
DeepSeek-V4-Flash-0731-imatrix.gguf	-	Importance matrix generated from 802 x 512-token calibration chunks	0.47 GB / 0.44 GiB

The IQ expert quantizations were generated with the included importance matrix. Dense components were kept at higher precision because they are always active and represent only a small fraction of total parameters.

KLD quality evaluation

All models were tested against logits from the lossless MXFP4_MOE-BF16 reference using 50 x 512-token Wikitext-2 chunks. The test evaluated 12,750 output distributions. Lower KLD and higher same-top-token agreement are better.

Model	Mean KLD	PPL ratio	Same top token
Lossless MXFP4/BF16	0.000000	1.0076	100.000%
MXFP4/Q8_0	0.138636	1.0109	88.824%
IQ3_S experts/Q8_0	0.317128	1.1540	83.404%
IQ3_XXS experts/Q8_0	0.310419	1.1604	83.067%
IQ2_XS experts/Q8_0	0.600273	1.4658	75.082%

Comparison with Unsloth Dynamic quants

Both collections retain very good quality across their practical mid- and high-bit ranges, and they are neck and neck along much of the size-quality curve. The lossless bullerwins and Unsloth versions have byte-identical tensor payloads. Unsloth has the advantage around 144 GiB and 119 GiB: UD-Q4_K_XL is smaller than this repository's MXFP4/Q8_0 model at essentially the same KLD, while UD-Q3_K_M has clearly lower KLD than IQ3_S for about 1.1 GiB more. The bullerwins IQ3_XXS model wins an intermediate comparison by being smaller and lower-KLD than Unsloth UD-IQ3_S, and bullerwins IQ2_XS has better KLD than Unsloth UD-IQ1_M for less than 1 GiB additional size.

Source model

DeepSeek's recommended sampling settings are temperature = 1.0, with top_p = 0.95 for agentic scenarios and top_p = 1.0 otherwise. See the official model repository for architecture details, chat formatting, license, and intended usage.

---

Original model card

The following model card is from deepseek-ai/DeepSeek-V4-Flash-0731.

DeepSeek-V4-Flash-0731

---

Technical Report👁️

Introduction

DeepSeek-V4-Flash-0731 is the official release of DeepSeek-V4-Flash, superseding the preview version, with substantially enhanced agentic capabilities. It has the same model structure as DeepSeek-V4-Flash-DSpark, i.e. it comes with a speculative decoding module attached.

DeepSeek-V4-Flash-0731 outperforms DeepSeek-V4-Pro (Preview) on benchmarks listed below despite its far smaller activated parameter count, and is broadly competitive with the strongest proprietary models available.

Benchmark	DeepSeek-V4-Flash-0731	DeepSeek-V4-Flash (Preview)	DeepSeek-V4-Pro (Preview)	GLM-5.2	Opus-4.8
Terminal Bench 2.1	82.7	61.8	72.1	81.0	85.0
NL2Repo	54.2	39.4	38.5	48.9	69.7
Cybergym	76.7	38.7	52.7	-	83.1
DeepSWE	54.4	7.3	12.8	46.2	58.0
Toolathlon-Verified	70.3	49.7	55.9	59.9	76.2
Agents' Last Exam	25.2	15.8	16.5	23.8	25.7
AutomationBench Public	25.1	10.8	12.8	12.9	27.2
DSBench-FullStack †	68.7	37.0	41.8	61.8	71.6
DSBench-Hard †	59.6	25.8	31.1	54.5	71.7

Notes:

1. For the Code Agent tasks among the public benchmarks above, DeepSeek-V4-Flash-0731 is evaluated with the minimal mode of DeepSeek Harness (to be released) as the agent framework, using the max reasoning effort level with temperature = 1.0, top_p = 0.95.
2. † DSBench-FullStack is an internal full-stack development test set; DSBench-Hard is an internal test set of difficult coding-agent problems.

Chat Template

This release does not include a Jinja-format chat template. Instead, we provide a dedicated encoding folder with Python scripts and test cases demonstrating how to encode messages in OpenAI-compatible format into input strings for the model, and how to parse the model's text output. Please refer to the encoding folder for full documentation.

The reasoning_effort parameter now supports three levels — low, high, and max — which control how much deliberation the model spends before answering.

A brief example:

from encoding_dsv4 import encode_messages, parse_message_from_completion_text

messages = [
    {"role": "user", "content": "hello"},
    {"role": "assistant", "content": "Hello! I am DeepSeek.", "reasoning_content": "thinking..."},
    {"role": "user", "content": "1+1=?"}
]

# messages -> string
prompt = encode_messages(messages, thinking_mode="thinking", reasoning_effort="max")

# string -> tokens
import transformers
tokenizer = transformers.AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V4-Flash-0731")
tokens = tokenizer.encode(prompt)

How to Run with vLLM

DSpark speculative decoding is enabled with a single flag — add --speculative-config with method: dspark to your vLLM launch command:

--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

For example, the command below serves the model with vLLM on a single 4×GB300 node. See the vLLM recipe for detailed instructions and other hardware configurations.

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --trust-remote-code --kv-cache-dtype fp8 --block-size 256 \
  --data-parallel-size 4 --enable-expert-parallel \
  --moe-backend deep_gemm_mega_moe \
  --attention-config '{"use_fp4_indexer_cache": true}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

How to Run with SGLang

Enable DSpark with --speculative-algorithm DSPARK and do not set a separate --speculative-draft-model-path as the target and draft weights therefore come from the same checkpoint. See the SGLang cookbook for detailed instructions, benchmarks and other hardwares configurations.

sglang serve \
  --trust-remote-code \
  --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
  --tp 4 \
  --moe-runner-backend flashinfer_mxfp4 \
  --speculative-algorithm DSPARK \
  --mem-fraction-static 0.90 \
  --chunked-prefill-size 4096 \
  --swa-full-tokens-ratio 0.1 \

How to Run Locally

Please refer to the inference folder for detailed instructions on running DeepSeek-V4 locally, including model weight conversion and interactive chat demos.

For local deployment, we recommend setting the sampling parameters to temperature = 1.0, with top_p = 0.95 for agentic scenarios and top_p = 1.0 otherwise. For the high and max reasoning effort levels, we recommend a maximum output length of 384K tokens.

License

This repository and the model weights are licensed under the MIT License.

Citation

@misc{deepseekai2026deepseekv4,
      title={DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence},
      author={DeepSeek-AI},
      year={2026},
}

Contact

If you have any questions, please raise an issue or contact us at service@deepseek.com.