ox-ox/DeepSeek-V4-Flash-0731-GGUF

DeepSeek-V4-Flash-0731 — GGUF for ds4 (mixed 2+4 bit)

GGUF builds of deepseek-ai/DeepSeek-V4-Flash-0731 — the official release of DeepSeek V4 Flash (304B total parameters, hybrid CSA+HCA attention, 1M context) — for the ds4 / DwarfStar inference engine. Runs fully resident on a single 128 GB Apple Silicon machine.

Quantized in the asymmetric style of antirez/deepseek-v4-gguf: crush the routed experts (they are almost all of the weights), keep the decision-making parts high precision. The filename is the spec.

📊 The imatrix is published here too, not just the models it produced: imatrix/DeepSeek-V4-Flash-0731-chat-v2-routed-moe-ds4-1p5m.dat — 430 MB, collected on the 0731 weights themselves, 2729 prompts / 1.5M tokens / 387M routed-expert observations, 129 entries (43 layers × gate/up/down, full coverage, zero missing tensors). Reuse it with deepseek4-quantize --imatrix to build your own mix, or to check mine.

This is not my recipe. The recipe, the quantizer (gguf-tools/deepseek4-quantize), the imatrix pipeline and the engine are all antirez and the DwarfStar contributors. All I did was run that toolchain against the newer 0731 checkpoint and collect a fresh imatrix on those weights. If antirez ships official 0731 builds, prefer them — watch huggingface.co/antirez/deepseek-v4-gguf (@antirez on the Hub).

⚠️ Needs ds4 / DwarfStar. This is not a generic GGUF. It will not load in llama.cpp, Ollama or LM Studio — the tensor layout, quant mix and metadata are specific to the DS4 engine.

Not affiliated with DeepSeek or antirez. Weights © DeepSeek, released under MIT.

Installation

git clone https://github.com/antirez/ds4
cd ds4
make                  # macOS Metal

Then download one of the files below into gguf/ and point -m at it.

Files

File	Size	imatrix	status
DeepSeek-V4-Flash-0731-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf	97.6 GB	—	✅ available
DeepSeek-V4-Flash-0731-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf	97.6 GB	✅ collected on 0731	✅ available
imatrix/DeepSeek-V4-Flash-0731-chat-v2-routed-moe-ds4-1p5m.dat	430 MB	—	✅ available

Either works. On held-out wikitext the two are statistically indistinguishable (−1.36 %, p ≈ 0.098 — full numbers). The imatrix build is the one to prefer a priori, since its statistics come from the 0731 weights, but I have no measurement proving it on general text. The plain build is the intermediate the imatrix was collected on, published so the comparison can be reproduced rather than taken on trust.

Usage

./ds4 -m gguf/DeepSeek-V4-Flash-0731-...-imatrix.gguf \
  -p "Explain Redis streams in one paragraph."

./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192

Sampling, per DeepSeek: temperature = 1.0, top_p = 0.95 for agentic scenarios, top_p = 1.0 otherwise.

0731 introduces a reasoning_effort parameter with three levels — low, high, max — controlling how much the model deliberates before answering. The preview's card documented no such parameter (it exposed thinking_mode only), so this is an addition. 0731 is also reported to spend substantially more tokens thinking than the preview; budget context and output limits accordingly, or use --nothink.

Quantization recipe

Unchanged from antirez's mixed 2+4 bit build:

Tensor class	Quant	Notes
blk.37..42.ffn_{gate,up,down}_exps	Q4_K	last 6 layers, most sensitive
blk.*.ffn_{gate,up}_exps	IQ2_XXS	routed expert gate/up
blk.*.ffn_down_exps	Q2_K	K-quant: down is far more sensitive
blk.*.ffn_{gate,up,down}_shexp	Q8_0	shared experts
blk.*.attn_q_a/q_b/kv/output_a/output_b	Q8_0	MLA + low-rank output
output.weight	Q8_0
token_embd.weight	F16
blk.*.ffn_gate_inp	F16	router — never touch this
router bias, attn_sinks, all *_norm	F32
blk.*.ffn_gate_tid2eid	I32	hash-routing tables, first 3 layers
attn_compressor_*, indexer_*, hc_*	F16 / F32	DSv4-specific blocks

Routed experts are the overwhelming majority of the parameters, but each individual expert only sees a fraction of the tokens — so aggressive quantization there costs little on average. Everything every token passes through stays high precision.

How this was built

The source weights are already 4-bit

Worth stating plainly, because it is easy to assume otherwise: DeepSeek never published a BF16 version of this model. The routed experts ship at FP4 because that is how they were trained — the paper (2606.19348, §5.2.1) applies FP4 quantization-aware training to the MoE expert weights and the indexer QK path during post-training, and states that "the routed expert parameters utilize FP4 precision".

Reading the safetensors headers across all 48 shards:

dtype	tensors	bytes	what it is
I8	35 328	148.18 GB	FP4 weights, two per byte
F8_E8M0	35 718	9.26 GB	block scales, one per 32 weights
F8_E4M3	390	6.30 GB
BF16	445	2.97 GB	attention / shared / embeddings
F32	433	0.15 GB	norms, biases
total	72 317	166.88 GB

Declared shapes are the packed shapes. Taken literally they sum to 165B parameters. Unpacking gives the real count — note these are bytes on the left, parameters on the right:

148.18 GB of I8   -> 148.18e9 bytes x 2 weights/byte = 296.36e9 FP4 weights
                   +   1.48e9 BF16  +  6.30e9 F8_E4M3
                   = 304.14e9 parameters

which is the advertised count.

The scale count confirms the unpacking independently. If I8 really holds two FP4 weights per byte in blocks of 32, there must be 296.36e9 / 32 = 9.26e9 block scales — and F8_E8M0 is exactly 9.26 GB, i.e. 9.26e9 one-byte scales. The two numbers are derived from different fields of the header and agree, so this isn't a guess about the layout.

Two different bit figures follow, and they should not be confused:

FP4 expert path	4.25 bits/weight — 4 bits + one 8-bit scale per 32 weights
whole model on disk	4.39 bits/param — 166.88 GB × 8 / 304.1B, including the BF16/F32 tensors that were never quantized

So this is a 4-bit → 2-bit requantization: one lossy step, not two. The tensors crushed to IQ2_XXS are exactly the ones DeepSeek trained at FP4. The quantizer unpacks FP4/FP8 before re-encoding.

Two passes, because the imatrix needs a running model

Collecting an imatrix means running the model — the DwarfStar collector hooks the layer-major Metal prefill graph and accumulates sum(x[column]^2) per routed expert. That needs a loadable GGUF, which doesn't exist yet before the first quantization:

1. Quantize without imatrix → the plain file above.
2. Run it over the calibration corpus to collect the imatrix.
3. Re-quantize with --imatrix → the final file.

Budget disk for it: the 167 GB of source safetensors have to stay available for step 3, and each GGUF is 97.6 GB, so plan for ~360 GB free if you want to keep both builds around. Drop the intermediate after step 2 if you don't.

antirez collects on the Q4 build for Flash, but Q4 for 0731 is ~165 GB and won't stay resident on 128 GB. So this imatrix was collected on the 2-bit build — which is exactly what the upstream README does for DeepSeek-V4-Pro, for the same reason.

That's defensible because of the recipe itself: the router is F16, attention projections and shared experts are Q8_0. Only the routed experts are degraded, so the routing decisions observed during collection are essentially the full-precision model's.

Why a fresh imatrix and not the preview's

The collector records, for down tensors, the routed SwiGLU row after route weighting — so the statistics depend directly on which experts fire. 0731 moves a lot versus the preview on exactly that axis (DeepSWE 7.3 → 54.4, Terminal-Bench 61.8 → 82.7), and the calibration corpus is heavily agent/code weighted (1106 of 4690 prompts are agent, 2074 are source). A stale imatrix would be most wrong precisely on ffn_down_exps — the tensor the recipe protects with Q2_K because it's the most fragile. So it was recollected.

Calibration corpus

Upstream's, unmodified — gguf-tools/imatrix/dataset/rendered_prompts.txt from antirez/ds4. Not mine, and not regenerated:

ds4 commit	54b36ed
last commit touching the dataset	b166a73
rendered_prompts.txt	12 MB, sha256 1159b0e7f1eff1c9…
prompts	4690
tokens	~2.92M (bytes/4 estimate, per manifest.json)

Category mix, straight from upstream's manifest.json: source 2074, agent 1106, language 1024, translation 180, eval_reasoning 150, programming 48, general 40, long_context 36, algorithms 32.

Use the tracked file, don't regenerate it. build_ds4_imatrix_dataset.py builds part of the corpus from the ds4 repository's own C/Metal sources, so regenerating at a different commit yields a different corpus and a non-comparable imatrix. The tracked file was used verbatim, which is what makes this reproducible: check out 54b36ed, verify the sha256, and you have the same input.

Why the Hub sidebar says 284B

The model tree widget reads the GGUF metadata, which comes from the --template file — a preview-checkpoint build. So it reports 284B params and architecture deepseek4, the preview's numbers. The tensor data is 0731 (that is what --compare-tensor verifies), and the 304.1B figure derived from the safetensors headers above is the correct one for this checkpoint. Fixing the sidebar would mean rewriting the metadata block; it does not affect inference.

Template GGUF

deepseek4-quantize regenerates tensor bytes from safetensors but takes metadata, tokenizer, tensor order and logical shapes from an existing DS4 GGUF passed as --template. The preview-checkpoint GGUF was used. Safe here because the tokenizer is byte-identical across the two releases — tokenizer.json, tokenizer_config.json and generation_config.json share the same blob hashes on the Hub. The only config.json differences are four added DSpark keys (dspark_block_size, dspark_noise_token_id, dspark_target_layer_ids, dspark_markov_rank).

Pre-flight checks

Run before writing anything, as the upstream README requires:

--dspark-manifest   dspark_stages=3   unknown_dspark_tensors=0
--dry-run           n_tensors=1328    type_changes=0    97 591 747 168 bytes
--compare-tensor    blk.0.attn_q_a.weight → bytes match, hashes differ

type_changes=0 proves the recipe was reproduced exactly from the template. The --compare-tensor mismatch is the expected result and the whole point of the check: identical byte counts prove the shape and target type are right, differing hashes prove the new 0731 weights are actually being read and not the template's.

0731 also carries 4 705 mtp.* tensors for the DSpark speculative decoding module that the preview template knows nothing about. The dry-run confirms they're correctly excluded from the main model (1328 tensors out). Convert them separately with --dspark-support if you want --dspark.

Does the imatrix actually help?

Honest answer: not measurably, on general text. Three runs, each widening the scored window on the same held-out corpus:

ctx	tokens scored	ppl plain	ppl imatrix	delta	significance
512	480	5.580490	5.250970	−5.90 %	0.9 σ
8192	8160	5.020238	4.847609	−3.44 %	2.1 σ
32768	32736	4.848056	4.782034	−1.36 %	1.7 σ, p ≈ 0.098

The gap shrinks as the sample grows — −5.90 → −3.44 → −1.36 %. That is the signature of an effect collapsing toward zero, not of a real one being measured more precisely. At 32k tokens it is not significant at the conventional threshold.

So the imatrix build is not demonstrably better on Wikipedia prose. It is not worse either. If you were hoping for a number that justifies picking one file over the other on general text, this measurement does not provide it, and I am not going to dress it up.

Method, so it can be repeated:

./ds4 -m <build>.gguf --perplexity-file wikitext2_test_300kb.txt -c 32768

• Corpus: wikitext-2 raw test split, first 300 KB. Deliberately not rendered_prompts.txt — the imatrix was collected on that, measuring there would be circular. Verified: zero overlap with the calibration corpus.
• Identical file, context length and tokenizer on both runs, 32736 tokens scored out of 68186 read. Teacher-forced NLL, no sampling, so no seed to fix.
• Significance uses a conservative per-token σ ≈ 1.5 and treats the two runs as independent. They are actually paired — same corpus, same tokens, same order — so a proper paired test on per-token log-probs would have more power. ds4 --perplexity-file only returns the aggregate NLL, so that test could not be run here. The figures above are therefore a lower bound on significance, not an upper one.

⚠️ Not comparable to published wikitext perplexities. ds4 --perplexity-file scores a single context window, not a sliding window over the whole test set like llama-perplexity. These numbers are valid against each other and nothing else — and this GGUF cannot be loaded by llama-perplexity at all.

What this does not test. The imatrix targets the routed-expert distribution seen in agent and code work, and it records the routed SwiGLU row after route weighting. General prose exercises a different mix of experts. A held-out code/agent slice would be the measurement that matters for the case this build is actually meant for — it has not been run yet. Until it is, treat the two builds as equivalent and pick either.

Performance

Apple M3 Max, 128 GB. Single-run Metal CLI, --ctx 32768 --nothink --temp 0 -n 256, short prompt — the same conditions upstream uses for its own table:

	imatrix build
prefill	45.08 t/s
generation	27.01 t/s

For reference, upstream reports 58.52 / 26.68 t/s for the uniform q2 preview build on the same machine class. Generation matches; prefill is lower here because this is the mixed 2+4 bit recipe — layers 37-42 stay at Q4_K, so there are more bytes to move per token during prompt processing.

Memory, measured at load:

resident model	90.88 GiB
KV @ 32k ctx	0.61 GiB (raw 0.36 + compressed 0.25)
total planned	91.74 GiB
model residency	~35-40 s from SSD

Build cost on the same machine, for anyone reproducing it:

step	time
quantization pass (--threads 16, CPU-bound, GPU idle)	~1 h 05
imatrix collection, 1.5M tokens (Metal, GPU-bound)	~4 h
second quantization pass with --imatrix	~1 h 10

Caveats

• Community build, not endorsed by antirez or DeepSeek.
• Not scored against the official DeepSeek continuation vectors that antirez uses as a release gate (tests/test-vectors). The perplexity delta above is a same-engine A/B between these two files, not an absolute quality claim against the FP4 original.
• The imatrix was collected on the 2-bit build, not a Q4 one — see above for why that is a reasonable compromise, but it is a compromise.

Credits

• DeepSeek — base model and weights (MIT)
• antirez and the DwarfStar contributors — recipe, quantizer, imatrix pipeline, inference engine. This repo is a straight application of their work to a newer checkpoint.
• llama.cpp / GGML — quant formats and the groundwork all of the above rests on

License

MIT, following the base model's release terms.