jmilnz/DeepSeek-V4-Flash-0731-antirez-ds4-GGUF

DeepSeek-V4-Flash-0731 — Mixed Q2/Q4 GGUF for ds4 (128 GB Macs)

Mixed-precision GGUFs of DeepSeek-V4-Flash-0731, built for antirez/ds4 (DwarfStar) on 128 GB Apple Silicon. Reproduces the canonical antirez "layers 37-42 Q4 / everything else Q2" recipe, applied to the 0731 weights.

The file

DeepSeek-V4-Flash-0731-Layers37-42Q4K-mixed-realimatrix-v2.gguf — 91 GiB (97,426,596,736 bytes), quantized with a real activation imatrix.

An earlier build of this same recipe used the quantizer's synthetic weight-energy fallback instead. It has been removed in favour of this one, which measures better on every metric tested (see Quality below). The two had an identical tensor-type layout and identical runtime cost — they differed only in which quantization codes were chosen for the routed experts.

Why this exists

deepseek-ai/DeepSeek-V4-Flash-0731 ships as ~155 GB of fp8 safetensors — too large for a 128 GB Mac alongside KV cache and the OS. These files compress the 0731 snapshot to ~91 GB using antirez's mixed-precision strategy, so they drop straight into an existing ds4 setup.

Quantization recipe

Tensors	Quant	Rationale
Routed MoE experts, layers 37–42 (the "hot" layers)	Q4_K	Higher fidelity where it matters most
Routed MoE experts, all other layers	IQ2_XXS (~2-bit)	The cold bulk, aggressively compressed
Down projections	Q2_K	2-bit
Attention projections	Q8_0	Small, fires every token — keep sharp
Shared experts	Q8_0	Always-on dense path
Output head	Q8_0	Critical, small

Verified layout: 18 × Q4_K + 37 × Q2_K + 74 × IQ2_XXS = 129 routed-expert tensors across 43 layers.

v2 is built in a single deepseek4-quantize pass, using --tensor-type overrides to promote layers 37-42 rather than quantizing a separate ~153 GiB Q4 donor and splicing it in:

args=()
for L in 37 38 39 40 41 42; do
  for T in gate down up; do
    args+=(--tensor-type "blk.$L.ffn_${T}_exps.weight=q4_k")
  done
done

gguf-tools/deepseek4-quantize \
  --hf   /path/to/DeepSeek-V4-Flash-0731 \
  --template DeepSeek-V4-Flash-0731-Q2-imatrix.gguf \
  --imatrix  DeepSeek-V4-Flash-0731-routed-moe-real.dat \
  --out  DeepSeek-V4-Flash-0731-Layers37-42Q4K-mixed-realimatrix-v2.gguf \
  "${args[@]}" --threads 16

This produces a byte-identical layout to the splice route (same 1328 tensors, same 18 type changes) while skipping an entire quantize pass.

The imatrix, and a trap worth documenting

v2 uses a real 0731-native routed-MoE imatrix collected with ds4 --imatrix-out (129 entries = 43 layers × {gate, down, up}, 2752 chunks — comparable calibration depth to antirez's own reference imatrix). v1 used the quantizer's synthetic weight-energy fallback.

Do not pass --imatrix-strict with a routed-MoE imatrix. The first v2 attempt did, and produced a silently truncated 73 GiB file. The flag means "fail if a quantized tensor has no matching imatrix vector" — but a routed-MoE imatrix by construction only covers routed experts. It therefore exit(1)s on token_embd.weight, the first non-expert tensor in write order, at byte offset 77,922,699,264 (72.6 GiB), killing the process mid-write. This is deterministic: strict mode can never succeed with a routed-MoE imatrix. Plain --imatrix is correct usage — the fallback for non-expert tensors is the intended path, not silent data loss.

Two related gotchas:

• Template must match the revision. Templating off the older chat-v2 GGUF instead of the 0731 Q2 shifts the planned size by ~165 MB.
• The tool prints decimal bytes; ls -lh prints GiB. A correct file is 97,426,596,736 B, which ls -lh shows as 91G. A real-imatrix build is also exactly 288 bytes larger than a non-imatrix one — those are the four quantize.imatrix.* provenance keys, not corruption.

Quality: real imatrix vs the synthetic build it replaced

Scored with gguf-tools/quality-testing/score_official against the 100 tracked DeepSeek API continuations in data/flash, compared via compare_scores.py. The metric is target-token negative log likelihood — how much probability each GGUF assigns to the official continuation, token by token. This avoids judging quality from a single sampled answer.

Metric	v1 (synthetic)	v2 (real imatrix)
Mean NLL (lower is better)	0.564260	0.522266 (−7.44%)
Cases won (of 100)	38	62 (0 ties)
First-token match	44/100	53/100
Avg greedy longest-common-prefix	4.770	5.490 tokens

All four metrics move the same direction, which is stronger evidence than the mean alone — a mean shift can come from a couple of outliers, but a 62/38 case split with better first-token agreement and longer greedy prefixes indicates a broad improvement.

Honest caveat on strength of evidence: the 62/38 split is exact-binomial p = 0.0105 — significant at p < 0.05, but not at p < 0.01, and this is a single 100-case run. Treat it as a real and consistent improvement, not a dramatic one. v2 still loses 38 cases; it wins more of them and loses them by less.

Hardware / performance

Tested on Apple M5 Max, 128 GB unified memory:

• Model maps to ~92.9 GB; loads with comfortable headroom for KV cache
• ~34.7 tokens/sec generation, ~23–58 tokens/sec prefill (ds4, Metal)
• Native ds4 --ctx 200000 works

Throughput figures were measured on v1. v2 has an identical tensor-type histogram, so runtime cost is unchanged — the imatrix affects which codes are chosen, not how many bits are read per weight.

Usage

# ds4 (DwarfStar) — intended runtime
./ds4 -m DeepSeek-V4-Flash-0731-Layers37-42Q4K-mixed-realimatrix-v2.gguf --ctx 200000

# As a drop-in for an existing ds4 install:
ln -sfn /path/to/this/file.gguf ds4flash.gguf
./ds4-server --ctx 200000

Not a general GGUF. Like all antirez-recipe files, this requires the ds4 engine's specific tensor layout — it will not load in a generic llama.cpp/Ollama build.

Comparison vs prior revision (0730)

Basic reasoning (greedy, temp=0) is equivalent on math, coding, and knowledge prompts. The 0731 finetune shows a more verbose structured "show your work" style. Across four agentic tasks via ds4-agent (file read, CSV filtering, log scanning, and an implement→write→run→verify loop), 0730 and 0731 were equivalent — 0731's distinguishing behavior was more transparent self-monitoring and error recovery rather than flawless execution. A rigorous multi-turn agent benchmark would be needed to confirm 0731's "enhanced agentic capabilities" claim; that probe only established parity at the basic-to-intermediate level.

Known caveats

• DSpark/MTP speculative decoding: 0731 adds dspark_* config fields. Base inference works fully; whether ds4's MTP path exploits 0731's DSpark config is unverified. The speculative-decoding speedup may not transfer without engine work or a DSpark-support companion GGUF.
• Built and tested on M5 Max only; behavior on other Apple Silicon (M3/M4) is expected to be equivalent but not separately validated.
• Quality evaluation is a single 100-case NLL run (see caveat above).

Acknowledgements

Recipe and tooling (deepseek4-quantize, the quality-testing harness, the mixed-precision strategy) are entirely from antirez/ds4. This is a community quant applying that tooling to the 0731 weights, not an official DeepSeek or antirez release.