DeepSeek-V4-Flash-JANGTQ-K

JANGTQ-K — the canonical max-quality JANG quantization of deepseek-ai/DeepSeek-V4-Flash for Apple Silicon, in the ~80 GiB tier.

Source	deepseek-ai/DeepSeek-V4-Flash
License	MIT, inherited from upstream
Format	JANG (MXTQ routed + affine non-routed, critical controls F32)
Profile	JANGTQ_K
Bundle size	~80 GiB (85.9 GB) across 80 shards
Tensor keys	2610
Context length	1,048,576 (1M)
Native cache schema	deepseek_v4_v7

What this is

The first DSV4-Flash JANGTQ bundle that survives the relaxed multi-turn reasoning_effort=max quality-chat gate. It pairs:

• 2-bit MXTQ routed experts on 38 of 43 layers,
• 4-bit MXTQ routed experts on the 5 layers that lost the most quality at 2-bit (L23, L25, L28, L34, L36), selected from a real-activation probe over all 43 layers,
• 8-bit affine for every non-routed module (attention, shared expert, Compressor, Indexer, embed, lm_head),
• float32 preserved for HC controls, attention sinks, and APE,
• MTP head dropped (DSV4-Flash ships one MTP module of ~6.5 B params; not used at inference).

Live Quality

Passed: 4-turn reasoning_effort=max chat gate. Correct arithmetic, memory recall (remembered HARBOR-17 and CERULEAN), final 3-bullet format honored, no loops, no empty visible answers. Paged DSV4 cache hit on turns 2–4 with 399 tokens saved per hit; block-L2 wrote composite DSV4 blocks. Speed: 14.6 – 17.6 tok/s on M3 Ultra Mac Studio.

Not yet passed: strict exact-copy gate. Short exact-marker rows can flip to neighboring BPE tokens. The failure reproduces with prefix-cache bypass and is not a UI, gateway, streaming, or disk-cache artifact. Raw-max (VMLINUX_DSV4_RAW_MAX=1) had a 2-turn smoke only and is not advertised as fully cleared.

Quantization Recipe

Category	Bits	Codec	Notes
Routed experts (38 layers default)	2	MXTQ	hash layers L0/L1/L2 included at 2-bit
Routed experts (5 lifted layers)	4	MXTQ	L23, L25, L28, L34, L36
Attention wq_a/wq_b/wkv/wo_a/wo_b	8	affine, gsz=32
Shared expert	8	affine, gsz=32
Compressor + Indexer + Indexer.Compressor	8	affine, gsz=32
embed_tokens + lm_head	8	affine, gsz=32
Norms / router gate / hc_* fn matrices	16	passthrough
hc_*_base, hc_*_scale, attn_sink, ape	32	source-f32
MTP head	—	dropped	drop_mtp=true; num_nextn_predict_layers=0

JANG vs MLX comparison

Benchmarks pending. To follow Eric's README standard (no TBD), this section will be filled in before any social promotion. The intended axes are:

• MMLU (reasoning + no-reasoning)
• HumanEval pass@1
• Korean / Chinese subset accuracy
• Decode tok/s on M3 Ultra, M5 Max
• Bundle size on disk

The MLX comparison baseline will be the smallest MLX-affine quant of DSV4-Flash that loads on the same hardware; if MLX has no published DSV4 quant at this size class, that absence will be stated.

Architecture (for downstream tooling)

• 43 decoder layers, no MTP.
• MoE: 256 routed + 1 shared per layer, top-6, routed_scaling_factor=1.5, topk_method=noaux_tc.
• MLA: q_lora_rank=1024, o_lora_rank=1024, head_dim=512, qk_rope_head_dim=64.
• Sparse Indexer: index_n_heads=64, index_head_dim=128, index_topk=512, compress_rope_theta=160000.
• HC controls: hc_mult=4, hc_sinkhorn_iters=20, hc_eps=1e-6.
• Sliding window 128; SwiGLU clip 10.0; rope_theta=10000.
• Context length 1,048,576 with YaRN.
• Reasoning modes chat / thinking; reasoning_effort ∈ {max, high, null}.
• Tool calling: DSML parser (<｜DSML｜> block).

Runtime contract

Bundle metadata pins:

• DSV4 native model family deepseek_v4, batch path DSV4BatchGenerator.
• Cache schema deepseek_v4_v7; layers 0,1 = KVCache, 2..42 = DeepseekV4Cache.
• Generic TurboQuant KV: off for DSV4 (composite cache).
• DSV4 pool quant: off by default; opt-in only with DSV4_POOL_QUANT=1.
• Paged cache block size: 256 (loader upgrades stale 64-token settings).
• Block disk L2: on.
• Chunked prefill: off; single-shot prefill is the safe path.

JANGTQ runtime sidecar

jangtq_runtime.safetensors carries the JANGTQ MXTQ codebook + signs tables required by the Swift loader. Six tensors total (codebook.{2048,4096}.{2,4} + signs.{2048,4096}.42).

Sampling defaults

temperature=0.6, top_p=0.95
repetition_penalty=1.0 (1.0 thinking, 1.05 chat)
max_new_tokens=4096

Files

• config.json — DSV4 HF config.
• jang_config.json — JANG profile, recipe, bit plan, runtime requirements, chat encoder, sampling defaults, lineage.
• model-00001-of-00080.safetensors … model-00080-of-00080.safetensors — sharded weights.
• model.safetensors.index.json — tensor → shard map (2610 keys).
• jangtq_runtime.safetensors — JANGTQ MXTQ runtime sidecar.
• tokenizer.json, tokenizer_config.json — preserved upstream tokenizer.
• generation_config.json — HF defaults.
• encoding/encoding_dsv4.py — DSV4 chat encoder (Python).
• LICENSE — MIT, upstream.

Lineage

Converter variant V3 (now collapsing into K), plan routed_only_worst5_23_25_28_34_36 (sha256 3db0b31fe6f1b19d3e00cfdd15572ebf3af950ef25e1e8622e1f2791b1977619). Originally staged as DeepSeek-V4-Flash-JANGTQ-V3-WORST5-F32; renamed to JANGTQ-K on 2026-05-11 as the canonical DSV4 max-quality tier.

Contact

eric@jangq.ai