sh0wie/Qwen3.8-Flash-Next-REAP-384-MLX-4bit

Qwen3.8-Flash-Next REAP-384 (MLX, 4-bit)

	Disk	Resident memory	HumanEval pass@1
Base Q4 (512 experts)	98 GB	97 GB	93.9%
This build (384 experts)	80 GB (-18%)	51 GB	92.1%

The resident figure is measured with the n-gram table served from NVMe (see below); loaded fully in memory this build is ~80 GB resident.

Qwen3.8-Flash-Next with 384 of 512 experts per MoE layer, pruned with REAP saliency calibrated on the quantized weights, on the machine that serves them. This is the shallowest cut in our pruning ladder that we publish: at 1.8 points off the stock conversion on HumanEval it is the closest-to-stock option when your disk budget allows 80 GB. If you want the best quality-per-GB instead, take sh0wie/Qwen3.8-Flash-Next-REAP-288-MLX-4bit (68 GB, 91.5%). Loads on stock mlx-vlm with no patches.

• 180B-parameter class: 125B main model, 51B n-gram embedding table, 48 layers alternating Gated DeltaNet and Qwen sparse attention, each with a 384-expert MoE routing top-10
• Affine 4-bit throughout (group size 64; n-gram table group size 32)
• Multimodal weights (vision tower) are intact but only text quality has been evaluated

Run it

Requires mlx-vlm with qwen4_exp MTP support (git main after 2026-08-27, or any release that includes it):

pip install git+https://github.com/Blaizzy/mlx-vlm.git

Generate:

python -m mlx_vlm.generate \
  --model sh0wie/Qwen3.8-Flash-Next-REAP-384-MLX-4bit \
  --prompt "Refactor this function to add input validation." \
  --max-tokens 512

Serve (OpenAI-compatible):

python -m mlx_vlm.server \
  --model sh0wie/Qwen3.8-Flash-Next-REAP-384-MLX-4bit --port 8080

Speculative decoding with the model's own MTP head, using the companion drafter sh0wie/Qwen3.8-Flash-Next-MTP-Drafter-MLX-bf16:

python -m mlx_vlm.generate \
  --model sh0wie/Qwen3.8-Flash-Next-REAP-384-MLX-4bit \
  --draft-model sh0wie/Qwen3.8-Flash-Next-MTP-Drafter-MLX-bf16 \
  --draft-kind mtp \
  --prompt "..." --max-tokens 512

A note on speculative speed: the drafter's acceptance rate is healthy (~44-68% depending on sampling), but the net speedup depends on how cheaply your hardware runs the verification pass. M5-class GPUs report 1.5-2.6x; on M4 it is roughly break-even. Quality is unaffected either way, since the target model verifies every drafted token.

Where 384 sits on the ladder

We measured an eleven-point pruning ladder; 384 is the runner-up, and the right choice when you are optimizing for quality rather than footprint:

Build (experts)	Disk	HumanEval pass@1
512 (stock conversion)	98 GB	93.9%
384 (this build)	80 GB	92.1%
320	72 GB	90.9%
288 (efficiency pick)	68 GB	91.5%
256	65 GB	88.4%

All legs ran the same harness on the same machine: 164 HumanEval problems, unit-test verified, one run per build. Routing width is untouched at the trained top-10. The kept-expert manifest ships in this repo as reap_kept_experts.json, which makes the prune reproducible from the source conversion.

The NVMe n-gram mode

Per token the model reads only a few hundred bytes of the 51B n-gram table, so the table does not need to be resident. A row-granular disk-read patch (ours, not yet upstream in mlx-vlm) serves it from NVMe with logits bit-identical to the in-memory path, dropping resident memory from ~80 GB to 51 GB measured (MLX peak 50.6 GiB during a short greedy run).

Provenance and what was fixed

• Qwen/Qwen3.8-Flash-Next: upstream weights
• Sawfwair/Qwen3.8-Flash-Next-MLX-4bit: MLX affine 4-bit conversion (group size 64; n-gram table group size 32)
• This build: REAP expert pruning 512 -> 384 per layer, calibrated on-device over ~686K tokens of agentic-coding traffic

Two defects of the source conversion are corrected in the weights, so no loader patches are needed: RMSNorm tensors stored un-centered (+1) are re-centered to the zero-centered convention the runtime's (1 + w) norm expects, and the n-gram table tensors plus their per-tensor quantization overrides are renamed shard_N -> shards.N to match the runtime module path. Everything else is byte-identical to the pruned source. Stock-runtime logits on this build match our patched-runtime reference (max abs diff 0.0 at the final prefill position).

Limitations

• Calibration reflects one team's agentic-coding distribution. Retention numbers should not be read as general-domain; domains far from code may degrade more.
• Single-run evaluations, no confidence intervals. Differences of a point or two between neighboring builds are within noise.
• Vision input is untested after pruning.

License

Qwen Community License 1.0, inherited from the base model; see LICENSE.