ddalcu/Qwen3.8-Flash-Next-MLX-Serve-mixed-4-8bit

Qwen3.8-Flash-Next for mlx-serve (4-bit experts, 8-bit rest)

mlx-serve pack of Qwen/Qwen3.8-Flash-Next, the Qwen4 preview architecture (model_type: qwen4_exp). Runs on a 128 GB Mac with about 75 GB resident. Includes the MTP head and the vision tower (image and video input).

mlx-serve --model ddalcu/Qwen3.8-Flash-Next-MLX-Serve-mixed-4-8bit --serve

Measured on an M4 Max 128 GB with mlx-serve 26.8.11: ~75 GB resident, decode ~60 tok/s serial and 78 tok/s with MTP (--mtp; +41% on code, a few percent slower on prose), prefill ~730 tok/s, a needle at 24.8k tokens recovered with sparse attention engaged. Prefix cache on, images included.

What is different about this model

This is not a Qwen3.5-style pack. Three things around the usual GDN + MoE trunk:

• Gated residual streams. The residual is 4 streams wide (4 x 2560). Every block reads a sigmoid-mixed average of the normalized streams and writes back through per-stream scalar gates. The final mixer replaces the usual final norm.
• N-gram embedding (51B parameters). A second embedding table indexed by hashed bigrams and trigrams of the token ids: 16 heads, each a prime-sized bucket space of ~20M rows, 160 dims per row, injected once before layer 1. It is a lookup, no compute, which is why Qwen quotes the model as 125B: the full checkpoint is 125B trunk + 51B n-gram + 4B MTP = 180B (360 GB bf16).
• Qwen Sparse Attention. Past 2048 tokens each attention layer only reads the 512 most relevant 4-token blocks per query (picked by a small indexer), plus the query's own partial block. Attention cost stays flat with context. Native 262k context.

How this pack stores the n-gram table

The 51B table is NOT in the safetensors shards. It is one merged 4-bit table in ngram_table.bin (32.0 GB, safetensors format, .bin so nothing mlx-loads it). mlx-serve mmaps the file and, per token, dequantizes the 16 rows it needs on the CPU (16 x 80 bytes) and uploads only the resulting 2560-vector. The table never becomes resident: its cost is page cache, which the OS evicts as needed. That is the difference between this pack and mlx-lm style packs that ship the table as 128 quantized tensors and load it onto the GPU (+32 GB resident, ~107 GB total for a 4-bit pack).

Expected effect: decode speed unchanged (16 tiny reads against a ~20 ms step), cold-cache prefill of very long prompts may pay up to ~1 s per 8k tokens of random reads on the SSD, warm cache is free. No user-space cache is needed, the page cache already is an LRU over exactly this access pattern.

Widths

tensors	width
routed experts (512 x 48 layers, the 121B)	4-bit, group 64
attention, GDN, hyper-connections, indexer, shared experts	8-bit, group 64
lm_head	8-bit, group 64
embed_tokens	4-bit, group 64
n-gram table	4-bit, group 32 (row width 160)
routers, inject gates, norms, convs, SSM state	bf16
MTP head	same policy as the trunk

Every (1 + w) RMSNorm has the +1 folded into the stored weight; depthwise convs are transposed to MLX's [C, K, 1]; experts.gate_up_proj is split into switch_mlp.gate_proj / up_proj. The vision tower ships dense bf16 in model-vision.safetensors (~0.9 GB).

Serving notes

• Memory. ~75 GB resident plus KV cache. mlx-serve sizes the context to what fits; --kv-quant 8 halves the cache.
• MTP. The checkpoint's own 1-layer speculative head is loaded from the pack (--mtp or per-request "enable_mtp": true). Measured on an M4 Max vs serial decode: code +41%, prose -4%, an 8.5k-token prompt -4%. It is opt-in for now because the win depends on the prompt.
• v1 limits in mlx-serve. One request at a time (no batched decode), PLD/DFlash speculation off (MTP is the speculative path). Prefix cache is on, images included, so follow-up turns skip the re-prefill. Very long prompts (past ~64k) want a smaller --prefill-chunk because the sparse-attention selection is built per chunk.
• Thinking is on by default ("enable_thinking": false turns it off). Tools use Qwen3.8's XML call format; mlx-serve parses and schema-coerces it.
• Images and video go through the Qwen3-VL-style tower (model.visual.*, dense bf16). MTP is declined on image turns (serial decode).

Conversion

tests/convert_qwen38_flash_next.py in the mlx-serve repo. It streams the 360 GB bf16 checkpoint shard by shard from the Hub (download, quantize, delete), so it converts on a machine with ~150 GB free. The engine was validated against HF transformers (trunk) and the vLLM/SGLang MTP math on a tiny random model before the full conversion.