unsloth

/

Qwen3.8-Flash-Next-GGUF

like 501

Follow

Unsloth AI 33.4k

Tuning results on RTX 5060 Ti 16GB / Ryzen 9 9950X / 128GB DDR5-5600, UD-IQ4_XS, llama.cpp PR #27742 @ af1ffaf37 (build 10706).

The speculative decoding variant matters far more than anything else, and the commonly recommended one isn't the best. Same server, one variable changed, measured on a copy-heavy code edit and on freeform prose:

⚠️ ngram-cache looks best on code and costs −24% on prose. If you only benchmark the workload speculation helps, you'll pick it and ship a silent regression on everything else. ngram-map-k takes nearly all the gain with prose fractionally faster than no speculation at all.

Other measured findings:

• --spec-draft-n-max has no effect on any n-gram variant (tested 2, 4, 6, 8, 12 — identical speed and identical acceptance every time).
• KV q4_0 → q8_0 is free (20.75 vs 20.70 code). Only 12 of 48 layers are full-attention with 2 KV heads, so KV is tiny. Take the better-validated q8_0.
• -c 131072 measured no slower than 32K (32.00 vs 30.66). 262144 hits the VRAM ceiling — 15.6/16 GB and drops to 22.54.
• Threads are flat (t8 20.98 / t12 21.46 / t16 21.13). CPU sits ~37% and the cores are stalled on memory, not computing.
• --tensor-read-lazy auto did nothing here (30.59 vs 30.66, RAM 89 GB either way) — it's redundant with --load-mode none, which forces the model resident anyway.
• --load-mode none is a prefill lever only: ~45% better PP, zero effect on decode, at the cost of 43 s load vs 10 s.

Expert residency is worth only ~0.66% speed per point, so filling the GPU barely helps:

Two consequences: a second 16GB card isn't worth buying for this, and guides suggesting --n-cpu-moe 38 or 36 won't fit on a 16GB card — expert weights are 1.34 GB/layer with an 8.1 GB zero-expert baseline.

Degradation with a genuinely loaded cache (real mixed documents and code, cache_prompt=false): 19.43 t/s at 1.1K tokens in → 19.03 at 8K → 18.00 at 16K → 16.40 at 29K. That's −15.6% across the span with prefill flat at ~123 t/s — it degrades gracefully rather than falling off a cliff.

Note --spec-type draft-mtp fails with "model doesn't contain MTP layers" — the MTP head isn't in the GGUF.

Config landed on (128K context, ~14.3 GB VRAM, ~90 GB RAM, 43 s load):

-ngl 999 --n-cpu-moe 44 -c 131072 -np 1 -fa on \
-b 2048 -ub 512 -ctk q8_0 -ctv q8_0 -t 12 \
--load-mode none --spec-type ngram-map-k \
--jinja --reasoning-format deepseek

Full data, harness scripts and raw JSON: https://github.com/solarkyle/qwen38-flashnext-16gb

Single machine, one run per config unless noted — differences under ~3% are noise.