kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF

Qwen3.8-Flash-Next — ROCmFP4 STRIX GGUF — AMD Ryzen AI Max+ 395 / Strix Halo / gfx1151

STRIX is the quality tier of the 4-bit family: Q4_0_ROCMFP4_STRIX — attention K/V on the base Q4_0_ROCMFP4 type instead of the fast layout, the per-layer embedding (PLE) table at Q8_0, token embeddings at Q6_K, and a Q6_K output head. Quantized directly from my own BF16 conversion of the release weights. 5.51 bpw overall, 113.5 GiB.

The PLE table is the whole story here: at Q8_0 it alone is 48.8 GiB. That is what buys this tier its quality margin over FAST, and it is why the file lands at 5.51 bpw.

tensor	type	size
attention K/V	Q4_0_ROCMFP4 (base type, not fast)	~2.5 GiB
MoE expert + remaining weights	Q4_0_ROCMFP4 family	~61 GiB
per_layer_token_embd.weight (the PLE table)	Q8_0	48.8 GiB
token_embd.weight	Q6_K	0.9 GiB
output.weight (lm head)	Q6_K	0.5 GiB

The Q6_K head

output.weight is kept at Q6_K, never 4-bit. An unprotected head ruins a 4-bit build: every token you sample passes through the lm head, so its quantization error lands directly in the argmax. On a sparse-MoE model the head is also one of the few dense matrices left, which makes its error stand out even more. It is 0.3% of the master weights — pinning it to Q6_K costs under half a GiB here and removes that entire error class.

⚠ Patched llama.cpp required

These files need a patched llama.cpp — PR #27742 merged into the ROCmFPX fork. Stock builds will NOT load them: both the qwen4exp architecture support and the Q4_0_ROCMFP4_* tensor types live in that fork. Build it for gfx1151:

-DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON

Measured

On my box — Ryzen AI MAX+ 395 (gfx1151, Radeon 8060S, ROCm 7.2.4), full offload -ngl 999, 49/49 layers on GPU (the server's own load log: load_tensors: offloaded 49/49 layers to GPU):

• generation: 14.5 tok/s (single stream, greedy)
• prompt processing: not measured yet

Slower than the 4-bit tiers — that is the Q8_0 PLE table: more bytes per token, bought on purpose.

Files

Distributed as a single 113.5 GB file: Qwen3.8-Flash-Next-ROCmFP4-STRIX.gguf.

This tier cannot be sharded under the 50 GB mark. The PLE table at Q8_0 is a single 54.4 GB tensor, and GGUF cannot split one tensor across shard files — any shard set necessarily contains a file of at least 54.4 GB. So I ship one whole file instead; the Hub takes single files up to 500 GB.

file	size
Qwen3.8-Flash-Next-ROCmFP4-STRIX.gguf	113.5 GB

Usage

llama-server \
  --model Qwen3.8-Flash-Next-ROCmFP4-STRIX.gguf \
  --host 127.0.0.1 --port 8080 \
  --n-gpu-layers 999 --fit off --no-warmup \
  --ctx-size 2048 --threads 16 --jinja

Memory

The weights are 113.5 GiB. On my 128 GB Strix Halo all 49 layers place with -ngl 999 while another ~25 GB model is resident — unified memory absorbs it — but a clean GPU makes placement obvious. Plan for ~115 GiB free.

Acknowledgements

This build would not exist without the work below. Please star and follow these projects — the quantisation format used here is their engineering, not mine.

ROCmFPX — the fork that defines the ROCmFP4 / ROCmFPX tensor formats and carries the qwen4exp architecture support merged from PR #27742. Every ROCmFP4 file in this repository was produced with its llama-quantize and runs on its runtime. Licensed MIT, based on upstream llama.cpp.

llama.cpp — ggml-org and contributors The inference engine, GGUF format and conversion tooling everything here is built on.

AMD ROCm — the compute platform these builds target (ROCm 7.2.4 on gfx1151).

Qwen team — the base model. See base_model for the source release; license is qwen-community-1.0.