unsloth

/

Qwen3.6-35B-A3B-MTP-GGUF

like 392

Follow

Unsloth AI 23.1k

Hardware: AMD RX 7800 XT (gfx1101) · 16 GB VRAM
OS: Xubuntu 24.04
ROCm: 6.4.0 kernel 6.17.
Fork: NJannasch/llama.cpp mtp-turboquant branch
Model: unsloth/Qwen3.6-35B-A3B-MTP-GGUF — Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf
Date tested: 2026-05-18

Tested on a consumer desktop, not a server rack.

Written by Claude Code · Edited by User

---

Most writeups on MTP + TurboQuant are NVIDIA. This is a ROCm-specific report from an RDNA3 card. There are a few things that will bite you that I didn't see documented anywhere. Posting this so you don't lose hours to the same issues.

---

Why This Combination

UD-IQ3_XXS quantization is 13 GB on disk — fits in 16 GB VRAM with room for KV cache.

The catch: at Q3-range quantization the KV cache overhead is proportionally larger. Without TurboQuant, a 32K context window adds another ~640 MB of f16 KV cache on top of the model. With turbo4 KV (-ctk turbo4 -ctv turbo4), that same 32K context costs ~170 MB. That's the margin that makes longer conversations viable on a 16 GB card.

MTP (Multi-Token Prediction) is baked into Qwen3.6-35B-A3B at training time — nextn_predict_layers = 1 in the model metadata. The NJannasch fork activates it with --spec-type draft-mtp.

---

Build

Single GPU setup (most people): straightforward. Build for your card's gfx target only.

git clone --branch mtp-turboquant --depth 1 https://github.com/NJannasch/llama.cpp
cd llama.cpp
cmake -B build \
  -DGGML_HIP=ON \
  -DAMDGPU_TARGETS="gfx1101" \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server -j$(nproc)

Build time: ~20 minutes on a mid-range CPU.

Multi-GPU setup — read this before you build:

If you have a secondary AMD GPU in the system, the temptation is to list both targets in AMDGPU_TARGETS. Don't.

This fork does not have a rocBLAS TensileLibrary path for every gfx target. Specifically, gfx1032 (RX 6600 XT / 6700 XT family) is not covered. If you include it, the build succeeds but inference crashes at runtime:

rocBLAS error: Cannot read TensileLibrary.dat: Illegal seek for GPU arch: gfx1032

This is not a VRAM issue. The crash happens because the compiled binary has no kernel path for that architecture. Not due to memory size. A 6600 XT with unlimited VRAM would crash the same way. Mainline llama.cpp handles gfx1032 fine via a gfx1030 fallback — this fork does not.

Fix: build for your primary card's target only, and restrict GPU visibility at launch with HIP_VISIBLE_DEVICES (see Launch Flags below). The secondary card stays available for other processes; this binary just won't touch it.

---

Model Download

The filename in the MTP GGUF repo does not include "MTP" in the filename itself:

wget -c -O Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf \
  'https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf'

hf download worked but left an .incomplete file at exit code 0 in my session. Use wget -c with the direct resolve URL to be safe.

---

Launch Flags That Work

HIP_VISIBLE_DEVICES=1 ./build/bin/llama-server \
  --model Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf \
  --port 8080 \
  --host 127.0.0.1 \
  -ngl 999 \
  --n-cpu-moe 0 \
  --ctx-size 32768 \
  --jinja \
  -np 1 \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  -ctk turbo4 \
  -ctv turbo4 \
  --reasoning-budget 512

HIP_VISIBLE_DEVICES=1 assumes the 7800 XT is your second GPU. Adjust to 0 if it's your only GPU.

---

Results

Benchmark (API level)

At 32K context turbo4 saves ~470 MB vs f16. Speed barely changes between ctx 8K and 32K with turbo4 — that's the point.

MTP acceptance at 86-100% is strong. Qwen3.6 was trained with MTP so draft quality is high.

---

UX Testing (web UI, thinking mode on, --reasoning-budget 512)

Tested against the built-in llama-server web UI after full service setup. Scoring is subjective: Pass / Degraded / Fail against the criterion listed.

Live data note: The model has no web access. Without a --reasoning-budget cap, open-ended factual queries trigger an uncapped thinking loop where the model enumerates everything it knows from training data. This accumulated hundreds of tokens of internal reasoning, driving generation speed down from ~110 t/s to ~9 t/s as attention overhead compounded. With --reasoning-budget 512 of the thinking cuts that off Then the model states plainly that it can't provide live data — which is the correct answer. The degraded score reflects the behaviour without the budget; with it, this category becomes a Pass with a graceful "I don't have live data" response.

Thinking mode: The reasoning phase is not visible in the web UI by default — responses appear after the think completes. For tasks where the thinking budget is hit, the model produces a brief response from wherever reasoning ended. This is working as designed. For direct conversational use, disable thinking per-request via the API ("chat_template_kwargs": {"enable_thinking": false}) or set --reasoning-budget 0 at launch.

---

Issues to Watch For

1. The Silent CPU Fallback — Most Dangerous

Symptom: Model appears to load normally. VRAM reads correctly (14+ GB used). But generation is 5-10 t/s instead of 100+ t/s. CPU is pegged at 200-300% usage. GPU utilization reads 0%.

Cause: The ROCm runtime silently falls back to CPU compute when GPU initialization fails. The VRAM reading is misleading — the weights are loaded to GPU memory via mmap but computed on CPU.

How to catch it: Check the server log for compute buffer:

# GPU — correct
sched_reserve: ROCm0 compute buffer size = 493.00 MiB

# CPU — silent fallback
sched_reserve: CPU compute buffer size = 497.00 MiB

Also check startup for:

ggml_cuda_init: failed to initialize ROCm: no ROCm-capable device is detected

If you see that line, nothing is running on GPU regardless of what rocm-smi shows.

VRAM is not a reliable GPU-usage indicator. Always check compute buffer in the log.

---

2. ROCR_VISIBLE_DEVICES + HIP_VISIBLE_DEVICES Integer Conflict

If you're running multiple GPUs and use systemd (or any launcher that sets environment variables explicitly), watch out for this:

ROCR_VISIBLE_DEVICES and HIP_VISIBLE_DEVICES use different index spaces.

Setting both to 1 targets different physical devices. The ROCm runtime sees a conflict and reports no capable device — then silently falls back to CPU. This is the root cause of the silent CPU fallback above in systemd contexts.

Fix: Use HIP_VISIBLE_DEVICES alone. Don't combine both with integer indices.

# Correct — one variable, GPU-only index
HIP_VISIBLE_DEVICES=1 ./llama-server ...

# Wrong — conflicting index spaces
ROCR_VISIBLE_DEVICES=1 HIP_VISIBLE_DEVICES=1 ./llama-server ...

If you want UUID-based targeting (more stable across reboots), use ROCR_VISIBLE_DEVICES=<UUID> + HIP_VISIBLE_DEVICES=0 (since only one device is visible, it becomes index 0).

---

3. Hybrid Model Requires MTP to Load

Qwen3.6-35B-A3B is a hybrid architecture — it alternates between full attention layers and SSM (Gated Delta Net / Mamba-style) recurrent layers every 4 blocks. This is not a standard transformer.

In the NJannasch fork, attempting to load this model without --spec-type draft-mtp causes an assert crash during slot initialization:

GGML_ASSERT(rollback >= 1 && rollback <= (llama_pos) n_rs_seq) failed

Stack trace: llama_memory_recurrent::seq_rm → llama_memory_hybrid::seq_rm → common_context_can_seq_rm → server_context_impl::load_model

--no-warmup does not fix this. -np 1 does not fix this. Only adding --spec-type draft-mtp resolves it.

This appears to be a fork-specific bug where the recurrent sequence memory (n_rs_seq) is only initialized correctly when MTP is active. The model metadata contains nextn_predict_layers = 1 — this model expects MTP.

--spec-type draft-mtp -np 1 are both required, not optional.

---

4. Thinking Model + No Reasoning Budget = Recall Spiral

Qwen3.6-35B-A3B is a thinking model. By default, it reasons before answering. On open-ended queries (especially anything involving enumeration from training data — "what restaurants are in X", "list all Y"), it can run an unbounded think block that accumulates hundreds to thousands of tokens.

This causes a secondary performance problem: as the thinking context grows, the O(n) attention overhead across the model's 10 full-attention layers compounds. A fresh 16-token prompt runs at ~118 t/s. The same session after 1000 tokens of accumulated thinking: ~9 t/s.

Fix: Set a token budget for the thinking phase:

--reasoning-budget 512

512 tokens is enough for genuine multi-step reasoning. Not enough for exhaustive training-data recall. Adjust upward if you need deeper analysis on complex tasks.

To disable thinking entirely per-request via API:

"chat_template_kwargs": {"enable_thinking": false}

Note: the /no_think prompt token does not work with this fork/model. Use the API parameter.

---

5. gfx1032 (RX 6600 XT) Not Supported by This Fork — and It's Not a VRAM Issue

Clarifying this because it's easy to misread: the problem is not that the 6600 XT has 8 GB of VRAM and the model is 13 GB.

In standard llama.cpp builds, insufficient VRAM is handled gracefully — the runtime calculates how many layers fit on GPU and offloads the rest to CPU. Slow, but not a crash. That behavior works fine.

The crash here is different. If you have a multi-GPU system with a 6600 XT alongside your main card and expose both GPUs (e.g. by removing HIP_VISIBLE_DEVICES), you get:

rocBLAS error: Cannot read TensileLibrary.dat: Illegal seek for GPU arch: gfx1032

This happens because the mtp-turboquant fork's compiled rocBLAS kernels do not include a path for gfx1032. VRAM size is irrelevant — a hypothetical 6600 XT with 64 GB would crash the same way. The architecture simply isn't in this binary's kernel set.

The mainline llama.cpp binary handles gfx1032 via a gfx1030 fallback path. This fork does not. The fix is to restrict GPU visibility to your supported card only:

HIP_VISIBLE_DEVICES=<index of gfx1101 card> ./llama-server ...

If you only have a gfx1032 card and want to run this fork: it will crash. Use mainline llama.cpp instead — you'll get CPU offload for layers that don't fit, but no rocBLAS crash.

---

Thinking Mode

The model ships with thinking on. For direct conversational use, disable per-request:

{
  "chat_template_kwargs": {"enable_thinking": false}
}

For a persistent no-think default, set --reasoning 0 at launch (disables thinking for all requests).

---

What I'd Still Like to Test

• Higher ctx (65536+) with turbo4 — the VRAM math says it's viable but I haven't validated it
• Quantized KV types other than turbo4 (q8_0 baseline comparison on this fork)
• Whether the n_rs_seq assert is fixed in newer commits on the branch

---

TL;DR for AMD Users

• ✅ MTP works on ROCm gfx1101 at 86-100% acceptance — the NVIDIA-only assumption is wrong
• ✅ TurboQuant turbo4 KV works on ROCm — 160 MB → 47 MB at ctx 8K, 640 MB → 170 MB at ctx 32K
• ⚠️ Check compute buffer in logs — VRAM usage will look normal even if you're on CPU
• ⚠️ Don't combine ROCR_VISIBLE_DEVICES and HIP_VISIBLE_DEVICES with integers
• ⚠️ --spec-type draft-mtp -np 1 required to load — not optional for this hybrid model
• ⚠️ Set --reasoning-budget 512 or thinking loops will tank your t/s on open queries
• ⚠️ Build with gfx1101 only — this fork has no rocBLAS kernel path for gfx1032. Not a VRAM issue — a 6600 XT with unlimited VRAM would crash the same way. Mainline llama.cpp handles gfx1032 fine (CPU layer offload); this fork does not.

---

Relevant upstream discussion: TurboQuant KV Cache Compression — Full HIP/ROCm Port