AMD / Vulcan ; Lower T/S // Cpu offload of part of the model / fixes
I added 2 special "AMD/VULCAN" quants with "AMD" in the name in IQ4_XS and Q6_K. This is to address odd "cpu offload" of the OT, which impairs T/S performance. The "AMD" version has the OT in f16 [as opposed to bf16]. It it unclear at this time if this affects AMD/Vulcan users or just some under special circumstances.
Hi, I'm running the model (IQ3_M mtp) on an AMD RX 9070 XT with llama.cpp (Vulkan) and I'm also experiencing this issue. I would really like to use it, but it's just so slow compared to the base model (from Unsloth). It's running with 10 t/s while the base model is running with 50 t/s in comparison (both mtp). Unfortunately, I can not use the special IQ4_XS/Q6_K quants because they are too large to fit into the 16 GB VRAM. I've tried the AMD IQ4_XS quant, but it's running with just 10 t/s too. It's obviously not running entirely in VRAM though. I'm happy to help with testing.
Still experiencing the CPU offload issue on the AMD IQ4_XS quant on my 7900XTX too. Roughly a 60/40 split.
@rboluyt
Try dropping context down to 8k and see if this helps.
What AI App[s] are you using?
I've tried context windows from 8K to 65K, no change in the offload.
I typically use JanAI(0.84), both in-app, and through its OpenAI API server when I work on my custom storyboard(WIP) app(Tool? Platform? Not sure what I want to call it yet). I also observed the same offload in LM Studio Bionic 1.03 during my troubleshooting. Think both JanAI and LM Studio use Ollama though.
Hmm ; sounds like 16 bit is not supported and/or offloading to cpu for some reason.
The ideal is to use the quants with 16 bit as the OT is critical for final output.
I have put in a request to ask team Mradermarcher for "standard" / reg "imatrix" quants.
I tried to load the UD released Q4_K_XL model and compare to the AMD IQ4_XS. The graph splits of UD_Q4_K_XL is 2 and AMD IQ4_XS increased to 3 which cause the lower performance of AMD IQ4_XS. I'm using 7900XTX with Vulkan backend. After comparing the tensors definition of these two models and the loading logs from llama.cpp, the 7900XTX as Vulkan device did not support bf16 or f16 as output.weight type to prefered buffer. The UD_Q4_K_XL uses Q6_K as the tensor type instead which could be supported by Vulkan device. May be you can set it to Q6_K or Q8_0 for better compatibility while still provide reasonable precision afterall.
It may be related to a buffer allocation size limit. The output.weights tensor is 2425MB when quantized as f16 / bf16.
Requantizing that layer to f32 (4850MB) makes things worse. Requantizing that layer to q8_0 (1288MB) makes the problem go away.
edit: I believe I was able to reproduce the problem seperately by taking one of unsloth's models (Qwen3.6-27B-UD-IQ2_M.gguf) and requantizing the output.weights layer to be bf16.
With llama-server Vulkan build b10155 on windows, split layer-wise on two radeon d-gpus.
edit 2: using the command vulkaninfo | findstr /i "maxMemoryAllocationSize" indicates my system's maximum buffer allocation size is 0x80000000 or 2GB.
@LocoLlamaEnth You are correct, it's not because of the data type not support by Vulkan backend but the default limitation of buffer allocation size in the system. I tried to override the default 2GB limitation with setting environment variable GGML_VK_FORCE_MAX_BUFFER_SIZE=3221225472 to 3GB and restart the llama.cpp server. The compute graph splits became 2 again and the infrerence speed increased from ~30 tps to ~60 tps with MTP draft by 2. It's still slower than UD_Q4_K_XL with the same MTP draft configuration which could reach ~75 tps. However, considering the output.weight is f16 which must introduced more cost for inferencing. Looking forward to see a q8_0 version of this model.