Love the prose it writes, but it constantly (AMD) splits 60% to the CPU, 20-30% to my GPU
This behavior seems specific to this model too, other models I have installed load my 7900XTX to 100% when working. This model splits 60 to 80% to the CPU, regardless of the settings. I typically set my models up with JanAI, with the GPU layers set to -1 or 100. MTP on or off. It makes no difference, it constantly dumps to my CPU. I love the prose, but when it loads so much to the CPU, I barely get 10-15 tok/s with it.
I was going to try LM Studio, to see if I get the same effect. But does anyone know why this happens?
Same problem, 80%gpu 20%cpu, which makes 2x slower gen speed(seemingly). Ryzen iGPU 32gb unified memory. All other models run 100% gpu speed.
vulkan llama.cpp
I cycled back a few llama.cpp versions; From b9967 to b9743 to b9568, without any change. I also swapped the backend from 'win-vulkan-common_CPUs-x64', my usual default, to 'Win-vulkan-x64', also without success. I also installed LM Studio Bionic, loaded the model and tried it, and got the same CPU/GPU split. Other models don't do this, including 'DavidAU/Qwen3.6-27B-Heretic-Uncensored-FINETUNE-NEO-CODE-Di-IMatrix-MAX-GGUF', which has been my goto writer agent lately.
I'd really like to figure this out, if anyone has any ideas, since this Fable Fusion version writes noticeably better prose. But this split to the CPU craters the tok/s.
Yea, we need a solution, because this model might be a good replacement for stock qwen27b. But because my system already is very slow, no real gpu just igpu, 7ts/s for q4km mtp, I don't want it more slower than that
I would probably suggest to try the latest beta build of llama.cpp. i use an nvidia 5090 and i have no cpu offloading. but it may have something to do with this line of the model card:
In addition the output tensor (10-20% of output) was modified to full precision - 16 bit - for all quants.
it could be that whatever runtime loader you guys are using or however amd's handling of it, it is not compatible. but try the latest llama.cpp beta build to rule that out.
I would probably suggest to try the latest beta build of llama.cpp. i use an nvidia 5090 and i have no cpu offloading. but it may have something to do with this line of the model card:
In addition the output tensor (10-20% of output) was modified to full precision - 16 bit - for all quants.
it could be that whatever runtime loader you guys are using or however amd's handling of it, it is not compatible. but try the latest llama.cpp beta build to rule that out.
Interesting. I grabbed llama.cpp 'b10107' and will test it when I have time later today.
Interesting. I grabbed llama.cpp 'b10107' and will test it when I have time later today.
no, that won't help for sure. llama.cpp is fine
Interesting. I grabbed llama.cpp 'b10107' and will test it when I have time later today.
no, that won't help for sure. llama.cpp is fine
Correct. It didn't. No change, still a 60/40 split.
Please try the "LOW" version[s] of the quants and see if this addresses CPU issue.
Please report back and I will proceed with possible fix[es] .
Issue seems to be related to BF16 Output tensor, and this being "processed" on CPU ; which maybe due to F16/BF16 "bug".
This same issue affected Nvidia GPUs at one point.
Please try the "LOW" version[s] of the quants and see if this addresses CPU issue.
Please report back and I will proceed with possible fix[es] .Issue seems to be related to BF16 Output tensor, and this being "processed" on CPU ; which maybe due to F16/BF16 "bug".
This same issue affected Nvidia GPUs at one point.
I'm using the 'IQ4_XS' versions currently, tried both the MTP and non-MTP versions. I can try the next step down, IQ3_M<?>, though I don't normally go below Q4s.
No luck with the IQ3_M version. It actually displays higher GPU Compute utilization during the Reading and Thinking parts, before generating output.
You need the version[s] with "LOW" in the quant name.
These have lower OT settings.
You need the version[s] with "LOW" in the quant name.
These have lower OT settings.
Found it, my bad. Still learning to navigate Huggingface's screens. Grabbing 'Qwen3.6-27B-Fable-Fus-711-UnHeretic-NM-DAU-NEO-MAX-NEO-LOW-MTP-IQ4_XS.gguf'.
Using 'Qwen3.6-27B-Fable-Fus-711-UnHeretic-NM-DAU-NEO-MAX-NEO-LOW-MTP-IQ4_XS.gguf' seems to work. 100% load on my 7900XTX, normal load (~10%) on the CPU. Token speed jumps from 15 tok/s to 40 tok/s.
Is that F16/BF16 "bug" something that can be fixed or something that's going to be inherent with your Fable Fusion 711 models?
CPU offload issues - Vulcan and/or AMD related:
I am uploading(ed) TWO quants:
IQ4XS and Q6 with "AMD" in the name.
This changes the output tensor to f16 (from bf16).
This should give you close to max performance, without CPU offloading.
For user with the cpu offload issue, can you post feedback and see if this addresses the cpu offload issue(s)?
These will be uploaded about 1/2 hr or so from this post's timestamp.
Thank you ;
For user with the cpu offload issue, can you post feedback and see if this addresses the cpu offload issue(s)?
I'm afraid it's the same problem again... (AMD-MTP-IQ4_XS.gguf) 😢
But at least the LOW quant works... And it seems to perform well enough.
For user with the cpu offload issue, can you post feedback and see if this addresses the cpu offload issue(s)?
I'm afraid it's the same problem again... (AMD-MTP-IQ4_XS.gguf) 😢
But at least the LOW quant works... And it seems to perform well enough.
Same here. 60/40 CPU/GPU split on AMD-MTP-IQ4_XS. Whether MTP is enabled or disabled, if that makes a difference. I need to do more comparisons with the LOW quant to see how its prose writing compares; I normally use the 'DavidAU/Qwen3.6-27B-Heretic-Uncensored-FINETUNE-NEO-CODE-Di-IMatrix-MAX-GGUF' at Q5_K_S for writing, with the 40B Opus Deckard for creative development; plot structure, character development, outlining . . . things not dependent on speed. T
Hmm ; sounds like 16 bit is not supported and/or offloading to cpu for some reason.
The ideal is to use the quants with 16 bit as the OT is critical for final output.
I have put in a request to ask team Mradermarcher for "standard" / reg "imatrix" quants.
Hmm ; sounds like 16 bit is not supported and/or offloading to cpu for some reason.
The ideal is to use the quants with 16 bit as the OT is critical for final output.
I have put in a request to ask team Mradermarcher for "standard" / reg "imatrix" quants.
Let me know if you you need information from me. Training and building the models is beyond my skill levels, but let me know what I can do to help solve this.