Can you just please uploaf q4ks without mtp...
İ have low VRAM and q4 k xl doesnt fit on my CPU+gpu,q4ks fits but its slow because of mtp layer,it will ve so good if we get q4 KS without mtp.
Before I give you a solution to a misunderstood problem: I'm a bit confused by your statement "q4ks fits but it's slow because of mtp layer".
If MTP slows you down on this model, it's usually because you have not tuned n_max and p_min in llama.cpp to your system and prompt. If you ask you coding agent to do a parameter sweep across different numbers for these two parameters on a representative prompt (coding or whatever) then MTP will almost always make the model much faster. On my system Nail goes from 75 to 105 tokens per second generation with MTP and tuned parameters. And you don't have to activate the MTP head for every task, you turn it on and off yourself. So if you are doing creative writing or other unstructured text at high temperature, just turn off MTP. Is your problem that q4ks with MTP takes a bit more RAM than without? If you want max speed for coding and still want to fit full context kv in (V)RAM, your best bet would then be to use UD-IQ4_XS (18.2 GB) with MTP. If you help me understand your situation I might help you find a better solution, or whether a MTP-less q4ks is indeed the right shape for you.
Before I give you a solution to a misunderstood problem: I'm a bit confused by your statement "q4ks fits but it's slow because of mtp layer".
If MTP slows you down on this model, it's usually because you have not tuned n_max and p_min in llama.cpp to your system and prompt. If you ask you coding agent to do a parameter sweep across different numbers for these two parameters on a representative prompt (coding or whatever) then MTP will almost always make the model much faster. On my system Nail goes from 75 to 105 tokens per second generation with MTP and tuned parameters. And you don't have to activate the MTP head for every task, you turn it on and off yourself. So if you are doing creative writing or other unstructured text at high temperature, just turn off MTP. Is your problem that q4ks with MTP takes a bit more RAM than without? If you want max speed for coding and still want to fit full context kv in (V)RAM, your best bet would then be to use UD-IQ4_XS (18.2 GB) with MTP. If you help me understand your situation I might help you find a better solution, or whether a MTP-less q4ks is indeed the right shape for you.
i understood but in my situation,an additional mtp layer(~300mb) slows the model because of my limited ram,but i will try iq4xs,thanks!