The Best Kimi Quant!
Q4_X is the best Kimi quant, thank you!
Thanks for smol-IQ2_KL!
It was the usable quant I could run for K2.5 (Though I swapped the embedding / output to q8_0 for faster textgen).
Thanks y'all! had some help with jukofyork and AesSedai too to get the Q4_X going which matches perplexity with AesSedai's too!
I also snagged the mmproj from AesSedai https://huggingface.co/AesSedai/Kimi-K2.6-GGUF/tree/main for my own use haha
@sousekd The Q4_X quants should be basically the same (since it's a static quant without an imatrix), but ubergarm focuses on recipes that include ik_llama.cpp-specific quantizations eg QX_KS and QX_KT variants. I focus on mainline llama.cpp quants instead.
But... that makes it even worse. I can only run one and don't want to hurt anyone's feelings π.
If I discuss it here but secretly use yours, wouldn't that count as cheating?
Yes, very good Q4-X, tested hours ago, 677Gb RAM.
This is model number 92 test (from year 2020 GPT2).
23:39:52-566266 INFO Loaded "USB-DATA/kimi/Q4_X/Kimi-K2.6-Q4_X-00001-of-00014.gguf" in 2438.57 seconds.
23:39:52-569095 INFO LOADER: "llama.cpp"
23:39:52-570905 INFO CONTEXT LENGTH: 65536
23:56:11-383450 INFO Renaming "user_data/logs/instruct/20260421-23-55-59.json" to "user_data/logs/instruct/kimi26-moz.json"
My coding music test all previous Kimis fail in quantized form (quantization destroying coding always).
But this version in ik_llama quantization of Q4_X (in fp16 cache, context 65536, uses 677Gb RAM total) resolved my test in 1st try, creating a working melody in ChucK language, and become the 5th model by all time which have made that (to the list of GLM5.0 in Q6-K-XL; GLM4.5 in Q8; Minimax M2.0 in F16 and DeepseekV2.5-1210 in Q8).
Although it takes almost 4 hours for it on my ANCIENT Xeon from year 2016 to write this 10 tunes, but other models cant do even that.
03:53:35-837407 INFO Output generated in 14237.99 seconds (1.07 tokens/s, 15198 tokens, context 339, seed 525598635)
ik_llama supplied in the app "Textgen" with user interface (aka oobabooga aka Text-generation-WebUI https://github.com/oobabooga/textgen/releases). Deterministic preset, special system prompt (will try without separately).
#My Hardware# Intel Xeon E5-2699v4 LGA2011-3 22 cores 44 threads (2016) $110 # Gigabyte C612 chipset 12 RAM slots VGA motherboard year 2016 $150 # Samsung-Hynix ECC RAM 12x64Gb=768Gb ~$900 # VGA monitor # IKEA chair # NO GPU # Run: Trillions Deepseeks, Kimis in Q5-Q6, 400-500billions in BF16, super high quality 1 token/sec testai.neocities.org
Wow, 40 minutes of loading time? Glad to see K2.6 passed your test!
Kimi-K2.6-Q4_X is effectively unquantized.
GLM5.0 in Q6-K-XL; GLM4.5 in Q8; Minimax M2.0 in F16 and DeepseekV2.5-1210 in Q8
Does that mean all the smaller quants failed? eg: DeepseekV2.5-1210 in Q4 ? Or you just happened to test Q8?
Have you tried: https://huggingface.co/ubergarm/Qwen3.5-122B-A10B-GGUF ?
Wow, 40 minutes of loading time? Glad to see K2.6 passed your test!
Thats not the test, it is USB 3.0, downloading models (how is your ISP speed?) on internal drive is kinda dumb waste of energy of main system, you can use laptop with USB drive for that. USB 3 works weird sometimes, one time its super fast by exchanging hundreds gigabytes between drives, sometimes its no more than 200Mb/s read-write. Its obvious that much faster internal NVMe are already filled with other models, i know by practice it loads there much faster, but USB storage much usable. Sarcasm failing by technical negligence.
If anyone interested in result, request and i will publish it, but its very lengthy text(which not appreciated here) - it thinking and reasoning for near 4 hours how to make small tune in ChucK, but result is good, for writing music itself like Mozart its still too early for any model existing.
Kimi-K2.6-Q4_X is effectively unquantized.
GLM5.0 in Q6-K-XL; GLM4.5 in Q8; Minimax M2.0 in F16 and DeepseekV2.5-1210 in Q8
Does that mean all the smaller quants failed? eg: DeepseekV2.5-1210 in Q4 ? Or you just happened to test Q8?
Have you tried: https://huggingface.co/ubergarm/Qwen3.5-122B-A10B-GGUF ?
Smaller always waste of time, i've tried with Deepseek Q5 vs Q6, hallucinations and quality always worse on lower and large models esp hallucinate high than smaller models. Im not trying anymore testing smaller quants and frankly dont have time for this, there's 10 models in list i need to check (new GLM5.1, Nemotron, etc-etc+i need to put that on Neocities, theres no time). Downloading almost terabyte on my ISP takes whole night, maybe 1.5 days, i can't just download every quant possible. And electricity with 50% discount is only from 23 to 7 am for testing itself.
Quality radically change even in one quant of one model just by changing cache from F16 to Q8 or Q4, i've started to check that and difference present by less Ram, more speed, sometimes interesting quality of same prompt.
About Qwen, recently tested 3.6-35B-A3B-BF16 (80,5Gb RAM), but havent checked for errors yet; and 3.5-397B-A17B-BF16-Opus finetune (797Gb RAM) failed result, same in Q8-K-XL (470Gb RAM) also failed, Qwen3-Coder-Next-BF16 (164Gb RAM) also fail. I haven't found use for Qwen models specifically yet, their output are noticeable by first view by overuse of emotes. Like interesting self-improving bug in Mirothinker which cant stop generation and self polishing its answer to unlimited iterations is cheaper by tokens than using agents(i leave it overnight, wasnt able to make other models work like that) or very interesting hallucinations in very large models which can be used (i was able to trick one chinese Trillion model (not Kimi) that its year 2040 to bring me archival records (hallucination), it makes very good specifications of new declassified machine learning language, but getting full compiler of it almost...breaks by context size of writing full raw code of it.
I understand that wasn't the test, I'm just commenting that I'm glad it worked after loading for 40 minutes because if it crashed that would have been frustrating!
Now i've tested Q4-X more and can say that it's much more productive model, because i really not anticipated to receive a 10 ready webdesign html codes in 1 prompt without stop, all very different (you can see them in my blog testai.neocities.org). Maybe ik_llama in Textgen aka oobabooga not so stable for now, but few times i've got fatal errors after one generation, after which you need to reload model again.
In non-thinking mode it writes right away but more errors (in thinking reasoning very long but result may be non erroneous), same with changing cache from fp16 to q8. The longest stable work of it i've got only today (completed docu for drawing machine test and repaired its mistakes in Mozart song test, all in total takes all night) and its still not crashed in GUI launcher.
I present here its Mozart song in ChucK code, which takes 3 full prompting repairs(~3 days), its made first in non-thinking mode in q8 cache and was unplayable by many errors, but by size was looking like a very long melody. After it repaired the size of code shrinks and now its 5 seconds tune, im not sure its exact Mozart or not but very melodic, so not a garbled collection of noises. (you can play it in Stanford player https://chuck.stanford.edu/ide/)
// Mozart: Sonata No.16 K.545, I. Allegro β Bulletproof ChucK
// Reduced to essential architecture that definitely produces audio
// ============================================================
// VOICE: single oscillator + envelope, connected to dac
// ============================================================
class Voice {
TriOsc osc;
ADSR env;
fun void init() {
// CRITICAL FIX: connected to dac
osc => env => dac;
env.set(5::ms, 80::ms, 0.6, 250::ms);
}
fun void play(float freq, float velocity, dur length) {
freq => osc.freq;
velocity * 0.2 => osc.gain;
env.keyOn();
length => now;
env.keyOff();
250::ms => now;
}
}
// 8 voices, globally accessible
Voice v[8];
for (0 => int i; i < 8; i++) {
v[i].init();
}
0 => int vNext;
// allocate next voice and spork
fun void playOne(float freq, float vel, dur len) {
vNext => int i;
(vNext + 1) % 8 => vNext;
spork ~ v[i].play(freq, vel, len);
}
// melody note with optional bass
fun void note(float freq, float bassFreq, float vel, dur len) {
if (bassFreq > 0.0) {
vNext => int i;
(vNext + 1) % 8 => vNext;
spork ~ v[i].play(bassFreq, vel * 0.45, len);
}
playOne(freq, vel, len);
len => now;
}
// chord (3 notes simultaneously)
fun void chord(float a, float b, float c, float vel, dur len) {
playOne(a, vel, len);
playOne(b, vel, len);
playOne(c, vel, len);
len => now;
}
// ============================================================
// TEMPO
// ============================================================
0.48::second => dur Q; // quarter note ~125 bpm, lively
// ============================================================
// EXPOSITION
// ============================================================
fun void exposition() {
note(523.25, 130.81, 0.85, Q / 2);
note(659.25, 0, 0.75, Q / 2);
note(783.99, 0, 0.80, Q / 2);
note(1046.50, 0, 0.90, Q);
note(783.99, 196.00, 0.85, Q / 4);
note(880.00, 0, 0.70, Q / 4);
note(987.77, 0, 0.75, Q / 4);
note(1046.50, 0, 0.85, Q / 2);
note(987.77, 0, 0.70, Q / 4);
note(880.00, 0, 0.75, Q / 4);
note(783.99, 196.00, 0.65, Q / 4);
note(698.46, 0, 0.60, Q / 4);
note(659.25, 0, 0.55, Q / 4);
note(587.33, 0, 0.65, Q / 2);
note(523.25, 0, 0.55, Q / 4);
note(587.33, 0, 0.60, Q / 4);
note(659.25, 130.81, 0.70, Q / 2);
note(587.33, 0, 0.60, Q / 4);
note(493.88, 0, 0.55, Q / 4);
note(523.25, 0, 0.85, Q + Q / 2);
Q => now;
note(659.25, 146.83, 0.70, Q / 2);
note(783.99, 0, 0.65, Q / 2);
note(880.00, 0, 0.70, Q / 2);
note(1046.50, 0, 0.80, Q);
note(987.77, 174.61, 0.70, Q / 2);
note(880.00, 0, 0.65, Q / 4);
note(783.99, 0, 0.60, Q / 4);
note(739.99, 0, 0.65, Q / 2);
note(783.99, 0, 0.75, Q);
note(783.99, 196.00, 0.80, Q);
note(880.00, 0, 0.70, Q / 2);
note(783.99, 0, 0.65, Q / 2);
note(698.46, 0, 0.75, Q);
Q => now;
note(659.25, 196.00, 0.75, Q);
note(587.33, 0, 0.65, Q / 2);
note(523.25, 0, 0.60, Q / 2);
note(587.33, 0, 0.70, Q);
Q => now;
note(659.25, 146.83, 0.75, Q / 2);
note(739.99, 0, 0.65, Q / 2);
note(783.99, 196.00, 0.85, Q + Q);
1::second => now;
}
// ============================================================
// DEVELOPMENT
// ============================================================
fun void development() {
note(783.99, 196.00, 0.70, Q / 2);
note(739.99, 207.65, 0.65, Q / 2);
note(698.46, 0, 0.60, Q / 2);
note(659.25, 0, 0.55, Q / 2);
note(622.25, 0, 0.60, Q);
note(587.33, 130.81, 0.70, Q / 4);
note(659.25, 0, 0.65, Q / 4);
note(698.46, 0, 0.60, Q / 4);
note(783.99, 0, 0.70, Q / 4);
note(880.00, 0, 0.75, Q / 2);
note(783.99, 0, 0.65, Q / 2);
for (0 => int i; i < 3; i++) {
note(783.99, 196.00, 0.45 + i * 0.12, Q / 3);
note(880.00, 0, 0.45 + i * 0.12, Q / 3);
note(987.77, 0, 0.45 + i * 0.12, Q / 3);
}
note(783.99, 196.00, 0.90, Q + Q / 2);
1::second => now;
}
// ============================================================
// RECAPITULATION
// ============================================================
fun void recapitulation() {
note(523.25, 130.81, 0.85, Q / 2);
note(659.25, 0, 0.75, Q / 2);
note(783.99, 0, 0.80, Q / 2);
note(1046.50, 0, 0.90, Q);
note(783.99, 130.81, 0.80, Q / 4);
note(880.00, 0, 0.70, Q / 4);
note(987.77, 0, 0.65, Q / 4);
note(1046.50, 0, 0.80, Q / 2);
note(987.77, 0, 0.70, Q / 4);
note(880.00, 0, 0.65, Q / 4);
note(880.00, 164.81, 0.80, Q);
note(987.77, 0, 0.70, Q / 2);
note(880.00, 0, 0.65, Q / 2);
note(783.99, 0, 0.75, Q);
Q => now;
note(659.25, 130.81, 0.75, Q / 2);
note(739.99, 0, 0.65, Q / 2);
note(783.99, 0, 0.80, Q * 2);
note(698.46, 196.00, 0.65, Q / 2);
note(659.25, 0, 0.60, Q / 2);
note(587.33, 0, 0.70, Q);
chord(329.63, 523.25, 659.25, 0.55, Q);
chord(392.00, 493.88, 587.33, 0.60, Q / 2);
chord(261.63, 329.63, 523.25, 0.90, Q * 2);
3::second => now;
}
// ============================================================
// MAIN
// ============================================================
exposition();
development();
recapitulation();