Curiosity about MTP size and performance for Q8_0
I've used Bartowski Q8_0 quants since forever. I want the highest precision possible short of BF16. Saw someone recommend your AutoRound quant as highly precise.
The Bartowski quant has a separate mtp model file, loaded into llama.cpp with --draft-model , and when the server starts, it notes:
load_model: [spec] estimated memory usage of draft model is 2136.04 MiB
Your draft model appears to be integrated into the model file, and when the server starts, it notes:
load_model: [spec] estimated memory usage of MTP context is 417.33 MiB
I noticed that on the same prompt over a 30k context, the Bartowski quant averages 9.01 t/s, where your model averages 8.72 t/s. I'm wondering if it has to do with the MTP model size being smaller, and thus less accurate. I could be reading it all wrong, but just curious if anyone has any thoughts on this. I'm still learning.
Thanks!
Great question! The difference you're seeing is actually just how the MTP weights are stored, not a quality difference.
Bartowski's approach ships the MTP head as a separate draft model file (--draft-model). The 2136 MiB reported is the weight memory of that separate file, loaded and allocated independently. In our GGUFs, the MTP head weights are integrated directly into the main model inside blk.64.* tensors. The 417 MiB is the runtime KV cache/context overhead for MTP inference, not the weights themselves, since they're already part of the model.
Both approaches use the exact same MTP architecture from the original Qwen3.6 (one NextN transformer block with attention projections, a shared expert hidden projection, and two layer norms). The size difference is just what llama.cpp is counting, weights vs. context overhead.
As for the ~0.3 t/s speed difference, this is unlikely to be MTP-related. The MTP head is a single forward pass regardless of how it's loaded. The difference is more likely from measurement variance at 30k context, minor memory layout differences affecting cache behavior, or possibly different quantization schemes applied to the draft model vs. the integrated weights.
TL;DR: Same MTP head, different packaging. The speed difference is negligible and likely noise.
That said, if you're looking for the highest precision Q8_0, you might want to try our latest re-upload. We've switched to new method which uses way more iterations of gradient-based optimization per block. At Q2_K_S this showed 41.5% lower perplexity. At Q8_0 the improvement will be more modest since there's less quantization error to correct, but it should still be slightly more precise than a standard quant. Would be curious to hear how it compares to the Bartowski version if you give it a try! :3
Wow, thanks for such a detailed reply. It is much appreciated. I saw the difference in text outputs but didn't have the presence of mind to note context meant KV cache. Thanks and I'll check out your latest re-upload.
I'll try a few things and see if I can notice a difference. It's sometimes difficult to notice a difference between quants at this level, but if I think I notice something significant I'll let you know.