Pure NVFP4 vs NVFP4-MXFP6 Hybrid
First off, thank you! Both of these 27B Quants are amazing.
I’m having a hard time deciding between the two you’ve created. Pure NVFP4 is so fast, I can get regularly 120 t/s on an RTX 5090. The hybrid gets around 80-90, roughly. This is of course roughly mid-way through 131k context in a Hermes Agent use case. Just peeking llama server logs every so often.
Question: how should choose between the two? Do you find that the hybrid is better in terms of quality?
Again, thank you for sharing your work with us!
Hi @andrewdoto13
Thanks for the kind words! I have been using them myself too, and I too am not sure which is better. The truth of all of it is that every NVFP4 implementation is a hybrid, there are always some layers that are not NVFP4, if you made everything pure the model would never work, so the hardest part is always deciding what to keep in higher precision (this is also why I have the mixed MXFP6 version: on Blackwell the hardware also supports native MXFP6 acceleration and so it's also very fast for 6-bit, just not as fast as NVFP4 is).
So far in the evals I've run they seem very similar, but on the mathematical tests there's a clear distinction, but it may not matter. If you're a bit farsighted or getting older and need reading glasses, you may still be able to read this message 100% correctly and see every word as I've written it, it might just not be as clear; or if you turn the volume up on speakers you can still hear the same words perfectly if the signal is clear, it's just louder - that's a lot of what these numbers also mean, so while it's cool to have on paper to be 'better', in some ways it makes absolutely no difference and a lot of effort is spent on proving what's 'a better model' but chasing the wrong metric (which is true for all sorts of evaluations for things in the world besides LLMs). Both of the versions I made here are using the same dataset, so they're biased to the same message, and that's likely what makes a big part of the difference from one version of a model to another. There's more to it, but I hope that is a really good analogy!
Hi @andrewdoto13
Thanks for the kind words! I have been using them myself too, and I too am not sure which is better. The truth of all of it is that every NVFP4 implementation is a hybrid, there are always some layers that are not NVFP4, if you made everything pure the model would never work, so the hardest part is always deciding what to keep in higher precision (this is also why I have the mixed MXFP6 version: on Blackwell the hardware also supports native MXFP6 acceleration and so it's also very fast for 6-bit, just not as fast as NVFP4 is).
So far in the evals I've run they seem very similar, but on the mathematical tests there's a clear distinction, but it may not matter. If you're a bit farsighted or getting older and need reading glasses, you may still be able to read this message 100% correctly and see every word as I've written it, it might just not be as clear; or if you turn the volume up on speakers you can still hear the same words perfectly if the signal is clear, it's just louder - that's a lot of what these numbers also mean, so while it's cool to have on paper to be 'better', in some ways it makes absolutely no difference and a lot of effort is spent on proving what's 'a better model' but chasing the wrong metric (which is true for all sorts of evaluations for things in the world besides LLMs). Both of the versions I made here are using the same dataset, so they're biased to the same message, and that's likely what makes a big part of the difference from one version of a model to another. There's more to it, but I hope that is a really good analogy!
I need to correct myself and was thinking about a different model when you were talking about pure NVFP4, and I made a "turbo" vs "HQ" version. Working on too many things! On this model and on the NVFP4-MXFP6 blend for Qwen3.6-27B, they are using the standard wikitrain dataset for the imatrix which biases the model in that direction. https://huggingface.co/michaelw9999/Qwen3.6-35B-A3B-NVFP4-MTP-GGUF is the one using my own first try dataset. https://huggingface.co/michaelw9999/Qwopus3.6-27B-Coder-MTP-NVFP4-GGUF skipped using an imatrix altogether.
Thank you for responding!
Yea I've been going back and forth between:
michaelw9999/Qwen3.6-27B-NVFP4-MTP-GGUF
vs
michaelw9999/Qwen3.6-27B-NVFP4-MXFP6-MTP-GGUF
I find myself being entranced by the raw speed of the NVFP4 "only" version. The MXFP6 hybrid is still fast, just not quite as blazing. Maybe my use cases don't challenge the model enough to where I notice a difference. I use it for some coding and also web research mainly with Hermes Agent. I find that the MXFP6 hybrid maybe is more "precise," but I'm not sure if I'm just confirming my bias of what I think it should be. It's really hard to tell.
And for MXFP6 hybrid, I am using your llama cpp with NVFP4 repack, to get the most out of it. Let me know if this makes sense to you. And again, thank you for sharing this amazing work!