noctrex

/

Nemotron-3-Nano-30B-A3B-MXFP4_MOE-GGUF

like 17

Oh and just in case you didn't try: for this specific model, even if the Q4_K_XL from Unsloth is 22.8GB, with 24GB of VRAM I can entirely fit it without even needing to quantize the kv cache, with a 131K token window ;) It grows really slowly as the context is increased

@owao at the end of day, this is another quant type. a 4-bit that uses floating point instead of integer. I have seem some differences on tool calling for example on other models, so I thought of just quantizing them in this format and just upload them if anyone want to use them.

@cmp-nct why do you think that? if you have a look at the other 30B models I've quantized, they are the same size. I didn't use any custom code. Used the official llama.cpp tools to create it.

Oh ok! Interesting! Thanks for sharing, and even if I don't use this one, thanks for the quants!

@owao at the end of day, this is another quant type. a 4-bit that uses floating point instead of integer. I have seem some differences on tool calling for example on other models, so I thought of just quantizing them in this format and just upload them if anyone want to use them.

Interesting convo. I've noticed MXFP4 was ok, but for some reason your vanilla Qwen3 30B A3B 2507 Thinking MXFP4 performed worse than Coder and Instruct in some of my tasks. I have to recheck this. Probably just my mind.

@owao at the end of day, this is another quant type. a 4-bit that uses floating point instead of integer. I have seem some differences on tool calling for example on other models, so I thought of just quantizing them in this format and just upload them if anyone want to use them.

Interesting convo. I've noticed MXFP4 was ok, but for some reason your vanilla Qwen3 30B A3B 2507 Thinking MXFP4 performed worse than Coder and Instruct in some of my tasks. I have to recheck this. Probably just my mind.

Yeah, it will need actual extended benchmarking, but I don't have the resources to do some, as I only have my main rig with a powerful enough card to do it, but I need it for work.

@cmp-nct why do you think that? if you have a look at the other 30B models I've quantized, they are the same size. I didn't use any custom code. Used the official llama.cpp tools to create it.

Your quants are fine, GG likely is right that MXFP4 is not optimal for most models. Q4_K is likely a bit better in quality at same size.
I was wondering about the size because all Q2_K quants are also 18GB of size, I checked them now and they are labelled as 2 bit but actually 4-5 bit inside.

@owao at the end of day, this is another quant type. a 4-bit that uses floating point instead of integer. I have seem some differences on tool calling for example on other models, so I thought of just quantizing them in this format and just upload them if anyone want to use them.

Interesting convo. I've noticed MXFP4 was ok, but for some reason your vanilla Qwen3 30B A3B 2507 Thinking MXFP4 performed worse than Coder and Instruct in some of my tasks. I have to recheck this. Probably just my mind.

Yeah, it will need actual extended benchmarking, but I don't have the resources to do some, as I only have my main rig with a powerful enough card to do it, but I need it for work.

Totally get you. Thanks anyway for these quants. They're amazing.

I thought you had the quantized MXFP4 for the Qwen3 30B A3B 2507 Thinking but I cannot find it? I had someone elses MXFP4 against the UD-Q4_K_XL when I realized it wasn't your quant.

The MXFP4 did better by ~2% than UD-Q4_K_XL on some mathematical calculations on my financial work books. However the MXFP4 succeeded in refactoring 1 monolith NextJS component into 9 parts, albeit it did not keep the design as requested but it wasn't broken - a similar behaviour with Coder variant. The UD-Q4_K_XL kept looping different fixes without success.

It's not a sanitized test but since I know my work better than some random benchmarks, there's less risk of over-fitting. I did used GPT 5.2 Thinking against my mathematical calcs as baseline, so it could be temperature variance or else.

Considering the weights I do suppose it's fine like cmp-nct touched.

The best part is, I was able to do the same bench I did above on your Nemotron and it was 5-7% better on the same mathematical equations from my books.

Refactoring the NextJS component was extremely good. It completed exactly as it was originally. Spent <155k tokens to complete on my framework, whereas some of my other models did between 800k-4.7M on the same refactor. Heck, even GLM 4.6 did that in 647k.

Too early to tell on anything else but this was very good. I think NVIDIA cooked here.

Thank you for sharing this valuable information.
Yes it seems that it's better to see it on actual workflows, because at this point I don't know how much we can trust many of the benchmarks released, cause of bechmaxxing.
That said, I found some interesting benches I could maybe run.
https://github.com/chigkim/Ollama-MMLU-Pro
https://github.com/muayyad-alsadi/HalluBench
The MMLU one takes a long time to complete, and the hallubench seems promising, in order to see who hallucinates less

Thank you for sharing this valuable information.
Yes it seems that it's better to see it on actual workflows, because at this point I don't know how much we can trust many of the benchmarks released, cause of bechmaxxing.
That said, I found some interesting benches I could maybe run.
https://github.com/chigkim/Ollama-MMLU-Pro
https://github.com/muayyad-alsadi/HalluBench
The MMLU one takes a long time to complete, and the hallubench seems promising, in order to see who hallucinates less

Never tried them over the fact if models know this stuff or not.

I was able to load your quant at 512k context with virtually 30GB without KV Cache quantizations. Couldn't try it fully as I had something else to cook.

This doesn't seem right? Way too good but not sure processing powers gets there. Was running about 150 t/s generation early on.