NVFP4?
For us blackwell/5090 owners - for high speed inference with full fp8 context?
Or just post a bf16 version so we can cook up our own! NVFP4 mlp and FP8 attention FTW!
AN interesting model, would be good if you released the safe tensors or made a NVFP4, ive done some preliminary testing on my 5090 and think its good but i would wanna through it on my spark for real benchmarking before id put swap up my current 27b
Source is here, on my partner's repo - Nightmedia:
https://huggingface.co/nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451
This is EXACT source.
Ah thankyou much appreciated i'll take a look
I created a version from F32 source. It is not enhanced, just simple conversion with the mlx-vlm tools.
The F32 version does not reach as high as BF16 with mxfp8, but it helps on the lower quants.
The mxfp4 performance is identical between BF16 and F32
https://huggingface.co/nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-F32-nvfp4
arc arc/e boolq hswag obkqa piqa wino
mxfp8 0.709,0.880,0.909
qx64-hi 0.706,0.873,0.908
nvfp4 0.704,0.868,0.908
mxfp4 0.701,0.873,0.909
Quant Perplexity Peak Memory Tokens/sec
mxfp8 3.782 ± 0.023 34.74 GB 164
qx64-hi 3.751 ± 0.023 27.03 GB 166
nvfp4 3.805 ± 0.023 22.14 GB 138
mxfp4 3.854 ± 0.024 21.30 GB 176
BF16
qx64-hi 0.702,0.873,0.909,0.794,0.514,0.822,0.750
nvfp4 0.701,0.866,0.907
mxfp4 0.701,0.873,0.909,0.786,0.488,0.813,0.759
The nvfp4 I created from BF16 source shows lower metrics, so F32 is a better base for this quant.
Speed is as tested on a Mac, it would obviously be different on the target platform(hopefully better)
This is the first nvfp4 I created, so I did not want to start with optimizations right away. I am aware people are enhancing layers, and will soon try that, once I find out how to do it :)
@nightmedia Thanks for this. Will check this out soon - havent really tried nvfp4 made using mlx-vlm . Typically, people use modelopt (Nvidia), and theres some custom higher precision options prismaquant ,etc. That being said, 0.704 could be the ballpark for this level of quantization - the other nvfp4 strategies could very well land here anyway. Cheers, and amazing effort by you guys on these. Keep pushing the tensors lol.
I created a version from F32 source. It is not enhanced, just simple conversion with the mlx-vlm tools.
The F32 version does not reach as high as BF16 with mxfp8, but it helps on the lower quants.
The mxfp4 performance is identical between BF16 and F32
https://huggingface.co/nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-F32-nvfp4
arc arc/e boolq hswag obkqa piqa wino mxfp8 0.709,0.880,0.909 qx64-hi 0.706,0.873,0.908 nvfp4 0.704,0.868,0.908 mxfp4 0.701,0.873,0.909 Quant Perplexity Peak Memory Tokens/sec mxfp8 3.782 ± 0.023 34.74 GB 164 qx64-hi 3.751 ± 0.023 27.03 GB 166 nvfp4 3.805 ± 0.023 22.14 GB 138 mxfp4 3.854 ± 0.024 21.30 GB 176 BF16 qx64-hi 0.702,0.873,0.909,0.794,0.514,0.822,0.750 nvfp4 0.701,0.866,0.907 mxfp4 0.701,0.873,0.909,0.786,0.488,0.813,0.759The nvfp4 I created from BF16 source shows lower metrics, so F32 is a better base for this quant.
Speed is as tested on a Mac, it would obviously be different on the target platform(hopefully better)
This is the first nvfp4 I created, so I did not want to start with optimizations right away. I am aware people are enhancing layers, and will soon try that, once I find out how to do it :)
Thankyou this saved me a step im also experimenting the first time with NVFP4 ill let you know the results :)
Hey Just wanted to say Architect-Polaris2-Fable-B-F451 Is a neat little model
One thing worth flagging for your is that -F451-F32-nvfp4 upload is an MLX quant (mlx-vlm)
o it only runs on Apple Silicon.
On CUDA/vLLM it can't load — vLLM binds quantization off quantization_config.quant_method (it expects modelopt or compressed-tensors), and the MLX config ({group_size, bits, mode: nvfp4}) has no quant_method for it to read.
So I took your bf16 F451 merge and ran it through NVIDIA's ModelOpt PTQ into a proper CUDA NVFP4 mixed precision, FP4 on the MLP/linear layers with the linear-attention state, vision tower, MTP head and embeddings kept in higher precision (matching how NVIDIA quantizes the base Qwen3.6-27B). It loads and serves on vLLM and the output holds up well.
Happy to upload it but wanted to check in with you first as its useful for NVIDIA users but i dont want to look like im stealing your work otherwise i can send a doc instead on how i did it
@PassingByPixels would be great if you can upload the CUDA NVFP4 . You can credit the original model in the model card. Thats how all quants on huggingface are. Looking forward to it.
I would greatly appreciate it--as I said, I converted it naively using the tools I had :)
@PassingByPixels
Excellent.
I will add links to it to ; please link back/comment @nightmedia repo and GGUF source here.
Project is a colab between myself and nightmedia and other people too.
OK ive uploaded it https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-NVFP4
My first time posting any models so if ive missed something let me know , averages a solid 18-20 toks on a DGX spark, i reckon i can get it over 30+ with DFlash but i need to tinker with it more
Thanks @PassingByPixels for the quant!
I noticed in my attempt to create it that processing from F32 source returns better results in NVFP4, if you want to try it from F32 I can give you access to source.
-G
HI @nightmedia
Sure happy to give that a go but i think the repo you gave me access to https://huggingface.co/nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-F32-nvfp4 is already pre-converted and has the same mlx issue
or atleast i don't see the link to the F32 files im probably doing something wrong 🙃 if you can point me to the F32's im happy to try running the foundry on that
Oh, forgot to unlock it
https://huggingface.co/nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-F32
Have at it :)
P.S. Forgot to gate it. Can't remember how ;)
Ran into a snag which is that loading the F32 is 105gb and after the overheads i OOM on the DGX spark, Sorry id need to wait till i get a second to do it
Ran it by a few other models that all agreed it probably wouldnt move the needle much but theres no way for more to cheat the memory need 😔
So, if you can tell me how I could do this on a Mac, I would. First time didn't go well :)
Im fairly certain you cannot run NVIDIA ModelOpt on mac.
From what i can see it needs ;
Linux (or Windows)
NVIDIA GPU + CUDA 12.x/13.x
PyTorch with CUDA
Anyone with a single spark can convert your B16 its like 50gb but the NF32 because its 105gb+ OOM's on a single, if i pick up a second spark ill come back and convert it for you if someone out there with more gear doesnt do it.
In the meantime i will try to make a DFlash version of the B16 which will bump it from 20 toks to 40+
Switched to passingbypixels version thank you
Just to close the loop the MTP and Dflash versions are up now aswell at https://huggingface.co/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-NVFP4 & https://github.com/PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-NVFP4_Dflash_DGX_recipe
A very small detail about the naming: there is actually a Qwen3.6-27B-Architect-Polaris2-Fable-B without the F451, that I did not upload yet, because it was a building block for the F451 and I am very tight on space. By itself it is a fairly competent--let me know if you want to try it out :)
models:
- model: Qwen3.6-27B-Architect-Polaris-Fable
parameters:
weight: 1.4
- model: DavidAU/Qwen3.5-27B-Polar-Rev1-Uncensored-Heretic
parameters:
weight: 0.6
merge_method: nuslerp
dtype: bfloat16
name: Qwen3.6-27B-Architect-Polaris2-Fable-B
mxfp8 0.706,0.875,0.911,0.788,0.516,0.821,0.769
qx86-hi 0.701,0.873,0.911,0.787,0.506,0.823,0.758
Ah! your right i missed out the F451 in the naming, i'll have to work out how i correct that