Qwen3.6-27B-NVIDIA-NVFP4-MTP-GGUF

GGUF conversion of nvidia/Qwen3.6-27B-NVFP4, preserving NVIDIA NVFP4 tensors, with MTP speculative decoding and a BF16 vision projector. Benchmarked on an RTX 5090 with llama-benchy.

Highlights

• NVFP4 preserved: 193 NVFP4 tensors are preserved from NVIDIA's ModelOpt quantized checkpoint.
• MTP included: the GGUF keeps the extra MTP layer for draft-mtp speculative decoding.
• Vision supported: includes a BF16 mmproj file for image input.
• RTX 5090 tested: measured with llama-benchy using MTP depth d=3.

Provenance

Component	Source
Base model	Qwen/Qwen3.6-27B
NVFP4 source checkpoint	nvidia/Qwen3.6-27B-NVFP4
Runtime target	llama.cpp

Files

File	Size	Description
Qwen3.6-27B-NVIDIA-NVFP4-MTP.gguf	14.66 GiB / 15,747,650,944 bytes	Main GGUF. NVFP4 tensors preserved; MTP layer included.
mmproj-Qwen3.6-27B-NVIDIA-NVFP4-BF16.gguf	~888 MiB / 931,146,304 bytes	BF16 vision projector for image input.

llama.cpp example

llama-server \
  -m Qwen3.6-27B-NVIDIA-NVFP4-MTP.gguf \
  --mmproj mmproj-Qwen3.6-27B-NVIDIA-NVFP4-BF16.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  -ngl 999

For text-only use, omit --mmproj mmproj-Qwen3.6-27B-NVIDIA-NVFP4-BF16.gguf.

Benchmarks

Measured on RTX 5090, llama-benchy 0.3.8, MTP depth d=3, 3 runs, generation-latency mode.

Mode	Prompt / generation length	Prefill (pp) tok/s	Generation (tg) tok/s	Peak generation tok/s
Text-only	pp512 / tg512	2676.94 ± 51.88	112.69 ± 3.13	130.33 ± 7.41
Text-only	pp4096 / tg512	3401.62 ± 94.87	116.38 ± 3.25	136.33 ± 4.50
Vision + mmproj	pp512 / tg512	3337.94 ± 382.60	111.31 ± 7.18	126.33 ± 8.38
Vision + mmproj	pp4096 / tg512	3650.71 ± 86.68	114.15 ± 3.52	131.00 ± 2.94

For reference, the no-speculative baseline measured in the same project notes was about 70.75 tok/s on pp512/tg512, so MTP d=3 gives roughly +59% generation throughput in that test.

Model details

Field	Value
GGUF architecture	qwen35
License metadata	apache-2.0
Text layers	64
MTP layers	1
qwen35.block_count	65
qwen35.nextn_predict_layers	1
Vision projector architecture	clip / qwen3vl_merger
Vision projector precision	BF16/F32

Tensor composition of the main GGUF

Tensor type	Count
NVFP4	193
Q4_K	279
Q6_K	34
F32	746
Total	1252

The conversion keeps the original NVFP4 tensors instead of requantizing all weights to K-quants. Non-NVFP4 tensors are stored as K-quants or F32 as appropriate for the final GGUF.

License and attribution

The upstream models are licensed under Apache-2.0:

• Qwen/Qwen3.6-27B
• nvidia/Qwen3.6-27B-NVFP4

This repository redistributes a GGUF format conversion of the NVIDIA NVFP4 checkpoint under the same Apache-2.0 license. Please also follow the terms and usage guidance of the upstream model cards.

Acknowledgements

Thanks to the Qwen team for the base model, NVIDIA for the NVFP4 ModelOpt checkpoint, and the llama.cpp project for GGUF, NVFP4, MTP, and multimodal runtime support.