Qwen3.6-35B-A3B-NVFP4-GGUF

GGUF conversion of nvidia/Qwen3.6-35B-A3B-NVFP4 for use with llama.cpp.

This is an NVFP4-quantized standalone Mixture-of-Experts language model.

The converted GGUF preserves the NVFP4 tensors and the model's native MTP tensors. It is not a separate draft model and does not require an external speculator.

Model Details

• Source model: nvidia/Qwen3.6-35B-A3B-NVFP4
• Base model: Qwen/Qwen3.6-35B-A3B
• Format: GGUF
• Quantization: NVIDIA NVFP4 / ModelOpt
• Architecture: Qwen3.6 Mixture-of-Experts with hybrid attention
• Parameters: 35B total, approximately 3B activated per token
• Purpose: Local inference and native MTP speculative decoding with llama.cpp

NVIDIA quantized the weights and activations of linear operators inside the MoE transformer blocks to NVFP4. Other tensors may remain in higher-precision formats.

This repository contains the complete target model. It is not an MTP, EAGLE3, or DFlash draft-only checkpoint.

Compatibility

A recent version of llama.cpp with Qwen3.6 MoE, NVFP4, and native MTP support is required.

Tested with:

• Windows
• NVIDIA GeForce RTX 5070 Ti 16 GB
• NVIDIA GeForce RTX 5060 Ti 16 GB
• llama.cpp CUDA backend
• Native Qwen3.6 MTP speculative decoding

Older llama.cpp builds may fail to recognize the nvfp4 tensor type, may not correctly load associated scale tensors, or may lack compatible Qwen3.6 MoE support.

Performance may vary with llama.cpp build, GPU split, context size, KV-cache format, prompt, sampling settings, and other runtime options.

Usage

llama-server

llama-server \
  -m Qwen3.6-35B-A3B-NVFP4.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --flash-attn on \
  -ngl 99

llama-cli

llama-cli \
  -m Qwen3.6-35B-A3B-NVFP4.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --flash-attn on \
  -ngl 99

Multi-GPU example

llama-server \
  -m Qwen3.6-35B-A3B-NVFP4.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --split-mode layer \
  --tensor-split 1,1 \
  --flash-attn on \
  -ngl 99

The best tensor split depends on available VRAM, GPU speed, PCIe topology, context length, and KV-cache placement. An even split is only a starting point.

Suggested Settings

Based on the mixed-task benchmark below:

• n_max = 3 gave the best aggregate result of the tested settings.
• n_max = 2 gave the highest aggregate acceptance rate and was clearly better for translation and other high-entropy natural-language output.
• n_max = 4 produced the highest per-request throughput for highly predictable output such as repeated patterns, JSON, and code completion, but did not improve aggregate throughput over n_max = 3.
• Increasing the draft length beyond n_max = 3 sharply reduced acceptance on creative writing, conceptual explanation, translation, and long code review.

A reasonable general-purpose starting point is:

--spec-draft-n-max 3

For translation, role-play, creative writing, conversational output, or open-ended explanations, start with:

--spec-draft-n-max 2

For JSON, fixed templates, repeated patterns, and deterministic code completion, try:

--spec-draft-n-max 4

Conversion

Converted from the original Hugging Face ModelOpt checkpoint using a recent convert_hf_to_gguf.py from llama.cpp:

python convert_hf_to_gguf.py \
  /path/to/Qwen3.6-35B-A3B-NVFP4 \
  --outfile Qwen3.6-35B-A3B-NVFP4.gguf \
  --outtype auto \
  --verbose

Do not pass a conventional GGUF quantization type such as q4_k_m when the goal is to preserve the original NVFP4 tensors. The conversion repackages the ModelOpt checkpoint into GGUF rather than requantizing it into a K-quant.

No neural-network weights were retrained. The original NVFP4 weights and their associated scale tensors were converted to GGUF-compatible representations.

File Size

The converted GGUF may be larger than the original Hugging Face ModelOpt checkpoint.

The conversion does not copy the original safetensors byte-for-byte. NVFP4 weights are repacked into the GGML layout, and the GGUF stores the scale and input-scale tensors required by llama.cpp.

A larger GGUF file does not mean that the model was converted to FP16 or to a conventional Q4 format. The tensor dump should still show the main linear weights as:

type = nvfp4

Windows reports file sizes using binary units even though File Explorer labels them as GB. For example, a file shown as 28.2 GB by Hugging Face may appear as approximately 26.2 GB in Windows.

File size should not be treated as the exact amount of VRAM required at runtime. Additional memory is needed for the KV cache, CUDA context, compute buffers, graph workspace, and native MTP execution.

Verifying the GGUF

The converted file can be checked with llama-gguf:

.\llama-gguf.exe E:\HF_MODELS\Qwen3.6-35B-A3B-NVFP4.gguf r

A successful conversion should show NVFP4 tensors and their associated scale tensors, for example:

tensor: name = blk.0.ffn_down.weight,       type = nvfp4
tensor: name = blk.0.ffn_down.scale,        type = f32
tensor: name = blk.0.ffn_down.input_scale,  type = f32

This confirms that the checkpoint was not converted into a conventional Q4 type and is not an empty metadata-only GGUF.

Benchmark

Benchmarked using mtp-bench with native Qwen3.6 MTP speculative decoding.

Aggregate Results

n_max	Draft acceptance	Predicted tokens	Draft tokens	Accepted tokens	Wall time	Effective batch throughput
2	82.2%	1,824	1,370	1,126	15.83 s	115.22 tok/s
3	72.8%	1,807	1,690	1,230	14.81 s	122.01 tok/s
4	64.2%	1,804	2,015	1,294	14.91 s	121.00 tok/s

Effective batch throughput is calculated from total predicted tokens divided by total wall time. Per-request tok/s values may be higher because aggregate wall time can also include prompt processing and benchmark overhead.

The predicted-token totals differ slightly between runs because some test outputs terminated at different lengths. Aggregate throughput is therefore more useful than wall time alone when comparing these runs.

These results do not include a non-speculative baseline and therefore should not be interpreted as a direct speedup ratio over standard decoding.

Detailed Results

n_max = 2

code_python        pred= 192 draft= 132 acc= 125 rate=0.947 tok/s=149.5
code_cpp           pred=  54 draft=  36 acc=  36 rate=1.000 tok/s=151.5
explain_concept    pred= 192 draft= 172 acc= 104 rate=0.605 tok/s=117.4
summarize          pred=  50 draft=  38 acc=  31 rate=0.816 tok/s=135.7
qa_factual         pred= 192 draft= 142 acc= 120 rate=0.845 tok/s=143.9
translation        pred=  17 draft=  12 acc=  10 rate=0.833 tok/s=134.3
creative_short     pred=  39 draft=  38 acc=  20 rate=0.526 tok/s=107.6
stepwise_math      pred= 192 draft= 135 acc= 123 rate=0.911 tok/s=147.5
json_output        pred= 192 draft= 132 acc= 124 rate=0.939 tok/s=146.5
long_reasoning     pred= 192 draft= 145 acc= 118 rate=0.814 tok/s=138.7
repeat_pattern     pred= 192 draft= 127 acc= 127 rate=1.000 tok/s=153.9
code_completion    pred= 128 draft=  86 acc=  85 rate=0.988 tok/s=153.8
long_code_review   pred= 192 draft= 175 acc= 103 rate=0.589 tok/s=113.4

Aggregate:
  requests:            13
  predicted tokens:    1824
  draft tokens:        1370
  accepted tokens:     1126
  acceptance rate:     0.8219
  total wall time:     15.83 s

n_max = 3

code_python        pred= 192 draft= 150 acc= 141 rate=0.940 tok/s=167.6
code_cpp           pred=  54 draft=  45 acc=  39 rate=0.867 tok/s=155.6
explain_concept    pred= 192 draft= 232 acc= 112 rate=0.483 tok/s=112.3
summarize          pred=  50 draft=  48 acc=  33 rate=0.688 tok/s=134.5
qa_factual         pred= 192 draft= 174 acc= 132 rate=0.759 tok/s=149.8
translation        pred=  17 draft=  21 acc=  11 rate=0.524 tok/s=103.8
creative_short     pred=  39 draft=  48 acc=  22 rate=0.458 tok/s=107.2
stepwise_math      pred= 192 draft= 166 acc= 135 rate=0.813 tok/s=153.3
json_output        pred= 192 draft= 156 acc= 139 rate=0.891 tok/s=162.3
long_reasoning     pred= 192 draft= 181 acc= 130 rate=0.718 tok/s=141.6
repeat_pattern     pred= 192 draft= 144 acc= 143 rate=0.993 tok/s=176.8
code_completion    pred= 111 draft=  87 acc=  83 rate=0.954 tok/s=168.8
long_code_review   pred= 192 draft= 238 acc= 110 rate=0.462 tok/s=109.3

Aggregate:
  requests:            13
  predicted tokens:    1807
  draft tokens:        1690
  accepted tokens:     1230
  acceptance rate:     0.7278
  total wall time:     14.81 s

n_max = 4

code_python        pred= 192 draft= 168 acc= 148 rate=0.881 tok/s=170.2
code_cpp           pred=  54 draft=  48 acc=  44 rate=0.917 tok/s=167.8
explain_concept    pred= 192 draft= 293 acc= 117 rate=0.399 tok/s=110.0
summarize          pred=  50 draft=  52 acc=  37 rate=0.712 tok/s=149.3
qa_factual         pred= 192 draft= 212 acc= 137 rate=0.646 tok/s=145.8
translation        pred=  17 draft=  28 acc=  12 rate=0.429 tok/s=96.1
creative_short     pred=  36 draft=  72 acc=  20 rate=0.278 tok/s=84.8
stepwise_math      pred= 192 draft= 187 acc= 143 rate=0.765 tok/s=159.9
json_output        pred= 192 draft= 174 acc= 147 rate=0.845 tok/s=171.5
long_reasoning     pred= 192 draft= 215 acc= 137 rate=0.637 tok/s=142.8
repeat_pattern     pred= 192 draft= 153 acc= 152 rate=0.994 tok/s=191.2
code_completion    pred= 111 draft= 100 acc=  88 rate=0.880 tok/s=177.6
long_code_review   pred= 192 draft= 313 acc= 112 rate=0.358 tok/s=101.4

Aggregate:
  requests:            13
  predicted tokens:    1804
  draft tokens:        2015
  accepted tokens:     1294
  acceptance rate:     0.6422
  total wall time:     14.91 s

Observations

• n_max = 3 delivered the highest effective aggregate throughput at approximately 122.01 tok/s.
• n_max = 4 was nearly tied in aggregate throughput at approximately 121.00 tok/s, but required substantially more draft work and reduced acceptance to 64.2%.
• n_max = 2 retained the highest aggregate acceptance rate at 82.2%.
• Compared with n_max = 2, n_max = 3 improved effective aggregate throughput by approximately 5.9%.
• Repeated-pattern generation reached 191.2 tok/s with n_max = 4.
• Code completion reached 177.6 tok/s with n_max = 4.
• JSON output reached 171.5 tok/s with n_max = 4.
• Translation was fastest with n_max = 2, reaching 134.3 tok/s.
• Creative writing dropped from 107.6 tok/s at n_max = 2 to 84.8 tok/s at n_max = 4.
• Conceptual explanation, translation, creative writing, and long code review showed rapidly diminishing returns as the draft length increased.

The results show a clear task-dependent trade-off: predictable output benefits from a longer native MTP draft, while high-entropy natural-language generation generally favors a shorter draft.

Notes

• This GGUF preserves NVIDIA's NVFP4 tensor type; it is not equivalent to Q4_K_M, Q4_K_S, or an Unsloth dynamic quant.
• The model is a 35B-total-parameter MoE with approximately 3B parameters activated per token; runtime memory requirements still depend on the full stored checkpoint rather than only the active parameter count.
• NVIDIA's source checkpoint was prepared for ModelOpt and vLLM. llama.cpp support is a separate community implementation and may behave differently from NVIDIA's reference runtime.
• Model size, VRAM requirements, and speed should not be estimated as though every parameter were stored as a plain 4-bit scalar. NVFP4 uses block scales and leaves some tensors at higher precision.
• Native MTP accelerates token generation but does not improve prompt-prefill speed in the same way.
• Very short benchmark outputs, especially translation and C++ in these runs, are more sensitive to run-to-run variance.
• Results are specific to the tested hardware, llama.cpp build, prompts, runtime options, and context configuration.

Credits

• Qwen Team / Alibaba Cloud — Qwen3.6-35B-A3B
• NVIDIA — ModelOpt and the original NVFP4 checkpoint
• ggml-org — llama.cpp, GGUF, NVFP4 inference support, and native MTP support

License

The source model is distributed under the Apache License 2.0.

Users should review the upstream nvidia/Qwen3.6-35B-A3B-NVFP4 and Qwen/Qwen3.6-35B-A3B model cards before redistribution or commercial use.