selimaktas/Qwen3.6-27B-NVFP4-FP8-MTP

WARNING! This model has not yet been fully verified as functional and fast on SM120 (RTX 5000 Series).

Qwen3.6-27B · Mixed W4A8 (NVFP4) + FP8

A mixed-precision quantization of Qwen3.6-27B (dense hybrid Gated-DeltaNet + attention, vision-language) for Blackwell GPUs. The dense FFN is NVFP4 W4A8 (4-bit weight × FP8 activation); the attention and the validated linear-attention projections are FP8 W8A8; the SSM-critical gates, embeddings, LM head, vision tower and MTP head stay bf16.

The headline: it serves at a ~22 GiB weight footprint (vs ~54 GB BF16 and ~28 GB FP8) with no measurable quality loss.

Requires a small vLLM plugin + CUDA kernel to serve the W4A8 FFN — see Requirements. Stock vLLM cannot run the 4-bit-weight × 8-bit-activation FFN on its own.

Quantization recipe

layers	precision	bytes/param
FFN mlp.{gate,up,down}_proj	NVFP4 W4A8 — 4-bit E2M1 weight, group-16 E4M3 scale, FP8 dynamic-per-token act	~0.56
Attention self_attn.{q,k,v,o}_proj, GDN linear_attn.{in_proj_qkv, in_proj_z, out_proj}	FP8 W8A8 — per-channel weight, per-token act	1.0
GDN gates (in_proj_a/b/ba, conv1d), norm, embed_tokens, lm_head, visual.*, mtp.*	bf16 (unquantized)	2.0

Data-free quantization (memoryless_minmax); format is standard compressed-tensors.

Memory footprint (measured, Qwen3.6-27B, B300)

serving mode	weight footprint
bf16 (base)	~54 GiB
default (rowwise, fp8 weights in HBM)	27.65 GiB
VLLM_NVFP4_VRAM=1 (4-bit weights in HBM)	21.77 GiB

⚠️ You must set VLLM_NVFP4_VRAM=1 to get the 22 GiB footprint. Without it, the 4-bit FFN is expanded to fp8 at load and you land at ~27.6 GiB. The MTP head adds ~0.8 GiB only when speculative decoding is enabled; for normal serving the footprint is unchanged.

Requirements

GPU	Blackwell — B200 (sm_100), B300 (sm_103), or RTX 5090 / PRO 6000 (sm_120)
vLLM	0.23.x
extras	the nvfp4_fp8 kernel wheel + vllm_nvfp4_fp8 plugin (registers the nvfp4_fp8_w4a8 quant method)

If you are using CUDA 13.0.x, Torch 2.11 and Python 3.12 prebuilt wheels are included in the repo. Self-compilation required otherwise.

To use the prebuilt wheels:

pip install nvfp4_fp8-0.1.0-cp312-cp312-linux_x86_64.whl
pip install vllm_nvfp4_fp8-0.1.0-py3-none-any.whl

Usage

VLLM_NVFP4_VRAM=1 \
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
vllm serve selimaktas/Qwen3.6-27B-NVFP4-FP8-MTP \
    --tensor-parallel-size 1 \
    --tokenizer Qwen/Qwen3.6-27B \
    --gpu-memory-utilization 0.92 \
    --attention-backend FLASHINFER \
    --max-model-len 262144 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3 \
    --default-chat-template-kwargs '{"enable_thinking": true, "preserve_thinking": true}' \
    --trust-remote-code

For text-only serving, add --language-model-only For MTP, add --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":3}'

If you run out of VRAM, add --enforce-eager. This will make the model run slower, but with a lower VRAM footprint.

Evaluation

SWE-Bench Verified (Single-turn, no tool use, avg@3):

Tasks	Qwen/Qwen3.6-27B	selimaktas/Qwen3.6-27B-NVFP4-FP8-MTP (this)
swe_bench_verified	26.9%	26.8%

More soon!

Throughput Numbers

Soon!

What's preserved

• Vision tower — full 333-tensor ViT, bf16, untouched. Multimodal inputs work.
• MTP head — restored as bf16 (the quantizer's model loader drops it; merged back from the base with vllm_plugin/integrate_aux.py).

Hardware notes

• B200 / B300: the full path runs on the SM100 tensor-core (tcgen05) datapath.
• RTX 5090 / PRO 6000 (SM120): same 4-bit footprint and the same decode/prefill kernels (portable CUDA + cuBLAS fp8); the one tcgen05-only fallback is auto-replaced by a tiled GEMV. Compiled and correctness-checked for sm_120a, further being tested.

Limitations

• Inherits the base model's capabilities and biases; quantization adds no new alignment.
• Serving requires the kernel + plugin wheels — it will not load on stock vLLM.