Qwen3.8-27B-NVFP4-MTP-GGUF

A family of nine GGUF files of Qwen3.8-27B (the native vision-language 27B dense model, Gated DeltaNet + Gated Attention hybrid layout, 262,144-token native context, MTP speculative head), converted from unsloth/Qwen3.8-27B-NVFP4. The MTP (multi-token prediction) speculative head is baked into every file — no separate drafter needed.

• ORIG — the source-preserving conversion: native NVFP4 MLP backbone + BF16 attention/embeddings (the source's F8 attention is dequantized to BF16 because GGML has no F8 tensor type). This is the largest file and the one all tiers are derived from.
• VERY-LOW / COMPACT-LOW / LOW / MEDIUM / MID-HIGH / HIGH / VERY-HIGH — a compact family sharing a byte-identical 448-tensor NVFP4 backbone (all attention + MLP re-quantized to NVFP4), differing only in the 10 "extra" tensors (LM head, token embedding, MTP draft head). COMPACT-LOW fills the gap between VERY-LOW and LOW — slightly smaller than LOW while keeping a materially stronger LM head than VERY-LOW (Q4_K vs Q3_K). MID-HIGH sits between MEDIUM and HIGH with all three head groups at Q8_0.
• HIGHEST — the top tier: keeps the source's native NVFP4 MLP (layers 0-55) exactly as in ORIG, restores Q8_0 for attention + the late MLP layers + the LM head, and keeps the token embedding + MTP head in BF16. The closest compact approximation of the source layout, for high-end GPUs.

The goal: keep native NVFP4 density across the whole model for Blackwell, and offer a size/precision ladder for the tensors that most affect output quality and decode speed. On our dual 16 GB Blackwell setup every tier fits and runs (see notes before treating any numbers as meaningful).

Vision works. The model is a native VLM (images and video). Pair any of these GGUFs with the mmproj-BF16.gguf included in this repo (byte-identical to unsloth/Qwen3.8-27B-GGUF's projector) via --mmproj.

For 16 GB VRAM users who want NVFP4 as small as possible: the same NVFP4 backbone, squeezed further — the MTP head stripped out and the LM head / embeddings trimmed to the absolute minimum — is published as BUDGET / STARVED in esatapedico/Qwen3.8-27B-NVFP4-BUDGET-GGUF.

Follow along & support

I post updates on new conversions, benchmarks, and what I'm working on over on Ko-fi. Follow along there to keep up with new releases and the work in progress. If you'd like to support more of it, a coffee is always welcome. I do this on consumer hardware and like seeing how far it goes. More is on the way.

☕ ko-fi.com/esatapedico. Updates, work-in-progress, and an optional coffee.

The nine files

File	Size	lm_head (output.weight)	token_embd	MTP head (blk.64)	Attention
Qwen3.8-27B-NVFP4-MTP-ORIG.gguf	33.13 GB	BF16	BF16	BF16	BF16 (source F8 dequant)
Qwen3.8-27B-NVFP4-MTP-VERY-LOW.gguf	14.86 GB	Q3_K	Q2_K	Q2_K	NVFP4
Qwen3.8-27B-NVFP4-MTP-COMPACT-LOW.gguf	15.16 GB	Q4_K	Q3_K	Q2_K	NVFP4
Qwen3.8-27B-NVFP4-MTP-LOW.gguf	15.53 GB	Q5_0	IQ4_XS	IQ4_XS	NVFP4
Qwen3.8-27B-NVFP4-MTP-MEDIUM.gguf	16.38 GB	Q8_0	Q6_K	IQ4_XS	NVFP4
Qwen3.8-27B-NVFP4-MTP-MID-HIGH.gguf	16.90 GB	Q8_0	Q8_0	Q8_0	NVFP4
Qwen3.8-27B-NVFP4-MTP-HIGH.gguf	17.57 GB	BF16	Q6_K	IQ4_XS	NVFP4
Qwen3.8-27B-NVFP4-MTP-VERY-HIGH.gguf	19.69 GB	BF16	BF16	BF16	NVFP4
Qwen3.8-27B-NVFP4-MTP-HIGHEST.gguf	23.19 GB	Q8_0	BF16	BF16	Q8_0
mmproj-BF16.gguf	931 MB	— (vision projector)

Tensor layout

ORIG (1,202 tensors): the direct --outtype auto conversion of the pre-processed source checkpoint.

GGML type	Tensors	Size	Component
NVFP4	168	8.42 GB	MLP gate/up/down of layers 0-55 (source NVFP4)
BF16	338	24.69 GB	attention (self_attn + linear_attn, source F8 dequantized), MLP layers 56-63, LM head, embeddings, MTP head
F32	696	0.01 GB	norms, scales

VERY-LOW–VERY-HIGH (1,202 tensors): the 7 compact tiers share a byte-identical 448-tensor NVFP4 backbone (all 64 layers' attention + MLP, 13.69 GB) and differ only in the 10 extra tensors:

GGML type	Tensors	Component
NVFP4	448	all 64 transformer blocks (attention QKV/output + FFN gate/up/down) — identical in all tiers
F32	744	norms, gates, scales
tier-dependent	10	output.weight, token_embd.weight, blk.64.* MTP draft block (incl. nextn.eh_proj)

MID-HIGH (1,202 tensors): the same compact 448-tensor NVFP4 backbone with all three head groups at Q8_0 (LM head, token embedding, MTP draft head) — the highest-precision heads the compact backbone can carry. Same tensor layout as the table above, with the 10 tier-dependent tensors at Q8_0 (3.15 GB).

HIGHEST (1,202 tensors): mirrors the source layout — the NVFP4 MLP of layers 0-55 is byte-identical to ORIG, the BF16 tensors (attention, MLP 56-63, LM head) are restored to Q8_0, and the token embedding + MTP head stay BF16.

GGML type	Tensors	Size	Component
NVFP4	168	8.42 GB	MLP gate/up/down of layers 0-55 (byte-identical to ORIG)
Q8_0	281	11.30 GB	attention (self_attn + linear_attn), MLP layers 56-63, LM head
BF16	9	3.39 GB	token_embd, MTP head (blk.64)
F32	744	0.01 GB	norms, gates, scales

The MTP draft head is embedded in every GGUF (blk.64.nextn.*), so no separate drafter file is needed. Enable it in llama.cpp with --spec-type draft-mtp.

Why the ORIG file exists, and why the tiers are the way they are

The source checkpoint, unsloth/Qwen3.8-27B-NVFP4, is a mixed-precision compressed-tensors model:

• group_0 — F8 float-quantized (float-quantized, 8-bit): self_attn.q/k/v/o_proj, linear_attn.in_proj_qkv/in_proj_z/out_proj, lm_head, and the MLP of layers 56-63
• group_1 — NVFP4 (nvfp4-pack-quantized, 4-bit): the MLP of layers 0-55

llama.cpp's converter does not yet accept a compressed-tensors checkpoint with more than one config group unless all are NVFP4 (it raises NotImplementedError). To convert this model we therefore:

1. Pre-processed the checkpoint: dequantized the 233 F8 tensors to BF16 in place (per-channel weight × scale), removed their weight_scale tensors, kept the NVFP4-packed MLP tensors untouched, and pinned the config to a single nvfp4-pack-quantized group.
2. Converted with convert_hf_to_gguf.py --outtype auto → ORIG (NVFP4 MLP preserved natively, F8 attention materialized as BF16).
3. Built the 8 tiers with llama-quantize --tensor-type-file <overrides> from the ORIG parent: the 7 compact tiers re-quantize the BF16 attention to NVFP4 for a compact uniform backbone and pin the extra tensors per tier; MID-HIGH is the compact tier with all three head groups at Q8_0; HIGHEST instead restores attention/MLP56-63/lm_head to Q8_0 while keeping the NVFP4 MLP and BF16 embedding/MTP head.

Honest trade-off note: the compact tiers re-quantize attention from BF16 (which was F8 in the source) down to NVFP4. That is a second quantization step on those tensors (F8 → BF16 → NVFP4) and does lose some attention precision vs the source. HIGHEST keeps the same NVFP4 MLP as ORIG and uses Q8_0 (instead of BF16) for the attention path — much closer to the source's effective precision at ~10 GB less than ORIG. The ORIG file preserves the source's attention quality at the cost of a 33 GB file. If you want the highest-fidelity version, use ORIG or HIGHEST; if you need the compact footprint, the tiers are ~2× smaller.

Attribution & provenance

This is a derivative work built entirely from existing Apache-2.0 artifacts. Nothing here was trained or fine-tuned. Credit belongs to:

1. Alibaba / Qwen team for the base model, Qwen/Qwen3.8-27B (Apache-2.0): 27B dense, 64 layers, Gated DeltaNet + Gated Attention hybrid, native vision-language, native 262,144-token context, MTP head.
2. Unsloth for the NVFP4 quantization unsloth/Qwen3.8-27B-NVFP4 (Apache-2.0) and the vision projector unsloth/Qwen3.8-27B-GGUF (Apache-2.0, mmproj-BF16.gguf).
3. This repo's author for the F8→BF16 pre-processing, the GGUF conversion, and the tier splicing.

The NVFP4 tensors are native GGML type 40, preserved from the source checkpoint through GGUF conversion (for the MLP backbone) with no re-quantization round trip.

How this was made

1. unsloth/Qwen3.8-27B-NVFP4 (safetensors, mixed F8+NVFP4) was pre-processed by dequantizing F8→BF16 and keeping the NVFP4-packed tensors (see above) — this is the only step that touches values beyond type conversion, and only on the 233 F8 tensors.
2. Converted to GGUF with convert_hf_to_gguf.py --outtype auto → ORIG (33.13 GB).
3. Each tier produced with llama-quantize --tensor-type-file <overrides> from the ORIG parent. The per-tier override maps are in this repo (overrides-very-low.txt, overrides-compact-low.txt, overrides-low.txt, overrides-medium.txt, overrides-mid-high.txt, overrides-high.txt, overrides-very-high.txt, overrides-highest.txt, 1,202 entries each).

The NVFP4 backbone is byte-identical across the family: the 448-tensor backbone of the seven compact tiers is the same set of bytes in every tier, and HIGHEST's 168 NVFP4 MLP tensors are byte-identical to ORIG's (verified: per-tensor SHA-256).

Repository contents

• Qwen3.8-27B-NVFP4-MTP-ORIG.gguf (33.13 GB) — source-preserving conversion
• Qwen3.8-27B-NVFP4-MTP-VERY-LOW.gguf (14.86 GB)
• Qwen3.8-27B-NVFP4-MTP-COMPACT-LOW.gguf (15.16 GB)
• Qwen3.8-27B-NVFP4-MTP-LOW.gguf (15.53 GB)
• Qwen3.8-27B-NVFP4-MTP-MEDIUM.gguf (16.38 GB)
• Qwen3.8-27B-NVFP4-MTP-MID-HIGH.gguf (16.90 GB)
• `Qwen3.8-27B-NVFP4-MTP-HIGH.gguf (17.57 GB)
• Qwen3.8-27B-NVFP4-MTP-VERY-HIGH.gguf (19.69 GB)
• Qwen3.8-27B-NVFP4-MTP-HIGHEST.gguf (23.19 GB)
• mmproj-BF16.gguf (931 MB) — vision projector (byte-identical to unsloth's)
• overrides-very-low.txt, overrides-compact-low.txt, overrides-low.txt, overrides-medium.txt, overrides-mid-high.txt, overrides-high.txt, overrides-very-high.txt, overrides-highest.txt — per-tensor quantization maps (1,202 entries each) for reproduction

SHA-256

f18098dfc32ca398f63093c113d48bd962684c78a3e93d4158f8f440699f7cae  Qwen3.8-27B-NVFP4-MTP-ORIG.gguf
74ea17ea05e0e0241af8d5b29cdea38b3f4509f66d9b96c1ab05f0e1f0e537d9  Qwen3.8-27B-NVFP4-MTP-VERY-LOW.gguf
ac0ef9c5eceb5a5dc9b266eacc9372508158713c6c35618cf735675451fdd3ac  Qwen3.8-27B-NVFP4-MTP-COMPACT-LOW.gguf
ce66a629d4a3516bba27ca91de29372f086f90f72ddb92fe298de67b8bb88bbc  Qwen3.8-27B-NVFP4-MTP-LOW.gguf
f0b4c538c75037f026bde3b650f0ca639d382c128a4572769dce1183db86253a  Qwen3.8-27B-NVFP4-MTP-MEDIUM.gguf
79b032f7a118fb34f1445c4d7ae50bc7b304c74161035c3df8bd5526c12899b9  Qwen3.8-27B-NVFP4-MTP-MID-HIGH.gguf
d57008707b0558bde05ce61d7402e4e668ffd45a4c97d03d2ad97db73f98d403  Qwen3.8-27B-NVFP4-MTP-HIGH.gguf
3e52d6280ee650520a2d901002121c11cf9d23ac75f23c52a362bf285d561d81  Qwen3.8-27B-NVFP4-MTP-VERY-HIGH.gguf
6a202c2faf67f79d4c8c61ec940da7a62bd59a87608508fe8048131630cc4ba6  Qwen3.8-27B-NVFP4-MTP-HIGHEST.gguf
83ee4f4f205fa514161778c41df1ea14144faa0f713510893b63c2395f5c2d53  mmproj-BF16.gguf

First observations (naive, single-run, not a benchmark)

We did not run a proper benchmark. What follows are informal first impressions from single-stream runs, included only so others know what to expect. Do not treat these as claims.

On reading the numbers: prefill t/s is a mean over the run and degrades as context grows, so figures from different context settings are not directly comparable — the 180k-context runs (645-650 t/s), the 32k-context runs (1250-2290 t/s) and the community RTX 5090 numbers all sit at different context sizes, hardware, and prompts. Compare numbers only within the same context and harness.

Our run (180k payload, one fresh pod per tier)

• Hardware: 2x NVIDIA Blackwell 16 GB (RTX 5070 Ti + RTX 5060 Ti), split-mode: tensor, flash attention on, quantized KV cache.
• Prompt: one 180,052-token payload to a ~1.2k-2k-token essay, max_tokens 20,000 (repetition_analysis.py). This payload was highly repetitive (the same boilerplate unit ~2,000 times), which likely inflates MTP acceptance and speedups.
• Sampling: temperature 0.6, top_p 0.95, top_k 20, min_p 0. We previously recommended DRY anti-loop sampling for this family; we no longer do — it interferes with verbatim reproduction of long strings (paths, identifiers, tool arguments), which matters for coding and tool use.
• MTP: --spec-type draft-mtp, spec_n_max 6, spec_p_min 0.75.
• Context: 307,200 (LOW, MEDIUM) / 221,184 (HIGH) / 204,800 (VERY-HIGH, MID-HIGH) — the larger HIGH/VERY-HIGH weights don't fit a 307,200-token KV cache on the 16 GB-per-GPU split.

Tier	Prefill t/s	Decode t/s	n_dec	MTP acc	MTP len	req s
LOW (Q5_0 / IQ4_XS / IQ4_XS)	645.6	18.44	1,151	0.726	2.88	361
MEDIUM (Q8_0 / Q6_K / IQ4_XS)	649.3	18.48	1,907	0.749	2.91	401
MID-HIGH (Q8_0 / Q8_0 / Q8_0)	649.0	18.00	2,465	0.765	2.82	427
HIGH (BF16 / Q6_K / IQ4_XS)	647.6	15.54	1,795	0.751	2.65	416
VERY-HIGH (BF16 / BF16 / BF16)	650.4	15.42	2,007	0.729	2.77	429

Naive first impressions:

• Prefill is effectively identical across tiers (~645-650 t/s) — expected: the 448-tensor NVFP4 backbone is byte-identical in all five files.
• Decode splits into two groups: LOW/MEDIUM/MID-HIGH (Q5_0/Q8_0 lm_head) at ~18-18.5 t/s vs HIGH/VERY-HIGH (BF16 lm_head) at ~15.5 t/s — the BF16 LM head costs more per MTP verification pass, the same pattern a community tester later measured on an RTX 5090. MID-HIGH (all-Q8_0 heads) tracks the Q8_0-lm_head group and stopped naturally at 2,465 tokens (finish: stop, with 6.2k chars of content) instead of hitting the 20k cap. Tokens generated differ per tier (1.2k-2.5k), so exact numbers are indicative.
• MTP acceptance 0.73-0.75 on this long repetitive payload — higher than short-prompt runs (see community numbers below); the draft head exploits the document's repetitive passages. Lower than our 711-family numbers on the same harness (0.87-0.88) — different model, not apples-to-apples.
• All "suspicious" repetition flags are reasoning-segment only; content is clean on every tier (distinct-5-gram ≥ 0.99, no adjacent-dup stutter) under the sampling preset used for this run. MEDIUM (Q8_0 head) shows the worst reasoning repetition (a 5-gram repeated 6×); HIGH is the cleanest compact tier.

New tiers at 32k context (one fresh pod per tier)

The newest tiers were exercised at 32,768 context with quantized q4_0 KV, on the same dual-16GB box, with a 28k-token payload (source-28k.txt) and max_tokens 2000 (COMPACT-LOW ran the same harness at max_tokens 20000).

Tier	Prefill t/s	Decode t/s	MTP acc	MTP len	req s
VERY-LOW (Q3_K / Q2_K / Q2_K)	—	—	—	—	not yet benchmarked
COMPACT-LOW (Q4_K / Q3_K / Q2_K)	2281.1	25.52	—	—	94.8
MID-HIGH (Q8_0 / Q8_0 / Q8_0)	1451.6	25.98	0.771	2.87	108.8
HIGHEST (Q8_0 attn / BF16 embd / BF16 MTP)	1250.4	23.49	0.765	2.88	122.6

First impressions:

• COMPACT-LOW runs at the same prefill speed as the compact tiers (~2281 vs ~2288 t/s) and slightly lower decode than LOW (25.5 vs ~27 t/s — Q4_K vs Q5_0 lm_head is the only decode-relevant difference, and the difference is small). Output was clean (content distinct-5-gram 0.978, no adjacent-dup stutter); the run stopped naturally (finish_reason: stop). MTP was left disabled in the deployed config for this run.
• A note on the COMPACT-LOW numbers: at 32k context the 14.1 GB model fits comfortably on a single 16 GB card, so this run likely exercised one GPU — which is consistent with its prefill matching the single-card BUDGET/STARVED numbers (~2287-2289 t/s) rather than a dual-GPU-split result. Treat these as single-GPU figures at 32k context.
• HIGHEST prefill is ~1.8× slower than the compact tiers at the same context (1250 vs ~2288 t/s) and decode ~15% slower (23.5 vs ~27 t/s) — the price of Q8_0 attention/lm_head + BF16 token_embd. MTP acceptance 0.765 / mean len 2.88, comparable to the compact tiers.
• MID-HIGH (16.9 GB) is too large for a single 16 GB card at 32k context (weights + KV exceed 16 GB), so this is a dual-GPU split-mode result: prefill 1451.6 t/s (between the single-card compact tiers at ~2280 and HIGHEST's split-mode 1250) and decode 25.98 t/s — the fastest split-mode decode in the family, ahead of HIGHEST's 23.49. MTP acceptance 0.771 / mean len 2.87, on par with the other tiers. The run hit the 2,000-token cap with reasoning-only output (distinct_ratio 0.91, suspicious flag) — the known heavy-reasoning behavior for this checkpoint on the repetitive 28k payload.
• HIGHEST is too large to pair with a big KV cache on 16 GB-per-GPU splits; it serves comfortably at 131,072 context and below. At 32k context it fits a single 16 GB card (BUDGET/STARVED of the sibling repo peak ~15.7-15.9 GiB).
• The BUDGET / STARVED ultra-compact tiers in the sibling repo (MTP head stripped to fit 16 GB) hit 2287-2289 t/s prefill and 27.0-27.4 t/s decode at 32k context.

Community benchmark on RTX 5090 (llama.cpp b10434) — by PierpaoloPernici

A community user benchmarked LOW / MEDIUM / VERY-HIGH on a single RTX 5090 32 GB (9 short prompts, seed 42, --spec-draft-n-max 4, 192K ctx, KV q8_0, temp 1.0). These are their numbers, not ours — see the full gist for the complete results (per-prompt breakdown, method, configs).

Tier	MTP acc	Avg tok/s	Peak tok/s	VRAM @192K
LOW (Q5_0 / IQ4_XS / IQ4_XS)	49.0%	148.3	203	25.0 GiB
MEDIUM (Q8_0 / Q6_K / IQ4_XS)	44.3%	133.7	182	25.3 GiB
VERY-HIGH (BF16 / BF16 / BF16)	49.5%	111.9	139	26.9 GiB
Qwen3.6-27B NVFP4-MTP ref (michaelw9999)	69.7%	113.8	133	—

Their main points:

• LOW is the throughput winner (~148 avg, up to 203 tok/s): the small Q5_0 LM head makes every target verification + MTP draft pass cheaper while acceptance stays ~49%.
• VERY-HIGH is the odd one out: it accepts ~5pp more drafts than MEDIUM but is slower — the BF16 LM head costs more on each verification pass than the acceptance gain saves. BF16 buys output quality, not speed.
• No quality cliff on LOW in their spot-checks (coherent outputs — e.g. answers a "two sentences" prompt with exactly 2 sentences).
• All three tiers beat the Qwen3.6 NVFP4 baseline on throughput despite lower acceptance.
• Long context (64K) raises acceptance: LOW 49→56%, MEDIUM 44→53% — the draft head exploits the document's repetitive passages.
• --spec-draft-n-max sweep (LOW): wall-time peak at n-max 4 (11.5 s / 9 prompts); n-max 2 is the most draft-efficient (68% accepted) but ~10% slower in wall time; n-max 6/8 collapse acceptance (38%/28%) and get slower. Our deployed config uses spec_n_max 6 — on their hardware 4 was fastest; try 1-6 and keep whatever is fastest on yours.
• Gotcha: a stale HF cache produced 0% MTP acceptance (mean len 1.00) until the file was re-downloaded — verify the file SHA before debugging anything else. They confirmed the re-uploaded hashes (LOW ce66a629…, MEDIUM f0b4c538…, VERY-HIGH 3e52d628…) match this repo's current files.

Benchmark tool: they published mtp-bench.py (9 prompts, seed 42, accept-rate / tok/s / wall, --long-context, --diff).

Usage

llama.cpp / llama-server

llama-server \
  --model Qwen3.8-27B-NVFP4-MTP-MEDIUM.gguf \
  --mmproj mmproj-BF16.gguf \
  --ctx-size 262144 \
  --flash-attn on \
  --spec-type draft-mtp \
  --spec-draft-n-max 6 \
  --spec-draft-p-min 0.75 \
  --temp 0.7 --top-p 0.95 --top-k 20

• Requires a recent llama.cpp with NVFP4 (GGML type 40) CUDA kernels and sm_120 support (Blackwell).
• Requires the draft-mtp spec path (merged upstream as LLAMA_CONTEXT_TYPE_MTP).
• For vision input, pass --mmproj mmproj-BF16.gguf.
• Qwen3.8's official sampling presets: thinking mode temp 1.0 / top_p 0.95 / top_k 20; instruct mode temp 0.7 / top_p 0.80 / top_k 20. Pick per use case.
• MTP performance is hardware-dependent: try --spec-draft-n-max values 1 through 6 and keep whatever is fastest on your system.

Notes

• Hardware: our test box is 2× NVIDIA Blackwell 16 GB (RTX 5070 Ti + RTX 5060 Ti). All tiers fit a 16 GB card at reasonable context; the larger tiers prefer a split or smaller context.
• First observations and a community RTX 5090 benchmark are in the section above (naive, single-run — do not treat as claims).
• The ORIG file is 33 GB — it fits a single 16 GB GPU only with heavy context reduction or CPU offload. It is included for fidelity and as the tier-derivation source, not as the recommended serving file.
• For ultra-compact NVFP4 tiers aimed at 16 GB VRAM (everything trimmed to the minimum, including removing the MTP head), see the sibling repository esatapedico/Qwen3.8-27B-NVFP4-BUDGET-GGUF.
• Non-NVFP4 (K-quant) GGUFs of the base model are available in Unsloth's original repository, unsloth/Qwen3.8-27B-GGUF — they run on any GPU but drop the native NVFP4 density this family is built around.

License

Apache-2.0, identical to every upstream artifact. The base model license governs; GGUF conversion and quantization are transformations, not new training. When redistributing, please retain attribution to Qwen (Alibaba) and Unsloth as above.

"Qwen" is a trademark of Alibaba. Trademarks are used here only to identify upstream models; this repository is not affiliated with, sponsored by, or endorsed by Alibaba or Unsloth.

Note on this card

This model card was written by an AI assistant at the request of the repository author, who did the engineering. As with any AI-generated text, there may be errors; please verify anything important (hashes, sizes, commands) against the file itself before relying on it.