s-batman/Agents-A1-NVFP4-MTP-GGUF

Agents-A1 NVFP4/MXFP4 GGUF with MTP

Quantized GGUF files for InternScience/Agents-A1, a Qwen3.5-35B-A3B MoE multimodal model.

Attribution

• Original model: InternScience/Agents-A1 — a Qwen3.5-35B-A3B MoE multimodal derivative
• MTP heads: Grafted from unsloth/Qwen3.6-35B-A3B-MTP-GGUF
• Quantization framework: llama.cpp with NVFP4/MXFP4 MoE routing patches (commits 1f8583c6b, 6e761dbc9)
• Tensor mapping: Follows Unsloth's gold standard MoE quantization strategy
• Quantized by: s-batman

Files

File	Format	Size	BPW	Tensor Type	Notes
agents-a1-NVFP4-MTP.gguf	NVFP4 (type 40)	19.8 GB	4.80	80 NVFP4 + 320 Q8_0 + 308 F32 + 5 MTP donor	Blackwell-native FP4, MTP-grafted
agents-a1-MXFP4-MTP.gguf	MXFP4 (type 39)	18.9 GB	4.57	80 MXFP4 + 320 Q8_0 + 308 F32 + 5 MTP donor	MXFP4, MTP-grafted
mmproj-agents-a1-f16.gguf	F16	858 MB	16	334 F16 tensors	Vision tower (mmproj), never quantized

Tensor Type Mapping

Follows Unsloth's gold standard MoE quantization mapping:

Tensor Category	Count	Quant Format
3D expert tensors (ffn_*_exps)	80	NVFP4 / MXFP4
Sensitive 2D (attn, embeddings, output, shared experts)	312	Q8_0
Routers (ffn_gate_inp, ffn_gate_inp_shexp)	2	F32
Norms, biases	301	F32
MTP block (blk.40.*)	20	Mixed (from MTP donor)

MTP (Multi-Token Prediction)

MTP heads grafted from unsloth/Qwen3.6-35B-A3B-MTP-GGUF for speculative decoding support.

• block_count: 40 → 41
• nextn_predict_layers: 1
• 20 MTP tensors at blk.40.* (separate gate/up experts, not fused)

Usage with MTP speculative decoding

llama-server \
  -m agents-a1-NVFP4-MTP.gguf \
  --mmproj mmproj-agents-a1-f16.gguf \
  -c 4096 -ngl 99 -fa 1 \
  -b 1024 -ub 4096 \
  --spec-type draft-mtp --spec-draft-n-max 8

Architecture

• Architecture: Qwen3.5 MoE (qwen35moe)
• Layers: 40 (30 linear attention + 10 full attention, every 4th)
• Experts: 256 total, 8 per token, 1 shared expert
• Hidden size: 2048
• Head dim: 256
• Vocab: 248,320
• Context: 262,144
• Vision: ViT depth 27, hidden 1152 (multimodal)

Quantization Details

Converted from BF16 GGUF using llama-quantize with:

• --output-tensor-type q8_0
• --token-embedding-type q8_0
• --no-mtp --fuse-gate-up-exps (during HF→GGUF conversion)
• MoE routing: 3D expert tensors (ne[2] > 1) → FP4, 2D linear tensors → Q8_0, norms → F32

Benchmarks

All benchmarks run on NVIDIA GB10 (DGX Spark) — 124 GB unified memory, compute capability 12.1, BLACKWELL_NATIVE_FP4=1. Benchmark tool: llama-bench (build 9277) and llama-server with chat completions.

Prompt Processing (prefill) — varying ubatch size

Fixed: b=4096, pp2048, ngl=99, fa=1, 3 repetitions.

Model	ub=512	ub=1024	ub=2048	ub=4096
NVFP4	1,142 t/s	1,254 t/s	1,470 t/s	1,642 t/s
MXFP4	1,320 t/s	1,562 t/s	1,619 t/s	1,618 t/s

Best prefill: NVFP4 at ub=4096 (1,642 t/s), MXFP4 at ub=2048 (1,619 t/s). Larger ubatch consistently improves prefill throughput.

Token Generation (decode) — varying batch size

Fixed: ub=512, tg256, ngl=99, fa=1, 3 repetitions.

Model	b=512	b=1024	b=2048	b=4096
NVFP4	23.1 t/s	25.4 t/s	25.4 t/s	23.6 t/s
MXFP4	24.8 t/s	25.4 t/s	25.3 t/s	22.2 t/s

Best decode: Both formats at b=1024 (25.4 t/s). Decode performance is relatively flat across batch sizes, with b=4096 showing slight degradation.

Chat completions — MTP speculative decoding sweep

Fixed: b=1024, ub=4096, c=4096, ngl=99, fa=1, 256 max tokens, temperature 0.7. Values are averages of 3 runs. MTP heads grafted from unsloth/Qwen3.6-35B-A3B-MTP-GGUF.

NVFP4 — varying --spec-draft-n-max

n-max	Decode (t/s)	Accept Rate	Accepted/Generated	vs Baseline
— (baseline)	69.3	—	—	—
2	84.1	62.2%	423/680	+21.3%
3	77.9	50.5%	460/911	+12.4%
4	71.6	~43.7%	483/1113	+3.3%
8	58.0	36.0%	~190/502	−16.3%

Key finding: --spec-draft-n-max 2 is the optimal setting on GB10, delivering a 21% speedup over baseline (84.1 t/s vs 69.3 t/s). The accept rate at n-max=2 is 62% — substantially higher than at n-max=8 (36%) because shorter draft sequences have higher per-token match probability and lower overhead.

At n-max=8, the overhead of generating 8 draft tokens per step with only 36% acceptance makes MTP net-negative. The break-even point on GB10 is around n-max=4.

MXFP4 — n-max=8 (prior runs, with mmproj)

n-max	Decode (t/s)	Accept Rate	vs Baseline
— (baseline)	69.0	—	—
8	58.2	35%	−15.7%

MXFP4 shows the same pattern as NVFP4 at n-max=8. Based on the NVFP4 sweep, n-max=2 is expected to provide a similar ~20% speedup for MXFP4 as well.

Note: The tg256 numbers from llama-bench (25 t/s) differ from the chat completion numbers (69 t/s) because llama-bench measures raw decode throughput with no chat template overhead, while the chat tests include Jinja template processing, thinking mode, and warmup effects. The relative comparison between MTP and no-MTP is valid within the same test type.

Recommended configuration

# Best decode with MTP speculative decoding (recommended)
llama-server -m agents-a1-NVFP4-MTP.gguf --mmproj mmproj-agents-a1-f16.gguf \
  -c 4096 -ngl 99 -fa 1 -b 1024 -ub 4096 -np 1 \
  --spec-type draft-mtp --spec-draft-n-max 2

# Baseline without MTP
llama-server -m agents-a1-NVFP4-MTP.gguf --mmproj mmproj-agents-a1-f16.gguf \
  -c 4096 -ngl 99 -fa 1 -b 1024 -ub 4096 -np 1

License

MIT

---

Quantized by s-batman. For additional GGUF quants, benchmarks, and articles on LLM memory and context engineering, visit sbatman.com.