localweights/Qwen3.6-35B-A3B-MTP-BF16-GGUF

Qwen3.6-35B-A3B-MTP (BF16 GGUF)

Pure BF16 GGUF checkpoint (71.1 GB) with baked NextN/MTP block for speculative decoding.
Architecture: Mixture‑of‑Experts (35 B total, 3 B active per token, 256 experts, 8 active). Base model: Qwen/Qwen3.6-35B-A3B. License: Apache‑2.0.

---

What is this?

localweights/Qwen3.6-35B-A3B-MTP-bf16.gguf is a lossless conversion of the original BF16‑trained Safetensors checkpoint into the GGUF format used by llama.cpp.
During conversion the NextN / Multi‑Token Prediction (MTP) speculative‑decoding head is baked directly into the model graph, enabling self‑speculative generation without any external adapter.

---

When to use BF16 vs IQ4_XS

Variant	Size	Precision	Typical hardware	Quality
BF16 (this)	71.1 GB	Full BF16 (lossless)	80 GB+ GPU RAM or hybrid GPU + CPU offload	Baseline (identical to original HF checkpoint)
IQ4_XS‑Q8nextn (sister repo)	18.5 GB	4‑bit IQ4_XS + Q8nextn for speculative head	24 GB GPU (e.g., RTX 4090)	Near‑baseline, ~0.3 % perplexity loss

Choose BF16 when you need exact model fidelity (research, benchmarking, fine‑tuning). Use the quantized IQ4_XS variant when GPU memory is limited and a small quality trade‑off is acceptable.

---

Hardware requirements

• GPU‑only: ≥ 80 GB VRAM (A100 80GB, H100 80GB, etc.) – runs at full speed, ~30 tok/s on A100.
• Hybrid GPU + CPU offload: 24 GB GPU + ≥ 96 GB system RAM.
  Example: -ngl 8 (8 GB GPU layers) on a Ryzen 9 9950X + RTX 3090 Ti yields ~12 tok/s.
• CPU‑only: Feasible with DDR5 memory, ~3–5 tok/s. Expect higher latency; useful for inference on servers without a high‑memory GPU.

---

Quick start (llama.cpp)

# Clone and build llama.cpp with MoE and speculative decoding support
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
git checkout f0e9c2b   # commit with --cpu-moe support
make LLAMA_BUILD_MOE=1 LLAMA_BUILD_SPECULATIVE=1

# Run the model (GPU‑only, 80 GB VRAM)
./main -m localweights/Qwen3.6-35B-A3B-MTP-bf16.gguf \
       -p "Explain quantum entanglement in simple terms." \
       -c 2048 -ngl 0 \
       --speculative-mtp

# Hybrid offload (24 GB GPU, 8 GB GPU layers)
./main -m localweights/Qwen3.6-35B-A3B-MTP-bf16.gguf \
       -p "Write a short poem about sunrise." \
       -c 2048 -ngl 8 \
       --cpu-moe --n-cpu-moe 8 \
       --speculative-mtp

• --cpu-moe tells llama.cpp to keep inactive experts on the CPU. Only the 8 active experts per token (~13 % of total MoE weights) travel across the PCIe bus.
• --n-cpu-moe N controls how many expert groups stay on CPU (default = 8). Adjust to fit your CPU‑GPU bandwidth budget.

---

MoE offload guidance

Benchmarks from chair-moe-offload-bench-2026-05-07 (Ryzen 9 9950X + RTX 3090 Ti, DDR5‑5600, PCIe 4.0):

Offload mode	GPU memory used	Generation speed (tok/s)	Remarks
Full GPU (no offload)	> 80 GB (not feasible)	–	BF16 exceeds 24 GB limit
-ncmoe 16 (16 GB GPU MoE)	~12.1 GB	106 tok/s	Active experts on GPU, rest on CPU
-cmoe (CPU‑only MoE)	~1.9 GB	67 tok/s	Frees > 21 GB GPU for concurrent specialist models
-ngl 8 (hybrid layer offload)	~8 GB (layers) + MoE offload	~12 tok/s	Good balance for 24 GB GPUs

Tips

1. Start with -ncmoe 16; if GPU memory is still tight, switch to -cmoe.
2. Monitor PCIe traffic; the MoE offload adds ~0.8 GB/s per active expert group.
3. Combine -ngl with MoE offload for maximal flexibility (e.g., -ngl 4 -ncmoe 12).

---

Speculative decoding (MTP head)

The checkpoint contains a baked NextN block that implements the Multi‑Token Prediction (MTP) speculative decoding scheme described in the Qwen3 paper. When used with a llama.cpp build that includes LLAMA_BUILD_SPECULATIVE, the model can:

• Generate ≈ 2.7× speed‑up over vanilla single‑token decoding (single‑slot speculative).
• Perform self‑speculative decoding without an external draft model; the MTP head predicts a short token sequence that is later verified against the main model.
• Require modest extra memory (≈ 200 MB) for the speculative KV cache.

Usage is simply the --speculative-mtp flag shown in the quick‑start commands.

---

Sister releases (localweights collection)

Variant	File	Size	Intended use
IQ4_XS‑Q8nextn	Qwen3.6-35B-A3B-MTP-iq4xs-q8nextn.gguf	18.5 GB	24 GB GPUs, near‑baseline quality
Q4_K_M‑Q8nextn	Qwen3.6-35B-A3B-MTP-q4k_m-q8nextn.gguf	12.2 GB	Extreme memory‑constrained environments
BF16 (this)	Qwen3.6-35B-A3B-MTP-bf16.gguf	71.1 GB	Research, fine‑tuning, exact replication

All variants share the same baked MTP block and MoE architecture.

---

Building from source (conversion)

If you need to regenerate the GGUF file from the original HF checkpoint:

git clone https://github.com/localweights/gguf-converter
cd gguf-converter
pip install -r requirements.txt

# Convert the HF safetensors checkpoint to BF16 GGUF
python convert_hf_to_gguf.py \
    --model_dir  Qwen/Qwen3.6-35B-A3B \
    --outtype    bf16 \
    --output    Qwen3.6-35B-A3B-MTP-bf16.gguf \
    --add-mtp   # injects the NextN/MTP block

Note: The conversion script applies a patched Qwen3NextModel.filter_tensors routine that keeps the MTP tensors intact. The patch is bundled with this repository’s quantizer.

---

Citation

If you use this model in research, please cite the original Qwen3 family and the speculative decoding work:

@article{tong2024qwen3,
  title   = {Qwen3: A New Generation of Large Language Models},
  author  = {Tong, Zhi and others},
  journal = {arXiv preprint arXiv:2405.09388},
  year    = {2024}
}

@inproceedings{zhang2024mtp,
  title     = {Multi‑Token Prediction for Efficient Speculative Decoding},
  author    = {Zhang, Wei and Li, Hao and others},
  booktitle = {Proceedings of the 2024 Conference on Neural Information Processing Systems},
  year      = {2024}
}

@misc{llamacpp2024,
  author       = {Georgi Gerganov},
  title        = {llama.cpp: Inference of LLaMA models in pure C/C++},
  year         = {2024},
  howpublished = {\url{https://github.com/ggerganov/llama.cpp}}
}

---

License

The model is released under the Apache‑2.0 license, identical to the base model Qwen/Qwen3.6-35B-A3B. See the LICENSE file in the original repository for the full text. Use of the model is subject to the terms of that license.