---
license: apache-2.0
base_model: Qwen/Qwen3.6-35B-A3B
tags:
  - gguf
  - quantized
  - apex
  - apex-mtp
  - moe
  - mixture-of-experts
  - qwen3
  - qwen3.6
  - speculative-decoding
  - self-speculative
  - mtp
---

<!-- apex-banner-v2 -->
<div style="background-color: #f59e0b; color: white; padding: 20px; border-radius: 10px; text-align: center; margin: 20px 0;">
<h2 style="color: white; margin: 0 0 10px 0;">⚡ Each donation = another big MoE quantized</h2>
<p style="font-size: 18px; margin: 0 0 15px 0;">I host <b>30+ free APEX MoE quantizations</b> as independent research. My only local hardware is an <b>NVIDIA DGX Spark</b> (122 GB unified memory), enough for ~30-50B-class MoEs, but <b>bigger ones (200B+) require rented compute</b> on H100/H200/Blackwell, typically $20-100 per quant.<br>If APEX quants are useful to you, your support directly funds those bigger runs.</p>
<p style="font-size: 20px; margin: 0;">
<a href="https://www.patreon.com/cw/mudler" style="color: white; text-decoration: underline;">🎉 Patreon (Monthly)</a> &nbsp;|&nbsp;
<a href="https://www.buymeacoffee.com/mudler" style="color: white; text-decoration: underline;">☕ Buy Me a Coffee</a> &nbsp;|&nbsp;
<a href="https://github.com/sponsors/mudler" style="color: white; text-decoration: underline;">⭐ GitHub Sponsors</a>
</p>
</div>

# Qwen3.6-35B-A3B — APEX-MTP GGUF

**APEX (Adaptive Precision for EXpert Models)** quantizations of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B), with the **MTP (multi-token prediction) head bundled** for in-the-box self-speculative decoding.

**Brought to you by the [LocalAI](https://github.com/mudler/LocalAI) team** | [APEX Project](https://github.com/mudler/apex-quant) | [Technical Report](https://github.com/mudler/apex-quant/blob/main/paper/APEX_Technical_Report.pdf)

## What's different from the plain APEX repo?

These GGUFs bundle the model's **MTP (multi-token prediction) head** alongside the trunk in a single file, courtesy of [llama.cpp PR #22673](https://github.com/ggml-org/llama.cpp/pull/22673). With a recent llama.cpp (>= commit 255582687) you can enable self-speculative decoding using just this one file — no separate draft model needed:

```bash
llama-server -m Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf --draft-mtp
```

The non-MTP version is still available at [mudler/Qwen3.6-35B-A3B-APEX-GGUF](https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-GGUF) — slightly smaller, but no self-spec.

## File sizes

Each quant is ~2.5% larger than its non-MTP counterpart (one extra transformer-block worth of weights, no embedding duplication since MTP shares the trunk's embed_tokens).

## MTP draft head precision

The bundled MTP head (`blk.40.*` including the `nextn.*` projection + norms) is
quantized to **Q8_0** (near-lossless) on **every tier except I-Nano**. I-Nano keeps
the trunk-tier precision on the MTP block (Q3_K routed experts, Q4_K attention)
but pins `blk.40.nextn.eh_proj` to Q4_K — see the [explainer below](#why-the-mtp-head-doesnt-use-imatrix).

This keeps draft accuracy high (important for spec-decode acceptance rate) at a
modest ~1 GB cost per file vs. trunk-tier precision.

### Why the MTP head doesn't use imatrix

`llama-imatrix` runs normal forward passes that only activate the trunk
(`blk.0..blk.39`). The MTP head only fires during `--draft-mtp` spec decoding,
so its tensors get no imatrix activation data. We work around this by
quantizing the MTP head with static K-quant / Q8_0 which doesn't require
imatrix.

(A patch to `llama-imatrix` that records MTP activations during collection
is in progress at [mudler/llama.cpp#mtp-imatrix](https://github.com/mudler/llama.cpp/tree/mtp-imatrix)
— once upstream this will let us push the drafter to lower bit-widths cleanly.)

## What is APEX?

APEX is a MoE-aware mixed-precision quantization strategy. Per-tensor-role gradient: routed experts compress hardest, shared experts kept high (always active), attention/Mamba uniform; 5+5 symmetric edge gradient across the 40 trunk layers + MTP layer 40 at edge precision. I-variants use diverse imatrix calibration (chat, code, reasoning, tool-calling, agentic traces, Wikipedia).

See the [APEX project](https://github.com/mudler/apex-quant) for full details.

## Architecture

- **Base**: Qwen 3.6 35B-A3B family (Qwen3_5MoeForCausalLM)
- **Layers**: 40 trunk + 1 MTP (bundled)
- **Experts**: 256 routed + 1 shared (8 active per token)
- **Hidden size**: 2048
- **Calibration**: v1.3 diverse dataset

## Credits

- **APEX quantization**: [LocalAI](https://github.com/mudler/LocalAI) team
- **MTP support**: llama.cpp PR #22673 by Aman Gupta + ggerganov
- Built on [llama.cpp](https://github.com/ggerganov/llama.cpp)