README.md

---
library_name: transformers
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/LICENSE
pipeline_tag: image-text-to-text
base_model:
- DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF
tags:
- unsloth
- qwen
- qwen3_5
- GGUF
- llama.cpp
- MTP
datasets:
- tristandruyen/calibration_data_v5_rc
metrics:
- perplexity 
---

# Qwen3.6-27B-MTP-48Gb
## Overview

This repository hosts a production-ready `Q8_0` quantization of the exceptional Qwen3.6-27B fine-tune made by [DavidAU](https://huggingface.co/DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF) and [Nightmedia](https://huggingface.co/nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451), specifically optimized to **run within 48 GB of VRAM** (dual RTX 3090s via NVLink). The weights were calibrated using the great iMatrix file [calibration_data_v5_rc](https://gist.github.com/tristandruyen/9e207a95c7d75ddf37525d353e00659c) to precisely target layer-wise quantization sensitivity. On the tested hardware, this configuration achieves up to **~2,000 t/s prefill** and **~65–80 t/s decode** throughput with Multi-Token Prediction (MTP) enabled. The `Q8_0` format was selected for its optimal speed-to-accuracy ratio, lower-precision variants (e.g., Q6_K) showed a 5 to 10% throughput drop alongside reduced long-context stability, capping reliable usage at ~242k tokens. By preserving critical precision in attention and KV cache operations, this quantization enables stable inference at the model’s maximum context window.

> *Note: Due to VRAM constraints, the MMPROJ vision encoder must be offloaded to CPU when running multimodal tasks.*

## Research & Methodology
### Selective precision Quantization for high-speed inference

As previously disclosed, `Q8_0` is the proper format but the model became too big for the 48Gb target. Therefore, the strategy was to use Q6_K on just a few tensors. I managed to identify which were the least sensitive to quantization thanks to the iMatrix, which also identified the few tensors that needed more care in order to maintain the model's capacities.
The schema identified was :

| Tensor | Layers | Format |
| -- | --: | --: |
| `ssm_alpha` | 57, 58, 60, 61, 62 | `F16`  |
| `ssm_beta` | 57, 58, 60, 61, 62 | `F16`  |
| `attn_q`, `attn_k`, `attn_v` | 3, 7 | `F16`  |
| `attn_output` | 3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43, 47, 51, 55 | `Q6_K` |
| `ssm_out` | 0–2, 4–6, 8–10, 12–14, 16–18, 20–22, 24–26, 28–30, 32–34, 36–38, 40–42, 44–46, 48–50, 52–54, 56–58 | `Q6_K` |

Therefore, **93.91%** of the model is **`Q8_0`**, 5.44% is `Q6_K` and 0.61% is `F16`.

### iMatrix Calibration

The model was calibrated using the very well, and rightfully, known calibration data [`calibration_data_v5_rc`](https://gist.github.com/tristandruyen/9e207a95c7d75ddf37525d353e00659c). I tested several iMatrix dataset, including of my own making, but this file achieved the best results.

### Accuracy Analysis

To evaluate the precision loss after the mixed-quantization scheme, I compared the perplexity on Wiki-Text-raw and a custom made dataset, composed of code mainly found on llama.cpp's repo:

| Metric | Code | WikiText-Raw |
| ------------------- | -------------------: | -------------------: |
| **Base Mean Perplexity** | **1.879303 ± 0.005783** | **6.320168 ± 0.039392** |
| **Quant Mean Perplexity** | **1.879671 ± 0.005789** | **6.324147 ± 0.039467** |
| **PPL(Q) / PPL(Base)** | **1.000196 ± 0.000094** | **1.000630 ± 0.000145** |
| Cor(ln PPL) | 99.95% | 99.97% |
| **Mean KLD** | **0.000795 ± 0.000069** | **0.000746 ± 0.000023** |
| 99.9% KLD | 0.032576 | 0.031017 |
| 99.0% KLD | 0.008418 | 0.006217 |
| 95.0% KLD | 0.002935 | 0.002095 |
| Median KLD | 0.000029 | 0.000304 |
| Max KLD | 12.675076 | 1.559480 |
| **RMS Δp** | **0.885 ± 0.030 %** | **0.773 ± 0.031 %** |
| **Same Top-p** | *N/A* | **98.806 ± 0.028 %** |

**Key Findings:**
* **Near-Lossless:** The perplexity degradation is minimal at **+0.003979** on Wiki-Text-Raw, indicating that this mixed precision layout preserves the original model's reasoning and coding capabilities.
* **Reaches full-Q8_0 capacities**: Other tests made on a 'pure' Q8_0 quantization shown the same perplexity and KL-Div results, proof that this work has achieved its goals, lower memory footprint with no accuracy loss.

## Recommended Usage

To replicate the optimal performance (262K context, F16 K-Cache, Multi-GPU) using [`llama.cpp`](https://github.com/ggml-org/llama.cpp), use the following `llama-server` command.
Note the specific use of `--split-mode tensor` and `--tensor-split 1,1` for optimal PCIe bandwidth management across dual RTX 3090s. This command appeared to be the best one I could come across using an NVLink.
As previously disclosed, the `mmproj` file will **not** hold on the GPU, if you want to use it you need to add `--no-mmproj-offload` and `--image-min-tokens 1024` to the following command, alongside the `--mmproj /path/to/mmproj-BF16.gguf`.

``` bash
/path/to/llama.cpp/build/bin/llama-server \
    -m /path/to/Qwen3.6-27B-Fusion-711-Uncensored-MTP-Q8_0-Opt.gguf \
    --split-mode tensor \
    --tensor-split 1,1 \
    --host 0.0.0.0 \
    --port 8080 \
    --ctx-size 262144 \
    --parallel 1 \
    --gpu-layers 999 \
    --cache-type-k f16 \
    --cache-type-v q8_0 \
    --flash-attn on \
    -b 2048 -ub 2048 \
    --spec-type draft-mtp \
    --alias Qwen3.6-27b \
    --jinja
```

I also developed a proxy to enable users to select thinking or non-thinking behaviors, apply the recommended sampling parameters AND the `"Preserve Thinking"` option. You may find it on my [GitHub](https://github.com/AlexanderKyng/Qwen3.6-reasoning-toggle-proxy).

## Hardware Requirements
- **Target VRAM:** 48 GB (Tested on 2x NVIDIA RTX 3090 24GB).
- **RAM:** Minimum 32GB system RAM (Prompt caching, system overhead and MMPROJ).
- **Context limit:** The command above loads ~21GB of KV cache across the two GPUs. If you experience OOM (Out of Memory) errors, consider reducing `--ctx-size` or using 8-bit K-cache (`--cache-type-k q8_0`).

## Acknowledgments

This project was made possible thanks to the outstanding tools and contributions from the open-source AI community. Special thanks to:

*   **DavidAU and Nightmedia**: For their immense work done on this exceptionnal fine-tuned version.
*   **The Qwen Team**: For researching and releasing the exceptional Qwen3.6 architecture.
*   **llama.cpp**: Using the new Tensor split mode, it finally achieves extremelly high performances on dual-GPU setups.
*   **Froggeric**: For the immense work done on fixing Qwen3.5 and 3.6 chat_template and the base BF16 model used for this project.
*   **tristandruyen**: For the great calibration_data_v5-rc.txt file.