steampunque/Qwen3.5-27B-MP-GGUF

Mixed Precision GGUF layer quantization of Qwen3.5-27B by Qwen

Original model: https://huggingface.co/Qwen/Qwen3.5-27B

The hybrid quant employs different quantization levels on a per layer basis to enable both high performance and small file size at the same time. The quants employed are all K to avoid slow CPU or older GPU processing of IQ quants. An extended layer definition E quant Q4_E_H for the model is defined as follows (updated 8/25/2026):

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOSD"],
   [0 ,"Q5_K_6666"],[1 ,"Q5_K_6665"],[2 ,"Q4_K_6555"],[3 ,"Q4_K_6555"],
   [4 ,"Q4_K_5555"],[5 ,"Q4_K_5555"],[6 ,"Q4_K_4444"],[7 ,"Q4_K_5554"],
   [8 ,"Q4_K_4444"],[9 ,"Q4_K_4444"],[10,"Q4_K_4444"],[11,"Q4_K_5554"],
   [12,"Q4_K_4444"],[13,"Q4_K_4444"],[14,"Q4_K_4444"],[15,"Q4_K_5554"],
   [16,"Q4_K_4444"],[17,"Q4_K_4444"],[18,"Q4_K_4444"],[19,"Q4_K_5554"],
   [20,"Q4_K_4444"],[21,"Q4_K_4444"],[22,"Q4_K_4444"],[23,"Q4_K_5554"],
   [24,"Q4_K_4444"],[25,"Q4_K_4444"],[26,"Q4_K_4444"],[27,"Q4_K_5554"],
   [28,"Q4_K_5444"],[29,"Q4_K_5444"],[30,"Q4_K_5444"],[31,"Q4_K_5554"],
   [32,"Q4_K_5454"],[33,"Q4_K_5454"],[34,"Q4_K_5454"],[35,"Q4_K_5555"],
   [36,"Q4_K_5454"],[37,"Q4_K_5454"],[38,"Q4_K_5454"],[39,"Q4_K_5555"],
   [40,"Q4_K_5554"],[41,"Q4_K_5554"],[42,"Q4_K_5554"],[43,"Q4_K_5555"],
   [44,"Q4_K_5554"],[45,"Q4_K_5554"],[46,"Q4_K_5554"],[47,"Q4_K_5555"],
   [48,"Q4_K_5554"],[49,"Q4_K_5554"],[50,"Q4_K_5554"],[51,"Q4_K_5555"],
   [52,"Q4_K_5555"],[53,"Q4_K_5555"],[54,"Q4_K_5555"],[55,"Q4_K_6665"],
   [56,"Q4_K_5555"],[57,"Q4_K_5555"],[58,"Q4_K_5555"],[59,"Q5_K_5565"],
   [60,"Q5_K_6665"],[61,"Q5_K_6666"],[62,"Q5_K_6668"],[63,"Q6_K_8666"],
   [64,"Q4_K_6554"]
   ]'
   FLAGS="--token-embedding-type Q4_K --output-tensor-type Q6_K --layer-types-high"

The quant was sized at ~Q4_K_M bpw and optimized for strong performance over a small set of curated reasoning prompts.

Comparison:

Quant	size	PPL	Comment
Q4_K_M	16.8e9	8.3	Q4_K_M with default embedding and output
Q4_E_H	16.7e9	8.5	Hybrid quant with Q4_K embedding Q6_K output (includes 0.3B MTP layer)

Usage:

Qwen3.5-27B is a vision capable dense RL model. It can be used together with its multimedia projector layers to process images and text inputs and generate text outputs. The mmproj file is made available in this repository.

Due to the attention scheme used in the model, straightforward speculation approaches cannot be used. Later versions of llama.cpp as of 4/23/2026 attempt to circumvent this problem however efficiency will not be good so use of speculation prior to Qwen3.5 MTP patches is not recommended. As of llama.cpp b9180 MTP support for the model was added to upstream and may be experimented with by using the Q4_E_H_MTP quant.

On a 2x 4070 setup (1 RPC) approx performance is:

Q	QKV	NKV	gen tps
Q4_E_H	F16	100k +	25
Q4_E_H	Q8_0	160k +	23

The model appears to be trained to decide itself whether to do a think block or not. When it does a think block it can fall into very heavy overthinking on ambiguous/trick prompts. Over a small set of eval prompts the model did extremely well. To avoid the overthinking inject think start and think stop tokens first thing after assistant prompt:

THINK_START="<think>\n"
THINK_STOP="\n</think>\n\n"

If the model doesnt feel like doing thinking on a given prompt it will automatically do this. To force the model into a think block inject a bootstrap think start following the assistant prompt:

"<think>\n"

The model was found to be highly capable on reasoning tasks when skipping think block.

VISION:

The model was tested in vision mode on a couple pretty tough bird ID image and did extremely well handling the images with concise, informative, accurate, and efficient reasoning blocks.

CODE:

The model was tested across a small set of code gen prompts and found to be quite capable in its ability to generate working code, though it did fall into a rep loop with think enabled on one of the test prompts.

Benchmarks:

A full set of both math and vision benchmarks for the model will eventually be given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

Link	Type	Size/e9 B	Notes
Qwen3.5-27B.Q4_E_H.gguf	Q4_E_H	16.7e9 B	~Q4_K_M bpw, includes 0.3B MPT layer
Qwen3.5-27B.mmproj.gguf	F16	0.93e9 B	multimedia projector

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040