---
license: apache-2.0
tags:
- qwen
- qwen3.5
- multimodal
- vision
- gguf
- ssm
- quantization
base_model: Qwen/Qwen3.8-27B
---

# Qwen3.8-27B — GGUF quantizations (XYZ)

GGUF quantizations of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B)
(multimodal, hybrid SSM + attention, 64 layers + 1 MTP layer), converted with
upstream llama.cpp. All files include the built-in **MTP block**
(`nextn_predict_layers=1`), usable for speculative decoding.

## v2 — main series

v2 files carry the `-v2` suffix, except `Q3` and `Q3.5` which are also v2 but
kept without the suffix.

| File | Size (GiB) | BPW |
|---|---|---|
| Qwen3.8-27B-Q3-XYZ.gguf | 10.39 | 3.26 |
| Qwen3.8-27B-Q3.5-XYZ.gguf | 11.95 | 3.75 |
| Qwen3.8-27B-Q4-XYZ-v2.gguf | 14.03 | 4.41 |
| Qwen3.8-27B-Q4.5-XYZ-v2.gguf | 14.62 | 4.59 |
| Qwen3.8-27B-Q5-XYZ-v2.gguf | 16.61 | 5.22 |
| Qwen3.8-27B-Q5.5-XYZ-v2.gguf | 19.51 | 6.13 |
| Qwen3.8-27B-Q6-XYZ-v2.gguf | 20.94 | 6.58 |
| Qwen3.8-27B-Q7-XYZ-v2.gguf | 24.18 | 7.60 |
| Qwen3.8-27B-Q8-XYZ-v2.gguf | 26.71 | 8.39 |
| Qwen3.8-27B-ULTRA-XYZ-v2.gguf | 15.76 | 4.93 |

## v2 — low-bit / emergency files

| File | Size (GiB) | BPW |
|---|---|---|
| Qwen3.8-27B-Q1Z-XYZ-v2.gguf | 7.12 | 2.24 |
| Qwen3.8-27B-Q2-XYZ-v2.gguf | 8.15 | 2.56 |
| Qwen3.8-27B-Q1Q-XYZ-v2.gguf | 6.15 | 1.98 |

## v1 — previous generation (kept, still usable)

v1 files have **no version tag** in the name: `Q4`, `Q4.5`, `Q5`, `Q5.5`,
`Q6` without `-v2` are all v1.

| File | Size (GiB) | BPW (approx) |
|---|---|---|
| Qwen3.8-27B-Q4-XYZ.gguf (v1) | 14.15 | ~4.4 |
| Qwen3.8-27B-Q4.5-XYZ.gguf (v1) | 14.45 | ~4.5 |
| Qwen3.8-27B-Q5-XYZ.gguf (v1) | 16.21 | ~5.1 |
| Qwen3.8-27B-Q5.5-XYZ.gguf (v1) | 18.19 | ~5.7 |
| Qwen3.8-27B-Q6-XYZ.gguf (v1) | 21.00 | ~6.6 |

> Naming: `-v2` suffix = v2 recipe. No suffix on `Q4`–`Q6` = v1 (older
> recipe). `Q3`/`Q3.5` with no suffix are v2 (they predate the suffix
> convention). `Q7` is the former `Q6` full-BF16-attention build (7.60 BPW).

## Usage

Serve with llama-server (OpenAI-compatible API):

```bash
llama-server -m Qwen3.8-27B-Q5-XYZ-v2.gguf --port 1234
```

With the vision projector (multimodal):

```bash
llama-server -m Qwen3.8-27B-Q5-XYZ-v2.gguf --mmproj mmproj-Qwen3.8-27B-f32.gguf --port 1234
```

With MTP speculative decoding:

```bash
llama-server -m Qwen3.8-27B-Q5-XYZ-v2.gguf --spec-type mtp --port 1234
```

**Recommended sampling**: `--temp 0.6` for balanced, coherent output.

### Extreme quantizations (Q1/Q2) — sampling to reduce hallucinations

The low-bit files (Q1Q/Q2) are more prone to hallucination — use this tuned
sampling with llama-server for much more stable output:

```bash
llama-server -m Qwen3.8-27B-Q2-XYZ-v2.gguf --host 0.0.0.0 --port 1234 \
  --temp 0.6 \
  --top-p 0.9 \
  --top-k 40 \
  --repeat-penalty 1.10 \
  --repeat-last-n 512 \
  --dry-multiplier 0.8 \
  --dry-base 1.75 \
  --dry-allowed-length 2
```

The DRY sampler (repetition-aware) is the key flag for low-bit models —
`--dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 2` keeps generation
on-track without over-penalizing. Works for Q1Q-XYZ-v2 too.

## Notes

- Model is Apache-2.0, architecture `Qwen3_5ForConditionalGeneration` (hybrid
  SSM + attention, full attention every 4th layer), vocab 248,320,
  `tie_word_embeddings=false`.
- Quantized with upstream llama.cpp, 4 threads, `--tensor-type` per-tensor
  recipe with imatrix. Key tensors (attention K/V, SSM params, MTP head) are
  kept in **BF16** — full precision where it matters.
- The Q1/Q2 files are experimental: below ~2.5 BPW a reasoning model can get
  stuck in the thinking loop on some prompts.
- Sizes are exact file sizes on disk (GiB, base 1024 — matches the sizes shown
  on the HF file browser).