README.md

---
license: apache-2.0
base_model:
- Qwen/Qwen3.6-27B
tags:
- Qwen3.6-27B
- abliterated
- Uncensored
- Smaller
- TurboQuant
- gguf
- IQ4_XS
pipeline_tag: image-text-to-text
---

# Qwen3.6-27B-uncensored-abliterated-i1-IQ4_XS-GGUF(Smaller)

This model is equivalent to https://huggingface.co/lemonyins/Qwen3.6-27B-abliterated-i1-IQ4_XS-GGUF-Smaller

## Innovation

This model refers to the fully optimized [Qwen3.6-27B-i1-IQ4_XS](https://huggingface.co/cHunter789/Qwen3.6-27B-i1-IQ4_XS-GGUF), which has restored the `attn_qkv` layer to pure `IQ4_XS`. Furthermore, we introduce a novel hybrid precision quantization strategy: **the FFN layer uses `IQ3_S`**, which achieves significantly smaller file sizes while maintaining core inference capabilities through support from TurboQuant KV caching. Additionally, we use an Huihui abliterated version of the base model for quantization, making it convenient for users to conduct in-depth research.

## Motivation

The original `llama.cpp` quantization heuristics upgrade `attn_qkv` layers to `q5_K` under certain conditions (e.g., `n_gqa >= 4`), causing noticeable file bloat. cHunter789's fix restores `attn_qkv` to pure `IQ4_XS`, saving ~375 MiB.

Taking this further: **FFN layers (ffn_down, ffn_up, ffn_gate) account for ~2/3 of total model parameters**, yet they have higher redundancy than attention layers. Downgrading them from `IQ4_XS` to `IQ3_S` is a natural next step — it yields substantial size reduction with minimal quality impact, especially when attention layers (which dominate inference quality) remain at `IQ4_XS`.

## Methodology

1. **Base model**: [Huihui-Qwen3.6-27B-abliterated](https://huggingface.co/mradermacher/Huihui-Qwen3.6-27B-abliterated-i1-GGUF) by mradermacher (abliterated, F16) — an uncensored version optimized for inference
2. **Quantization tool**: llama.cpp with TurboQuant support, built from [TheTom/llama-cpp-turboquant](https://github.com/TheTom/llama-cpp-turboquant/releases)
3. **Quantization types**:
   - `attn_qkv`, `attn_k`, `attn_v`, `attn_output`, `output`: `IQ4_XS`
   - `ffn_down`, `ffn_up`, `ffn_gate`: `IQ3_S`
   - Other layers: default `IQ4_XS`
4. **Importance matrix (imatrix)**: sourced from mradermacher's [Qwen3.6-27B-i1-GGUF](https://huggingface.co/mradermacher/Qwen3.6-27B-i1-GGUF)

### Quantization Commands

**Option A — Pure IQ4_XS baseline:**

```bash
llama-quantize.exe \
  --imatrix Huihui-Qwen3.6-27B-abliterated.imatrix.gguf \
  Huihui-Qwen3.6-27B-abliterated-BF16.gguf \
  Huihui-Qwen3.6-27B-abliterated-i1-IQ4_XS.gguf \
  IQ4_XS
```

**Option B — Recommended: IQ4_XS-FFN-IQ3_S (smaller + longer context):**

```bash
llama-quantize.exe ^
  --imatrix Huihui-Qwen3.6-27B-abliterated.imatrix.gguf ^
  Huihui-Qwen3.6-27B-abliterated-BF16.gguf ^
  Huihui-Qwen3.6-27B-abliterated-i1-IQ4_XS-FFN-IQ3_S.gguf ^
  IQ4_XS ^
  --tensor-type "blk.*.ffn_down" iq3_s ^
  --tensor-type "blk.*.ffn_up" iq3_s ^
  --tensor-type "blk.*.ffn_gate" iq3_s
```

> **Note on TurboQuant**: This model is recommended to be used with llama.cpp (https://github.com/TheTom/llama-cpp-turboquant) that supports TurboQuant KV caching. TurboQuant allows the KV cache to use a separate, more compact quantization format (turbo4 / turbo3), dramatically reducing memory usage even when the model weights themselves remain at IQ4_XS. Of course, it is also possible to use vllm or other inference frameworks that support TurboQuant technology, but the author used llama.cpp for the test.

## Memory Performance (with TurboQuant KV Cache)

| Version | Context | KV Cache | VRAM Usage |
| :--- | :--- | :--- | :--- |
| `IQ4_XS` (baseline) | 60K | turbo4 | **15.3 GB** |
| `IQ4_XS` (baseline) | 80K | turbo3 | **15.3 GB** |
| **`IQ4_XS-FFN-IQ3_S`** | **160K** | turbo4 | **15.4 GB** |
| **`IQ4_XS-FFN-IQ3_S`** | **200K** | turbo3 | **15.3 GB** |

### Key Takeaways

- **IQ4_XS baseline** reaches 60-80K context before VRAM saturation
- **IQ4_XS-FFN-IQ3_S** extends context window to **160-200K** — more than **2x** — with the same VRAM budget, thanks to the smaller FFN weight footprint enabling more KV cache capacity
- Quality-critical attention layers remain at `IQ4_XS`, so the model's reasoning and instruction-following capabilities are largely preserved

### Inference Speed

Tested on **NVIDIA RTX 4060 Ti 16GB**:

| Scenario | Speed |
| :--- | :--- |
| Text-only inference | **18-20 tokens/s** |

### Vision Support (Optional)

This model supports vision input when paired with the Vision Modality Projector (`mmproj-BF16.gguf`). If you load the vision module, the available context window will be reduced by approximately **10K** to accommodate the vision encoder's memory footprint.

**Example with vision support:**

```bash
llama-server.exe ^
  -m Huihui-Qwen3.6-27B-abliterated-i1-IQ4_XS-FFN-IQ3_S.gguf ^
  -mmproj mmproj-BF16.gguf ^
  -c 153840 ^
  -ngl 99 ^
  --flash-attn on ^
  --cache-type-k turbo4 ^
  --cache-type-v turbo4 ^
  --host 0.0.0.0
```

## 🧠 Intelligence (Perplexity) Comparison

Test using Chinese novel: 
| Model Version | Perplexity (PPL) | Quality Drop |
| :--- | :--- | :--- |
| Q4_K_M | 13.1909 +/- 0.06037 | Baseline |
| IQ4_XS | 13.2138 +/- 0.06054 | 0.17% |
| IQ4_XS-FFN-IQ3_S | 13.6056 +/- 0.06159 | 3.14% |

Test with code: 
| Model Version | Perplexity (PPL) | Quality Drop |
| :--- | :--- | :--- |
| IQ4_XS | 1.2217 +/- 0.00156 | Baseline |
| IQ4_XS-FFN-IQ3_S | 1.2324 +/- 0.00158 | 0.87% |


## Caveats

- **TurboQuant is mandatory**: This model relies on TurboQuant KV cache for the listed memory figures. Standard llama.cpp builds without TurboQuant will consume significantly more VRAM.
- **FFN layers at IQ3_S**: While the quality impact is expected to be minimal for most tasks, some degradation may be observable in tasks that heavily depend on FFN-related capabilities (e.g., certain factual recall scenarios). The attention layers remain at IQ4_XS to preserve core inference quality.
- **Verification pending**: Perplexity benchmarks with the standard IQ4_XS baseline are planned to quantify the quality difference precisely.

## How to Use

You need a **TurboQuant-enabled** llama.cpp build from [TheTom/llama-cpp-turboquant](https://github.com/TheTom/llama-cpp-turboquant).

> **Note**: This model is quantized from an **abliterated (uncensored)** base model. The base model removes content restrictions for research and development purposes.

**Recommended: IQ4_XS-FFN-IQ3_S version — 160K context on ~15 GB VRAM**

```bash
llama-server.exe ^
  -m Huihui-Qwen3.6-27B-abliterated-i1-IQ4_XS-FFN-IQ3_S.gguf ^
  -c 163840 ^
  -ngl 99 ^
  --flash-attn on ^
  --cache-type-k turbo4 ^
  --cache-type-v turbo4 ^
  --host 0.0.0.0
```

**Alternative: Pure IQ4_XS version — shorter context, slightly larger file**

```bash
llama-server.exe ^
  -m Huihui-Qwen3.6-27B-abliterated-i1-IQ4_XS.gguf ^
  -c 65536 ^
  -ngl 99 ^
  --flash-attn on ^
  --cache-type-k turbo4 ^
  --cache-type-v turbo4 ^
  --host 0.0.0.0
```

## Acknowledgments

- **[cHunter789](https://huggingface.co/cHunter789)** — for the `attn_qkv → IQ4_XS` fix and the original fully optimized IQ4_XS GGUF
- **[mradermacher](https://huggingface.co/mradermacher)** — for the base abliterated model and imatrix
- **[TheTom](https://github.com/TheTom)** — for [llama-cpp-turboquant](https://github.com/TheTom/llama-cpp-turboquant), the TurboQuant KV cache implementation
- **[llama.cpp](https://github.com/ggml-org/llama.cpp)** — the ggml/llama.cpp team for the base quantization framework