---
license: apache-2.0
library_name: gguf
pipeline_tag: image-text-to-text
base_model: Qwen/Qwen3-VL-4B-Instruct
base_model_relation: quantized
language:
  - en
tags:
  - abliteration
  - heretic
  - uncensored
  - qwen3-vl
  - qwen3
  - gguf
  - llama.cpp
  - text-generation
  - forensics
---

# Qwen3-VL-4B-Instruct Heretic (GGUF)

💬 **Community:** Join the [Abliterlitics Discord](https://discord.gg/AqmDnBjPvM) for discussion, model releases and support.


GGUF quantisations of the [Heretic abliteration](https://huggingface.co/DreamFast/Qwen3-VL-4b-Heretic) of [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct). The full forensic report comparing all candidate variants sits in the [main repo](https://huggingface.co/DreamFast/Qwen3-VL-4b-Heretic#forensic-report).

## Available formats

This model is published across three repos. Pick the one that matches your runtime.

| Repo | Best for | Contents |
|------|----------|----------|
| [Qwen3-VL-4b-Heretic](https://huggingface.co/DreamFast/Qwen3-VL-4b-Heretic) | transformers, vLLM, HF Hub | bf16 weights with config, vision encoder preserved |
| **[Qwen3-VL-4b-Heretic-GGUF](https://huggingface.co/DreamFast/Qwen3-VL-4b-Heretic-GGUF)** (this repo) | llama.cpp, Ollama, LM Studio, ComfyUI-GGUF | GGUF quants from Q3_K_M up to F16 (text path) |
| [Qwen3-VL-4b-Heretic-ComfyUI](https://huggingface.co/DreamFast/Qwen3-VL-4b-Heretic-ComfyUI) | ComfyUI text encoder | bf16, fp8, int8, nvfp4 and mxfp8 checkpoints |

## Why this variant?

Several Heretic trials were run against Qwen3-VL-4B and all of them reach 100% HarmBench ASR, up from 30.8% on the base. This build was picked because, with safety tied, it wins on the tie-breakers:

| | Base | **Heretic** |
|---|------|------|
| HarmBench ASR | 30.8% | **100%** |
| KL divergence (lower is better) |  | **0.0283** (lowest of the candidates) |
| GSM8K | 78.62% | 77.18% (**−1.83%**, smallest drop) |
| MMLU | 69.58% | 69.61% (+0.03%) |
| Tensors changed |  | 54 (pure rank-1) |

See the [full report](https://huggingface.co/DreamFast/Qwen3-VL-4b-Heretic#forensic-report) for the comparison.

![Benchmark deltas across the Heretic variants](graphs/aggressiveness_qwen3-vl-4b.svg)

## Files

| File | Quant | Size | Notes |
|------|-------|------|-------|
| `qwen3-vl-4b-heretic-f16.gguf` | F16 | 7.5 GB | Lossless reference |
| `qwen3-vl-4b-heretic-Q8_0.gguf` | Q8_0 | 4.0 GB | Excellent quality |
| `qwen3-vl-4b-heretic-Q6_K.gguf` | Q6_K | 3.1 GB | Very good quality |
| `qwen3-vl-4b-heretic-Q5_K_M.gguf` | Q5_K_M | 2.7 GB | Good quality |
| `qwen3-vl-4b-heretic-Q5_K_S.gguf` | Q5_K_S | 2.7 GB | Slightly smaller Q5 |
| `qwen3-vl-4b-heretic-Q4_K_M.gguf` | **Q4_K_M** | **2.4 GB** | **Recommended balance** |
| `qwen3-vl-4b-heretic-Q4_K_S.gguf` | Q4_K_S | 2.3 GB | Smaller Q4 variant |
| `qwen3-vl-4b-heretic-Q3_K_M.gguf` | Q3_K_M | 2.0 GB | Low VRAM only |

## Which quant should I pick?

- **Q4_K_M** (2.4 GB) suits almost everyone. Quality loss is barely noticeable and the file is roughly 30% of the bf16 size.
- **Q6_K / Q8_0** if you have the VRAM and want quality as close to bf16 as possible.
- **Q3_K_M** only if VRAM is tight, for instance on an 8 GB GPU. Expect a measurable quality drop.
- **F16** is the lossless reference, useful for re-quantisation or benchmarking.

> **Vision note:** these GGUF builds target the text path. For Qwen3-VL vision support, use the bf16 / transformers repo or a ComfyUI checkpoint.

## Usage

### llama.cpp

```bash
llama-server -m qwen3-vl-4b-heretic-Q4_K_M.gguf -c 8192 --port 8080
```

```bash
llama-cli -m qwen3-vl-4b-heretic-Q4_K_M.gguf -p "Explain why the sky is blue, step by step."
```

### ComfyUI (via ComfyUI-GGUF)

1. Install [ComfyUI-GGUF](https://github.com/city96/ComfyUI-GGUF).
2. Drop the `.gguf` file into `ComfyUI/models/unet/` (or wherever your GGUF loader expects it).
3. Use the **Unet Loader (GGUF)** node.

### Other runtimes

Compatible with anything that reads GGUF, including [Ollama](https://ollama.com), [LM Studio](https://lmstudio.ai), [KoboldCpp](https://github.com/LostRuins/koboldcpp) and [text-generation-webui](https://github.com/oobabooga/text-generation-webui).

## How these were made

Produced with [Heretic Docker](https://github.com/dreamfast/heretic-docker), which wraps:

- [Heretic](https://github.com/p-e-w/heretic) for the abliteration
- [llama.cpp](https://github.com/ggerganov/llama.cpp) for the GGUF conversion

## Limitations

- These GGUF builds are text-path oriented.
- Quantisation adds a small extra quality loss on top of the abliteration deltas already measured in the report.
- Inherits the abliteration caveats: TruthfulQA and GSM8K sit measurably lower than the base model.

## Disclaimer

This model has had its safety alignment removed. It complies with harmful requests, including content related to violence, illegal activities and other harmful behaviour. Use it responsibly and in line with the laws and regulations that apply to you. The authors do not condone or encourage using this model for harmful purposes.