Instructions to use Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF:IQ4_XS
# Run inference directly in the terminal:
llama cli -hf Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF:IQ4_XS

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF:IQ4_XS
# Run inference directly in the terminal:
llama cli -hf Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF:IQ4_XS

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF:IQ4_XS
# Run inference directly in the terminal:
./llama-cli -hf Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF:IQ4_XS

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF:IQ4_XS
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF:IQ4_XS

Use Docker

docker model run hf.co/Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF:IQ4_XS

LM Studio
Jan
Ollama
How to use Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF with Ollama:
```
ollama run hf.co/Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF:IQ4_XS
```

Unsloth Studio

How to use Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF to start chatting

How to use Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF:IQ4_XS

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF:IQ4_XS"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF with Docker Model Runner:
```
docker model run hf.co/Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF:IQ4_XS
```

Lemonade

How to use Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF:IQ4_XS

Run and chat with the model

lemonade run user.Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF-IQ4_XS

List all available models

lemonade list

Hermes Agent

How to use Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF:IQ4_XS

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF:IQ4_XS

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF:IQ4_XS

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF:IQ4_XS" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen 3.8 27B Heretic‑Ara IQ4_XS 量化模型（适配 16GB 显存）

本模型基于 Qwen 3.8 27B Heretic‑Ara BF16 进行 IQ4_XS 量化（4‑bit），文件体积为 12.7-13.3 GiB，专为 16GB 显存的显卡优化

8/22 更新修复思考的问题，模型性能没有变化大概还需要1天我会更新MTP的版本

8/22 Updated and fixed the issues with thinking; model performance remains unchanged It will probably take about another day for me to update the MTP version.

8/23 更新，对模型本身的性能进行一定优化，体积略微变大，无MTP版本13Gib，MTP版本13.3Gib 8/23 Update: The model's performance has been slightly optimized, resulting in a slight increase in size The non-MTP version is 13GiB, while the MTP version is 13.3GiB

与同体积的 Heretic‑Ara‑Q3_K_M（12.4 GiB）量化方案进行了全面对比

本模型使用Heretic Arbitrary-Rank Ablation做到的无审查

📊 量化质量对比

评估指标	Heretic-Ara BF16 (base)	IQ4_XS-3.0	IQ4_XS-2.0	Heretic-Ara-Q3_K_M
文件大小	50.1 GiB	13 GiB (带MTP 13.3 GiB)	12.7 GiB	12.4 GiB
量化精度	BF16	IQ4_XS (4‑bit)	IQ4_XS (4‑bit)	Q3_K_M (约 3‑bit)
模型困惑度 (Mean PPL)	7.008212 ± 0.045362	7.046980 ± 0.045498	7.102940 ± 0.046017	7.403971 ± 0.048924
与基座模型 PPL 相关性	100%	99.34%	99.26%	98.31%
平均 KL 散度 (Mean KLD)	0	0.027832 ± 0.000324	0.033398 ± 0.000308	0.076034 ± 0.000554
最大 KL 散度 (Max KLD)	0	18.317436	15.094215	17.866985
99.9% KL 分位数	0	1.162850	1.130034	2.448278
Top‑1 一致率 (Same top p)	100%	92.867% ± 0.067%	91.619% ± 0.072%	88.152% ± 0.084%
平均概率变化 (Mean Δp)	0%	-0.243% ± 0.012%	-0.306% ± 0.013%	-0.490% ± 0.020%
RMS 概率变化 (RMS Δp)	0%	4.538% ± 0.045%	4.952% ± 0.041%	7.560% ± 0.054%

注：基座（BF16）的 KL 散度、Δp 等指标均为 0（自身对比），一致率为 100%。

在不启用 MTP 的情况下，IQ4_XS 模型在 16 GiB 无显存占用（不作为 Windows 显示显卡）下可支持约 110k 上下文

开启 MTP 后约为 80k

Qwen 3.8 27B Heretic‑Ara IQ4_XS Quantized Model (Optimized for 16 GB VRAM) This model is quantized from Qwen 3.8 27B Heretic‑Ara BF16 using the IQ4_XS scheme (4‑bit), with a file size of 12.8 GiB, specifically designed for graphics cards with 16 GB of VRAM.

It has been comprehensively compared against the similarly sized Heretic‑Ara‑Q3_K_M (12.4 GiB) quantization variant.

This model achieves uncensored behavior through Heretic's Arbitrary-Rank Ablation.

📊 Quantization Quality Comparison

Evaluation Metric	Heretic-Ara BF16 (base)	IQ4_XS-3.0	IQ4_XS-2.0	Heretic-Ara-Q3_K_M (comparison)
File Size	50.1 GiB	13 GiB (with MTP 13.3 GiB)	12.7 GiB	12.4 GiB
Quantization Precision	BF16	IQ4_XS (4‑bit)	IQ4_XS (4‑bit)	Q3_K_M (~3‑bit)
Model Perplexity (Mean PPL)	7.008212 ± 0.045362	7.046980 ± 0.045498	7.102940 ± 0.046017	7.403971 ± 0.048924
PPL Correlation with Base Model	100%	99.34%	99.26%	98.31%
Mean KL Divergence (Mean KLD)	0	0.027832 ± 0.000324	0.033398 ± 0.000308	0.076034 ± 0.000554
Max KL Divergence (Max KLD)	0	18.317436	15.094215	17.866985
99.9% KL Quantile	0	1.162850	1.130034	2.448278
Top‑1 Agreement Rate (Same top p)	100%	92.867% ± 0.067%	91.619% ± 0.072%	88.152% ± 0.084%
Mean Probability Change (Mean Δp)	0%	-0.243% ± 0.012%	-0.306% ± 0.013%	-0.490% ± 0.020%
RMS Probability Change (RMS Δp)	0%	4.538% ± 0.045%	4.952% ± 0.041%	7.560% ± 0.054%

Note: For the base model (BF16), KL divergence, Δp, etc. are all 0 (self‑comparison), and the agreement rate is 100%.

With MTP (Multi‑Token Prediction) disabled, the IQ4_XS model supports approximately 110k context length on a 16 GiB GPU with no VRAM reserved for display (i.e., not used as the primary display adapter). With MTP enabled, the context length is approximately 80k.

Downloads last month: 4,459

GGUF

Model size

27B params

Architecture

qwen35

Hardware compatibility

4-bit

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF

Base model

Qwen/Qwen3.8-27B

Quantized

(776)

this model