Instructions to use barozp/Qwen3.8-27B-Opus-Distill-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use barozp/Qwen3.8-27B-Opus-Distill-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf barozp/Qwen3.8-27B-Opus-Distill-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf barozp/Qwen3.8-27B-Opus-Distill-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf barozp/Qwen3.8-27B-Opus-Distill-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf barozp/Qwen3.8-27B-Opus-Distill-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf barozp/Qwen3.8-27B-Opus-Distill-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf barozp/Qwen3.8-27B-Opus-Distill-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf barozp/Qwen3.8-27B-Opus-Distill-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf barozp/Qwen3.8-27B-Opus-Distill-GGUF:Q4_K_M

Use Docker

docker model run hf.co/barozp/Qwen3.8-27B-Opus-Distill-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use barozp/Qwen3.8-27B-Opus-Distill-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "barozp/Qwen3.8-27B-Opus-Distill-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "barozp/Qwen3.8-27B-Opus-Distill-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/barozp/Qwen3.8-27B-Opus-Distill-GGUF:Q4_K_M

Ollama
How to use barozp/Qwen3.8-27B-Opus-Distill-GGUF with Ollama:
```
ollama run hf.co/barozp/Qwen3.8-27B-Opus-Distill-GGUF:Q4_K_M
```

Unsloth Studio

How to use barozp/Qwen3.8-27B-Opus-Distill-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for barozp/Qwen3.8-27B-Opus-Distill-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for barozp/Qwen3.8-27B-Opus-Distill-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for barozp/Qwen3.8-27B-Opus-Distill-GGUF to start chatting

How to use barozp/Qwen3.8-27B-Opus-Distill-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf barozp/Qwen3.8-27B-Opus-Distill-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "barozp/Qwen3.8-27B-Opus-Distill-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use barozp/Qwen3.8-27B-Opus-Distill-GGUF with Docker Model Runner:
```
docker model run hf.co/barozp/Qwen3.8-27B-Opus-Distill-GGUF:Q4_K_M
```

Lemonade

How to use barozp/Qwen3.8-27B-Opus-Distill-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull barozp/Qwen3.8-27B-Opus-Distill-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.8-27B-Opus-Distill-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use barozp/Qwen3.8-27B-Opus-Distill-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf barozp/Qwen3.8-27B-Opus-Distill-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default barozp/Qwen3.8-27B-Opus-Distill-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use barozp/Qwen3.8-27B-Opus-Distill-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf barozp/Qwen3.8-27B-Opus-Distill-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "barozp/Qwen3.8-27B-Opus-Distill-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.8-27B-Opus-Distill-GGUF / README.md

barozp

Upload README.md with huggingface_hub

c594717 verified 18 minutes ago

preview code

Raw

History Blame Contribute Delete

8.79 kB

metadata

license: apache-2.0
base_model: barozp/Qwen3.8-27B-Opus-Distill
base_model_relation: quantized
library_name: llama.cpp
pipeline_tag: image-text-to-text
tags:
  - gguf
  - llama.cpp
  - qwen
  - reasoning
  - opus-distill
  - vision
  - mtp
  - imatrix
  - quantized

Qwen3.8-27B-Opus-Distill-GGUF

GGUF quantizations of barozp/Qwen3.8-27B-Opus-Distill — a Qwen3.8-27B fine-tuned with LoRA on Claude Opus reasoning traces (merged), with the native vision tower and native MTP head carried over untouched.

Highlights

Reasoning-distilled, not just quantized. The LoRA was trained on 14,250 Opus chain-of-thought traces and merged into the base weights. Quantization only converts the weights — the reasoning gains travel with them unchanged.
Full multimodal. Native vision tower ships as a separate mmproj file (~0.9 GB). Text-only users can ignore it entirely.
Native MTP for self-speculative decoding. The model was released with its MTP head trained in — unlike grafted MTP setups, no approximation involved. Free speedups on compute-bound hardware.
imatrix-calibrated. All quants below Q3_K_M use an importance matrix built from the model's own reasoning-distillation data (see Imatrix).

Known issues

Reasoning loop under stacked output-format constraints. Reported by zxbc2023 (full writeup, discussion #1). Combining "no prose" with a second output-format constraint (e.g. "no markdown" or "no comments") can send this model into a non-converging self-verification reasoning loop -- it burns the entire token budget with zero visible output. Fully deterministic and reproducible at temp=0. Root cause: traced to part of the training data being sourced from reconstructed (not verbatim) Opus reasoning traces, not a capability gap.

Fixed in barozp/Qwen3.8-27B-Opus-Distill-v2-clean -- retrained on a rebuilt dataset where every row is traced to a verified genuine source. If you're hitting this, switch to v2-clean.

Workaround if staying on this version: avoid combining "no prose" with another format constraint, or raise the generation token budget to >=4096 for constrained code-gen tasks.

Quality benchmarks (of the source safetensors model)

Measured with lm-evaluation-harness: 0-shot, loglikelihood (multiple-choice), chat template OFF, QUICK mode (--limit 500). Base and distill ran with the identical harness, so the Δ column is the meaningful signal.

Task	Metric	Base	Distill	Δ
wikitext	word perplexity ↓	8.434	8.344	−0.09
mmlu	acc	0.849	0.849	−0.001
hellaswag	acc_norm	0.742	0.740	−0.002
arc_challenge	acc_norm	0.588	0.630	+0.042
gpqa_diamond	acc_norm	0.232	0.495	+0.263

Reading the table:

Reasoning improved (ARC +4.2pt, GPQA +26pt), knowledge stayed flat (MMLU −0.001) and language modeling stayed flat (wikitext −0.09 ppl).
GPQA caveat: measured with thinking disabled (loglikelihood) — the base scores near random (25%) because it gets no chance to deliberate. The +26pt Δ is a valid same-protocol comparison, but do not compare 0.495 to Qwen's published 89.2 (measured with thinking ON, different harness).
ARC-Challenge is saturated for modern models; treat it as continuity with the Qwen3.6 release — GPQA is the stronger reasoning signal here.

Speed (MTP self-speculative decoding)

Not yet benchmarked for this exact model. On the Qwen3.6 sibling (same MTP mechanism, grafted there), measured with llama.cpp: +39% tok/s full offload, +67% partial offload with spec-decode ON. Native MTP (this model) is trained in and typically does at least as well. Guidance:

Compute-bound (full offload, strong GPU) → enable --spec-type draft-mtp.
Memory-bandwidth-bound (partial offload) → keep spec off.

Available quantizations

File	Size	Bits/w	Use case
`Qwen3.8-27B-Opus-Distill-BF16.gguf`	54.7 GB	16.0	reference / re-quantization source
`Qwen3.8-27B-Opus-Distill-Q8_0.gguf`	29.0 GB	8.5	near-lossless
`Qwen3.8-27B-Opus-Distill-Q6_K.gguf`	22.4 GB	6.6	high quality
`Qwen3.8-27B-Opus-Distill-Q5_K_M.gguf`	19.5 GB	5.7	quality / balanced
`Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf`	16.8 GB	4.9	recommended all-rounder
`Qwen3.8-27B-Opus-Distill-Q3_K_M.gguf`	13.5 GB	4.0	tight VRAM
`Qwen3.8-27B-Opus-Distill-IQ3_XXS.gguf`	11.4 GB	3.3	low-bit, imatrix
`Qwen3.8-27B-Opus-Distill-IQ2_XXS.gguf`	8.7 GB	2.5	very low-bit, imatrix
`Qwen3.8-27B-Opus-Distill-IQ1_M.gguf`	7.9 GB	2.3	extreme low-bit, imatrix

K-quants (Q8_0–Q3_K_M) are plain llama-quantize passes, no imatrix needed. IQ-quants (IQ3_XXS and below) require an importance matrix to run at all in current llama.cpp and are built from the one in this repo (see below).

Which one to pick:

Best quality with headroom → Q6_K or Q8_0
Best quality/size balance → Q4_K_M (default recommendation)
24 GB card → Q4_K_M; 16 GB card → Q3_K_M (partial offload)
Below that → IQ quants, accept the quality hit

Imatrix

imatrix.dat in this repo (512 samples from barozp/opus-reasoning-distill-train, context 512) was used to build the IQ quants above. It applies to any GGUF with this same architecture — including the base Qwen/Qwen3.8-27B — so it can be reused for re-quantization without recomputing it:

llama-quantize --imatrix imatrix.dat model-BF16.gguf model-IQ4_XS.gguf IQ4_XS

Note on IQ1_M: the MTP head (blk.64, the nextn.* decoder layer) is never exercised by a normal forward pass, so the imatrix has no data for it. llama-quantize pins that block to q4_K instead of failing, which is why IQ1_M lands at ~2.3 bits/weight (7.9 GB) rather than the ~1.8 a "pure" IQ1_M would suggest — the MTP head alone accounts for the difference, the rest of the model is quantized normally.

Vision (mmproj)

The vision tower is in Qwen3.8-27B-Opus-Distill-mmproj-f16.gguf (~0.9 GB) in this repo. Load it alongside any quant for image/video input:

llama-server -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf --mmproj Qwen3.8-27B-Opus-Distill-mmproj-f16.gguf

Text-only usage does not need mmproj and runs fine without it.

Quick start

# build llama.cpp with CUDA, then:

# text-only chat
llama-cli -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf -no-cnv

# multimodal server
llama-server -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf --mmproj Qwen3.8-27B-Opus-Distill-mmproj-f16.gguf

# with self-speculative decoding (compute-bound hardware)
llama-cli -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf -no-cnv --spec-type draft-mtp -fa on

Training details (source safetensors model)

Base: Qwen/Qwen3.8-27B — dense 27B, hybrid Gated-DeltaNet / full-attention, 64 layers
Method: LoRA r=64, alpha=64, dropout 0.05, merged into base weights
LoRA targets: attention q/k/v/o_proj on the 16 full-attention layers; FFN gate/up/down_proj on all 64 layers (Gated-DeltaNet projections untouched)
Data: barozp/opus-reasoning-distill-train (14,250) + -validation (750, held out)
Run: 1 epoch (891 steps), lr 1e-4 cosine + 3% warmup, effective batch 16, MAX_SEQ 4096, bf16, ~5h52m on A100 80GB
Final validation loss: 0.4647
Vision + MTP: carried over byte-for-byte from the base checkpoint — never trained

Notes

Thinking mode is on by default (same as the base model). The GGUF embeds the chat template; how thinking is toggled depends on the llama.cpp version / frontend (e.g., LM Studio exposes the setting in its UI).
Conversion: llama.cpp convert_hf_to_gguf.py from the corrected multimodal config (nested text_config + vision_config).
No chaining: every quant was produced directly from the BF16 GGUF, so errors do not accumulate across the ladder.

Source chain

Qwen/Qwen3.8-27B (base) → barozp/Qwen3.8-27B-Opus-Distill (LoRA finetune, safetensors) → this repo (GGUF quantizations)

See Known Issues above -- if you're hitting the reasoning-loop bug, barozp/Qwen3.8-27B-Opus-Distill-v2-clean-GGUF fixes it.