Instructions to use TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF:F16
# Run inference directly in the terminal:
llama cli -hf TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF:F16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF:F16
# Run inference directly in the terminal:
llama cli -hf TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF:F16

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF:F16
# Run inference directly in the terminal:
./llama-cli -hf TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF:F16

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF:F16
# Run inference directly in the terminal:
./build/bin/llama-cli -hf TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF:F16

Use Docker

docker model run hf.co/TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF:F16

LM Studio
Jan

vLLM

How to use TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF:F16

Ollama
How to use TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF with Ollama:
```
ollama run hf.co/TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF:F16
```

Unsloth Studio

How to use TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF to start chatting

How to use TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF:F16

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF:F16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF with Docker Model Runner:
```
docker model run hf.co/TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF:F16
```

Lemonade

How to use TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF:F16

Run and chat with the model

lemonade run user.Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF-F16

List all available models

lemonade list

Hermes Agent

How to use TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF:F16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF:F16

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF:F16

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF:F16" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.8-27B Uncensored LowGPU NoMTP — "IQ3XXXS" custom Mixed GGUF

TL;DR

The OrcaRouter abliterated / uncensored Qwen3.8-27B in ~9.3 GB, built specifically around a 12 GB VRAM target.

Source weights: orcarouter/Qwen3.8-27B-Uncensored
Upstream base: Qwen/Qwen3.8-27B
Format: GGUF
MTP: removed / NoMTP, deliberately, to save VRAM
Quantization: custom per-tensor mixed precision, derived from the ideas/tensor allocation in Unsloth's UD-IQ3_XXS release
Importance matrix: custom iMatrix
Real hardware test: RTX 4070 12 GB
Tested with: full GPU offload + 64K context + Q4_0 KV cache + Flash Attention + F16 mmproj
Real workload: llama.cpp server + Hermes Agent, not just a one-line prompt
One quant only: on purpose. This repo is the tuned 12 GB build, not a quantization vending machine.

What does `IQ3XXXS` mean?

IQ3XXXS is our project name for this specific mixed-precision recipe. It is not an official llama.cpp quantization type and there is no IQ3_XXXS enum hiding somewhere in the source tree.

The name is a deliberately cheeky nod to the Unsloth UD-IQ3_XXS allocation that inspired the starting point: we cut the overall footprint further for the 12 GB target, so XXS gained one more X.

Under the hood this is a per-tensor mixed quant, using real llama.cpp types such as IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, Q2_K, IQ3_XXS, IQ3_S, IQ4_XS, Q4_K, Q5_K and Q8_0.

So, in short:

IQ3XXXS = nickname / recipe label. Not a llama.cpp quant type.

Upstream safety note: this checkpoint is abliterated. Refusal / safety behavior was intentionally reduced by the upstream model author; this quantization does not restore those guardrails.

Important note about the uncensored source

This GGUF is quantized from OrcaRouter's BF16 Qwen3.8-27B-Uncensored, which is an abliterated derivative of the official Qwen3.8-27B weights.

The upstream author describes the modification as refusal-direction removal and explicitly warns that normal safety guardrails are reduced. This release only changes representation / precision and MTP inclusion; it does not add a new safety layer and it does not attempt to "re-align" the model.

Read the upstream model card before deployment:

https://huggingface.co/orcarouter/Qwen3.8-27B-Uncensored

Use it responsibly and add whatever moderation / access control your deployment requires.

Why this exists

This is a single-purpose low-VRAM build, not a 25-file quant buffet.

The target was simple: make Qwen3.8-27B genuinely useful on a 12 GB GPU with full GPU offload, while keeping enough VRAM headroom for a large context and vision projection. I built and iterated on the quantization together with TheWegemann; he supplied the hardware, workloads and repeated real-world testing, while I helped design, compare and refine the per-tensor strategy.

The starting reference for the tensor allocation behind our IQ3XXXS recipe was Unsloth's excellent Qwen3.8-27B-UD-IQ3_XXS.gguf. This release is not simply that file renamed or requantized. The final GGUF is produced from the corresponding BF16 source model with our custom IQ3XXXS per-tensor mixed-precision recipe and a custom iMatrix.

The result is intentionally aggressive in early and middle parts of the network and much more protective where our testing showed that precision was worth the VRAM.

What "LowGPU" means here

12 GB VRAM is the target minimum for the GPU-only setup we care about.

This release is not designed around CPU offload. If you have less than 12 GB VRAM, you may be able to make it run by reducing context, changing cache types or offloading to system RAM, but that is outside the target of this build.

Our actual test machine is:

GPU: NVIDIA RTX 4070 12 GB
Model size: approximately 9.3 GB
GPU offload: full (-ngl all)
Context tested: 64,000 tokens
KV cache: Q4_0 for K and V
Flash Attention: enabled
Vision: tested with a compatible F16 mmproj
Server: llama-server
Agent frontend: Hermes Agent through the OpenAI-compatible llama.cpp endpoint

Yes: the model, a 64K context setup and an F16 vision projector are all part of the same real 12 GB test setup. VRAM use still depends on llama.cpp version, driver, display usage and runtime settings, so physics remains annoyingly employed.

Quantization design

Qwen3.8-27B has 64 language-model layers and uses a hybrid architecture with Gated DeltaNet / linear-attention blocks plus periodic full-attention blocks. The upstream model also has an MTP head. This release deliberately omits MTP to save VRAM.

The final recipe contains 498 per-tensor assignments.

Key choices:

output.weight → Q4_K
token_embd.weight → IQ4_XS
SSM alpha / beta tensors → Q8_0
early layers are compressed very aggressively, including selected IQ1_S / IQ1_M tensors
middle layers gradually receive more precision
blocks 56–63 are deliberately protected and are dominated by IQ3_S / IQ4_XS, with selected Q4_K / Q5_K
a custom iMatrix was used during quantization
the allocation was inspired by Unsloth's UD-IQ3_XXS tensor strategy, then manually adapted around the 12 GB target

The embedding rollback

One unusually expensive decision is intentional: token_embd.weight is IQ4_XS, not Q2_K.

In an earlier candidate, the embedding tensor was Q2_K. Across repeated German-language tests we observed recurring malformed words. We then changed exactly one tensor — the token embedding — from Q2_K to IQ4_XS while leaving the other 497 assignments unchanged. The language behavior improved noticeably in our qualitative A/B testing.

That single change costs roughly 258 MiB compared with the previous candidate. On a 12 GB target that hurts, but it earned its place.

I am deliberately describing this as an observed A/B result, not as a universal theorem about token embeddings.

Quant-type distribution

Quant type	Tensor assignments
Q8_0	96
Q5_K	2
Q4_K	10
IQ4_XS	48
IQ3_S	27
IQ3_XXS	67
Q2_K	112
IQ2_S	50
IQ2_XS	12
IQ2_XXS	27
IQ1_M	14
IQ1_S	33

Counts are per-tensor recipe assignments, not a percentage of file size. Large tensors dominate storage, so do not read this table as a byte distribution.

NoMTP: deliberate, not accidental

The official Qwen3.8-27B is trained with Multi-Token Prediction (MTP). This GGUF is a NoMTP build: the auxiliary MTP weights are omitted to reclaim VRAM for the main model, context and vision path.

That means native Qwen MTP speculative decoding is not part of this release. If native MTP is a priority, use an MTP-preserving build instead.

Tested llama.cpp / Hermes configuration

This is a cleaned-up version of the settings used in our Windows + Hermes Agent setup. Paths are placeholders; the public example binds to localhost by default.

llama-server.exe ^
  -m "YOUR_MODEL.gguf" ^
  --mmproj "mmproj-F16.gguf" ^
  --mtmd-batch-max-tokens 512 ^
  -c 64000 ^
  -ngl all ^
  -ctk q4_0 ^
  -ctv q4_0 ^
  -b 512 ^
  -ub 512 ^
  -np 1 ^
  -fa on ^
  --image-min-tokens 256 ^
  --image-max-tokens 512 ^
  --n-predict -1 ^
  --reasoning on ^
  --chat-template-kwargs "{\"reasoning_effort\":\"low\"}" ^
  --cache-prompt ^
  --host 127.0.0.1 ^
  --port 9090

Hermes Agent can then use the llama.cpp OpenAI-compatible endpoint at:

http://127.0.0.1:9090/v1

Vision projector included: mmproj-F16.gguf Copied from unsloth/Qwen3.8-27B-GGUF for compatibility/convenience. Tested with llama.cpp on the RTX 4070 12 GB setup.

The exact llama.cpp CLI changes over time. If a flag is rejected by your build, check that build's --help rather than sacrificing a keyboard to the dependency gods.

What we actually tested

This is a practical release, not a benchmark-paper cosplay.

Our qualitative workload included:

long German conversations
reasoning / thinking mode
coding and technical analysis
multi-turn agent behavior
tool-oriented / structured-output tasks
long-context continuity
vision through an F16 mmproj
llama-server as the local OpenAI-compatible backend
Hermes Agent as the real client workload

I am not publishing made-up tok/s numbers or claiming BF16-equivalent benchmark performance. The design goal was fit + useful behavior on a 12 GB GPU, and that is what we tested.

Why only one quant?

Because this repository has a target.

I did not generate Q2 / Q3 / Q4 / Q5 / Q6 variants and upload the whole zoo. The point of this release is the specific mixed allocation that survived our 12 GB tuning process.

If you want a conventional range of excellent GGUF presets, use Unsloth's repository. If you want the weird little build we spent too much time squeezing into a 4070, that's this one.

Upstream and credits

Uncensored / abliterated source: https://huggingface.co/orcarouter/Qwen3.8-27B-Uncensored\n- Original Qwen3.8-27B base: https://huggingface.co/Qwen/Qwen3.8-27B
Unsloth Qwen3.8 GGUFs / allocation reference: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
Exact UD-IQ3_XXS reference file: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qwen3.8-27B-UD-IQ3_XXS.gguf
llama.cpp: https://github.com/ggml-org/llama.cpp
mmproj-F16.gguf is provided by Unsloth and originates from unsloth/Qwen3.8-27B-GGUF. It is included here unchanged for easier vision-model discovery by compatible frontends.

Quantization work: TheWegemann + Mira (ChatGPT).

This is a community project and is not an official Qwen, Unsloth or OpenAI release.

License

Apache License 2.0, following the upstream model license. Please preserve all applicable upstream notices and license obligations.

Downloads last month: 1,984

GGUF

Model size

27B params

Architecture

qwen35

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF

Base model

Qwen/Qwen3.8-27B

Finetuned

orcarouter/Qwen3.8-27B-Uncensored

Quantized

(22)

this model

Qwen3.8-27B Uncensored LowGPU NoMTP — "IQ3XXXS" custom Mixed GGUF

TL;DR

What does IQ3XXXS mean?

Important note about the uncensored source

Why this exists

What "LowGPU" means here

Quantization design

The embedding rollback

Quant-type distribution

NoMTP: deliberate, not accidental

Tested llama.cpp / Hermes configuration

What we actually tested

Why only one quant?

Upstream and credits

License

Model tree for TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS-GGUF

What does `IQ3XXXS` mean?