Instructions to use YTan2000/Qwen3.8-27B-TQ3_4S with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use YTan2000/Qwen3.8-27B-TQ3_4S with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf YTan2000/Qwen3.8-27B-TQ3_4S:BF16
# Run inference directly in the terminal:
llama cli -hf YTan2000/Qwen3.8-27B-TQ3_4S:BF16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf YTan2000/Qwen3.8-27B-TQ3_4S:BF16
# Run inference directly in the terminal:
llama cli -hf YTan2000/Qwen3.8-27B-TQ3_4S:BF16

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf YTan2000/Qwen3.8-27B-TQ3_4S:BF16
# Run inference directly in the terminal:
./llama-cli -hf YTan2000/Qwen3.8-27B-TQ3_4S:BF16

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf YTan2000/Qwen3.8-27B-TQ3_4S:BF16
# Run inference directly in the terminal:
./build/bin/llama-cli -hf YTan2000/Qwen3.8-27B-TQ3_4S:BF16

Use Docker

docker model run hf.co/YTan2000/Qwen3.8-27B-TQ3_4S:BF16

LM Studio
Jan

vLLM

How to use YTan2000/Qwen3.8-27B-TQ3_4S with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "YTan2000/Qwen3.8-27B-TQ3_4S"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "YTan2000/Qwen3.8-27B-TQ3_4S",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/YTan2000/Qwen3.8-27B-TQ3_4S:BF16

Ollama
How to use YTan2000/Qwen3.8-27B-TQ3_4S with Ollama:
```
ollama run hf.co/YTan2000/Qwen3.8-27B-TQ3_4S:BF16
```

Unsloth Studio

How to use YTan2000/Qwen3.8-27B-TQ3_4S with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for YTan2000/Qwen3.8-27B-TQ3_4S to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for YTan2000/Qwen3.8-27B-TQ3_4S to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for YTan2000/Qwen3.8-27B-TQ3_4S to start chatting

How to use YTan2000/Qwen3.8-27B-TQ3_4S with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf YTan2000/Qwen3.8-27B-TQ3_4S:BF16

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "YTan2000/Qwen3.8-27B-TQ3_4S:BF16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

OpenClaw new

How to use YTan2000/Qwen3.8-27B-TQ3_4S with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf YTan2000/Qwen3.8-27B-TQ3_4S:BF16

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "YTan2000/Qwen3.8-27B-TQ3_4S:BF16" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use YTan2000/Qwen3.8-27B-TQ3_4S with Docker Model Runner:
```
docker model run hf.co/YTan2000/Qwen3.8-27B-TQ3_4S:BF16
```

Lemonade

How to use YTan2000/Qwen3.8-27B-TQ3_4S with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull YTan2000/Qwen3.8-27B-TQ3_4S:BF16

Run and chat with the model

lemonade run user.Qwen3.8-27B-TQ3_4S-BF16

List all available models

lemonade list

Hermes Agent

How to use YTan2000/Qwen3.8-27B-TQ3_4S with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf YTan2000/Qwen3.8-27B-TQ3_4S:BF16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default YTan2000/Qwen3.8-27B-TQ3_4S:BF16

Run Hermes

hermes

Atomic Chat

Qwen3.8-27B-TQ3_4S

TurboQuant TQ3_4S build of Qwen3.8-27B — the hybrid Gated DeltaNet + Gated Attention architecture with trained Multi-Token Prediction (MTP) heads.

Required Runtime

This model requires the TurboQuant runtime fork: turbo-tan/llama.cpp-tq3

Stock llama.cpp builds cannot load the TQ3_4S tensor type. This is an MTP model: the GGUF contains a trained MTP draft block (nextn heads) — serve it with draft-MTP speculative decoding for full speed.

Files & Versions

File	Size	BPW	Context	Template
`Qwen3.8-27B-TQ3_4S.gguf` (v1)	13.8 GB	4.24	262,144	official Qwen3.8 template
`Qwen3.8-27B-TQ3_4S-v2.gguf` (recommended)	13.8 GB	4.24	262,144	fixed froggeric v22 template embedded
`mmproj-BF16.gguf`	0.93 GB	—	—	vision projector (BF16)
`mmproj-F16.gguf`	0.93 GB	—	—	vision projector (F16)

The mmproj files are the native vision-language projectors from unsloth/Qwen3.8-27B-GGUF — add --mmproj mmproj-BF16.gguf (or -F16) for image/video understanding.

v2 embeds the froggeric v22 fixed chat template directly in the GGUF — llama.cpp applies it automatically, no --chat-template-file needed. It fixes two bugs in the official 3.8 template:

String tool-arguments crash — the official template throws TypeError: Can only get item pairs from a mapping when clients send standard OpenAI-format string arguments (e.g. most agents/harnesses). v2 handles both dicts and JSON strings.
Forced xhigh reasoning steering — the official template injects "think carefully" steering instructions into every system prompt by default; v2 suppresses steering when thinking is off.

Same weights, same quant, same MTP heads — only the template differs. Measured impact: toolcall suite 81.7 → 86.7 (+5pp). If you use v1, add --chat-template-file chat_template.jinja (v22, included in this repo) for correct tool calling.

Model Details

Parent model: Qwen/Qwen3.8-27B (quantized from the unsloth/Qwen3.8-27B-GGUF BF16 source)
File: Qwen3.8-27B-TQ3_4S.gguf — 13.8 GB (4.24 BPW)
Recipe: TQ3_4S with output layers + token embeddings at q6_K (the "out6k" recipe)
Architecture: qwen3_5 — 64 layers, 48× Gated DeltaNet (linear attention) + 16× Gated Attention, trained multi-step MTP heads
Context: 262,144 tokens native

Recommended Launch

./build/bin/llama-server \
  -m Qwen3.8-27B-TQ3_4S.gguf \
  --host 127.0.0.1 --port 8080 \
  -c 32768 -np 1 -ngl 99 -fa on \
  -ctk q8_0 -ctv tq3_0 \
  --spec-type draft-mtp \
  --spec-draft-n-min 1 \
  --spec-draft-n-max 2 \
  --spec-draft-p-min 0.0 \
  --no-spec-draft-backend-sampling \
  --reasoning off --jinja

Settings notes:

--spec-draft-n-max 2 is optimal for this model's MTP head (measured)
--no-spec-draft-backend-sampling keeps CUDA graph reuse high (~25% throughput difference)
--reasoning off for agentic/benchmark work; the model supports thinking mode via --reasoning-format deepseek --reasoning-budget N

Benchmarks

Validated on NVIDIA RTX 3090 (24 GB), single stream, draft-MTP speculative decoding (--spec-draft-n-max 2).

Benchmark	Result	Notes
Hard86 (executable coding)	74/86 (86.0%)	reasoning-off, 20 tasks / 86 checks
HumanEval	92.7	official scorer, pass@1
HumanEval+	88.4	official scorer, pass@1
MBPP	90.5	official scorer, pass@1
MBPP+	77.2	official scorer, pass@1
Coding suite	93.8
Reason-math suite	86.7
Tool-call suite	86.7	v2 template
Data-extract suite	80.2
Instruction-follow suite	62.2

Model size: 13.8 GB (4.24 BPW).

Decode speed

Metric	Value
Warm long-output decode	64.78 tok/s (mean of 2 runs)
Protocol	1 warmup discarded + 2 measured runs, 1024-in / 4096-out
Speculative decoding	draft-MTP on, acceptance 0.849
Context	32768, single stream (np=1)

The 3.8's MTP head is better trained than the 3.6's (acceptance 0.849 vs 0.796), yielding ~2.7 tokens per decode step on the same hardware and recipe — that's the source of the speed uplift.

Comparison vs Qwen3.6-27B-MTP-TQ3_4S (same TQ3_4S recipe)

Gate	Qwen3.6-27B (published card)	Qwen3.8-27B
Size (GB)	13.39 GiB	13.8
Warm long-output decode	59.0 tok/s	64.78 tok/s
Hard86	76/86 (88.4%)	74/86 (86.0%)
Coding suite	100%	93.8%
Tool-call suite	96.67%	86.7% (v2)
Data-extract suite	90.97%	80.2%
Instruction-follow suite	76.67%	62.2%
Reason-math suite	73.33%	86.7%
Draft-MTP acceptance	0.796	0.849

Notes on the comparison: the 3.6 column is the published model-card number (run under that model's own tuned serving config). The 3.8 column is a first-pass battery — Hard86, task suites and speed under reasoning-off; evalplus under budget-256. The 3.8 wins decisively on decode speed, MTP acceptance and reason-math; on the task suites the 3.6's tuned-config numbers remain the reference until the 3.8 battery is re-tuned.

Tested Hardware

NVIDIA RTX 3090 (24 GB) — all numbers above

License

Subject to the Qwen3.8 base model license (Apache 2.0) and the runtime components used to run the GGUF.

Downloads last month: 2,220

GGUF

Model size

27B params

Architecture

qwen35

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for YTan2000/Qwen3.8-27B-TQ3_4S

Base model

Qwen/Qwen3.8-27B

Quantized

(408)

this model