Instructions to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

Use Docker

docker model run hf.co/ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

SGLang

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with Ollama:
```
ollama run hf.co/ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M
```

Unsloth Studio

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF to start chatting

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with Docker Model Runner:
```
docker model run hf.co/ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M
```

Lemonade

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.6-27B-A3B-Coder-MTP-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.6-27B-A3B-Coder

A code-specialist expert prune of Qwen3.6-35B-A3B: the MoE is reduced from 256 experts to 184 (72 dropped per layer, ~35B→27B, still A3B active) using a code-targeted competence map (LiveCodeBench + MultiPL-E competence classes). Same router, attention, norms, MTP head and vision tower as the base — only the expert keep-set changes.

Served at top-10 (num_experts_per_tok = 10, baked as the default). This is a routing-recovery lever: after pruning to 184 experts, activating the top-10 (vs the base top-8) recovers instruction-following at no cost to code (see below). No fine-tuning, no distillation — pure expert selection + a routing-width dial.

Recipe

Competence map: the 256e teacher is profiled per-expert on a balanced corpus + targeted LiveCodeBench and MultiPL-E (Rust/Java/JS) PASS-response classes.
Drop map: wmax aggregation with the LCB + MPE classes up-weighted (1.5) → 72/256 experts dropped per layer, protecting the code-competent experts.
Top-10 routing (num_experts_per_tok = 10) baked into the config → the shipped default. Pass --override-kv qwen35moe.expert_used_count=int:8 to any llama.cpp tool to A/B back to native top-8.

Evaluation (Q6_K, llama.cpp, temp 0.6 / top-p 0.95 / top-k 20)

Benchmark	This model	Qwen3.6-35B-A3B (256e)	coder (LCB-only)
GPQA-Diamond	0.773	0.833	0.793
MATH-500	0.620	0.730	0.620
AIME	0.733	0.633	0.767
LiveCodeBench (v6, 77q)	0.688	0.714	0.688
IFEval	0.730	0.960	0.840
HumanEval	0.970	0.970	0.963
GSM8K	0.970	0.960	0.980
ARC-Challenge	0.944	0.935	0.933
MultiPL-E	0.840	0.827	0.670
Average	0.808	0.840	0.806

Highlights: best code profile of any prune — MultiPL-E 0.840 (above the teacher; +17pp over the LCB-only coder that this model supersedes), LiveCodeBench 0.688 (tied best), HumanEval 0.970. Average 0.808 sits at the LCB-coder level and within 0.03 of the full teacher.

Verbosity / rumination (length breakdown per eval)

Aggressive expert pruning makes the model verbose on open-ended reasoning — it over-thinks before answering. This is largely inherited from the base (the 256e teacher does the same on GPQA/AIME) and is bounded by the generation cap; it does not affect the code benches, which have a natural termination anchor.

Response length in characters (content + reasoning), this model vs the 256e teacher; runaway = responses > 20k chars (of 100, or 30/198 for GPQA, 30 for AIME):

Benchmark	p50	p90	max	runaway	256e runaway
GPQA	13.8k	58.8k	129k	58	58 (same)
AIME	52.9k	85.5k	96k	25	29
IFEval	12.1k	56.2k	81k	30	11
MATH-500	2.3k	14.5k	76k	9	12
GSM8K	2.5k	12.5k	108k	6	3
ARC	1.4k	2.3k	59k	7	0
HumanEval	0.8k	1.4k	24k	1	2
LiveCodeBench / MultiPL-E	— code path —			tight	tight

Reading it: GPQA/AIME verbosity is essentially the base model (58 vs 58, 25 vs 29). Only IFEval shows prune-added rumination (30 vs 11) — the trade for the code-targeted drop map. Code and math-with-boxing tasks terminate cleanly. If you want tighter output, a repetition/length penalty at serve time (or top-8 via the override above) reduces the tail.

Reasoning budget and thinking stop phrase (llama.cpp)

Qwen 3.6 reasons at length by design, and on a hard prompt it can consume the whole context window before it answers. llama.cpp can bound the thinking block with a sampler, and — the part that actually matters — tell the model why the block is being closed.

Needs llama.cpp b8508 or newer for the flags, b10091 or newer for the per-request overrides.

Serve with a bounded thinking block

llama-server -m Qwen3.6-27B-A3B-Coder-Q4_K_M.gguf -c 32768 -ngl 99 \
    --jinja \
    --reasoning-budget 8192 \
    --reasoning-budget-message $'\n\nConsidering the limited time by the user, I have to give the solution based on the thinking directly now.\n' \
    --temp 0.6 --top-k 20 --top-p 0.95

flag	meaning
`--reasoning-budget N`	`-1` unrestricted (default), `0` close the block immediately, `N > 0` cap it at N tokens
`--reasoning-budget-message`	text written into the block just before the closing tag is forced
`--jinja`	required — the delimiters come from the chat template (`<think>` … `</think>`). Without it llama.cpp has no tags to count and the budget silently does nothing

Both flags also read from the environment: LLAMA_ARG_THINK_BUDGET and LLAMA_ARG_THINK_BUDGET_MESSAGE.

--reasoning-format is not part of this. It only decides how the thinking is handed back — message.reasoning_content versus left inline in message.content — and never whether the budget is enforced: the delimiters the sampler counts are set by the chat template regardless, so the cap binds under auto, deepseek and none alike. The default auto already extracts reasoning and is behaviourally identical to deepseek (they differ only in name; the sole branch in the parser is != none). Leave it at the default so the model's own tool-call and channel handling stays in play, and pin deepseek only when a harness needs the thinking kept out of content.

--reasoning-budget on its own forces the closing tag the moment the budget runs out, wherever the model happens to be. When that lands mid-thought the model frequently does not register that it was interrupted: it carries on reasoning, now inside the visible answer. The stop phrase is what prevents that — it gives the model a reason to be finishing.

Two wordings that work

# "qwen" — the string Qwen's own service uses, from their docs
--reasoning-budget-message $'\n\nConsidering the limited time by the user, I have to give the solution based on the thinking directly now.\n'

# "voice" — shorter, in the model's own reasoning voice
--reasoning-budget-message $'\n\nOK, I have enough to answer now.\n'

Wording is model-specific: Qwen note that the ability to act on such a message "is not explicitly trained but emerges naturally", so it is worth trying both on your own workload. Leading and trailing newlines matter — they keep the phrase off whatever half-finished line the cut landed on.

What it measures out to

Measured on the Qwen3.6-35B-A3B base this model is pruned from. Three hard questions, temperature 0.6, fixed seed, answer characters with wall time in brackets. Every run answered all three correctly, and thinking length is unchanged by the message in every row:

budget	no message	`qwen`	`voice`
2048	1907 (69 s)	1838 (42 s)	1615 (41 s)
4096	18015 (170 s)	2642 (78 s)	1441 (104 s)
8192	3642 (158 s)	1848 (129 s)	2023 (175 s)

The 4096 row is the failure this exists for: the cap lands mid-thought and the reasoning simply continues in the answer, ten times longer and 2.2x the wall time, for the same three correct answers. Both phrases remove it.

Per request, instead of per server

The server accepts both as request fields, overriding the command line:

{
  "messages": [ ... ],
  "thinking_budget_tokens": 8192,
  "reasoning_budget_message": "\n\nOK, I have enough to answer now.\n"
}

On the raw /completion endpoint the delimiters are not inferred, so they have to be supplied with the budget:

{
  "prompt": "...",
  "reasoning_budget_tokens": 8192,
  "reasoning_budget_start_tag": "<think>",
  "reasoning_budget_end_tag": "</think>",
  "reasoning_budget_message": "\n\nOK, I have enough to answer now.\n"
}

On b10091 the message field must be present on /completion requests even when empty: llama.cpp builds the sequence it forces from message + end_tag inside that field's handler, so omitting it leaves the budget with nothing to force — the sampler logs as though the cap fired while the thinking block stays open.

Rules of thumb

Keep -c several times larger than the budget. A budget equal to the context lets the thinking phase fill the window on its own.
A quarter of the context is a sensible starting point: 8192 at -c 32768.
Qwen recommend keeping a thinking budget above 1024 tokens; below that the cap tends to land before the model has committed to an approach.
The budget is per thinking block, not per response — the sampler re-arms when it sees a new opening tag, so a multi-turn agent gets a fresh window each time.

Formats

GGUF (this repo family): full imatrix quant sweep (Q8_0 → IQ2, plus ContribDynamic CD-* per-layer quants) in Qwen3.6-27B-A3B-Coder-MTP-GGUF. Includes the native MTP head (speculative decoding) and a -vision mmproj for multimodal use. imatrix.dat archived in-repo.
Ollama: mannix/qwen3.6-27b-a3b-coder (text) and …-vision tags (with mmproj).

Quantization quality

Every K-quant (Q4_K_S → Q6_K and the _L variants) is built with imatrix. On this model the imatrix is load-bearing at 4-bit — the opposite of Gemma-4, where imatrix degrades K-quants. Measured on a deterministic greedy MultiPL-E code probe, the entire imatrix K-family and the ContribDynamic CD-* tiers sit at full-precision parity (within measurement noise of the F16 anchor). The only outlier was a plain, imatrix-free Q4_K_M, which fell ~12pp below parity — which is why imatrix is now the default for every tier in this repo.

Recommended tier: CD-IQ4_K_M (~16 GB) — full-precision-parity code quality at the smallest at-parity size.
Q4_0 / Q4_1 are not shipped — superseded by the imatrix K-quants (legacy round-to-nearest tiers offered no quality at their size).

⚠️ Hardware note — low i-quants (IQ2_M, IQ3_M) on Blackwell (sm_120) GPUs

The low i-quant tiers IQ2_M and IQ3_M produce incoherent output ("token salad") on NVIDIA Blackwell GPUs (sm_120, e.g. RTX PRO 6000) — under both stock llama.cpp and opencoti-llamafile, which share the same ggml CUDA IQ2_S/IQ3_S kernel. The weights are fine: the identical GGUF is fully coherent and produces correct code on CPU and on Ampere/Ada GPUs (verified on an RTX 3090). This is a llama.cpp/ggml CUDA-kernel issue on the sm_120 IQ2_S/IQ3_S path, not a defect in these files.

On a Blackwell GPU, instead use: the K-quants (Q2_K_L, Q3_K_S/Q3_K_M/Q3_K_L), or IQ2_XS / IQ4_XS / the CD-* tiers — all coherent on Blackwell in the same size band (Q2_K_L/IQ2_XS ≈ the IQ2_M band; Q3_K_M/Q3_K_L ≈ the IQ3_M band).
IQ2_M / IQ3_M are kept in the repo because they are correct on CPU and Ampere/Ada GPUs.

Notes

Top-10 is baked as the default; the model was selected and evaluated at top-10.
Same tokenizer, chat template, MTP head and vision tower as the base.
Research checkpoint. Verbosity on open-ended prompts is a known, base-inherited trait.

Downloads last month: 107,133

GGUF

Model size

26B params

Architecture

qwen35moe

Hardware compatibility

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Model tree for ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF

Base model

Qwen/Qwen3.6-35B-A3B

Quantized

(755)

this model

Collection including ManniX-ITA/Qwen3.6-27B-A3B-Coder-MTP-GGUF

Qwen-3.6

Collection

7 items • Updated Jul 19