Instructions to use ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve with MLX:

# Make sure mlx-lm is installed
# pip install --upgrade mlx-lm

# Generate text with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve")

prompt = "Write a story about Einstein"
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True
)

text = generate(model, tokenizer, prompt=prompt, verbose=True)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve"

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve

Run Hermes

hermes

OpenClaw new

How to use ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

MLX LM

How to use ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve with MLX LM:

Generate or start a chat session

# Install MLX LM
uv tool install mlx-lm
# Interactive chat REPL
mlx_lm.chat --model "ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve"

Run an OpenAI-compatible server

# Install MLX LM
uv tool install mlx-lm
# Start the server
mlx_lm.server --model "ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve"
# Calling the OpenAI-compatible server with curl
curl -X POST "http://localhost:8000/v1/chat/completions" \
   -H "Content-Type: application/json" \
   --data '{
     "model": "ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve",
     "messages": [
       {"role": "user", "content": "Hello"}
     ]
   }'

Qwen3.6-27B 4-bit + MTP — up to 2× faster decode with mlx-serve

Qwen 3.6 27B that drafts its own future. This is the standard mlx-community/Qwen3.6-27B-4bit conversion (trunk shards and config byte-identical) plus the model's multi-token-prediction head (mtp/weights.safetensors, ~500 MB) — the one-layer draft head Qwen trains and ships in Qwen/Qwen3.6-27B but that gets dropped by most conversions. The head here is extracted directly from Qwen's original bf16 weights, INT8-quantized (near-lossless for a draft head).

With a runtime that knows how to use it, the model speculates on itself: the MTP head drafts the next tokens, the trunk verifies them in one batched pass with exact rejection sampling — the same output distribution as regular decoding, at temperature 0 or your real sampling settings, just faster. No second model eating your RAM, no greedy shortcut.

Quick start

mlx-serve is a native (Zig, no Python) inference server for Apple Silicon with OpenAI- and Anthropic-compatible APIs, plus MLX Core, a macOS menu-bar app with chat, agent mode, and MCP tool calling. It detects the MTP sidecar in this repo and enables speculative decoding automatically — zero configuration:

# Install mlx-serve
brew tap ddalcu/mlx-serve https://github.com/ddalcu/mlx-serve
brew install mlx-serve          # CLI server
brew install --cask mlx-core    # optional: GUI menu-bar app

# Get the model
hf download ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve --local-dir Qwen3.6-27B-4bit-MTP

# Serve it
mlx-serve --model ./Qwen3.6-27B-4bit-MTP --serve --port 11234

curl http://127.0.0.1:11234/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"default","messages":[{"role":"user","content":"hi"}],"stream":true}'

Works with Claude Code out of the box (ANTHROPIC_BASE_URL=http://localhost:11234 claude), plus the OpenAI SDK, Continue, Cursor, Open WebUI — /v1/chat/completions, /v1/messages, /v1/completions (FIM), and /v1/responses, streaming and non-streaming. Speculation engages on every endpoint; opt out per request with "enable_mtp": false or at launch with --no-mtp.

Project: github.com/ddalcu/mlx-serve · Site: ddalcu.github.io/mlx-serve · App + CLI downloads: latest release

Up to 2× faster — measured

Speculative decoding pays off most where the next tokens are predictable: agent loops, file edits, refactors, code generation. Measured with mlx-serve on an M4 Max (greedy, identical output to non-speculative decoding):

Workload	Without MTP	With MTP	Speedup
Agent-style file edit / echo	29.0 tok/s	51.6 tok/s (90% draft acceptance)	1.78×
Code generation	29.0 tok/s	41.6 tok/s (93% acceptance)	1.43×
Creative writing	29.3 tok/s	32.4 tok/s (73% acceptance)	1.11×

Newer chips with more compute headroom (M5 generation) push the speculative win further — batched verification gets cheaper relative to memory-bound autoregressive decoding, which is how this technique reaches the 2× range. mlx-serve also watches the per-request acceptance rate and adapts draft depth on the fly, so there is no tuning pass and no regression on hard-to-predict content — --mtp-depth is there if you want to experiment.

These numbers stack on top of mlx-serve's baseline lead: on identical 4-bit MLX weights it decodes ~35% faster (geomean) than LM Studio on the same hardware — see the project README for the full benchmark suite.

Also works as a plain Qwen 3.6

The trunk shards and config.json are unmodified from the mlx-community 4-bit (group 64) conversion, so this repo loads anywhere that model does — mlx-lm, LM Studio, etc. — which simply ignore the mtp/ folder and run standard autoregressive decoding.

What's in the box

File	What it is
`model-*.safetensors`	Byte-identical to `mlx-community/Qwen3.6-27B-4bit` (4-bit, group 64)
`mtp/weights.safetensors`	Qwen's original MTP head (fc + one gated full-attention layer + norms), extracted from `Qwen/Qwen3.6-27B` and INT8-quantized (group 32)
`config.json`	Byte-identical to the mlx-community config (mlx-serve detects the head from the `mtp/` folder and infers its quantization from tensor geometry)

Attribution

Component	Source	License
Base model + MTP head	Qwen/Qwen3.6-27B	Apache-2.0
4-bit MLX trunk conversion	mlx-community/Qwen3.6-27B-4bit	Apache-2.0
Head extraction/quantization + MTP serving	mlx-serve	MIT

See NOTICE for the full derivation chain.

Downloads last month: 626

Safetensors

Model size

5B params

Tensor type

BF16

U32

MLX

Hardware compatibility

4-bit

Model tree for ddalcu/Qwen3.6-27B-4bit-MTP-MLX-Serve

Base model

Qwen/Qwen3.6-27B

Quantized

mlx-community/Qwen3.6-27B-4bit

Quantized

(1)

this model