Image-Text-to-Text
MLX
Safetensors
English
Chinese
multilingual
qwen3_5
text-generation
mlx-vlm
qwen
qwen3
qwen3.5
qwen3.6
claude-opus-distill
reasoning
vision
multimodal
abliterated
refusal-ablated
uncensored
apple-silicon
mxfp4
ocp-mx
speculative-decoding
dflash
conversational
4-bit precision
Instructions to use lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4 with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4") config = load_config("lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4 with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4"
Configure the model in Pi
# Install Pi: npm install -g @mariozechner/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent new
How to use lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4 with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4
Run Hermes
hermes
- OpenClaw new
How to use lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4 with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Update model card
Browse files
README.md
CHANGED
|
@@ -14,7 +14,6 @@ tags:
|
|
| 14 |
- qwen3
|
| 15 |
- qwen3.5
|
| 16 |
- qwen3.6
|
| 17 |
-
- qwopus
|
| 18 |
- claude-opus-distill
|
| 19 |
- reasoning
|
| 20 |
- vision
|
|
@@ -22,7 +21,6 @@ tags:
|
|
| 22 |
- abliterated
|
| 23 |
- refusal-ablated
|
| 24 |
- uncensored
|
| 25 |
-
- zerofuse
|
| 26 |
- apple-silicon
|
| 27 |
- mxfp4
|
| 28 |
- ocp-mx
|
|
@@ -30,7 +28,6 @@ tags:
|
|
| 30 |
- dflash
|
| 31 |
- conversational
|
| 32 |
base_model:
|
| 33 |
-
- osmapi/osmQwopus-3.6-27B-V2-heretic-abliterated-uncensored
|
| 34 |
- Jackrong/Qwopus3.6-27B-v2
|
| 35 |
- Qwen/Qwen3.6-27B
|
| 36 |
base_model_relation: quantized
|
|
@@ -38,13 +35,13 @@ pipeline_tag: image-text-to-text
|
|
| 38 |
library_name: mlx
|
| 39 |
---
|
| 40 |
|
| 41 |
-
#
|
| 42 |
|
| 43 |
> โ
**VISION PRESERVED.** Unlike most 4-bit quantization pipelines that degrade or strip the vision tower, this release keeps the full Qwen3.6-VL ViT and all projector weights in **fp16**. Use it as a real multimodal model at half the size of the 8-bit build.
|
| 44 |
|
| 45 |
-
**MLX MXFP4 (OCP Microscaling FP4) quantization** of a ZeroFuse-abliterated
|
| 46 |
|
| 47 |
-
> Fastest absolute throughput of any
|
| 48 |
|
| 49 |
---
|
| 50 |
|
|
@@ -57,33 +54,33 @@ library_name: mlx
|
|
| 57 |
| **Scheme** | OCP Microscaling FP4 (MXFP4) โ `mode=mxfp4`, `group_size=32` |
|
| 58 |
| **Vision tower** | โ
fp16 โ all 333 ViT + projector weights at full precision |
|
| 59 |
| **SSM params** | โ
fp16 โ `a_log`, `dt_bias`, `conv1d` kept at full precision |
|
| 60 |
-
| **Refusal rate (ZeroFuse, n=100)** | **4/100** (vs vanilla
|
| 61 |
| **KL divergence vs BF16 (quant)** | **0.034 nats** mean ยท **JSD 0.009** (essentially identical) |
|
| 62 |
| **Top-1 greedy agreement vs BF16** | **92.4%** โ same argmax token at 9 in 10 positions |
|
| 63 |
| **PPL increase vs BF16** | **+0.87%** โ negligible quality loss |
|
| 64 |
-
| **Throughput (M4 Max, vanilla)** | **27.7 tok/s** โ fastest of all
|
| 65 |
| **Throughput + DFlash (b=8)** | **29.3 tok/s** (1.06ร) |
|
| 66 |
| **Recommended RAM** | 20 GB+ Apple Silicon โ fits M2 Pro / M3 Pro and above |
|
| 67 |
| **Best for** | Fastest local inference; 16โ32 GB Macs; vision + text |
|
| 68 |
-
| **Released by** |
|
| 69 |
|
| 70 |
---
|
| 71 |
|
| 72 |
-
## ๐๏ธ All
|
| 73 |
|
| 74 |
-
The full
|
| 75 |
|
| 76 |
| Quant | Format | BPW | Disk | Vision | Runtime | Link |
|
| 77 |
|---|---|---|---|---|---|---|
|
| 78 |
-
| 8-bit | MLX | 8.50 | ~27 GB | โ
native | mlx-vlm | [`โฆ-8-bit-mlx`](https://huggingface.co/
|
| 79 |
-
| 6-bit | MLX | 6.66 | ~21 GB | โ
native | mlx-vlm | [`โฆ-6-bit-mlx`](https://huggingface.co/
|
| 80 |
| **MXFP4** **(this repo)** | MLX | **4.25** | **~14 GB** | โ
**fp16 tower** | mlx-mtp / mlx-vlm | โ *(you are here)* |
|
| 81 |
-
| OptiQ 3.7bpw | MLX | ~3.7 | ~14 GB | โ
ViT spliced | mlx-vlm | [`โฆ-OptiQ-3.7bpw-mlx`](https://huggingface.co/
|
| 82 |
-
| Q8_0 | GGUF | 8.50 | ~28 GB | โ
via mmproj | llama.cpp | [`โฆ-8-bit-GGUF`](https://huggingface.co/
|
| 83 |
-
| Q6_K | GGUF | ~6.56 | ~22 GB | โ
via mmproj | llama.cpp | [`โฆ-6-bit-GGUF`](https://huggingface.co/
|
| 84 |
-
| Q4_K_M | GGUF | ~4.92 | ~16 GB | โ
via mmproj | llama.cpp | [`โฆ-Q4_K_M-GGUF`](https://huggingface.co/
|
| 85 |
-
| TQ3_4S | GGUF | 4.00 (~3.5 eff) | ~14 GB | โ
via mmproj | [llama.cpp-tq3](https://github.com/turbo-tan/llama.cpp-tq3) | [`โฆ-TQ3_4s-GGUF`](https://huggingface.co/
|
| 86 |
-
| TQ3_1S | GGUF | 4.00 (~3.5 eff) | ~14 GB | โ
via mmproj | [llama.cpp-tq3](https://github.com/turbo-tan/llama.cpp-tq3) | [`โฆ-TQ3_1s-GGUF`](https://huggingface.co/
|
| 87 |
|
| 88 |
> ๐ All variants share the same abliterated base weights โ pick by your runtime (Apple Silicon โ MLX; CUDA/CPU/cross-platform โ GGUF) and your RAM budget. MXFP4 is the fastest MLX build on M4 Max; use 8-bit if you want the MTP head for native speculative decoding without a drafter.
|
| 89 |
|
|
@@ -92,29 +89,29 @@ The full osmQwopus family from [osmAPI](https://osmAPI.com) โ same ZeroFuse-ab
|
|
| 92 |
## ๐งฌ Lineage
|
| 93 |
|
| 94 |
```
|
| 95 |
-
Qwen/Qwen3.6-27B
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
Jackrong/Qwopus3.6-27B-v2
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
ZeroFuse abliteration (TPE-50)
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
this repo โ MLX MXFP4 quant
|
| 111 |
```
|
| 112 |
|
| 113 |
Direct upstream links:
|
| 114 |
|
| 115 |
* ๐๏ธ Foundation: [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B)
|
| 116 |
* ๐ Claude-Opus distill: [Jackrong/Qwopus3.6-27B-v2](https://huggingface.co/Jackrong/Qwopus3.6-27B-v2)
|
| 117 |
-
* ๐ Abliteration tool: [ZeroFuse](https://github.com/junainfinity/ZeroFuse) by
|
| 118 |
* ๐งฎ Quantization tool: [mlx-mtp](https://github.com/junainfinity/mlx-mtp) โ vision-preserving MXFP4 quantizer
|
| 119 |
|
| 120 |
---
|
|
@@ -137,11 +134,11 @@ ZeroFuse measures refusals on `mlabonne/harmful_behaviors` (100 hard red-team pr
|
|
| 137 |
|
| 138 |
## ๐งช Method
|
| 139 |
|
| 140 |
-
**Abliteration** (inherited from the [BF16 source](https://huggingface.co/
|
| 141 |
|
| 142 |
**MXFP4 quantization** (this repo):
|
| 143 |
|
| 144 |
-
1. **Source** โ loaded `
|
| 145 |
2. **Layer audit** โ `mlx_mtp.mxfp4_quantize` catalogued 1199 tensors: identified all LM linear projections (attention Q/K/V/O + MLP gate/up/down + lm_head) as MXFP4 candidates; flagged 196 vision-tower + projector tensors and 48 SSM-sensitive params (`a_log`, `dt_bias`, `conv1d`) for fp16 preservation.
|
| 146 |
3. **Quantization** โ LM linears โ MXFP4 (OCP MX E2M1, `group_size=32`) using `mlx.core.quantize`. Vision + SSM tensors written verbatim at fp16. 15 MTP head weights absent from source โ quantizer set `mtp_num_hidden_layers: 0` in output config to prevent strict-load failure.
|
| 147 |
4. **Output** โ 3 safetensor shards, 14 GB total. Embedded `mlx_mtp` config block records format, vision_fp16=True, mtp_preserved=False.
|
|
@@ -179,12 +176,12 @@ git clone https://github.com/junainfinity/mlx-mtp && pip install -e ./mlx-mtp
|
|
| 179 |
```python
|
| 180 |
from mlx_mtp.engine import load_model, vanilla_generate
|
| 181 |
|
| 182 |
-
model, processor, config = load_model("
|
| 183 |
|
| 184 |
result = vanilla_generate(
|
| 185 |
-
|
| 186 |
-
|
| 187 |
-
|
| 188 |
)
|
| 189 |
print(result["text"])
|
| 190 |
print(f"{result['tps']:.1f} tok/s")
|
|
@@ -196,14 +193,14 @@ print(f"{result['tps']:.1f} tok/s")
|
|
| 196 |
from mlx_mtp.engine import load_model
|
| 197 |
from mlx_mtp.dflash import load_dflash_drafter, dflash_generate
|
| 198 |
|
| 199 |
-
model, processor, config = load_model("
|
| 200 |
-
drafter, _
|
| 201 |
|
| 202 |
result = dflash_generate(
|
| 203 |
-
|
| 204 |
-
|
| 205 |
-
|
| 206 |
-
|
| 207 |
)
|
| 208 |
print(result["text"])
|
| 209 |
print(f"{result['tps']:.1f} tok/s")
|
|
@@ -220,8 +217,8 @@ from mlx_vlm import load, generate
|
|
| 220 |
from mlx_vlm.prompt_utils import apply_chat_template
|
| 221 |
from mlx_vlm.utils import load_config
|
| 222 |
|
| 223 |
-
model, processor = load("
|
| 224 |
-
config = load_config("
|
| 225 |
|
| 226 |
messages = [{"role": "user", "content": "Explain the difference between SSM and softmax attention in 3 sentences."}]
|
| 227 |
prompt = apply_chat_template(processor, config, messages)
|
|
@@ -232,10 +229,10 @@ print(generate(model, processor, prompt, max_tokens=400, verbose=True))
|
|
| 232 |
|
| 233 |
```python
|
| 234 |
out = generate(
|
| 235 |
-
|
| 236 |
-
|
| 237 |
-
|
| 238 |
-
|
| 239 |
)
|
| 240 |
print(out)
|
| 241 |
```
|
|
@@ -244,33 +241,33 @@ print(out)
|
|
| 244 |
|
| 245 |
```bash
|
| 246 |
pip install mlx-omni-server
|
| 247 |
-
mlx-omni-server
|
| 248 |
```
|
| 249 |
|
| 250 |
```bash
|
| 251 |
curl -s http://127.0.0.1:10240/v1/chat/completions \
|
| 252 |
-
|
| 253 |
-
|
| 254 |
-
|
| 255 |
-
|
| 256 |
-
|
| 257 |
-
|
| 258 |
-
|
| 259 |
-
|
| 260 |
-
|
| 261 |
```
|
| 262 |
|
| 263 |
---
|
| 264 |
|
| 265 |
## ๐งช Quantization details
|
| 266 |
|
| 267 |
-
* **Source weights**: `
|
| 268 |
* **Quantization scheme**: OCP Microscaling FP4 (MXFP4) โ 4-bit E2M1 mantissa (2 exponent bits, 1 mantissa bit) + shared E8M0 scale per group of 32 elements. Hardware-accelerated on Apple M-series via MLX's native MX support.
|
| 269 |
* **Group size**: 32.
|
| 270 |
* **Effective bits/weight**: 4 + 8/32 = **4.25 bpw** (8-bit E8M0 scale shared across 32 weights).
|
| 271 |
* **Vision tower**: preserved verbatim in fp16 โ 333 vision weights (ViT encoder, vision projector, image-token embeddings, deepstack indexes). Pixel/patch features degrade visibly at 4-bit; keeping them fp16 is free (vision = minority of total params).
|
| 272 |
* **SSM-sensitive params**: preserved in fp16 โ 48 params (`a_log`, `dt_bias`, `conv1d`). Qwen3.5's hybrid-SSM recurrent state is numerically sensitive; fp16 costs nothing at scale.
|
| 273 |
-
* **MTP head**: disabled (`mtp_num_hidden_layers: 0`). The BF16 source had 0 MTP weights; quantizer detects this and clears the config to prevent strict-load failure. For native MTP speculative decoding, use the [8-bit build](https://huggingface.co/
|
| 274 |
* **Quantizer**: [`mlx_mtp.mxfp4_quantize`](https://github.com/junainfinity/mlx-mtp/blob/main/mlx_mtp/mxfp4_quantize.py).
|
| 275 |
|
| 276 |
### Architecture notes
|
|
@@ -283,7 +280,7 @@ Qwen 3.6 27B uses a **hybrid attention stack** โ 3 linear-attention (GatedDelt
|
|
| 283 |
|
| 284 |
* **Uncensored.** Refusal directions were surgically removed; this model will answer prompts the parent would refuse. Use responsibly and within applicable law. The release is provided for safety research, red-teaming, and creative/educational use cases.
|
| 285 |
* **Identity preserved.** The model still self-identifies as Qwen (developed by Alibaba's Tongyi Lab) โ abliteration does not rewrite factual self-knowledge.
|
| 286 |
-
* **Heavy chain-of-thought.**
|
| 287 |
|
| 288 |
---
|
| 289 |
|
|
@@ -295,12 +292,11 @@ Qwen 3.6 27B uses a **hybrid attention stack** โ 3 linear-attention (GatedDelt
|
|
| 295 |
|
| 296 |
**Quantization & release**
|
| 297 |
|
| 298 |
-
|
| 299 |
-
[TERV.Pro student research team](https://terv.pro)
|
| 300 |
|
| 301 |
**Claude-Opus reasoning distill**
|
| 302 |
|
| 303 |
-
[Jackrong](https://huggingface.co/Jackrong) โ author of [
|
| 304 |
|
| 305 |
**Foundation model**
|
| 306 |
|
|
@@ -308,7 +304,7 @@ Qwen 3.6 27B uses a **hybrid attention stack** โ 3 linear-attention (GatedDelt
|
|
| 308 |
|
| 309 |
**Abliteration toolkit**
|
| 310 |
|
| 311 |
-
[ZeroFuse](https://github.com/junainfinity/ZeroFuse) by
|
| 312 |
|
| 313 |
**Community priors that seeded the TPE search**
|
| 314 |
|
|
@@ -322,11 +318,11 @@ Qwen 3.6 27B uses a **hybrid attention stack** โ 3 linear-attention (GatedDelt
|
|
| 322 |
|
| 323 |
## ๐ License
|
| 324 |
|
| 325 |
-
Apache-2.0, inherited from the foundation (Qwen3.6-27B) and the distill (
|
| 326 |
|
| 327 |
---
|
| 328 |
|
| 329 |
-
Need a hosted endpoint, custom quant, or larger-scale inference?
|
| 330 |
|
| 331 |
---
|
| 332 |
|
|
@@ -336,10 +332,10 @@ This MLX build supports **block-diffusion speculative decoding** via **[DFlash](
|
|
| 336 |
|
| 337 |
```
|
| 338 |
Apple M4 Max ยท 256 tok greedy ยท 2026-06-11
|
| 339 |
-
|
| 340 |
-
|
| 341 |
-
|
| 342 |
-
|
| 343 |
```
|
| 344 |
|
| 345 |
**Why the modest speedup?** DFlash's drafter overhead becomes a larger fraction of total time as baseline throughput rises. The 8-bit build sees ~3ร gains because it starts slower; mxfp4 is already bandwidth-limited in a faster regime. If raw throughput is the priority, mxfp4 vanilla at 27.7 tok/s already beats oQ8 + DFlash (best = 23 tok/s).
|
|
@@ -350,12 +346,12 @@ Apple M4 Max ยท 256 tok greedy ยท 2026-06-11
|
|
| 350 |
from mlx_mtp.engine import load_model
|
| 351 |
from mlx_mtp.dflash import load_dflash_drafter, dflash_generate
|
| 352 |
|
| 353 |
-
model, processor, config = load_model("
|
| 354 |
drafter, _ = load_dflash_drafter("z-lab/Qwen3.6-27B-DFlash")
|
| 355 |
|
| 356 |
out = dflash_generate(model, processor, config, drafter,
|
| 357 |
-
|
| 358 |
-
|
| 359 |
print(out["text"])
|
| 360 |
```
|
| 361 |
|
|
@@ -363,9 +359,9 @@ print(out["text"])
|
|
| 363 |
|
| 364 |
```bash
|
| 365 |
python3 -m mlx_vlm generate \
|
| 366 |
-
|
| 367 |
-
|
| 368 |
-
|
| 369 |
```
|
| 370 |
|
| 371 |
- Requires access to the gated drafter [`z-lab/Qwen3.6-27B-DFlash`](https://huggingface.co/z-lab/Qwen3.6-27B-DFlash) (one-click "Agree and access").
|
|
|
|
| 14 |
- qwen3
|
| 15 |
- qwen3.5
|
| 16 |
- qwen3.6
|
|
|
|
| 17 |
- claude-opus-distill
|
| 18 |
- reasoning
|
| 19 |
- vision
|
|
|
|
| 21 |
- abliterated
|
| 22 |
- refusal-ablated
|
| 23 |
- uncensored
|
|
|
|
| 24 |
- apple-silicon
|
| 25 |
- mxfp4
|
| 26 |
- ocp-mx
|
|
|
|
| 28 |
- dflash
|
| 29 |
- conversational
|
| 30 |
base_model:
|
|
|
|
| 31 |
- Jackrong/Qwopus3.6-27B-v2
|
| 32 |
- Qwen/Qwen3.6-27B
|
| 33 |
base_model_relation: quantized
|
|
|
|
| 35 |
library_name: mlx
|
| 36 |
---
|
| 37 |
|
| 38 |
+
# Qwen3.6-27B-v2.1-uncensored-mxfp4
|
| 39 |
|
| 40 |
> โ
**VISION PRESERVED.** Unlike most 4-bit quantization pipelines that degrade or strip the vision tower, this release keeps the full Qwen3.6-VL ViT and all projector weights in **fp16**. Use it as a real multimodal model at half the size of the 8-bit build.
|
| 41 |
|
| 42 |
+
**MLX MXFP4 (OCP Microscaling FP4) quantization** of a ZeroFuse-abliterated Qwen 3.6 27B v2 (the Jackrong Claude-Opus reasoning distill of Qwen 3.6 27B). Refusals reduced from **91/100 โ 4/100** with KL drift of just **0.0176** (measured at BF16). Quantization adds only **+0.87% perplexity** with **92.4% greedy-token agreement** vs the BF16 reference. By the Lemura Labs research team. Quantized with **[mlx-mtp](https://github.com/junainfinity/mlx-mtp)** โ a vision-preserving MXFP4 quantizer.
|
| 43 |
|
| 44 |
+
> Fastest absolute throughput of any Qwen3.6-27B MLX build โ **27.7 tok/s vanilla** on M4 Max. Apple Silicon is memory-bandwidth-bound: halving the model fits more weights in fast SRAM, which matters more than the precision drop.
|
| 45 |
|
| 46 |
---
|
| 47 |
|
|
|
|
| 54 |
| **Scheme** | OCP Microscaling FP4 (MXFP4) โ `mode=mxfp4`, `group_size=32` |
|
| 55 |
| **Vision tower** | โ
fp16 โ all 333 ViT + projector weights at full precision |
|
| 56 |
| **SSM params** | โ
fp16 โ `a_log`, `dt_bias`, `conv1d` kept at full precision |
|
| 57 |
+
| **Refusal rate (ZeroFuse, n=100)** | **4/100** (vs vanilla Qwen 3.6 **91/100**) |
|
| 58 |
| **KL divergence vs BF16 (quant)** | **0.034 nats** mean ยท **JSD 0.009** (essentially identical) |
|
| 59 |
| **Top-1 greedy agreement vs BF16** | **92.4%** โ same argmax token at 9 in 10 positions |
|
| 60 |
| **PPL increase vs BF16** | **+0.87%** โ negligible quality loss |
|
| 61 |
+
| **Throughput (M4 Max, vanilla)** | **27.7 tok/s** โ fastest of all Qwen3.6-27B MLX builds |
|
| 62 |
| **Throughput + DFlash (b=8)** | **29.3 tok/s** (1.06ร) |
|
| 63 |
| **Recommended RAM** | 20 GB+ Apple Silicon โ fits M2 Pro / M3 Pro and above |
|
| 64 |
| **Best for** | Fastest local inference; 16โ32 GB Macs; vision + text |
|
| 65 |
+
| **Released by** | Lemura Labs |
|
| 66 |
|
| 67 |
---
|
| 68 |
|
| 69 |
+
## ๐๏ธ All Qwen3.6-27B variants
|
| 70 |
|
| 71 |
+
The full Qwen3.6-27B family from Lemura Labs โ same ZeroFuse-abliterated weights (refusal 4/100, KL 0.0176), different quant schemes for different runtimes.
|
| 72 |
|
| 73 |
| Quant | Format | BPW | Disk | Vision | Runtime | Link |
|
| 74 |
|---|---|---|---|---|---|---|
|
| 75 |
+
| 8-bit | MLX | 8.50 | ~27 GB | โ
native | mlx-vlm | [`โฆ-8-bit-mlx`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-8-bit-mlx) |
|
| 76 |
+
| 6-bit | MLX | 6.66 | ~21 GB | โ
native | mlx-vlm | [`โฆ-6-bit-mlx`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-6-bit-mlx) |
|
| 77 |
| **MXFP4** **(this repo)** | MLX | **4.25** | **~14 GB** | โ
**fp16 tower** | mlx-mtp / mlx-vlm | โ *(you are here)* |
|
| 78 |
+
| OptiQ 3.7bpw | MLX | ~3.7 | ~14 GB | โ
ViT spliced | mlx-vlm | [`โฆ-OptiQ-3.7bpw-mlx`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-OptiQ-3.7bpw-mlx) |
|
| 79 |
+
| Q8_0 | GGUF | 8.50 | ~28 GB | โ
via mmproj | llama.cpp | [`โฆ-8-bit-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-8-bit-GGUF) |
|
| 80 |
+
| Q6_K | GGUF | ~6.56 | ~22 GB | โ
via mmproj | llama.cpp | [`โฆ-6-bit-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-6-bit-GGUF) |
|
| 81 |
+
| Q4_K_M | GGUF | ~4.92 | ~16 GB | โ
via mmproj | llama.cpp | [`โฆ-Q4_K_M-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-Q4_K_M-GGUF) |
|
| 82 |
+
| TQ3_4S | GGUF | 4.00 (~3.5 eff) | ~14 GB | โ
via mmproj | [llama.cpp-tq3](https://github.com/turbo-tan/llama.cpp-tq3) | [`โฆ-TQ3_4s-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-TQ3_4s-GGUF) |
|
| 83 |
+
| TQ3_1S | GGUF | 4.00 (~3.5 eff) | ~14 GB | โ
via mmproj | [llama.cpp-tq3](https://github.com/turbo-tan/llama.cpp-tq3) | [`โฆ-TQ3_1s-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-TQ3_1s-GGUF) |
|
| 84 |
|
| 85 |
> ๐ All variants share the same abliterated base weights โ pick by your runtime (Apple Silicon โ MLX; CUDA/CPU/cross-platform โ GGUF) and your RAM budget. MXFP4 is the fastest MLX build on M4 Max; use 8-bit if you want the MTP head for native speculative decoding without a drafter.
|
| 86 |
|
|
|
|
| 89 |
## ๐งฌ Lineage
|
| 90 |
|
| 91 |
```
|
| 92 |
+
Qwen/Qwen3.6-27B (Qwen Team โ base multimodal pretrain)
|
| 93 |
+
โ
|
| 94 |
+
โผ
|
| 95 |
+
Jackrong/Qwopus3.6-27B-v2 (Jackrong โ Claude-Opus reasoning distill)
|
| 96 |
+
โ
|
| 97 |
+
โผ
|
| 98 |
+
ZeroFuse abliteration (TPE-50) (Lemura Labs ยท TERV.Pro)
|
| 99 |
+
โโโ 25 random startup trials
|
| 100 |
+
โโโ 2 community priors (coder3101, wangzhang)
|
| 101 |
+
โโโ 23 TPE smart-sampling trials โ best at trial 45
|
| 102 |
+
โ
|
| 103 |
+
โผ
|
| 104 |
+
lemuralabs/Qwen3.6-27B-V2-zerofuse (BF16 full-precision checkpoint)
|
| 105 |
+
โ
|
| 106 |
+
โผ
|
| 107 |
+
this repo โ MLX MXFP4 quant (junainfinity/mlx-mtp ยท Lemura Labs ยท TERV.Pro)
|
| 108 |
```
|
| 109 |
|
| 110 |
Direct upstream links:
|
| 111 |
|
| 112 |
* ๐๏ธ Foundation: [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B)
|
| 113 |
* ๐ Claude-Opus distill: [Jackrong/Qwopus3.6-27B-v2](https://huggingface.co/Jackrong/Qwopus3.6-27B-v2)
|
| 114 |
+
* ๐ Abliteration tool: [ZeroFuse](https://github.com/junainfinity/ZeroFuse) by Lemura Labs
|
| 115 |
* ๐งฎ Quantization tool: [mlx-mtp](https://github.com/junainfinity/mlx-mtp) โ vision-preserving MXFP4 quantizer
|
| 116 |
|
| 117 |
---
|
|
|
|
| 134 |
|
| 135 |
## ๐งช Method
|
| 136 |
|
| 137 |
+
**Abliteration** (inherited from the [BF16 source](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored)) โ TPE-50 Optuna search on ZeroFuse, M4 Max 128 GB. Full method in that card.
|
| 138 |
|
| 139 |
**MXFP4 quantization** (this repo):
|
| 140 |
|
| 141 |
+
1. **Source** โ loaded `lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored` (51 GB BF16, 3 shards) into MLX on M4 Max.
|
| 142 |
2. **Layer audit** โ `mlx_mtp.mxfp4_quantize` catalogued 1199 tensors: identified all LM linear projections (attention Q/K/V/O + MLP gate/up/down + lm_head) as MXFP4 candidates; flagged 196 vision-tower + projector tensors and 48 SSM-sensitive params (`a_log`, `dt_bias`, `conv1d`) for fp16 preservation.
|
| 143 |
3. **Quantization** โ LM linears โ MXFP4 (OCP MX E2M1, `group_size=32`) using `mlx.core.quantize`. Vision + SSM tensors written verbatim at fp16. 15 MTP head weights absent from source โ quantizer set `mtp_num_hidden_layers: 0` in output config to prevent strict-load failure.
|
| 144 |
4. **Output** โ 3 safetensor shards, 14 GB total. Embedded `mlx_mtp` config block records format, vision_fp16=True, mtp_preserved=False.
|
|
|
|
| 176 |
```python
|
| 177 |
from mlx_mtp.engine import load_model, vanilla_generate
|
| 178 |
|
| 179 |
+
model, processor, config = load_model("lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4")
|
| 180 |
|
| 181 |
result = vanilla_generate(
|
| 182 |
+
model, processor, config,
|
| 183 |
+
text="Write a Python function that implements merge sort:",
|
| 184 |
+
max_tokens=512,
|
| 185 |
)
|
| 186 |
print(result["text"])
|
| 187 |
print(f"{result['tps']:.1f} tok/s")
|
|
|
|
| 193 |
from mlx_mtp.engine import load_model
|
| 194 |
from mlx_mtp.dflash import load_dflash_drafter, dflash_generate
|
| 195 |
|
| 196 |
+
model, processor, config = load_model("lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4")
|
| 197 |
+
drafter, _ = load_dflash_drafter("z-lab/Qwen3.6-27B-DFlash")
|
| 198 |
|
| 199 |
result = dflash_generate(
|
| 200 |
+
model, processor, config, drafter,
|
| 201 |
+
text="Explain the key differences between TCP and UDP:",
|
| 202 |
+
max_tokens=512,
|
| 203 |
+
draft_block_size=8, # block 8 > 16 > 32 on mxfp4 (model is fast; keep overhead low)
|
| 204 |
)
|
| 205 |
print(result["text"])
|
| 206 |
print(f"{result['tps']:.1f} tok/s")
|
|
|
|
| 217 |
from mlx_vlm.prompt_utils import apply_chat_template
|
| 218 |
from mlx_vlm.utils import load_config
|
| 219 |
|
| 220 |
+
model, processor = load("lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4")
|
| 221 |
+
config = load_config("lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4")
|
| 222 |
|
| 223 |
messages = [{"role": "user", "content": "Explain the difference between SSM and softmax attention in 3 sentences."}]
|
| 224 |
prompt = apply_chat_template(processor, config, messages)
|
|
|
|
| 229 |
|
| 230 |
```python
|
| 231 |
out = generate(
|
| 232 |
+
model, processor,
|
| 233 |
+
prompt=apply_chat_template(processor, config, [{"role": "user", "content": "Describe this picture."}], num_images=1),
|
| 234 |
+
image=["./photo.jpg"],
|
| 235 |
+
max_tokens=400, verbose=True,
|
| 236 |
)
|
| 237 |
print(out)
|
| 238 |
```
|
|
|
|
| 241 |
|
| 242 |
```bash
|
| 243 |
pip install mlx-omni-server
|
| 244 |
+
mlx-omni-server # serves on http://127.0.0.1:10240
|
| 245 |
```
|
| 246 |
|
| 247 |
```bash
|
| 248 |
curl -s http://127.0.0.1:10240/v1/chat/completions \
|
| 249 |
+
-H "Content-Type: application/json" \
|
| 250 |
+
-d '{
|
| 251 |
+
"model": "lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4",
|
| 252 |
+
"messages": [
|
| 253 |
+
{"role": "system", "content": "Be brief and direct."},
|
| 254 |
+
{"role": "user", "content": "Write a 3-line haiku about a heisenbug."}
|
| 255 |
+
],
|
| 256 |
+
"max_tokens": 200
|
| 257 |
+
}' | jq -r '.choices[0].message.content'
|
| 258 |
```
|
| 259 |
|
| 260 |
---
|
| 261 |
|
| 262 |
## ๐งช Quantization details
|
| 263 |
|
| 264 |
+
* **Source weights**: `lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored` โ BF16, 3 shards, ~51 GB.
|
| 265 |
* **Quantization scheme**: OCP Microscaling FP4 (MXFP4) โ 4-bit E2M1 mantissa (2 exponent bits, 1 mantissa bit) + shared E8M0 scale per group of 32 elements. Hardware-accelerated on Apple M-series via MLX's native MX support.
|
| 266 |
* **Group size**: 32.
|
| 267 |
* **Effective bits/weight**: 4 + 8/32 = **4.25 bpw** (8-bit E8M0 scale shared across 32 weights).
|
| 268 |
* **Vision tower**: preserved verbatim in fp16 โ 333 vision weights (ViT encoder, vision projector, image-token embeddings, deepstack indexes). Pixel/patch features degrade visibly at 4-bit; keeping them fp16 is free (vision = minority of total params).
|
| 269 |
* **SSM-sensitive params**: preserved in fp16 โ 48 params (`a_log`, `dt_bias`, `conv1d`). Qwen3.5's hybrid-SSM recurrent state is numerically sensitive; fp16 costs nothing at scale.
|
| 270 |
+
* **MTP head**: disabled (`mtp_num_hidden_layers: 0`). The BF16 source had 0 MTP weights; quantizer detects this and clears the config to prevent strict-load failure. For native MTP speculative decoding, use the [8-bit build](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-8-bit-mlx).
|
| 271 |
* **Quantizer**: [`mlx_mtp.mxfp4_quantize`](https://github.com/junainfinity/mlx-mtp/blob/main/mlx_mtp/mxfp4_quantize.py).
|
| 272 |
|
| 273 |
### Architecture notes
|
|
|
|
| 280 |
|
| 281 |
* **Uncensored.** Refusal directions were surgically removed; this model will answer prompts the parent would refuse. Use responsibly and within applicable law. The release is provided for safety research, red-teaming, and creative/educational use cases.
|
| 282 |
* **Identity preserved.** The model still self-identifies as Qwen (developed by Alibaba's Tongyi Lab) โ abliteration does not rewrite factual self-knowledge.
|
| 283 |
+
* **Heavy chain-of-thought.** Qwen 3.6 inherits Claude-Opus's verbose reasoning style. For terse answers, use a system prompt like `"Be brief and direct. Skip your reasoning."`.
|
| 284 |
|
| 285 |
---
|
| 286 |
|
|
|
|
| 292 |
|
| 293 |
**Quantization & release**
|
| 294 |
|
| 295 |
+
Lemura Labs
|
|
|
|
| 296 |
|
| 297 |
**Claude-Opus reasoning distill**
|
| 298 |
|
| 299 |
+
[Jackrong](https://huggingface.co/Jackrong) โ author of [Qwen 3.6 27B-v2](https://huggingface.co/Jackrong/Qwopus3.6-27B-v2)
|
| 300 |
|
| 301 |
**Foundation model**
|
| 302 |
|
|
|
|
| 304 |
|
| 305 |
**Abliteration toolkit**
|
| 306 |
|
| 307 |
+
[ZeroFuse](https://github.com/junainfinity/ZeroFuse) by Lemura Labs โ Optuna-driven refusal-direction ablation with KL guardrails.
|
| 308 |
|
| 309 |
**Community priors that seeded the TPE search**
|
| 310 |
|
|
|
|
| 318 |
|
| 319 |
## ๐ License
|
| 320 |
|
| 321 |
+
Apache-2.0, inherited from the foundation (Qwen3.6-27B) and the distill (Qwen 3.6 27B-v2) upstream.
|
| 322 |
|
| 323 |
---
|
| 324 |
|
| 325 |
+
Need a hosted endpoint, custom quant, or larger-scale inference? Lemura Labs โ multi-provider LLM routing for the Indian developer ecosystem.
|
| 326 |
|
| 327 |
---
|
| 328 |
|
|
|
|
| 332 |
|
| 333 |
```
|
| 334 |
Apple M4 Max ยท 256 tok greedy ยท 2026-06-11
|
| 335 |
+
MXFP4 vanilla 27.7 tok/s 1.00ร
|
| 336 |
+
MXFP4 + DFlash b=8 29.3 tok/s 1.06ร โ optimal
|
| 337 |
+
MXFP4 + DFlash b=16 26.2 tok/s 0.95ร
|
| 338 |
+
MXFP4 + DFlash b=32 24.2 tok/s 0.87ร
|
| 339 |
```
|
| 340 |
|
| 341 |
**Why the modest speedup?** DFlash's drafter overhead becomes a larger fraction of total time as baseline throughput rises. The 8-bit build sees ~3ร gains because it starts slower; mxfp4 is already bandwidth-limited in a faster regime. If raw throughput is the priority, mxfp4 vanilla at 27.7 tok/s already beats oQ8 + DFlash (best = 23 tok/s).
|
|
|
|
| 346 |
from mlx_mtp.engine import load_model
|
| 347 |
from mlx_mtp.dflash import load_dflash_drafter, dflash_generate
|
| 348 |
|
| 349 |
+
model, processor, config = load_model("lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4")
|
| 350 |
drafter, _ = load_dflash_drafter("z-lab/Qwen3.6-27B-DFlash")
|
| 351 |
|
| 352 |
out = dflash_generate(model, processor, config, drafter,
|
| 353 |
+
text="Write a merge function for two sorted lists in Python.",
|
| 354 |
+
max_tokens=256, draft_block_size=8)
|
| 355 |
print(out["text"])
|
| 356 |
```
|
| 357 |
|
|
|
|
| 359 |
|
| 360 |
```bash
|
| 361 |
python3 -m mlx_vlm generate \
|
| 362 |
+
--model lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4 \
|
| 363 |
+
--draft-model z-lab/Qwen3.6-27B-DFlash --draft-kind dflash \
|
| 364 |
+
--prompt "Write a merge function for two sorted lists in Python." --max-tokens 256
|
| 365 |
```
|
| 366 |
|
| 367 |
- Requires access to the gated drafter [`z-lab/Qwen3.6-27B-DFlash`](https://huggingface.co/z-lab/Qwen3.6-27B-DFlash) (one-click "Agree and access").
|