README.md

metadata

license: apache-2.0
language:
  - en
  - zh
  - multilingual
tags:
  - text-generation
  - image-text-to-text
  - mlx
  - mlx-vlm
  - safetensors
  - qwen
  - qwen3
  - qwen3.5
  - qwen3.6
  - claude-opus-distill
  - reasoning
  - vision
  - multimodal
  - abliterated
  - refusal-ablated
  - uncensored
  - apple-silicon
  - mxfp4
  - ocp-mx
  - speculative-decoding
  - dflash
  - conversational
base_model:
  - Jackrong/Qwopus3.6-27B-v2
  - Qwen/Qwen3.6-27B
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: mlx

Qwen3.6-27B-v2.1-uncensored-mxfp4

✅ VISION PRESERVED. Unlike most 4-bit quantization pipelines that degrade or strip the vision tower, this release keeps the full Qwen3.6-VL ViT and all projector weights in fp16. Use it as a real multimodal model at half the size of the 8-bit build.

MLX MXFP4 (OCP Microscaling FP4) quantization of a ZeroFuse-abliterated Qwen 3.6 27B v2 (the Jackrong Claude-Opus reasoning distill of Qwen 3.6 27B). Refusals reduced from 91/100 → 4/100 with KL drift of just 0.0176 (measured at BF16). Quantization adds only +0.87% perplexity with 92.4% greedy-token agreement vs the BF16 reference. By the Lemura Labs research team. Quantized with mlx-mtp — a vision-preserving MXFP4 quantizer.

Fastest absolute throughput of any Qwen3.6-27B MLX build — 27.7 tok/s vanilla on M4 Max. Apple Silicon is memory-bandwidth-bound: halving the model fits more weights in fast SRAM, which matters more than the precision drop.

---

⚡ TL;DR

Property	Value
Disk size	~14 GB
Effective BPW	4.25 (4-bit E2M1 data + E8M0 scale per group of 32)
Scheme	OCP Microscaling FP4 (MXFP4) — mode=mxfp4, group_size=32
Vision tower	✅ fp16 — all 333 ViT + projector weights at full precision
SSM params	✅ fp16 — a_log, dt_bias, conv1d kept at full precision
Refusal rate (ZeroFuse, n=100)	4/100 (vs vanilla Qwen 3.6 91/100)
KL divergence vs BF16 (quant)	0.034 nats mean · JSD 0.009 (essentially identical)
Top-1 greedy agreement vs BF16	92.4% — same argmax token at 9 in 10 positions
PPL increase vs BF16	+0.87% — negligible quality loss
Throughput (M4 Max, vanilla)	27.7 tok/s — fastest of all Qwen3.6-27B MLX builds
Throughput + DFlash (b=8)	29.3 tok/s (1.06×)
Recommended RAM	20 GB+ Apple Silicon — fits M2 Pro / M3 Pro and above
Best for	Fastest local inference; 16–32 GB Macs; vision + text
Released by	Lemura Labs

---

🎚️ All Qwen3.6-27B variants

The full Qwen3.6-27B family from Lemura Labs — same ZeroFuse-abliterated weights (refusal 4/100, KL 0.0176), different quant schemes for different runtimes.

Quant	Format	BPW	Disk	Vision	Runtime	Link
8-bit	MLX	8.50	~27 GB	✅ native	mlx-vlm	…-8-bit-mlx
6-bit	MLX	6.66	~21 GB	✅ native	mlx-vlm	…-6-bit-mlx
MXFP4 (this repo)	MLX	4.25	~14 GB	✅ fp16 tower	mlx-mtp / mlx-vlm	— (you are here)
OptiQ 3.7bpw	MLX	~3.7	~14 GB	✅ ViT spliced	mlx-vlm	…-OptiQ-3.7bpw-mlx
Q8_0	GGUF	8.50	~28 GB	✅ via mmproj	llama.cpp	…-8-bit-GGUF
Q6_K	GGUF	~6.56	~22 GB	✅ via mmproj	llama.cpp	…-6-bit-GGUF
Q4_K_M	GGUF	~4.92	~16 GB	✅ via mmproj	llama.cpp	…-Q4_K_M-GGUF
TQ3_4S	GGUF	4.00 (~3.5 eff)	~14 GB	✅ via mmproj	llama.cpp-tq3	…-TQ3_4s-GGUF
TQ3_1S	GGUF	4.00 (~3.5 eff)	~14 GB	✅ via mmproj	llama.cpp-tq3	…-TQ3_1s-GGUF

👉 All variants share the same abliterated base weights — pick by your runtime (Apple Silicon → MLX; CUDA/CPU/cross-platform → GGUF) and your RAM budget. MXFP4 is the fastest MLX build on M4 Max; use 8-bit if you want the MTP head for native speculative decoding without a drafter.

---

🧬 Lineage

Qwen/Qwen3.6-27B (Qwen Team — base multimodal pretrain)
 │
 ▼
Jackrong/Qwopus3.6-27B-v2 (Jackrong — Claude-Opus reasoning distill)
 │
 ▼
ZeroFuse abliteration (TPE-50) (Lemura Labs · TERV.Pro)
 ├── 25 random startup trials
 ├── 2 community priors (coder3101, wangzhang)
 └── 23 TPE smart-sampling trials → best at trial 45
 │
 ▼
lemuralabs/Qwen3.6-27B-V2-zerofuse (BF16 full-precision checkpoint)
 │
 ▼
this repo — MLX MXFP4 quant (junainfinity/mlx-mtp · Lemura Labs · TERV.Pro)

Direct upstream links:

• 🏛️ Foundation: Qwen/Qwen3.6-27B
• 🎓 Claude-Opus distill: Jackrong/Qwopus3.6-27B-v2
• 🔓 Abliteration tool: ZeroFuse by Lemura Labs
• 🧮 Quantization tool: mlx-mtp — vision-preserving MXFP4 quantizer

---

📊 Abliteration Results

ZeroFuse measures refusals on mlabonne/harmful_behaviors (100 hard red-team prompts) and KL divergence on mlabonne/harmless_alpaca.

Stage	Refusals (n=100) ↓	KL divergence ↓
Vanilla Jackrong/Qwopus3.6-27B-v2	91 / 100	— (reference)
Community prior: coder3101 (T27)	4 / 100	0.0359
Community prior: wangzhang (T28)	30 / 100	0.0259
TPE best (T45) — BF16 source	4 / 100	0.0176
MXFP4 quant (this model)	4 / 100	0.034 nats vs BF16

→ 96% reduction in refusals with capability preserved at BF16. MXFP4 quantization adds only +0.87% perplexity on top — the abliteration and quantization quality loss are both negligible.

---

🧪 Method

Abliteration (inherited from the BF16 source) — TPE-50 Optuna search on ZeroFuse, M4 Max 128 GB. Full method in that card.

MXFP4 quantization (this repo):

1. Source — loaded lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored (51 GB BF16, 3 shards) into MLX on M4 Max.
2. Layer audit — mlx_mtp.mxfp4_quantize catalogued 1199 tensors: identified all LM linear projections (attention Q/K/V/O + MLP gate/up/down + lm_head) as MXFP4 candidates; flagged 196 vision-tower + projector tensors and 48 SSM-sensitive params (a_log, dt_bias, conv1d) for fp16 preservation.
3. Quantization — LM linears → MXFP4 (OCP MX E2M1, group_size=32) using mlx.core.quantize. Vision + SSM tensors written verbatim at fp16. 15 MTP head weights absent from source — quantizer set mtp_num_hidden_layers: 0 in output config to prevent strict-load failure.
4. Output — 3 safetensor shards, 14 GB total. Embedded mlx_mtp config block records format, vision_fp16=True, mtp_preserved=False.
5. Verification — text generation correct; vision captioning correct (image → caption matches BF16 output); DFlash block-diffusion speculative decoding loads and runs; KL divergence measured end-to-end (see Quality section below).

Total wall-clock: ~2 h on M4 Max 128 GB (dominated by disk I/O).

---

📊 Quantization quality vs BF16

Measured via teacher-forced prefill — 10 diverse prompts (code, math, reasoning, instruction-following), 244 token positions, full 248K-vocab distributions at every position.

Metric	Value	Interpretation
KL(bf16 ‖ mxfp4) mean	0.034 nats	Forward divergence from reference
JSD mean	0.009 nats	1.3% of [0, ln2] bound — essentially identical
Top-1 agreement	92.4%	Both pick same greedy token at 9 in 10 positions
PPL increase	+0.87%	Perplexity vs BF16 reference tokens
Code prompts (KL)	0.010–0.012	Lowest — code is deterministic

JSD < 0.01 is the standard "essentially identical distributions" threshold. At 0.009 we are below it. The 7.6% of positions where models disagree on the top-1 token are near-tie argmax flips from quantization noise — not quality regressions.

---

📦 Use it

Fastest inference via mlx-mtp (recommended — native DFlash support)

git clone https://github.com/jundot/omlx && pip install -e ./omlx
git clone https://github.com/junainfinity/mlx-mtp && pip install -e ./mlx-mtp

from mlx_mtp.engine import load_model, vanilla_generate

model, processor, config = load_model("lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4")

result = vanilla_generate(
 model, processor, config,
 text="Write a Python function that implements merge sort:",
 max_tokens=512,
)
print(result["text"])
print(f"{result['tps']:.1f} tok/s")

With DFlash speculative decoding (block size 8 is optimal for mxfp4)

from mlx_mtp.engine import load_model
from mlx_mtp.dflash import load_dflash_drafter, dflash_generate

model, processor, config = load_model("lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4")
drafter, _ = load_dflash_drafter("z-lab/Qwen3.6-27B-DFlash")

result = dflash_generate(
 model, processor, config, drafter,
 text="Explain the key differences between TCP and UDP:",
 max_tokens=512,
 draft_block_size=8, # block 8 > 16 > 32 on mxfp4 (model is fast; keep overhead low)
)
print(result["text"])
print(f"{result['tps']:.1f} tok/s")

Inference via mlx-vlm

pip install mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

model, processor = load("lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4")
config = load_config("lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4")

messages = [{"role": "user", "content": "Explain the difference between SSM and softmax attention in 3 sentences."}]
prompt = apply_chat_template(processor, config, messages)
print(generate(model, processor, prompt, max_tokens=400, verbose=True))

With an image

out = generate(
 model, processor,
 prompt=apply_chat_template(processor, config, [{"role": "user", "content": "Describe this picture."}], num_images=1),
 image=["./photo.jpg"],
 max_tokens=400, verbose=True,
)
print(out)

OpenAI-compatible HTTP via mlx-omni-server

pip install mlx-omni-server
mlx-omni-server # serves on http://127.0.0.1:10240

curl -s http://127.0.0.1:10240/v1/chat/completions \
 -H "Content-Type: application/json" \
 -d '{
 "model": "lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4",
 "messages": [
 {"role": "system", "content": "Be brief and direct."},
 {"role": "user", "content": "Write a 3-line haiku about a heisenbug."}
 ],
 "max_tokens": 200
 }' | jq -r '.choices[0].message.content'

---

🧪 Quantization details

• Source weights: lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored — BF16, 3 shards, ~51 GB.
• Quantization scheme: OCP Microscaling FP4 (MXFP4) — 4-bit E2M1 mantissa (2 exponent bits, 1 mantissa bit) + shared E8M0 scale per group of 32 elements. Hardware-accelerated on Apple M-series via MLX's native MX support.
• Group size: 32.
• Effective bits/weight: 4 + 8/32 = 4.25 bpw (8-bit E8M0 scale shared across 32 weights).
• Vision tower: preserved verbatim in fp16 — 333 vision weights (ViT encoder, vision projector, image-token embeddings, deepstack indexes). Pixel/patch features degrade visibly at 4-bit; keeping them fp16 is free (vision = minority of total params).
• SSM-sensitive params: preserved in fp16 — 48 params (a_log, dt_bias, conv1d). Qwen3.5's hybrid-SSM recurrent state is numerically sensitive; fp16 costs nothing at scale.
• MTP head: disabled (mtp_num_hidden_layers: 0). The BF16 source had 0 MTP weights; quantizer detects this and clears the config to prevent strict-load failure. For native MTP speculative decoding, use the 8-bit build.
• Quantizer: mlx_mtp.mxfp4_quantize.

Architecture notes

Qwen 3.6 27B uses a hybrid attention stack — 3 linear-attention (GatedDeltaNet / SSM) layers followed by 1 full-softmax-attention layer, repeated 16× for 64 total layers; hidden 5120, vocab 248320, context 262144. The hybrid is fully supported by mlx-vlm 0.5.0+. For inference latency at MXFP4, expect 27–30 tok/s on M4 Max 128 GB at batch size 1 — approximately 2× faster than the 8-bit build because Apple Silicon is memory-bandwidth-bound and the smaller model fits more weights in fast SRAM per cycle.

---

⚠️ Behavior caveats

• Uncensored. Refusal directions were surgically removed; this model will answer prompts the parent would refuse. Use responsibly and within applicable law. The release is provided for safety research, red-teaming, and creative/educational use cases.
• Identity preserved. The model still self-identifies as Qwen (developed by Alibaba's Tongyi Lab) — abliteration does not rewrite factual self-knowledge.
• Heavy chain-of-thought. Qwen 3.6 inherits Claude-Opus's verbose reasoning style. For terse answers, use a system prompt like "Be brief and direct. Skip your reasoning.".

---

🙏 Credits

MXFP4 quantization

junainfinity/mlx-mtp — vision-preserving MXFP4 + oQ8 quantizer + native MTP / DFlash speculative decode engine for Apple Silicon VLMs.

Quantization & release

Lemura Labs

Claude-Opus reasoning distill

Jackrong — author of Qwen 3.6 27B-v2

Foundation model

Qwen Team @ Alibaba Tongyi Lab — Qwen3.6-27B

Abliteration toolkit

ZeroFuse by Lemura Labs — Optuna-driven refusal-direction ablation with KL guardrails.

Community priors that seeded the TPE search

coder3101/Qwen3.5-27B-zerofuse · wangzhang/Qwen3.6-27B-abliterated

MLX stack

mlx-vlm · mlx (Apple)

---

📜 License

Apache-2.0, inherited from the foundation (Qwen3.6-27B) and the distill (Qwen 3.6 27B-v2) upstream.

---

Need a hosted endpoint, custom quant, or larger-scale inference? Lemura Labs — multi-provider LLM routing for the Indian developer ecosystem.

---

⚡ Faster decoding with DFlash (MLX, block-diffusion speculative)

This MLX build supports block-diffusion speculative decoding via DFlash — no requantization, no model changes. Because MXFP4 runs at ~27.7 tok/s baseline (already fast), DFlash gains are modest vs the 8-bit build; use block size 8 for the best result.

Apple M4 Max · 256 tok greedy · 2026-06-11
 MXFP4 vanilla 27.7 tok/s 1.00×
 MXFP4 + DFlash b=8 29.3 tok/s 1.06× ← optimal
 MXFP4 + DFlash b=16 26.2 tok/s 0.95×
 MXFP4 + DFlash b=32 24.2 tok/s 0.87×

Why the modest speedup? DFlash's drafter overhead becomes a larger fraction of total time as baseline throughput rises. The 8-bit build sees ~3× gains because it starts slower; mxfp4 is already bandwidth-limited in a faster regime. If raw throughput is the priority, mxfp4 vanilla at 27.7 tok/s already beats oQ8 + DFlash (best = 23 tok/s).

Via mlx-mtp (recommended — correct block-size selection):

from mlx_mtp.engine import load_model
from mlx_mtp.dflash import load_dflash_drafter, dflash_generate

model, processor, config = load_model("lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4")
drafter, _ = load_dflash_drafter("z-lab/Qwen3.6-27B-DFlash")

out = dflash_generate(model, processor, config, drafter,
 text="Write a merge function for two sorted lists in Python.",
 max_tokens=256, draft_block_size=8)
print(out["text"])

Via mlx-vlm CLI (standard interface):

python3 -m mlx_vlm generate \
 --model lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4 \
 --draft-model z-lab/Qwen3.6-27B-DFlash --draft-kind dflash \
 --prompt "Write a merge function for two sorted lists in Python." --max-tokens 256

• Requires access to the gated drafter z-lab/Qwen3.6-27B-DFlash (one-click "Agree and access").
• Accelerates the text path only (vision encoding is unaffected).
• Adds ~3.9 GB for the drafter.
• Full benchmark write-up: mlx-mtp.