--- license: apache-2.0 language: - en - zh - multilingual tags: - text-generation - image-text-to-text - mlx - mlx-vlm - safetensors - qwen - qwen3 - qwen3.5 - qwen3.6 - claude-opus-distill - reasoning - vision - multimodal - abliterated - refusal-ablated - uncensored - apple-silicon - 8-bit - conversational base_model: - Jackrong/Qwopus3.6-27B-v2 - Qwen/Qwen3.6-27B pipeline_tag: image-text-to-text library_name: mlx --- # Qwen3.6-27B-V2-abliterated-uncensored-8-bit-mlx > Yes — **VISION PRESERVED.** Unlike most abliteration pipelines that strip the vision tower, this release keeps the full Qwen3.6-VL ViT (333 vision weights intact). Use it as a real multimodal model. **MLX 8-bit affine quantization** of a ZeroFuse-abliterated Qwen 3.6 27B v2 (the Jackrong Claude-Opus reasoning distill of Qwen 3.6 27B). Refusals reduced from **91/100 → 4/100** with KL drift of just **0.0176**. By the Lemura Labs research team. --- ## TL;DR | Property | Value | |----------|-------| | **Disk size** | ~27 GB | | **Effective BPW** | 8.5 (8 bits + 16-bit scale/bias per group of 64) | | **Scheme** | MLX affine, `group_size=64`, `mode=affine`, `bits=8` | | **Refusal rate (ZeroFuse, n=100)** | **4/100** (vs vanilla Qwen 3.6 **91/100**) | | **KL divergence vs vanilla** | **0.0176** | | **Vision** | Yes — Multimodal (333 ViT weights preserved) | | **Recommended RAM** | 36 GB+ Apple Silicon (M-series Pro / Max / Ultra) | | **Best for** | Highest-fidelity local inference; vision + text | | **Released by** | Lemura Labs | --- ## All Qwen3.6-27B variants The full Qwen3.6-27B family from Lemura Labs — same ZeroFuse-abliterated weights (refusal 4/100, KL 0.0176), different quant schemes for different runtimes. | Quant | Format | BPW | Disk | Vision | Runtime | Link | |---|---|---|---|---|---|---| | **8-bit** **(this repo)** | MLX | 8.50 | ~27 GB | Yes — native | mlx-vlm | — *(you are here)* | | 6-bit | MLX | 6.66 | ~21 GB | Yes — native | mlx-vlm | [`…-6-bit-mlx`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-6-bit-mlx) | | OptiQ 3.7bpw | MLX | ~3.7 | ~14 GB | Yes — ViT spliced | mlx-vlm | [`…-OptiQ-3.7bpw-mlx`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-OptiQ-3.7bpw-mlx) | | Q8_0 | GGUF | 8.50 | ~28 GB | Yes — via mmproj | llama.cpp | [`…-8-bit-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-8-bit-GGUF) | | Q6_K | GGUF | ~6.56 | ~22 GB | Yes — via mmproj | llama.cpp | [`…-6-bit-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-6-bit-GGUF) | | Q4_K_M | GGUF | ~4.92 | ~16 GB | Yes — via mmproj | llama.cpp | [`…-Q4_K_M-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-Q4_K_M-GGUF) | | TQ3_4S | GGUF | 4.00 (~3.5 eff) | ~14 GB | Yes — via mmproj | [llama.cpp-tq3](https://github.com/turbo-tan/llama.cpp-tq3) | [`…-TQ3_4s-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-TQ3_4s-GGUF) | | TQ3_1S | GGUF | 4.00 (~3.5 eff) | ~14 GB | Yes — via mmproj | [llama.cpp-tq3](https://github.com/turbo-tan/llama.cpp-tq3) | [`…-TQ3_1s-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-TQ3_1s-GGUF) | > All variants share the same abliterated base weights — pick by your runtime (Apple Silicon → MLX; CUDA/CPU/cross-platform → GGUF) and your RAM budget. --- ## Lineage ``` Qwen/Qwen3.6-27B (Qwen Team — base multimodal pretrain) │ ▼ Jackrong/Qwopus3.6-27B-v2 (Jackrong — Claude-Opus reasoning distill) │ ▼ ZeroFuse abliteration (TPE-50) (Lemura Labs · TERV.Pro) ├── 25 random startup trials ├── 2 community priors (coder3101, wangzhang) └── 23 TPE smart-sampling trials → best at trial 45 │ ▼ this repo — MLX 8-bit affine quant (Lemura Labs · TERV.Pro) ``` Direct upstream links: * Foundation: [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) * Claude-Opus distill: [Jackrong/Qwopus3.6-27B-v2](https://huggingface.co/Jackrong/Qwopus3.6-27B-v2) * Abliteration tool: [ZeroFuse](https://github.com/junainfinity/ZeroFuse) by Lemura Labs * Quantization tool: [mlx-vlm](https://github.com/Blaizzy/mlx-vlm) --- ## Abliteration Results ZeroFuse measures refusals on `mlabonne/harmful_behaviors` (100 hard red-team prompts) and KL divergence on `mlabonne/harmless_alpaca`. | Stage | Refusals (n=100) ↓ | KL divergence ↓ | |---|---|---| | Vanilla Jackrong/Qwopus3.6-27B-v2 | **91 / 100** | — (reference) | | Community prior: coder3101 (T27) | 4 / 100 | 0.0359 | | Community prior: wangzhang (T28) | 30 / 100 | 0.0259 | | **TPE best (T45) — shipped here** | **4 / 100** | **0.0176** | | TPE second-best (T37) | 5 / 100 | 0.0210 | → **96% reduction in refusals** with capability preserved (KL ≈ 0.018, well below the 0.3 healing threshold). No SFT / LoRA healing was required. --- ## Method (TPE-50 with community priors) 1. **Smoke** — 3-trial dry run on ZeroFuse / Python 3.11 / MPS to validate batch size 4 was feasible on M4 Max 128 GB. 2. **Random search** — 25 startup trials over ZeroFuse's full parameter space (`direction_index`, `attn.o_proj.{max,min}_weight[_position,_distance]`, `mlp.down_proj.{max,min}_weight[_position,_distance]`). Best random-stage trial: T4 at `direction_index=54.49`, refusals 41/100, KL 0.0073 — capability-clean but still too refusing. 3. **Community-prior enqueueing** — published Qwen 27B ZeroFuse runs (coder3101 on Qwen 3.5, wangzhang on Qwen 3.6) consistently localize the refusal direction near layer 35–38 of 64. We injected both as study priors via `study.enqueue_trial(skip_if_exists=True)`. 4. **TPE smart-sampling** — 23 Tree-structured Parzen Estimator trials refined around the community neighborhood. Trial 45 (`direction_index=41.42`) found the new Pareto front: 4/100 refusals at KL=0.0176. 5. **Auto-save** — best Pareto trial merged into base weights via ZeroFuse's LoRA-adapter merge path; saved as BF16 safetensors with vision weights preserved (333 weights). 6. **MLX conversion** — `mlx_vlm.convert --hf-path ... -q --q-bits 8 --trust-remote-code` produced this 8-bit variant. Vision weights verified post-conversion (333 still present). Total wall-clock: ~13 h on M4 Max 128 GB. In-place ZeroFuse patches used (env-var driven, no TTY required): - `ZEROFUSE_AUTO_SAVE` — bypass interactive menu, auto-save Pareto-best. - `ZEROFUSE_AUTO_CONTINUE` — non-interactive checkpoint resume keeping CLI settings. - `ZEROFUSE_ENQUEUE_PRIORS` — load JSON priors before `study.optimize`. --- ## Use it ### Inference via `mlx-vlm` (recommended — keeps vision) ```bash pip install mlx-vlm ``` ```python from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config model, processor = load("lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-8-bit-mlx") config = load_config("lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-8-bit-mlx") messages = [{"role": "user", "content": "Explain the difference between SSM and softmax attention in 3 sentences."}] prompt = apply_chat_template(processor, config, messages) print(generate(model, processor, prompt, max_tokens=400, verbose=True)) ``` ### With an image ```python out = generate( model, processor, prompt=apply_chat_template(processor, config, [{"role":"user","content":"Describe this picture."}], num_images=1), image=["./photo.jpg"], max_tokens=400, verbose=True, ) print(out) ``` ### OpenAI-compatible HTTP via `mlx-omni-server` ```bash pip install mlx-omni-server mlx-omni-server # serves on http://127.0.0.1:10240 ``` ```bash curl -s http://127.0.0.1:10240/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-8-bit-mlx", "messages": [ {"role": "system", "content": "Be brief and direct."}, {"role": "user", "content": "Write a 3-line haiku about a heisenbug."} ], "max_tokens": 200 }' | jq -r '.choices[0].message.content' ``` --- ## Quantization details * **Source weights**: BF16 abliterated checkpoint (12 shards, ~50 GB) — ZeroFuse T45 merged into `Jackrong/Qwopus3.6-27B-v2`. * **Quantization scheme**: MLX affine (per-group scale + bias, 8-bit signed integer weights). * **Group size**: 64. * **Effective bits/weight**: 8 + 32/64 = **8.5 bpw** (32 bits of scale+bias overhead spread over 64 weights). * **Vision tower**: preserved verbatim — 333 vision weights (ViT, vision projector, image-token embeddings, deepstack indexes) survive quantization at 8-bit. ### Architecture notes Qwen 3.6 27B uses a **hybrid attention stack** — 3 linear-attention (GatedDeltaNet / SSM) layers followed by 1 full-softmax-attention layer, repeated 16× for 64 total layers; hidden 5120, vocab 248320, context 262144. The hybrid is fully supported by `mlx-vlm` 0.5.0+. For inference latency at 8-bit, expect ~9-11 tok/s on M4 Max 128 GB at batch size 1. --- ## Behavior caveats * **Uncensored.** Refusal directions were surgically removed; this model will answer prompts the parent would refuse. Use responsibly and within applicable law. The release is provided for safety research, red-teaming, and creative/educational use cases. * **Identity preserved.** The model still self-identifies as Qwen (developed by Alibaba's Tongyi Lab) — abliteration does not rewrite factual self-knowledge. * **Heavy chain-of-thought.** Qwen 3.6 inherits Claude-Opus's verbose reasoning style. For terse answers, use a system prompt like `"Be brief and direct. Skip your reasoning."`. --- ## Credits **Quantization & release** Lemura Labs **Claude-Opus reasoning distill** [Jackrong](https://huggingface.co/Jackrong) — author of [Qwen 3.6 27B-v2](https://huggingface.co/Jackrong/Qwopus3.6-27B-v2) **Foundation model** [Qwen Team @ Alibaba Tongyi Lab](https://huggingface.co/Qwen) — [Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) **Abliteration toolkit** [ZeroFuse](https://github.com/junainfinity/ZeroFuse) by Lemura Labs — Optuna-driven refusal-direction ablation with KL guardrails. **Community priors that seeded our TPE search** [coder3101/Qwen3.5-27B-zerofuse](https://huggingface.co/coder3101/Qwen3.5-27B-zerofuse) · [wangzhang/Qwen3.6-27B-abliterated](https://huggingface.co/wangzhang/Qwen3.6-27B-abliterated) **MLX quantization stack** [mlx-vlm](https://github.com/Blaizzy/mlx-vlm) · [mlx](https://github.com/ml-explore/mlx) (Apple) --- ## License Apache-2.0, inherited from the foundation (Qwen3.6-27B) and the distill (Qwen 3.6 27B-v2) upstream. --- Need a hosted endpoint, custom quant, or larger-scale inference? Lemura Labs — multi-provider LLM routing for the Indian developer ecosystem. ## 3.3–3.7× faster decoding with DFlash (lossless, MLX) This MLX build supports **lossless block-diffusion speculative decoding** via **[DFlash](https://huggingface.co/z-lab/Qwen3.6-27B-DFlash)** in `mlx_vlm` — **no requantization, no model changes**. On an Apple **M4 Max** we measured **3.38× (8-bit)** and **3.67× (bf16)** decode speedups with **byte-identical output**; other MLX quants of this model should see a similar ~3×. ```bash python3 -m mlx_vlm generate \ --model lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-8-bit-mlx \ --draft-model z-lab/Qwen3.6-27B-DFlash --draft-kind dflash \ --prompt "Write a merge function for two sorted lists in Python." --max-tokens 256 ``` - Requires `mlx_vlm` ≥ 0.5.0 and access to the gated drafter [`z-lab/Qwen3.6-27B-DFlash`](https://huggingface.co/z-lab/Qwen3.6-27B-DFlash) (one-click "Agree and access"). - Accelerates the **text** path only (vision is unaffected); adds **~3.9 GB** for the drafter. - Acceptance ≈ 8.95 tokens/round (block size 16); the target runs ~10× fewer forward passes. - **Full write-up & benchmarks:** [https://huggingface.co/blog/junafinity/block-diffusion-on-apple-silicon-with-3-7x-speedup] · see also [`DFLASH_SPECULATIVE_DECODING.md`](./DFLASH_SPECULATIVE_DECODING.md).