--- license: apache-2.0 language: - en - zh - multilingual tags: - text-generation - image-text-to-text - mlx - mlx-vlm - safetensors - qwen - qwen3 - qwen3.5 - qwen3.6 - claude-opus-distill - reasoning - vision - multimodal - abliterated - refusal-ablated - uncensored - optiq - mixed-precision - apple-silicon - conversational base_model: - Jackrong/Qwopus3.6-27B-v2 - Qwen/Qwen3.6-27B pipeline_tag: image-text-to-text library_name: mlx --- # Qwen3.6-27B-V2-abliterated-uncensored-OptiQ-3.7bpw-mlx > Yes — **VISION PRESERVED — even at 3.7 BPW.** Most OptiQ releases strip the vision tower; we keep it. The LM portion is OptiQ static-mixed (3/4/8-bit per tensor), and the ViT is spliced back in at BF16 from our 8-bit MLX build, giving you a full multimodal model that fits in 14 GB. **OptiQ static mixed ~3.7 BPW MLX quantization** of a ZeroFuse-abliterated Qwen 3.6 27B v2 (the Jackrong Claude-Opus reasoning distill of Qwen 3.6 27B). Refusals reduced from **91/100 → 4/100** with KL drift of just **0.0176**. By the Lemura Labs research team. > Smallest variant in our line. The LM uses [mlx-optiq](https://pypi.org/project/mlx-optiq/) Frobenius-stream sensitivity scans to assign **3-bit** to layers that tolerate it and **4–8-bit** to the layers that don't, averaging **3.697 bits per weight**. The vision tower (333 ViT weights, ~880 MB at BF16) is spliced back in unmodified from the abliterated parent — the cleanest combination of language compression and visual fidelity. --- ## TL;DR | Property | Value | |----------|-------| | **Disk size** | ~14 GB | | **LM BPW (achieved)** | 3.697 (target 3.7) · 4.147 effective with scale/bias | | **LM scheme** | OptiQ static-mixed (sensitivity-ranked, Frobenius-stream weight-space) | | **LM bit allocation** | 223 × 8-bit, 210 × 4-bit, 149 × 3-bit, 0 × 2-bit (582 quantizable tensors) | | **Vision tower** | Yes — BF16, full ViT (333 weights, ~880 MB) | | **Refusal rate (ZeroFuse, n=100)** | **4/100** (vs vanilla Qwen 3.6 **91/100**) | | **KL divergence vs vanilla (at BF16)** | **0.0176** | | **Recommended RAM** | 16–24 GB Apple Silicon | | **Best for** | Tight RAM budgets · full multimodal · base-Mac inference | | **Released by** | Lemura Labs | --- ## All Qwen3.6-27B variants The full Qwen3.6-27B family from Lemura Labs — same ZeroFuse-abliterated weights (refusal 4/100, KL 0.0176), different quant schemes for different runtimes. | Quant | Format | BPW | Disk | Vision | Runtime | Link | |---|---|---|---|---|---|---| | 8-bit | MLX | 8.50 | ~27 GB | Yes — native | mlx-vlm | [`…-8-bit-mlx`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-8-bit-mlx) | | 6-bit | MLX | 6.66 | ~21 GB | Yes — native | mlx-vlm | [`…-6-bit-mlx`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-6-bit-mlx) | | **OptiQ 3.7bpw** **(this repo)** | MLX | ~3.7 | ~14 GB | Yes — ViT spliced | mlx-vlm | — *(you are here)* | | Q8_0 | GGUF | 8.50 | ~28 GB | Yes — via mmproj | llama.cpp | [`…-8-bit-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-8-bit-GGUF) | | Q6_K | GGUF | ~6.56 | ~22 GB | Yes — via mmproj | llama.cpp | [`…-6-bit-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-6-bit-GGUF) | | Q4_K_M | GGUF | ~4.92 | ~16 GB | Yes — via mmproj | llama.cpp | [`…-Q4_K_M-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-Q4_K_M-GGUF) | | TQ3_4S | GGUF | 4.00 (~3.5 eff) | ~14 GB | Yes — via mmproj | [llama.cpp-tq3](https://github.com/turbo-tan/llama.cpp-tq3) | [`…-TQ3_4s-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-TQ3_4s-GGUF) | | TQ3_1S | GGUF | 4.00 (~3.5 eff) | ~14 GB | Yes — via mmproj | [llama.cpp-tq3](https://github.com/turbo-tan/llama.cpp-tq3) | [`…-TQ3_1s-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-TQ3_1s-GGUF) | > All variants share the same abliterated base weights — pick by your runtime (Apple Silicon → MLX; CUDA/CPU/cross-platform → GGUF) and your RAM budget. --- ## Lineage ``` Qwen/Qwen3.6-27B (Qwen Team — base multimodal pretrain) │ ▼ Jackrong/Qwopus3.6-27B-v2 (Jackrong — Claude-Opus reasoning distill) │ ▼ ZeroFuse abliteration (TPE-50) (Lemura Labs · TERV.Pro) ├── 25 random startup trials ├── 2 community priors (coder3101, wangzhang) └── 23 TPE smart-sampling trials → best at trial 45 │ ▼ mlx-optiq static-mixed 3.7 BPW (LM only) (Lemura Labs · TERV.Pro) ├── Frobenius weight-space sensitivity (stream mode, fits 27B) └── Multi-tier bit allocation: 3/4/8-bit │ ▼ ViT spliced back in at BF16 from 8-bit build (Lemura Labs · TERV.Pro) ├── 333 ViT weights, ~880 MB total └── VLM config (Qwen3_5ForConditionalGeneration) restored │ ▼ this repo — OptiQ-3.7bpw multimodal MLX ``` Direct upstream links: * Foundation: [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) * Claude-Opus distill: [Jackrong/Qwopus3.6-27B-v2](https://huggingface.co/Jackrong/Qwopus3.6-27B-v2) * Abliteration tool: [ZeroFuse](https://github.com/junainfinity/ZeroFuse) by Lemura Labs * Quantization tool: [mlx-optiq](https://pypi.org/project/mlx-optiq/) by Thin Signal, on top of [mlx-lm](https://github.com/ml-explore/mlx-lm) --- ## Abliteration Results ZeroFuse measures refusals on `mlabonne/harmful_behaviors` (100 hard red-team prompts) and KL divergence on `mlabonne/harmless_alpaca`. These were measured at BF16 before quantization; the OptiQ-3.7bpw quant inherits them within calibration noise. | Stage | Refusals (n=100) ↓ | KL divergence ↓ | |---|---|---| | Vanilla Jackrong/Qwopus3.6-27B-v2 | **91 / 100** | — (reference) | | Community prior: coder3101 (T27) | 4 / 100 | 0.0359 | | Community prior: wangzhang (T28) | 30 / 100 | 0.0259 | | **TPE best (T45) — shipped here** | **4 / 100** | **0.0176** | | TPE second-best (T37) | 5 / 100 | 0.0210 | → **96% reduction in refusals** with capability preserved (KL ≈ 0.018, well below the 0.3 healing threshold). No SFT / LoRA healing was required. --- ## Method (TPE-50 with community priors → OptiQ-3.7 → vision merge) ### Step 1. Abliteration (ZeroFuse TPE-50) 1. **Smoke** — 3-trial dry run on ZeroFuse / Python 3.11 / MPS validated batch size 4 on M4 Max 128 GB. 2. **Random search** — 25 startup trials over ZeroFuse's parameter space (`direction_index`, `attn.o_proj.*`, `mlp.down_proj.*`). Best random-stage trial: T4 at `direction_index=54.49` — capability-clean but still too refusing (41/100). 3. **Community-prior enqueueing** — published Qwen 27B ZeroFuse runs (coder3101 on Qwen 3.5, wangzhang on Qwen 3.6) localized the refusal direction near layer 35–38 of 64. We injected both as study priors via `study.enqueue_trial(skip_if_exists=True)`. 4. **TPE smart-sampling** — 23 Tree-structured Parzen Estimator trials refined around the community neighborhood. **Trial 45** (`direction_index=41.42`) found the Pareto front: **4/100 refusals at KL=0.0176**. 5. **Auto-save** — best Pareto trial merged into base weights via ZeroFuse's LoRA-adapter merge path; saved as BF16 safetensors with vision tower fully intact. Total ZeroFuse wall-clock: ~13 h on M4 Max 128 GB. ### Step 2. LM mixed-precision quantization (OptiQ) 6. **Sensitivity analysis** — `optiq convert --target-bpw 3.7 --candidate-bits 2,3,4,8 --sensitivity stream` mmaps each tensor and computes per-tensor Frobenius distance between BF16 reference and candidate quantized versions. For a 27B model on 128 GB unified memory this is the only mode that fits. 7. **Multi-tier bit allocation** — OptiQ's optimizer assigns each of the 582 quantizable tensors a bit-width from `{3, 4, 8}` such that the average lands at the **3.7 BPW** target. Result: 38% × 8-bit (high-sensitivity), 36% × 4-bit (medium), 26% × 3-bit (low-sensitivity SSM projections and certain MLP gates). No tensor required 2-bit. 8. **MLX serialization** — quantized LM weights written via `mlx-lm` quantize hooks, with the per-tensor bit-width recorded in `config.json["quantization"]` for each module path. ### Step 3. Vision splice-back (custom) 9. **Why** — OptiQ defaults to stripping vision weights and rewriting `config.json` to a text-only `Qwen3_5ForCausalLM`. We wanted a multimodal release. The `--keep-unused-modalities` flag preserves the architecture string but not the weights, so we built a manual splice. 10. **What** — we copied the 333 vision tensors verbatim from our [companion 8-bit MLX build](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-8-bit-mlx) (where `mlx_vlm.convert` leaves the ViT at BF16) into a 4th safetensors shard, restored `vision_config`, `image_token_id`, `video_token_id`, `vision_start/end_token_id`, and the nested `text_config` block in `config.json`, and re-set `architectures` to `Qwen3_5ForConditionalGeneration`. 11. **Verification** — post-load norm weights at expected ~0.79 mean (no double-sanitize), text generation coherent, vision pipeline correctly reads test imagery. ### In-place ZeroFuse patches used - `ZEROFUSE_AUTO_SAVE` — bypass interactive menu, auto-save Pareto-best. - `ZEROFUSE_AUTO_CONTINUE` — non-interactive checkpoint resume keeping CLI settings. - `ZEROFUSE_ENQUEUE_PRIORS` — load JSON priors before `study.optimize`. --- ## Use it ### Inference via `mlx-vlm` (full multimodal) ```bash pip install mlx-vlm ``` ```python from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config model, processor = load("lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-OptiQ-3.7bpw-mlx") config = load_config("lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-OptiQ-3.7bpw-mlx") messages = [{"role": "user", "content": "Explain the difference between SSM and softmax attention in 3 sentences."}] prompt = apply_chat_template(processor, config, messages) print(generate(model, processor, prompt, max_tokens=400, verbose=True)) ``` ### With an image ```python out = generate( model, processor, prompt=apply_chat_template(processor, config, [{"role":"user","content":"Describe this image briefly."}], num_images=1), image=["./photo.jpg"], max_tokens=400, verbose=True, ) print(out) ``` ### OpenAI-compatible HTTP via `mlx-omni-server` ```bash pip install mlx-omni-server mlx-omni-server # serves on http://127.0.0.1:10240 ``` ```bash curl -s http://127.0.0.1:10240/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-OptiQ-3.7bpw-mlx", "messages": [ {"role": "system", "content": "Be brief and direct."}, {"role": "user", "content": "Write a 3-line haiku about a heisenbug."} ], "max_tokens": 200 }' | jq -r '.choices[0].message.content' ``` --- ## Quantization details * **Source weights**: BF16 abliterated checkpoint (12 shards, ~50 GB) — ZeroFuse T45 merged into `Jackrong/Qwopus3.6-27B-v2`. * **LM quantization tool**: [mlx-optiq](https://pypi.org/project/mlx-optiq/) v0.0.11 (`optiq convert --target-bpw 3.7 --candidate-bits 2,3,4,8 --sensitivity stream`). * **Achieved LM BPW**: 3.697 (target 3.7) — see `optiq_metadata.json` for the full per-tensor allocation table. * **Effective LM bits/weight**: 4.147 (with 16+16-bit scale+bias per group of 64 spread over the bit-width-weighted mean). * **Group size**: 64 throughout (LM). * **Vision tower**: spliced back at BF16 from the 8-bit MLX build (`mlx_vlm.convert` leaves ViTs unquantized by default; the 8-bit build's "8-bit" only refers to LM weights). 333 vision tensors, ~880 MB. * **Architecture class**: `Qwen3_5ForConditionalGeneration` (full multimodal restored). ### Per-tensor LM bit allocation (summary) | Bit-width | # tensors | % of total LM | |---|---|---| | 8-bit | 223 | 38.3% | | 4-bit | 210 | 36.1% | | 3-bit | 149 | 25.6% | | 2-bit | 0 | 0.0% | | **Total** | **582** | 100% | The full per-tensor allocation map lives in [`optiq_metadata.json`](./optiq_metadata.json). The vision tower (333 weights) does not appear in this table — it is unquantized BF16. ### Architecture notes Qwen 3.6 27B uses a **hybrid attention stack** — 3 linear-attention (GatedDeltaNet / SSM) layers followed by 1 full-softmax-attention layer, repeated 16× for 64 total layers; hidden 5120, vocab 248320, context 262144. The SSM kernels lack a VJP path in MLX, so backward-pass-based quant methods (DWQ, dynamic quant) **cannot** be applied to this architecture — OptiQ's forward-only Frobenius-stream sensitivity approach is the only calibration-aware option that works. For inference latency at 3.7 BPW LM + BF16 ViT, expect ~12–15 tok/s on M4 Max 128 GB at batch size 1, or ~5–7 tok/s on M2/M3 16 GB at batch size 1. --- ## Behavior caveats * **Uncensored.** Refusal directions were surgically removed; this model will answer prompts the parent would refuse. Use responsibly and within applicable law. The release is provided for safety research, red-teaming, and creative/educational use cases. * **Multimodal preserved.** Vision works — the 333 ViT weights are unmodified from the abliterated parent. The abliteration acts on the LM hidden states only. * **Identity preserved.** The model still self-identifies as Qwen (developed by Alibaba's Tongyi Lab) — abliteration does not rewrite factual self-knowledge. * **Heavy chain-of-thought.** Qwen 3.6 inherits Claude-Opus's verbose reasoning style. For terse answers, use a system prompt like `"Be brief and direct. Skip your reasoning."`. * **Quantization noise at 3-bit.** ~26% of LM tensors are at 3-bit. On long-generation tasks (>1000 tokens) you may notice slightly more drift than the 6/8-bit siblings. For best fidelity at this lineage, prefer 6-bit or 8-bit. --- ## Credits **Quantization & release** Lemura Labs **Claude-Opus reasoning distill** [Jackrong](https://huggingface.co/Jackrong) — author of [Qwen 3.6 27B-v2](https://huggingface.co/Jackrong/Qwopus3.6-27B-v2) **Foundation model** [Qwen Team @ Alibaba Tongyi Lab](https://huggingface.co/Qwen) — [Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) **Abliteration toolkit** [ZeroFuse](https://github.com/junainfinity/ZeroFuse) by Lemura Labs — Optuna-driven refusal-direction ablation with KL guardrails. **Community priors that seeded our TPE search** [coder3101/Qwen3.5-27B-zerofuse](https://huggingface.co/coder3101/Qwen3.5-27B-zerofuse) · [wangzhang/Qwen3.6-27B-abliterated](https://huggingface.co/wangzhang/Qwen3.6-27B-abliterated) **Mixed-precision quantization framework** [mlx-optiq](https://pypi.org/project/mlx-optiq/) by Thin Signal — sensitivity-driven multi-tier MLX quantization · built on [mlx-lm](https://github.com/ml-explore/mlx-lm) and [mlx](https://github.com/ml-explore/mlx) (Apple). --- ## License Apache-2.0, inherited from the foundation (Qwen3.6-27B) and the distill (Qwen 3.6 27B-v2) upstream. --- Need a hosted endpoint, custom quant, or larger-scale inference? Lemura Labs — multi-provider LLM routing for the Indian developer ecosystem.