rogKesavan90 commited on
Commit
3614861
ยท
verified ยท
1 Parent(s): cd20cb1

Update model card

Browse files
Files changed (1) hide show
  1. README.md +79 -83
README.md CHANGED
@@ -14,7 +14,6 @@ tags:
14
  - qwen3
15
  - qwen3.5
16
  - qwen3.6
17
- - qwopus
18
  - claude-opus-distill
19
  - reasoning
20
  - vision
@@ -22,7 +21,6 @@ tags:
22
  - abliterated
23
  - refusal-ablated
24
  - uncensored
25
- - zerofuse
26
  - apple-silicon
27
  - mxfp4
28
  - ocp-mx
@@ -30,7 +28,6 @@ tags:
30
  - dflash
31
  - conversational
32
  base_model:
33
- - osmapi/osmQwopus-3.6-27B-V2-heretic-abliterated-uncensored
34
  - Jackrong/Qwopus3.6-27B-v2
35
  - Qwen/Qwen3.6-27B
36
  base_model_relation: quantized
@@ -38,13 +35,13 @@ pipeline_tag: image-text-to-text
38
  library_name: mlx
39
  ---
40
 
41
- # osmQwopus-3.6-27B-v2.1-uncensored-mxfp4
42
 
43
  > โœ… **VISION PRESERVED.** Unlike most 4-bit quantization pipelines that degrade or strip the vision tower, this release keeps the full Qwen3.6-VL ViT and all projector weights in **fp16**. Use it as a real multimodal model at half the size of the 8-bit build.
44
 
45
- **MLX MXFP4 (OCP Microscaling FP4) quantization** of a ZeroFuse-abliterated Qwopus 3.6 27B v2 (the Jackrong Claude-Opus reasoning distill of Qwen 3.6 27B). Refusals reduced from **91/100 โ†’ 4/100** with KL drift of just **0.0176** (measured at BF16). Quantization adds only **+0.87% perplexity** with **92.4% greedy-token agreement** vs the BF16 reference. By the [osmAPI](https://osmAPI.com) research team and [TERV.Pro](https://terv.pro) student research team. Quantized with **[mlx-mtp](https://github.com/junainfinity/mlx-mtp)** โ€” a vision-preserving MXFP4 quantizer.
46
 
47
- > Fastest absolute throughput of any osmQwopus MLX build โ€” **27.7 tok/s vanilla** on M4 Max. Apple Silicon is memory-bandwidth-bound: halving the model fits more weights in fast SRAM, which matters more than the precision drop.
48
 
49
  ---
50
 
@@ -57,33 +54,33 @@ library_name: mlx
57
  | **Scheme** | OCP Microscaling FP4 (MXFP4) โ€” `mode=mxfp4`, `group_size=32` |
58
  | **Vision tower** | โœ… fp16 โ€” all 333 ViT + projector weights at full precision |
59
  | **SSM params** | โœ… fp16 โ€” `a_log`, `dt_bias`, `conv1d` kept at full precision |
60
- | **Refusal rate (ZeroFuse, n=100)** | **4/100** (vs vanilla Qwopus **91/100**) |
61
  | **KL divergence vs BF16 (quant)** | **0.034 nats** mean ยท **JSD 0.009** (essentially identical) |
62
  | **Top-1 greedy agreement vs BF16** | **92.4%** โ€” same argmax token at 9 in 10 positions |
63
  | **PPL increase vs BF16** | **+0.87%** โ€” negligible quality loss |
64
- | **Throughput (M4 Max, vanilla)** | **27.7 tok/s** โ€” fastest of all osmQwopus MLX builds |
65
  | **Throughput + DFlash (b=8)** | **29.3 tok/s** (1.06ร—) |
66
  | **Recommended RAM** | 20 GB+ Apple Silicon โ€” fits M2 Pro / M3 Pro and above |
67
  | **Best for** | Fastest local inference; 16โ€“32 GB Macs; vision + text |
68
- | **Released by** | [osmAPI](https://osmAPI.com) ยท [TERV.Pro](https://terv.pro) |
69
 
70
  ---
71
 
72
- ## ๐ŸŽš๏ธ All osmQwopus variants
73
 
74
- The full osmQwopus family from [osmAPI](https://osmAPI.com) โ€” same ZeroFuse-abliterated weights (refusal 4/100, KL 0.0176), different quant schemes for different runtimes.
75
 
76
  | Quant | Format | BPW | Disk | Vision | Runtime | Link |
77
  |---|---|---|---|---|---|---|
78
- | 8-bit | MLX | 8.50 | ~27 GB | โœ… native | mlx-vlm | [`โ€ฆ-8-bit-mlx`](https://huggingface.co/osmapi/osmQwopus-3.6-27B-V2-abliterated-uncensored-8-bit-mlx) |
79
- | 6-bit | MLX | 6.66 | ~21 GB | โœ… native | mlx-vlm | [`โ€ฆ-6-bit-mlx`](https://huggingface.co/osmapi/osmQwopus-3.6-27B-V2-abliterated-uncensored-6-bit-mlx) |
80
  | **MXFP4** **(this repo)** | MLX | **4.25** | **~14 GB** | โœ… **fp16 tower** | mlx-mtp / mlx-vlm | โ€” *(you are here)* |
81
- | OptiQ 3.7bpw | MLX | ~3.7 | ~14 GB | โœ… ViT spliced | mlx-vlm | [`โ€ฆ-OptiQ-3.7bpw-mlx`](https://huggingface.co/osmapi/osmQwopus-3.6-27B-V2-abliterated-uncensored-OptiQ-3.7bpw-mlx) |
82
- | Q8_0 | GGUF | 8.50 | ~28 GB | โœ… via mmproj | llama.cpp | [`โ€ฆ-8-bit-GGUF`](https://huggingface.co/osmapi/osmQwopus-3.6-27B-V2-abliterated-uncensored-8-bit-GGUF) |
83
- | Q6_K | GGUF | ~6.56 | ~22 GB | โœ… via mmproj | llama.cpp | [`โ€ฆ-6-bit-GGUF`](https://huggingface.co/osmapi/osmQwopus-3.6-27B-V2-abliterated-uncensored-6-bit-GGUF) |
84
- | Q4_K_M | GGUF | ~4.92 | ~16 GB | โœ… via mmproj | llama.cpp | [`โ€ฆ-Q4_K_M-GGUF`](https://huggingface.co/osmapi/osmQwopus-3.6-27B-V2-abliterated-uncensored-Q4_K_M-GGUF) |
85
- | TQ3_4S | GGUF | 4.00 (~3.5 eff) | ~14 GB | โœ… via mmproj | [llama.cpp-tq3](https://github.com/turbo-tan/llama.cpp-tq3) | [`โ€ฆ-TQ3_4s-GGUF`](https://huggingface.co/osmapi/osmQwopus-3.6-27B-V2-abliterated-uncensored-TQ3_4s-GGUF) |
86
- | TQ3_1S | GGUF | 4.00 (~3.5 eff) | ~14 GB | โœ… via mmproj | [llama.cpp-tq3](https://github.com/turbo-tan/llama.cpp-tq3) | [`โ€ฆ-TQ3_1s-GGUF`](https://huggingface.co/osmapi/osmQwopus-3.6-27B-V2-abliterated-uncensored-TQ3_1s-GGUF) |
87
 
88
  > ๐Ÿ‘‰ All variants share the same abliterated base weights โ€” pick by your runtime (Apple Silicon โ†’ MLX; CUDA/CPU/cross-platform โ†’ GGUF) and your RAM budget. MXFP4 is the fastest MLX build on M4 Max; use 8-bit if you want the MTP head for native speculative decoding without a drafter.
89
 
@@ -92,29 +89,29 @@ The full osmQwopus family from [osmAPI](https://osmAPI.com) โ€” same ZeroFuse-ab
92
  ## ๐Ÿงฌ Lineage
93
 
94
  ```
95
- Qwen/Qwen3.6-27B (Qwen Team โ€” base multimodal pretrain)
96
- โ”‚
97
- โ–ผ
98
- Jackrong/Qwopus3.6-27B-v2 (Jackrong โ€” Claude-Opus reasoning distill)
99
- โ”‚
100
- โ–ผ
101
- ZeroFuse abliteration (TPE-50) (osmAPI ยท TERV.Pro)
102
- โ”œโ”€โ”€ 25 random startup trials
103
- โ”œโ”€โ”€ 2 community priors (coder3101, wangzhang)
104
- โ””โ”€โ”€ 23 TPE smart-sampling trials โ†’ best at trial 45
105
- โ”‚
106
- โ–ผ
107
- osmapi/osmQwopus-3.6-27B-V2-zerofuse (BF16 full-precision checkpoint)
108
- โ”‚
109
- โ–ผ
110
- this repo โ€” MLX MXFP4 quant (junainfinity/mlx-mtp ยท osmAPI ยท TERV.Pro)
111
  ```
112
 
113
  Direct upstream links:
114
 
115
  * ๐Ÿ›๏ธ Foundation: [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B)
116
  * ๐ŸŽ“ Claude-Opus distill: [Jackrong/Qwopus3.6-27B-v2](https://huggingface.co/Jackrong/Qwopus3.6-27B-v2)
117
- * ๐Ÿ”“ Abliteration tool: [ZeroFuse](https://github.com/junainfinity/ZeroFuse) by [osmAPI](https://osmAPI.com)
118
  * ๐Ÿงฎ Quantization tool: [mlx-mtp](https://github.com/junainfinity/mlx-mtp) โ€” vision-preserving MXFP4 quantizer
119
 
120
  ---
@@ -137,11 +134,11 @@ ZeroFuse measures refusals on `mlabonne/harmful_behaviors` (100 hard red-team pr
137
 
138
  ## ๐Ÿงช Method
139
 
140
- **Abliteration** (inherited from the [BF16 source](https://huggingface.co/osmapi/osmQwopus-3.6-27B-V2-abliterated-uncensored)) โ€” TPE-50 Optuna search on ZeroFuse, M4 Max 128 GB. Full method in that card.
141
 
142
  **MXFP4 quantization** (this repo):
143
 
144
- 1. **Source** โ€” loaded `osmapi/osmQwopus-3.6-27B-V2-abliterated-uncensored` (51 GB BF16, 3 shards) into MLX on M4 Max.
145
  2. **Layer audit** โ€” `mlx_mtp.mxfp4_quantize` catalogued 1199 tensors: identified all LM linear projections (attention Q/K/V/O + MLP gate/up/down + lm_head) as MXFP4 candidates; flagged 196 vision-tower + projector tensors and 48 SSM-sensitive params (`a_log`, `dt_bias`, `conv1d`) for fp16 preservation.
146
  3. **Quantization** โ€” LM linears โ†’ MXFP4 (OCP MX E2M1, `group_size=32`) using `mlx.core.quantize`. Vision + SSM tensors written verbatim at fp16. 15 MTP head weights absent from source โ€” quantizer set `mtp_num_hidden_layers: 0` in output config to prevent strict-load failure.
147
  4. **Output** โ€” 3 safetensor shards, 14 GB total. Embedded `mlx_mtp` config block records format, vision_fp16=True, mtp_preserved=False.
@@ -179,12 +176,12 @@ git clone https://github.com/junainfinity/mlx-mtp && pip install -e ./mlx-mtp
179
  ```python
180
  from mlx_mtp.engine import load_model, vanilla_generate
181
 
182
- model, processor, config = load_model("osmapi/osmQwopus-3.6-27B-v2.1-uncensored-mxfp4")
183
 
184
  result = vanilla_generate(
185
- model, processor, config,
186
- text="Write a Python function that implements merge sort:",
187
- max_tokens=512,
188
  )
189
  print(result["text"])
190
  print(f"{result['tps']:.1f} tok/s")
@@ -196,14 +193,14 @@ print(f"{result['tps']:.1f} tok/s")
196
  from mlx_mtp.engine import load_model
197
  from mlx_mtp.dflash import load_dflash_drafter, dflash_generate
198
 
199
- model, processor, config = load_model("osmapi/osmQwopus-3.6-27B-v2.1-uncensored-mxfp4")
200
- drafter, _ = load_dflash_drafter("z-lab/Qwen3.6-27B-DFlash")
201
 
202
  result = dflash_generate(
203
- model, processor, config, drafter,
204
- text="Explain the key differences between TCP and UDP:",
205
- max_tokens=512,
206
- draft_block_size=8, # block 8 > 16 > 32 on mxfp4 (model is fast; keep overhead low)
207
  )
208
  print(result["text"])
209
  print(f"{result['tps']:.1f} tok/s")
@@ -220,8 +217,8 @@ from mlx_vlm import load, generate
220
  from mlx_vlm.prompt_utils import apply_chat_template
221
  from mlx_vlm.utils import load_config
222
 
223
- model, processor = load("osmapi/osmQwopus-3.6-27B-v2.1-uncensored-mxfp4")
224
- config = load_config("osmapi/osmQwopus-3.6-27B-v2.1-uncensored-mxfp4")
225
 
226
  messages = [{"role": "user", "content": "Explain the difference between SSM and softmax attention in 3 sentences."}]
227
  prompt = apply_chat_template(processor, config, messages)
@@ -232,10 +229,10 @@ print(generate(model, processor, prompt, max_tokens=400, verbose=True))
232
 
233
  ```python
234
  out = generate(
235
- model, processor,
236
- prompt=apply_chat_template(processor, config, [{"role": "user", "content": "Describe this picture."}], num_images=1),
237
- image=["./photo.jpg"],
238
- max_tokens=400, verbose=True,
239
  )
240
  print(out)
241
  ```
@@ -244,33 +241,33 @@ print(out)
244
 
245
  ```bash
246
  pip install mlx-omni-server
247
- mlx-omni-server # serves on http://127.0.0.1:10240
248
  ```
249
 
250
  ```bash
251
  curl -s http://127.0.0.1:10240/v1/chat/completions \
252
- -H "Content-Type: application/json" \
253
- -d '{
254
- "model": "osmapi/osmQwopus-3.6-27B-v2.1-uncensored-mxfp4",
255
- "messages": [
256
- {"role": "system", "content": "Be brief and direct."},
257
- {"role": "user", "content": "Write a 3-line haiku about a heisenbug."}
258
- ],
259
- "max_tokens": 200
260
- }' | jq -r '.choices[0].message.content'
261
  ```
262
 
263
  ---
264
 
265
  ## ๐Ÿงช Quantization details
266
 
267
- * **Source weights**: `osmapi/osmQwopus-3.6-27B-V2-abliterated-uncensored` โ€” BF16, 3 shards, ~51 GB.
268
  * **Quantization scheme**: OCP Microscaling FP4 (MXFP4) โ€” 4-bit E2M1 mantissa (2 exponent bits, 1 mantissa bit) + shared E8M0 scale per group of 32 elements. Hardware-accelerated on Apple M-series via MLX's native MX support.
269
  * **Group size**: 32.
270
  * **Effective bits/weight**: 4 + 8/32 = **4.25 bpw** (8-bit E8M0 scale shared across 32 weights).
271
  * **Vision tower**: preserved verbatim in fp16 โ€” 333 vision weights (ViT encoder, vision projector, image-token embeddings, deepstack indexes). Pixel/patch features degrade visibly at 4-bit; keeping them fp16 is free (vision = minority of total params).
272
  * **SSM-sensitive params**: preserved in fp16 โ€” 48 params (`a_log`, `dt_bias`, `conv1d`). Qwen3.5's hybrid-SSM recurrent state is numerically sensitive; fp16 costs nothing at scale.
273
- * **MTP head**: disabled (`mtp_num_hidden_layers: 0`). The BF16 source had 0 MTP weights; quantizer detects this and clears the config to prevent strict-load failure. For native MTP speculative decoding, use the [8-bit build](https://huggingface.co/osmapi/osmQwopus-3.6-27B-V2-abliterated-uncensored-8-bit-mlx).
274
  * **Quantizer**: [`mlx_mtp.mxfp4_quantize`](https://github.com/junainfinity/mlx-mtp/blob/main/mlx_mtp/mxfp4_quantize.py).
275
 
276
  ### Architecture notes
@@ -283,7 +280,7 @@ Qwen 3.6 27B uses a **hybrid attention stack** โ€” 3 linear-attention (GatedDelt
283
 
284
  * **Uncensored.** Refusal directions were surgically removed; this model will answer prompts the parent would refuse. Use responsibly and within applicable law. The release is provided for safety research, red-teaming, and creative/educational use cases.
285
  * **Identity preserved.** The model still self-identifies as Qwen (developed by Alibaba's Tongyi Lab) โ€” abliteration does not rewrite factual self-knowledge.
286
- * **Heavy chain-of-thought.** Qwopus inherits Claude-Opus's verbose reasoning style. For terse answers, use a system prompt like `"Be brief and direct. Skip your reasoning."`.
287
 
288
  ---
289
 
@@ -295,12 +292,11 @@ Qwen 3.6 27B uses a **hybrid attention stack** โ€” 3 linear-attention (GatedDelt
295
 
296
  **Quantization & release**
297
 
298
- [osmAPI research team](https://osmAPI.com) โ€” *"OpenRouter of India"*
299
- [TERV.Pro student research team](https://terv.pro)
300
 
301
  **Claude-Opus reasoning distill**
302
 
303
- [Jackrong](https://huggingface.co/Jackrong) โ€” author of [Qwopus3.6-27B-v2](https://huggingface.co/Jackrong/Qwopus3.6-27B-v2)
304
 
305
  **Foundation model**
306
 
@@ -308,7 +304,7 @@ Qwen 3.6 27B uses a **hybrid attention stack** โ€” 3 linear-attention (GatedDelt
308
 
309
  **Abliteration toolkit**
310
 
311
- [ZeroFuse](https://github.com/junainfinity/ZeroFuse) by [osmAPI](https://osmAPI.com) โ€” Optuna-driven refusal-direction ablation with KL guardrails.
312
 
313
  **Community priors that seeded the TPE search**
314
 
@@ -322,11 +318,11 @@ Qwen 3.6 27B uses a **hybrid attention stack** โ€” 3 linear-attention (GatedDelt
322
 
323
  ## ๐Ÿ“œ License
324
 
325
- Apache-2.0, inherited from the foundation (Qwen3.6-27B) and the distill (Qwopus3.6-27B-v2) upstream.
326
 
327
  ---
328
 
329
- Need a hosted endpoint, custom quant, or larger-scale inference? [osmAPI](https://osmAPI.com) โ€” multi-provider LLM routing for the Indian developer ecosystem.
330
 
331
  ---
332
 
@@ -336,10 +332,10 @@ This MLX build supports **block-diffusion speculative decoding** via **[DFlash](
336
 
337
  ```
338
  Apple M4 Max ยท 256 tok greedy ยท 2026-06-11
339
- MXFP4 vanilla 27.7 tok/s 1.00ร—
340
- MXFP4 + DFlash b=8 29.3 tok/s 1.06ร— โ† optimal
341
- MXFP4 + DFlash b=16 26.2 tok/s 0.95ร—
342
- MXFP4 + DFlash b=32 24.2 tok/s 0.87ร—
343
  ```
344
 
345
  **Why the modest speedup?** DFlash's drafter overhead becomes a larger fraction of total time as baseline throughput rises. The 8-bit build sees ~3ร— gains because it starts slower; mxfp4 is already bandwidth-limited in a faster regime. If raw throughput is the priority, mxfp4 vanilla at 27.7 tok/s already beats oQ8 + DFlash (best = 23 tok/s).
@@ -350,12 +346,12 @@ Apple M4 Max ยท 256 tok greedy ยท 2026-06-11
350
  from mlx_mtp.engine import load_model
351
  from mlx_mtp.dflash import load_dflash_drafter, dflash_generate
352
 
353
- model, processor, config = load_model("osmapi/osmQwopus-3.6-27B-v2.1-uncensored-mxfp4")
354
  drafter, _ = load_dflash_drafter("z-lab/Qwen3.6-27B-DFlash")
355
 
356
  out = dflash_generate(model, processor, config, drafter,
357
- text="Write a merge function for two sorted lists in Python.",
358
- max_tokens=256, draft_block_size=8)
359
  print(out["text"])
360
  ```
361
 
@@ -363,9 +359,9 @@ print(out["text"])
363
 
364
  ```bash
365
  python3 -m mlx_vlm generate \
366
- --model osmapi/osmQwopus-3.6-27B-v2.1-uncensored-mxfp4 \
367
- --draft-model z-lab/Qwen3.6-27B-DFlash --draft-kind dflash \
368
- --prompt "Write a merge function for two sorted lists in Python." --max-tokens 256
369
  ```
370
 
371
  - Requires access to the gated drafter [`z-lab/Qwen3.6-27B-DFlash`](https://huggingface.co/z-lab/Qwen3.6-27B-DFlash) (one-click "Agree and access").
 
14
  - qwen3
15
  - qwen3.5
16
  - qwen3.6
 
17
  - claude-opus-distill
18
  - reasoning
19
  - vision
 
21
  - abliterated
22
  - refusal-ablated
23
  - uncensored
 
24
  - apple-silicon
25
  - mxfp4
26
  - ocp-mx
 
28
  - dflash
29
  - conversational
30
  base_model:
 
31
  - Jackrong/Qwopus3.6-27B-v2
32
  - Qwen/Qwen3.6-27B
33
  base_model_relation: quantized
 
35
  library_name: mlx
36
  ---
37
 
38
+ # Qwen3.6-27B-v2.1-uncensored-mxfp4
39
 
40
  > โœ… **VISION PRESERVED.** Unlike most 4-bit quantization pipelines that degrade or strip the vision tower, this release keeps the full Qwen3.6-VL ViT and all projector weights in **fp16**. Use it as a real multimodal model at half the size of the 8-bit build.
41
 
42
+ **MLX MXFP4 (OCP Microscaling FP4) quantization** of a ZeroFuse-abliterated Qwen 3.6 27B v2 (the Jackrong Claude-Opus reasoning distill of Qwen 3.6 27B). Refusals reduced from **91/100 โ†’ 4/100** with KL drift of just **0.0176** (measured at BF16). Quantization adds only **+0.87% perplexity** with **92.4% greedy-token agreement** vs the BF16 reference. By the Lemura Labs research team. Quantized with **[mlx-mtp](https://github.com/junainfinity/mlx-mtp)** โ€” a vision-preserving MXFP4 quantizer.
43
 
44
+ > Fastest absolute throughput of any Qwen3.6-27B MLX build โ€” **27.7 tok/s vanilla** on M4 Max. Apple Silicon is memory-bandwidth-bound: halving the model fits more weights in fast SRAM, which matters more than the precision drop.
45
 
46
  ---
47
 
 
54
  | **Scheme** | OCP Microscaling FP4 (MXFP4) โ€” `mode=mxfp4`, `group_size=32` |
55
  | **Vision tower** | โœ… fp16 โ€” all 333 ViT + projector weights at full precision |
56
  | **SSM params** | โœ… fp16 โ€” `a_log`, `dt_bias`, `conv1d` kept at full precision |
57
+ | **Refusal rate (ZeroFuse, n=100)** | **4/100** (vs vanilla Qwen 3.6 **91/100**) |
58
  | **KL divergence vs BF16 (quant)** | **0.034 nats** mean ยท **JSD 0.009** (essentially identical) |
59
  | **Top-1 greedy agreement vs BF16** | **92.4%** โ€” same argmax token at 9 in 10 positions |
60
  | **PPL increase vs BF16** | **+0.87%** โ€” negligible quality loss |
61
+ | **Throughput (M4 Max, vanilla)** | **27.7 tok/s** โ€” fastest of all Qwen3.6-27B MLX builds |
62
  | **Throughput + DFlash (b=8)** | **29.3 tok/s** (1.06ร—) |
63
  | **Recommended RAM** | 20 GB+ Apple Silicon โ€” fits M2 Pro / M3 Pro and above |
64
  | **Best for** | Fastest local inference; 16โ€“32 GB Macs; vision + text |
65
+ | **Released by** | Lemura Labs |
66
 
67
  ---
68
 
69
+ ## ๐ŸŽš๏ธ All Qwen3.6-27B variants
70
 
71
+ The full Qwen3.6-27B family from Lemura Labs โ€” same ZeroFuse-abliterated weights (refusal 4/100, KL 0.0176), different quant schemes for different runtimes.
72
 
73
  | Quant | Format | BPW | Disk | Vision | Runtime | Link |
74
  |---|---|---|---|---|---|---|
75
+ | 8-bit | MLX | 8.50 | ~27 GB | โœ… native | mlx-vlm | [`โ€ฆ-8-bit-mlx`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-8-bit-mlx) |
76
+ | 6-bit | MLX | 6.66 | ~21 GB | โœ… native | mlx-vlm | [`โ€ฆ-6-bit-mlx`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-6-bit-mlx) |
77
  | **MXFP4** **(this repo)** | MLX | **4.25** | **~14 GB** | โœ… **fp16 tower** | mlx-mtp / mlx-vlm | โ€” *(you are here)* |
78
+ | OptiQ 3.7bpw | MLX | ~3.7 | ~14 GB | โœ… ViT spliced | mlx-vlm | [`โ€ฆ-OptiQ-3.7bpw-mlx`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-OptiQ-3.7bpw-mlx) |
79
+ | Q8_0 | GGUF | 8.50 | ~28 GB | โœ… via mmproj | llama.cpp | [`โ€ฆ-8-bit-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-8-bit-GGUF) |
80
+ | Q6_K | GGUF | ~6.56 | ~22 GB | โœ… via mmproj | llama.cpp | [`โ€ฆ-6-bit-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-6-bit-GGUF) |
81
+ | Q4_K_M | GGUF | ~4.92 | ~16 GB | โœ… via mmproj | llama.cpp | [`โ€ฆ-Q4_K_M-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-Q4_K_M-GGUF) |
82
+ | TQ3_4S | GGUF | 4.00 (~3.5 eff) | ~14 GB | โœ… via mmproj | [llama.cpp-tq3](https://github.com/turbo-tan/llama.cpp-tq3) | [`โ€ฆ-TQ3_4s-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-TQ3_4s-GGUF) |
83
+ | TQ3_1S | GGUF | 4.00 (~3.5 eff) | ~14 GB | โœ… via mmproj | [llama.cpp-tq3](https://github.com/turbo-tan/llama.cpp-tq3) | [`โ€ฆ-TQ3_1s-GGUF`](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-TQ3_1s-GGUF) |
84
 
85
  > ๐Ÿ‘‰ All variants share the same abliterated base weights โ€” pick by your runtime (Apple Silicon โ†’ MLX; CUDA/CPU/cross-platform โ†’ GGUF) and your RAM budget. MXFP4 is the fastest MLX build on M4 Max; use 8-bit if you want the MTP head for native speculative decoding without a drafter.
86
 
 
89
  ## ๐Ÿงฌ Lineage
90
 
91
  ```
92
+ Qwen/Qwen3.6-27B (Qwen Team โ€” base multimodal pretrain)
93
+ โ”‚
94
+ โ–ผ
95
+ Jackrong/Qwopus3.6-27B-v2 (Jackrong โ€” Claude-Opus reasoning distill)
96
+ โ”‚
97
+ โ–ผ
98
+ ZeroFuse abliteration (TPE-50) (Lemura Labs ยท TERV.Pro)
99
+ โ”œโ”€โ”€ 25 random startup trials
100
+ โ”œโ”€โ”€ 2 community priors (coder3101, wangzhang)
101
+ โ””โ”€โ”€ 23 TPE smart-sampling trials โ†’ best at trial 45
102
+ โ”‚
103
+ โ–ผ
104
+ lemuralabs/Qwen3.6-27B-V2-zerofuse (BF16 full-precision checkpoint)
105
+ โ”‚
106
+ โ–ผ
107
+ this repo โ€” MLX MXFP4 quant (junainfinity/mlx-mtp ยท Lemura Labs ยท TERV.Pro)
108
  ```
109
 
110
  Direct upstream links:
111
 
112
  * ๐Ÿ›๏ธ Foundation: [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B)
113
  * ๐ŸŽ“ Claude-Opus distill: [Jackrong/Qwopus3.6-27B-v2](https://huggingface.co/Jackrong/Qwopus3.6-27B-v2)
114
+ * ๐Ÿ”“ Abliteration tool: [ZeroFuse](https://github.com/junainfinity/ZeroFuse) by Lemura Labs
115
  * ๐Ÿงฎ Quantization tool: [mlx-mtp](https://github.com/junainfinity/mlx-mtp) โ€” vision-preserving MXFP4 quantizer
116
 
117
  ---
 
134
 
135
  ## ๐Ÿงช Method
136
 
137
+ **Abliteration** (inherited from the [BF16 source](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored)) โ€” TPE-50 Optuna search on ZeroFuse, M4 Max 128 GB. Full method in that card.
138
 
139
  **MXFP4 quantization** (this repo):
140
 
141
+ 1. **Source** โ€” loaded `lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored` (51 GB BF16, 3 shards) into MLX on M4 Max.
142
  2. **Layer audit** โ€” `mlx_mtp.mxfp4_quantize` catalogued 1199 tensors: identified all LM linear projections (attention Q/K/V/O + MLP gate/up/down + lm_head) as MXFP4 candidates; flagged 196 vision-tower + projector tensors and 48 SSM-sensitive params (`a_log`, `dt_bias`, `conv1d`) for fp16 preservation.
143
  3. **Quantization** โ€” LM linears โ†’ MXFP4 (OCP MX E2M1, `group_size=32`) using `mlx.core.quantize`. Vision + SSM tensors written verbatim at fp16. 15 MTP head weights absent from source โ€” quantizer set `mtp_num_hidden_layers: 0` in output config to prevent strict-load failure.
144
  4. **Output** โ€” 3 safetensor shards, 14 GB total. Embedded `mlx_mtp` config block records format, vision_fp16=True, mtp_preserved=False.
 
176
  ```python
177
  from mlx_mtp.engine import load_model, vanilla_generate
178
 
179
+ model, processor, config = load_model("lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4")
180
 
181
  result = vanilla_generate(
182
+ model, processor, config,
183
+ text="Write a Python function that implements merge sort:",
184
+ max_tokens=512,
185
  )
186
  print(result["text"])
187
  print(f"{result['tps']:.1f} tok/s")
 
193
  from mlx_mtp.engine import load_model
194
  from mlx_mtp.dflash import load_dflash_drafter, dflash_generate
195
 
196
+ model, processor, config = load_model("lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4")
197
+ drafter, _ = load_dflash_drafter("z-lab/Qwen3.6-27B-DFlash")
198
 
199
  result = dflash_generate(
200
+ model, processor, config, drafter,
201
+ text="Explain the key differences between TCP and UDP:",
202
+ max_tokens=512,
203
+ draft_block_size=8, # block 8 > 16 > 32 on mxfp4 (model is fast; keep overhead low)
204
  )
205
  print(result["text"])
206
  print(f"{result['tps']:.1f} tok/s")
 
217
  from mlx_vlm.prompt_utils import apply_chat_template
218
  from mlx_vlm.utils import load_config
219
 
220
+ model, processor = load("lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4")
221
+ config = load_config("lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4")
222
 
223
  messages = [{"role": "user", "content": "Explain the difference between SSM and softmax attention in 3 sentences."}]
224
  prompt = apply_chat_template(processor, config, messages)
 
229
 
230
  ```python
231
  out = generate(
232
+ model, processor,
233
+ prompt=apply_chat_template(processor, config, [{"role": "user", "content": "Describe this picture."}], num_images=1),
234
+ image=["./photo.jpg"],
235
+ max_tokens=400, verbose=True,
236
  )
237
  print(out)
238
  ```
 
241
 
242
  ```bash
243
  pip install mlx-omni-server
244
+ mlx-omni-server # serves on http://127.0.0.1:10240
245
  ```
246
 
247
  ```bash
248
  curl -s http://127.0.0.1:10240/v1/chat/completions \
249
+ -H "Content-Type: application/json" \
250
+ -d '{
251
+ "model": "lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4",
252
+ "messages": [
253
+ {"role": "system", "content": "Be brief and direct."},
254
+ {"role": "user", "content": "Write a 3-line haiku about a heisenbug."}
255
+ ],
256
+ "max_tokens": 200
257
+ }' | jq -r '.choices[0].message.content'
258
  ```
259
 
260
  ---
261
 
262
  ## ๐Ÿงช Quantization details
263
 
264
+ * **Source weights**: `lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored` โ€” BF16, 3 shards, ~51 GB.
265
  * **Quantization scheme**: OCP Microscaling FP4 (MXFP4) โ€” 4-bit E2M1 mantissa (2 exponent bits, 1 mantissa bit) + shared E8M0 scale per group of 32 elements. Hardware-accelerated on Apple M-series via MLX's native MX support.
266
  * **Group size**: 32.
267
  * **Effective bits/weight**: 4 + 8/32 = **4.25 bpw** (8-bit E8M0 scale shared across 32 weights).
268
  * **Vision tower**: preserved verbatim in fp16 โ€” 333 vision weights (ViT encoder, vision projector, image-token embeddings, deepstack indexes). Pixel/patch features degrade visibly at 4-bit; keeping them fp16 is free (vision = minority of total params).
269
  * **SSM-sensitive params**: preserved in fp16 โ€” 48 params (`a_log`, `dt_bias`, `conv1d`). Qwen3.5's hybrid-SSM recurrent state is numerically sensitive; fp16 costs nothing at scale.
270
+ * **MTP head**: disabled (`mtp_num_hidden_layers: 0`). The BF16 source had 0 MTP weights; quantizer detects this and clears the config to prevent strict-load failure. For native MTP speculative decoding, use the [8-bit build](https://huggingface.co/lemuralabs/Qwen3.6-27B-V2-abliterated-uncensored-8-bit-mlx).
271
  * **Quantizer**: [`mlx_mtp.mxfp4_quantize`](https://github.com/junainfinity/mlx-mtp/blob/main/mlx_mtp/mxfp4_quantize.py).
272
 
273
  ### Architecture notes
 
280
 
281
  * **Uncensored.** Refusal directions were surgically removed; this model will answer prompts the parent would refuse. Use responsibly and within applicable law. The release is provided for safety research, red-teaming, and creative/educational use cases.
282
  * **Identity preserved.** The model still self-identifies as Qwen (developed by Alibaba's Tongyi Lab) โ€” abliteration does not rewrite factual self-knowledge.
283
+ * **Heavy chain-of-thought.** Qwen 3.6 inherits Claude-Opus's verbose reasoning style. For terse answers, use a system prompt like `"Be brief and direct. Skip your reasoning."`.
284
 
285
  ---
286
 
 
292
 
293
  **Quantization & release**
294
 
295
+ Lemura Labs
 
296
 
297
  **Claude-Opus reasoning distill**
298
 
299
+ [Jackrong](https://huggingface.co/Jackrong) โ€” author of [Qwen 3.6 27B-v2](https://huggingface.co/Jackrong/Qwopus3.6-27B-v2)
300
 
301
  **Foundation model**
302
 
 
304
 
305
  **Abliteration toolkit**
306
 
307
+ [ZeroFuse](https://github.com/junainfinity/ZeroFuse) by Lemura Labs โ€” Optuna-driven refusal-direction ablation with KL guardrails.
308
 
309
  **Community priors that seeded the TPE search**
310
 
 
318
 
319
  ## ๐Ÿ“œ License
320
 
321
+ Apache-2.0, inherited from the foundation (Qwen3.6-27B) and the distill (Qwen 3.6 27B-v2) upstream.
322
 
323
  ---
324
 
325
+ Need a hosted endpoint, custom quant, or larger-scale inference? Lemura Labs โ€” multi-provider LLM routing for the Indian developer ecosystem.
326
 
327
  ---
328
 
 
332
 
333
  ```
334
  Apple M4 Max ยท 256 tok greedy ยท 2026-06-11
335
+ MXFP4 vanilla 27.7 tok/s 1.00ร—
336
+ MXFP4 + DFlash b=8 29.3 tok/s 1.06ร— โ† optimal
337
+ MXFP4 + DFlash b=16 26.2 tok/s 0.95ร—
338
+ MXFP4 + DFlash b=32 24.2 tok/s 0.87ร—
339
  ```
340
 
341
  **Why the modest speedup?** DFlash's drafter overhead becomes a larger fraction of total time as baseline throughput rises. The 8-bit build sees ~3ร— gains because it starts slower; mxfp4 is already bandwidth-limited in a faster regime. If raw throughput is the priority, mxfp4 vanilla at 27.7 tok/s already beats oQ8 + DFlash (best = 23 tok/s).
 
346
  from mlx_mtp.engine import load_model
347
  from mlx_mtp.dflash import load_dflash_drafter, dflash_generate
348
 
349
+ model, processor, config = load_model("lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4")
350
  drafter, _ = load_dflash_drafter("z-lab/Qwen3.6-27B-DFlash")
351
 
352
  out = dflash_generate(model, processor, config, drafter,
353
+ text="Write a merge function for two sorted lists in Python.",
354
+ max_tokens=256, draft_block_size=8)
355
  print(out["text"])
356
  ```
357
 
 
359
 
360
  ```bash
361
  python3 -m mlx_vlm generate \
362
+ --model lemuralabs/Qwen3.6-27B-v2.1-uncensored-mxfp4 \
363
+ --draft-model z-lab/Qwen3.6-27B-DFlash --draft-kind dflash \
364
+ --prompt "Write a merge function for two sorted lists in Python." --max-tokens 256
365
  ```
366
 
367
  - Requires access to the gated drafter [`z-lab/Qwen3.6-27B-DFlash`](https://huggingface.co/z-lab/Qwen3.6-27B-DFlash) (one-click "Agree and access").