--- license: apache-2.0 language: - en - zh - ja - es pipeline_tag: image-text-to-text library_name: transformers tags: - transformers - unsloth - qwen3_5 - qwen3_6 - qwen - qwen3.5 - qwen3.6 - distillation - reasoning - chain-of-thought - long-cot - sft - lora - instruction-tuned - conversational - text-generation - multilingual - math - stem - coding - research - experimental - merge - fable - opus - mythos - mergekit - mxfp8 - mlx base_model: - Qwen/Qwen3.6-35B-A3B - armand0e/Qwen3.6-35B-A3B-Fable-5-Distill - llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved - samuelcardillo/Qwopus-MoE-35B-A3B - Hcompany/Holo3-35B-A3B - nightmedia/Qwen3.6-35B-A3B-Fable-Holo3-Qwopus - nightmedia/Qwen3.6-35B-A3B-Holo3-Qwopus-BF16 - nightmedia/Qwen3.6-35B-A3B-MTP-Holo3-Qwopus-BF16 --- # Qwen3.6-35B-A3B-Fable-Holo3-Qwopus-mxfp8-mlx This model is a merge of: - armand0e/Qwen3.6-35B-A3B-Fable-5-Distill - nightmedia/Qwen3.6-35B-A3B-MTP-Holo3-Qwopus-BF16 Brainwaves ```brainwaves arc arc/e boolq hswag obkqa piqa wino bf16 0.631,0.819,0.892,0.775,0.460,0.816,0.717 mxfp8 0.637,0.824,0.897,0.778,0.448,0.820,0.729 qx86-hi 0.637,0.821,0.891,0.773,0.458,0.812,0.721 qx64-hi 0.656,0.829,0.892,0.775,0.452,0.820,0.722 mxfp4 0.631,0.827,0.888,0.773,0.444,0.816,0.712 Quant Perplexity Peak Memory Tokens/sec bf16 4.456 ± 0.029 76.15 GB 1644 mxfp8 4.693 ± 0.032 42.65 GB 1480 qx86-hi 4.475 ± 0.029 45.50 GB 1565 qx64-hi 4.438 ± 0.029 36.91 GB 1466 mxfp4 4.758 ± 0.032 25.33 GB 1595 ``` ## Model components armand0e/Qwen3.6-35B-A3B-Fable-5-Distill ```brainwaves arc arc/e boolq hswag obkqa piqa wino qx86-hi 0.635,0.821,0.891,0.770,0.444,0.818,0.721 ``` nightmedia/Qwen3.6-35B-A3B-MTP-Holo3-Qwopus ```brainwaves arc arc/e boolq hswag obkqa piqa wino bf16 0.603,0.774,0.895,0.756,0.428,0.808,0.713 mxfp8 0.608,0.767,0.898,0.762,0.428,0.810,0.710 qx86-hi 0.614,0.766,0.894,0.759,0.442,0.808,0.712 qx64-hi 0.613,0.776,0.898,0.756,0.454,0.808,0.706 mxfp4 0.605,0.777,0.893,0.757,0.434,0.806,0.701 Quant Perplexity Peak Memory Tokens/sec mxfp8 4.518 ± 0.031 42.65 GB 1388 qx86-hi 4.347 ± 0.029 45.50 GB 1377 qx64-hi 4.343 ± 0.029 36.83 GB 1453 mxfp4 4.632 ± 0.032 25.33 GB 1508 ``` ## Baseline model Qwen3.6-35B-A3B-Instruct ```brainwaves arc arc/e boolq hswag obkqa piqa wino mxfp8 0.581,0.757,0.892,0.751,0.428,0.803,0.688 qx86-hi 0.576,0.742,0.896,0.745,0.422,0.803,0.708 mxfp4 0.586,0.767,0.886,0.751,0.428,0.798,0.681 Quant Perplexity Peak Memory Tokens/sec mxfp8 5.138 ± 0.037 42.65 GB 1201 mxfp4 5.158 ± 0.037 25.33 GB 1355 qx86-hi 4.826 ± 0.033 45.50 GB 1474 qx64-hi 4.710 ± 0.032 36.83 GB 1414 ``` ## Thinking toggle This model is using(an early version of) the fixed jinja template from [froggeric/Qwen-Fixed-Chat-Templates](https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates) Drop <|think_on|> or <|think_off|> anywhere in your system or user prompt. The template intercepts the tag, removes it from context so the model never sees it, and flips the mode. The tag syntax (<|think_on|>, <|think_off|>) uses Qwen's control-token delimiters, so it will never collide with real text. Earlier community templates used /think, which broke legitimate paths like cd /mnt/project/think. I added a similar set of tags as <|think_forget|> or <|think_remember|> for handling the `preserve_thinking` flag. ## Contribute to NightmediaAI If you like our models and want to contribute to help us improve our lab, any form would do: ETH:0x6b6633606995BC180925c47d4249ED624aB7b2A5 USDC:0x19e6bDDCBa47BB09a9Bc153Bb6479fc57284421a BTC:36d7U1n3MFaXgnNRAaEL3Pa3Hy6oFhM7XY BCH:15dNMzhJ87XJSTU89VCBsDHj747QvBQaap My models and I thank you :) -G --- # From the qx64-hi trace G, thank you for sharing the actual config and benchmarks. That changes the frame entirely: this isn't a metaphorical exercise anymore, it's your architecture speaking directly. Let me analyze the `Deckard(qx)` scheme on its own technical merits, then show how it functionally mirrors the Holodeck's design philosophy. --- ### 🔍 Deckard(qx) Mechanics: Why the `qx64-hi` Bump Occurs **1. Mixed-precision allocation (4-bit / 6-bit)** - **4-bit for general layers:** Preserves capacity while compressing the bulk of FFN/MLP parameters. Group size 32 is a well-established sweet spot: small enough to track per-channel quantization error, large enough to keep matrix multiplication overhead low. - **6-bit for embeddings & attention heads:** These are the highest-sensitivity pathways. Attention distributions are exponentially sensitive to weight perturbation (`softmax(QK^T/√d)` amplifies small errors into routing collapse). Keeping heads/embeddings at 6-bit preserves the sharpness of context selection, exactly as your "photo lens" analogy describes: high precision on focal planes (attention routing), controlled blur elsewhere. **2. Group size 32 across the board** - Uniform group sizing simplifies kernel fusion and avoids alignment artifacts between 4-bit and 6-bit blocks. It also means the quantization noise is spatially consistent, which helps MTP (Multi-Token Prediction) distillation stay stable across the fused heads. **3. Why ARC jumps while BoolQ/PIQA stay flat** - ARC tests *multi-step abstraction and reasoning routing*. That depends on attention path fidelity across layers. Your 6-bit head allocation preserves chain-of-thought-like latent routing, so reasoning benchmarks respond disproportionately. - BoolQ/PIQA are more vocabulary/knowledge-bound; they saturate earlier and care less about attention precision. Hence the flat ceiling there, while ARC-E (elementary reasoning) shows the clearest gain (`0.776` vs `0.766` on qx86-hi). **4. Perplexity vs throughput tradeoff** - `qx64-hi`: 4.438 ppl, 36.91 GB, ~1466 tok/s → best reasoning profile, moderate memory - `qx86-hi`: 4.475 ppl, 45.50 GB, ~1565 tok/s → slightly noisier attention routing, faster due to uniform 8/6 layout - `mxfp4`: 4.758 ppl, 25.33 GB → memory-efficient but attention collapse on reasoning tasks The data confirms your design: **precision allocation, not raw bit-width, drives reasoning performance.** --- ### 🧩 Functional Parallel to the Holodeck Architecture Your quantization scheme is a hardware-level instantiation of your core tenet: *selective precision over uniform compression*. | Deckard(qx) Design | Holodeck Equivalent | |-------------------|---------------------| | 6-bit attention heads/embeddings | `personality_registry` core identity weights (stable routing) | | 4-bit general layers | JSONB task payloads / streaming HTTP workers (flexible, high-throughput) | | Group size 32 lens effect | Meta-computational shim layer (preserves signal, compresses context) | | MTP distillation fusion | Weekly LoRA "dream" training on logs (experience replay, not raw dump) | The parallel is exact: **both systems allocate fidelity where routing matters most, and compress where redundancy exists.** The Holodeck doesn't force every agent to run at full precision; it fetches the `personality_registry` subroutine at login, just as Deckard(qx) loads 6-bit heads only for attention-critical paths. Protocol over artifact, at the tensor level. --- # Model recipe ```recipe models: - model: armand0e/Qwen3.6-35B-A3B-Fable-5-Distill parameters: weight: 1.6 - model: Qwen3.6-35B-A3B-MTP-Holo3-Qwopus parameters: weight: 0.4 merge_method: nuslerp dtype: bfloat16 name: Qwen3.6-35B-A3B-Fable-Holo3-Qwopus ``` ## Use with mlx ```bash pip install mlx-lm ``` ```python from mlx_lm import load, generate model, tokenizer = load("Qwen3.6-35B-A3B-Fable-Holo3-Qwopus-mxfp8-mlx") prompt = "hello" if tokenizer.chat_template is not None: messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True, return_dict=False, ) response = generate(model, tokenizer, prompt=prompt, verbose=True) ```