A mixed-precision MLX conversion of Qwen/Qwen3.8-Flash-Next, quantised directly from the official BF16 checkpoint with its native MTP draft block preserved.
Original model · Qwen overview · MLX-VLM · Qwen Community License 1.0
## About this conversion This repository contains an oQ6 mixed-precision MLX conversion produced directly from Qwen's BF16 weights. Layer sensitivity was measured with a validated local 4-bit proxy. Every final tensor, including the native MTP head, was rebuilt from the official BF16 checkpoint. Group size 32 supports the model's 160-wide hashed n-gram embedding tables. | Item | Value | | --- | --- | | Repository | `Vontra/Qwen3.8-Flash-Next-MLX-oQ6-MTP` | | Base model | [`Qwen/Qwen3.8-Flash-Next`](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) | | Format | MLX safetensors | | Quantisation | oQ6 mixed precision, 6-bit affine base with protected modules at 6/8-bit | | Base group size | 32 | | Native MTP | Included, one Qwen4Exp draft block | | Weight tensors | 3,747 total, including 76 converted MTP tensors | | Weight shards | 31 | | Weight size | 158.07 GB, 147.21 GiB | | Configured context | 262,144 tokens | | Architecture | `qwen4_exp` vision-language sparse MoE | The upstream tokenizer, chat template, vision processor, generation configuration, and native MTP configuration are preserved. > [!IMPORTANT] > This checkpoint requires an oMLX or MLX-VLM runtime with explicit `qwen4_exp` native-MTP support. Stock runtimes that do not construct the Qwen4Exp MTP module may reject the 76 MTP tensors during strict weight loading. > [!CAUTION] > Do not attach a Qwen3.8 27B drafter to this model. Flash Next has different hidden dimensions and ships its own matching MTP block here. ## Download and use ```bash hf download Vontra/Qwen3.8-Flash-Next-MLX-oQ6-MTP \ --local-dir Qwen3.8-Flash-Next-MLX-oQ6-MTP ``` Add the downloaded directory to an MTP-capable oMLX installation, select the model, and enable native MTP in its model settings. ## Apple M3 Studio performance Validated with native MTP enabled in oMLX on an Apple M3 Studio: | Test | Output tokens | Speed | | --- | ---: | ---: | | Raw completion, warmed | 32 | 29.6 tokens/s | | Exact-instruction chat | 27 | 28.3 tokens/s | | Casual chat | 68 | 36.6 tokens/s | The two deterministic raw-completion runs produced identical output. The exact-instruction test returned `hello`, and the longer chat completed coherently without repetition or cache-reconciliation errors. Native MTP remained active throughout, with 70.8–84.2% draft acceptance across these runs. For scale only, the earlier non-MTP oQ6 backbone reached 23.1 tokens/s in a short standalone MLX test, 21.1 tokens/s over a 142-token response, and 19.5 tokens/s over a warmed 512-token oMLX response on an Apple M3 Studio. Those figures are a backbone reference, not an MTP speed claim for this repository. Results vary with prompt length, cache state, sampling settings, runtime version, and memory pressure. ## Architecture Qwen3.8 Flash Next combines Gated DeltaNet, Qwen Sparse Attention, sparse mixture-of-experts layers, widened gated residual streams, hashed bigram and trigram embeddings, and a native next-token prediction block for speculative decoding. | Architecture detail | Upstream value | | --- | ---: | | Language-model parameters | 125B total / 6B active | | N-gram embedding | 51B parameters | | Layers | 48 | | Routed / active experts | 512 / 10, plus 1 shared | | Attention heads / KV heads | 24 / 2 | | Hidden size | 2,560 | | Native configured context | 262,144 tokens | | Native MTP draft blocks | 1 | For upstream evaluations, intended use, limitations, safety guidance, and the complete architecture discussion, see the [original model card](https://huggingface.co/Qwen/Qwen3.8-Flash-Next). ## Conversion and validation - The converter read the official BF16 checkpoint directly. - Sensitivity-guided allocation used a 6-bit base and 232 protected or overridden modules. - Structural validation checked all 3,747 indexed tensors and all 31 shards. - MTP validation confirmed one configured draft layer and 76 converted MTP tensor entries. - Live generation validation covered deterministic completion, exact-instruction chat, coherent chat, MTP telemetry, and cache rollback. This is a community conversion, not an official Qwen release. ## License and attribution The upstream model is released under the **Qwen Community License 1.0**. The required licence text is included in this repository. Model design, training, evaluations, and upstream documentation belong to Qwen and the original contributors. The MLX conversion and packaging are provided by [Vontra](https://huggingface.co/Vontra).