--- base_model: - Qwen/Qwen3.8-Flash-Next-FP8 base_model_relation: quantized library_name: mlx license: other license_name: qwen-community-1.0 license_link: LICENSE pipeline_tag: text-generation tags: - mlx - safetensors - qwen4_exp - omlx - oq - quantized - apple-silicon - mixture-of-experts - 4-bit - long-context - conversational - tool-calling - speculative-decoding - mtp --- # Qwen3.8 Flash Next oQ4e MTP 128K for oMLX This is a community, text-only dynamic quantization of [Qwen/Qwen3.8-Flash-Next-FP8](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) for Apple Silicon and oMLX. It is designed for a 131,072-token context window on a Mac with 128 GiB of unified memory. The conversion removes the vision encoder and retains the released one-layer Multi-Token Prediction head for depth-1 speculative decoding. It is not an official Qwen release. ## Quantization The model uses oMLX enhanced importance-matrix quantization with 4-bit affine, group-size 128 weights as the default. Precision is assigned by tensor family: - The token embedding and language-model head use 4-bit affine quantization. - Large linear and routed-expert weights use importance-matrix-guided mixed precision. - Sensitive attention and shared-expert matrices retain higher precision, including 8-bit floors. - Routers, recurrent state, convolutions, normalization, residual controls, and related control tensors retain BF16. - The MTP head's 512 routed experts use importance-weighted 4-bit affine quantization at group size 128. Their three projections are stacked, avoiding 3,072 separate expert weight and scale arrays. - MTP attention, shared-expert, and layer mixer matrices use 8-bit affine quantization. Fusion projections, the final mixer, routers, norms, gates, and injection controls retain BF16. - The 51.2B-parameter PLE n-gram bank remains split into 128 shards. Shards 0 through 3 use 2-bit affine, group size 32. Shards 4 through 127 use 3-bit affine, group size 32. The importance matrix was collected from 1,024 tool-heavy multilingual calibration samples at sequence length 512. It observed 73,665 of 73,728 backbone expert-projection slots. The 63 unobserved slots retained conservative fallback precision. The included `oq_imatrix_report.json` records the allocation inputs and complete coverage details. The MTP expert quantizer uses the calibrated activation importance of the analogous final backbone layer as a proxy. This preserves expert-specific weighting without claiming that the non-MTP calibration cache directly measured the draft head. The resulting checkpoint contains 22 safetensors files and occupies 87.959 GiB. Its index contains 2,916 tensors, including 62 MTP tensors. ## Runtime requirements This checkpoint is not compatible with stock Transformers, stock MLX-LM, or an oMLX release without Qwen4 experimental support. The exact text model, cache integration, and mixed-bit SSD-backed PLE implementation used for validation are included under `omlx_support`. The tested software stack is: - Apple Silicon - MLX and mlx-metal 0.32.1 - The MLX-LM package bundled with oMLX 0.6.3rc3 - oMLX 0.6.3rc3 - Depth-1 MTP enabled through oMLX's generic native-MTP scheduler The Qwen4 implementation is based on [oMLX pull request #3161](https://github.com/jundot/omlx/pull/3161) at commit `79e11cf86360ca800db5bdc45cec4ee582ada6b1`, with an adaptation for this checkpoint's per-shard 2-bit and 3-bit PLE metadata. The MTP equations follow draft [pull request #3163](https://github.com/jundot/omlx/pull/3163) at commit `b115fa5d928c11acf4588a55814a307d9161baff`, adapted to the sparse 128K text path and oMLX's installed generic speculative scheduler. Install the tested MLX runtime inside the downloaded model directory, then launch the bundled oMLX wrapper: ```sh hf download jedisct1/Qwen3.8-Flash-Next-oQ4e-MTP-128k \ --local-dir models/Qwen3.8-Flash-Next-oQ4e-MTP-128k cd models/Qwen3.8-Flash-Next-oQ4e-MTP-128k uv pip install --target .mlx-runtime mlx==0.32.1 mlx-metal==0.32.1 zsh omlx_support/serve ``` The launcher expects oMLX at `/Applications/oMLX.app`, listens only on `127.0.0.1:8766`, and uses `omlx` as its default local API key. Set `OMLX_APP_RESOURCES`, `OMLX_API_KEY`, `OMLX_PORT`, `OMLX_CACHE_DIR`, or `OMLX_MLX_RUNTIME` to override those paths and values. It installs the included model settings into a fresh oMLX base path, permits one concurrent request, selects oMLX's balanced memory guard, disables the in-memory hot cache, memory-maps the PLE safetensors, and uses an SSD prompt cache. MTP is fixed at depth 1. Prompt priming is capped at 8,192 tokens so a long request cannot retain a second full-context attention timeline. The context budget is total input plus generated output, so keep that total at or below 131,072 tokens. ## Recommended generation settings Thinking mode is enabled by default. The tested generation settings are split between `generation_config.json` and the bundled `omlx_support/model_settings.json`: ```text temperature: 1.0 top_p: 0.95 top_k: 20 min_p: 0.0 presence_penalty: 0.0 repetition_penalty: 1.0 ``` For multi-turn agents, preserve `reasoning_content` between tool turns and keep `preserve_thinking` enabled. Use `reasoning_effort = "medium"` inside `extra_body` as the general-purpose default. It retains native thinking without the extra system instruction added by the `xhigh` template tier. It is a prompt policy, not a hard reasoning-token budget. Reserve `xhigh` for unusually difficult tasks where the additional latency and reasoning tokens are justified. ## Swival setup The end-to-end agent checks used Swival 1.0.41 at commit `f969cf5f9ab8f46ea350fb050275b27f4916bf8a`. After starting the bundled oMLX launcher, add this profile to `~/.config/swival/config.toml`: ```toml [profiles.qwen38-flash-next] provider = "generic" base_url = "http://127.0.0.1:8766" api_key = "omlx" model = "Qwen3.8-Flash-Next-oQ4e-MTP-128k" max_context_tokens = 131072 max_output_tokens = 32768 temperature = 1.0 top_p = 0.95 extra_body = { reasoning_effort = "medium", top_k = 20, min_p = 0.0, presence_penalty = 0.0, repetition_penalty = 1.0, chat_template_kwargs = { enable_thinking = true, preserve_thinking = true } } ``` For this unregistered local model, Swival 1.0.41's top-level `reasoning_effort` is removed by LiteLLM's unsupported-parameter filter. Keeping the value inside `extra_body` sends the dedicated OpenAI-compatible field to oMLX. The field is named `reasoning_effort`; `thinking_effort` is not supported by this stack. Run a task from its workspace so Swival can apply the intended filesystem and command boundaries: ```sh swival --profile qwen38-flash-next --base-dir /path/to/workspace \ "Inspect the project and complete the requested task." ``` The tested Swival revision rejects duplicate identical tool calls before execution, permits at most one repair response, and fails closed if the repair is still invalid. Keep equivalent guards if using another agent runner. ## Validation The artifact validator and strict lazy load found: - 22 expected and readable safetensors files - 2,916 indexed and stored tensors, including 62 MTP tensors - No missing, unindexed, duplicated, or misplaced tensors - No remaining per-expert MTP tensors after stacking - All 128 PLE mixed-bit overrides applied as intended - No missing or mismatched importance-matrix entries for measurable modules On an M5 Max with 128 GiB of unified memory, the SSD-backed PLE implementation loaded 64.65 GB of Metal-backed model arrays. A fresh request containing 131,071 uncached prompt tokens generated one token, for exactly 131,072 total tokens, in 873.41 seconds with oMLX's balanced memory guard enabled. The guard reclaimed 4.86 GB and 5.87 GB of pooled Metal buffers during prefill instead of weakening Apple's 107.52 GiB working-set ceiling. Live depth-1 requests activated the MTP path. The direct structured-call check accepted 39 of 45 draft tokens, or 86.7 percent. Across the six-turn Swival workflow, per-turn acceptance ranged from 72.5 to 87.7 percent. Acceptance is prompt-dependent and is not a quality guarantee. ## Tool calling The original chat template and XML tool-call protocol are included. With MTP active, direct structured tool calls, continuation after a tool result, and a six-turn Swival file-write, checksum, read-back, and exact-byte comparison workflow were tested successfully. No identical tool invocation executed twice in that workflow. Tool calling is stochastic and is not guaranteed to be flawless. Synthetic exact-call testing passed 17 of 18 cases with resident PLE and 16 of 18 cases with SSD-backed PLE. One failed response emitted duplicate calls. Agents should reject duplicate identical calls before execution, allow at most one bounded repair attempt, and fail closed if the repaired response remains invalid. The dedicated MTP run passed 17 of 18 exact structured-call cases at the official thinking preset. The failed case emitted ordinary text claiming that it had called the requested tool but contained no structured call. The same case passed on an immediate isolated retry. MTP therefore matched the best prior non-MTP soak but did not eliminate the model's stochastic protocol failures. The duplicate-call protection used during validation is implemented by Swival and is not part of these model weights. ## Limitations - This is a text-only conversion. Image and video inputs are unsupported. - MTP requires the bundled Qwen4 compatibility hook and oMLX's native-MTP scheduler. Stock MLX-LM cannot use this head. - Prompt priming is intentionally disabled for prompts above 8,192 tokens. Long prompts still run correctly but begin drafting without a primed head history. - The architecture supports 262,144 native tokens, but this conversion was validated and operationally limited to 131,072 total tokens. - Only the stated Apple Silicon and oMLX configuration was tested. - Quantization can reduce quality relative to the FP8 source. - Greedy MTP-on and MTP-off checks produced the same final JSON answer but not a bit-identical hidden reasoning sequence. Multi-token target verification can take numerically different matrix paths. - The model can produce incorrect content, malformed tool calls, duplicate tool calls, or unsafe actions. Tool execution requires independent validation and appropriate permission boundaries. - The model inherits the limitations and intended-use considerations of the upstream Qwen model. ## Source and license The conversion was produced from revision `bcd9f01ddc9cff2316eb84281bebcd5b058bddce` of [Qwen/Qwen3.8-Flash-Next-FP8](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8). The weights remain subject to the Qwen Community License 1.0. Read `LICENSE` before using or redistributing the model, particularly its terms for commercial Model as a Service and AI Work Assistant businesses. The bundled `omlx_support` code is distributed under the Apache License 2.0 in `omlx_support/LICENSE` and includes modifications to the implementations proposed in oMLX pull requests #3161 and #3163. ## Citation ```bibtex @techreport{qwen2026design, title = {On the Design of {Qwen3.8-Next} Architecture: Evaluation, Efficiency, and Training Stability}, author = {{Qwen Team}}, institution = {Alibaba Group}, month = {August}, year = {2026} } @misc{qwen3.8flashnext, title = {{Qwen3.8-Flash-Next}: A New Architecture, Towards Ultimate Cost-Efficiency}, author = {{Qwen Team}}, month = {August}, year = {2026}, url = {https://qwen.ai/blog?id=qwen3.8-flash-next} } ```