jedisct1/Qwen3.8-Flash-Next-oQ4e-MTP-128k

Qwen3.8 Flash Next oQ4e MTP 128K for oMLX

This is a community, text-only dynamic quantization of Qwen/Qwen3.8-Flash-Next-FP8 for Apple Silicon and oMLX.

It is designed for a 131,072-token context window on a Mac with 128 GiB of unified memory. The conversion removes the vision encoder and retains the released one-layer Multi-Token Prediction head for depth-1 speculative decoding. It is not an official Qwen release.

Quantization

The model uses oMLX enhanced importance-matrix quantization with 4-bit affine, group-size 128 weights as the default. Precision is assigned by tensor family:

• The token embedding and language-model head use 4-bit affine quantization.
• Large linear and routed-expert weights use importance-matrix-guided mixed precision.
• Sensitive attention and shared-expert matrices retain higher precision, including 8-bit floors.
• Routers, recurrent state, convolutions, normalization, residual controls, and related control tensors retain BF16.
• The MTP head's 512 routed experts use importance-weighted 4-bit affine quantization at group size 128. Their three projections are stacked, avoiding 3,072 separate expert weight and scale arrays.
• MTP attention, shared-expert, and layer mixer matrices use 8-bit affine quantization. Fusion projections, the final mixer, routers, norms, gates, and injection controls retain BF16.
• The 51.2B-parameter PLE n-gram bank remains split into 128 shards. Shards 0 through 3 use 2-bit affine, group size 32. Shards 4 through 127 use 3-bit affine, group size 32.

The importance matrix was collected from 1,024 tool-heavy multilingual calibration samples at sequence length 512. It observed 73,665 of 73,728 backbone expert-projection slots. The 63 unobserved slots retained conservative fallback precision. The included oq_imatrix_report.json records the allocation inputs and complete coverage details.

The MTP expert quantizer uses the calibrated activation importance of the analogous final backbone layer as a proxy. This preserves expert-specific weighting without claiming that the non-MTP calibration cache directly measured the draft head.

The resulting checkpoint contains 22 safetensors files and occupies 87.959 GiB. Its index contains 2,916 tensors, including 62 MTP tensors.

Runtime requirements

This checkpoint is not compatible with stock Transformers, stock MLX-LM, or an oMLX release without Qwen4 experimental support. The exact text model, cache integration, and mixed-bit SSD-backed PLE implementation used for validation are included under omlx_support.

The tested software stack is:

• Apple Silicon
• MLX and mlx-metal 0.32.1
• The MLX-LM package bundled with oMLX 0.6.3rc3
• oMLX 0.6.3rc3
• Depth-1 MTP enabled through oMLX's generic native-MTP scheduler

The Qwen4 implementation is based on oMLX pull request #3161 at commit 79e11cf86360ca800db5bdc45cec4ee582ada6b1, with an adaptation for this checkpoint's per-shard 2-bit and 3-bit PLE metadata. The MTP equations follow draft pull request #3163 at commit b115fa5d928c11acf4588a55814a307d9161baff, adapted to the sparse 128K text path and oMLX's installed generic speculative scheduler.

Install the tested MLX runtime inside the downloaded model directory, then launch the bundled oMLX wrapper:

hf download jedisct1/Qwen3.8-Flash-Next-oQ4e-MTP-128k \
  --local-dir models/Qwen3.8-Flash-Next-oQ4e-MTP-128k
cd models/Qwen3.8-Flash-Next-oQ4e-MTP-128k
uv pip install --target .mlx-runtime mlx==0.32.1 mlx-metal==0.32.1
zsh omlx_support/serve

The launcher expects oMLX at /Applications/oMLX.app, listens only on 127.0.0.1:8766, and uses omlx as its default local API key. Set OMLX_APP_RESOURCES, OMLX_API_KEY, OMLX_PORT, OMLX_CACHE_DIR, or OMLX_MLX_RUNTIME to override those paths and values.

It installs the included model settings into a fresh oMLX base path, permits one concurrent request, selects oMLX's balanced memory guard, disables the in-memory hot cache, memory-maps the PLE safetensors, and uses an SSD prompt cache. MTP is fixed at depth 1. Prompt priming is capped at 8,192 tokens so a long request cannot retain a second full-context attention timeline. The context budget is total input plus generated output, so keep that total at or below 131,072 tokens.

Recommended generation settings

Thinking mode is enabled by default. The tested generation settings are split between generation_config.json and the bundled omlx_support/model_settings.json:

temperature: 1.0
top_p: 0.95
top_k: 20
min_p: 0.0
presence_penalty: 0.0
repetition_penalty: 1.0

For multi-turn agents, preserve reasoning_content between tool turns and keep preserve_thinking enabled. Use reasoning_effort = "medium" inside extra_body as the general-purpose default. It retains native thinking without the extra system instruction added by the xhigh template tier. It is a prompt policy, not a hard reasoning-token budget. Reserve xhigh for unusually difficult tasks where the additional latency and reasoning tokens are justified.

Swival setup

The end-to-end agent checks used Swival 1.0.41 at commit f969cf5f9ab8f46ea350fb050275b27f4916bf8a. After starting the bundled oMLX launcher, add this profile to ~/.config/swival/config.toml:

[profiles.qwen38-flash-next]
provider = "generic"
base_url = "http://127.0.0.1:8766"
api_key = "omlx"
model = "Qwen3.8-Flash-Next-oQ4e-MTP-128k"
max_context_tokens = 131072
max_output_tokens = 32768
temperature = 1.0
top_p = 0.95
extra_body = { reasoning_effort = "medium", top_k = 20, min_p = 0.0, presence_penalty = 0.0, repetition_penalty = 1.0, chat_template_kwargs = { enable_thinking = true, preserve_thinking = true } }

For this unregistered local model, Swival 1.0.41's top-level reasoning_effort is removed by LiteLLM's unsupported-parameter filter. Keeping the value inside extra_body sends the dedicated OpenAI-compatible field to oMLX. The field is named reasoning_effort; thinking_effort is not supported by this stack.

Run a task from its workspace so Swival can apply the intended filesystem and command boundaries:

swival --profile qwen38-flash-next --base-dir /path/to/workspace \
  "Inspect the project and complete the requested task."

The tested Swival revision rejects duplicate identical tool calls before execution, permits at most one repair response, and fails closed if the repair is still invalid. Keep equivalent guards if using another agent runner.

Validation

The artifact validator and strict lazy load found:

• 22 expected and readable safetensors files
• 2,916 indexed and stored tensors, including 62 MTP tensors
• No missing, unindexed, duplicated, or misplaced tensors
• No remaining per-expert MTP tensors after stacking
• All 128 PLE mixed-bit overrides applied as intended
• No missing or mismatched importance-matrix entries for measurable modules

On an M5 Max with 128 GiB of unified memory, the SSD-backed PLE implementation loaded 64.65 GB of Metal-backed model arrays. A fresh request containing 131,071 uncached prompt tokens generated one token, for exactly 131,072 total tokens, in 873.41 seconds with oMLX's balanced memory guard enabled. The guard reclaimed 4.86 GB and 5.87 GB of pooled Metal buffers during prefill instead of weakening Apple's 107.52 GiB working-set ceiling.

Live depth-1 requests activated the MTP path. The direct structured-call check accepted 39 of 45 draft tokens, or 86.7 percent. Across the six-turn Swival workflow, per-turn acceptance ranged from 72.5 to 87.7 percent. Acceptance is prompt-dependent and is not a quality guarantee.

Tool calling

The original chat template and XML tool-call protocol are included. With MTP active, direct structured tool calls, continuation after a tool result, and a six-turn Swival file-write, checksum, read-back, and exact-byte comparison workflow were tested successfully. No identical tool invocation executed twice in that workflow.

Tool calling is stochastic and is not guaranteed to be flawless. Synthetic exact-call testing passed 17 of 18 cases with resident PLE and 16 of 18 cases with SSD-backed PLE. One failed response emitted duplicate calls. Agents should reject duplicate identical calls before execution, allow at most one bounded repair attempt, and fail closed if the repaired response remains invalid.

The dedicated MTP run passed 17 of 18 exact structured-call cases at the official thinking preset. The failed case emitted ordinary text claiming that it had called the requested tool but contained no structured call. The same case passed on an immediate isolated retry. MTP therefore matched the best prior non-MTP soak but did not eliminate the model's stochastic protocol failures.

The duplicate-call protection used during validation is implemented by Swival and is not part of these model weights.

Limitations

• This is a text-only conversion. Image and video inputs are unsupported.
• MTP requires the bundled Qwen4 compatibility hook and oMLX's native-MTP scheduler. Stock MLX-LM cannot use this head.
• Prompt priming is intentionally disabled for prompts above 8,192 tokens. Long prompts still run correctly but begin drafting without a primed head history.
• The architecture supports 262,144 native tokens, but this conversion was validated and operationally limited to 131,072 total tokens.
• Only the stated Apple Silicon and oMLX configuration was tested.
• Quantization can reduce quality relative to the FP8 source.
• Greedy MTP-on and MTP-off checks produced the same final JSON answer but not a bit-identical hidden reasoning sequence. Multi-token target verification can take numerically different matrix paths.
• The model can produce incorrect content, malformed tool calls, duplicate tool calls, or unsafe actions. Tool execution requires independent validation and appropriate permission boundaries.
• The model inherits the limitations and intended-use considerations of the upstream Qwen model.

Source and license

The conversion was produced from revision bcd9f01ddc9cff2316eb84281bebcd5b058bddce of Qwen/Qwen3.8-Flash-Next-FP8.

The weights remain subject to the Qwen Community License 1.0. Read LICENSE before using or redistributing the model, particularly its terms for commercial Model as a Service and AI Work Assistant businesses.

The bundled omlx_support code is distributed under the Apache License 2.0 in omlx_support/LICENSE and includes modifications to the implementations proposed in oMLX pull requests #3161 and #3163.

Citation

@techreport{qwen2026design,
    title       = {On the Design of {Qwen3.8-Next} Architecture: Evaluation, Efficiency, and Training Stability},
    author      = {{Qwen Team}},
    institution = {Alibaba Group},
    month       = {August},
    year        = {2026}
}

@misc{qwen3.8flashnext,
    title  = {{Qwen3.8-Flash-Next}: A New Architecture, Towards Ultimate Cost-Efficiency},
    author = {{Qwen Team}},
    month  = {August},
    year   = {2026},
    url    = {https://qwen.ai/blog?id=qwen3.8-flash-next}
}