README.md

---
base_model:
- Qwen/Qwen3.8-Flash-Next-FP8
base_model_relation: quantized
library_name: mlx
license: other
license_name: qwen-community-1.0
license_link: LICENSE
pipeline_tag: text-generation
tags:
- mlx
- safetensors
- qwen4_exp
- omlx
- oq
- quantized
- apple-silicon
- mixture-of-experts
- 4-bit
- long-context
- conversational
- tool-calling
---

# Qwen3.8 Flash Next oQ4e 128K for oMLX

This is a community, text-only dynamic quantization of [Qwen/Qwen3.8-Flash-Next-FP8](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) for Apple Silicon and oMLX.

It is designed for a 131,072-token context window on a Mac with 128 GiB of unified memory. The conversion removes the vision encoder and MTP head. It is not an official Qwen release.

## Quantization

The model uses oMLX enhanced importance-matrix quantization with 4-bit affine, group-size 128 weights as the default. Precision is assigned by tensor family:

- The token embedding and language-model head use 4-bit affine quantization.
- Large linear and routed-expert weights use importance-matrix-guided mixed precision.
- Sensitive attention and shared-expert matrices retain higher precision, including 8-bit floors.
- Routers, recurrent state, convolutions, normalization, residual controls, and related control tensors retain BF16.
- The 51.2B-parameter PLE n-gram bank remains split into 128 shards. Shards 0 through 3 use 2-bit affine, group size 32. Shards 4 through 127 use 3-bit affine, group size 32.

The importance matrix was collected from 1,024 tool-heavy multilingual calibration samples at sequence length 512. The included `oq_imatrix_report.json` records the allocation inputs and coverage.

The resulting checkpoint contains 18 safetensors shards and occupies 86.608 GiB. Its index contains 2,854 tensors.

## Runtime requirements

This checkpoint is not compatible with stock Transformers, stock MLX-LM, or an oMLX release without Qwen4 experimental support. The exact text model, cache integration, and mixed-bit SSD-backed PLE implementation used for validation are included under `omlx_support`.

The tested software stack is:

- Apple Silicon
- MLX and mlx-metal 0.32.1
- The MLX-LM package bundled with oMLX 0.6.3rc3
- oMLX 0.6.3rc3
- MTP disabled

The Qwen4 implementation is based on [oMLX pull request #3161](https://github.com/jundot/omlx/pull/3161) at commit `79e11cf86360ca800db5bdc45cec4ee582ada6b1`, with an adaptation for this checkpoint's per-shard 2-bit and 3-bit PLE metadata. [Pull request #3163](https://github.com/jundot/omlx/pull/3163) adds experimental MTP decoding, but MTP was not used here.

Install the tested MLX runtime inside the downloaded model directory, then launch the bundled oMLX wrapper:

```sh
hf download jedisct1/Qwen3.8-Flash-Next-oQ4e-128k \
  --local-dir models/Qwen3.8-Flash-Next-oQ4e-128k
cd models/Qwen3.8-Flash-Next-oQ4e-128k
uv pip install --target .mlx-runtime mlx==0.32.1 mlx-metal==0.32.1
zsh omlx_support/serve
```

The launcher expects oMLX at `/Applications/oMLX.app`, listens only on `127.0.0.1:8766`, and uses `omlx` as its default local API key. Set `OMLX_APP_RESOURCES`, `OMLX_API_KEY`, `OMLX_PORT`, `OMLX_CACHE_DIR`, or `OMLX_MLX_RUNTIME` to override those paths and values.

It installs the included model settings into a fresh oMLX base path, permits one concurrent request, selects oMLX's balanced memory guard, disables the in-memory hot cache, memory-maps the PLE safetensors, and uses an SSD prompt cache. The context budget is total input plus generated output, so keep that total at or below 131,072 tokens.

## Recommended generation settings

Thinking mode is enabled by default. The upstream recommended settings are included in `generation_config.json`:

```text
temperature: 1.0
top_p: 0.95
top_k: 20
min_p: 0.0
presence_penalty: 0.0
repetition_penalty: 1.0
```

For multi-turn agents, preserve `reasoning_content` between tool turns and keep `preserve_thinking` enabled.

## Validation

The artifact validator found:

- 18 expected and readable safetensors shards
- 2,854 indexed and stored tensors
- No missing, unindexed, duplicated, or misplaced tensors
- All 128 PLE mixed-bit overrides applied as intended
- No missing or mismatched importance-matrix entries for measurable modules

On an M5 Max with 128 GiB of unified memory, the SSD-backed PLE implementation loaded 63.30 GB of Metal-backed model arrays. A fresh request containing 131,071 uncached prompt tokens generated one token, for exactly 131,072 total tokens, with oMLX's balanced memory guard enabled.

## Tool calling

The original chat template and XML tool-call protocol are included. Direct structured tool calls, continuation after a tool result, and a multi-turn file-write, checksum, and read-back workflow were tested successfully through oMLX and Swival.

Tool calling is stochastic and is not guaranteed to be flawless. Synthetic exact-call testing passed 17 of 18 cases with resident PLE and 16 of 18 cases with SSD-backed PLE. One failed response emitted duplicate calls. Agents should reject duplicate identical calls before execution, allow at most one bounded repair attempt, and fail closed if the repaired response remains invalid.

The duplicate-call protection used during validation is implemented by Swival and is not part of these model weights.

## Limitations

- This is a text-only conversion. Image and video inputs are unsupported.
- MTP speculative decoding is unsupported by this checkpoint.
- The architecture supports 262,144 native tokens, but this conversion was validated and operationally limited to 131,072 total tokens.
- Only the stated Apple Silicon and oMLX configuration was tested.
- Quantization can reduce quality relative to the FP8 source.
- The model can produce incorrect content, malformed tool calls, duplicate tool calls, or unsafe actions. Tool execution requires independent validation and appropriate permission boundaries.
- The model inherits the limitations and intended-use considerations of the upstream Qwen model.

## Source and license

The conversion was produced from revision `bcd9f01ddc9cff2316eb84281bebcd5b058bddce` of [Qwen/Qwen3.8-Flash-Next-FP8](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8).

The weights remain subject to the Qwen Community License 1.0. Read `LICENSE` before using or redistributing the model, particularly its terms for commercial Model as a Service and AI Work Assistant businesses.

The bundled `omlx_support` code is distributed under the Apache License 2.0 in `omlx_support/LICENSE` and includes modifications to the implementation proposed in oMLX pull request #3161.

## Citation

```bibtex
@techreport{qwen2026design,
    title       = {On the Design of {Qwen3.8-Next} Architecture: Evaluation, Efficiency, and Training Stability},
    author      = {{Qwen Team}},
    institution = {Alibaba Group},
    month       = {August},
    year        = {2026}
}

@misc{qwen3.8flashnext,
    title  = {{Qwen3.8-Flash-Next}: A New Architecture, Towards Ultimate Cost-Efficiency},
    author = {{Qwen Team}},
    month  = {August},
    year   = {2026},
    url    = {https://qwen.ai/blog?id=qwen3.8-flash-next}
}
```