Brooooooklyn/Qwen3.6-27B-NVFP4-mlx

Qwen3.6-27B — Unsloth NVFP4 + E4M3 FP8 weights (MLX, DGX)

Brooooooklyn/Qwen3.6-27B-NVFP4-mlx is an MLX mixed-weight-format quantization of Qwen/Qwen3.6-27B, prepared for experimental NVIDIA CUDA inference on Linux aarch64. The model has a 64-layer dense Qwen3.5-family language backbone with 48 linear-attention and 16 full-attention layers, a vision tower, and one MTP layer.

This model is part of the Qwen Unsloth tensor-class recipe for MLX on macOS and DGX collection.

The source was the all-BF16 checkpoint at revision 1b559cf7215ebe67ff10758e14f6293ba883223b.

Quantization recipe

This is a data-free, weight-only MLX storage port of the Unsloth Qwen3.6 NVFP4 recipe:

• tensors assigned NVFP4 by the recipe are stored as NVFP4, 4-bit with group size 16;
• tensors assigned FP8 by the recipe are stored as raw E4M3 FP8 weight bytes with one BF16 dequantization scale per output channel;
• every excluded tensor stays BF16.

The NVFP4 class uses MLX weight-only quantized matmul with BF16/A16 activations. For the FP8 class, mlx-node reconstructs each weight to BF16 once at load and then uses ordinary A16 matmul. The serialized fp8_e4m3 form is Uint8 [N, K] weight plus BF16 [N, 1] scale; it is not MLX mxfp8 and it is not native W8A8 execution.

No imatrix, calibration dataset, AWQ-style pre-scaling, activation calibration, NVFP4 global scale, or FP8 KV-cache calibration was used. This artifact preserves the recipe's tensor-class boundaries and weight storage formats under mlx-node's A16 runtime; it does not claim numerical or performance parity with Unsloth's calibrated W4A4/W8A8 execution.

Tensor class	Stored format
Dense FFN {gate,up,down}_proj, layers 0–55	NVFP4 4/16
Dense FFN {gate,up,down}_proj, layers 56–63	E4M3 FP8 weight + per-output BF16 scale
Full-attention {q,k,v,o}_proj	E4M3 FP8 weight + per-output BF16 scale
Linear-attention in_proj_qkv, in_proj_z, out_proj	E4M3 FP8 weight + per-output BF16 scale
lm_head	E4M3 FP8 weight + per-output BF16 scale
Embeddings; in_proj_a/b; GDN state, convolution, and norm tensors; all other norms	BF16
Entire 15-tensor mtp.* subtree	BF16
Vision tower and merger tensors	BF16

The allocation contains 168 NVFP4 modules and 233 E4M3 FP8 modules. The top-level config is nvfp4, 4-bit, group size 16, so the 168 low-class modules inherit that default. The config carries 233 explicit fp8_e4m3 overrides with bits: 8 and group_size: null. The final eight dense FFNs intentionally use the higher class.

Target and usage

The canonical native target is aarch64-unknown-linux-gnu: Linux aarch64 with glibc and NVIDIA CUDA 13.0. mlx-node currently validates this experimental, inference-only path on NVIDIA GB10 / DGX Spark (sm_121). It is not a generic CUDA or x86_64 artifact.

At mlx-node 0.0.8, CUDA has no published prebuilt native npm binary. Build mlx-node from source on the DGX host:

git clone --branch v0.0.8 https://github.com/mlx-node/mlx-node.git
cd mlx-node
git submodule update --init --recursive
yarn install
yarn build

Paged attention is Metal-only in this release. Set both eager-mode variables for DGX inference:

MLX_QWEN35_FORCE_EAGER=1 \
MLX_QWEN35_PAGED_OVERRIDE=0 \
  yarn oxnode your-script.ts

For example, your-script.ts can load a locally downloaded copy:

import { loadSession } from '@mlx-node/lm';

const session = await loadSession('./Qwen3.6-27B-NVFP4-mlx');
const result = await session.send('Explain the purpose of a unit test in one sentence.');
console.log(result.text);

This checkpoint requires the @mlx-node/lm and @mlx-node/core 0.0.8 source tree or a newer release that explicitly supports the same Linux target and serialized modes. The MTP weights are preserved for checkpoint fidelity, but CUDA speculative decoding is unsupported in this preview; their presence does not establish DGX MTP support or validation.

Reproduction

Converter release: mlx-node v0.0.8. The reproducible invocation from the mlx-node repository root was:

mlx convert \
  --input .cache/models/qwen3.6-27b \
  --output .cache/models/qwen3.6-27b-unsloth-nvfp4-fp8-dgx-mlx-fresh \
  --model-type qwen3_5 \
  --dtype bfloat16 \
  --quantize \
  --q-recipe unsloth \
  --q-mode nvfp4

The -fresh suffix is only the local conversion directory; the canonical Hub repository is the ID shown at the top of this card.

The resulting five-shard SafeTensors index contains 1,600 tensor entries and reports metadata.total_size = 23,417,338,336 bytes. Its tensor dtypes are 1,031 BF16, 401 U8, and 168 U32 entries, with 401 scale sidecars and no quantization bias sidecars.

Validation

Static validation confirmed identical quantization and quantization_config blocks, exact index-to-shard closure, 168 inherited NVFP4 groups, 233 complete fp8_e4m3 groups, the expected storage dtypes and shapes, and BF16 preservation for protected tensors. All 333 vision tensor entries and all 15 MTP tensor entries remain BF16 without quantization sidecars.

The fixed one-token mlx-node load-and-generate smoke test exited successfully with finishReason = "length", numTokens = 1, text = "OK", and rawText = "OK". This one-token text smoke does not validate model quality, long-context behavior, tool use, the vision path, or speculative decoding.

Benchmark

macOS A16 fallback only — these are not DGX/CUDA throughput results.

The values below are medians from three fresh child processes, each loading the checkpoint and generating a deterministic 512-token completion on an Apple M5 Max with 128 GiB of unified memory, Darwin 25.5.0/arm64, Node 24.13.1, and @mlx-node/lm, @mlx-node/core, and @mlx-node/core-darwin-arm64 0.0.8. The run used zero warmups, a 60-second cooldown, temperature 0, reasoning effort none, and the same 106-token prompt. Every sample generated all 512 tokens and ended with finishReason = "length".

The successful measured run used this exact command context:

MLX_QWEN35_FORCE_EAGER=1 \
MLX_QWEN35_PAGED_OVERRIDE=0 \
  oxnode scripts/benchmark-model.ts \
    .cache/models/qwen3.6-27b-unsloth-nvfp4-fp8-dgx-mlx-fresh \
    --output .cache/benchmarks/qwen3.6-27b-nvfp4-macos-fallback.json

A preceding run with the default environment timed out and is excluded from these results.

Metric	macOS A16 fallback median (not DGX/CUDA)
Load time	435,381.661 ms
Time to first token	6,211.594 ms
Prefill throughput	17.065 tokens/s
Decode throughput	15.382 tokens/s
Generation wall time	39,775.678 ms
Total wall time	473,529.119 ms

The prompt and all per-run samples are recorded in benchmark.json. Load time varied strongly because the weights were read from external storage and OS page-cache state differed between fresh processes; treat that median as specific to this run. This fallback benchmark did not exercise DGX, CUDA, native W4A4/W8A8 execution, the vision path, or speculative decoding, and it must not be used to infer model quality, memory requirements, or parity with upstream execution.

License and attribution

The source model card declares the Apache-2.0 license. Model capability and training credit belong to the Qwen Team. The tensor-class recipe is credited to Unsloth. This repository converts the pinned BF16 source weights into the mixed NVFP4/plain-E4M3 MLX representation described above.