Nemotron Labs 3 Puzzle 75B A9B, MLX mixed 4/6-bit
Mixed-precision MLX conversion of NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-BF16, tested locally on an Apple M2 Max with 64 GB unified memory.
This is a community conversion, not an NVIDIA release.
Why this checkpoint exists
Puzzle 75B A9B has 75.3B total parameters and 9.3B active parameters per token. A uniform conversion is not the best fit for a 64 GB Mac because most parameters live in the routed expert bank, while dense paths and the output head are more sensitive.
This checkpoint applies precision by role:
| Component | Precision |
|---|---|
| Routed MoE expert projections | 4-bit affine |
| Dense projections and shared experts | 6-bit affine |
| Token embeddings | 6-bit affine |
| Output head | BF16 |
| Quantization group size | 64 |
The ordinary autoregressive backbone is included. Training-only MTP tensors were intentionally excluded.
Measured M2 Max results
Tested on an Apple M2 Max with 64 GB unified memory using MLX 0.32.0.
| Measurement | Result |
|---|---|
| Directory size | 42.03 GiB |
| Peak MLX memory | 49.6835 GB |
| Maximum process RSS | 45.3008 GB |
| Average generation speed | 14.2722 tok/s |
| Local task checks | 24/30 |
| Long-context retrieval checks | 4/5 |
A separate 256-token Python-generation run measured 24.623 tok/s and 45.269 GB peak process memory. That run used a different prompt and should not be compared directly with the 30-run average.
The local checks are regression and instruction-following checks, not a standardized model-quality benchmark. See the benchmark methodology and raw metrics.
MLX support
This model needs the Nemotron Puzzle support implemented in:
Until the upstream pull request is merged and released, install the feature branch:
git clone --branch feat/nemotron-h-puzzle-support \
https://github.com/sxuff/mlx-lm.git
cd mlx-lm
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e .
Generate
mlx_lm.generate \
--model tekosML/Nemotron-Labs-3-Puzzle-75B-A9B-MLX-4bit-experts-6bit \
--trust-remote-code \
--prompt "Write a Python function that returns the first n Fibonacci numbers." \
--max-tokens 256 \
--temp 1.0 \
--top-p 0.95
Only use --trust-remote-code after inspecting the repository's configuration and modeling files. They were copied from the NVIDIA source checkpoint during conversion.
Conversion
The conversion script and exact heterogeneous quantization predicate are available at:
Equivalent conversion command:
python scripts/convert_puzzle_mixed.py \
/path/to/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-BF16 \
/path/to/output \
--expert-bits 4
Recommended sampling settings used during validation:
temperature = 1.0
top_p = 0.95
Numerical compatibility work
The supporting MLX implementation preserves Puzzle's BF16 softplus(dt + dt_bias) boundary while retaining FP32 state-transition and recurrence arithmetic. First-layer SSM cosine similarity against NVIDIA's reference improved from 0.8832 to 0.999998 after correcting that boundary.
Puzzle-specific RMSNorm, router, projection-bias, and timestep behavior are gated to nemotron_h_puzzle, preserving existing ordinary Nemotron-H behavior.
Files and provenance
- Source weights:
nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-BF16 - Conversion runtime: MLX 0.32.0
- Format: sharded safetensors
- Weight shards: 9
- Total local files: 20
- File-content size: 45,104,700,954 bytes
License
Use and redistribution are governed by the original model's OpenMDW-1.1 license. The full license text and NVIDIA copyright notice are included in this repository.
The files configuration_nemotron_h.py and modeling_nemotron_h.py are separately licensed under Apache-2.0 as stated in the included license.
Original model copyright:
Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Users are responsible for reviewing and complying with the original model license.
- Downloads last month
- 80
4-bit