README.md

---
license: other
license_name: nvidia-open-model-license
license_link: https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf
base_model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16
tags: [nemotron, multimodal, mamba2, moe, quantized, rotorquant, mlx, apple-silicon,
  mlx-lm, text-tower-only]
library_name: mlx
pipeline_tag: text-generation
language: [en]
datasets: [nvidia/Nemotron-Image-Training-v3]
inference: false
---

> [!TIP]
> **KV-cache quantization without any fork (recommended, 2026):** upstream
> llama.cpp/Ollama now cover this natively — use `-ctk q8_0 -ctv q8_0`
> (~half KV memory, negligible quality loss: perplexity +0.002–0.05) or
> `-ctk q4_0 -ctv q4_0` (~quarter memory, ≈7.6% perplexity increase). In
> Ollama: `OLLAMA_KV_CACHE_TYPE=q8_0` with `OLLAMA_FLASH_ATTENTION=1`. Keep
> K and V types symmetric to stay on the fast fused Flash-Attention path.
> Since April 2026, mainline llama.cpp also applies Hadamard rotation to
> KV activations ([PR #21038](https://github.com/ggml-org/llama.cpp/pull/21038)),
> which greatly improves low-bit KV quality (opt-out:
> `LLAMA_ATTN_ROT_DISABLE=1`).
>
> The RotorQuant/TurboQuant fork flow below is **experimental/legacy**: the
> TurboQuant llama.cpp PR was closed without merging (June 2026) and the fork
> is unmaintained relative to mainline. It is NOT required to use this model.
<!-- kv-upstream-note -->

# Nemotron-3-Nano-Omni-30B-A3B-Reasoning - RotorQuant MLX 2-bit

MLX 2-bit quantization of the **text tower** of `Nemotron-3-Nano-Omni-30B-A3B-Reasoning` (`nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16`)
with RotorQuant weight method. Apple Silicon native via `mlx-lm`.

This variant covers the LLM backbone only. Vision (CRADIO v4-H) + audio (Parakeet-TDT-0.6B-v2)
encoders are NOT included — MLX-VLM Nemotron-Omni model class is **pending upstream support**
(no PR observed as of 2026-05-04). For multimodal inference, use the GGUF variants with
`llama-mtmd-cli` instead.

For the matched-KV stack — RotorQuant weights + RotorQuant KV-cache modifier —
For the runtime KV-cache modifier itself, see
`majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-RotorQuant`.

## Quickstart

```python
# Today (mlx-lm 0.31.x): the NemotronH_Nano_Omni_Reasoning_V3 model class
# is not yet registered in mlx-lm. The cell below is the API shape that WILL
# work once upstream lands the class (track ml-explore/mlx-lm#386).

from mlx_lm import load, generate

model, tokenizer = load("majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-RotorQuant-MLX-2bit")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Solve: 17 * 23"}],
    add_generation_prompt=True,
    enable_thinking=False,  # set True to enable extended reasoning (default)
)

response = generate(
    model, tokenizer,
    prompt=prompt,
    max_tokens=512,
    sampler=lambda x: x.argmax(axis=-1),  # or use mlx_lm.sample_utils.make_sampler(temp=0.6, top_p=0.95)
)
print(response)
```

> ⚠️ This variant covers the **text tower only**. For multimodal inference (vision + audio + video), use the GGUF variants with `llama-mtmd-cli` — see the GGUF cards in this family.

## Modality matrix

| Modality | Encoder | Quantization in this variant |
|---|---|---|
| Text | LLM backbone (Mamba-2 + Transformer hybrid Sparse MoE) | per the variant suffix |
| Image | CRADIO v4-H | **BF16** (kept full-precision in every non-GGUF variant; GGUF uses mmproj-F16 split file) |
| Audio | Parakeet-TDT-0.6B-v2 | **BF16** (same rationale) |
| Video | Parakeet-TDT-0.6B-v2 + frame sampler | **BF16** (≤ 2 min, 256 frames @ 2 FPS) |

NVIDIA's official FP8 / NVFP4 recipe keeps both encoders + the cross-modal
MLP projectors in BF16 to preserve multimodal accuracy. We follow that
convention in every quantized variant we ship.

## Runtime quirks

### MLX-LM (text-only)

This variant covers the LLM backbone only. Vision + audio encoders
are NOT included — MLX-VLM Nemotron-Omni model class is
**pending upstream support** (no PR observed as of 2026-05-04).

Use the `mlx_lm.generate` API; `enable_thinking` is a runtime flag
(see below).

### Reasoning mode

`enable_thinking` defaults to `True`. To disable extended reasoning
(e.g., for latency-sensitive cases), pass `enable_thinking=False`
to the chat template / generate call. No separate "no-think"
variant card exists — this is a runtime flag, not a model variant.

## Quant trade-off (MLX lane)

| Bits | Approx size | Use case | Recommendation |
|---|---|---|---|
| **2-bit** | ~8.1 GB | Aggressive quantization | **Very low-RAM Macs** |
| 3-bit | ~11 GB | Lossy but small | Low-RAM Macs |
| 4-bit | ~13 GB | Balanced default | Recommended for most Macs |
| 5-bit | ~16 GB | Higher fidelity | Quality-sensitive |
| 6-bit | ~19 GB | Approaching FP16 quality | High-fidelity |
| 8-bit | ~24 GB | Near-lossless reference | Fidelity-critical work |

(Current variant — **2bit** — is bolded.)

## Variants in this family

(Showing 56 sibling variants under `majentik/nemotron3-nano-omni-30b-*`. The current variant — `RotorQuant-MLX-2bit` — is **bolded**.)

| Variant | Runtime | Approx size | Use case |
|---|---|---|---|
| **RotorQuant-MLX-2bit** | mlx-lm | ~9.6 GB | Apple Silicon, smallest |

## About the RotorQuant / TurboQuant labels

RotorQuant and TurboQuant are this project's **release labels**, not distinct
quantization algorithms — for any given tier, both brand repos carry
byte-identical weights produced with the standard MLX / llama.cpp quantizers.
No brand-specific speedup is claimed or measured.