Nemotron — quantized
Collection
Quantized GGUF and MLX packs of the Nemotron 3 model family (Nano, Nano-Omni, Super). • 60 items • Updated
KV-cache quantization without any fork (recommended, 2026): upstream llama.cpp/Ollama now cover this natively — use
-ctk q8_0 -ctv q8_0(half KV memory, negligible quality loss: perplexity +0.002–0.05) orquarter memory, ≈7.6% perplexity increase). In Ollama:-ctk q4_0 -ctv q4_0(OLLAMA_KV_CACHE_TYPE=q8_0withOLLAMA_FLASH_ATTENTION=1. Keep K and V types symmetric to stay on the fast fused Flash-Attention path. Since April 2026, mainline llama.cpp also applies Hadamard rotation to KV activations (PR #21038), which greatly improves low-bit KV quality (opt-out:LLAMA_ATTN_ROT_DISABLE=1).The RotorQuant/TurboQuant fork flow below is experimental/legacy: the TurboQuant llama.cpp PR was closed without merging (June 2026) and the fork is unmaintained relative to mainline. It is NOT required to use this model.
Nemotron-3-Nano-4B - RotorQuant MLX 8-bit
8-bit weight-quantized MLX version of nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 with the legacy RotorQuant KV-cache fork (superseded by upstream llama.cpp KV options). Optimized for Apple Silicon inference via the MLX framework. The dense hybrid Mamba-2 + Attention architecture supports up to 262K context length.
Approximate model size: ~4 GB
Model Specifications
| Property | Value |
|---|---|
| Base Model | nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 |
| Parameters | 4 billion (dense) |
| Architecture | Hybrid Mamba-2 + Attention (dense) |
| Context Length | 262,144 tokens (262K) |
| License | NVIDIA Open Model License (commercial use OK) |
| Weight Quantization | 8-bit (~4 GB) |
| KV-Cache Quantization | RotorQuant |
| Framework | MLX (Apple Silicon) |
Quickstart
from mlx_lm import load, generate
from rotorquant import IsoQuantCache
model, tokenizer = load("majentik/Nemotron-3-Nano-4B-RotorQuant-MLX-8bit")
prompt = "Explain the theory of relativity."
response = generate(model, tokenizer, prompt=prompt, max_tokens=512)
print(response)
About the RotorQuant / TurboQuant labels
RotorQuant and TurboQuant are this project's release labels, not distinct
quantization algorithms — for any given tier, both brand repos carry
byte-identical weights produced with the standard MLX / llama.cpp quantizers.
No brand-specific speedup is claimed or measured. The KV-cache fork these
labels originally referred to is legacy; for KV-cache memory savings use the
upstream options described above (-ctk/-ctv q8_0, OLLAMA_KV_CACHE_TYPE).
KV-Cache Quantization Comparison
| Method | Prefill Speed | Decode Speed | Memory Savings | Reference |
|---|---|---|---|---|
| TurboQuant | 1x (baseline) | 1x (baseline) | High | arXiv: 2504.19874 |
Memory Estimates (Nemotron-3-Nano-4B)
| Precision | Approximate Size | MLX Variant |
|---|---|---|
| BF16 (original) | ~8 GB | -- |
| 8-bit quantized | ~4 GB | This model |
| 4-bit quantized | ~2.3 GB | RotorQuant-MLX-4bit |
| 2-bit quantized | ~1.2 GB | RotorQuant-MLX-2bit |
Hardware Requirements
This model requires approximately 4 GB of unified memory. Recommended hardware:
- Apple M1 (8 GB+)
- Apple M2 (8 GB+)
- Apple M3 (8 GB+)
- Apple M4 (8 GB+)
- Any Apple Silicon Mac with 8 GB+ unified memory
See Also
- nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 -- Base model
- majentik/Nemotron-3-Nano-4B-RotorQuant-MLX-4bit -- MLX 4-bit variant
- majentik/Nemotron-3-Nano-4B-RotorQuant-MLX-2bit -- MLX 2-bit variant
- majentik/Nemotron-3-Nano-4B-TurboQuant-MLX-8bit -- TurboQuant MLX 8-bit variant
- RotorQuant GitHub
- MLX Framework
Quant trade-off (MLX lane)
| Bits | Approx size | Use case | Recommendation |
|---|---|---|---|
| 2-bit | ~1.0 GB | Aggressive quantization | Very low-RAM Macs |
| 3-bit | ~1.4 GB | Lossy but small | Low-RAM Macs |
| 4-bit | ~1.7 GB | Balanced default | Recommended for most Macs |
| 5-bit | ~2.0 GB | Higher fidelity | Quality-sensitive |
| 6-bit | ~2.4 GB | Approaching FP16 quality | High-fidelity |
| 8-bit | ~3.0 GB | Near-lossless reference | Fidelity-critical work |
(Current variant — 8bit — is bolded.)
Variants in this family
(Showing 13 sibling variants under majentik/nemotron3-nano-4b-*. The current variant — RotorQuant-MLX-8bit — is bolded.)
| Variant | Runtime | Approx size | Use case |
|---|---|---|---|
| RotorQuant-MLX-8bit | mlx-lm | ~4.7 GB | Apple Silicon reference |
- Downloads last month
- 115
Model size
1B params
Tensor type
BF16
·
U32 ·
Hardware compatibility
Log In to add your hardware
8-bit
Model tree for majentik/Nemotron-3-Nano-4B-RotorQuant-MLX-8bit
Base model
nvidia/NVIDIA-Nemotron-Nano-12B-v2-Base Finetuned
nvidia/NVIDIA-Nemotron-Nano-12B-v2 Finetuned
nvidia/NVIDIA-Nemotron-Nano-9B-v2 Finetuned
nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("majentik/Nemotron-3-Nano-4B-RotorQuant-MLX-8bit") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True)