Instructions to use greghavens/nemotron-3-super-120b-bnb-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use greghavens/nemotron-3-super-120b-bnb-4bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="greghavens/nemotron-3-super-120b-bnb-4bit", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("greghavens/nemotron-3-super-120b-bnb-4bit", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("greghavens/nemotron-3-super-120b-bnb-4bit", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use greghavens/nemotron-3-super-120b-bnb-4bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "greghavens/nemotron-3-super-120b-bnb-4bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "greghavens/nemotron-3-super-120b-bnb-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/greghavens/nemotron-3-super-120b-bnb-4bit

SGLang

How to use greghavens/nemotron-3-super-120b-bnb-4bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "greghavens/nemotron-3-super-120b-bnb-4bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "greghavens/nemotron-3-super-120b-bnb-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "greghavens/nemotron-3-super-120b-bnb-4bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "greghavens/nemotron-3-super-120b-bnb-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use greghavens/nemotron-3-super-120b-bnb-4bit with Docker Model Runner:
```
docker model run hf.co/greghavens/nemotron-3-super-120b-bnb-4bit
```

NVIDIA-Nemotron-3-Super-120B-A12B — bitsandbytes NF4

4-bit NF4 quantization of unsloth/NVIDIA-Nemotron-3-Super-120B-A12B in bitsandbytes format. ~62 GB, down from 230 GB BF16.

Load it

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "greghavens/nemotron-3-super-120b-bnb-4bit",
    trust_remote_code=True,
    device_map="auto",
)

quantization_config is in config.json, so no BitsAndBytesConfig is needed and nothing is quantized at load time — the weights are already NF4.

What was quantized


format	NF4, blocksize 64, double quantization
compute dtype	bfloat16
storage dtype	bfloat16
quantized	every 2-D `nn.Linear` weight
left in BF16	`backbone.embeddings`, `lm_head`, Conv1D (3-D), all 1-D params (`A_log`, `D`, `dt_bias`, norms)

bnb_4bit_quant_storage=bfloat16 matters for FSDP: FSDP only shards float dtypes, and 4-bit weights default to uint8. With the default storage dtype FSDP leaves the base replicated on every rank instead of sharding it.

Precision vs the official Nemotron 3 Super checkpoints

NVIDIA's published precision table for the Super FP8 checkpoint, against this build:

component	NVIDIA Super	this build
Mamba GEMM (`in_proj` / `out_proj`)	FP8	NF4
Mamba 1D Conv	BF16	BF16
MoE GEMM (sparse + shared experts)	FP8	NF4
Embeddings / output layers	BF16	BF16
Attention GEMM (QKV + Out Projection)	BF16	NF4
Router	FP32	NF4
MoE Latent Projection GEMM	BF16	NF4

This build quantizes uniformly, so the attention projections, router, and MoE latent projections are NF4 here where NVIDIA keeps them at higher precision. For serving, the official FP8 and NVFP4 checkpoints follow NVIDIA's recipe.

How it was built

Streamed one shard at a time — download BF16 shard, quantize via bitsandbytes.functional.quantize_4bit, write the NF4 shard, delete the BF16 shard — keeping peak disk near the 62 GB output instead of the 292 GB a load-and-save needs.

License

Inherits the license of NVIDIA-Nemotron-3-Super-120B-A12B. Quantization does not alter licensing.

Downloads last month: 18

Safetensors

Model size

36B params

Tensor type

F32

BF16

Model tree for greghavens/nemotron-3-super-120b-bnb-4bit

Base model

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16

Finetuned

unsloth/NVIDIA-Nemotron-3-Super-120B-A12B

Quantized

(4)

this model