thetom-ai/Nemotron-H-8B-Base-8K-Config-I-GGUF

Nemotron-H-8B-Base-8K — Config-I (TQ4_1S) GGUF

First Config-I / tq4_1s quantization of a Nemotron-H (Mamba-2 hybrid) model.

This is a TurboQuant Config-I weight quantization (TQ4_1S — RHT/Walsh-Hadamard-rotated 4-bit, 16-level Lloyd-Max codebook, dual half-block scales, ~5.0 bpw) of NVIDIA's nvidia/Nemotron-H-8B-Base-8K — a hybrid Mamba-2 / attention language model (52 layers: interleaved Mamba-2 SSM, self-attention, and squared-ReLU FFN blocks). Config-I / tq4_1s had previously only been validated on dense and MoE Transformer (Qwen) models — this is its first application to the Nemotron-H / Mamba-2-hybrid architecture family.

Sizes

Variant	Size	vs BF16	vs Q8_0
BF16 (source GGUF)	16.2 GB	—	—
Q8_0 (baseline)	8.0 GB	−51%	—
Config-I (tq4_1s)	4.8 GB	−70%	−40%

At 4.8 GB the whole model fits comfortably in the memory of a single consumer GPU / NVIDIA DGX Spark (GB10) / RTX 5090, with plenty of headroom for context.

Quality — perplexity (wikitext, ctx 512)

Variant	PPL	Δ vs Q8_0
Q8_0 (baseline)	6.8211	—
Config-I (tq4_1s)	7.0190	+0.198 (+2.9%)

A 2.9% perplexity increase for a 40%-smaller file (4-bit vs 8-bit) — the SSM recurrence (A/Δt/conv/D), attention, and squared-ReLU FFN all survive the rotated-4-bit quantization intact. This is, to our knowledge, the first evidence that TurboQuant's RHT-rotated 4-bit format transfers cleanly to a Mamba-2-hybrid architecture, not just to Transformers.

Intended use & limitations

This is a base (pretrained) model — completion / few-shot only. It has no chat template and is not instruction-tuned; prompt it with plain text continuations (e.g. "The capital of France is"), not chat turns. Context length is 8K (Nemotron-H-8B-Base-8K). Quantization is lossy: expect the +2.9% perplexity shown above vs the 8-bit baseline. eos_token_id is [2, 11] — pass both if you want generation to stop on either.

Running it

tq4_1s is a custom TurboQuant weight type and Nemotron-H is a hybrid Mamba-2 architecture, so this GGUF requires the TheTom/llama-cpp-turboquant fork (which implements both TQ4_1S decode and the nemotron_h / nemotron_h_moe inference graphs). Stock llama.cpp will not load it.

llama-cli -m Nemotron-H-8B-Base-8K-Config-I-tq4_1s.gguf -p "The capital of France is" -n 64

Backend notes

• CUDA (e.g. NVIDIA DGX Spark GB10, RTX 50-series) — the intended runtime, and verified on a DGX Spark (GB10): PPL 7.0037, coherent generation at ~24 tok/s decode / ~50 tok/s prefill, the whole 4.9 GB fitting comfortably alongside other resident models. TQ4_1S decodes via a warp-cooperative dequant-to-float path that applies the inverse-RHT before the matmul, so the default path is numerically exact (no flags needed).
• CPU — supported; verified (PPL 7.0190 on the CPU dequant-to-float path).
• Metal (Apple Silicon) — supported with the environment flag TQ_NO_ROTATE=1, e.g. TQ_NO_ROTATE=1 llama-cli -m Nemotron-H-8B-Base-8K-Config-I-tq4_1s.gguf -p "...". Verified PPL 7.0072 (matches CPU/CUDA to fp16 noise). The fork's default fused "rotate-act" TQ matmul (which pre-rotates activations in-place and uses a no-RHT weight dequant) produces NaN on Nemotron-H's squared-ReLU FFN on Metal; TQ_NO_ROTATE=1 routes TQ weights through the standard mul_mm with the exact inverse-RHT dequant instead. Transformer (Qwen) models are unaffected on the default path.

Reproduction

# 1. convert the HF safetensors to a bf16 GGUF (needs a converter with NemotronHForCausalLM support)
python convert_hf_to_gguf.py Nemotron-H-8B-Base-8K --outtype bf16 --outfile nemoh8b-bf16.gguf
# 2. Config-I quantize with the fork's llama-quantize
llama-quantize nemoh8b-bf16.gguf Nemotron-H-8B-Base-8K-Config-I-tq4_1s.gguf TQ4_1S
# (Q8_0 baseline for comparison: llama-quantize nemoh8b-bf16.gguf nemoh8b-Q8_0.gguf Q8_0)
# 3. perplexity check
llama-perplexity -m Nemotron-H-8B-Base-8K-Config-I-tq4_1s.gguf -f wikitext.txt -c 512

Attribution

• Base model: © NVIDIA, Nemotron-H-8B-Base-8K, under the NVIDIA Open Model License.
• Quantization method: TurboQuant (RHT-rotated extreme weight compression) + the tq4_1s / Config-I weight format from the llama-cpp-turboquant fork.