Instructions to use audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF",
	filename="nemotron-cascade-2-attn-repeat-L5-Q4_K_M.gguf",
)

output = llm(
	"Once upon a time,",
	max_tokens=512,
	echo=True
)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF:Q4_K_M

Use Docker

docker model run hf.co/audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF:Q4_K_M

Ollama
How to use audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF with Ollama:
```
ollama run hf.co/audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF:Q4_K_M
```

Unsloth Studio new

How to use audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF to start chatting

Docker Model Runner
How to use audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF with Docker Model Runner:
```
docker model run hf.co/audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF:Q4_K_M
```

Lemonade

How to use audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.nemotron-cascade-2-attn-repeat-L5-GGUF-Q4_K_M

List all available models

lemonade list

Nemotron-Cascade-2-30B-A3B with Attention Repeat (Layer 5) - GGUF Q4_K_M

A modified GGUF of NVIDIA Nemotron-Cascade-2-30B-A3B that applies the RYS (Repeat Your Steps) layer duplication technique from shi3z/nemotron-cascade-2-attn-repeat-L5, but in GGUF format for use with ollama and llama.cpp.

What changed

Layer 5 (the first GQA Attention layer) is physically duplicated in the GGUF. The model goes from 52 to 53 layers, with layers 5 and 6 being identical attention blocks. All subsequent layers are shifted by 1.

This is the GGUF equivalent of shi3z's BlockRepeatWrapper approach. No weights were modified -- only duplicated.

Per-layer metadata

The GGUF includes correct 53-element per-layer arrays for attention.head_count_kv and feed_forward_length, so ollama correctly classifies each layer (Mamba2 / GQA Attention / MoE).

Architecture

53 layers total (was 52):

Mamba-2: 23 layers (SSM-based, sequential)
MoE: 23 layers (128 routed experts + 1 shared, top-6)
GQA Attention: 7 layers (was 6) at positions 5, 6, 13, 20, 27, 34, 43

Parameters: 31.6B total, ~3B active per token. Quantization: Q4_K_M.

Performance

Benchmarked on NVIDIA DGX Spark (GB10 Blackwell, 120GB unified LPDDR5X):

Metric	Base cascade-2	attn-repeat-L5
Generation speed	74.5 tok/s	62.5 tok/s
Prompt eval	81.7 tok/s	299.0 tok/s
Model size	23 GB	22.6 GB
VRAM loaded	26 GB	27 GB

~16% slower generation, ~3.7x faster prompt processing. Per shi3z's benchmarks, +6.7 percentage points on BBH-style reasoning.

Usage with ollama

# Download and create model
ollama create nemotron-cascade-2-attn-repeat -f Modelfile

# Run
ollama run nemotron-cascade-2-attn-repeat "Hello!"

Modelfile:

FROM nemotron-cascade-2-attn-repeat-L5-Q4_K_M.gguf
TEMPLATE {{ .Prompt }}
RENDERER nemotron-3-nano
PARSER nemotron-3-nano
PARAMETER temperature 1
PARAMETER top_p 0.95

How this GGUF was built

The GGUF was produced by a Python script that:

Reads the original nemotron-cascade-2 Q4_K_M GGUF from ollama
Duplicates all 5 tensors of block 5 (attn_q, attn_k, attn_v, attn_output, attn_norm) as block 6
Shifts all blocks >= 6 up by 1
Updates block_count from 52 to 53
Rebuilds the per-layer attention.head_count_kv and feed_forward_length arrays with the inserted entry

Source: built on DGX Spark running ollama 0.20.0, using gguf-py 0.18.0.

Credits

NVIDIA for Nemotron-Cascade-2-30B-A3B
shi3z for the attn-repeat-L5 discovery and RYS technique
Built with assistance from Claude Code (Anthropic)

License

NVIDIA Open Model License (inherited from base model).

Downloads last month: 26

GGUF

Model size

32B params

Architecture

nemotron_h_moe

Hardware compatibility

4-bit

Model tree for audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF

Base model

nvidia/Nemotron-Cascade-2-30B-A3B

Quantized

(30)

this model