---
license: other
base_model: nvidia/Nemotron-3-Super-120B
tags:
  - nemotron
  - nemotron-h
  - sft
  - example
  - deepspeed
  - moe
library_name: transformers
pipeline_tag: text-generation
---

# Nemotron-3-Super-120B - Example SFT

This is an **example** supervised fine-tune (SFT) of NVIDIA's Nemotron-3-Super-120B
(`NemotronHForCausalLM`, a Mamba-2 / MoE hybrid with 512 routed experts, top-22).
It was produced as an end-to-end pipeline validation run, **not** a production model.

## Training summary

- **Base:** Nemotron-3-Super-120B (~120.7B params, bf16)
- **Method:** Full-parameter SFT (no LoRA), DeepSpeed ZeRO-3, 24x H200 over RDMA
- **Data:** Alpaca (2,000 examples), chat format
- **Steps:** 84 (one full epoch; 2000 / 24 ranks)
- **Loss:** 2.88 -> best 0.3534
- **Key fix:** `set_z3_leaf_modules(model, [NemotronHMoE])` to avoid a ZeRO-3
  parameter-gather deadlock caused by data-dependent MoE expert routing.

## Notes / caveats

- This checkpoint **omits the auxiliary `mtp` (multi-token-prediction) head**
  (dropped by `save_16bit_model`); it is not required for standard generation.
- Behaviour: shifts the base reasoning model toward concise, Alpaca-style direct answers.
- Use `trust_remote_code=True` when loading (custom Nemotron-H modeling code is included).

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained('tzchen07/Nemotron-Super-120B-sft-example',
                                         trust_remote_code=True, torch_dtype='bfloat16', device_map='auto')
t = AutoTokenizer.from_pretrained('tzchen07/Nemotron-Super-120B-sft-example', trust_remote_code=True)
```