---
library_name: transformers
license: other
license_name: nvidia-oneway-noncommercial-license
license_link: https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B/blob/main/license/NVIDIA-OneWay-Noncommercial-License.docx
pipeline_tag: text-generation
track_downloads: true
language:
- en
tags:
- nvidia
- nemotron-labs-audex
- reasoning
- general-purpose
- SFT
- RL
- audio-language-modeling
- audio-understanding
- text-to-speech
- text-to-audio
- speech-recognition
- speech-translation
---
# Nemotron-Labs-Audex-30B-A3B
[](https://arxiv.org/abs/2607.05196)
[](https://huggingface.co/collections/nvidia/nemotron-labs-audex)
## Introduction
We're excited to introduce [Nemotron-Labs-Audex-30B-A3B](https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B), a unified audio-text LLM built on [Nemotron-Cascade-2-30B-A3B](https://huggingface.co/nvidia/Nemotron-Cascade-2-30B-A3B), a strong text-only MoE LLM with 30B MoE model with 3B activated parameters. Audex-30B-A3B extends the vocabulary for discrete audio tokens used for speech and general audio outputs, as well as an audio encoder for speech and general audio inputs. Audex-30B-A3B delivers strong abilities on audio tasks (audio understanding, speech recognition
and translation, text-to-speech, audio generation, and speech-to-speech generation) while preserving
very compelling reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only LLM
backbone with marginal or no regression. Audex-30B-A3B operates in both **thinking** and **instruct** (non-thinking) modes.
### Model Architecture
### Templates
### Multi-Stage-SFT and Cascaded-RL Pipelines
## Quick Start
- Audex-30B-A3B follows the ChatML template and supports both thinking and instruct (non-thinking) modes. Reasoning content is enclosed within `` and `` tags. To activate the instruct (non-thinking) mode, we prepend `` to the beginning of the assistant’s response.
- Audex-30B-A3B supports up to a 1M-token context length.
- Audex-30B-A3B follows Nemotron-Cascade-2 on text evaluation.
- Audex-30B-A3B has different recommended inference setups per audio-related task as described below.
### Environment
We use **vLLM 0.20.0 container image**: [vllm/vllm-openai:v0.20.0-cu129](https://hub.docker.com/layers/vllm/vllm-openai/v0.20.0-cu129/images/sha256-f4ace3494896eeda800dee284d1fc42ca7f5626f31ceae8e24d1383d770567c2)
- **vLLM inference** — text-only reasoning, text-to-speech, text-to-audio, and audio understanding / speech recognition / speech translation: runs on **vLLM 0.20.0**.
- **Hugging Face / transformers inference** — requires transformers >= 4.53.0 (tested with 4.53.3) and also works with transformers >= 5.0. This additionally needs `mamba-ssm` and `causal-conv1d` (build against your CUDA toolchain, e.g. ```pip install --no-build-isolation causal-conv1d==1.6.2.post1 mamba-ssm==2.3.2.post1```).
**Audio extras:** ```vllm/vllm-openai:v0.20.0``` image does not include audio codecs. This command installs audio-related packages: ```python3 -m pip install "vllm[audio]"```.
### Audio QA Inference
Audio QA includes audio understanding, speech recognition, and speech translation (see templates in Introduction).
- **vLLM (recommended)** — offline ```LLM.generate``` and an OpenAI-compatible ```audio_url``` server.
- **Hugging Face / transformers** — requires transformers >= 4.53.0 (we tested with 4.53.3), and also works with transformers >= 5.
#### Inputs
To prepare inputs, create a JSON file in the following format with the ```\n``` placeholder:
```json
[
{
"id": "sample_0",
"sound": "/path/to/audio_0.wav",
"conversations": [
{"from": "human", "value": "\nDescribe the audio in detail."},
{"from": "gpt", "value": "N/A"}
]
},
{
"id": "sample_1",
"sound": "/path/to/audio_1.wav",
"conversations": [
{"from": "human", "value": "\n{prompt}"},
{"from": "gpt", "value": "N/A"}
]
},
...
]
```
#### Inference recipes
- For audio understanding, we use top_p=0.9 and temperature=0.7.
- For speech recognition and translation, we use greedy sampling.
#### vLLM (recommended)
To install environments:
```bash
python3 -m pip install "vllm[audio]" # audio input decoding; skip if your image already bundles it (see Environment)
pip install -e inference_scripts_vllm/audioqa_scripts --no-deps --no-build-isolation
```
- Offline (JSON in → JSONL out):
```bash
python inference_scripts_vllm/audioqa_scripts/run_audioqa_vllm.py \
--model-path "$(pwd)/checkpoint_folder_full" \
--input-json ./inputs.json \
--output-jsonl ./audioqa_outputs/results.jsonl \
--tensor-parallel-size 8
```
- OpenAI-compatible server + client:
```bash
bash inference_scripts_vllm/audioqa_scripts/serve_audioqa_vllm.sh "$(pwd)/checkpoint_folder_full" 8000
python inference_scripts_vllm/audioqa_scripts/client_audioqa.py --audio /path/to/audio.wav --prompt "Describe this audio."
```
#### Hugging Face / transformers
- Example inference script: ```bash inference_scripts_hf/inference_example.sh```.
- Task instruction examples: audio understanding — a question about the audio; speech recognition — ```Transcribe the speech in the input audio.\n```; speech translation — a translation instruction such as ```Translate the speech in the input audio into English.\n```.
### Audio Generation Inference
Audio generation includes text-to-speech and text-to-audio generation.
First, prepare vLLM inference using ```bash model_conversion_scripts/prepare_audiogen_vllm_checkpoint.sh``` (which only creates symlinks of safetensors under ```checkpoint_folder_audiogen```).
#### Text-to-audio (TTA)
Download **XCodec1** ([hf-audio/xcodec-hubert-general-balanced](https://huggingface.co/hf-audio/xcodec-hubert-general-balanced)) via
```
hf download hf-audio/xcodec-hubert-general-balanced --local-dir /path/to/xcodec1
```
Prepare a folder ```/path/to/caption_txt_dir/``` with all .txt files where each contains one caption. Run ```cd inference_scripts_vllm/audiogen_scripts/``` and
set ```--tensor-parallel-size``` to the number of GPUs. Run
```
XCODEC1_PATH=/path/to/xcodec1 python3 run_audio_gen_vllm_rvq_logit_mask.py \
--task tta \
--model-path $(pwd)/../../checkpoint_folder_audiogen/ \
--dataset-path /path/to/caption_txt_dir/ \
--output-dir ../../tta_outputs/dataset_name/ \
--tensor-parallel-size 8 \
--temperature 1.0 \
--top-k 80 \
--max-tokens 2048 \
--cfg-scale 3.0 \
--cfg-pairs-per-batch 2
```
Finally (optional), apply the 48 kHz enhancement VAE to the generated waveforms; see ```enhancement_VAE/README.md```.
#### Text-to-speech (TTS)
we recommend using the standalone Audex causal speech decoder in ```audex_causal_speech_decoder``` (default).
```
./run_tts_vllm.sh --transcription "The weather is so good, and I want to enjoy the beautiful morning in the park." \
--output-dir ./tts_outputs --utt-id the_weather_is_so_good
```
Alternatively, users can download the original XCodec2 via [this repo](https://github.com/zhenye234/X-Codec-2.0) and decode the tokens after full generation; this has better quality but is not streaming.
### Text-Only Reasoning
The text reasoning follows [Nemotron-Cascade-2-30B-A3B](https://huggingface.co/nvidia/Nemotron-Cascade-2-30B-A3B).
- To create the checkpoint that completely matches their setup, run ```python model_conversion_scripts/convert_full_HF_to_textonly_HF.py``` to remove the audio-related vocabularies.
- Then, the text-only inference completely follows Nemotron-Cascade-2-30B-A3B. See a simple reasoning example with ```cd inference_scripts_vllm/textonly_scripts/; python run_text_vllm_example.py --model-path $(pwd)/../../checkpoint_folder_textonly```.
- Note: you could also avoid model conversion by using ```sampling_params = SamplingParams(allowed_token_ids=list(range(131072)))``` in vLLM inference to mask the audio tokens, although we did not thoroughly test this approach.
### Demo: speech-to-speech interaction with text reasoning
See `inference_scripts_vllm/unified_s2s_scripts/README.md`.
## Reproducibility
The benchmark numbers below use the following setups:
- **Text-to-speech (TTS):** the original non-streaming XCodec2 decoder.
- **Text-to-audio (TTA):** XCodec1 followed by the enhancement VAE.
- **Text:** same as Nemotron-Cascade-2.
- **Audio understanding:** transformers 4.53.3 and Megatron-LM's native inference.
## Detailed Benchmark Results
### Text Results
### Text-To-Speech Results
### Text-To-Audio Results
### Speech Recognition and Translation Results
### Audio Understanding Results
### Speech-To-Speech Results
## Release Date
June 8, 2026
## License
Your use of this model is governed by the [NVIDIA Oneway Noncommercial License](license/NVIDIA-OneWay-Noncommercial-License.docx)
## Citation
```
@article{Nemotron-Labs-Audex,
title={Unified Audio Intelligence Without Regressing on Text Intelligence},
author={Kong, Zhifeng and Lee, Sang-gil and Kim, Jaehyeon and Wang, Boxin and Liu, Zihan and Kim, Sungwon and Chen, Yang and Goel, Arushi and Roy, Rajarshi and Dai, Wenliang and Yang, Zhuolin and Chen, Yangyi and Jiang, Dongfu and Ghosh, Sreyan and Rintamaki, Tuomas and Tao, Andrew and Raiman, Jonathan and Shoeybi, Mohammad and Catanzaro, Bryan and Ping, Wei},
year={2026}
}
```