# NVIDIA NemotronLabs VoiceChat 12B > ▶ **Hear it first.** Natural turn-taking, barge-in and live tool calling. | Sample | What to listen for | Player | |---|---|---| | Natural turn-taking | Smooth back-and-forth, ~450 ms response | | | Barge-in / interruption | User cuts in; model yields instantly | | | Tool calling live | Tools are called when needed | | # Model Overview ## Description: NVIDIA NemotronLabs VoiceChat is a **12B end-to-end, real-time speech full duplex (FD)** model for conversational AI that jointly performs streaming speech understanding and speech generation [1]. Unlike traditional cascaded stacks (ASR → LLM → TTS), this model achieves full duplex, real-time, seamless voice interaction in one unified architecture, eliminating the need for multiple models or API handoffs, thus reducing end-to-end latency. It sets new benchmarks by bringing open, robust, and highly natural conversation capabilities. Moreover, NVIDIA NemotronLabs VoiceChat is **the first open full-duplex model to support tool calling** while maintaining a natural conversation flow during tool execution. For each tool a specific “on-hold” message can be defined that will be spoken by the agent as soon as the LLM generates the text that will trigger the tool call and response. The model operates on audio signals, which are encoded using a fast conformer module. The resulting audio tokens are inputted into a Nemotron Nano V2 9B LLM backbone to predict text tokens, which are fed to a TTS decoder [2] to predict audio codes for generating the agent's speech. A separate output channel is used to predict tool calling scripts. NemotronLabs VoiceChat offers an unprecedented trade-off between intelligence and latency in the space of open-source voice agents, as highlighted by our benchmarking results [below](#testingevaluation-dataset). > This model is ready for research purposes only. ### Highlights | Parameters | Turn-taking latency | VoiceBench (open FD) | Tool calling | Architecture | |---|---|---|---|---| | 12B | ~450 ms | #2 | 1st open FD | 1 unified model | ### License/Terms of Use GOVERNING TERMS: The use of this model is governed by the [OpenMDW License Agreement, version 1.1](https://github.com/OpenMDW/OpenMDW/blob/main/1.1/LICENSE.OpenMDW-1.1). ### Use Case:
NemotronLabs VoiceChat is targeted for researchers, developers, and professionals in the field of natural language processing (NLP) and speech technology for purposes such as automatic speech recognition (ASR), text-to-speech synthesis (TTS), and voice assistant development. ### Deployment Geography Global ### Release Date:
July 16, 2026. ## Model Architecture **Architecture Type:** Hybrid Mamba/Transformer  ·  **Parameters:** 12B **Network Architecture:**
NemotronLabs VoiceChat uses: * Fast Conformer Speech Encoder from [Nemotron-Speech-Streaming-En-0.6b](https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b) * [NVIDIA Nemotron Nano v2 LLM backbone](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2) * NVIDIA TTS decoder and codec [2] * Separate output channel for tool-calling scripts ![NemotronLabs VoiceChat Architecture](./VoiceChat-v1-TC%20voicechatv1.png) ### Input | Property | Value | |---|---| | Input Type(s) | Text (prompt), Audio (user speech) | | Input Format | String, WAV/WebAudio | | Sample rate | 16 kHz | ### Output | Property | Value | |---|---| | Output Type(s) | Text (agent), Audio (agent speech), Text (user transcription) | | Output Format | String, WAV/WebAudio | | Sample rate | 22.05 kHz | ## How NemotronLabs VoiceChat Compares to Other Open FD Models | Model | Params | Tool calling | Big Bench Audio | |---|---|---|---| | **NemotronLabs VoiceChat** | 12B | ✔ | 37.0% | | PersonaPlex 7B | 7B | ✘ | 19.1% | | Moshi | 7B | ✘ | 4.4% | | Freeze-Omni | 7B | ✘ | 33.4% | ## Software Integration Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
**Runtime Engine:** vLLM
**Supported Hardware Microarchitecture Compatibility:**
* NVIDIA A100 * NVIDIA H100 * NVIDIA B200 * NVIDIA RTX-6000 **Preferred/Supported Operating System(s):** * Linux The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
## Model Version(s) * v1.0 ## Quickstart A tutorial is under preparation and will be shared shortly. Visit back soon! ## Training, Testing, and Evaluation Datasets: ### Training Dataset: **Data Modality:** Audio (speech) and Text
**Audio Training Data Size:** ~550k hours
VoiceChat has been trained on a blend of different datasets comprising both real audio datasets and synthetic speech datasets generated using various TTS systems on text training corpora, including: - Nemotron 5.5 pre-training and SFT text data - Brainy-mantis text data - Greteal AI v1 and v2 text data - [Ultrachat text data](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) - Blackwell studio recordings real speech data - [Fisher real speech data](https://catalog.ldc.upenn.edu/LDC2004S13) - [LibriVox](https://librivox.org/) - [LibriTTS](https://www.openslr.org/60/) - [HiFi-TTS](https://www.openslr.org/109/) - Riva Speakers: Internal Dataset - Publicly available internet scale data - [PromptTTS](https://speechresearch.github.io/prompttts/) - [VCTK](https://datashare.ed.ac.uk/handle/10283/3443) - [Voxmovies](https://mmai.io/datasets/voxmovies/) - [JL-Corpus](https://huggingface.co/datasets/CLAPv2/JL-Corpus) - Nemotron Nano v3 function calling data - Persona Plex training datasets **Data Collection Method by dataset:** Hybrid: Human, Synthetic, Automated.
**Labeling Method by dataset:** Automated.
### Testing/Evaluation Dataset: #### VoiceBench VoiceBench is a benchmark dataset developed to evaluate large language model (LLM)-based voice assistants, focusing on real-world spoken interactions rather than just text or clean speech recognition. It combines audio and text data. The dataset includes multiple subsets covering tasks like open-ended questions, multiple-choice QA, instruction following, and adversarial cases — sourced from both real human speech and synthetic text-to-speech examples. **Link:** [VoiceBench](https://github.com/MatthewCYM/VoiceBench)
**Data Collection Method by dataset:** Hybrid: Human, Synthetic, Automated.
**Labeling Method by dataset:** Automated.
**Properties:** VoiceChat is #2 amongst all open full-duplex models on [VoiceBench](https://arxiv.org/abs/2410.17196).
**Benchmark Scores:** | Metric | Value | |------------------------------------------|-------| | Text-output average accuracy | 55.1 | #### Artificial Analysis Big Bench Audio Big Bench Audio is a benchmark dataset used to evaluate the reasoning capabilities of audio-language and native speech-to-speech models. It adapts 1,000 questions from Big Bench Hard into audio format, testing whether models can understand spoken prompts and solve reasoning tasks directly from audio rather than relying only on text input. The benchmark is used by Artificial Analysis to compare speech models on audio reasoning performance across challenging question-answering tasks. **Link:** [Artificial Analysis Big Bench Audio](https://artificialanalysis.ai/speech-to-speech?utm_source=chatgpt.com)
**Data Collection Method by dataset:** Hybrid: Synthetic, Automated.
**Labeling Method by dataset:** Automated.
**Properties:** TODO
**Benchmark Scores:** | Metric | Value | |------------------------------------------|-------| | Text-output average accuracy | 37.0% | #### Full-Duplex-Bench 1.0 Full-Duplex-Bench 1.0 is a benchmark designed to evaluate the interactive capabilities of full-duplex spoken dialogue models like VoiceChat. It focuses on measuring natural, human-like conversational behaviors such as pause handling, backchanneling, smooth turn-taking, and user interruption management, using automatic metrics to provide consistent, reproducible assessments of model performance. **Link:** [Full-Duplex-Bench 1.0](https://arxiv.org/abs/2503.04721)
**Data Collection Method by dataset:** Hybrid: Human, Synthetic, Automated.
**Labeling Method by dataset:** Automated.
**Properties:** NemotronLabs VoiceChat is #2 amongst all open models on [FullDuplexBench 1.0](https://arxiv.org/abs/2503.04721).
**Benchmark Scores:** | Metric | Value | |------------------------------------------|-------| | Pause Handling(Synthetic): TOR↓ | 0.153 | | Pause Handling(Candor): TOR↓ | 0.255 | | Smooth Turn Taking: TOR↑ | 0.82 | | Smooth Turn Taking: Latency↓ | 448 ms | | User Interruption: TOR↑ | 1 | | User Interruption: Latency↓ | 480 ms | | User Interruption: GPT-4o↑ | 4.33 | #### AU Harness BFCL-v3 (tool calling) This is the subset of the [AU Harness benchmark](https://github.com/ServiceNow/AU-Harness) focused on the evaluation of tool calling within audio conversation. It was created by systematically converting the original [BFCL-v3](https://openreview.net/pdf?id=2GmDdhBdDk) textual instructions into spoken counterparts. **Link:** [AU Harness](https://github.com/ServiceNow/AU-Harness)
**Data Collection Method by dataset:** Hybrid: Human, Synthetic, Automated.
**Properties:** NemotronLabs VoiceChat is the first open full-duplex model to support tool calling while maintaining a natural conversation flow during tool execution.
**Benchmark Scores:** | Metric | Value | |--------------------|-------| | Simple | 58.5% | | Multiple | 62.5% | | Parallel | 42.5% | | Parallel Multiple | 27.5% | | Irrelevance | 89.6% | | Average | 56.1% | #### Full-Duplex-Bench v3 (tool calling) [Full-Duplex-Bench-v3 (FDB-v3)](https://arxiv.org/abs/2604.04847) is a benchmark for evaluating spoken language models under naturalistic speech conditions and multi-step tool use. **Link:** [Full-Duplex-Bench v3](https://daniellin94144.github.io/FDB-v3-demo/)
**Data Collection Method by dataset:** Human.
**Properties:** NemotronLabs VoiceChat is competitive with frontier models in terms of tool selection accuracy on this benchmark.
**Benchmark Scores:** | Metric | Value | |--------------------|-------| | Tool Selection | 82.5% | | Argument accuracy | 44.2% | | Pass@1 | 33% | # Inference: **Acceleration Engine:** vLLM
**Test Hardware:** NVIDIA H100
## Known Limitations NemotronLabs VoiceChat is trained with no more than 2-min long audio context windows. Conversational context beyond this window may not be retained reliably. NemotronLabs VoiceChat has been optimized to achieve a trade-off between general knowledge and natural conversation, hence it may not be as performant as the LLM backbone it is derived from, namely NVIDIA Nemotron Nano v2, in terms of knowledge, instruction-following capabilities and safety. The model can make reasoning errors or provide incorrect/incomplete information NemotronLabs VoiceChat was not explicitly trained for reasoning or alignment. Its performance on tasks requiring multi-step reasoning, arithmetic, or safety-aligned behavior may therefore be limited. Further, the model may not yet be able to systematically handle user backchannelling. As far as tool calling is concerned, we recommend a maximum of 5 tools per session; more tools may degrade performance. Also, the model cannot yet reliably call multiple tools simultaneously. Long tool responses may cause delays before the agent speaks (mitigate with on-hold messages). The user cannot interrupt the agent during tool calling execution. Further, in mixed conversations (general chat + tool requests), the model may answer from its own knowledge instead of calling the appropriate tool. Finally, the model is not suitable for noisy or highly reverberant environments, especially where background speech may occur. ## Ethical Considerations NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/). ## References: [1] [SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model](https://arxiv.org/abs/2505.15670)
[2] [Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models](https://arxiv.org/abs/2507.08128)
[3] [PersonaPlex: Voice and role control for full duplex conversational speech models](https://research.nvidia.com/labs/adlr/files/personaplex/personaplex_preprint.pdf)