README.md

---
language:
- ar
- be
- bg
- bn
- cs
- cy
- da
- de
- el
- en
- es
- et
- fa
- fi
- fr
- gl
- hi
- hu
- it
- ja
- ka
- lt
- lv
- mk
- mr
- nl
- pl
- pt
- ro
- ru
- sk
- sl
- sr
- sv
- sw
- ta
- th
- tr
- uk
- ur
- vi
- zh
license: mit
library_name: transformers
metrics:
- bleu
pipeline_tag: audio-text-to-text
---

# Model Card for Ultravox

Ultravox is a multimodal Speech LLM built around a pretrained LLM (GLM 4.6 in this case) and a speech encoder ([whisper-large-v3-turbo](https://huggingface.co/openai/whisper-large-v3-turbo)) backbone.

See https://ultravox.ai for the GitHub repo and more information.


## Model Details

### Model Description

Ultravox is a multimodal model that can consume both speech and text as input (e.g., a text system prompt and voice user message). 
The input to the model is given as a text prompt with a special `<|audio|>` pseudo-token, and the model processor will replace this magic token with embeddings derived from the input audio. Using the merged embeddings as input, the model will then generate output text as usual. 

In the v0.7 series, the Ultravox model is trained on GLM 4.6, taking the lead on audio reasoning tasks over closed source models like gpt4o-audio, while retaining advantages in speech understanding from previous versions. 

- **Developed by:** Ultravox.ai
- **License:** MIT

### Model Sources

- **Repository:** https://ultravox.ai
- **Demo:** See repo

## Usage

Coming soon.


## Training Details

The model uses a pre-trained LLM (GLM, Llama, Gemma, Qwen, etc) backbone as well as the encoder part of [whisper-large-v3-turbo](https://huggingface.co/openai/whisper-large-v3-turbo).

The multi-modal adapter is trained, the Whisper encoder is fine-tuned, and the LLM is kept frozen.

We use a knowledge-distillation loss where Ultravox is trying to match the logits of the text-based LLM backbone.

### Training Data

The training dataset is a mix of ASR datasets, extended with continuations generated by Llama 3.1 8B, speech translation datasets, and noise datasets.

### Training Procedure

Supervised speech instruction finetuning via knowledge-distillation. For more info, see [training code in Ultravox repo](https://github.com/fixie-ai/ultravox/blob/main/ultravox/training/train.py).


#### Training Hyperparameters

- **Training regime:** BF16 mixed precision training
- **Hardware used:** 8x B200 GPUs

## Evaluation

Evaluations are conducted [big bench audio](https://huggingface.co/blog/big-bench-audio-release) (audio reasoning measured in accuracy), [VoiceBench](https://github.com/MatthewCYM/VoiceBench) (overall score averaged across multiple evaluations), as well as on covost2 (speech translation measured in BLEU), and LibriSpeech (speech recognition measured in WER).

### Audio Reasoning & General Understanding
| | **v0_7-glm-4_6 w/ reasoning** | **v0_7-glm-4_6 w/o reasoning** | v0_6-llama-3_3-70b | v0_6-gemma-3-27b | v0_6-qwen-3-32b | **gpt4o-audio** |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| **big bench audio** | **97.00** |  **91.80** | 85.48 | 83.84 | 84.22 | 82.80 |
| **voicebench overall** | **90.75** | **87.05** | 81.81 | – | – | 86.75 |


### Speech Translation & Recognition 
| | **v0_7-glm-4_6** | v0_6-llama-3_3-70b | v0_6-gemma-3-27b | v0_6-qwen-3-32b |
| --- | ---: | ---: | ---: | ---: |
| **covost2 en_ar** | **22.89** | 18.92 | 22.68 | 16.91 |
| **covost2 en_ca** | **41.48** | 38.73 | 39.67 | 33.63 |
| **covost2 en_de** | **35.44** | 33.69 | 34.76 | 31.09 |
| **covost2 es_en** | **42.88** | 41.39 | 41.11 | 41.20 |
| **covost2 ru_en** | **50.30** | 43.73 | 49.29 | 47.08 |
| **covost2 zh_en** | **23.85** | 17.81 | 20.88 | 22.24 |
| **librispeech** | **2.28** | 2.55 | 2.73 | 2.88 |

Note that custom system prompts are used with v0_7-glm-4_6 on VoiceBench and BigBenchAudio:
- BigBenchAudio in non-reasoning mode:
  ```
  You are a helpful assistant. Answer the question at the end of your response.
  ```
- VoiceBench in both reasoning and non-reasoning modes:
   ```
   You are a helpful assistant. When answering questions: For multiple choice questions (A/B/C/D options): End your response with "The answer is [X]" where X is the letter (A, B, C, or D)
   ```