fixie-ai
/

ultravox-v0_7-glm-4_6

@@ -1,199 +1,146 @@
 ---
 library_name: transformers
-tags: []
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
 ## Model Details
 ### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
 ### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
 ### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
 #### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
 ## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
+language:
+- ar
+- be
+- bg
+- bn
+- cs
+- cy
+- da
+- de
+- el
+- en
+- es
+- et
+- fa
+- fi
+- fr
+- gl
+- hi
+- hu
+- it
+- ja
+- ka
+- lt
+- lv
+- mk
+- mr
+- nl
+- pl
+- pt
+- ro
+- ru
+- sk
+- sl
+- sr
+- sv
+- sw
+- ta
+- th
+- tr
+- uk
+- ur
+- vi
+- zh
+license: mit
 library_name: transformers
+metrics:
+- bleu
+pipeline_tag: audio-text-to-text
 ---
+# Model Card for Ultravox
+Ultravox is a multimodal Speech LLM built around a pretrained LLM (GLM 4.6 in this case) and a speech encoder ([whisper-large-v3-turbo](https://huggingface.co/openai/whisper-large-v3-turbo)) backbone.
+See https://ultravox.ai for the GitHub repo and more information.
 ## Model Details
 ### Model Description
+Ultravox is a multimodal model that can consume both speech and text as input (e.g., a text system prompt and voice user message).
+The input to the model is given as a text prompt with a special `<|audio|>` pseudo-token, and the model processor will replace this magic token with embeddings derived from the input audio. Using the merged embeddings as input, the model will then generate output text as usual.
+In the v0.7 series, the Ultravox model is trained on GLM 4.6, taking the lead on audio reasoning tasks over closed source models like gpt4o-audio, while retaining advantages in speech understanding from previous versions.
+- **Developed by:** Ultravox.ai
+- **License:** MIT
+### Model Sources
+- **Repository:** https://ultravox.ai
+- **Demo:** See repo
+## Usage
+Think of the model as an LLM that can also hear and understand speech. As such, it can be used as a voice agent, and also to do speech-to-speech translation, analysis of spoken audio, etc.
+To use the model, try the following:
+```python
+# pip install transformers peft librosa
+import transformers
+import numpy as np
+import librosa
+pipe = transformers.pipeline(model='model_placeholder', trust_remote_code=True)
+path = "<path-to-input-audio>"  # TODO: pass the audio here
+audio, sr = librosa.load(path, sr=16000)
+turns = [
+  {
+    "role": "system",
+    "content": "You are a friendly and helpful character. You love to answer questions for people."
+  },
+]
+pipe({'audio': audio, 'turns': turns, 'sampling_rate': sr}, max_new_tokens=30)
+```
+## Training Details
+The model uses a pre-trained LLM (GLM, Llama, Gemma, Qwen, etc) backbone as well as the encoder part of [whisper-large-v3-turbo](https://huggingface.co/openai/whisper-large-v3-turbo).
+The multi-modal adapter is trained, the Whisper encoder is fine-tuned, and the LLM is kept frozen.
+We use a knowledge-distillation loss where Ultravox is trying to match the logits of the text-based LLM backbone.
 ### Training Data
+The training dataset is a mix of ASR datasets, extended with continuations generated by Llama 3.1 8B, speech translation datasets, and noise datasets.
 ### Training Procedure
+Supervised speech instruction finetuning via knowledge-distillation. For more info, see [training code in Ultravox repo](https://github.com/fixie-ai/ultravox/blob/main/ultravox/training/train.py).
 #### Training Hyperparameters
+- **Training regime:** BF16 mixed precision training
+- **Hardware used:** 8x B200 GPUs
 ## Evaluation
+Evaluations are conducted on covost2 (speech translation measured in BLEU), big bench audio (audio reasoning measured in accuracy), as well as [VoiceBench](https://github.com/MatthewCYM/VoiceBench) .
+### Audio Reasoning & General Understanding
+| | **v0_7-glm (reasoning / no reasoning)** | v0_6-llama-3_3-70b | v0_6-gemma-3-27b | **gpt4o-audio** |
+| --- | ---: | ---: | ---: | ---: |
+| **big bench audio** | **97.00 / 91.80** | 85.48 | 83.84 | 82.80 |
+| **voicebench overall** | **90.75 / 87.05** | 81.81 | – | 86.75 |
+### Speech Understanding
+| | **v0_7-glm** | v0_6-llama-3_1-8b | v0_6-llama-3_3-70b | v0_6-gemma-3-27b | v0_6-qwen-3-32b |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| **covost2 en_ar** | **22.89** | 12.94 | 18.92 | 22.68 | 16.91 |
+| **covost2 en_ca** | **41.48** | 31.47 | 38.73 | 39.67 | 33.63 |
+| **covost2 en_de** | **35.44** | 28.66 | 33.69 | 34.76 | 31.09 |
+| **covost2 es_en** | **42.88** | 40.36 | 41.39 | 41.11 | 41.20 |
+| **covost2 ru_en** | **50.30** | 42.41 | 43.73 | 49.29 | 47.08 |
+| **covost2 zh_en** | **23.85** | 17.24 | 17.81 | 20.88 | 22.24 |
+| **librispeech (WER)** | 2.28 | **2.09** | 2.55 | 2.73 | 2.88 |