--- license: other language: - en tags: - audio - video - image - audio-visual - multimodal - reasoning - video understanding - long-video - audio understanding - ASR - timestamp-grounding - instruction-tuned - dynamic-s2 - CRTE - TAVIT pipeline_tag: video-text-to-text library_name: transformers --- # Model Overview

Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos

Audio-Visual Flamingo (AVF) is a fully open audio-visual large language model for joint understanding and reasoning over audio, images, and videos. `nvidia/audio-visual-flamingo-hf` is the instruction-tuned checkpoint for multimodal understanding and text generation over long, complex, real-world videos. AVF is designed for settings where the answer depends on both what is visible and what is audible: speech, background sounds, music, visual actions, scene transitions, temporal order, and cross-modal alignment. This checkpoint is text-output only. The broader AVF system includes a streaming TTS component, but speech output is not part of this checkpoint. **This model is for non-commercial research purposes only.** ## Description AVF extends the Audio Flamingo line from audio-only understanding into joint audio-visual reasoning. The model combines: - a SigLIP vision tower for images and sampled video frames - Dynamic-S2 visual preprocessing for high-resolution images and long videos - an AF-Whisper audio encoder for speech, sound, and music - separate 2-layer projectors for visual and audio features - temporal audio-visual interleaving with Constrained Rotary Time Embeddings (CRTE) - a Qwen2.5-7B-family decoder-only language backbone Unlike systems that process video frames and audio as independent streams, AVF aligns synchronized visual and audio chunks, interleaves them along the time axis, and feeds the fused sequence to the language model. This helps the model attend across co-occurring visual and auditory events. ## Model Details - **Developed by:** NVIDIA and University of Maryland - **Model type:** audio-visual large language model for conditional text generation - **Backbone:** Qwen2.5-7B-family decoder-only language model - **Modalities:** text, image, audio, video - **Language:** English-focused release - **License:** NVIDIA OneWay Noncommercial License ## Model Sources - **Checkpoint:** https://huggingface.co/nvidia/audio-visual-flamingo-hf - **Demo:** https://huggingface.co/spaces/nvidia/audio-visual-flamingo - **Project page:** https://avflamingo.github.io/ - **Audio Flamingo repository:** https://github.com/NVIDIA/audio-flamingo - **Transformers integration fork:** https://github.com/lashahub/transformers/tree/add_AudioVisualFlamingo ## Best For - video question answering where both audio and visual evidence matter - detailed audio-visual captioning for real-world videos - temporal reasoning over event order, scene changes, and long-range context - audio-visual event alignment, such as identifying which sound matches a visual moment - reasoning over speech, environmental sounds, music, and visual actions together - image, audio, video, and text prompts in a single chat-template interface ## Usage ### Install AVF support is currently available through the AVF Transformers integration fork while the upstream Transformers PR is pending. ```bash pip install --upgrade pip pip install "transformers[video] @ git+https://github.com/lashahub/transformers.git@add_AudioVisualFlamingo" pip install accelerate torch torchaudio librosa soundfile ``` For video and audio decoding, make sure `ffmpeg` and `libsndfile1` are available in your system environment. ### Notes - Use `load_audio_in_video=True` when a video file should contribute both sampled frames and its audio track. - The processor expects audio at `16 kHz` mono internally and extracts `128`-bin log-mel features. - Audio is processed in `30`-second windows. - The released processor defaults to `num_video_frames=32`; examples below use `128` frames for stronger video coverage. - AVF is trained for long videos up to `15` minutes and `32K` context in long-context stages. Very long, dense videos can still be challenging. - Prompting matters. Ask explicitly for audio-visual evidence, timestamps, scene transitions, speech, sounds, or temporal order when those details matter. ### Video + Audio From One Container ```python import torch from transformers import AudioVisualFlamingoForConditionalGeneration, AutoProcessor model_id = "nvidia/audio-visual-flamingo-hf" dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32 model = AudioVisualFlamingoForConditionalGeneration.from_pretrained( model_id, device_map="auto", dtype=dtype, load_audio_in_video=True, ).eval() processor = AutoProcessor.from_pretrained( model_id, padding_side="left", use_fast=False, load_audio_in_video=True, num_video_frames=128, audio_chunk_length="max_3600", ) conversation = [ { "role": "user", "content": [ {"type": "video", "video": "/path/to/video.mp4"}, { "type": "text", "text": ( "Describe the full video. Include the visual scene, spoken content, " "important sounds or music, and how the audio and visuals align over time." ), }, ], } ] inputs = processor.apply_chat_template( conversation, tokenize=True, add_generation_prompt=True, return_dict=True, ).to(model.device) generated_ids = model.generate( **inputs, max_new_tokens=1024, do_sample=False, ) new_tokens = generated_ids[:, inputs["input_ids"].shape[1] :] print(processor.batch_decode(new_tokens, skip_special_tokens=True)[0]) ``` ### Image, Audio, Video, and Text Items AVF uses one chat-template interface for all modalities: ```python conversation = [ { "role": "user", "content": [ {"type": "image", "image": "/path/to/reference_frame.jpg"}, {"type": "audio", "path": "/path/to/audio.wav"}, {"type": "video", "video": "/path/to/video.mp4"}, { "type": "text", "text": "What does the audio reveal that is not obvious from the image and video frames alone?", }, ], } ] ``` ### Prompt Guide | Task | Prompt pattern | | --- | --- | | Detailed AV captioning | `Generate a detailed caption of the full video, covering both visual events and the audio track.` | | Audio-visual alignment | `Explain how the audio and visuals align over time. Mention key timestamps or scene transitions.` | | Temporal order | `Arrange the major audio-visual events in chronological order and explain the evidence.` | | Spoken-content analysis | `Transcribe the speech, identify speakers if possible, and relate the speech to the visual scene.` | | Sound-grounded QA | `What sound occurs when the camera shows , and what visual evidence supports it?` | | Visual-grounded audio QA | `Which visible action best explains the sound heard around ?` | | Long-video reasoning | `Answer using evidence from the whole video, not a single moment. Cite the relevant audio and visual cues.` | ## Architecture

At a high level, AVF contains: - **Vision encoder:** SigLIP vision tower with `448 x 448` image processing and Dynamic-S2 multi-scale tiling. - **Audio encoder:** AF-Whisper-style encoder with `128` mel bins, `16 kHz` input audio, and long-audio chunking. - **Projectors:** separate visual and audio MLP projectors that map encoder outputs into the LLM hidden size. - **Temporal fusion:** synchronized visual and audio chunks are interleaved over time and augmented with CRTE. - **Language model:** Qwen2.5-7B-family decoder-only backbone for multimodal reasoning and text generation. The released config uses: - `vision_tower_cfg.hidden_size = 1152` - `vision_tower_cfg.image_size = 448` - `sound_tower_cfg.d_model = 1280` - `sound_tower_cfg.encoder_layers = 32` - `llm_cfg.hidden_size = 3584` - `llm_cfg.num_hidden_layers = 28` - `llm_cfg.max_position_embeddings = 32768` - `num_video_frames = 32` by default - `interleaved_video_segment_duration = 30` - `load_audio_in_video = true` - `interleaved_vis_aud_in_video = true` ## Training Summary AVF uses a three-stage training curriculum: | Stage | Main data mixture | Max input / context | Output checkpoint | | --- | --- | --- | --- | | Short-context SFT | AV-Skills-Short plus audio, music, image, video, and speech data | `5` min / `16K` tokens | AVF short-context model | | Long-context SFT | AV-Skills-Long plus downsampled short/audio/music/speech data | `15` min / `32K` tokens | **AVF-Instruct** | | CoT post-training | AV-Think reasoning triplets with SFT and GRPO | `15` min / `32K` tokens | AVF-Think | The AV-Skills data mixture includes:

- `100K` hours of short video and `3.8M` AV-Skills-Short training instances - about `140K` hours of long video and `3.2M` long-video caption/QA instances - about `4.8M` QA pairs across the full AV-Skills collection - long-video skill categories such as needle-in-the-haystack reasoning, temporal reasoning, sub-scene understanding, holistic reasoning, counting, audio-visual referring, event alignment, event sequence reasoning, inference, comparative reasoning, and context understanding - AV-Think, with about `24K` temporally grounded audio-visual reasoning samples Training uses bf16 precision on `512` NVIDIA H100 GPUs, with ZeRO-3, sequence parallelism for long-context stages, cosine learning-rate decay, warmup ratio `0.03`, and GRPO post-training for the AVF-Think variant. ## Results

ACC is higher-is-better and WER is lower-is-better. This repository hosts AVF-Instruct. AVF-Think entries summarize the post-trained reasoning variant where it was evaluated. | Area | Benchmark | AVF-Instruct | AVF-Think | | --- | --- | ---: | ---: | | Omni understanding | WorldSense ACC | **50.3** | 51.6 | | Omni understanding | DailyOmni ACC | **72.4** | 73.9 | | Omni understanding | OmniBench ACC | **48.5** | 50.6 | | Omni understanding | MMOU ACC | 56.9 | 60.2 | | Audio hallucination | CMM Hallucination ACC | **86.7** | - | | Audio reasoning | MMAR ACC | **60.1** | - | | Audio reasoning | MMSU ACC | **61.5** | - | | Audio reasoning | MMAU avg ACC | **73.49** | - | | Video understanding | Video-MME ACC, no subtitles / with subtitles | **70.7 / 71.2** | - | | Video understanding | LongVideoBench ACC | 60.1 | - | | Video understanding | MVHBench ACC | **71.7** | - | | ASR | LibriSpeech test-clean / test-other WER | **1.64 / 3.5** | - | | ASR | SPGISpeech WER | **2.8** | - | | ASR | TEDLIUM WER | 3.0 | - | | ASR | GigaSpeech WER | 10.2 | - | | ASR | VoxPopuli WER | **5.8** | - | AVF-Instruct improves over OmniVinci on WorldSense, DailyOmni, Video-MME, MMAR, and several ASR settings, while AVF-Think further improves temporally grounded reasoning where evaluated. ## Input and Output **Input** - **Modalities:** text, image, audio, video, or mixed audio-visual video containers - **Image format:** common image formats supported by the Transformers image loader - **Audio format:** WAV/MP3/FLAC or audio track from video containers, decoded to mono `16 kHz` - **Video format:** local or remote video files supported by the Transformers video loader and PyAV/ffmpeg stack - **Recommended video setting:** `load_audio_in_video=True` for joint frame/audio understanding **Output** - **Type:** generated text - **Format:** UTF-8 string - **Typical max generation:** task dependent; examples use `512` to `1024` new tokens ## Limitations Known limitations: - AV-Skills is built from public datasets and open-internet videos, which can introduce source bias and possible overlap with prior training data. - Very long and highly dense videos remain difficult, especially when the relevant evidence is sparse, subtle, or far apart in time. - Current benchmarks do not fully capture open-ended real-world deployment. - The public checkpoint generates text only; it does not include the broader AVF streaming TTS component. - The Transformers integration is currently available through an AVF fork while upstream support is pending. Users should evaluate the model on their own target data before deployment, especially for safety-sensitive, privacy-sensitive, or high-stakes settings. ## Ethical Considerations AVF can support beneficial applications such as accessibility, lecture and documentary understanding, audio description, media analysis, and multimodal research. It can also be misused for surveillance, deepfake analysis workflows, or multimodal disinformation at scale. The model and associated assets are released for non-commercial research use only, and downstream users are responsible for applying appropriate safeguards, privacy controls, and domain-specific evaluation. Please report security vulnerabilities or NVIDIA AI concerns through [NVIDIA's vulnerability disclosure channel](https://www.nvidia.com/en-us/support/submit-security-vulnerability/). ## License / Terms of Use The model is released under the [NVIDIA OneWay Noncommercial License](static/NVIDIA-OneWay-Noncommercial-License_22Mar2022-academic.docx). Underlying datasets and components retain their original licenses and distribution terms. AVF uses Qwen2.5-7B as the base LLM and Apache 2.0 component, with AF-Whisper/OmniVinci-derived components under NVIDIA OneWay Noncommercial terms. ## Deployment Geography Global. ## Intended Use This checkpoint is intended for researchers and developers exploring: - joint audio-visual understanding - long-video question answering and captioning - temporal audio-visual reasoning - multimodal assistant research - evaluation of open audio-visual models It is not intended for production surveillance, biometric identification, rights enforcement, medical diagnosis, legal decision-making, or other high-stakes decisions without extensive independent validation and safeguards. ## Software Integration **Runtime engine:** PyTorch / Hugging Face Transformers AVF integration fork **Recommended hardware:** NVIDIA A100/H100-class GPUs for long video inputs **Supported OS:** Linux The integration of foundation and fine-tuned models into AI systems requires use-case-specific testing. Developers should validate quality, latency, safety, privacy, and failure modes on representative data before deployment. ## Citation ```bibtex @misc{ghosh2026audiovisualflamingo, title={Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos}, author={Sreyan Ghosh and Arushi Goel and Kaousheik Jayakumar and Lasha Koroshinadze and Nishit Anand and Zhifeng Kong and Siddharth Gururani and Sang-gil Lee and Jaehyeon Kim and Aya Aljafari and Chao-Han Huck Yang and Sungwon Kim and Ramani Duraiswami and Dinesh Manocha and Mohammad Shoeybi and Bryan Catanzaro and Ming-Yu Liu and Wei Ping}, year={2026}, howpublished={Technical report}, url={https://avflamingo.github.io/} } ``` ## Acknowledgements Audio-Visual Flamingo builds on the Audio Flamingo series, AF-Whisper, OmniVinci, Qwen2.5, NVILA-style Dynamic-S2 processing, and the broader open audio, video, and multimodal learning community.