Add model card

.gitattributes

@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,199 +1,369 @@
 ---
+license: other
+language:
+- en
+tags:
+- audio
+- video
+- image
+- audio-visual
+- multimodal
+- reasoning
+- video understanding
+- long-video
+- audio understanding
+- ASR
+- timestamp-grounding
+- instruction-tuned
+- dynamic-s2
+- CRTE
+- TAVIT
+pipeline_tag: video-text-to-text
 library_name: transformers
-tags: []
 ---
-
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
-
+# Model Overview
+
+<div align="center" style="display: flex; justify-content: center; align-items: center; text-align: center;">
+  <a href="https://avflamingo.github.io/" style="text-decoration: none; display: flex; align-items: center;">
+    <img src="static/avf_logo.png" alt="Audio-Visual Flamingo" width="150">
+  </a>
+</div>
+<div align="center" style="display: flex; justify-content: center; align-items: center; text-align: center;">
+  <h2>
+    Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos
+  </h2>
+</div>
+
+<div align="center" style="display: flex; justify-content: center; margin-top: 10px; flex-wrap: wrap; gap: 5px;">
+  <a href="https://avflamingo.github.io/"><img src="https://img.shields.io/badge/Project%20Page-AVF-DC2626" alt="Project Page"></a>
+  <a href="https://github.com/NVIDIA/audio-flamingo"><img src="https://img.shields.io/badge/GitHub-Audio--Flamingo-EA580C" alt="GitHub"></a>
+  <a href="https://github.com/NVIDIA/audio-flamingo/stargazers"><img src="https://img.shields.io/github/stars/NVIDIA/audio-flamingo.svg?style=social" alt="GitHub stars"></a>
+  <a href="https://github.com/lashahub/transformers/tree/add_AudioVisualFlamingo"><img src="https://img.shields.io/badge/Transformers-AVF%20fork-2563EB" alt="Transformers fork"></a>
+</div>
+
+<div align="center" style="display: flex; justify-content: center; margin-top: 10px; flex-wrap: wrap; gap: 5px;">
+  <a href="https://huggingface.co/nvidia/audio-visual-flamingo-hf">
+    <img src="https://img.shields.io/badge/HF-Checkpoint-ED5A22.svg" alt="Checkpoint">
+  </a>
+  <a href="https://huggingface.co/spaces/nvidia/audio-visual-flamingo">
+    <img src="https://img.shields.io/badge/HF-Gradio%20Demo-5F9EA0.svg" alt="Gradio Demo">
+  </a>
+  <img src="https://img.shields.io/badge/AV--Skills-7M%20caption%20and%20QA%20instances-F97316.svg" alt="AV-Skills">
+  <img src="https://img.shields.io/badge/TAVIT-Temporal%20AV%20Reasoning-DC2626.svg" alt="TAVIT">
+</div>
+
+Audio-Visual Flamingo (AVF) is a fully open audio-visual large language model for joint understanding and reasoning over audio, images, and videos. `nvidia/audio-visual-flamingo-hf` is the instruction-tuned checkpoint for multimodal understanding and text generation over long, complex, real-world videos.
+
+AVF is designed for settings where the answer depends on both what is visible and what is audible: speech, background sounds, music, visual actions, scene transitions, temporal order, and cross-modal alignment. This checkpoint is text-output only. The broader AVF system includes a streaming TTS component, but speech output is not part of this checkpoint.
+
+**This model is for non-commercial research purposes only.**
+
+## Description
+
+AVF extends the Audio Flamingo line from audio-only understanding into joint audio-visual reasoning. The model combines:
+
+- a SigLIP vision tower for images and sampled video frames
+- Dynamic-S2 visual preprocessing for high-resolution images and long videos
+- an AF-Whisper audio encoder for speech, sound, and music
+- separate 2-layer projectors for visual and audio features
+- temporal audio-visual interleaving with Constrained Rotary Time Embeddings (CRTE)
+- a Qwen2.5-7B-family decoder-only language backbone
+
+Unlike systems that process video frames and audio as independent streams, AVF aligns synchronized visual and audio chunks, interleaves them along the time axis, and feeds the fused sequence to the language model. This helps the model attend across co-occurring visual and auditory events.
 
 ## Model Details
 
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
+- **Developed by:** NVIDIA and University of Maryland
+- **Model type:** audio-visual large language model for conditional text generation
+- **Backbone:** Qwen2.5-7B-family decoder-only language model
+- **Modalities:** text, image, audio, video
+- **Language:** English-focused release
+- **License:** NVIDIA OneWay Noncommercial License
+
+## Model Sources
+
+- **Checkpoint:** https://huggingface.co/nvidia/audio-visual-flamingo-hf
+- **Demo:** https://huggingface.co/spaces/nvidia/audio-visual-flamingo
+- **Project page:** https://avflamingo.github.io/
+- **Audio Flamingo repository:** https://github.com/NVIDIA/audio-flamingo
+- **Transformers integration fork:** https://github.com/lashahub/transformers/tree/add_AudioVisualFlamingo
+
+## Best For
+
+- video question answering where both audio and visual evidence matter
+- detailed audio-visual captioning for real-world videos
+- temporal reasoning over event order, scene changes, and long-range context
+- audio-visual event alignment, such as identifying which sound matches a visual moment
+- reasoning over speech, environmental sounds, music, and visual actions together
+- image, audio, video, and text prompts in a single chat-template interface
+
+## Usage
+
+### Install
+
+AVF support is currently available through the AVF Transformers integration fork while the upstream Transformers PR is pending.
+
+```bash
+pip install --upgrade pip
+pip install "transformers[video] @ git+https://github.com/lashahub/transformers.git@add_AudioVisualFlamingo"
+pip install accelerate torch torchaudio librosa soundfile
+```
+
+For video and audio decoding, make sure `ffmpeg` and `libsndfile1` are available in your system environment.
+
+### Notes
+
+- Use `load_audio_in_video=True` when a video file should contribute both sampled frames and its audio track.
+- The processor expects audio at `16 kHz` mono internally and extracts `128`-bin log-mel features.
+- Audio is processed in `30`-second windows.
+- The released processor defaults to `num_video_frames=32`; examples below use `128` frames for stronger video coverage.
+- AVF is trained for long videos up to `15` minutes and `32K` context in long-context stages. Very long, dense videos can still be challenging.
+- Prompting matters. Ask explicitly for audio-visual evidence, timestamps, scene transitions, speech, sounds, or temporal order when those details matter.
+
+### Video + Audio From One Container
+
+```python
+import torch
+from transformers import AudioVisualFlamingoForConditionalGeneration, AutoProcessor
+
+model_id = "nvidia/audio-visual-flamingo-hf"
+dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32
+
+model = AudioVisualFlamingoForConditionalGeneration.from_pretrained(
+    model_id,
+    device_map="auto",
+    dtype=dtype,
+    load_audio_in_video=True,
+).eval()
+
+processor = AutoProcessor.from_pretrained(
+    model_id,
+    padding_side="left",
+    use_fast=False,
+    load_audio_in_video=True,
+    num_video_frames=128,
+    audio_chunk_length="max_3600",
+)
+
+conversation = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "video", "video": "/path/to/video.mp4"},
+            {
+                "type": "text",
+                "text": (
+                    "Describe the full video. Include the visual scene, spoken content, "
+                    "important sounds or music, and how the audio and visuals align over time."
+                ),
+            },
+        ],
+    }
+]
+
+inputs = processor.apply_chat_template(
+    conversation,
+    tokenize=True,
+    add_generation_prompt=True,
+    return_dict=True,
+).to(model.device)
+
+generated_ids = model.generate(
+    **inputs,
+    max_new_tokens=1024,
+    do_sample=False,
+)
+
+new_tokens = generated_ids[:, inputs["input_ids"].shape[1] :]
+print(processor.batch_decode(new_tokens, skip_special_tokens=True)[0])
+```
+
+### Image, Audio, Video, and Text Items
+
+AVF uses one chat-template interface for all modalities:
+
+```python
+conversation = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "image": "/path/to/reference_frame.jpg"},
+            {"type": "audio", "path": "/path/to/audio.wav"},
+            {"type": "video", "video": "/path/to/video.mp4"},
+            {
+                "type": "text",
+                "text": "What does the audio reveal that is not obvious from the image and video frames alone?",
+            },
+        ],
+    }
+]
+```
+
+### Prompt Guide
+
+| Task | Prompt pattern |
+| --- | --- |
+| Detailed AV captioning | `Generate a detailed caption of the full video, covering both visual events and the audio track.` |
+| Audio-visual alignment | `Explain how the audio and visuals align over time. Mention key timestamps or scene transitions.` |
+| Temporal order | `Arrange the major audio-visual events in chronological order and explain the evidence.` |
+| Spoken-content analysis | `Transcribe the speech, identify speakers if possible, and relate the speech to the visual scene.` |
+| Sound-grounded QA | `What sound occurs when the camera shows <event>, and what visual evidence supports it?` |
+| Visual-grounded audio QA | `Which visible action best explains the sound heard around <time/event>?` |
+| Long-video reasoning | `Answer using evidence from the whole video, not a single moment. Cite the relevant audio and visual cues.` |
+
+## Architecture
+
+<center><img src="static/avf_architecture.png" width="900"></center>
+
+At a high level, AVF contains:
+
+- **Vision encoder:** SigLIP vision tower with `448 x 448` image processing and Dynamic-S2 multi-scale tiling.
+- **Audio encoder:** AF-Whisper-style encoder with `128` mel bins, `16 kHz` input audio, and long-audio chunking.
+- **Projectors:** separate visual and audio MLP projectors that map encoder outputs into the LLM hidden size.
+- **Temporal fusion:** synchronized visual and audio chunks are interleaved over time and augmented with CRTE.
+- **Language model:** Qwen2.5-7B-family decoder-only backbone for multimodal reasoning and text generation.
+
+The released config uses:
+
+- `vision_tower_cfg.hidden_size = 1152`
+- `vision_tower_cfg.image_size = 448`
+- `sound_tower_cfg.d_model = 1280`
+- `sound_tower_cfg.encoder_layers = 32`
+- `llm_cfg.hidden_size = 3584`
+- `llm_cfg.num_hidden_layers = 28`
+- `llm_cfg.max_position_embeddings = 32768`
+- `num_video_frames = 32` by default
+- `interleaved_video_segment_duration = 30`
+- `load_audio_in_video = true`
+- `interleaved_vis_aud_in_video = true`
+
+## Training Summary
+
+AVF uses a three-stage training curriculum:
+
+| Stage | Main data mixture | Max input / context | Output checkpoint |
+| --- | --- | --- | --- |
+| Short-context SFT | AV-Skills-Short plus audio, music, image, video, and speech data | `5` min / `16K` tokens | AVF short-context model |
+| Long-context SFT | AV-Skills-Long plus downsampled short/audio/music/speech data | `15` min / `32K` tokens | **AVF-Instruct** |
+| CoT post-training | AV-Think reasoning triplets with SFT and GRPO | `15` min / `32K` tokens | AVF-Think |
 
-#### Metrics
+The AV-Skills data mixture includes:
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+<center><img src="static/avf_data_categories.png" width="760"></center>
 
-[More Information Needed]
+- `100K` hours of short video and `3.8M` AV-Skills-Short training instances
+- about `140K` hours of long video and `3.2M` long-video caption/QA instances
+- about `4.8M` QA pairs across the full AV-Skills collection
+- long-video skill categories such as needle-in-the-haystack reasoning, temporal reasoning, sub-scene understanding, holistic reasoning, counting, audio-visual referring, event alignment, event sequence reasoning, inference, comparative reasoning, and context understanding
+- AV-Think, with about `24K` temporally grounded audio-visual reasoning samples
 
-### Results
+Training uses bf16 precision on `512` NVIDIA H100 GPUs, with ZeRO-3, sequence parallelism for long-context stages, cosine learning-rate decay, warmup ratio `0.03`, and GRPO post-training for the AVF-Think variant.
 
-[More Information Needed]
+## Results
 
-#### Summary
+<center><img src="static/avf_results_radar.png" width="520"></center>
 
+ACC is higher-is-better and WER is lower-is-better.
 
+This repository hosts AVF-Instruct. AVF-Think entries summarize the post-trained reasoning variant where it was evaluated.
 
-## Model Examination [optional]
+| Area | Benchmark | AVF-Instruct | AVF-Think |
+| --- | --- | ---: | ---: |
+| Omni understanding | WorldSense ACC | **50.3** | 51.6 |
+| Omni understanding | DailyOmni ACC | **72.4** | 73.9 |
+| Omni understanding | OmniBench ACC | **48.5** | 50.6 |
+| Omni understanding | MMOU ACC | 56.9 | 60.2 |
+| Audio hallucination | CMM Hallucination ACC | **86.7** | - |
+| Audio reasoning | MMAR ACC | **60.1** | - |
+| Audio reasoning | MMSU ACC | **61.5** | - |
+| Audio reasoning | MMAU avg ACC | **73.49** | - |
+| Video understanding | Video-MME ACC, no subtitles / with subtitles | **70.7 / 71.2** | - |
+| Video understanding | LongVideoBench ACC | 60.1 | - |
+| Video understanding | MVHBench ACC | **71.7** | - |
+| ASR | LibriSpeech test-clean / test-other WER | **1.64 / 3.5** | - |
+| ASR | SPGISpeech WER | **2.8** | - |
+| ASR | TEDLIUM WER | 3.0 | - |
+| ASR | GigaSpeech WER | 10.2 | - |
+| ASR | VoxPopuli WER | **5.8** | - |
 
-<!-- Relevant interpretability work for the model goes here -->
+AVF-Instruct improves over OmniVinci on WorldSense, DailyOmni, Video-MME, MMAR, and several ASR settings, while AVF-Think further improves temporally grounded reasoning where evaluated.
 
-[More Information Needed]
+## Input and Output
 
-## Environmental Impact
+**Input**
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+- **Modalities:** text, image, audio, video, or mixed audio-visual video containers
+- **Image format:** common image formats supported by the Transformers image loader
+- **Audio format:** WAV/MP3/FLAC or audio track from video containers, decoded to mono `16 kHz`
+- **Video format:** local or remote video files supported by the Transformers video loader and PyAV/ffmpeg stack
+- **Recommended video setting:** `load_audio_in_video=True` for joint frame/audio understanding
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+**Output**
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+- **Type:** generated text
+- **Format:** UTF-8 string
+- **Typical max generation:** task dependent; examples use `512` to `1024` new tokens
 
-## Technical Specifications [optional]
+## Limitations
 
-### Model Architecture and Objective
+Known limitations:
 
-[More Information Needed]
+- AV-Skills is built from public datasets and open-internet videos, which can introduce source bias and possible overlap with prior training data.
+- Very long and highly dense videos remain difficult, especially when the relevant evidence is sparse, subtle, or far apart in time.
+- Current benchmarks do not fully capture open-ended real-world deployment.
+- The public checkpoint generates text only; it does not include the broader AVF streaming TTS component.
+- The Transformers integration is currently available through an AVF fork while upstream support is pending.
 
-### Compute Infrastructure
+Users should evaluate the model on their own target data before deployment, especially for safety-sensitive, privacy-sensitive, or high-stakes settings.
 
-[More Information Needed]
+## Ethical Considerations
 
-#### Hardware
+AVF can support beneficial applications such as accessibility, lecture and documentary understanding, audio description, media analysis, and multimodal research. It can also be misused for surveillance, deepfake analysis workflows, or multimodal disinformation at scale. The model and associated assets are released for non-commercial research use only, and downstream users are responsible for applying appropriate safeguards, privacy controls, and domain-specific evaluation.
 
-[More Information Needed]
+Please report security vulnerabilities or NVIDIA AI concerns through [NVIDIA's vulnerability disclosure channel](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
 
-#### Software
+## License / Terms of Use
 
-[More Information Needed]
+The model is released under the [NVIDIA OneWay Noncommercial License](static/NVIDIA-OneWay-Noncommercial-License_22Mar2022-academic.docx). Underlying datasets and components retain their original licenses and distribution terms. AVF uses Qwen2.5-7B as the base LLM and Apache 2.0 component, with AF-Whisper/OmniVinci-derived components under NVIDIA OneWay Noncommercial terms.
 
-## Citation [optional]
+## Deployment Geography
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+Global.
 
-**BibTeX:**
+## Intended Use
 
-[More Information Needed]
+This checkpoint is intended for researchers and developers exploring:
 
-**APA:**
+- joint audio-visual understanding
+- long-video question answering and captioning
+- temporal audio-visual reasoning
+- multimodal assistant research
+- evaluation of open audio-visual models
 
-[More Information Needed]
+It is not intended for production surveillance, biometric identification, rights enforcement, medical diagnosis, legal decision-making, or other high-stakes decisions without extensive independent validation and safeguards.
 
-## Glossary [optional]
+## Software Integration
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+**Runtime engine:** PyTorch / Hugging Face Transformers AVF integration fork
 
-[More Information Needed]
+**Recommended hardware:** NVIDIA A100/H100-class GPUs for long video inputs
 
-## More Information [optional]
+**Supported OS:** Linux
 
-[More Information Needed]
+The integration of foundation and fine-tuned models into AI systems requires use-case-specific testing. Developers should validate quality, latency, safety, privacy, and failure modes on representative data before deployment.
 
-## Model Card Authors [optional]
+## Citation
 
-[More Information Needed]
+```bibtex
+@misc{ghosh2026audiovisualflamingo,
+  title={Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos},
+  author={Sreyan Ghosh and Arushi Goel and Kaousheik Jayakumar and Lasha Koroshinadze and Nishit Anand and Zhifeng Kong and Siddharth Gururani and Sang-gil Lee and Jaehyeon Kim and Aya Aljafari and Chao-Han Huck Yang and Sungwon Kim and Ramani Duraiswami and Dinesh Manocha and Mohammad Shoeybi and Bryan Catanzaro and Ming-Yu Liu and Wei Ping},
+  year={2026},
+  howpublished={Technical report},
+  url={https://avflamingo.github.io/}
+}
+```
 
-## Model Card Contact
+## Acknowledgements
 
-[More Information Needed]
+Audio-Visual Flamingo builds on the Audio Flamingo series, AF-Whisper, OmniVinci, Qwen2.5, NVILA-style Dynamic-S2 processing, and the broader open audio, video, and multimodal learning community.