File size: 3,674 Bytes
cc59e50 c52758e 665c4ed c52758e 665c4ed 187293d c52758e fd5935c c52758e 522930f c52758e 522930f 665c4ed c52758e 665c4ed c52758e 665c4ed c52758e 665c4ed c52758e 665c4ed c52758e 187293d c52758e 665c4ed c52758e fd5935c c52758e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 | ---
license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- omni-modal
- video-captioning
- instruction-following
- audio-visual
- qwen2.5-omni
---
# **OmniCaptioner-IF-3B**
[](https://github.com/NJU-LINK/OmniCap-IF)
[](https://nju-link.github.io/OmniCap-IF/)
[](https://arxiv.org/abs/2606.08572)
[](https://huggingface.co/datasets/NJU-LINK/OmniCap-IF-54K)
[](https://huggingface.co/datasets/NJU-LINK/OmniCap-IF)
## Quick Start
### Installation
```bash
conda create -n omnicap_if python=3.12
conda activate omnicap_if
pip install torch torchvision
pip install transformers==4.57.1
pip install accelerate
pip install flash-attn --no-build-isolation
pip install qwen-omni-utils[decord] -U
```
### Usage
```python
import torch
from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor
from qwen_omni_utils import process_mm_info
MODEL_ID = "NJU-LINK/OmniCaptioner-IF-3B"
VIDEO_PATH = "example_video.mp4"
INSTRUCTION = (
"Please describe this video in a Markdown table with columns "
"'Timestamp', 'Visual Action', and 'Audio Content'. Include precise timestamps "
"and mention the key audio-visual events."
)
MAX_PIXELS = 297920
VIDEO_MAX_PIXELS = 297920
model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
MODEL_ID,
torch_dtype=torch.bfloat16,
device_map="cuda",
attn_implementation="flash_attention_2"
)
processor = Qwen2_5OmniProcessor.from_pretrained(MODEL_ID)
model.disable_talker()
conversation = [
{
"role": "user",
"content": [
{"type": "text", "text": INSTRUCTION},
{
"type": "video",
"video": VIDEO_PATH,
"max_pixels": MAX_PIXELS,
"max_frames": 160,
"fps": 1.0,
"video_max_pixels": VIDEO_MAX_PIXELS
}
],
},
]
text = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)
audios, images, videos = process_mm_info(conversation, use_audio_in_video=True)
inputs = processor(
text=text,
audio=audios,
images=images,
videos=videos,
return_tensors="pt",
padding=True,
use_audio_in_video=True
)
inputs = inputs.to(model.device).to(model.dtype)
with torch.inference_mode():
text_ids = model.generate(
**inputs,
use_audio_in_video=True,
return_audio=False,
thinker_max_new_tokens=1536,
talker_max_tokens=1536
)
response = processor.decode(text_ids[0][inputs.input_ids[0].size(0):], skip_special_tokens=True)
print(response)
```
## Citation
```bibtex
@misc{wang2026omnicapifbenchmarkingimprovinginstruction,
title={OmniCap-IF: Benchmarking and Improving Instruction Following Abilities for Omni-Video Captioning},
author={Jiahao Wang and An Ping and Yanghai Wang and Yuanxing Zhang and Shihao Li and Hanyan Bian and Yichi Ren and Yize Zhang and Han Wang and Haowen Chen and Junze Li and Jiaqi Wang and Yiyang Hu and Zhuze Xu and Zijie Zhang and Jiaheng Liu},
year={2026},
eprint={2606.08572},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2606.08572},
}
```
|