Update model card README

README.md

@@ -1,52 +1,119 @@
 ---
-frameworks:
-- Pytorch
 license: apache-2.0
-tags: []
-tasks:
-- image-text-to-text
-
-#model-type:
-##如 gpt、phi、llama、chatglm、baichuan 等
-#- gpt
+library_name: transformers
+pipeline_tag: image-text-to-text
+tags:
+- omni-modal
+- video-captioning
+- instruction-following
+- audio-visual
+- qwen2.5-omni
+---
 
-#domain:
-##如 nlp、cv、audio、multi-modal
-#- nlp
+# **OmniCaptioner-IF-3B**
 
-#language:
-##语言代码列表 https://help.aliyun.com/document_detail/215387.html?spm=a2c4g.11186623.0.0.9f8d7467kni6Aa
-#- cn 
+[![GitHub](https://img.shields.io/badge/GitHub-OmniCap--IF-181717?logo=github&logoColor=white)](https://github.com/wang-jiahao/OmniCap-IF)
+ 
+[![Project Page](https://img.shields.io/badge/Project%20Page-OmniCap--IF-1B2838?logo=githubpages&logoColor=white)](https://nju-link.github.io/OmniCap-IF/)
+ 
+[![Paper](https://img.shields.io/badge/arXiv-2606.xxxxx-b31b1b)](https://arxiv.org/abs/2606.xxxxx)
+ 
+[![Trainset](https://img.shields.io/badge/Trainset-OmniCap--IF--54K-059669)](https://huggingface.co/datasets/NJU-LINK/OmniCap-IF-54K)
+ 
+[![Testset](https://img.shields.io/badge/Testset-OmniCap--IF-d97706)](https://huggingface.co/datasets/NJU-LINK/OmniCap-IF)
 
-#metrics:
-##如 CIDEr、Blue、ROUGE 等
-#- CIDEr
+## Quick Start
 
-#tags:
-##各种自定义，包括 pretrained、fine-tuned、instruction-tuned、RL-tuned 等训练方法和其他
-#- pretrained
+### Installation
 
-#tools:
-##如 vllm、fastchat、llamacpp、AdaSeq 等
-#- vllm
----
-### 当前模型的贡献者未提供更加详细的模型介绍。模型文件和权重，可浏览“模型文件”页面获取。
-#### 您可以通过如下git clone命令，或者ModelScope SDK来下载模型
-
-SDK下载
 ```bash
-#安装ModelScope
-pip install modelscope
+conda create -n omnicap_if python=3.12
+conda activate omnicap_if
+pip install torch torchvision
+pip install transformers==4.57.1
+pip install accelerate
+pip install flash-attn --no-build-isolation
+pip install qwen-omni-utils[decord] -U
 ```
+
+### Usage
+
 ```python
-#SDK模型下载
-from modelscope import snapshot_download
-model_dir = snapshot_download('jiahaowang1/OmniCaptioner-IF-3B')
-```
-Git下载
-```
-#Git模型下载
-git clone https://www.modelscope.cn/jiahaowang1/OmniCaptioner-IF-3B.git
+import torch
+from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor
+from qwen_omni_utils import process_mm_info
+
+MODEL_ID = "wang-jiahao/OmniCaptioner-IF-3B"
+VIDEO_PATH = "example_video.mp4"
+INSTRUCTION = (
+    "Please describe this video in a Markdown table with columns "
+    "'Timestamp', 'Visual Action', and 'Audio Content'. Include precise timestamps "
+    "and mention the key audio-visual events."
+)
+
+MAX_PIXELS = 297920
+VIDEO_MAX_PIXELS = 297920
+
+model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
+    MODEL_ID,
+    torch_dtype=torch.bfloat16,
+    device_map="cuda",
+    attn_implementation="flash_attention_2"
+)
+processor = Qwen2_5OmniProcessor.from_pretrained(MODEL_ID)
+model.disable_talker()
+
+conversation = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "text", "text": INSTRUCTION},
+            {
+                "type": "video",
+                "video": VIDEO_PATH,
+                "max_pixels": MAX_PIXELS,
+                "max_frames": 160,
+                "fps": 1.0,
+                "video_max_pixels": VIDEO_MAX_PIXELS
+            }
+        ],
+    },
+]
+
+text = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)
+audios, images, videos = process_mm_info(conversation, use_audio_in_video=True)
+
+inputs = processor(
+    text=text,
+    audio=audios,
+    images=images,
+    videos=videos,
+    return_tensors="pt",
+    padding=True,
+    use_audio_in_video=True
+)
+inputs = inputs.to(model.device).to(model.dtype)
+
+with torch.inference_mode():
+    text_ids = model.generate(
+        **inputs,
+        use_audio_in_video=True,
+        return_audio=False,
+        thinker_max_new_tokens=1536,
+        talker_max_tokens=1536
+    )
+
+response = processor.decode(text_ids[0][inputs.input_ids[0].size(0):], skip_special_tokens=True)
+print(response)
 ```
 
-<p style="color: lightgrey;">如果您是本模型的贡献者，我们邀请您根据<a href="https://modelscope.cn/docs/ModelScope%E6%A8%A1%E5%9E%8B%E6%8E%A5%E5%85%A5%E6%B5%81%E7%A8%8B%E6%A6%82%E8%A7%88" style="color: lightgrey; text-decoration: underline;">模型贡献文档</a>，及时完善模型卡片内容。</p>
+## Citation
+
+```bibtex
+@article{wang2026omnicapif,
+  title   = {OmniCap-IF: Benchmarking and Improving Instruction Following Abilities for Omni-Video Captioning},
+  author  = {Wang, Jiahao and Ping, An and Wang, Yanghai and Zhang, Yuanxing and Li, Shihao and Bian, Hanyan and Ren, Yichi and Zhang, Yize and Wang, Han and Chen, Haowen and Li, Junze and Wang, Jiaqi and Hu, Yiyang and Xu, Zhuze and Zhang, Zijie and Liu, Jiaheng},
+  journal = {Preprint},
+  year    = {2026}
+}
+```