--- library_name: transformers license: apache-2.0 base_model: interlive/video-ma2mba-1.3b tags: - video-understanding - frame-selection - long-video - reinforcement-learning - policy-gradient - mamba - multimodal - plug-and-play language: - en pipeline_tag: video-classification model-index: - name: ReFoCUS-1.3B results: - task: type: video-classification name: Query-Conditioned Frame Selection dataset: type: custom name: Video-MME (w/o sub) metrics: - type: accuracy value: 68.5 name: Overall (w/ Qwen3-VL-8B) - type: accuracy value: 62.6 name: Overall (w/ LLaVA-OneVision-7B) - task: type: video-classification name: Query-Conditioned Frame Selection dataset: type: custom name: LongVideoBench metrics: - type: accuracy value: 63.3 name: Acc. val (w/ Qwen3-VL-8B) - type: accuracy value: 61.0 name: Acc. val (w/ LLaVA-OneVision-7B) - task: type: video-classification name: Query-Conditioned Frame Selection dataset: type: custom name: MLVU metrics: - type: accuracy value: 72.5 name: m-avg (w/ Qwen3-VL-8B) - type: accuracy value: 68.5 name: m-avg (w/ LLaVA-OneVision-7B) - task: type: video-classification name: Query-Conditioned Frame Selection dataset: type: custom name: Video-MMMU metrics: - type: accuracy value: 61.1 name: Overall (w/ Qwen3-VL-8B) - type: accuracy value: 35.7 name: Overall (w/ LLaVA-OneVision-7B) --- # ReFoCUS-1.3B **ReFoCUS** (**Re**inforcement-guided **F**rame **O**ptimization for **C**ontextual **U**nder**S**tanding) is a lightweight frame-selection model for long-video understanding. Given a video and a question, it identifies the frames that are **truly essential for answering the question** and hands them to a downstream Video-LLM, replacing uniform sampling with query-conditioned visual evidence.

arXiv Project Page GitHub HF

> **Paper**: *ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding* > > Hosu Lee\*, Junho Kim\*, Hyunjun Kim, Yong Man Ro > > KAIST · UIUC ## What is ReFoCUS? ReFoCUS is a framework for finding the visual evidence scattered along a video's temporal axis that is *truly essential* for answering a given query. The evidence needed to answer a question is usually sparse and unevenly distributed, yet most video-LLMs still consume a fixed set of uniformly sampled frames, and heuristic or retrieval-based selection is never jointly optimized with the model's own reasoning, so it often conflates raw visual dynamics with true semantic relevance. ReFoCUS is the **first framework to integrate online policy-gradient reinforcement learning into frame-level optimization for video-LLMs**. Instead of aligning the *textual outputs* of an LMM with preferences, it optimizes the *visual inputs* the model attends to: - **Policy model (this model)**: reads the dense video together with the question and autoregressively picks the frames that best support answering, each choice conditioned on the query and the frames already chosen. - **Reward model (frozen)**: a reference video-LLM scores each selected frame set by how confidently it favors the correct answer, capturing its internal utility of that visual evidence. - **Reinforcement learning, no frame labels**: the policy learns from these reward signals alone; frame-level supervision is never required, and semantically and temporally coherent frame compositions emerge implicitly. ### Plug-and-Play Pipeline ``` Video (4 fps, up to 512 frames) + Question │ ▼ [ReFoCUS Policy Model] ← this model │ │ 32 query-relevant frames (temporally sorted) ▼ [Downstream Video-LLM] ← frozen, any off-the-shelf │ ▼ Answer ``` Starting from the special `<|startofframe|>` token, the policy attends over the pool of frame embeddings and samples frames one at a time without replacement, so every choice is conditioned on the question and all prior selections. Its Mamba-2 (state-space) backbone keeps computation and memory **linear** in video length, and the whole selection runs in a single feed-forward autoregressive pass, with no iterative evaluation or post-processing. ## Results ### Video QA Benchmarks | Model | LLM Size | Video-MME | LongVideoBench | MLVU | Video-MMMU | |---|:---:|:---:|:---:|:---:|:---:| | *Closed Source* | | | | | | | Gemini 2.5 Flash | - | 66.0 | 47.9 | 52.8 | 40.6 | | **+ ReFoCUS** | - | 69.5 (+3.5) | 50.9 (+3.0) | 58.0 (+5.2) | 45.6 (+5.0) | | GPT-4o | - | 58.8 | 49.5 | 58.7 | 62.9 | | **+ ReFoCUS** | - | 60.8 (+2.0) | 52.9 (+3.4) | 65.1 (+6.4) | 62.1 (-0.8) | | *Open Source* | | | | | | | LLaVA-OneVision | 0.5B | 43.5 | 44.7 | 44.8 | 17.3 | | **+ ReFoCUS** | 0.5B | 47.1 (+3.6) | 48.7 (+4.0) | 50.3 (+5.5) | 19.4 (+2.1) | | InternVL3 | 1B | 50.0 | 47.6 | 54.0 | 27.7 | | **+ ReFoCUS** | 1B | 53.6 (+3.6) | 50.6 (+3.0) | 58.9 (+4.9) | 29.3 (+1.6) | | VideoLLaMA 3 | 2B | 43.1 | 48.8 | 46.8 | 28.7 | | **+ ReFoCUS** | 2B | 47.1 (+4.0) | 53.7 (+4.9) | 50.2 (+3.4) | 29.2 (+0.5) | | InternVL3 | 2B | 58.4 | 50.9 | 62.7 | 38.3 | | **+ ReFoCUS** | 2B | 60.7 (+2.3) | 54.9 (+4.0) | 68.0 (+5.3) | 39.3 (+1.0) | | InternVL3.5 | 4B | 62.7 | 57.7 | 66.6 | 52.0 | | **+ ReFoCUS** | 4B | 65.9 (+3.2) | 62.6 (+4.9) | 71.5 (+4.9) | 53.3 (+1.3) | | Qwen3-VL | 4B | 62.1 | 57.4 | 63.1 | 54.0 | | **+ ReFoCUS** | 4B | 66.4 (+4.3) | 61.9 (+4.5) | 71.9 (+8.8) | 56.4 (+2.4) | | VideoLLaMA 3 | 7B | 59.0 | 54.8 | 52.9 | 32.8 | | **+ ReFoCUS** | 7B | 62.2 (+3.2) | 57.0 (+2.2) | 59.8 (+6.9) | 34.4 (+1.6) | | LLaVA-OneVision | 7B | 58.4 | 55.0 | 63.7 | 34.1 | | **+ ReFoCUS** | 7B | 62.6 (+4.2) | 61.0 (+6.0) | 68.5 (+4.8) | 35.7 (+1.6) | | InternVL3 | 8B | 64.3 | 57.8 | 68.1 | 49.3 | | **+ ReFoCUS** | 8B | 67.0 (+2.7) | 62.0 (+4.2) | 72.7 (+4.6) | 50.6 (+1.3) | | InternVL3.5 | 8B | 64.4 | 59.7 | 67.3 | 50.0 | | **+ ReFoCUS** | 8B | 66.7 (+2.3) | 64.1 (+4.4) | 70.6 (+3.3) | 53.2 (+3.2) | | Qwen3-VL | 8B | 65.0 | 56.6 | 63.0 | 59.1 | | **+ ReFoCUS** | 8B | 68.5 (+3.5) | 63.3 (+6.7) | 72.5 (+9.5) | 61.1 (+2.0) | ### Open-Ended Video QA | Model | LLM Size | NExT-QA (WUPS) | ActivityNet-QA Score | Video-ChatGPT Score | |---|:---:|:---:|:---:|:---:| | VideoLLaMA 3 | 7B | 25.8 | 70.2 | 60.3 | | **+ ReFoCUS** | 7B | 26.5 (+0.7) | 72.2 (+2.0) | 62.7 (+2.4) | | LLaVA-OneVision | 7B | 16.2 | 68.8 | 59.9 | | **+ ReFoCUS** | 7B | 16.4 (+0.2) | 69.7 (+0.9) | 61.4 (+1.5) | | InternVL3 | 8B | 26.6 | 69.4 | 59.4 | | **+ ReFoCUS** | 8B | 26.8 (+0.2) | 70.4 (+1.0) | 61.0 (+1.6) | | InternVL3.5 | 8B | 24.3 | 67.7 | 59.5 | | **+ ReFoCUS** | 8B | 24.7 (+0.4) | 68.8 (+1.1) | 60.8 (+1.3) | | Qwen3-VL | 8B | 25.3 | 65.9 | 60.3 | | **+ ReFoCUS** | 8B | 25.7 (+0.4) | 67.2 (+1.3) | 62.3 (+2.0) | ## Usage The selected frame indices are temporally sorted and can be forwarded to **any** off-the-shelf video-LLM. For the full selection/inference pipeline and evaluation scripts, please refer to the [ReFoCUS GitHub repository](https://github.com/interlive-team/ReFoCUS). ## Citation ```bibtex @InProceedings{Lee_2026_CVPR, author = {Lee, Hosu and Kim, Junho and Kim, Hyunjun and Ro, Yong Man}, title = {ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Findings}, month = {June}, year = {2026}, pages = {8291-8302} } ``` ## License This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).