--- license: other tags: - humomni - proactive-videoqa - video-language-model - mmd duet2 - qwen2.5-vl - multi-stage-inference - competition --- # Lenormand HumOmni Track 2 System This repository contains the reproducibility materials for Team **Lenormand** in HumOmni Track 2 ProactivEval. ## Base Repository Our system is built on top of the public MMDuet2 repository: - MMDuet2: https://github.com/yellow-binary-tree/MMDuet2 We use MMDuet2 as the streaming video-language backbone and modify the inference and response-control pipeline for HumOmni Track 2. ## Overview The final system is a **multi-stage agentic inference harness** rather than a single-pass VLM generation pipeline. We found that the streaming VLM should not simultaneously act as the visual reader, memory controller, response gate, and final answer writer. Therefore, we decouple the system into several functional modules. The core idea is to treat the VLM as a mostly stateless visual reader, while external modules handle timing, answer selection, memory/state tracking, and post-processing. ## System Components ### 1. V36 Reader / Timing Agent We use the MMDuet2-style streaming pipeline to propose candidate response times. The raw V36-style output is not directly used as the final submission because it may contain long responses, repeated content, and late-stage narrative contamination. ### 2. Causal Visual Answer Regeneration For each candidate response time `t`, the system regenerates a concise answer using only visual frames up to `t`. No future frames are used when generating a response at time `t`. ### 3. QEA Selection Module We include a lightweight Question–Evidence–Answer selection module to choose concise and question-relevant answers from candidate generations. This module compares the question, candidate answer, and causal visual evidence. It does not use hidden labels, ground-truth reply spans, or leaderboard feedback. ### 4. External Memory Agent The external memory module stores only question-relevant textual states rather than raw visual history. It checks whether a candidate turn is new, relevant, non-redundant, and not contaminated by previous narrative outputs. This module is designed to reduce answer self-contamination in streaming generation. The VLM is used mainly for visual reading, while memory and response control are handled outside the autoregressive dialogue context. ### 5. Clean-Light-Plus Post-processing The final global post-processing step removes only obvious generation artifacts, such as: - `NO REPLY` - `NS REPLY` - `NSWYR` - `REPLYING:` - `no image displayed` - `No active movement occurs` - `No actions take place` - `No description available` - `No response available` - `No additional context` - `No additional elements` This filtering is global and rule-based. It does not use ground-truth labels, hidden reply spans, leaderboard feedback, or video-specific manual edits. ## Final Submission Variant The final prediction file submitted to the official Google Drive is generated by: `memory_agent_top6_clean_light_plus` Final sanity check: ```text N_records = 500 unique_question_ids = 500 duplicate_question_ids = 0 empty_outputs = 0 avg_turns_per_video = 3.03 max_turns_per_video = 6 time_min = 4.0 time_max = 30.0 chronological_per_video = True all turns have time and content = True ``` ## Base Models and Checkpoints We do not duplicate large public checkpoints in this repository. The system uses public base resources, including MMDuet2-style streaming components and Qwen2.5-VL-based vision-language inference components. This repository provides the modified inference harness, post-processing code, configuration notes, and reproducibility notebook needed to regenerate the submitted system output. ## Modified Components Compared with the public MMDuet2-style baseline, our submitted system modifies the response-control pipeline in the following ways: 1. **Deterministic decoding and concise generation** - We use deterministic decoding for stable outputs. - We favor concise visual answers instead of long dialogue-style responses. 2. **KV-cache / writeback control** - We avoid feeding long generated answers back into the streaming context. - We use short summary/marker-style writeback to reduce narrative self-contamination. 3. **Timestamp correction and causal frame filtering** - Each response at time `t` is generated only from frames up to `t`. - The pipeline corrects timestamp alignment for frame sampling. 4. **Per-turn answer regeneration** - V36-style outputs are used as timing proposals. - Final answers are regenerated at candidate times using causal visual context. 5. **QEA selection and targeted fallback** - Candidate answers are selected using a lightweight QEA-style selection module. - Targeted fallback generation is used to rescue low-confidence or generic answers. 6. **External Memory Agent** - The memory/state module filters redundant, irrelevant, or contaminated turns. - It keeps only question-relevant textual states instead of storing the full visual history. 7. **Clean-Light-Plus filtering** - Obvious generation artifacts are removed using a global rule-based filter. - No test labels, hidden reply spans, leaderboard feedback, or sample-specific manual edits are used. ## Repository Files The repository includes: ```text README.md requirements.txt HumOmni_V4_FINAL_K1_MEMORY_AGENT_TOP6_CLEAN_PLUS_FULL500.ipynb patch_mmd duet2_inference.py clean_light_plus_cell.py submission_generation_notes.md export_modified_inference_notes.md ``` If a fully patched local `inference.py` is exported from the Colab runtime, it can also be uploaded as `modified_inference.py` for convenience. ## Environment The system was run on Google Colab Pro with an NVIDIA A100 GPU. Main dependencies are listed in `requirements.txt`. The environment used the public MMDuet2 codebase with the provided inference patch and a Qwen2.5-VL-based visual-language inference setup. ## Reproducibility To reproduce the final submission: 1. Prepare the official HumOmni Track 2 Phase 1 test data. 2. Clone or prepare the public MMDuet2 repository. 3. Apply the provided inference patch / modified inference script. 4. Run `HumOmni_V4_FINAL_K1_MEMORY_AGENT_TOP6_CLEAN_PLUS_FULL500.ipynb`. 5. Generate the `memory_agent_top6` output. 6. Run the Clean-Light-Plus post-processing cell. 7. Verify the final JSONL format using the final audit cell. The official prediction JSONL is submitted separately through the official Google Drive prediction folder. ## Notes on Fairness and Test Usage The submitted system does not use hidden test labels, ground-truth reply spans, or manual video-specific edits. The final filtering rules are global, deterministic, and applied uniformly across all test samples.