README.md

---
license: apache-2.0
datasets:
- psp-dada/SENTINEL
language:
- en
base_model:
- llava-hf/llava-v1.6-vicuna-13b-hf
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- PEFT
- lora
- bf16
- llava
---

# Model Card for ``psp-dada/LLaVA-v1.6-Vicuna-13B-SENTINEL`` | ICCV 2025 | SENTINEL:<br>Mitigating Object Hallucinations via Sentence-Level Early Intervention <!-- omit in toc -->

<a href='https://arxiv.org/abs/2507.12455'>
<img src='https://img.shields.io/badge/Paper-Arxiv-purple'></a>
<a href='https://huggingface.co/datasets/psp-dada/SENTINEL'>
<img src='https://img.shields.io/badge/Datasets-HF-Green'></a>
<a href='https://github.com/pspdada/SENTINEL'>
<img src='https://img.shields.io/badge/Github-Repo-orange'></a>
<a href='https://huggingface.co/papers/2507.12455'>
<img src='https://img.shields.io/badge/Discussion-HF-blue'></a>
<a href='https://github.com/pspdada/SENTINEL/blob/main/LICENSE'>
<img src='https://img.shields.io/badge/LICENSE-Apache_2.0-yellow'></a>

<a href='https://modelscope.cn/datasets/pspdada/SENTINEL'>
<img src='https://img.shields.io/badge/Datasets-🤖ModelScope-pink'></a>
<a href='https://modelscope.cn/collections/pspdada/SENTINEL'>
<img src='https://img.shields.io/badge/Models-🤖ModelScope-red'></a>

## 🎊 News <!-- omit in toc -->

- [2025.07.30] 🔍 Our work has been featured and explained by 52CV, check it out [here](https://mp.weixin.qq.com/s/Sfr1wdUCkeOLmj7NVWNUnw).
- [2025.07.21] 📖 All code, data, and models are released!
- [2025.06.26] 🎉 Our SENTINEL is accepted by **ICCV 2025**!

## 🚀 Overview <!-- omit in toc -->

**SENTINEL** introduces an automatic, sentence‑level early intervention strategy to prevent and mitigate object hallucinations in multimodal large language models (MLLMs). Key advantages:

- **Annotation‑free**: No human labeling required.

- **Model-agnostic**: Compatible with any MLLM architecture.

- **Efficient**: Lightweight LoRA fine‑tuning.

## 🔑 Key Features

- 🧠 **Early intervention halts hallucination propagation**. We find that hallucinations of MLLMs predominantly arise in early sentences and propagate through the rest of the output. SENTINEL interrupts this chain early to maximize mitigation.
<table align="center">
    <p align="center">
      <img src="https://github.com/pspdada/SENTINEL/raw/main/docs/figures/figure2.png" width="80%" />
    </p>
</table>

- 🔍 **In-domain contextual preference learning without human labels**. SENTINEL constructs hallucinated/factual samples via detector cross-validation and builds context-aware preference data without relying on proprietary LLMs or manual annotations.
<table align="center">
    <p align="center">
      <img src="https://github.com/pspdada/SENTINEL/raw/main/docs/figures/figure3.png" width="80%" />
    </p>
</table>

- 💡 **Context matters: rich coherence drives robustness**. By prioritizing context-coherent positive samples over hallucinated ones, SENTINEL significantly boosts generalization.
<table align="center">
    <p align="center">
      <img src="https://github.com/pspdada/SENTINEL/raw/main/docs/figures/figure4.png" width="80%" />
    </p>
</table>

- ♻️ **Iterative contextual bootstrapping for diverse hallucination-free contexts**. Our pipeline dynamically grows non-hallucinated contexts and expands coverage across varied scenes, improving robustness across generations.
<table align="center">
    <p align="center">
      <img src="https://github.com/pspdada/SENTINEL/raw/main/docs/figures/figure5.png" width="80%" />
    </p>
</table>

- 📊 **State-of-the-art results across benchmarks**.
SENTINEL achieves **up to 92% reduction** in hallucinations and outperforms prior SOTA methods across Object HalBench, AMBER, and HallusionBench, while maintaining or improving general task performance.
<table align="center">
    <p align="center">
      <img src="https://github.com/pspdada/SENTINEL/raw/main/docs/figures/table1.png" width="80%" />
    </p>
</table>

## How to use

This model is a PEFT (LoRA) adapter. You first need to load the base model (`llava-hf/llava-v1.6-vicuna-13b-hf`) and then load this adapter on top of it.

**For the details of this model, please refer to the [documentation](https://github.com/pspdada/SENTINEL?tab=readme-ov-file#-model-weights) of the GitHub repo.**

## 📝 Citation

If you find our model/code/data/paper helpful, please consider citing our papers 📝 and starring us ⭐️！

```bibtex
@inproceedings{peng2025mitigating,
  title={Mitigating object hallucinations via sentence-level early intervention},
  author={Peng, Shangpin and Yang, Senqiao and Jiang, Li and Tian, Zhuotao},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={635--646},
  year={2025}
}
```

## 📧 Contact us <!-- omit in toc -->

If you have any questions, comments, or suggestions, please do not hesitate to submit an issue or PR to help advance research in this area.