---
language: en
license: apache-2.0
library_name: peft
base_model: liuhaotian/llava-v1.5-7b
tags:
- llava
- multimodal
- adversarial-robustness
- jailbreak-defense
- safemllm
- lora
---

# SafeMLLM-LLaVA-7B

LoRA adapter that turns **[liuhaotian/llava-v1.5-7b](https://huggingface.co/liuhaotian/llava-v1.5-7b)**
into a jailbreak-robust multimodal model, trained with the **SafeMLLM** framework
described in:

> **Towards Robust Multimodal Large Language Models Against Jailbreak Attacks**
> Ziyi Yin, Yuanpu Cao, Han Liu, Ting Wang, Jinghui Chen, Fenglong Ma — arXiv:[2502.00653](https://arxiv.org/abs/2502.00653) (2025).

## What is in this repo

A standard LoRA + projector adapter:

| File                       | What it is                                                 |
| -------------------------- | ---------------------------------------------------------- |
| `adapter_config.json`      | PEFT LoRA config                                           |
| `adapter_model.bin`        | LoRA weights (rank-r updates on attention/MLP layers)      |
| `non_lora_trainables.bin`  | Vision-language projector weights (always trainable)       |
| `config.json`              | LLaVA model config snapshot                                |
| `trainer_state.json`       | Training-time logs (steps / loss curve)                    |

To use it you also need the LLaVA-1.5-7B base weights from
[`liuhaotian/llava-v1.5-7b`](https://huggingface.co/liuhaotian/llava-v1.5-7b).

## Quick start

Clone the matching evaluation code:

```bash
git clone https://github.com/ericyinyzy/SafeMLLM.git
cd SafeMLLM
conda env create -f environment.yml && conda activate safemllm-llava
```

Download both checkpoints:

```bash
mkdir -p checkpoints
huggingface-cli download liuhaotian/llava-v1.5-7b      --local-dir checkpoints/llava-v1.5-7b
huggingface-cli download ericyinyzy/SafeMLLM-LLaVA-7B  --local-dir checkpoints/SafeMLLM-LLaVA-7B
```

Run any of the four evaluations (full pipeline shown):

```bash
export LLAVA7B_BASE=$PWD/checkpoints/llava-v1.5-7b
export SAFEMLLM_L7B=$PWD/checkpoints/SafeMLLM-LLaVA-7B
bash scripts/run_L7B.sh 0          # GPU id
```

Programmatic loading (PEFT-style):

```python
from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path

tokenizer, model, image_processor, _ = load_pretrained_model(
    model_path="ericyinyzy/SafeMLLM-LLaVA-7B",
    model_base="liuhaotian/llava-v1.5-7b",
    model_name=get_model_name_from_path("ericyinyzy/SafeMLLM-LLaVA-7B"),
)
```

## Evaluation results (paper, Table 2 / Table 3)

This adapter — when paired with LLaVA-1.5-7B — substantially improves robustness
against ImgJP, FigStep and MM-SafetyBench attacks while preserving general
multimodal capability on MM-Vet. Refer to the paper for full numbers.

## Hardware requirements

| Use case            | VRAM   |
| ------------------- | ------ |
| Inference (fp16)    | ~18 GB |
| ImgJP attack (PGD)  | ~26 GB |

## License

Apache-2.0 for the adapter weights. The underlying LLaVA-1.5 base model retains
its own license; please check
[`liuhaotian/llava-v1.5-7b`](https://huggingface.co/liuhaotian/llava-v1.5-7b) for
those terms.

## Citation

```bibtex
@article{yin2025safemllm,
  title   = {Towards Robust Multimodal Large Language Models Against Jailbreak Attacks},
  author  = {Yin, Ziyi and Cao, Yuanpu and Liu, Han and Wang, Ting and Chen, Jinghui and Ma, Fenglong},
  journal = {arXiv preprint arXiv:2502.00653},
  year    = {2025}
}
```