avnisinghal001/disasterlens-v2-multimodal-disaster-response

DisasterLens V2: Multimodal Disaster-Response Classifier

DisasterLens V2 is a two-stage multimodal disaster-response classifier built using computer vision, transfer learning, NLP, and late probability fusion.

It analyzes a disaster-related image and optional social media text to predict:

1. Whether the post is informative for disaster response.
2. If informative, the humanitarian category.

Architecture

Stage 1: Informativeness Detection

Predicts:

• informative
• not_informative

Models:

• ConvNeXt-Tiny image classifier with test-time augmentation
• DistilBERT text classifier
• Late fusion when text is available

Fusion weights:

• image: 0.55
• text: 0.45

Stage 2: Humanitarian Category Classification

Predicts one of 5 grouped humanitarian categories:

• not_humanitarian
• damage
• rescue_or_donation
• affected_people
• other_relevant_information

Models:

• ConvNeXt-Tiny image classifier with test-time augmentation
• DistilBERT text classifier
• Late fusion when text is available

Fusion weights:

• image: 0.65
• text: 0.35

Final Results

Component	Input	Accuracy	F1 Type	F1
Informativeness image-only	Image	0.8494	Binary F1	0.8565
Informativeness fusion	Image + Text	0.8632	Binary F1	0.8701
Humanitarian image-only	Image	0.7948	Macro F1	0.7375
Humanitarian fusion	Image + Text	0.7993	Macro F1	0.7436
Full V2 cascade	Image + Text	0.8485	Macro F1	0.7837

Final full V2 cascade:

• Accuracy: 84.85%
• Macro F1: 78.37%
• Weighted F1: 84.44%

Usage

Run from project root:

python scripts/34_predict_disasterlens_v2.py \
  --image_path "path/to/disaster_image.jpg" \
  --text "optional tweet or post text" \
  --json

Image only mode:

python scripts/34_predict_disasterlens_v2.py \
  --image_path "path/to/disaster_image.jpg" \
  --json

Example Output:

{
  "mode": "multimodal",
  "informativeness": {
    "prediction": "informative",
    "confidence": 0.9507
  },
  "humanitarian": {
    "mode": "image_text_fusion",
    "prediction": "other_relevant_information",
    "confidence": 0.9878
  }
}

Dataset

This model was trained on CrisisMMD v2.0-style disaster image/text data.

The original dataset is not redistributed in this repository. Please obtain the dataset from the official source and follow its license and terms.

Intended Use

This project is intended for:

• disaster-response research
• crisis informatics experiments
• multimodal ML demonstrations
• portfolio and educational use

It should not be used as the sole decision-making system in real emergency response workflows.

Limitations

• Performance varies by disaster type and visual clarity.
• Text-only humanitarian classification is weak alone, but useful in fusion.
• Some categories are visually and semantically overlapping.
• The dataset is imbalanced, especially for affected people-related labels.