---
license: other
license_name: university-of-warwick-tsam-license
license_link: LICENSE
pipeline_tag: video-classification
tags:
  - emotion-recognition
  - video-ads
  - temporal-shift
  - multimodal
  - audio-visual
  - resnet50
  - tsam
datasets:
  - dnamodel/adcumen-viewer-emotions
language:
  - en
library_name: pytorch
---

# TSAM — Temporal Shift Audio-Visual Model for Viewer Emotion Recognition

Pre-trained model weights for the paper ["Decoding Viewer Emotions in Video Ads"](https://www.nature.com/articles/s41598-024-76968-9) by Alexey Antonov, Shravan Sampath Kumar, Jiefei Wei, William Headley, Orlando Wood, and Giovanni Montana, published in _Nature Scientific Reports_.

**Code:** [github.com/gmontana/DecodingViewerEmotions](https://github.com/gmontana/DecodingViewerEmotions)
**Dataset:** [dnamodel/adcumen-viewer-emotions](https://huggingface.co/datasets/dnamodel/adcumen-viewer-emotions)

## Model Description

TSAM (Temporal Shift Augmented Module) is a deep learning model that predicts viewer emotional responses to video advertisements. It processes both visual frames and audio tracks from 5-second video clips to classify emotional reactions across seven categories.

### Architecture

- **Backbone**: ResNet50 pre-trained on ImageNet-21K
- **Temporal modeling**: Temporal Shift Module (TSM) for efficient video understanding
- **Audio-visual fusion**: Multimodal fusion of visual and audio features
- **Output**: 7-class emotion classification

### Emotion Classes

| ID | Emotion   |
|----|-----------|
| 0  | Anger     |
| 1  | Contempt  |
| 2  | Disgust   |
| 3  | Fear      |
| 4  | Happiness |
| 5  | Sadness   |
| 6  | Surprise  |

## Files

| File | Description |
|------|-------------|
| `backbone_weights.tar` | ResNet50 backbone pre-trained on ImageNet-21K |
| `tsam_weights.tar` | Trained TSAM model checkpoint (best balanced accuracy) |

## Usage

### Download weights

```python
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="dnamodel/tsam-viewer-emotions",
    local_dir="./tsam-weights"
)
```

### Inference

See the [code repository](https://github.com/gmontana/DecodingViewerEmotions) for full training and inference instructions.

```bash
# 1. Clone the code repo
git clone https://github.com/gmontana/DecodingViewerEmotions.git
cd DecodingViewerEmotions

# 2. Install dependencies
pip install -r requirements.txt

# 3. Download dataset and model weights
# 4. Run setup_data.py to extract frames and audio
# 5. Run predict.py for inference
python predict.py
```

### Requirements

- Python 3.10+
- PyTorch 2.5+
- FFmpeg
- CUDA-capable GPU

## Training Details

- **Training data**: 21,392 five-second video clips from video advertisements
- **Validation data**: 2,856 clips
- **Test data**: 2,387 clips
- **Annotation**: Each original advertisement annotated by ~75 viewers using System1's "Test Your Ad" tool
- **Selection criterion**: Best balanced accuracy on the validation set

## Citation

```bibtex
@article{antonov2024decoding,
  title={Decoding viewer emotions in video ads},
  author={Antonov, Alexey and Kumar, Shravan Sampath and Wei, Jiefei and Headley, William and Wood, Orlando and Montana, Giovanni},
  journal={Scientific Reports},
  volume={14},
  pages={25680},
  year={2024},
  publisher={Nature Publishing Group},
  doi={10.1038/s41598-024-76968-9}
}
```

## License

The TSAM software and associated weights are provided under a custom license from the University of Warwick. Use is permitted solely for academic research and non-commercial evaluation. See the [LICENSE](LICENSE) file for full terms.

## Contact

- Questions or collaborations: Giovanni Montana — [g.montana@warwick.ac.uk](mailto:g.montana@warwick.ac.uk)
- Commercial licensing: Warwick Ventures — [ventures@warwick.ac.uk](mailto:ventures@warwick.ac.uk)