Hugging Face's logo

dnamodel
/
tsam-viewer-emotions

Video Classification
PyTorch
English
emotion-recognition
video-ads
temporal-shift
multimodal
audio-visual
resnet50
tsam
Model card Files
xet
 Community
tsam-viewer-emotions
File size: 3,831 Bytes
6adb570
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
---
license: other
license_name: university-of-warwick-tsam-license
license_link: LICENSE
pipeline_tag: video-classification
tags:
  - emotion-recognition
  - video-ads
  - temporal-shift
  - multimodal
  - audio-visual
  - resnet50
  - tsam
datasets:
  - dnamodel/adcumen-viewer-emotions
language:
  - en
library_name: pytorch
---

# TSAM — Temporal Shift Audio-Visual Model for Viewer Emotion Recognition

Pre-trained model weights for the paper ["Decoding Viewer Emotions in Video Ads"](https://www.nature.com/articles/s41598-024-76968-9) by Alexey Antonov, Shravan Sampath Kumar, Jiefei Wei, William Headley, Orlando Wood, and Giovanni Montana, published in _Nature Scientific Reports_.

**Code:** [github.com/gmontana/DecodingViewerEmotions](https://github.com/gmontana/DecodingViewerEmotions)
**Dataset:** [dnamodel/adcumen-viewer-emotions](https://huggingface.co/datasets/dnamodel/adcumen-viewer-emotions)

## Model Description

TSAM (Temporal Shift Augmented Module) is a deep learning model that predicts viewer emotional responses to video advertisements. It processes both visual frames and audio tracks from 5-second video clips to classify emotional reactions across seven categories.

### Architecture

- **Backbone**: ResNet50 pre-trained on ImageNet-21K
- **Temporal modeling**: Temporal Shift Module (TSM) for efficient video understanding
- **Audio-visual fusion**: Multimodal fusion of visual and audio features
- **Output**: 7-class emotion classification

### Emotion Classes

| ID | Emotion   |
|----|-----------|
| 0  | Anger     |
| 1  | Contempt  |
| 2  | Disgust   |
| 3  | Fear      |
| 4  | Happiness |
| 5  | Sadness   |
| 6  | Surprise  |

## Files

| File | Description |
|------|-------------|
| `backbone_weights.tar` | ResNet50 backbone pre-trained on ImageNet-21K |
| `tsam_weights.tar` | Trained TSAM model checkpoint (best balanced accuracy) |

## Usage

### Download weights

```python
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="dnamodel/tsam-viewer-emotions",
    local_dir="./tsam-weights"
)
```

### Inference

See the [code repository](https://github.com/gmontana/DecodingViewerEmotions) for full training and inference instructions.

```bash
# 1. Clone the code repo
git clone https://github.com/gmontana/DecodingViewerEmotions.git
cd DecodingViewerEmotions

# 2. Install dependencies
pip install -r requirements.txt

# 3. Download dataset and model weights
# 4. Run setup_data.py to extract frames and audio
# 5. Run predict.py for inference
python predict.py
```

### Requirements

- Python 3.10+
- PyTorch 2.5+
- FFmpeg
- CUDA-capable GPU

## Training Details

- **Training data**: 21,392 five-second video clips from video advertisements
- **Validation data**: 2,856 clips
- **Test data**: 2,387 clips
- **Annotation**: Each original advertisement annotated by ~75 viewers using System1's "Test Your Ad" tool
- **Selection criterion**: Best balanced accuracy on the validation set

## Citation

```bibtex
@article{antonov2024decoding,
  title={Decoding viewer emotions in video ads},
  author={Antonov, Alexey and Kumar, Shravan Sampath and Wei, Jiefei and Headley, William and Wood, Orlando and Montana, Giovanni},
  journal={Scientific Reports},
  volume={14},
  pages={25680},
  year={2024},
  publisher={Nature Publishing Group},
  doi={10.1038/s41598-024-76968-9}
}
```

## License

The TSAM software and associated weights are provided under a custom license from the University of Warwick. Use is permitted solely for academic research and non-commercial evaluation. See the [LICENSE](LICENSE) file for full terms.

## Contact

- Questions or collaborations: Giovanni Montana — [g.montana@warwick.ac.uk](mailto:g.montana@warwick.ac.uk)
- Commercial licensing: Warwick Ventures — [ventures@warwick.ac.uk](mailto:ventures@warwick.ac.uk)