| --- |
| license: other |
| license_name: university-of-warwick-tsam-license |
| license_link: LICENSE |
| pipeline_tag: video-classification |
| tags: |
| - emotion-recognition |
| - video-ads |
| - temporal-shift |
| - multimodal |
| - audio-visual |
| - resnet50 |
| - tsam |
| datasets: |
| - dnamodel/adcumen-viewer-emotions |
| language: |
| - en |
| library_name: pytorch |
| --- |
| |
| # TSAM — Temporal Shift Audio-Visual Model for Viewer Emotion Recognition |
|
|
| Pre-trained model weights for the paper ["Decoding Viewer Emotions in Video Ads"](https://www.nature.com/articles/s41598-024-76968-9) by Alexey Antonov, Shravan Sampath Kumar, Jiefei Wei, William Headley, Orlando Wood, and Giovanni Montana, published in _Nature Scientific Reports_. |
|
|
| **Code:** [github.com/gmontana/DecodingViewerEmotions](https://github.com/gmontana/DecodingViewerEmotions) |
| **Dataset:** [dnamodel/adcumen-viewer-emotions](https://huggingface.co/datasets/dnamodel/adcumen-viewer-emotions) |
|
|
| ## Model Description |
|
|
| TSAM (Temporal Shift Augmented Module) is a deep learning model that predicts viewer emotional responses to video advertisements. It processes both visual frames and audio tracks from 5-second video clips to classify emotional reactions across seven categories. |
|
|
| ### Architecture |
|
|
| - **Backbone**: ResNet50 pre-trained on ImageNet-21K |
| - **Temporal modeling**: Temporal Shift Module (TSM) for efficient video understanding |
| - **Audio-visual fusion**: Multimodal fusion of visual and audio features |
| - **Output**: 7-class emotion classification |
|
|
| ### Emotion Classes |
|
|
| | ID | Emotion | |
| |----|-----------| |
| | 0 | Anger | |
| | 1 | Contempt | |
| | 2 | Disgust | |
| | 3 | Fear | |
| | 4 | Happiness | |
| | 5 | Sadness | |
| | 6 | Surprise | |
|
|
| ## Files |
|
|
| | File | Description | |
| |------|-------------| |
| | `backbone_weights.tar` | ResNet50 backbone pre-trained on ImageNet-21K | |
| | `tsam_weights.tar` | Trained TSAM model checkpoint (best balanced accuracy) | |
|
|
| ## Usage |
|
|
| ### Download weights |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| |
| snapshot_download( |
| repo_id="dnamodel/tsam-viewer-emotions", |
| local_dir="./tsam-weights" |
| ) |
| ``` |
|
|
| ### Inference |
|
|
| See the [code repository](https://github.com/gmontana/DecodingViewerEmotions) for full training and inference instructions. |
|
|
| ```bash |
| # 1. Clone the code repo |
| git clone https://github.com/gmontana/DecodingViewerEmotions.git |
| cd DecodingViewerEmotions |
| |
| # 2. Install dependencies |
| pip install -r requirements.txt |
| |
| # 3. Download dataset and model weights |
| # 4. Run setup_data.py to extract frames and audio |
| # 5. Run predict.py for inference |
| python predict.py |
| ``` |
|
|
| ### Requirements |
|
|
| - Python 3.10+ |
| - PyTorch 2.5+ |
| - FFmpeg |
| - CUDA-capable GPU |
|
|
| ## Training Details |
|
|
| - **Training data**: 21,392 five-second video clips from video advertisements |
| - **Validation data**: 2,856 clips |
| - **Test data**: 2,387 clips |
| - **Annotation**: Each original advertisement annotated by ~75 viewers using System1's "Test Your Ad" tool |
| - **Selection criterion**: Best balanced accuracy on the validation set |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{antonov2024decoding, |
| title={Decoding viewer emotions in video ads}, |
| author={Antonov, Alexey and Kumar, Shravan Sampath and Wei, Jiefei and Headley, William and Wood, Orlando and Montana, Giovanni}, |
| journal={Scientific Reports}, |
| volume={14}, |
| pages={25680}, |
| year={2024}, |
| publisher={Nature Publishing Group}, |
| doi={10.1038/s41598-024-76968-9} |
| } |
| ``` |
|
|
| ## License |
|
|
| The TSAM software and associated weights are provided under a custom license from the University of Warwick. Use is permitted solely for academic research and non-commercial evaluation. See the [LICENSE](LICENSE) file for full terms. |
|
|
| ## Contact |
|
|
| - Questions or collaborations: Giovanni Montana — [g.montana@warwick.ac.uk](mailto:g.montana@warwick.ac.uk) |
| - Commercial licensing: Warwick Ventures — [ventures@warwick.ac.uk](mailto:ventures@warwick.ac.uk) |
|
|
