---
language: en
tags:
- emotion-recognition
- audio-visual
- multimodal
- ravdess
license: mit
---

# AVERFormer (RAVDESS)

Multimodal Audio-Visual Emotion Recognition transformer trained on RAVDESS.

Source checkpoint: `best_model_epoch_161.pth`
Classes: ['neutral', 'calm', 'happy', 'sad', 'angry', 'fearful', 'disgust', 'surprised']

## Inputs

| Modality | Shape | Notes |
|---|---|---|
| `audio_waveform` | `[B, 2, 88000]` | stereo, 22000 Hz, 4 s |
| `mel_spectrogram` | `[B, 320, 343]` | 160 mels per channel, stereo concatenated |
| `mfcc` | `[B, 40, 343]` | 20 MFCC per channel, stereo concatenated |
| `video` | `[B, 3, 12, 180, 320]` | 12 uniformly sampled frames, ImageNet-normalised |

See repo https://github.com/anthropics/AVERFormer (or your fork) for the matching
model code under `models/averformer.py`.

## Loading

```python
import torch
from huggingface_hub import hf_hub_download
from models.averformer import AVERFormer

ckpt_path = hf_hub_download(repo_id="mhussainahmad/averformer-ravdess", filename="pytorch_model.pth")
model = AVERFormer(num_classes=8)
state = torch.load(ckpt_path, map_location="cpu", weights_only=False)
model.load_state_dict(state.get("model_state_dict", state), strict=False)
model.eval()
```