Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -4,25 +4,25 @@ tags:
|
|
| 4 |
- emotion-recognition
|
| 5 |
- audio-visual
|
| 6 |
- multimodal
|
| 7 |
-
-
|
| 8 |
license: apache-2.0
|
| 9 |
---
|
| 10 |
|
| 11 |
-
# AVERFormer-v4 (
|
| 12 |
|
| 13 |
-
Multimodal Audio-Visual-Text Emotion Recognition transformer trained on
|
| 14 |
Code: https://github.com/mhussainahmad/AVERFormer
|
| 15 |
|
| 16 |
## Reported numbers
|
| 17 |
-
- Best single-seed val wF1: **0.
|
| 18 |
-
- Best ensemble wF1: **0.
|
| 19 |
|
| 20 |
-
## Classes (
|
| 21 |
-
['
|
| 22 |
|
| 23 |
## Architecture
|
| 24 |
- Audio: `microsoft/wavlm-large` (16 kHz mono waveform)
|
| 25 |
-
- Video: `MCG-NJU/videomae-large` (16 frames @
|
| 26 |
- Text: `microsoft/deberta-v3-large` (speaker-aware ctx encoder)
|
| 27 |
- Fusion: 2-layer cross-modal transformer, dim=512, 8 heads
|
| 28 |
- Heads: face / voice / text / joint (all share class count)
|
|
@@ -34,8 +34,8 @@ import json, torch
|
|
| 34 |
from huggingface_hub import hf_hub_download
|
| 35 |
from models.averformer_v4 import AVERFormerV4
|
| 36 |
|
| 37 |
-
cfg = json.load(open(hf_hub_download(repo_id="mhussainahmad/averformer-
|
| 38 |
-
ckpt = hf_hub_download(repo_id="mhussainahmad/averformer-
|
| 39 |
|
| 40 |
model = AVERFormerV4(
|
| 41 |
audio_backbone=cfg["audio_backbone"],
|
|
@@ -54,23 +54,7 @@ model.eval()
|
|
| 54 |
## Live inference
|
| 55 |
|
| 56 |
```bash
|
| 57 |
-
python live_emotion_v4.py --repo_id mhussainahmad/averformer-
|
| 58 |
```
|
| 59 |
|
| 60 |
See `LIVE_INFERENCE_README.md` in the GitHub repo for full setup.
|
| 61 |
-
|
| 62 |
-
## Training command (reference)
|
| 63 |
-
|
| 64 |
-
```
|
| 65 |
-
python train_v5_spec.py --corpus CREMA-D --classes 6 \
|
| 66 |
-
--audio microsoft/wavlm-large \
|
| 67 |
-
--video MCG-NJU/videomae-large \
|
| 68 |
-
--text microsoft/deberta-v3-large \
|
| 69 |
-
--lora_r 16 \
|
| 70 |
-
--epochs 20 --bs 4 \
|
| 71 |
-
--grad_accum 4 \
|
| 72 |
-
--lr_head 0.0001 --lr_backbone 2e-05 \
|
| 73 |
-
--loss ce --class_weights sqrt-inv \
|
| 74 |
-
--select_metric wF1 \
|
| 75 |
-
--seed 100
|
| 76 |
-
```
|
|
|
|
| 4 |
- emotion-recognition
|
| 5 |
- audio-visual
|
| 6 |
- multimodal
|
| 7 |
+
- meld
|
| 8 |
license: apache-2.0
|
| 9 |
---
|
| 10 |
|
| 11 |
+
# AVERFormer-v4 (MELD)
|
| 12 |
|
| 13 |
+
Multimodal Audio-Visual-Text Emotion Recognition transformer trained on MELD.
|
| 14 |
Code: https://github.com/mhussainahmad/AVERFormer
|
| 15 |
|
| 16 |
## Reported numbers
|
| 17 |
+
- Best single-seed val wF1: **0.3815709082075283** (seed 1337, epoch 4)
|
| 18 |
+
- Best ensemble wF1: **0.6150 (ensemble), 0.6128 (10-seed avg)**
|
| 19 |
|
| 20 |
+
## Classes (7)
|
| 21 |
+
['neutral', 'joy', 'sadness', 'anger', 'fear', 'disgust', 'surprise']
|
| 22 |
|
| 23 |
## Architecture
|
| 24 |
- Audio: `microsoft/wavlm-large` (16 kHz mono waveform)
|
| 25 |
+
- Video: `MCG-NJU/videomae-large` (16 frames @ 224x224 RGB)
|
| 26 |
- Text: `microsoft/deberta-v3-large` (speaker-aware ctx encoder)
|
| 27 |
- Fusion: 2-layer cross-modal transformer, dim=512, 8 heads
|
| 28 |
- Heads: face / voice / text / joint (all share class count)
|
|
|
|
| 34 |
from huggingface_hub import hf_hub_download
|
| 35 |
from models.averformer_v4 import AVERFormerV4
|
| 36 |
|
| 37 |
+
cfg = json.load(open(hf_hub_download(repo_id="mhussainahmad/averformer-meld-v4", filename="config.json")))
|
| 38 |
+
ckpt = hf_hub_download(repo_id="mhussainahmad/averformer-meld-v4", filename="pytorch_model.pth")
|
| 39 |
|
| 40 |
model = AVERFormerV4(
|
| 41 |
audio_backbone=cfg["audio_backbone"],
|
|
|
|
| 54 |
## Live inference
|
| 55 |
|
| 56 |
```bash
|
| 57 |
+
python live_emotion_v4.py --repo_id mhussainahmad/averformer-meld-v4
|
| 58 |
```
|
| 59 |
|
| 60 |
See `LIVE_INFERENCE_README.md` in the GitHub repo for full setup.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|