mhussainahmad commited on
Commit
d136a92
·
verified ·
1 Parent(s): f86fe7c

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +11 -27
README.md CHANGED
@@ -4,25 +4,25 @@ tags:
4
  - emotion-recognition
5
  - audio-visual
6
  - multimodal
7
- - cremad
8
  license: apache-2.0
9
  ---
10
 
11
- # AVERFormer-v4 (CREMA-D)
12
 
13
- Multimodal Audio-Visual-Text Emotion Recognition transformer trained on CREMA-D.
14
  Code: https://github.com/mhussainahmad/AVERFormer
15
 
16
  ## Reported numbers
17
- - Best single-seed val wF1: **0.8320558803202199** (seed 100, epoch 12)
18
- - Best ensemble wF1: **0.8278 (top-3 ensemble)**
19
 
20
- ## Classes (6)
21
- ['anger', 'disgust', 'fear', 'happy', 'neutral', 'sad']
22
 
23
  ## Architecture
24
  - Audio: `microsoft/wavlm-large` (16 kHz mono waveform)
25
- - Video: `MCG-NJU/videomae-large` (16 frames @ 224×224 RGB)
26
  - Text: `microsoft/deberta-v3-large` (speaker-aware ctx encoder)
27
  - Fusion: 2-layer cross-modal transformer, dim=512, 8 heads
28
  - Heads: face / voice / text / joint (all share class count)
@@ -34,8 +34,8 @@ import json, torch
34
  from huggingface_hub import hf_hub_download
35
  from models.averformer_v4 import AVERFormerV4
36
 
37
- cfg = json.load(open(hf_hub_download(repo_id="mhussainahmad/averformer-cremad-v4", filename="config.json")))
38
- ckpt = hf_hub_download(repo_id="mhussainahmad/averformer-cremad-v4", filename="pytorch_model.pth")
39
 
40
  model = AVERFormerV4(
41
  audio_backbone=cfg["audio_backbone"],
@@ -54,23 +54,7 @@ model.eval()
54
  ## Live inference
55
 
56
  ```bash
57
- python live_emotion_v4.py --repo_id mhussainahmad/averformer-cremad-v4
58
  ```
59
 
60
  See `LIVE_INFERENCE_README.md` in the GitHub repo for full setup.
61
-
62
- ## Training command (reference)
63
-
64
- ```
65
- python train_v5_spec.py --corpus CREMA-D --classes 6 \
66
- --audio microsoft/wavlm-large \
67
- --video MCG-NJU/videomae-large \
68
- --text microsoft/deberta-v3-large \
69
- --lora_r 16 \
70
- --epochs 20 --bs 4 \
71
- --grad_accum 4 \
72
- --lr_head 0.0001 --lr_backbone 2e-05 \
73
- --loss ce --class_weights sqrt-inv \
74
- --select_metric wF1 \
75
- --seed 100
76
- ```
 
4
  - emotion-recognition
5
  - audio-visual
6
  - multimodal
7
+ - meld
8
  license: apache-2.0
9
  ---
10
 
11
+ # AVERFormer-v4 (MELD)
12
 
13
+ Multimodal Audio-Visual-Text Emotion Recognition transformer trained on MELD.
14
  Code: https://github.com/mhussainahmad/AVERFormer
15
 
16
  ## Reported numbers
17
+ - Best single-seed val wF1: **0.3815709082075283** (seed 1337, epoch 4)
18
+ - Best ensemble wF1: **0.6150 (ensemble), 0.6128 (10-seed avg)**
19
 
20
+ ## Classes (7)
21
+ ['neutral', 'joy', 'sadness', 'anger', 'fear', 'disgust', 'surprise']
22
 
23
  ## Architecture
24
  - Audio: `microsoft/wavlm-large` (16 kHz mono waveform)
25
+ - Video: `MCG-NJU/videomae-large` (16 frames @ 224x224 RGB)
26
  - Text: `microsoft/deberta-v3-large` (speaker-aware ctx encoder)
27
  - Fusion: 2-layer cross-modal transformer, dim=512, 8 heads
28
  - Heads: face / voice / text / joint (all share class count)
 
34
  from huggingface_hub import hf_hub_download
35
  from models.averformer_v4 import AVERFormerV4
36
 
37
+ cfg = json.load(open(hf_hub_download(repo_id="mhussainahmad/averformer-meld-v4", filename="config.json")))
38
+ ckpt = hf_hub_download(repo_id="mhussainahmad/averformer-meld-v4", filename="pytorch_model.pth")
39
 
40
  model = AVERFormerV4(
41
  audio_backbone=cfg["audio_backbone"],
 
54
  ## Live inference
55
 
56
  ```bash
57
+ python live_emotion_v4.py --repo_id mhussainahmad/averformer-meld-v4
58
  ```
59
 
60
  See `LIVE_INFERENCE_README.md` in the GitHub repo for full setup.