mhussainahmad/averformer-cremad-v4

AVERFormer-v4 (CREMA-D)

Multimodal Audio-Visual-Text Emotion Recognition transformer trained on CREMA-D. Code: https://github.com/mhussainahmad/AVERFormer

Reported numbers

• Best single-seed val wF1: 0.8320558803202199 (seed 100, epoch 12)
• Best ensemble wF1: 0.8278 (top-3 ensemble)

Classes (6)

['anger', 'disgust', 'fear', 'happy', 'neutral', 'sad']

Architecture

• Audio: microsoft/wavlm-large (16 kHz mono waveform)
• Video: MCG-NJU/videomae-large (16 frames @ 224×224 RGB)
• Text: microsoft/deberta-v3-large (speaker-aware ctx encoder)
• Fusion: 2-layer cross-modal transformer, dim=512, 8 heads
• Heads: face / voice / text / joint (all share class count)

Loading

import json, torch
from huggingface_hub import hf_hub_download
from models.averformer_v4 import AVERFormerV4

cfg = json.load(open(hf_hub_download(repo_id="mhussainahmad/averformer-cremad-v4", filename="config.json")))
ckpt = hf_hub_download(repo_id="mhussainahmad/averformer-cremad-v4", filename="pytorch_model.pth")

model = AVERFormerV4(
    audio_backbone=cfg["audio_backbone"],
    video_backbone=cfg["video_backbone"],
    text_backbone=cfg["text_backbone"],
    num_classes=cfg["num_classes"],
    fusion_layers=cfg["fusion_layers"],
    lora_r=cfg["lora_r"],
    use_text=True,
)
state = torch.load(ckpt, map_location="cpu", weights_only=False)
model.load_state_dict(state["model"], strict=False)
model.eval()

Live inference

python live_emotion_v4.py --repo_id mhussainahmad/averformer-cremad-v4

See LIVE_INFERENCE_README.md in the GitHub repo for full setup.

Training command (reference)

python train_v5_spec.py --corpus CREMA-D --classes 6 \
  --audio microsoft/wavlm-large \
  --video MCG-NJU/videomae-large \
  --text  microsoft/deberta-v3-large \
  --lora_r 16 \
  --epochs 20 --bs 4 \
  --grad_accum 4 \
  --lr_head 0.0001 --lr_backbone 2e-05 \
  --loss ce --class_weights sqrt-inv \
  --select_metric wF1 \
  --seed 100