NiDeYingZiD commited on
Commit
3dd252e
Β·
verified Β·
1 Parent(s): 43c0ef9

Upload 2 files

Browse files
Files changed (3) hide show
  1. .gitattributes +1 -0
  2. README.md +101 -0
  3. architecture.png +3 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ architecture.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -1,3 +1,104 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ language:
4
+ - en
5
+ library_name: pytorch
6
+ tags:
7
+ - dynamic-facial-expression-recognition
8
+ - dfer
9
+ - audio-visual
10
+ - emotion-recognition
11
+ - affective-computing
12
+ - multimodal
13
+ - mamba
14
+ - state-space-model
15
+ - www2026
16
  ---
17
+
18
+ <div align="center">
19
+
20
+ # BHGap Β· Trained Checkpoints
21
+
22
+ ### A Deep Iterative Prompting and Multi-stage Alignment Framework for Dynamic Facial Expression Recognition
23
+
24
+ <p>
25
+ <a href="https://doi.org/10.1145/3774904.3792417"><img src="https://img.shields.io/badge/Venue-WWW%202026%20(Oral)-1f6feb?style=flat-square" alt="WWW 2026 Oral"></a>
26
+ <a href="https://doi.org/10.1145/3774904.3792417"><img src="https://img.shields.io/badge/Paper-ACM%20DL-b31b1b?style=flat-square" alt="Paper"></a>
27
+ <a href="https://github.com/NDYZD666/-public-BHGap"><img src="https://img.shields.io/badge/Code-GitHub-181717?style=flat-square&logo=github&logoColor=white" alt="GitHub"></a>
28
+ <img src="https://img.shields.io/badge/Task-Audio--Visual%20DFER-8957e5?style=flat-square" alt="Task">
29
+ </p>
30
+
31
+ </div>
32
+
33
+ > πŸ€— This repository hosts the **trained model weights** for **BHGap** (WWW 2026 Oral).
34
+ > For the full code, training and evaluation pipeline, please visit the **[GitHub repository](https://github.com/NDYZD666/-public-BHGap)**.
35
+
36
+ <div align="center">
37
+ <img src="https://huggingface.co/NiDeYingZiD/BHGap-ckpt/resolve/main/architecture.png" width="92%" alt="BHGap architecture">
38
+ <br>
39
+ <em>Frozen audio-visual encoders + SDIC (iterative cross-modal prompting) + MSC2F (coarse-to-fine alignment) + lightweight fusion.</em>
40
+ </div>
41
+
42
+ ## πŸ“– Abstract
43
+
44
+ Dynamic Facial Expression Recognition (DFER) is a crucial part of affective computing, with broad applications in human-computer interaction and social media content analysis. Effectively integrating audio-visual signals remains the core challenge, as existing approaches are constrained by **(1) shallow, static fusion** that fails to capture the dynamic co-evolution of features, and **(2) implicit, coarse alignment** that cannot bridge the modality gap.
45
+
46
+ **BHGap** reformulates audio-visual collaboration from a one-shot fusion event into a continuous, reciprocal generation process spanning every layer of frozen backbone encoders:
47
+
48
+ - **SDIC** β€” an SSM (Mamba)-based **Cross-Modal Prompt Generator** dynamically produces *guidance prompts* for the counterpart modality at each encoding layer, enabling deep and fine-grained feature co-evolution.
49
+ - **MSC2F** β€” a coarse-to-fine alignment module that combines **low-rank adversarial alignment** (macro-level distribution & spatio-temporal congruity) with **MMD-driven implicit differentiation** (micro-level statistical & semantic consistency).
50
+
51
+ It achieves **state-of-the-art** performance on **DFEW** and **MAFW** using raw audio-visual inputs only.
52
+
53
+ ## πŸ“Š Results
54
+
55
+ Evaluated under 5-fold cross-validation (**WAR** = Weighted Average Recall, **UAR** = Unweighted Average Recall).
56
+
57
+ | Dataset | Modality | WAR (%) | UAR (%) |
58
+ | --- | :---: | :---: | :---: |
59
+ | **DFEW** | Audio + Visual | **78.80** | **69.03** |
60
+ | **MAFW** | Audio + Visual | **59.97** | **47.68** |
61
+
62
+ ## πŸ“¦ Checkpoints
63
+
64
+ This repository provides the trained BHGap checkpoints for **DFEW** and **MAFW** (5-fold). Each fold produces the best models selected by WAR / UAR (`model_best_war.pth`, `model_best_uar.pth`); see the **Files and versions** tab for the exact contents.
65
+
66
+ ## πŸš€ Usage
67
+
68
+ Download the checkpoints from the Hub:
69
+
70
+ ```bash
71
+ huggingface-cli download NiDeYingZiD/BHGap-ckpt --local-dir ./checkpoints
72
+ ```
73
+
74
+ ```python
75
+ from huggingface_hub import snapshot_download
76
+
77
+ snapshot_download(repo_id="NiDeYingZiD/BHGap-ckpt", local_dir="./checkpoints")
78
+ ```
79
+
80
+ Then evaluate with the code from the [GitHub repository](https://github.com/NDYZD666/-public-BHGap):
81
+
82
+ ```bash
83
+ python evaluate.py --dataset DFEW --checkpoint ./checkpoints/model_best_war.pth --fold 5
84
+ ```
85
+
86
+ > **Note** β€” These are task checkpoints for the BHGap pipeline (frozen MAE-Face / AudioMAE backbones + trainable SDIC & MSC2F modules), not a standalone `transformers` model. Please load them through the BHGap code.
87
+
88
+ ## πŸ“Œ Citation
89
+
90
+ ```bibtex
91
+ @inproceedings{zhang2026bhgap,
92
+ title = {BHGap: A Deep Iterative Prompting and Multi-stage Alignment Framework for Dynamic Facial Expression Recognition},
93
+ author = {Zhang, Yichi and Han, Yunqi and Ding, Jiayue and Chen, Liangyu},
94
+ booktitle = {Proceedings of the ACM Web Conference 2026 (WWW '26)},
95
+ year = {2026},
96
+ address = {Dubai, United Arab Emirates},
97
+ publisher = {Association for Computing Machinery},
98
+ doi = {10.1145/3774904.3792417}
99
+ }
100
+ ```
101
+
102
+ ## πŸ™ Acknowledgements
103
+
104
+ Built upon [MAE-Face](https://github.com/FuxiVirtualHuman/MAE-Face), [AudioMAE](https://github.com/facebookresearch/AudioMAE), [Mamba](https://github.com/state-spaces/mamba), [MMA-DFER](https://github.com/katerynaCh/MMA-DFER), and the [DFEW](https://dfew-dataset.github.io/) / [MAFW](https://mafw-database.github.io/MAFW/) datasets. Thank you for these excellent efforts!
architecture.png ADDED

Git LFS Details

  • SHA256: 3020febc57620713944f855f571fafd42d4b1190d8a8c765a628d2b1ddee40d0
  • Pointer size: 131 Bytes
  • Size of remote file: 337 kB