metadata
license: mit
tags:
- onnx
- facial-expression
- arkit
- vr
- mediapipe
- gru
library_name: onnx
pipeline_tag: other
FaceFill
Causal GRU that predicts 11 upper-face ARKit morphs (brows / eyes / blinks) from 30 lower-face channels when an HMD occludes the upper face.
~3.36M params · streaming step with GRU hidden state.
Files
| File | Description |
|---|---|
model.onnx |
Stateful single-frame step (opset 17) |
eval.json |
Autoregressive test metrics (held-out MEAD W009) |
Inputs / outputs
Inputs
| Name | Shape | Meaning |
|---|---|---|
current |
[B, 80] |
lower + vel + head + conf + blink state |
history |
[B, 176] |
prior full morphs + vel + obs mask + … |
u_prev |
[B, 11] |
previous upper prediction |
hidden_in |
[3, B, 384] |
GRU hidden |
Outputs
| Name | Shape |
|---|---|
u_hat |
[B, 11] |
u_abs |
[B, 11] |
u_delta |
[B, 11] |
blink_onset_logits |
[B, 3] |
blink_duration |
[B] |
blink_amplitude |
[B] |
hidden_out |
[3, B, 384] |
Combine rule: û = clamp(α·(u_prev + Δu) + (1−α)·u_abs), α=0.8.
Training data
Features derived from MEAD frontal videos via MediaPipe Face Landmarker (person-disjoint M033 / W015 / W009 for this release). Does not redistribute MEAD videos.
Eval (AR, test)
- mean MAE 0.075
- mean Pearson 0.82
- blink F1 0.73
Usage
import onnxruntime as ort
import numpy as np
sess = ort.InferenceSession("model.onnx")
B = 1
feeds = {
"current": np.zeros((B, 80), np.float32),
"history": np.zeros((B, 176), np.float32),
"u_prev": np.zeros((B, 11), np.float32),
"hidden_in": np.zeros((3, B, 384), np.float32),
}
outs = sess.run(None, feeds)
# u_hat, u_abs, u_delta, blink_onset_logits, blink_duration, blink_amplitude, hidden_out
Browser / WebXR companion: load with ONNX Runtime Web (see FaceFill / WebXREmotion projects).
License
MIT (this model). Respect MEAD terms for any further training on that dataset.