FaceFill

Causal GRU that predicts 11 upper-face ARKit morphs (brows / eyes / blinks) from 30 lower-face channels when an HMD occludes the upper face.

~3.36M params · streaming step with GRU hidden state.

Files

File	Description
model.onnx	Stateful single-frame step (opset 17)
eval.json	Autoregressive test metrics (held-out MEAD W009)

Inputs / outputs

Inputs

Name	Shape	Meaning
current	[B, 80]	lower + vel + head + conf + blink state
history	[B, 176]	prior full morphs + vel + obs mask + …
u_prev	[B, 11]	previous upper prediction
hidden_in	[3, B, 384]	GRU hidden

Outputs

Name	Shape
u_hat	[B, 11]
u_abs	[B, 11]
u_delta	[B, 11]
blink_onset_logits	[B, 3]
blink_duration	[B]
blink_amplitude	[B]
hidden_out	[3, B, 384]

Combine rule: û = clamp(α·(u_prev + Δu) + (1−α)·u_abs), α=0.8.

Training data

Features derived from MEAD frontal videos via MediaPipe Face Landmarker (person-disjoint M033 / W015 / W009 for this release). Does not redistribute MEAD videos.

Eval (AR, test)

• mean MAE 0.075
• mean Pearson 0.82
• blink F1 0.73

Usage

import onnxruntime as ort
import numpy as np

sess = ort.InferenceSession("model.onnx")
B = 1
feeds = {
    "current": np.zeros((B, 80), np.float32),
    "history": np.zeros((B, 176), np.float32),
    "u_prev": np.zeros((B, 11), np.float32),
    "hidden_in": np.zeros((3, B, 384), np.float32),
}
outs = sess.run(None, feeds)
# u_hat, u_abs, u_delta, blink_onset_logits, blink_duration, blink_amplitude, hidden_out

Browser / WebXR companion: load with ONNX Runtime Web (see FaceFill / WebXREmotion projects).

License

MIT (this model). Respect MEAD terms for any further training on that dataset.