Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,55 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
tags:
|
| 4 |
+
- egowm
|
| 5 |
+
- wan2.1
|
| 6 |
+
- hand-pose
|
| 7 |
+
- camera-pose
|
| 8 |
+
- egocentric
|
| 9 |
+
library_name: pytorch
|
| 10 |
+
---
|
| 11 |
+
|
| 12 |
+
# EgoWM Lane B — Wan2.1-T2V-1.3B + cam/hand heads (frozen trunk)
|
| 13 |
+
|
| 14 |
+
Project: [github.com/geng-haoran/EgoWM](https://github.com/geng-haoran/EgoWM)
|
| 15 |
+
|
| 16 |
+
This ckpt is Lane B's first real-finetune-at-scale on a single GH200:
|
| 17 |
+
the Wan2.1-T2V-1.3B DiT trunk is **frozen** and 8.37 M trainable parameters
|
| 18 |
+
on the camera + hand regression heads are tuned with rectified-flow latents
|
| 19 |
+
cached from 1,873 ARCTIC clips.
|
| 20 |
+
|
| 21 |
+
## Held-out evaluation (24-clip val pool, ARCTIC)
|
| 22 |
+
|
| 23 |
+
| metric | this ckpt | predict-train-mean baseline | Δ |
|
| 24 |
+
|---|---|---|---|
|
| 25 |
+
| Hand 21-keypoint MPJPE (mm) | 195.8 | 207.3 | **-5.5 %** |
|
| 26 |
+
| Hand wrist translation MAE (mm) | 181.8 | 178.1 | **+2.1 %** |
|
| 27 |
+
| Camera translation MAE (cm) | 7.53 | 3.64 | +107.2 % |
|
| 28 |
+
|
| 29 |
+
Training details: 2000 steps × 1 step/s on the cached-latent ARCTIC pool;
|
| 30 |
+
lr 5e-4; AdamW. Loss curves + per-ckpt eval in the repo (`_viz/21_train_curve.png`,
|
| 31 |
+
`_viz/30_final_summary.png`).
|
| 32 |
+
|
| 33 |
+
## Loading
|
| 34 |
+
|
| 35 |
+
```python
|
| 36 |
+
import torch
|
| 37 |
+
from egowm.models.egowm_toy import EgoWMToy
|
| 38 |
+
from egowm.models.wan_load import load_pretrained_wan
|
| 39 |
+
m = EgoWMToy(z_rgb=16, cond_ch=17, dim=1536, ffn_dim=8960, layers=30,
|
| 40 |
+
heads=12, model_type="t2v", cam_dim=7, hand_dim=61, n_hands=2)
|
| 41 |
+
load_pretrained_wan(m.dit, "Wan2.1-T2V-1.3B/diffusion_pytorch_model.safetensors")
|
| 42 |
+
sd = torch.load("ckpt_step001200.pt", weights_only=False)["state_dict"]
|
| 43 |
+
m.load_state_dict(sd)
|
| 44 |
+
```
|
| 45 |
+
|
| 46 |
+
## Manifest
|
| 47 |
+
|
| 48 |
+
Trained on: data_engine/manifests/train_egowm_arctic.jsonl (1873 ARCTIC clips with
|
| 49 |
+
both DA3 metric geometry and Wan2.1-VAE latents cached on disk).
|
| 50 |
+
|
| 51 |
+
## Lane B / Lane A handoff
|
| 52 |
+
|
| 53 |
+
This was produced by **h200_1 (Lane B)** of the EgoWM multi-machine sprint. See
|
| 54 |
+
`docs/WORK_CLAIMS.md` + `docs/17_lane_b_e2e_report.md` for the full pipeline,
|
| 55 |
+
HaWoR auto-label outputs, dataset deep-dive PPT, and remaining steps.
|