somaghr commited on
Commit
d64122c
·
verified ·
1 Parent(s): e10bdf9

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +55 -0
README.md ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags:
4
+ - egowm
5
+ - wan2.1
6
+ - hand-pose
7
+ - camera-pose
8
+ - egocentric
9
+ library_name: pytorch
10
+ ---
11
+
12
+ # EgoWM Lane B — Wan2.1-T2V-1.3B + cam/hand heads (frozen trunk)
13
+
14
+ Project: [github.com/geng-haoran/EgoWM](https://github.com/geng-haoran/EgoWM)
15
+
16
+ This ckpt is Lane B's first real-finetune-at-scale on a single GH200:
17
+ the Wan2.1-T2V-1.3B DiT trunk is **frozen** and 8.37 M trainable parameters
18
+ on the camera + hand regression heads are tuned with rectified-flow latents
19
+ cached from 1,873 ARCTIC clips.
20
+
21
+ ## Held-out evaluation (24-clip val pool, ARCTIC)
22
+
23
+ | metric | this ckpt | predict-train-mean baseline | Δ |
24
+ |---|---|---|---|
25
+ | Hand 21-keypoint MPJPE (mm) | 195.8 | 207.3 | **-5.5 %** |
26
+ | Hand wrist translation MAE (mm) | 181.8 | 178.1 | **+2.1 %** |
27
+ | Camera translation MAE (cm) | 7.53 | 3.64 | +107.2 % |
28
+
29
+ Training details: 2000 steps × 1 step/s on the cached-latent ARCTIC pool;
30
+ lr 5e-4; AdamW. Loss curves + per-ckpt eval in the repo (`_viz/21_train_curve.png`,
31
+ `_viz/30_final_summary.png`).
32
+
33
+ ## Loading
34
+
35
+ ```python
36
+ import torch
37
+ from egowm.models.egowm_toy import EgoWMToy
38
+ from egowm.models.wan_load import load_pretrained_wan
39
+ m = EgoWMToy(z_rgb=16, cond_ch=17, dim=1536, ffn_dim=8960, layers=30,
40
+ heads=12, model_type="t2v", cam_dim=7, hand_dim=61, n_hands=2)
41
+ load_pretrained_wan(m.dit, "Wan2.1-T2V-1.3B/diffusion_pytorch_model.safetensors")
42
+ sd = torch.load("ckpt_step001200.pt", weights_only=False)["state_dict"]
43
+ m.load_state_dict(sd)
44
+ ```
45
+
46
+ ## Manifest
47
+
48
+ Trained on: data_engine/manifests/train_egowm_arctic.jsonl (1873 ARCTIC clips with
49
+ both DA3 metric geometry and Wan2.1-VAE latents cached on disk).
50
+
51
+ ## Lane B / Lane A handoff
52
+
53
+ This was produced by **h200_1 (Lane B)** of the EgoWM multi-machine sprint. See
54
+ `docs/WORK_CLAIMS.md` + `docs/17_lane_b_e2e_report.md` for the full pipeline,
55
+ HaWoR auto-label outputs, dataset deep-dive PPT, and remaining steps.