Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
base_model: Wan-AI/Wan2.1-I2V-14B-480P-Diffusers
|
| 3 |
+
tags:
|
| 4 |
+
- lora
|
| 5 |
+
- qlora
|
| 6 |
+
- egocentric
|
| 7 |
+
- exo-to-ego
|
| 8 |
+
- video-generation
|
| 9 |
+
- egoX
|
| 10 |
+
---
|
| 11 |
+
|
| 12 |
+
# EgoX mix LoRA — rank-128 QLoRA (single 24 GB GPU reproduction)
|
| 13 |
+
|
| 14 |
+
A **LoRA adapter** for **Wan2.1-I2V-14B-480P** reproducing **EgoX** (exocentric → egocentric video
|
| 15 |
+
generation) on a **single 24 GB GPU** (vs the paper's 4× H200). Trained with the paper's
|
| 16 |
+
Geometry-Guided Self-Attention (GGA), scaled down to fit the hardware budget.
|
| 17 |
+
|
| 18 |
+

|
| 19 |
+
|
| 20 |
+
*exo input | our vitS ego-prior | ego pred (our prior) | ego pred (shipped prior)*
|
| 21 |
+
|
| 22 |
+
## Training
|
| 23 |
+
- **Adapter:** LoRA rank 128 / α 128 on the Wan-14B transformer (NF4 4-bit base + 8-bit AdamW)
|
| 24 |
+
- **Data:** 552 clips, **all 17 EgoX domains** (Ego-Exo4D); 495 train / 57 val
|
| 25 |
+
- **Resolution:** 49×176×704 · **LR** 2e-5, constant-with-warmup · effective batch 4
|
| 26 |
+
- **16 epochs / 1,984 steps**, held-out val loss **0.247 → 0.173** (checkpoint-1984)
|
| 27 |
+
|
| 28 |
+
## Use
|
| 29 |
+
Load as an unfused PEFT adapter onto an NF4-quantized Wan2.1-I2V-14B with GGA enabled
|
| 30 |
+
(see the [reproduction repo](https://github.com/skr3178/egoX): `EgoX/infer_nf4.py --use_GGA`).
|
| 31 |
+
|
| 32 |
+
## Caveats
|
| 33 |
+
Scaled-down repro: 176-res output, NF4-r128, 552 clips (vs the paper's ~3,510). Image-criteria
|
| 34 |
+
scores trail the paper (~11 vs 16 PSNR) due to resolution + budget, not pipeline fidelity.
|