Upload README.md with huggingface_hub

README.md

@@ -19,19 +19,88 @@ model_name: vitra-gigahands-allcam0-keypoints-mano-step60000
 
 # VITRA GigaHands all-cam0 keypoints_mano step60000
 
-This repository contains the finetuned VITRA checkpoint at training step 60000 for GigaHands all-cam0 training with `action_type=keypoints` and `keypoints_source=mano`.
+This repository contains the fine-tuned VITRA checkpoint at training step 60000 for GigaHands all-cam0 training with `action_type=keypoints` and `keypoints_source=mano`.
 
 ## Files
 - `checkpoints/epoch=0-step=60000.ckpt/weights.pt`: model weights
 - `checkpoints/epoch=0-step=60000.ckpt/meta.json`: checkpoint metadata
 - `config/human_pretrain_gigahands_real_all_cam0_keypoints_mano_vitra3b_linked.json`: training config
-- `eval/metrics_comparison.json`: base vs step60000 evaluation on 50 test clips
+- `eval/metrics_comparison.json`: base vs. step60000 evaluation on 50 test clips
 
-## Eval summary
-- action_mse: 16.103662 -> 0.670375
-- left_action_mse: 3.385067 -> 0.750329
-- right_action_mse: 45.174740 -> 0.487623
-- dual_hand_action_mse: 16.103662 -> 0.670375
+## Experimental Setup
+
+### Model
+This checkpoint is a fine-tuned version of the VITRA-VLA 3B base model on GigaHands egocentric hand-action data.
+
+### Dataset
+We fine-tuned on a converted GigaHands split built from all available egocentric `cam0` views.
+
+- Total clips: 29,901
+- Training clips: 28,405
+- Test clips: 1,496
+
+After conversion into the VITRA stage-1 format, this produced:
+
+- 7,307,829 train frame-level samples
+- 383,606 test frame-level samples
+- 7,691,435 total frame-level samples
+
+Camera distribution:
+
+- `brics-odroid-001_cam0`: 13,456 clips
+- `brics-odroid-002_cam0`: 13,619 clips
+- `brics-odroid-003_cam0`: 2,826 clips
+
+### Training Target
+The model is not trained to predict future RGB frames.
+
+Each training sample uses:
+
+- the current RGB frame,
+- the language instruction,
+- and the current hand state,
+
+to predict a 16-step future hand-action chunk.
+
+For this experiment:
+
+- `action_type = keypoints`
+- the target representation is derived from `keypoints_3d_mano`
+
+So the model is trained for vision-language-conditioned future hand motion prediction, not image generation.
+
+### Loss
+Training uses the native diffusion action loss in VITRA rather than a plain MSE loss.
+
+During evaluation, we report action-space MSE as an external metric:
+
+- `action_mse`
+- `left_action_mse`
+- `right_action_mse`
+
+These MSE values are used only for evaluation and comparison; they are not the optimization objective used during training.
+
+### Fine-tuning Setup
+The released checkpoint corresponds to a run fine-tuned from the VITRA 3B base model with:
+
+- global batch size: 2
+- prediction horizon: 16 future steps
+- released checkpoint: training step 60000
+
+### Evaluation
+We compare the fine-tuned checkpoint against the original VITRA 3B base model on the same GigaHands test split.
+
+On a 50-clip evaluation subset, the fine-tuned checkpoint improved over the base model from:
+
+- `action_mse`: 16.103662 -> 0.670375
+- `left_action_mse`: 3.385067 -> 0.750329
+- `right_action_mse`: 45.174740 -> 0.487623
+- `dual_hand_action_mse`: 16.103662 -> 0.670375
+
+This corresponds to large relative improvements, especially on right-hand motion prediction.
+
+## Qualitative Evaluation
+For qualitative analysis, we also generate RGB overlay videos comparing raw GT, the VITRA base model, and the fine-tuned model on selected test clips.
 
 ## Notes
 - Base checkpoint: `VITRA-VLA/VITRA-VLA-3B`