LeoJiangOR/vitra-gigahands-allcam0-keypoints-mano-step60000

VITRA GigaHands all-cam0 keypoints_mano step60000

This repository contains the fine-tuned VITRA checkpoint at training step 60000 for GigaHands all-cam0 training with action_type=keypoints and keypoints_source=mano.

Files

• checkpoints/epoch=0-step=60000.ckpt/weights.pt: model weights
• checkpoints/epoch=0-step=60000.ckpt/meta.json: checkpoint metadata
• config/human_pretrain_gigahands_real_all_cam0_keypoints_mano_vitra3b_linked.json: training config
• eval/metrics_comparison.json: base vs. step60000 evaluation on 50 test clips

Experimental Setup

Model

This checkpoint is a fine-tuned version of the VITRA-VLA 3B base model on GigaHands egocentric hand-action data.

Dataset

We fine-tuned on a converted GigaHands split built from all available egocentric cam0 views.

• Total clips: 29,901
• Training clips: 28,405
• Test clips: 1,496

After conversion into the VITRA stage-1 format, this produced:

• 7,307,829 train frame-level samples
• 383,606 test frame-level samples
• 7,691,435 total frame-level samples

Camera distribution:

• brics-odroid-001_cam0: 13,456 clips
• brics-odroid-002_cam0: 13,619 clips
• brics-odroid-003_cam0: 2,826 clips

Training Target

The model is not trained to predict future RGB frames.

Each training sample uses:

• the current RGB frame,
• the language instruction,
• and the current hand state,

to predict a 16-step future hand-action chunk.

For this experiment:

• action_type = keypoints
• the target representation is derived from keypoints_3d_mano

So the model is trained for vision-language-conditioned future hand motion prediction, not image generation.

Loss

Training uses the native diffusion action loss in VITRA rather than a plain MSE loss.

During evaluation, we report action-space MSE as an external metric:

• action_mse
• left_action_mse
• right_action_mse

These MSE values are used only for evaluation and comparison; they are not the optimization objective used during training.

Fine-tuning Setup

The released checkpoint corresponds to a run fine-tuned from the VITRA 3B base model with:

• global batch size: 2
• prediction horizon: 16 future steps
• released checkpoint: training step 60000

Evaluation

We compare the fine-tuned checkpoint against the original VITRA 3B base model on the same GigaHands test split.

On a 50-clip evaluation subset, the fine-tuned checkpoint improved over the base model from:

• action_mse: 16.103662 -> 0.670375
• left_action_mse: 3.385067 -> 0.750329
• right_action_mse: 45.174740 -> 0.487623
• dual_hand_action_mse: 16.103662 -> 0.670375

This corresponds to large relative improvements, especially on right-hand motion prediction.

Qualitative Evaluation

For qualitative analysis, we also generate RGB overlay videos comparing raw GT, the VITRA base model, and the fine-tuned model on selected test clips.

Notes

• Base checkpoint: VITRA-VLA/VITRA-VLA-3B
• Dataset setting: GigaHands all cam0 views
• Training target: action_type=keypoints
• Keypoint source: keypoints_3d_mano
• Optimizer state was not uploaded.