---
language:
- en
license: other
library_name: pytorch
tags:
- vitra
- paligemma
- hand-pose
- motion-prediction
- gigahands
- fine-tuned
base_model:
- VITRA-VLA/VITRA-VLA-3B
datasets:
- brown-ivl/GigaHands
model_name: vitra-gigahands-allcam0-keypoints-mano-step60000
---

# VITRA GigaHands all-cam0 keypoints_mano step60000

This repository contains the fine-tuned VITRA checkpoint at training step 60000 for GigaHands all-cam0 training with `action_type=keypoints` and `keypoints_source=mano`.

## Files
- `checkpoints/epoch=0-step=60000.ckpt/weights.pt`: model weights
- `checkpoints/epoch=0-step=60000.ckpt/meta.json`: checkpoint metadata
- `config/human_pretrain_gigahands_real_all_cam0_keypoints_mano_vitra3b_linked.json`: training config
- `eval/metrics_comparison.json`: base vs. step60000 evaluation on 50 test clips

## Experimental Setup

### Model
This checkpoint is a fine-tuned version of the VITRA-VLA 3B base model on GigaHands egocentric hand-action data.

### Dataset
We fine-tuned on a converted GigaHands split built from all available egocentric `cam0` views.

- Total clips: 29,901
- Training clips: 28,405
- Test clips: 1,496

After conversion into the VITRA stage-1 format, this produced:

- 7,307,829 train frame-level samples
- 383,606 test frame-level samples
- 7,691,435 total frame-level samples

Camera distribution:

- `brics-odroid-001_cam0`: 13,456 clips
- `brics-odroid-002_cam0`: 13,619 clips
- `brics-odroid-003_cam0`: 2,826 clips

### Training Target
The model is not trained to predict future RGB frames.

Each training sample uses:

- the current RGB frame,
- the language instruction,
- and the current hand state,

to predict a 16-step future hand-action chunk.

For this experiment:

- `action_type = keypoints`
- the target representation is derived from `keypoints_3d_mano`

So the model is trained for vision-language-conditioned future hand motion prediction, not image generation.

### Loss
Training uses the native diffusion action loss in VITRA rather than a plain MSE loss.

During evaluation, we report action-space MSE as an external metric:

- `action_mse`
- `left_action_mse`
- `right_action_mse`

These MSE values are used only for evaluation and comparison; they are not the optimization objective used during training.

### Fine-tuning Setup
The released checkpoint corresponds to a run fine-tuned from the VITRA 3B base model with:

- global batch size: 2
- prediction horizon: 16 future steps
- released checkpoint: training step 60000

### Evaluation
We compare the fine-tuned checkpoint against the original VITRA 3B base model on the same GigaHands test split.

On a 50-clip evaluation subset, the fine-tuned checkpoint improved over the base model from:

- `action_mse`: 16.103662 -> 0.670375
- `left_action_mse`: 3.385067 -> 0.750329
- `right_action_mse`: 45.174740 -> 0.487623
- `dual_hand_action_mse`: 16.103662 -> 0.670375

This corresponds to large relative improvements, especially on right-hand motion prediction.

## Qualitative Evaluation
For qualitative analysis, we also generate RGB overlay videos comparing raw GT, the VITRA base model, and the fine-tuned model on selected test clips.

## Notes
- Base checkpoint: `VITRA-VLA/VITRA-VLA-3B`
- Dataset setting: GigaHands all `cam0` views
- Training target: `action_type=keypoints`
- Keypoint source: `keypoints_3d_mano`
- Optimizer state was not uploaded.