--- datasets: - wantobcm/so101_orange_pick_gr00tn1.5 metrics: - mse base_model: - nvidia/GR00T-N1.5-3B model-index: - name: flrs/so101_orange_pick_gr00tn1.5_model results: - task: name: Imitation Policy Evaluation type: robotic-manipulation dataset: name: wantobcm/so101_orange_pick_gr00tn1.5 type: video metrics: - name: Train MSE type: mse value: 23.751963892617113 args: script: scripts/eval_policy.py model_path: flrs/so101_orange_pick_gr00tn1.5_model embodiment_tag: new_embodiment data_config: so100_dualcam dataset_path: wantobcm/so101_orange_pick_gr00tn1.5 modality_keys: - single_arm - gripper trajs: 40 pipeline_tag: robotics tags: - LeIsaac - LeRobot - SO-101 --- # Model Card for so101_orange_pick_gr00tn1.5_model [NVIDIA Isaac GR00T N1.5 model](https://huggingface.co/nvidia/GR00T-N1.5-3B) fine-tuned on the [so101_orange_pick dataset](https://huggingface.co/datasets/wantobcm/so101_orange_pick_gr00tn1.5) for experimenting with the robot arm [LeRobot SO-101](https://github.com/TheRobotStudio/SO-ARM100) in a dual-camera setup *The video preview has been generated from model training data in the dataset.* ## Model Details ### Model Description This model is a version of [NVIDIA's GR00T N1.5](https://research.nvidia.com/labs/gear/gr00t-n1_5/) fine-tuned on the [so101_orange_pick dataset](https://huggingface.co/datasets/wantobcm/so101_orange_pick_gr00tn1.5). The model is relevant of the context of [LightwheelAI's LeIsaac](https://github.com/LightwheelAI/leisaac), where it models the standard task "[LeIsaac-SO101-PickOrange-v0](https://github.com/LightwheelAI/leisaac/blob/7a2f6959d6bc961cf3c34d098c0057c2d8d6e308/source/leisaac/leisaac/tasks/pick_orange/__init__.py)". In this task, a SO-ARM101 robo arm picks up 3 oranges from a table and places them in a bowl, one after another. The robot is equipped with a front and wrist camera. - **Developed by:** Florian Roscheck, based on work of [wantobcm](https://huggingface.co/datasets/wantobcm/so101_orange_pick_gr00tn1.5) and [model](https://huggingface.co/nvidia/GR00T-N1.5-3B) authors - **License:** Non-commercial use (See 3.2, [NVIDIA license](https://developer.download.nvidia.com/licenses/NVIDIA-OneWay-Noncommercial-License-22Mar2022.pdf?t=eyJscyI6ImdzZW8iLCJsc2QiOiJodHRwczovL3d3dy5nb29nbGUuY29tLyIsIm5jaWQiOiJzby15b3V0LTg3MTcwMS12dDQ4In0=)) - **Finetuned from model:** [NVIDIA GR00T-N1.5-3B](https://huggingface.co/nvidia/GR00T-N1.5-3B) ## Uses The model is intended for researchers and hobbyists who would like to experiment with LeRobot, GR00T inference, LeIsaac and NVIDIA's [Isaac Sim](https://developer.nvidia.com/isaac/sim). ## How to Get Started with the Model To learn more about how to set up the environment for using the model as an inference service, refer to the instructions in the [Isaac-GR00T repo](https://github.com/NVIDIA/Isaac-GR00T). You can run the inference server for the model with the following command: ```bash python scripts/inference_service.py --model-path flrs/so101_orange_pick_gr00tn1.5_model --server --embodiment-tag new_embodiment --data-config so100_dualcam ``` ## Training Details ### Training Data See the [Dataset Card](https://huggingface.co/datasets/wantobcm/so101_orange_pick_gr00tn1.5) for more information on the training data. All data in the dataset was used for training and no test data was withheld. You can preview the dataset in the [LeRobot Dataset Visualizer](https://huggingface.co/spaces/lerobot/visualize_dataset?path=%2Fwantobcm%2Fso101_orange_pick_gr00tn1.5). ### Training Procedure Training was done on an NVIDIA L4 GPU, driver version 550.54.15, CUDA 12.4. 1. Set up the environment as described in the [Isaac-GR00T repo](https://github.com/NVIDIA/Isaac-GR00T). 2. Download the dataset from the [Hugging Face Hub](https://huggingface.co/datasets/wantobcm/so101_orange_pick_gr00tn1.5), e.g. via the Hugging Face CLI: ```bash hf download --repo-type dataset --local-dir ./dataset wantobcm/so101_orange_pick_gr00tn1.5 ``` 3. In order to train the model, you need a modality file. Create the file `dataset/meta/modality.json` with the following content: ```json { "state": { "single_arm": { "start": 0, "end": 5 }, "gripper": { "start": 5, "end": 6 } }, "action": { "single_arm": { "start": 0, "end": 5 }, "gripper": { "start": 5, "end": 6 } }, "video": { "front": { "original_key": "observation.images.front" }, "wrist": { "original_key": "observation.images.wrist" } }, "annotation": { "human.task_description": { "original_key": "task_index" } } } ``` 4. Train (fine-tune) the model with the following command: ```bash python scripts/gr00t_finetune.py \ --dataset-path ./dataset \ --num-gpus 1 \ --output-dir ./so101_orange_pick_gr00tn1.5_model \ --max-steps 6000 \ --data-config so100_dualcam \ --video-backend torchvision_av \ --no-tune_diffusion_model \ --dataloader-num-workers 1 \ --batch-size 16 \ --dataloader-prefetch-factor 1 ``` > **Note:** The following adjustments were made to accommodate for infrastructure limitations: > > - `--num-gpus 1` > - `--dataloader-num-workers 1` > - `--batch-size 16` > - `--dataloader-prefetch-factor 1` > - `--no-tune_diffusion_model` #### Training Hyperparameters Default hyperparameters for GR00T were used (as of commit [1259d62](https://github.com/NVIDIA/Isaac-GR00T/tree/1259d624f0405731b19a728c7e4f6bdf57063fa2)), except for the ones set via command line arguments above. ## Evaluation Evaluation was run via the following GR00T evaluation script: ```bash python scripts/eval_policy.py \ --model_path ./so101_orange_pick_gr00tn1.5_model \ --embodiment-tag new_embodiment \ --data-config so100_dualcam \ --dataset_path ./dataset \ --modality-keys single_arm gripper \ --trajs 40 ``` ### Testing Data, Factors & Metrics #### Testing Data Evaluation of the model was done on the full training set, as no separate test set was withheld. For the training set, see dataset card linked above. #### Metrics In accordance with the output of the evaluation script, MSE (Mean Squared Error) for all trajectories were used. ### Results The MSE for all 40 trajectories was 23.752. An example trajectory is visualized in the following image (image created via the evaluation script for trajectory 10): ## Environmental Impact Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). - **Hardware Type:** NVIDIA L4, 4 vCPUs, 16 GB memory, 300 GB SDD - **Hours used:** 3 - **Cloud Provider:** Google Cloud Platform (GCP) - **Compute Region:** us-east1-c - **Carbon Emitted:** 0.08 kg CO2eq (estimate via [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute)) ## Model Card Contact [Florian Roscheck](https://huggingface.co/flrs)