Squash history to reclaim LFS storage (keep only checkpoint-20000)

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,159 @@
+---
+license: other
+license_name: nvidia-open-model-license-agreement
+license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/
+library_name: gr00t
+base_model: nvidia/GR00T-N1.7-3B
+base_model_relation: finetune
+datasets:
+- cloudwalk-research/gr00t-g1-grab-bottle-right-hand-radius-20-merged
+tags:
+- gr00t
+- gr00t-n1.7
+- vla
+- vision-language-action
+- humanoid
+- robotics
+- imitation-learning
+- diffusion-policy
+- unitree-g1
+- sonic-wbc
+- arxiv:2503.14734
+- arxiv:2511.07820
+language:
+- en
+pipeline_tag: robotics
+---
+
+# GR00T N1.7-3B Fine-Tune v6 — Unitree G1 "grab the bottle" (right hand, SONIC WBC, merged radius-20 dataset)
+
+**v6 fine-tune of [NVIDIA GR00T N1.7-3B](https://huggingface.co/nvidia/GR00T-N1.7-3B) ([paper](https://arxiv.org/abs/2503.14734)) on a Unitree G1 humanoid driven by the [SONIC](https://arxiv.org/abs/2511.07820) whole-body controller.**
+
+Single-handed pick task ("grab the bottle", RIGHT hand). Trained at the **CloudWalk Robotics Lab (CW-RL)**, 2026-06, on a **502-episode / 120,017-frame** real-robot teleoperation dataset — the **largest and best-curated** of the lineage — for the `UNITREE_G1_SONIC` embodiment. Released as a reference fine-tune for teams building manipulation policies on the GR00T + SONIC + MuJoCo/G1 stack.
+
+**Predecessors:** [v2 (`…-210ep-v2-finetune`)](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-210ep-v2-finetune) — the current **validated production champion** (`checkpoint-20000`, grasps from poses where v1 failed) — [v4 (`…-417ep-v4-finetune`)](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-417ep-v4-finetune) (radius-5, aggressive curation, eval pending), [v5 (`…-314ep-v5-finetune`)](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-314ep-v5-finetune) (radius-20, eval pending), and [v1 (`…-105ep-v1-finetune`)](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-105ep-v1-finetune). v6's dataset is a **merge of two grab-bottle sets** — the 105-ep set + a new **115-ep "worst-positions" set** (hard poses) — re-curated at **radius 20** with two filtering improvements over prior versions: **(a) frame removal near the grasp moment is now prohibited** (a defect that had slipped through and likely hurt earlier versions' grasp behavior), and **(b) episodes 95–114 were kept as-is** (curation skipped). The result is **482 clean curated segments + 20 as-is = 502 episodes** (20.3% of frames / 29.5% of samples removed elsewhere), ~239 frames/ep — 2.4× the frames of v5.
+
+This is a behavior-cloning fine-tune of the full 3B model. **Status: trained 2026-06-24, eval pending** — `checkpoint-10000` and `checkpoint-20000` are published; the production checkpoint is selected by closed-loop comparison against the v2 champion (see [Evaluation](#evaluation)).
+
+## Quick facts
+
+|     |     |
+| --- | --- |
+| Base | [GR00T-N1.7-3B](https://huggingface.co/nvidia/GR00T-N1.7-3B) — Qwen3-VL vision-language backbone + flow-matching diffusion-transformer (DiT) action head |
+| Parameters | 3.14 B total / 1.62 B trainable (51.5%) |
+| Dataset | [cloudwalk-research/gr00t-g1-grab-bottle-right-hand-radius-20-merged](https://huggingface.co/datasets/cloudwalk-research/gr00t-g1-grab-bottle-right-hand-radius-20-merged) — **502 episodes, 120,017 frames** @ 50 Hz, 480×640 `ego_view` camera (no wrist cams); merge of the 105-ep + a new 115-ep "worst-positions" set, radius-20 curation with grasp-frame preservation (~239 frames/ep) |
+| Robot target | Unitree G1 (29-DoF body) + built-in G1 hands (7-DoF/hand) + SONIC whole-body controller |
+| Embodiment tag | `UNITREE_G1_SONIC` (`unitree_g1_sonic`) |
+| State space | 43-D (left_leg 6 + right_leg 6 + waist 3 + left_arm 7 + left_hand 7 + right_arm 7 + right_hand 7) |
+| Action space | `[40 × 78]` = 40-step horizon × (64 motion_token + 7 left_hand_joints + 7 right_hand_joints) |
+| Hardware | 6× NVIDIA B200 (sm_100 / Blackwell) |
+| Mixed precision | bf16 |
+| Optimizer | AdamW, lr 1e-4 cosine, warmup_ratio 0.05, weight_decay 1e-5 |
+| Steps / batch | 20,000 / global batch 48 (8 per GPU × 6 GPUs); checkpoints saved every 10,000 (→ `checkpoint-10000` + `checkpoint-20000`) |
+| Epochs | ~8.0 @ 20k (960k frame-views ÷ 120,017 frames); 10k ≈ 4.0 |
+| Augmentation | color jitter (brightness 0.3, contrast 0.4, saturation 0.5, hue 0.08) |
+| Wall-clock | **~65.3 min** (`train_runtime` 3915.8 s · 6× B200 @ **5.108 steps/s**, 245 samples/s) |
+| Final train loss | **0.0866** (mean over run; per-step @20k ≈ 0.0354) |
+| W&B run | `0p48c1z0` (offline → synced; project `g1_grab_bottle`) |
+
+## Repository contents
+
+```
+checkpoint-10000/    # 10k steps (~4.0 ep)
+checkpoint-20000/    # 20k steps (~8.0 ep)
+README.md            # this file
+```
+
+Each `checkpoint-NNNNN/` is a self-contained, deploy-ready snapshot:
+
+- `model-00001-of-00002.safetensors` + `model-00002-of-00002.safetensors` (~6.5 GB, bf16)
+- `model.safetensors.index.json`
+- `config.json`
+- `embodiment_id.json`  (contains `unitree_g1_sonic`)
+- `processor_config.json`
+- `statistics.json`  (dataset normalization stats)
+- `experiment_cfg/`  (training config snapshot)
+
+DeepSpeed ZeRO partition states, optimizer, scheduler and RNG files are intentionally omitted — they are needed only to resume training and would add several GB per checkpoint.
+
+## Evaluation
+
+**Why 10k + 20k, and why 20k is the goal.** This run saves a checkpoint every 10,000 steps, so only two are produced. On the 120,017-frame merged set, 10,000 steps is **~4.0 epochs** and 20,000 is **~8.0 epochs** — both **below** v2's validated sweet spot (~15.3 epochs), because the dataset is 2.4× larger than v5's. The live risk here is therefore **underfit**, not overfit. **20k is the v6 goal**: the two checkpoints are tested on the real robot first. The production checkpoint is chosen by **closed-loop** comparison, not by training loss. There is **no held-out split**; train loss is a fit probe, **not** a generalization measure.
+
+**Extension to 40k — no longer a true-resume.** A clean continuation from `checkpoint-20000` to 40,000 steps would have produced `checkpoint-30000` (~12 ep) and `checkpoint-40000` (~16 ep), bracketing v2's sweet spot. However, the cluster training state (DeepSpeed/optimizer/scheduler) was cleaned from NFS on 2026-06-24, so a true-resume is **no longer possible** — reaching 40k now requires re-training from scratch (`MAX_STEPS=40000`). The published deploy weights are unaffected (eval/inference unchanged).
+
+**Closed-loop evaluation — TBD.** `checkpoint-10000` and `checkpoint-20000` will be run closed-loop on the G1 + SONIC stack and compared against each other, the **v2 `checkpoint-20000` champion** (11/12 across hand-placed bottle poses), v4, and v5. If neither matches or beats the v2 champion, **v2 remains in production** (a 40k extension would now require a from-scratch re-train — see above). (This section is updated with the verdict once the eval runs.)
+
+## How to download
+
+```python
+from huggingface_hub import snapshot_download
+
+local = snapshot_download(
+    repo_id="cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-502ep-v6-finetune",
+    repo_type="model",
+    allow_patterns=["checkpoint-20000/*"],   # or "checkpoint-10000/*"
+)
+print(local)
+```
+
+## How to deploy
+
+Start the GR00T policy server (from an [Isaac-GR00T](https://github.com/NVIDIA/Isaac-GR00T) environment) pointing at the downloaded checkpoint:
+
+```bash
+python -m gr00t.eval.run_gr00t_server \
+    --model-path <local>/checkpoint-20000 \
+    --embodiment-tag UNITREE_G1_SONIC \
+    --device cuda:0 --host 0.0.0.0 --port 5550
+```
+
+Closed-loop control of the G1 (sim or real) is driven by the **SONIC whole-body controller** in [GR00T-WholeBodyControl](https://github.com/NVlabs/GR00T-WholeBodyControl): the policy emits `motion_token` + hand-joint targets that the SONIC WBC decodes into whole-body joint commands. The server must be launched with the **same `UNITREE_G1_SONIC` embodiment tag** used in training. See the NVlabs [VLA inference tutorial](https://nvlabs.github.io/GR00T-WholeBodyControl/tutorials/vla_inference.html). This checkpoint is **not** plug-and-play on hardware — it requires the SONIC C++ deploy stack and the matching G1 setup.
+
+## Known caveats
+
+1. **Right-hand-only, single task.** The dataset is one task ("grab the bottle") executed with the right hand. Left-hand and locomotion action dims reflect the (largely stationary) demonstrations; do not expect bimanual or locomotion behavior.
+2. **Single camera.** Only the `ego_view` (head) camera was recorded — no wrist cameras. The policy has never seen wrist views.
+3. **Deployment needs the SONIC stack.** The checkpoint outputs `motion_token` + hand joints for `UNITREE_G1_SONIC`; it only produces robot motion through the SONIC WBC + ZMQ deploy pipeline. It is not directly executable on a bare G1.
+4. **Train loss is not held-out.** No episode split; the loss is a smoothed train-fit probe, not a generalization metric.
+5. **Underfit risk (not overfit).** 20,000 steps = ~8.0 epochs on 120,017 frames — *below* v2's validated ~15.3-epoch sweet spot, because the dataset is 2.4× larger. The live risk is under-training, not overfitting. `checkpoint-20000` (~8.0 ep) is the goal; reaching 40k (~16 ep) is now a from-scratch re-train (the resume state was cleaned from NFS 2026-06-24) — see [Evaluation](#evaluation).
+6. **Coarse checkpoint cadence.** Only `checkpoint-10000` and `checkpoint-20000` exist (saved every 10k). There is no intermediate rung — a 40k extension (now only via from-scratch re-train) would add `checkpoint-30000` and `checkpoint-40000`.
+7. **Merged dataset provenance.** v6's data is a merge of two source sets (the 105-ep set + a new 115-ep "worst-positions" set), re-curated at radius 20 with grasp-frame preservation (frames near the grasp are no longer removed) and episodes 95–114 kept as-is (482 curated segments + 20 as-is = 502). The grasp-frame fix is a behavioral improvement over earlier versions whose effect is to be validated in closed-loop.
+8. **Training environment.** Trained on a shared B200 node with `NCCL_IB_DISABLE=1 NCCL_P2P_LEVEL=NVL` (InfiniBand off, P2P over NVLink) and W&B in offline mode (synced post-run). These affect only the training run, not the weights.
+
+## Lineage
+
+| Version | Dataset | Episodes | Frames | Epochs @20k | Closed-loop | Notes |
+| --- | --- | --- | --- | --- | --- | --- |
+| [v1](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-105ep-v1-finetune) | [105ep-v1](https://huggingface.co/datasets/cloudwalk-research/gr00t-g1-grab-bottle-right-hand-105ep-v1) | 105 | 70,680 | 13.6 | ✅ validated | First GR00T N1.7 + SONIC fine-tune at CW-RL; right-hand bottle pick. |
+| [v2](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-210ep-v2-finetune) | [right-hand-v2](https://huggingface.co/datasets/cloudwalk-research/gr00t-g1-grab-bottle-right-hand-v2) | 210 | 62,772 | 15.3 | ✅ validated (> v1) | Curated (episodes split into shorter windows, bad segments removed). **`checkpoint-20000` = current production champion (11/12).** |
+| [v4](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-417ep-v4-finetune) | [radius-5](https://huggingface.co/datasets/cloudwalk-research/gr00t-g1-grab-bottle-right-hand-zero-wandering-smooth-radius-5) | 417 | 48,577 | 19.8 | ⏳ TBD | Zero-wandering, **most aggressive** curation (radius 5 → short windows). Eval `checkpoint-20000` vs the v2 champion. |
+| [v5](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-314ep-v5-finetune) | [radius-20](https://huggingface.co/datasets/cloudwalk-research/gr00t-g1-grab-bottle-right-hand-zero-wandering-smooth-radius-20) | 314 | 50,496 | 19.0 | ⏳ TBD | Zero-wandering, **least aggressive** curation (radius 20 → longer windows). Saves only {10k, 20k}. |
+| **v6 (this)** | [radius-20-merged](https://huggingface.co/datasets/cloudwalk-research/gr00t-g1-grab-bottle-right-hand-radius-20-merged) | 502 | 120,017 | **8.0** | ⏳ TBD | **Merged** (105-ep + new 115-ep "worst-positions"), radius-20 with **grasp-frame preservation**. Largest set (2.4× v5) → 20k = only ~8 ep (below v2's sweet spot); 40k now needs a from-scratch re-train (NFS resume state cleaned 2026-06-24). |
+
+## References
+
+- **GR00T N1** — Open foundation model for generalist humanoid robots; base policy fine-tuned here. [Paper (arXiv:2503.14734)](https://arxiv.org/abs/2503.14734), [base model](https://huggingface.co/nvidia/GR00T-N1.7-3B).
+- **Isaac-GR00T** — Training / inference / deployment stack used for this fine-tune. [GitHub](https://github.com/NVIDIA/Isaac-GR00T).
+- **GR00T-WholeBodyControl (SONIC)** — Whole-body controller + teleop + VLA deployment for the G1. [GitHub](https://github.com/NVlabs/GR00T-WholeBodyControl), [SONIC paper (arXiv:2511.07820)](https://arxiv.org/abs/2511.07820).
+- **Training dataset** — [`cloudwalk-research/gr00t-g1-grab-bottle-right-hand-radius-20-merged`](https://huggingface.co/datasets/cloudwalk-research/gr00t-g1-grab-bottle-right-hand-radius-20-merged) (CloudWalk Research, 2026), PICO 4 Ultra teleoperation on the Unitree G1 with SONIC WBC; a merge of the 105-ep set + a new 115-ep "worst-positions" set, re-curated at radius 20 with grasp-frame preservation (502 episodes, 120,017 frames).
+
+## Attribution
+
+Developed by **cloudwalk-research** in the **CloudWalk Robotics Lab (CW-RL)**. Fine-tuned from [`nvidia/GR00T-N1.7-3B`](https://huggingface.co/nvidia/GR00T-N1.7-3B) using [Isaac-GR00T](https://github.com/NVIDIA/Isaac-GR00T); targets the Unitree G1 with the [SONIC](https://github.com/NVlabs/GR00T-WholeBodyControl) whole-body controller.
+
+## Citation
+
+```bibtex
+@misc{cwrl_gr00t_grab_bottle_v6_2026,
+  title        = {GR00T N1.7 Fine-Tune v6 --- Unitree G1 "grab the bottle" (right hand, SONIC WBC, merged radius-20 curated 502-ep dataset)},
+  author       = {{CloudWalk Robotics Lab}},
+  year         = {2026},
+  howpublished = {Hugging Face model repository},
+  url          = {https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-502ep-v6-finetune}
+}
+```
+
+## License
+
+Inherits the **NVIDIA Open Model License Agreement** of the base model [`nvidia/GR00T-N1.7-3B`](https://huggingface.co/nvidia/GR00T-N1.7-3B) — see the [license terms](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). This is a research preview: not intended for safety-critical use; closed-loop deployment on a physical humanoid requires human oversight.

checkpoint-20000/config.json

@@ -0,0 +1,88 @@
+{
+  "action_horizon": 40,
+  "add_pos_embed": true,
+  "apply_sincos_state_encoding": false,
+  "architectures": [
+    "Gr00tN1d7"
+  ],
+  "attn_dropout": 0.2,
+  "attn_implementation": null,
+  "backbone_embedding_dim": 2048,
+  "backbone_trainable_params_fp32": true,
+  "color_jitter_params": {
+    "brightness": 0.3,
+    "contrast": 0.4,
+    "hue": 0.08,
+    "saturation": 0.5
+  },
+  "crop_fraction": 0.95,
+  "diffusion_model_cfg": {
+    "attention_head_dim": 48,
+    "dropout": 0.2,
+    "final_dropout": true,
+    "interleave_self_attention": true,
+    "norm_type": "ada_norm",
+    "num_attention_heads": 32,
+    "num_layers": 32,
+    "output_dim": 1024,
+    "positional_embeddings": null
+  },
+  "dtype": "bfloat16",
+  "exclude_state": false,
+  "formalize_language": true,
+  "hidden_size": 1024,
+  "image_crop_size": [
+    230,
+    230
+  ],
+  "image_target_size": [
+    256,
+    256
+  ],
+  "letter_box_transform": false,
+  "load_bf16": false,
+  "max_action_dim": 132,
+  "max_num_embodiments": 32,
+  "max_seq_len": 1024,
+  "max_state_dim": 132,
+  "model_dtype": "bfloat16",
+  "model_name": "nvidia/Cosmos-Reason2-2B",
+  "model_type": "Gr00tN1d7",
+  "noise_beta_alpha": 1.5,
+  "noise_beta_beta": 1.0,
+  "noise_s": 0.999,
+  "num_inference_timesteps": 4,
+  "num_timestep_buckets": 1000,
+  "random_history_crop": true,
+  "random_rotation_angle": 0,
+  "reproject_vision": false,
+  "rtc_ramp_rate": 6.0,
+  "select_layer": 16,
+  "shortest_image_edge": 256,
+  "state_dropout_prob": 0.2,
+  "state_gaussian_noise_std": 0.0,
+  "transformers_version": "4.57.3",
+  "tune_diffusion_model": true,
+  "tune_linear": true,
+  "tune_llm": false,
+  "tune_projector": true,
+  "tune_top_llm_layers": 0,
+  "tune_visual": false,
+  "tune_vlln": true,
+  "use_albumentations": true,
+  "use_alternate_vl_dit": true,
+  "use_flash_attention": true,
+  "use_future_tokens": false,
+  "use_mean_std": false,
+  "use_percentiles": true,
+  "use_vl_self_attention": true,
+  "use_vlln": true,
+  "vl_self_attention_cfg": {
+    "attention_head_dim": 64,
+    "dropout": 0.2,
+    "final_dropout": true,
+    "num_attention_heads": 32,
+    "num_layers": 4,
+    "positional_embeddings": null
+  }
+}

checkpoint-20000/embodiment_id.json

@@ -0,0 +1,57 @@
+{
+  "robocasa_panda_omron": 13,
+  "oxe_droid": 17,
+  "oxe_fractal": 18,
+  "oxe_language_table": 19,
+  "oxe_bridge": 20,
+  "unknown": 22,
+  "gr1_unified": 20,
+  "agibot": 26,
+  "sim_behavior_r1_pro": 23,
+  "xdof": 24,
+  "xdof_oss_data": 25,
+  "unitree_g1_full_body_with_waist_height_nav_cmd": 25,
+  "real_r1_pro_sharpa": 27,
+  "real_r1_pro_sharpa_add_view": 27,
+  "real_r1_pro_sharpa_relative_arm_joint": 26,
+  "real_r1_pro_sharpa_delta_eef": 26,
+  "real_r1_pro_sharpa_absolute_eef": 26,
+  "real_r1_pro_sharpa_meanstd": 26,
+  "real_r1_pro_sharpa_relative_eef": 26,
+  "real_r1_pro_sharpa_relative_eef_add_view": 26,
+  "real_r1_pro_sharpa_relative_eef_relative_hand": 26,
+  "real_r1_pro_sharpa_relative_eef_human": 26,
+  "real_r1_pro_sharpa_relative_eef_human_add_view": 26,
+  "real_r1_pro_sharpa_relative_eef_human_relative_hand": 26,
+  "real_r1_pro_sharpa_relative_eef_egodex": 26,
+  "real_r1_pro_sharpa_relative_eef_egodex_relative_hand": 26,
+  "real_r1_pro_sharpa_relative_eef_egodex_wrist_only": 26,
+  "real_r1_pro_sharpa_relative_eef_maxinsights": 26,
+  "real_r1_pro_sharpa_relative_eef_maxinsights_relative_hand": 26,
+  "real_r1_pro_sharpa_relative_eef_mecka": 26,
+  "real_r1_pro_sharpa_relative_eef_mecka_relative_hand": 26,
+  "real_g1_relative_eef_absolute_joints": 25,
+  "real_g1_relative_eef_absolute_joints_wrist_cam": 25,
+  "real_g1_relative_eef_relative_joints": 25,
+  "real_r1_pro_sharpa_relative_eef_relative_hand_relative_joint": 26,
+  "real_r1_pro_sharpa_relative_joint": 29,
+  "oxe_droid_relative_eef_relative_joint": 24,
+  "oxe_droid_relative_eef_relative_joint_swapped": 24,
+  "oxe_droid_relative_eef_relative_joint_upweight_z": 24,
+  "oxe_droid_relative_eef_relative_joint_upweight_z_swapped": 24,
+  "oxe_droid_relative_eef_relative_joint_3view": 24,
+  "oxe_droid_relative_eef_relative_joint_3view_swapped": 24,
+  "oxe_droid_relative_eef": 24,
+  "oxe_droid_joint_position_relative": 24,
+  "xdof_relative_eef_relative_joint": 27,
+  "xdof_relative_eef_relative_joint_subtask": 27,
+  "xdof_relative_eef": 27,
+  "xdof_relative_joint": 28,
+  "simpler_env_google": 0,
+  "simpler_env_widowx": 1,
+  "libero_sim": 2,
+  "droid_sim": 3,
+  "unitree_g1_sonic": 11,
+  "new_embodiment": 10,
+  "robocasa_gr1_tabletop": 10
+}

checkpoint-20000/experiment_cfg/conf.yaml

@@ -0,0 +1,252 @@
+load_config_path: null
+model:
+  model_type: Gr00tN1d7
+  model_dtype: bfloat16
+  model_name: nvidia/Cosmos-Reason2-2B
+  backbone_model_type: qwen
+  model_revision: null
+  tune_top_llm_layers: 0
+  backbone_embedding_dim: 2048
+  tune_llm: false
+  tune_visual: false
+  select_layer: 12
+  reproject_vision: false
+  use_flash_attention: true
+  load_bf16: false
+  backbone_trainable_params_fp32: true
+  image_crop_size:
+  - 230
+  - 230
+  image_target_size:
+  - 256
+  - 256
+  shortest_image_edge: null
+  crop_fraction: null
+  random_rotation_angle: null
+  color_jitter_params:
+    brightness: 0.3
+    contrast: 0.4
+    saturation: 0.5
+    hue: 0.08
+  use_albumentations_transforms: true
+  extra_augmentation_config: null
+  formalize_language: true
+  apply_sincos_state_encoding: false
+  use_percentiles: true
+  use_relative_action: true
+  max_state_dim: 132
+  max_action_dim: 132
+  action_horizon: 40
+  hidden_size: 1024
+  input_embedding_dim: 1536
+  state_history_length: 1
+  add_pos_embed: true
+  attn_dropout: 0.2
+  use_vlln: true
+  max_seq_len: 1024
+  use_alternate_vl_dit: true
+  attend_text_every_n_blocks: 2
+  diffusion_model_cfg:
+    positional_embeddings: null
+    num_layers: 16
+    num_attention_heads: 32
+    attention_head_dim: 48
+    norm_type: ada_norm
+    dropout: 0.2
+    final_dropout: true
+    output_dim: 1024
+    interleave_self_attention: true
+  num_inference_timesteps: 4
+  noise_beta_alpha: 1.5
+  noise_beta_beta: 1.0
+  noise_s: 0.999
+  num_timestep_buckets: 1000
+  tune_projector: true
+  tune_diffusion_model: true
+  tune_vlln: true
+  state_dropout_prob: 0.2
+  exclude_state: false
+  use_mean_std: false
+  max_num_embodiments: 32
+data:
+  datasets:
+  - dataset_paths:
+    - <train_root>/data/g1_grab_bottle_rh_v6
+    embodiment_tag: unitree_g1_sonic
+    mix_ratio: 1.0
+    dataset_type: physical_embodiment
+    val_dataset_path: null
+  modality_configs:
+    unitree_g1_sonic:
+      video:
+        delta_indices:
+        - 0
+        modality_keys:
+        - ego_view
+        sin_cos_embedding_keys: null
+        mean_std_embedding_keys: null
+        action_configs: null
+      state:
+        delta_indices:
+        - 0
+        modality_keys:
+        - left_leg
+        - right_leg
+        - waist
+        - left_arm
+        - right_arm
+        - left_hand
+        - right_hand
+        - projected_gravity
+        sin_cos_embedding_keys: null
+        mean_std_embedding_keys: null
+        action_configs: null
+      action:
+        delta_indices:
+        - 0
+        - 1
+        - 2
+        - 3
+        - 4
+        - 5
+        - 6
+        - 7
+        - 8
+        - 9
+        - 10
+        - 11
+        - 12
+        - 13
+        - 14
+        - 15
+        - 16
+        - 17
+        - 18
+        - 19
+        - 20
+        - 21
+        - 22
+        - 23
+        - 24
+        - 25
+        - 26
+        - 27
+        - 28
+        - 29
+        - 30
+        - 31
+        - 32
+        - 33
+        - 34
+        - 35
+        - 36
+        - 37
+        - 38
+        - 39
+        modality_keys:
+        - motion_token
+        - left_hand_joints
+        - right_hand_joints
+        sin_cos_embedding_keys: null
+        mean_std_embedding_keys: null
+        action_configs:
+        - rep: ABSOLUTE
+          type: NON_EEF
+          format: DEFAULT
+          state_key: null
+        - rep: ABSOLUTE
+          type: NON_EEF
+          format: DEFAULT
+          state_key: null
+        - rep: ABSOLUTE
+          type: NON_EEF
+          format: DEFAULT
+          state_key: null
+      language:
+        delta_indices:
+        - 0
+        modality_keys:
+        - annotation.human.task_description
+        sin_cos_embedding_keys: null
+        mean_std_embedding_keys: null
+        action_configs: null
+  download_cache: false
+  shard_size: 1024
+  episode_sampling_rate: 0.1
+  num_shards_per_epoch: 100000
+  override_pretraining_statistics: true
+  mode: single_turn
+  random_chop: 0.0
+  mock_dataset_mode: false
+  shuffle: true
+  seed: 42
+  multiprocessing_context: fork
+  allow_padding: false
+  subsample_ratio: 1.0
+  image_crop_size:
+  - 244
+  - 244
+  image_target_size:
+  - 224
+  - 224
+  video_backend: torchcodec
+training:
+  output_dir: <train_root>/checkpoints/g1_grab_bottle_rh_v6
+  experiment_name: g1_grab_bottle_rh_v6
+  max_steps: 20000
+  global_batch_size: 48
+  per_gpu_batch_size: null
+  gradient_accumulation_steps: 1
+  learning_rate: 0.0001
+  lr_scheduler_type: cosine
+  weight_decay: 1.0e-05
+  warmup_ratio: 0.05
+  warmup_steps: 0
+  max_grad_norm: 1.0
+  optim: adamw_torch
+  start_from_checkpoint: nvidia/GR00T-N1.7-3B
+  skip_weight_loading: false
+  tf32: true
+  fp16: false
+  bf16: true
+  eval_bf16: true
+  logging_steps: 10
+  save_steps: 10000
+  save_total_limit: 5
+  save_vl_model: false
+  save_only_model: false
+  resume_from_checkpoint: false
+  upload_checkpoints: false
+  upload_every: 1000
+  upload_last_n_checkpoints: 5
+  max_concurrent_uploads: 2
+  eval_strategy: 'no'
+  eval_steps: 500
+  eval_set_split_ratio: 0.1
+  eval_batch_size: 2
+  save_best_eval_metric_name: ''
+  save_best_eval_metric_greater_is_better: true
+  deepspeed_stage: 2
+  gradient_checkpointing: false
+  transformers_trust_remote_code: true
+  transformers_local_files_only: false
+  transformers_cache_dir: null
+  transformers_access_token: null
+  use_ddp: false
+  ddp_bucket_cap_mb: 100
+  num_gpus: 6
+  dataloader_num_workers: 4
+  remove_unused_columns: false
+  use_wandb: true
+  wandb_project: g1_grab_bottle
+  enable_profiling: false
+  max_retries: 3
+  assert_loss_less_than: null
+  add_rl_callback: false
+  enable_open_loop_eval: false
+  open_loop_eval_traj_ids:
+  - 0
+  open_loop_eval_steps_per_traj: 100
+  open_loop_eval_plot_indices: null
+max_steps: 20000
+save_steps: 10000

checkpoint-20000/experiment_cfg/config.yaml

@@ -0,0 +1,250 @@
+load_config_path: null
+model:
+  model_type: Gr00tN1d7
+  model_dtype: bfloat16
+  model_name: nvidia/Cosmos-Reason2-2B
+  backbone_model_type: qwen
+  model_revision: null
+  tune_top_llm_layers: 0
+  backbone_embedding_dim: 2048
+  tune_llm: false
+  tune_visual: false
+  select_layer: 12
+  reproject_vision: false
+  use_flash_attention: true
+  load_bf16: false
+  backbone_trainable_params_fp32: true
+  image_crop_size:
+  - 230
+  - 230
+  image_target_size:
+  - 256
+  - 256
+  shortest_image_edge: null
+  crop_fraction: null
+  random_rotation_angle: null
+  color_jitter_params:
+    brightness: 0.3
+    contrast: 0.4
+    saturation: 0.5
+    hue: 0.08
+  use_albumentations_transforms: true
+  extra_augmentation_config: null
+  formalize_language: true
+  apply_sincos_state_encoding: false
+  use_percentiles: true
+  use_relative_action: true
+  max_state_dim: 132
+  max_action_dim: 132
+  action_horizon: 40
+  hidden_size: 1024
+  input_embedding_dim: 1536
+  state_history_length: 1
+  add_pos_embed: true
+  attn_dropout: 0.2
+  use_vlln: true
+  max_seq_len: 1024
+  use_alternate_vl_dit: true
+  attend_text_every_n_blocks: 2
+  diffusion_model_cfg:
+    positional_embeddings: null
+    num_layers: 16
+    num_attention_heads: 32
+    attention_head_dim: 48
+    norm_type: ada_norm
+    dropout: 0.2
+    final_dropout: true
+    output_dim: 1024
+    interleave_self_attention: true
+  num_inference_timesteps: 4
+  noise_beta_alpha: 1.5
+  noise_beta_beta: 1.0
+  noise_s: 0.999
+  num_timestep_buckets: 1000
+  tune_projector: true
+  tune_diffusion_model: true
+  tune_vlln: true
+  state_dropout_prob: 0.2
+  exclude_state: false
+  use_mean_std: false
+  max_num_embodiments: 32
+data:
+  datasets:
+  - dataset_paths:
+    - <train_root>/data/g1_grab_bottle_rh_v6
+    embodiment_tag: unitree_g1_sonic
+    mix_ratio: 1.0
+    dataset_type: physical_embodiment
+    val_dataset_path: null
+  modality_configs:
+    unitree_g1_sonic:
+      video:
+        delta_indices:
+        - 0
+        modality_keys:
+        - ego_view
+        sin_cos_embedding_keys: null
+        mean_std_embedding_keys: null
+        action_configs: null
+      state:
+        delta_indices:
+        - 0
+        modality_keys:
+        - left_leg
+        - right_leg
+        - waist
+        - left_arm
+        - right_arm
+        - left_hand
+        - right_hand
+        - projected_gravity
+        sin_cos_embedding_keys: null
+        mean_std_embedding_keys: null
+        action_configs: null
+      action:
+        delta_indices:
+        - 0
+        - 1
+        - 2
+        - 3
+        - 4
+        - 5
+        - 6
+        - 7
+        - 8
+        - 9
+        - 10
+        - 11
+        - 12
+        - 13
+        - 14
+        - 15
+        - 16
+        - 17
+        - 18
+        - 19
+        - 20
+        - 21
+        - 22
+        - 23
+        - 24
+        - 25
+        - 26
+        - 27
+        - 28
+        - 29
+        - 30
+        - 31
+        - 32
+        - 33
+        - 34
+        - 35
+        - 36
+        - 37
+        - 38
+        - 39
+        modality_keys:
+        - motion_token
+        - left_hand_joints
+        - right_hand_joints
+        sin_cos_embedding_keys: null
+        mean_std_embedding_keys: null
+        action_configs:
+        - rep: absolute
+          type: non_eef
+          format: default
+          state_key: null
+        - rep: absolute
+          type: non_eef
+          format: default
+          state_key: null
+        - rep: absolute
+          type: non_eef
+          format: default
+          state_key: null
+      language:
+        delta_indices:
+        - 0
+        modality_keys:
+        - annotation.human.task_description
+        sin_cos_embedding_keys: null
+        mean_std_embedding_keys: null
+        action_configs: null
+  download_cache: false
+  shard_size: 1024
+  episode_sampling_rate: 0.1
+  num_shards_per_epoch: 100000
+  override_pretraining_statistics: true
+  mode: single_turn
+  random_chop: 0.0
+  mock_dataset_mode: false
+  shuffle: true
+  seed: 42
+  multiprocessing_context: fork
+  allow_padding: false
+  subsample_ratio: 1.0
+  image_crop_size:
+  - 244
+  - 244
+  image_target_size:
+  - 224
+  - 224
+  video_backend: torchcodec
+training:
+  output_dir: <train_root>/checkpoints/g1_grab_bottle_rh_v6
+  experiment_name: g1_grab_bottle_rh_v6
+  max_steps: 20000
+  global_batch_size: 48
+  per_gpu_batch_size: null
+  gradient_accumulation_steps: 1
+  learning_rate: 0.0001
+  lr_scheduler_type: cosine
+  weight_decay: 1.0e-05
+  warmup_ratio: 0.05
+  warmup_steps: 0
+  max_grad_norm: 1.0
+  optim: adamw_torch
+  start_from_checkpoint: nvidia/GR00T-N1.7-3B
+  skip_weight_loading: false
+  tf32: true
+  fp16: false
+  bf16: true
+  eval_bf16: true
+  logging_steps: 10
+  save_steps: 10000
+  save_total_limit: 5
+  save_vl_model: false
+  save_only_model: false
+  resume_from_checkpoint: false
+  upload_checkpoints: false
+  upload_every: 1000
+  upload_last_n_checkpoints: 5
+  max_concurrent_uploads: 2
+  eval_strategy: 'no'
+  eval_steps: 500
+  eval_set_split_ratio: 0.1
+  eval_batch_size: 2
+  save_best_eval_metric_name: ''
+  save_best_eval_metric_greater_is_better: true
+  deepspeed_stage: 2
+  gradient_checkpointing: false
+  transformers_trust_remote_code: true
+  transformers_local_files_only: false
+  transformers_cache_dir: null
+  transformers_access_token: null
+  use_ddp: false
+  ddp_bucket_cap_mb: 100
+  num_gpus: 6
+  dataloader_num_workers: 4
+  remove_unused_columns: false
+  use_wandb: true
+  wandb_project: g1_grab_bottle
+  enable_profiling: false
+  max_retries: 3
+  assert_loss_less_than: null
+  add_rl_callback: false
+  enable_open_loop_eval: false
+  open_loop_eval_traj_ids:
+  - 0
+  open_loop_eval_steps_per_traj: 100
+  open_loop_eval_plot_indices: null

checkpoint-20000/experiment_cfg/dataset_statistics.json

@@ -0,0 +1,907 @@
+{
+  "unitree_g1_sonic": {
+    "state": {
+      "left_leg": {
+        "min": [
+          -0.14669117331504822,
+          -0.07608544826507568,
+          -0.06501765549182892,
+          0.13098491728305817,
+          -0.27601441740989685,
+          -0.09286714345216751
+        ],
+        "max": [
+          0.20649085938930511,
+          0.14269429445266724,
+          0.47703272104263306,
+          0.3493556082248688,
+          -0.08085808157920837,
+          0.050362009555101395
+        ],
+        "mean": [
+          0.052889496088027954,
+          0.062458690255880356,
+          0.2227516919374466,
+          0.20512205362319946,
+          -0.20019739866256714,
+          -0.03059203363955021
+        ],
+        "std": [
+          0.05077589675784111,
+          0.03317390754818916,
+          0.08180207014083862,
+          0.03545672073960304,
+          0.02584464661777013,
+          0.027249639853835106
+        ],
+        "q01": [
+          -0.057498712092638016,
+          -0.01865491084754467,
+          0.01471769344061613,
+          0.14020584523677826,
+          -0.2570301938056946,
+          -0.08588964492082596
+        ],
+        "q99": [
+          0.1694452166557312,
+          0.13594341993331907,
+          0.41892528533935547,
+          0.3165185225009918,
+          -0.14303868591785435,
+          0.033429062217473986
+        ]
+      },
+      "right_leg": {
+        "min": [
+          -0.222640261054039,
+          -0.1842055320739746,
+          -0.5060127973556519,
+          0.13547606766223907,
+          -0.3310108184814453,
+          -0.07219075411558151
+        ],
+        "max": [
+          0.21914619207382202,
+          0.060924604535102844,
+          0.05546923354268074,
+          0.4880615472793579,
+          -0.05241888388991356,
+          0.08624618500471115
+        ],
+        "mean": [
+          0.0333583690226078,
+          -0.05868099257349968,
+          -0.2565644681453705,
+          0.22718310356140137,
+          -0.19859550893306732,
+          0.029281744733452797
+        ],
+        "std": [
+          0.0720328837633133,
+          0.0333959124982357,
+          0.07502146810293198,
+          0.056596916168928146,
+          0.034849993884563446,
+          0.029130030423402786
+        ],
+        "q01": [
+          -0.1477806341648102,
+          -0.12219724774360656,
+          -0.42282232642173767,
+          0.15221486270427703,
+          -0.2814170789718628,
+          -0.049902956485748294
+        ],
+        "q99": [
+          0.16515859961509705,
+          0.031854305118322344,
+          -0.06510014593601232,
+          0.41485249161720267,
+          -0.10264042019844055,
+          0.07890639513730999
+        ]
+      },
+      "waist": {
+        "min": [
+          -0.26439619064331055,
+          -0.213858962059021,
+          -0.01679294742643833
+        ],
+        "max": [
+          0.31170976161956787,
+          0.16917863488197327,
+          0.024004876613616943
+        ],
+        "mean": [
+          0.01037809532135725,
+          0.005020632408559322,
+          0.0056914836168289185
+        ],
+        "std": [
+          0.08205181360244751,
+          0.04611584171652794,
+          0.004986464511603117
+        ],
+        "q01": [
+          -0.19955264031887054,
+          -0.1177203580737114,
+          -0.006733497343957424
+        ],
+        "q99": [
+          0.21181747555732722,
+          0.10909359306097027,
+          0.016945944800972938
+        ]
+      },
+      "left_arm": {
+        "min": [
+          -0.07312773913145065,
+          0.12179568409919739,
+          -0.6626437902450562,
+          0.5705689787864685,
+          -0.567321240901947,
+          -0.35752493143081665,
+          -0.6198586821556091
+        ],
+        "max": [
+          0.3904460668563843,
+          0.7534242868423462,
+          0.598564088344574,
+          1.3175097703933716,
+          0.4254399836063385,
+          0.14636477828025818,
+          0.06271680444478989
+        ],
+        "mean": [
+          0.14356407523155212,
+          0.27594050765037537,
+          0.019395535811781883,
+          1.0941842794418335,
+          -0.17310982942581177,
+          -0.04561951756477356,
+          -0.3147760331630707
+        ],
+        "std": [
+          0.07393965870141983,
+          0.07024486362934113,
+          0.20490798354148865,
+          0.08803243190050093,
+          0.12107257544994354,
+          0.05566338822245598,
+          0.11754066497087479
+        ],
+        "q01": [
+          -0.028638943061232557,
+          0.16339293122291565,
+          -0.578610360622406,
+          0.7969409322738648,
+          -0.4552040112018585,
+          -0.20082110166549683,
+          -0.5233304429054261
+        ],
+        "q99": [
+          0.3071978700160979,
+          0.492823462486267,
+          0.40476529836654657,
+          1.2583896923065185,
+          0.14082135140895805,
+          0.08796490252017972,
+          0.00109679636079818
+        ]
+      },
+      "right_arm": {
+        "min": [
+          -0.6237069964408875,
+          -0.9853190183639526,
+          -0.6880982518196106,
+          -0.5405125021934509,
+          -0.7846910953521729,
+          -0.4318923056125641,
+          -0.5493108630180359
+        ],
+        "max": [
+          0.5155733823776245,
+          -0.1550159603357315,
+          0.7213544845581055,
+          1.280202865600586,
+          0.877616822719574,
+          0.272458016872406,
+          0.940591037273407
+        ],
+        "mean": [
+          0.028888558968901634,
+          -0.3776170015335083,
+          -0.02223096787929535,
+          0.33385175466537476,
+          0.1310616135597229,
+          -0.04455765709280968,
+          0.19144287705421448
+        ],
+        "std": [
+          0.175581693649292,
+          0.13925781846046448,
+          0.26699480414390564,
+          0.48841339349746704,
+          0.1794845163822174,
+          0.07429497689008713,
+          0.2084822952747345
+        ],
+        "q01": [
+          -0.4000554883480072,
+          -0.7950397443771362,
+          -0.5589917182922363,
+          -0.3748914897441864,
+          -0.33935826539993286,
+          -0.22534055650234222,
+          -0.20451800525188446
+        ],
+        "q99": [
+          0.40518665313720703,
+          -0.20068784058094025,
+          0.6350081562995911,
+          1.2077462673187256,
+          0.6247764873504634,
+          0.13185609221458425,
+          0.6548680806159972
+        ]
+      },
+      "left_hand": {
+        "min": [
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0
+        ],
+        "max": [
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0
+        ],
+        "mean": [
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0
+        ],
+        "std": [
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0
+        ],
+        "q01": [
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0
+        ],
+        "q99": [
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0
+        ]
+      },
+      "right_hand": {
+        "min": [
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0
+        ],
+        "max": [
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0
+        ],
+        "mean": [
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0
+        ],
+        "std": [
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0
+        ],
+        "q01": [
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0
+        ],
+        "q99": [
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0
+        ]
+      },
+      "projected_gravity": {
+        "min": [
+          -0.07730928808450699,
+          -0.04716804251074791,
+          -1.0
+        ],
+        "max": [
+          0.09890831261873245,
+          0.05293969437479973,
+          -0.995090901851654
+        ],
+        "mean": [
+          -0.0043840291909873486,
+          0.010004215873777866,
+          -0.9998947381973267
+        ],
+        "std": [
+          0.027265189215540886,
+          0.01425170712172985,
+          0.0006761676631519937
+        ],
+        "q01": [
+          -0.0572731627523899,
+          -0.022329296618700027,
+          -0.9999953413009643
+        ],
+        "q99": [
+          0.05686203852295875,
+          0.0467790538072586,
+          -0.9977874636650086
+        ]
+      }
+    },
+    "action": {
+      "motion_token": {
+        "min": [
+          -0.375,
+          -0.3125,
+          -0.4375,
+          -0.25,
+          -0.375,
+          -0.25,
+          -0.1875,
+          -0.3125,
+          -0.4375,
+          -0.375,
+          -0.1875,
+          -0.3125,
+          -0.375,
+          -0.4375,
+          -0.5625,
+          -0.1875,
+          -0.375,
+          -0.375,
+          -0.3125,
+          -0.5,
+          -0.3125,
+          -0.4375,
+          -0.5,
+          -0.5,
+          -0.25,
+          -0.3125,
+          -0.125,
+          -0.4375,
+          -0.1875,
+          -0.1875,
+          0.0,
+          -0.3125,
+          0.0625,
+          -0.3125,
+          -0.125,
+          -0.375,
+          -0.375,
+          -0.5,
+          -0.0625,
+          -0.0625,
+          -0.3125,
+          -0.4375,
+          -0.3125,
+          -0.4375,
+          -0.5,
+          -0.1875,
+          0.0,
+          -0.25,
+          -0.125,
+          -0.1875,
+          -0.1875,
+          -0.3125,
+          -0.1875,
+          -0.25,
+          -0.125,
+          -0.4375,
+          -0.3125,
+          0.0625,
+          -0.25,
+          -0.5,
+          -0.3125,
+          -0.125,
+          -0.0625,
+          -0.4375
+        ],
+        "max": [
+          0.0625,
+          0.1875,
+          0.125,
+          0.3125,
+          0.3125,
+          0.125,
+          0.375,
+          0.3125,
+          0.25,
+          0.0625,
+          0.3125,
+          0.0625,
+          -0.0625,
+          0.0,
+          -0.1875,
+          0.125,
+          0.125,
+          0.0,
+          0.3125,
+          -0.125,
+          0.3125,
+          0.125,
+          0.1875,
+          -0.0625,
+          0.375,
+          0.0625,
+          0.3125,
+          0.125,
+          0.3125,
+          0.375,
+          0.3125,
+          0.0,
+          0.375,
+          0.3125,
+          0.3125,
+          0.5,
+          0.25,
+          -0.0625,
+          0.25,
+          0.375,
+          0.0625,
+          0.25,
+          0.25,
+          0.125,
+          -0.125,
+          0.0625,
+          0.375,
+          0.1875,
+          0.3125,
+          0.25,
+          0.4375,
+          0.25,
+          0.3125,
+          0.25,
+          0.4375,
+          0.0,
+          0.125,
+          0.5,
+          0.1875,
+          0.125,
+          0.25,
+          0.3125,
+          0.4375,
+          0.0625
+        ],
+        "mean": [
+          -0.13711702823638916,
+          -0.08371730893850327,
+          -0.11014533042907715,
+          0.024440288543701172,
+          -0.04960651323199272,
+          -0.10139813274145126,
+          0.1811576634645462,
+          0.04631322994828224,
+          -0.07858626544475555,
+          -0.15983985364437103,
+          0.07972099632024765,
+          -0.1953827440738678,
+          -0.22762921452522278,
+          -0.18921798467636108,
+          -0.3849152624607086,
+          -0.06569590419530869,
+          -0.08318248391151428,
+          -0.1585327535867691,
+          -0.05366375297307968,
+          -0.28489765524864197,
+          0.029177116230130196,
+          -0.1634419560432434,
+          -0.1573813557624817,
+          -0.2729748785495758,
+          0.0001171709009213373,
+          -0.12085162848234177,
+          0.05774806812405586,
+          -0.18254289031028748,
+          0.09354403614997864,
+          0.092095285654068,
+          0.13156834244728088,
+          -0.14989908039569855,
+          0.2321317046880722,
+          0.0823388546705246,
+          0.08006990700960159,
+          0.10722178965806961,
+          -0.03585377335548401,
+          -0.28966259956359863,
+          0.11201850324869156,
+          0.08083802461624146,
+          -0.11800151318311691,
+          -0.06649630516767502,
+          -0.0029980128165334463,
+          -0.09436475485563278,
+          -0.3098987936973572,
+          -0.07983139902353287,
+          0.17370352149009705,
+          -0.053218502551317215,
+          0.07664903998374939,
+          0.08144887536764145,
+          0.10868877172470093,
+          -0.0367041751742363,
+          0.10677654296159744,
+          0.011005732230842113,
+          0.1995811015367508,
+          -0.1268320381641388,
+          -0.05898747593164444,
+          0.31961721181869507,
+          -0.024935530498623848,
+          -0.1604132056236267,
+          -0.03035976178944111,
+          0.0727558359503746,
+          0.24019253253936768,
+          -0.23546351492404938
+        ],
+        "std": [
+          0.08427181094884872,
+          0.09132105112075806,
+          0.10239937156438828,
+          0.09662983566522598,
+          0.11844096332788467,
+          0.055486176162958145,
+          0.09144636988639832,
+          0.09764780849218369,
+          0.13288992643356323,
+          0.0999152809381485,
+          0.08530759811401367,
+          0.06189817190170288,
+          0.05665476247668266,
+          0.07613624632358551,
+          0.08372801542282104,
+          0.06037416681647301,
+          0.08315664529800415,
+          0.05492406710982323,
+          0.10013283044099808,
+          0.0646984875202179,
+          0.11560861021280289,
+          0.10104548186063766,
+          0.12369653582572937,
+          0.07488158345222473,
+          0.1057722344994545,
+          0.06476478278636932,
+          0.056242093443870544,
+          0.1215885579586029,
+          0.07282985001802444,
+          0.10904982686042786,
+          0.059046003967523575,
+          0.053109243512153625,
+          0.06036791950464249,
+          0.11286568641662598,
+          0.07332020998001099,
+          0.1950840801000595,
+          0.10334518551826477,
+          0.08399288356304169,
+          0.054664116352796555,
+          0.08029194921255112,
+          0.06380587071180344,
+          0.15343362092971802,
+          0.09377138316631317,
+          0.11054050922393799,
+          0.05195118859410286,
+          0.04799128323793411,
+          0.07289700955152512,
+          0.05934789776802063,
+          0.06626643985509872,
+          0.07222520560026169,
+          0.12431919574737549,
+          0.09400444477796555,
+          0.08693958073854446,
+          0.08674459159374237,
+          0.06900472939014435,
+          0.08344833552837372,
+          0.0683182030916214,
+          0.07598166167736053,
+          0.06619145721197128,
+          0.1398589164018631,
+          0.09648177027702332,
+          0.08966951072216034,
+          0.08390254527330399,
+          0.0911010354757309
+        ],
+        "q01": [
+          -0.3125,
+          -0.25,
+          -0.375,
+          -0.1875,
+          -0.3125,
+          -0.1875,
+          -0.0625,
+          -0.1875,
+          -0.3125,
+          -0.3125,
+          -0.125,
+          -0.3125,
+          -0.3125,
+          -0.3125,
+          -0.5625,
+          -0.1875,
+          -0.25,
+          -0.3125,
+          -0.25,
+          -0.4375,
+          -0.1875,
+          -0.375,
+          -0.375,
+          -0.4375,
+          -0.1875,
+          -0.25,
+          -0.0625,
+          -0.4375,
+          -0.0625,
+          -0.125,
+          0.0,
+          -0.25,
+          0.125,
+          -0.1875,
+          -0.125,
+          -0.3125,
+          -0.25,
+          -0.4375,
+          0.0,
+          -0.0625,
+          -0.25,
+          -0.375,
+          -0.25,
+          -0.375,
+          -0.4375,
+          -0.125,
+          0.0,
+          -0.1875,
+          -0.0625,
+          -0.0625,
+          -0.125,
+          -0.25,
+          -0.125,
+          -0.1875,
+          0.0625,
+          -0.3125,
+          -0.25,
+          0.125,
+          -0.1875,
+          -0.4375,
+          -0.25,
+          -0.125,
+          0.0625,
+          -0.375
+        ],
+        "q99": [
+          0.0,
+          0.1875,
+          0.0625,
+          0.25,
+          0.1875,
+          0.0,
+          0.375,
+          0.1875,
+          0.1875,
+          0.0625,
+          0.25,
+          -0.0625,
+          -0.125,
+          0.0,
+          -0.1875,
+          0.0625,
+          0.125,
+          -0.0625,
+          0.1875,
+          -0.125,
+          0.25,
+          0.0625,
+          0.0625,
+          -0.125,
+          0.25,
+          0.0,
+          0.1875,
+          0.0,
+          0.25,
+          0.3125,
+          0.3125,
+          -0.0625,
+          0.375,
+          0.25,
+          0.1875,
+          0.375,
+          0.1875,
+          -0.0625,
+          0.25,
+          0.3125,
+          0.0,
+          0.1875,
+          0.1875,
+          0.0625,
+          -0.1875,
+          0.0,
+          0.3125,
+          0.0625,
+          0.25,
+          0.1875,
+          0.375,
+          0.1875,
+          0.25,
+          0.1875,
+          0.3125,
+          0.0,
+          0.0625,
+          0.4375,
+          0.125,
+          0.0625,
+          0.25,
+          0.25,
+          0.375,
+          -0.0625
+        ]
+      },
+      "left_hand_joints": {
+        "min": [
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0
+        ],
+        "max": [
+          0.4000000059604645,
+          0.4000000059604645,
+          0.4000000059604645,
+          0.4000000059604645,
+          0.4000000059604645,
+          1.0,
+          0.0
+        ],
+        "mean": [
+          0.003951099701225758,
+          0.003951099701225758,
+          0.003951099701225758,
+          0.003951099701225758,
+          0.003951099701225758,
+          0.8622109889984131,
+          0.0
+        ],
+        "std": [
+          0.027563678100705147,
+          0.027563678100705147,
+          0.027563678100705147,
+          0.027563678100705147,
+          0.027563678100705147,
+          0.17302781343460083,
+          0.0
+        ],
+        "q01": [
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0
+        ],
+        "q99": [
+          0.20000000298023224,
+          0.20000000298023224,
+          0.20000000298023224,
+          0.20000000298023224,
+          0.20000000298023224,
+          0.8999999761581421,
+          0.0
+        ]
+      },
+      "right_hand_joints": {
+        "min": [
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0
+        ],
+        "max": [
+          1.0,
+          1.0,
+          1.0,
+          1.0,
+          1.0,
+          1.0,
+          0.0
+        ],
+        "mean": [
+          0.18415340781211853,
+          0.18415340781211853,
+          0.18415340781211853,
+          0.18415340781211853,
+          0.18415340781211853,
+          0.8806127905845642,
+          0.0
+        ],
+        "std": [
+          0.3695652186870575,
+          0.3695652186870575,
+          0.3695652186870575,
+          0.3695652186870575,
+          0.3695652186870575,
+          0.19645722210407257,
+          0.0
+        ],
+        "q01": [
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0,
+          0.0
+        ],
+        "q99": [
+          1.0,
+          1.0,
+          1.0,
+          1.0,
+          1.0,
+          1.0,
+          0.0
+        ]
+      }
+    },
+    "relative_action": {}
+  }
+}

checkpoint-20000/experiment_cfg/final_model_config.json

@@ -0,0 +1,55 @@
+{
+  "model_type": "Gr00tN1d7",
+  "model_dtype": "bfloat16",
+  "model_name": "nvidia/Cosmos-Reason2-2B",
+  "backbone_model_type": "qwen",
+  "model_revision": null,
+  "tune_top_llm_layers": 0,
+  "backbone_embedding_dim": 2048,
+  "tune_llm": false,
+  "tune_visual": false,
+  "select_layer": 16,
+  "reproject_vision": false,
+  "use_flash_attention": true,
+  "load_bf16": false,
+  "backbone_trainable_params_fp32": true,
+  "extra_augmentation_config": null,
+  "apply_sincos_state_encoding": false,
+  "use_percentiles": true,
+  "use_relative_action": false,
+  "max_state_dim": 132,
+  "max_action_dim": 132,
+  "action_horizon": 40,
+  "hidden_size": 1024,
+  "input_embedding_dim": 1536,
+  "state_history_length": 1,
+  "add_pos_embed": true,
+  "attn_dropout": 0.2,
+  "use_vlln": true,
+  "max_seq_len": 1024,
+  "use_alternate_vl_dit": true,
+  "attend_text_every_n_blocks": 2,
+  "diffusion_model_cfg": {
+    "attention_head_dim": 48,
+    "dropout": 0.2,
+    "final_dropout": true,
+    "interleave_self_attention": true,
+    "norm_type": "ada_norm",
+    "num_attention_heads": 32,
+    "num_layers": 32,
+    "output_dim": 1024,
+    "positional_embeddings": null
+  },
+  "num_inference_timesteps": 4,
+  "noise_beta_alpha": 1.5,
+  "noise_beta_beta": 1.0,
+  "noise_s": 0.999,
+  "num_timestep_buckets": 1000,
+  "tune_projector": true,
+  "tune_diffusion_model": true,
+  "tune_vlln": true,
+  "state_dropout_prob": 0.2,
+  "exclude_state": false,
+  "use_mean_std": false,
+  "max_num_embodiments": 32
+}

checkpoint-20000/model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:67820845aeadebf18fb78ed492af7a5fdab63619ca53eee3d1fe9631aba01e91
+size 4990519232

checkpoint-20000/model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:27a74b305d6f9574f5e78407f3d8f297bea0a0061918126354198414de21a75b
+size 1919980184

checkpoint-20000/processor_config.json

@@ -0,0 +1,1159 @@
+{
+  "processor_class": "Gr00tN1d7Processor",
+  "processor_kwargs": {
+    "modality_configs": {
+      "real_g1_relative_eef_relative_joints": {
+        "video": {
+          "delta_indices": [
+            -20,
+            0
+          ],
+          "modality_keys": [
+            "ego_view"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        },
+        "state": {
+          "delta_indices": [
+            0
+          ],
+          "modality_keys": [
+            "left_wrist_eef_9d",
+            "right_wrist_eef_9d",
+            "left_hand",
+            "right_hand",
+            "left_arm",
+            "right_arm",
+            "waist"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        },
+        "action": {
+          "delta_indices": [
+            0,
+            1,
+            2,
+            3,
+            4,
+            5,
+            6,
+            7,
+            8,
+            9,
+            10,
+            11,
+            12,
+            13,
+            14,
+            15,
+            16,
+            17,
+            18,
+            19,
+            20,
+            21,
+            22,
+            23,
+            24,
+            25,
+            26,
+            27,
+            28,
+            29,
+            30,
+            31,
+            32,
+            33,
+            34,
+            35,
+            36,
+            37,
+            38,
+            39
+          ],
+          "modality_keys": [
+            "left_wrist_eef_9d",
+            "right_wrist_eef_9d",
+            "left_hand",
+            "right_hand",
+            "left_arm",
+            "right_arm",
+            "waist",
+            "base_height_command",
+            "navigate_command"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": [
+            {
+              "rep": "RELATIVE",
+              "type": "EEF",
+              "format": "XYZ_ROT6D",
+              "state_key": "left_wrist_eef_9d"
+            },
+            {
+              "rep": "RELATIVE",
+              "type": "EEF",
+              "format": "XYZ_ROT6D",
+              "state_key": "right_wrist_eef_9d"
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "left_hand"
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "right_hand"
+            },
+            {
+              "rep": "RELATIVE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "left_arm"
+            },
+            {
+              "rep": "RELATIVE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "right_arm"
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "waist"
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "base_height_command"
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "navigate_command"
+            }
+          ]
+        },
+        "language": {
+          "delta_indices": [
+            0
+          ],
+          "modality_keys": [
+            "annotation.human.task_description"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        }
+      },
+      "real_r1_pro_sharpa_relative_eef_mecka": {
+        "video": {
+          "delta_indices": [
+            -30,
+            0
+          ],
+          "modality_keys": [
+            "ego_view_cropratio_res320x240_freq30"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        },
+        "state": {
+          "delta_indices": [
+            0
+          ],
+          "modality_keys": [
+            "left_wrist_eef",
+            "right_wrist_eef",
+            "left_hand_joints",
+            "right_hand_joints"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        },
+        "action": {
+          "delta_indices": [
+            0,
+            1,
+            2,
+            3,
+            4,
+            5,
+            6,
+            7,
+            8,
+            9,
+            10,
+            11,
+            12,
+            13,
+            14,
+            15,
+            16,
+            17,
+            18,
+            19,
+            20,
+            21,
+            22,
+            23,
+            24,
+            25,
+            26,
+            27,
+            28,
+            29,
+            30,
+            31,
+            32,
+            33,
+            34,
+            35,
+            36,
+            37,
+            38,
+            39
+          ],
+          "modality_keys": [
+            "left_wrist_eef",
+            "right_wrist_eef",
+            "left_hand_joints",
+            "right_hand_joints"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": [
+            {
+              "rep": "RELATIVE",
+              "type": "EEF",
+              "format": "XYZ_ROT6D",
+              "state_key": "left_wrist_eef"
+            },
+            {
+              "rep": "RELATIVE",
+              "type": "EEF",
+              "format": "XYZ_ROT6D",
+              "state_key": "right_wrist_eef"
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "left_hand_joints"
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "right_hand_joints"
+            }
+          ]
+        },
+        "language": {
+          "delta_indices": [
+            0
+          ],
+          "modality_keys": [
+            "annotation.human.coarse_action"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        }
+      },
+      "real_r1_pro_sharpa_relative_eef_human": {
+        "video": {
+          "delta_indices": [
+            -20,
+            0
+          ],
+          "modality_keys": [
+            "ego_view_res320x240_freq20",
+            "left_wrist_view_res320x240_freq20",
+            "right_wrist_view_res320x240_freq20"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        },
+        "state": {
+          "delta_indices": [
+            0
+          ],
+          "modality_keys": [
+            "left_wrist_eef",
+            "right_wrist_eef",
+            "left_hand_joints",
+            "right_hand_joints"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        },
+        "action": {
+          "delta_indices": [
+            0,
+            1,
+            2,
+            3,
+            4,
+            5,
+            6,
+            7,
+            8,
+            9,
+            10,
+            11,
+            12,
+            13,
+            14,
+            15,
+            16,
+            17,
+            18,
+            19,
+            20,
+            21,
+            22,
+            23,
+            24,
+            25,
+            26,
+            27,
+            28,
+            29,
+            30,
+            31,
+            32,
+            33,
+            34,
+            35,
+            36,
+            37,
+            38,
+            39
+          ],
+          "modality_keys": [
+            "left_wrist_eef",
+            "right_wrist_eef",
+            "left_hand_joints",
+            "right_hand_joints"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": [
+            {
+              "rep": "RELATIVE",
+              "type": "EEF",
+              "format": "XYZ_ROT6D",
+              "state_key": "left_wrist_eef"
+            },
+            {
+              "rep": "RELATIVE",
+              "type": "EEF",
+              "format": "XYZ_ROT6D",
+              "state_key": "right_wrist_eef"
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "left_hand_joints"
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "right_hand_joints"
+            }
+          ]
+        },
+        "language": {
+          "delta_indices": [
+            0
+          ],
+          "modality_keys": [
+            "annotation.human.coarse_action"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        }
+      },
+      "real_r1_pro_sharpa_relative_eef": {
+        "video": {
+          "delta_indices": [
+            -20,
+            0
+          ],
+          "modality_keys": [
+            "ego_view_res320x240_freq20",
+            "left_wrist_view_res320x240_freq20",
+            "right_wrist_view_res320x240_freq20"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        },
+        "state": {
+          "delta_indices": [
+            0
+          ],
+          "modality_keys": [
+            "left_wrist_eef",
+            "right_wrist_eef",
+            "left_hand_joints",
+            "right_hand_joints"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        },
+        "action": {
+          "delta_indices": [
+            0,
+            1,
+            2,
+            3,
+            4,
+            5,
+            6,
+            7,
+            8,
+            9,
+            10,
+            11,
+            12,
+            13,
+            14,
+            15,
+            16,
+            17,
+            18,
+            19,
+            20,
+            21,
+            22,
+            23,
+            24,
+            25,
+            26,
+            27,
+            28,
+            29,
+            30,
+            31,
+            32,
+            33,
+            34,
+            35,
+            36,
+            37,
+            38,
+            39
+          ],
+          "modality_keys": [
+            "left_wrist_eef",
+            "right_wrist_eef",
+            "left_hand_joints",
+            "right_hand_joints"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": [
+            {
+              "rep": "RELATIVE",
+              "type": "EEF",
+              "format": "XYZ_ROT6D",
+              "state_key": "left_wrist_eef"
+            },
+            {
+              "rep": "RELATIVE",
+              "type": "EEF",
+              "format": "XYZ_ROT6D",
+              "state_key": "right_wrist_eef"
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "left_hand_joints"
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "right_hand_joints"
+            }
+          ]
+        },
+        "language": {
+          "delta_indices": [
+            0
+          ],
+          "modality_keys": [
+            "annotation.human.coarse_action"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        }
+      },
+      "xdof_relative_eef_relative_joint": {
+        "video": {
+          "delta_indices": [
+            -30,
+            0
+          ],
+          "modality_keys": [
+            "top_camera-images-rgb_320_240",
+            "left_camera-images-rgb_320_240",
+            "right_camera-images-rgb_320_240"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        },
+        "state": {
+          "delta_indices": [
+            0
+          ],
+          "modality_keys": [
+            "left_wrist_eef",
+            "right_wrist_eef",
+            "left_gripper_pos",
+            "right_gripper_pos",
+            "left_joint_pos",
+            "right_joint_pos"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        },
+        "action": {
+          "delta_indices": [
+            0,
+            1,
+            2,
+            3,
+            4,
+            5,
+            6,
+            7,
+            8,
+            9,
+            10,
+            11,
+            12,
+            13,
+            14,
+            15,
+            16,
+            17,
+            18,
+            19,
+            20,
+            21,
+            22,
+            23,
+            24,
+            25,
+            26,
+            27,
+            28,
+            29,
+            30,
+            31,
+            32,
+            33,
+            34,
+            35,
+            36,
+            37,
+            38,
+            39
+          ],
+          "modality_keys": [
+            "left_wrist_eef",
+            "right_wrist_eef",
+            "left_gripper_pos",
+            "right_gripper_pos",
+            "left_joint_pos",
+            "right_joint_pos"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": [
+            {
+              "rep": "RELATIVE",
+              "type": "EEF",
+              "format": "XYZ_ROT6D",
+              "state_key": "left_wrist_eef"
+            },
+            {
+              "rep": "RELATIVE",
+              "type": "EEF",
+              "format": "XYZ_ROT6D",
+              "state_key": "right_wrist_eef"
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "left_gripper_pos"
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "right_gripper_pos"
+            },
+            {
+              "rep": "RELATIVE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "left_joint_pos"
+            },
+            {
+              "rep": "RELATIVE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "right_joint_pos"
+            }
+          ]
+        },
+        "language": {
+          "delta_indices": [
+            0
+          ],
+          "modality_keys": [
+            "annotation.task"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        }
+      },
+      "real_r1_pro_sharpa_relative_eef_maxinsights": {
+        "video": {
+          "delta_indices": [
+            -30,
+            0
+          ],
+          "modality_keys": [
+            "ego_view_cropratio_res320x240_freq30"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        },
+        "state": {
+          "delta_indices": [
+            0
+          ],
+          "modality_keys": [
+            "left_wrist_eef",
+            "right_wrist_eef",
+            "left_hand_joints",
+            "right_hand_joints"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        },
+        "action": {
+          "delta_indices": [
+            0,
+            1,
+            2,
+            3,
+            4,
+            5,
+            6,
+            7,
+            8,
+            9,
+            10,
+            11,
+            12,
+            13,
+            14,
+            15,
+            16,
+            17,
+            18,
+            19,
+            20,
+            21,
+            22,
+            23,
+            24,
+            25,
+            26,
+            27,
+            28,
+            29,
+            30,
+            31,
+            32,
+            33,
+            34,
+            35,
+            36,
+            37,
+            38,
+            39
+          ],
+          "modality_keys": [
+            "left_wrist_eef",
+            "right_wrist_eef",
+            "left_hand_joints",
+            "right_hand_joints"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": [
+            {
+              "rep": "RELATIVE",
+              "type": "EEF",
+              "format": "XYZ_ROT6D",
+              "state_key": "left_wrist_eef"
+            },
+            {
+              "rep": "RELATIVE",
+              "type": "EEF",
+              "format": "XYZ_ROT6D",
+              "state_key": "right_wrist_eef"
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "left_hand_joints"
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "right_hand_joints"
+            }
+          ]
+        },
+        "language": {
+          "delta_indices": [
+            0
+          ],
+          "modality_keys": [
+            "annotation.human.coarse_action"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        }
+      },
+      "xdof_relative_eef_relative_joint_subtask": {
+        "video": {
+          "delta_indices": [
+            -30,
+            0
+          ],
+          "modality_keys": [
+            "top_camera-images-rgb_320_240",
+            "left_camera-images-rgb_320_240",
+            "right_camera-images-rgb_320_240"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        },
+        "state": {
+          "delta_indices": [
+            0
+          ],
+          "modality_keys": [
+            "left_wrist_eef",
+            "right_wrist_eef",
+            "left_gripper_pos",
+            "right_gripper_pos",
+            "left_joint_pos",
+            "right_joint_pos"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        },
+        "action": {
+          "delta_indices": [
+            0,
+            1,
+            2,
+            3,
+            4,
+            5,
+            6,
+            7,
+            8,
+            9,
+            10,
+            11,
+            12,
+            13,
+            14,
+            15,
+            16,
+            17,
+            18,
+            19,
+            20,
+            21,
+            22,
+            23,
+            24,
+            25,
+            26,
+            27,
+            28,
+            29,
+            30,
+            31,
+            32,
+            33,
+            34,
+            35,
+            36,
+            37,
+            38,
+            39
+          ],
+          "modality_keys": [
+            "left_wrist_eef",
+            "right_wrist_eef",
+            "left_gripper_pos",
+            "right_gripper_pos",
+            "left_joint_pos",
+            "right_joint_pos"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": [
+            {
+              "rep": "RELATIVE",
+              "type": "EEF",
+              "format": "XYZ_ROT6D",
+              "state_key": "left_wrist_eef"
+            },
+            {
+              "rep": "RELATIVE",
+              "type": "EEF",
+              "format": "XYZ_ROT6D",
+              "state_key": "right_wrist_eef"
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "left_gripper_pos"
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "right_gripper_pos"
+            },
+            {
+              "rep": "RELATIVE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "left_joint_pos"
+            },
+            {
+              "rep": "RELATIVE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "right_joint_pos"
+            }
+          ]
+        },
+        "language": {
+          "delta_indices": [
+            0
+          ],
+          "modality_keys": [
+            "annotation.sub_task"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        }
+      },
+      "oxe_droid_relative_eef_relative_joint": {
+        "video": {
+          "delta_indices": [
+            -15,
+            0
+          ],
+          "modality_keys": [
+            "exterior_image_1_left",
+            "wrist_image_left"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        },
+        "state": {
+          "delta_indices": [
+            0
+          ],
+          "modality_keys": [
+            "eef_9d",
+            "gripper_position",
+            "joint_position"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        },
+        "action": {
+          "delta_indices": [
+            0,
+            1,
+            2,
+            3,
+            4,
+            5,
+            6,
+            7,
+            8,
+            9,
+            10,
+            11,
+            12,
+            13,
+            14,
+            15,
+            16,
+            17,
+            18,
+            19,
+            20,
+            21,
+            22,
+            23,
+            24,
+            25,
+            26,
+            27,
+            28,
+            29,
+            30,
+            31,
+            32,
+            33,
+            34,
+            35,
+            36,
+            37,
+            38,
+            39
+          ],
+          "modality_keys": [
+            "eef_9d",
+            "gripper_position",
+            "joint_position"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": [
+            {
+              "rep": "RELATIVE",
+              "type": "EEF",
+              "format": "XYZ_ROT6D",
+              "state_key": "eef_9d"
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "gripper_position"
+            },
+            {
+              "rep": "RELATIVE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": "joint_position"
+            }
+          ]
+        },
+        "language": {
+          "delta_indices": [
+            0
+          ],
+          "modality_keys": [
+            "annotation.language.language_instruction"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        }
+      },
+      "unitree_g1_sonic": {
+        "video": {
+          "delta_indices": [
+            0
+          ],
+          "modality_keys": [
+            "ego_view"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        },
+        "state": {
+          "delta_indices": [
+            0
+          ],
+          "modality_keys": [
+            "left_leg",
+            "right_leg",
+            "waist",
+            "left_arm",
+            "right_arm",
+            "left_hand",
+            "right_hand",
+            "projected_gravity"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        },
+        "action": {
+          "delta_indices": [
+            0,
+            1,
+            2,
+            3,
+            4,
+            5,
+            6,
+            7,
+            8,
+            9,
+            10,
+            11,
+            12,
+            13,
+            14,
+            15,
+            16,
+            17,
+            18,
+            19,
+            20,
+            21,
+            22,
+            23,
+            24,
+            25,
+            26,
+            27,
+            28,
+            29,
+            30,
+            31,
+            32,
+            33,
+            34,
+            35,
+            36,
+            37,
+            38,
+            39
+          ],
+          "modality_keys": [
+            "motion_token",
+            "left_hand_joints",
+            "right_hand_joints"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": [
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": null
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": null
+            },
+            {
+              "rep": "ABSOLUTE",
+              "type": "NON_EEF",
+              "format": "DEFAULT",
+              "state_key": null
+            }
+          ]
+        },
+        "language": {
+          "delta_indices": [
+            0
+          ],
+          "modality_keys": [
+            "annotation.human.task_description"
+          ],
+          "sin_cos_embedding_keys": null,
+          "mean_std_embedding_keys": null,
+          "action_configs": null
+        }
+      }
+    },
+    "image_crop_size": [
+      230,
+      230
+    ],
+    "image_target_size": [
+      256,
+      256
+    ],
+    "use_albumentations": true,
+    "random_rotation_angle": 0,
+    "color_jitter_params": {
+      "brightness": 0.3,
+      "contrast": 0.4,
+      "saturation": 0.5,
+      "hue": 0.08
+    },
+    "shortest_image_edge": 256,
+    "crop_fraction": 0.95,
+    "letter_box_transform": false,
+    "model_name": "nvidia/Cosmos-Reason2-2B",
+    "model_type": "qwen",
+    "formalize_language": true,
+    "max_state_dim": 132,
+    "max_action_dim": 132,
+    "max_action_horizon": 40,
+    "use_percentiles": true,
+    "use_mean_std": false,
+    "clip_outliers": true,
+    "apply_sincos_state_encoding": false,
+    "use_relative_action": true,
+    "exclude_state": false,
+    "state_dropout_prob": 0.2
+  }
+}