IDEAS-Lab-Northwestern/gr00t-n16-datagen-v1-clutter-joint-2cam-yanZ

GR00T-N1.6 - Clutter (joint, 2-cam)

NVIDIA Isaac GR00T-N1.6-3B fine-tuned on the ManiGuard clutter base task (sim Franka Panda). Part of the ManiGuard VLA benchmark - GR00T vs pi0.5 on the same task families with identical data, cameras, and controller.

Model

• Base: nvidia/GR00T-N1.6-3B - Eagle (nvidia/Eagle-Block2A-2B-v2) VLM + flow-matching DiT action head
• Embodiment: NEW_EMBODIMENT - Franka Panda, 8-D joint state/action (7 arm joints + 1 gripper)
• Cameras (2): image_left (overview) + wrist (256x256)
• Action: arm = state-relative chunks, gripper = absolute; 16-step horizon; NON_EEF (joint space)
• Tuning: GR00T-N1.6 default - VLM (LLM + visual) frozen, train projector + diffusion action head (no LoRA)

Training

• 8-card config, DeepSpeed ZeRO-2, bf16, global batch 256, 7044 steps (~2 epochs over 901,520 frames), cosine LR (peak 2e-4, sqrt-scaled), warmup 0.05
• Data: IDEAS-Lab-Northwestern/datagen-clutter-v1-joint-5cam; videos decoded as H.264 for GR00T's torchcodec loader

Usage

Load with Gr00tPolicy from Isaac-GR00T (n1d6), --embodiment-tag NEW_EMBODIMENT. The included processor/ carries the normalization stats + modality config.

WARNING - Convention (must match at eval): joint-space JointController (absolute joint targets, NON_EEF) + 2 cameras (image_left overview + wrist). A mismatched controller or camera set silently feeds an out-of-distribution input.