cloudwalk-research/psi0-g1-sneaker-199ep-v2-finetune

Psi0 Fine-Tune v2 — G1 Sneaker-in-Box, Bimanual (Inspire RH56DFTP)

v2 community fine-tune of Psi0 (paper) on a Unitree G1 humanoid with Inspire RH56DFTP hands, with bimanual hand-selection coverage.

Same task as the v1 fine-tune — "Pick up the sneaker and place it inside the box." — but trained on the bimanual 199-episode v2 dataset (108 LEFT-hand demos + 91 RIGHT-hand demos), with 6 episodes held out as the v2 eval set. The hypothesis under test: given roughly balanced LEFT/RIGHT data, can the policy learn to visually select which hand to use based on where the shoe is in the scene? Trained at the CloudWalk Robotics Lab (CW-RL), 2026-05.

This is a follow-up to the v1 fine-tune, which was the first publicly reported external reproduction of the Psi0 paper's full pipeline (collect → fine-tune → real-robot deploy). v2 directly addresses one of the three failure modes surfaced in v1's real-robot deployment (single-handed only); the other two (no recovery / no negative scenes) are deferred to v3.

Quick facts

Base	Psi0 (Qwen3-VL-2B + flow-based MM-DiT action expert)
Pretrained	pre.fast.1by1.2601091803.ckpt.ego200k.he30k + postpre.1by1.pad36.2601131206.ckpt.he30k
Training dataset	cloudwalk-research/psi0-g1-sneaker-199ep-v2 (199 ep, 60 656 frames @ 30 Hz, 640×480 ego cam, 108 LEFT / 91 RIGHT)
Held-out eval dataset	cloudwalk-research/psi0-g1-sneaker-6ep-v2-eval (6 ep, 1 791 frames, 3 LEFT / 3 RIGHT) — never seen during training
Canonical source	cloudwalk-research/psi0-g1-sneaker-205ep-v2-source (205 = 199 + 6, immutable union)
Robot target	Unitree G1 + Inspire RH56DFTP (NOT Dex3-1)
Action space	36-D (6 hand + 1 pad / side, 14 arm, 4 torso, 4 locomotion)
State space	32-D (padded to 36-D for the model)
Action chunk	30
Hardware	8× NVIDIA B200 (sm_100 / Blackwell), system PyTorch 2.10 (NVIDIA build)
Mixed precision	bf16
Optimizer	AdamW, lr=1e-4 cosine + min_lr 5e-7, warmup 1000, betas=(0.95, 0.999), wd=1e-6, grad-clip 1.0
Steps / batch	40 000 / global batch 128 (16 per GPU × 8 GPUs)
Effective epochs	84 (vs v1's 196 — deliberate intervention against v1 overfitting)
Checkpointing	every 10 000 steps (max_checkpoints_to_keep=5; 4 retained: 10k / 20k / 30k / 40k)
RTC	enabled, max_delay = 8
VLM	frozen (only action header trained)
Wall-clock	3 h 42 m 34 s (slightly faster than v1's 3 h 58 m despite 2× more data; system PyTorch 2.10 is faster than v1's pip 2.7 wheel)
Final eval/loss	0.081 (from 38.03 — ~470× reduction)
W&B run	3yifcdgs (CW-RL internal)

Repository contents

• checkpoints/ckpt_10000/model.safetensors (~5.9 GB — early, lr ~ 9e-5)
• checkpoints/ckpt_20000/model.safetensors (~5.9 GB — mid, lr ~ 7e-5)
• checkpoints/ckpt_30000/model.safetensors (~5.9 GB — late, lr ~ 2e-5)
• checkpoints/ckpt_40000/model.safetensors (~5.9 GB — final, lr → 0)
• run_config.json (tyro-serialized launch config)
• argv.txt (full command line)
• README.md (this file)

Optimizer / scheduler / sampler / random-state files are intentionally omitted — needed only to resume training, would add ~3.7 GB per checkpoint.

Evaluation

Validation runs on a fresh-batch sample of the training repo (Psi0 standard practice — val_repo_ids defaults to train_repo_ids). v2 also publishes a real held-out set (psi0-g1-sneaker-6ep-v2-eval, never seen during training); the open-loop evaluation section below uses the held-out set, while the in-process metrics in the table below are still in-distribution. Treat training-time eval/loss as a smoothed train-loss probe, not a generalization signal. See Open-loop evaluation (ID + OOD) below for the held-out result.

Metric	Step 0	Step 40k	Improvement	v1 reference (hjy0caz0)
eval/loss	38.03	0.081	~470×	0.0675
eval/err_l1_arm_joints	~3.6	0.111 rad	~32×	0.058 rad
eval/err_l1_hand_joints	~1.4	0.0135 rad	~104×	0.012 rad
eval/err_l1_height	~5.0e-3	1.0e-5 m	~500×	1.0e-5 m
eval/err_l1_{vx, vy, vyaw, target_yaw, torso_rpy}	0	0	task is stationary	0

Full training curves tracked internally (W&B run 3yifcdgs).

The v2 eval curve sat directly on top of v1's curve on shared step axes — same shape, same plateau timing, modestly higher final values. This is consistent with v2 having ~2.3× less per-example exposure (84 epochs vs v1's 196) and a wider arm trajectory distribution from bimanual mirroring. eval/err_l1_hand_joints = 0.0135 is the headline data-level result for the bimanual hypothesis: v2 averages over both LEFT and RIGHT active hand DoF (vs v1's RIGHT only) and the per-active-side error is essentially unchanged. Whether the policy learned to visually select which hand to use is the Phase 14 question, not answered by this in-distribution metric.

eval/err_l1_arm_joints = 0.111 rad is ~2× v1's 0.058 rad. The most plausible read is that bimanual mirroring widens the arm trajectory distribution v2 must fit (LEFT-side and RIGHT-side trajectories share a single network); the prediction was anchored to v1's single-side scope. Less likely: an artefact of v2's lower per-example exposure. Either way, this is in-distribution and won't predict real-robot performance.

Open-loop evaluation (ID + OOD) — 2026-05-07

TL;DR. v2 memorized rather than generalized. At ckpt_40000 the OOD/ID gap is 5.5×–7.2× on hand and arm joints — ID is healthy (the inference path is sane), OOD blows past the runbook's predicted thresholds by 5×/8×. Training past 30 k steps made it worse on OOD arm_joints (regression 0.609 → 0.628 → 0.632) while ID arm_joints kept improving — that's the textbook overfitting signature, and it makes ckpt_30000 the Pareto-better deploy candidate. The model is also 3× worse picking with the RIGHT hand (OOD hand_joints 0.214 vs LEFT 0.070 rad) despite the training set being only 1.19× LEFT-heavy (108 L / 91 R) — so RIGHT-side scene diversity (object pose, lighting, distractors) is the bottleneck, not raw episode count. Verdict: Phase 14 (real-robot eval grid) is gated; if shipping is forced, prefer ckpt_30000.

Biggest lesson for v3 (single highest-leverage intervention): wire the OOD probe into the inner training loop as an early-stopping signal. Today Psi0's eval/loss is in-distribution by default (val_repo_ids = train_repo_ids); the OOD curve is invisible during training. Without that probe, we trained blind past the 30 k OOD optimum.

Figure 1 — Glance view: each polygon is one checkpoint; bigger polygon area = better overall on that metric. Score per axis is 1 − (error / max_error_in_panel), so the outer edge = perfect (zero error) and the origin = worst observed. hand_joints panel: ckpt_30k (teal) and ckpt_40k (red) have nearly identical areas, both substantially larger than ckpt_20k (purple) — the latter collapses at the OOD-RIGHT axis, dragging its area down. arm_joints panel: every polygon collapses into a thin vertical spike pointing only at the ID axis (top) — all three checkpoints have near-zero scores on every OOD axis, visualizing that the model fails at OOD arm equally regardless of training step. Radial signature: hand-side ckpt selection matters; arm-side ckpt selection does not, because no ckpt generalizes there.

Figure 2 — Open-loop convergence on 4 evals × 3 checkpoints. ID (blue) keeps improving 20k → 40k on both metrics; OOD all-6 (red) plateaus on hand_joints and regresses on arm_joints past 30k; OOD-LEFT (green dashed) tracks OOD-all closely while OOD-RIGHT (orange dashed) sits visibly above on hand_joints — the per-side asymmetry. The growing ID-vs-OOD divergence IS the overfitting signature; ckpt_30000 sits at the OOD optimum.

Figure 3 — ckpt_40000 snapshot. ID (blue) is tiny next to the OOD bars — that's the memorization gap. hand_joints: OOD-RIGHT (0.213) is 3.06× OOD-LEFT (0.070) despite the training set being only 1.19× LEFT-heavy (108 L / 91 R) — episode-count alone cannot mechanically explain a 3× error gap, so RIGHT-side scene diversity is the load-bearing variable. arm_joints: LEFT and RIGHT are within 7% (0.661 vs 0.617) — the asymmetry is hand-specific (fingertip-trajectory / grip-timing), not reach-specific.

Figure 4 — Cross-checkpoint magnitude comparison (Figure 2 as bars, side-by-side). Reads top-down per ckpt: hand_joints ID is in monotone freefall (0.042 → 0.017 → 0.014); OOD all-6 (red) is stuck at ~0.10 from 30k onward; OOD-RIGHT (orange) bar towers above all others at every ckpt (0.231 / 0.212 / 0.213) and is the dominant contributor to the OOD-all bar. arm_joints ID drops 0.205 → 0.137 → 0.114; the three OOD bars are tightly clustered around 0.61–0.66, with a slight upward creep from 30k → 40k. The hand-side asymmetry is therefore the only place where ckpt selection moves the needle on OOD.

Figure 5 — Asymmetry evolution across training. The RIGHT-vs-LEFT hand_joints multiplier grows during training: 2.23× at 20k → 3.11× at 30k → 3.06× at 40k. The 30k step is where the LEFT-side error drops sharply (0.103 → 0.068, down 34%) while the RIGHT-side barely moves (0.231 → 0.212, down 8%) — extra training disproportionately benefits the better-covered side, widening the asymmetry rather than closing it. By 40k the asymmetry has plateaued and arm_joints has started regressing on both sides — late training is past the OOD optimum.

ID (random 6 episodes from training set) and OOD (held-out 6-episode psi0-g1-sneaker-6ep-v2-eval) using byte-for-byte identical eval code (openloop_eval_psi0.py, CW-RL internal). Only WHICH frames feed the loop changes between ID and OOD; the model's training-time normalization stats stay frozen. Single GPU (RTX 5090, sm_120), --stride 1 --num-inference-steps 10. Full per-side tables live on the eval-set dataset card.

Headline @ ckpt_40000 — denormalized L1 error norm (smaller is better)

Group	ID (train-v2 random 6)	OOD all-6 (eval-v2)	OOD-LEFT (idx 3,4,5)	OOD-RIGHT (idx 0,1,2)	OOD/ID ratio	per-side |L−R|/max
hand_joints	0.0138	0.1000	0.0698	0.2135	7.2×	67.3% (RIGHT 3.06× worse)
arm_joints	0.1140	0.6318	0.6614	0.6168	5.5×	6.7% (symmetric)
height (m)	6.7e-6	6.3e-6	6.0e-6	7.0e-6	~1.0×	—
rpy / vx / vy / torso_vyaw / target_yaw	0.0	0.0	0.0	0.0	—	—

Per-checkpoint OOD all-6 (the convergence signal)

ckpt	OOD hand_joints	OOD arm_joints	(ID hand for ref)	(ID arm for ref)
20 000	0.1201	0.6090 ← best arm	0.0417	0.2045
30 000	0.1009	0.6278	0.0173	0.1369
40 000	0.1000 ← best hand	0.6318	0.0138	0.1140

Reading

• OOD/ID gap is 5.5×–7.2× @ ckpt_40000 → the model fits the training distribution cleanly but generalizes weakly to held-out episodes. Memorization signature, not generalization.
• ckpt_30000 is the better deploy candidate than ckpt_40000: OOD arm_joints regresses past 30k (0.609 → 0.628 → 0.632), OOD hand_joints plateaus at 30k. Late training trades OOD performance for ID overfit. ID curve keeps improving past 30k; OOD curve does not.
• Per-side hand_joints on OOD is 3× worse on RIGHT-pickup episodes (RIGHT 0.214 vs LEFT 0.070 rad). Training set is 108 L / 91 R but the 1.19× episode-count ratio doesn't explain a 3× error gap — RIGHT-side scene diversity (sneaker pose, lighting, distractors) is the more likely bottleneck. arm_joints is symmetric across sides, so the asymmetry is finger / grip-timing-specific, not reach-specific.

Decision

Recommendation: do NOT proceed to Phase 14 (real-robot eval grid) on ckpt_40000 without further investigation. ID anchor is healthy (hand 0.0138 / arm 0.114 rad, in family with v1's 0.0117 / 0.058 rad) — the inference path is sane. But OOD blows past the runbook's predicted thresholds (hand_joints ≤ 0.020 → actual 0.100, 5× over; arm_joints ≤ 0.080 → actual 0.632, 8× over). If a real-robot test must happen, prefer ckpt_30000; expect coarser end-effector tracking on RIGHT-pickup scenes.

v3 priorities surfaced by these numbers (in expected-impact order): (1) RIGHT-side scene diversity (object pose, lighting, distractors); (2) recovery + negative scenes (carry-over from v1 punch list, never addressed in v2); (3) wire eval-v2 as an inner-loop OOD probe for early stopping; (4) raise episode count ~1.5–2× to keep effective-epochs lower at 40 k steps.

How to download

import os
from huggingface_hub import hf_hub_download

repo = "cloudwalk-research/psi0-g1-sneaker-199ep-v2-finetune"
local_dir = ".runs/finetune/sneaker-ext-left-real"
for f in [
    "run_config.json",
    "argv.txt",
    "checkpoints/ckpt_40000/model.safetensors",
]:
    hf_hub_download(repo_id=repo, filename=f, repo_type="model",
                    local_dir=local_dir)

How to evaluate (closed-loop sim)

bash sim/scripts/run_closed_loop.sh \
  "Pick up the sneaker and place it inside the box." \
  .runs/finetune/sneaker-ext-left-real 40000 \
  --fixed_base --live_viewer

For v2, watch whether both Inspire hands move in the rendered video. In v1 only the RIGHT hand exhibited motion (LEFT was held open at 1.0 throughout the dataset); v2 should show motion on both sides at different episodes. If only one moves, the bimanual coverage did not transfer through training.

Known caveats

1. Hand-selection hypothesis is not yet validated. This card reports the data-level signal that v2 absorbed bimanual coverage (per-active-side hand error matches v1). Whether the policy visually selects which hand to use based on where the shoe is in the scene is what Phase 14 (real-robot eval grid) is designed to measure. Until then, treat v2's bimanual capability as a hypothesis with strong data-level support, not a confirmed generalization.
2. Hand commands are still lockstep across the 6 active Inspire DoF (per side). The teleop dataset was collected with a Pico 4 Ultra trigger driving all 6 right-hand or 6 left-hand DoF in lockstep (single grip-strength scalar, ~21 quantized levels at 0.05 step). The model can only output what it saw — it cannot do per-finger dexterous control on this checkpoint. The bimanual extension to v2 only widens the which-hand distribution, not the per-finger one.
3. Task is bimanual but stationary. Action dims for locomotion (vx/vy/vyaw/target_yaw) and torso RPY are constant in the dataset; the model produces those constants at inference. Do not expect locomotion behavior.
4. In-process eval/loss is in-distribution. Psi0 LerobotDataConfig defaults val_repo_ids to train_repo_ids. Training-time eval and train share data; for the real generalization signal, see the Open-loop evaluation (ID + OOD) section above (numbers as of 2026-05-07).
5. Action-dim normalization tolerates zero-range dims. action[6], action[13], action[28..30], action[32..36] have min == max == 0 in the dataset stats. The repo action_norm_type=bounds path handles divide-by-zero internally; smoke-tested at 1 GPU and 8 GPUs.
6. Inspire deploy bridge is required (resolved in v1, inherited in v2). real/teleop/master_whole_body.py:206 of the Psi0 deploy stack hard-codes Dex3_1_Controller, which publishes hand commands to rt/dex3/{left,right}/cmd. The Inspire RH56DFTP listens on rt/inspire/cmd. The Inspire_VLA_Controller bridge implemented during v1 Phase 14 is the prerequisite — verify it is integrated in the deploy branch before driving v2 on the physical G1.

Lineage

Item	Repo
v1 fine-tune (RIGHT-hand only)	cloudwalk-research/psi0-g1-sneaker-94ep-v1-finetune
v2 fine-tune (this repo, bimanual)	cloudwalk-research/psi0-g1-sneaker-199ep-v2-finetune
v1 dataset (94 ep, RIGHT-only)	cloudwalk-research/psi0-g1-sneaker-94ep-v1
v2 training dataset (199 ep, 108 L / 91 R)	cloudwalk-research/psi0-g1-sneaker-199ep-v2
v2 held-out eval (6 ep, 3 L / 3 R)	cloudwalk-research/psi0-g1-sneaker-6ep-v2-eval
v2 canonical source (205 ep, immutable)	cloudwalk-research/psi0-g1-sneaker-205ep-v2-source

References

• Psi0 — Generalist VLA model, base policy fine-tuned here. Paper, GitHub.
• Pre-deploy checklist — Psi0 issue #39 (community Q&A by Songlin Wei, Psi0 author).
• v1 fine-tune — cloudwalk-research/psi0-g1-sneaker-94ep-v1-finetune; first publicly reported external reproduction of the Psi0 pipeline.
• Cluster runbook (v1 and v2) — End-to-end procedures (Phases 0–14) including the bimanual hypothesis, the held-out eval split, and Phase 14's real-robot eval grid; tracked internally at CW-RL.
• Sibling fine-tune (apple-to-plate, Isaac-Lab sim) — cloudwalk-research/psi0-apple-to-plate-teleop (dataset).

Attribution

Developed by cloudwalk-research in the CloudWalk Robotics Lab (CW-RL). Part of the Psi0 collection. Fine-tuned from the Psi0 base model; integrates with the LeRobot framework.

Citation

If you build on this checkpoint, cite both upstream Psi0 and this fine-tune:

@misc{cwrl_psi0_sneaker_v2_2026,
  title        = {Psi0 Fine-Tune v2 --- G1 Sneaker-in-Box, Bimanual (Inspire RH56DFTP)},
  author       = {{CloudWalk Robotics Lab}},
  year         = {2026},
  howpublished = {Hugging Face model repository},
  url          = {https://huggingface.co/cloudwalk-research/psi0-g1-sneaker-199ep-v2-finetune}
}

@article{psi0_2026,
  title         = {Psi0: A Generalist Vision-Language-Action Model for Humanoid Robots},
  author        = {Wei, Songlin and others},
  year          = {2026},
  eprint        = {2603.12263},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2603.12263}
}

@article{black2024pi0,
  title   = {{$\pi_0$}: A Vision-Language-Action Flow Model for General Robot Control},
  author  = {Black, Kevin and Brown, Noah and Driess, Danny and others},
  journal = {arXiv preprint arXiv:2410.24164},
  year    = {2024}
}

@software{lerobot2024,
  title  = {{LeRobot}: State-of-the-art Machine Learning for Real-World Robotics in {PyTorch}},
  author = {{Hugging Face}},
  url    = {https://github.com/huggingface/lerobot},
  year   = {2024}
}

License

Released under Apache-2.0. This is a research preview: not intended for safety-critical use; closed-loop deployment on a physical humanoid requires human oversight.