eval: publish closed-loop 12-position grid results + photo (checkpoint-20000)

Real-G1 closed-loop eval: bottle placed at 12 fixed positions on one table half-side
(same grid used for data collection), 1 trial/position, plus 5-trial fixed positions
where collected. Legend: red=knocked over, orange=indecision loop, green=success.

Results: 🟩 Success | 8/12 | 67%.
Adds the eval photo (eval_12_positions_20k.png), the procedure, the results table,
and a data-driven conclusion. Lineage table Closed-loop column now carries each
model's grid verdict; v2 champion updated 11/12 -> 10/12 (canonical).

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -32,9 +32,9 @@ pipeline_tag: robotics
 
 Single-handed pick task ("grab the bottle", RIGHT hand). Trained at the **CloudWalk Robotics Lab (CW-RL)**, 2026-06, on a **502-episode / 120,017-frame** real-robot teleoperation dataset — the **largest and best-curated** of the lineage — for the `UNITREE_G1_SONIC` embodiment. Released as a reference fine-tune for teams building manipulation policies on the GR00T + SONIC + MuJoCo/G1 stack.
 
-**Predecessors:** [v2 (`…-210ep-v2-finetune`)](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-210ep-v2-finetune) — the current **validated production champion** (`checkpoint-20000`, grasps from poses where v1 failed) — [v4 (`…-417ep-v4-finetune`)](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-417ep-v4-finetune) (radius-5, aggressive curation, eval pending), [v5 (`…-314ep-v5-finetune`)](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-314ep-v5-finetune) (radius-20, eval pending), and [v1 (`…-105ep-v1-finetune`)](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-105ep-v1-finetune). v6's dataset is a **merge of two grab-bottle sets** — the 105-ep set + a new **115-ep "worst-positions" set** (hard poses) — re-curated at **radius 20** with two filtering improvements over prior versions: **(a) frame removal near the grasp moment is now prohibited** (a defect that had slipped through and likely hurt earlier versions' grasp behavior), and **(b) episodes 95–114 were kept as-is** (curation skipped). The result is **482 clean curated segments + 20 as-is = 502 episodes** (20.3% of frames / 29.5% of samples removed elsewhere), ~239 frames/ep — 2.4× the frames of v5.
+**Predecessors:** [v2 (`…-210ep-v2-finetune`)](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-210ep-v2-finetune) — the current **validated production champion** (`checkpoint-20000`, grasps from poses where v1 failed) — [v4 (`…-417ep-v4-finetune`)](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-417ep-v4-finetune) (radius-5, 6/12), [v5 (`…-314ep-v5-finetune`)](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-314ep-v5-finetune) (radius-20, 9/12), and [v1 (`…-105ep-v1-finetune`)](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-105ep-v1-finetune). v6's dataset is a **merge of two grab-bottle sets** — the 105-ep set + a new **115-ep "worst-positions" set** (hard poses) — re-curated at **radius 20** with two filtering improvements over prior versions: **(a) frame removal near the grasp moment is now prohibited** (a defect that had slipped through and likely hurt earlier versions' grasp behavior), and **(b) episodes 95–114 were kept as-is** (curation skipped). The result is **482 clean curated segments + 20 as-is = 502 episodes** (20.3% of frames / 29.5% of samples removed elsewhere), ~239 frames/ep — 2.4× the frames of v5.
 
-This is a behavior-cloning fine-tune of the full 3B model. **Status: trained 2026-06-24, eval pending** — only `checkpoint-20000` is published (one checkpoint per repo, lineage storage policy); the production checkpoint is selected by closed-loop comparison against the v2 champion (see [Evaluation](#evaluation)).
+This is a behavior-cloning fine-tune of the full 3B model. **Status: trained 2026-06-24, closed-loop eval complete (8/12)** — only `checkpoint-20000` is published (one checkpoint per repo, lineage storage policy); the larger merged set underfit and did not beat the v2 champion (see [Evaluation](#evaluation)).
 
 ## Quick facts
 
@@ -82,7 +82,23 @@ DeepSpeed ZeRO partition states, optimizer, scheduler and RNG files are intentio
 
 **Extension to 40k — no longer a true-resume.** A clean continuation from `checkpoint-20000` to 40,000 steps would have produced `checkpoint-30000` (~12 ep) and `checkpoint-40000` (~16 ep), bracketing v2's sweet spot. However, the cluster training state (DeepSpeed/optimizer/scheduler) was cleaned from NFS on 2026-06-24, so a true-resume is **no longer possible** — reaching 40k now requires re-training from scratch (`MAX_STEPS=40000`). The published deploy weights are unaffected (eval/inference unchanged).
 
-**Closed-loop evaluation — TBD.** `checkpoint-20000` will be run closed-loop on the G1 + SONIC stack and compared against the **v2 `checkpoint-20000` champion** (11/12 across hand-placed bottle poses), v4, v5, and the other family members. If it does not match or beat the v2 champion, **v2 remains in production** (a 40k extension would now require a from-scratch re-train — see above). (This section is updated with the verdict once the eval runs.)
+**Closed-loop evaluation — completed (real G1).** `checkpoint-20000` was run closed-loop on the physical G1 + SONIC stack and measured against the **v2 `checkpoint-20000` champion**. Low BC loss is a healthy prerequisite, **not** evidence of a good policy — this closed-loop result is the load-bearing signal.
+
+**Procedure.** The bottle was placed at **12 fixed positions** spanning one half-side of the table (the table is fixed relative to the robot). These are the **same 12 positions used for data collection** (see the [source dataset card](https://huggingface.co/datasets/cloudwalk-research/gr00t-g1-grab-bottle-right-hand-105ep-v1)). **One trial per position** (12 trials) gives the grid result. Legend: 🟥 red = knocked the bottle over · 🟧 orange = stuck in an "indecision" loop, never reaching the bottle · 🟩 green = successfully grasped the bottle.
+
+![Closed-loop eval: 12-position grid, checkpoint-20000](eval_12_positions_20k.png)
+
+**Results — checkpoint-20000.**
+
+Grid (12 fixed positions, 1 trial each):
+
+| Outcome | Count | Rate |
+|---|---|---|
+| 🟩 Success | 8/12 | 67% |
+| 🟧 Non-convergence (indecision loop) | 3/12 | 25% |
+| 🟥 Error (knocked bottle over) | 1/12 | 8% |
+
+**Conclusion.** More data did not beat better data. v6 trains on the largest set (502 episodes / 120,017 frames, 2.4× v5) yet lands below both v2 (10/12) and v5 (9/12). Its bright spot is the tied-lowest knock-over rate (1/12, 8%, tied with v2) — the merged set's worst-position coverage taught it to avoid toppling the bottle — but the highest non-convergence (3/12, 25%) shows the 2.4× larger set at only ~8 epochs *underfit*: the policy often can't commit. The lineage already flagged underfit as the live risk; the closed-loop result confirms it. **v2 remains in production**.
 
 ## How to download
 
@@ -125,11 +141,11 @@ Closed-loop control of the G1 (sim or real) is driven by the **SONIC whole-body
 
 | Version | Dataset | Episodes | Frames | Epochs @20k | Closed-loop | Notes |
 | --- | --- | --- | --- | --- | --- | --- |
-| [v1](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-105ep-v1-finetune) | [105ep-v1](https://huggingface.co/datasets/cloudwalk-research/gr00t-g1-grab-bottle-right-hand-105ep-v1) | 105 | 70,680 | 13.6 | ✅ validated | First GR00T N1.7 + SONIC fine-tune at CW-RL; right-hand bottle pick. |
-| [v2](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-210ep-v2-finetune) | [right-hand-v2](https://huggingface.co/datasets/cloudwalk-research/gr00t-g1-grab-bottle-right-hand-v2) | 210 | 62,772 | 15.3 | ✅ validated (> v1) | Curated (episodes split into shorter windows, bad segments removed). **`checkpoint-20000` = current production champion (11/12).** |
-| [v4](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-417ep-v4-finetune) | [radius-5](https://huggingface.co/datasets/cloudwalk-research/gr00t-g1-grab-bottle-right-hand-zero-wandering-smooth-radius-5) | 417 | 48,577 | 19.8 | ⏳ TBD | Zero-wandering, **most aggressive** curation (radius 5 → short windows). |
-| [v5](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-314ep-v5-finetune) | [radius-20](https://huggingface.co/datasets/cloudwalk-research/gr00t-g1-grab-bottle-right-hand-zero-wandering-smooth-radius-20) | 314 | 50,496 | 19.0 | ⏳ TBD | Zero-wandering, **least aggressive** curation (radius 20 → longer windows, ~161 f/ep). |
-| **v6 (this)** | [radius-20-merged](https://huggingface.co/datasets/cloudwalk-research/gr00t-g1-grab-bottle-right-hand-radius-20-merged) | 502 | 120,017 | 8.0 | ⏳ TBD | **Merged** (105-ep + 115-ep "worst-positions"), radius-20 with grasp-frame preservation. Largest set → 20k = only ~8 ep (underfit risk). |
+| [v1](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-105ep-v1-finetune) | [105ep-v1](https://huggingface.co/datasets/cloudwalk-research/gr00t-g1-grab-bottle-right-hand-105ep-v1) | 105 | 70,680 | 13.6 | ✅ 4/12 (33%) | First GR00T N1.7 + SONIC fine-tune at CW-RL; right-hand bottle pick. |
+| [v2](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-210ep-v2-finetune) | [right-hand-v2](https://huggingface.co/datasets/cloudwalk-research/gr00t-g1-grab-bottle-right-hand-v2) | 210 | 62,772 | 15.3 | ✅ 10/12 (83%) — **champion** | Curated (episodes split into shorter windows, bad segments removed). **`checkpoint-20000` = current production champion (10/12).** |
+| [v4](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-417ep-v4-finetune) | [radius-5](https://huggingface.co/datasets/cloudwalk-research/gr00t-g1-grab-bottle-right-hand-zero-wandering-smooth-radius-5) | 417 | 48,577 | 19.8 | ✅ 6/12 (50%) | Zero-wandering, **most aggressive** curation (radius 5 → short windows). |
+| [v5](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-314ep-v5-finetune) | [radius-20](https://huggingface.co/datasets/cloudwalk-research/gr00t-g1-grab-bottle-right-hand-zero-wandering-smooth-radius-20) | 314 | 50,496 | 19.0 | ✅ 9/12 (75%) | Zero-wandering, **least aggressive** curation (radius 20 → longer windows, ~161 f/ep). |
+| **v6 (this)** | [radius-20-merged](https://huggingface.co/datasets/cloudwalk-research/gr00t-g1-grab-bottle-right-hand-radius-20-merged) | 502 | 120,017 | 8.0 | ✅ 8/12 (67%) | **Merged** (105-ep + 115-ep "worst-positions"), radius-20 with grasp-frame preservation. Largest set → 20k = only ~8 ep (underfit risk). |
 | [v7](https://huggingface.co/cloudwalk-research/gr00t-n17-g1-grab-bottle-rh-220ep-v7-finetune) | [speedup-3mm-v1](https://huggingface.co/datasets/cloudwalk-research/gr00t-g1-grab-bottle-right-hand-speedup-3mm-v1) | 220 | 60,163 | ~16.0 | ⏳ TBD | **DP speedup resampling** (wrist-Cartesian, 3 mm/frame, max_K=40) of the merged set; **no segment removal** — reference baseline. ≈16 ep @20k lands on v2's sweet spot. |
 
 ## References