Document ZeRO-2 optimizer resume layout

README.md

@@ -1,53 +1,22 @@
 ---
 license: other
-tags:
-- vggt-omega
-- robotics
-- oxe
-- mimicgen
-- robocasa365
-- checkpoint
 ---
 
-# VGGT-Omega 1B 256 OXE+MimicGen+RoboCasa365 Step 62000
+# VGGT-Omega 1B 256 OXE/MimicGen/RoboCasa365 Step 62000
 
-Latest consolidated checkpoint from the running CVLAB1 DA3/VGGT-Omega robot training run.
+This repo contains the portable model checkpoint `0062000.pt` and the DeepSpeed ZeRO-2 checkpoint shards needed for optimizer-state resume.
 
-## Files
+Important: because this run used DeepSpeed ZeRO-2, `0062000.pt` intentionally stores `optimizer: null`. Optimizer state lives in `checkpoints/0062000/`.
 
-- `0062000.pt`: consolidated PyTorch checkpoint
-- `config.yaml`: resolved training config saved with the run
-
-## Run
-
-- W&B run id: `8gxiadnc`
-- W&B name: `[VGGTOMEGA]_hf23000_H4_cvlab1_8gpu_mb28acc1_zero2_ckptfirst_evalmb1_eval1000_20260522_003836`
-- Local run dir: `logs/train/[VGGTOMEGA]_resume8gxiadnc_officialteacher_ckpt34000_H4_cvlab1_8gpu_mb28acc1_zero2_eval1000_20260522_124402/robot-20260522-124541`
-- Source/resume checkpoint: `.local/deepspeed_ckpt_aliases/hf23000_8gxiadnc_ckpt34000_checkpoints/0034000.pt`
-- Teacher checkpoint: `checkpoints/vggt_omega/vggt_omega_1b_256_text.pt`
-
-## Training Setup
-
-- Backbone: VGGT-Omega 1B, 256 input
-- History: H=4 only
-- `predictor.deep_temporal_causal_mask=true`
-- Batch: 8 GPUs, micro batch 28/GPU, global batch 224, grad accumulation 1
-- DeepSpeed: ZeRO-2
-- Eval/checkpoint interval: 1000 steps
-- Dense head: frozen, teacher depth from pristine VGGT-Omega teacher checkpoint
-
-## Metrics At Step 62000
-
-Train log:
-
-```text
-[step=0062000] total=0.3498 action_l1=0.0664 feat=0.0459 depth=0.0349 ray=0.0000 point=0.0000 rel@1/5/10=0.434/0.788/0.902 mse=0.02163 s/step=3.1 lr=5.2e-05/5.2e-04
-```
-
-Unified eval:
+To resume with optimizer/scheduler state in this codebase, keep the sibling layout:
 
 ```text
-[step=0062000] EVAL-UNIFIED H=4 l1_norm=0.1933 mse_norm=0.10018 l1_raw=0.0468 mse_raw=0.01675 r2_norm=0.386 rel@5/10=0.501/0.674 (104 samples, 104 batches)
+0062000.pt
+checkpoints/0062000/
+  bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
+  ...
+  bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
+  mp_rank_00_model_states.pt
 ```
 
-Uploaded from CVLAB1 on 2026-05-23.
+Then pass `--ckpt /path/to/0062000.pt` with the `checkpoints/0062000/` directory present; `train_robot.py` resolves the sibling ZeRO folder and calls DeepSpeed `load_checkpoint(tag="0062000")`.