Add pi05 block-transfer (lerobot, LoRA, relative actions) step-40000 + Run Record README

README.md

@@ -0,0 +1,167 @@
+---
+library_name: lerobot
+pipeline_tag: robotics
+license: apache-2.0
+base_model: lerobot/pi05_base
+tags:
+  - lerobot
+  - pi05
+  - robotics
+  - lora
+  - relative-actions
+  - bimanual
+  - widowxai
+  - model:pi05
+  - task:block-transfer
+  - framework:lerobot-latest
+  - robot:stationary
+  # - env:local-5090            # TODO: set local-5090 | cloud-runpod
+  # - study:RLE-NN              # TODO: set the RLE Study Epic number
+---
+
+# pi05-block-transfer-lerobot
+
+π₀.₅ fine-tune for bimanual red-cube handover / block transfer on the Trossen AI
+**stationary** (WidowX AI) platform. Trained with **lerobot** (LoRA, **relative actions**).
+This Run Record documents the **step-40000** checkpoint.
+
+> **Framework: lerobot-latest.** This is the **lerobot**-trained variant — distinct from the
+> openpi `pi05_trossen_transfer_block` fine-tune. RLE tracks framework as a first-class axis
+> (`framework:lerobot-latest` tag), so this is intentionally not conflated with the openpi model.
+> Method (LoRA + relative actions) lives in the tags and the sections below, not the repo name.
+
+> RLE Run Record. The Jira Run ticket is the project-management view; **this README is the
+> artifact-of-record**. Naming follows the RLE convention `<model>-<task>-<run-tag>`
+> (`pi05` · `block-transfer` · `lerobot`).
+
+## TL;DR
+
+- **Policy:** π₀.₅ — PaliGemma `gemma_2b` VLM + `gemma_300m` action expert (flow matching).
+- **Framework:** lerobot-latest (`--policy.type=pi05`).
+- **Method:** LoRA (r=32, α=32) on attention + MLP projections of both the VLM language
+  model and the action expert; base = [`lerobot/pi05_base`](https://huggingface.co/lerobot/pi05_base).
+- **Action space:** **relative/delta** on the 12 arm joints, **absolute** on the 2 grippers.
+- **Robot:** Trossen AI stationary bimanual, 14-DoF, 4 cameras, 30 fps.
+- **Checkpoint:** step 40000.
+
+## How it differs from the openpi fine-tune
+
+| Repo | Framework | Action space | Method |
+|---|---|---|---|
+| **pi05-block-transfer-lerobot** (this) | lerobot-latest | **relative** (arms) | **LoRA** |
+| openpi `pi05_trossen_transfer_block` | openpi | absolute | LoRA |
+
+## How it was trained (relative actions)
+
+Relative actions are implemented as processor steps baked into the saved pipeline (not a
+model flag), so evaluation handles them automatically:
+
+- **Preprocessing** — `delta_actions_processor` (`enabled: true`): `action -= state`.
+- **Postprocessing** — `absolute_actions_processor` (`enabled: true`): `action += state`.
+- `exclude_joints: [left_carriage_joint, right_carriage_joint]` → the two **grippers stay
+  absolute**; the 12 arm joints are relative.
+
+The server/loader re-wires the paired steps on load (`_reconnect_relative_absolute_steps`),
+so no special flag is needed at inference.
+
+## Model configuration (`config.json`)
+
+| Field | Value |
+|---|---|
+| type | `pi05` |
+| paligemma_variant / action_expert_variant | `gemma_2b` / `gemma_300m` |
+| dtype | `bfloat16` |
+| input_features | `observation.state` (14); images `cam_high`, `cam_low`, `cam_left_wrist`, `cam_right_wrist` (3×480×640) |
+| output_features | `action` (14) |
+| chunk_size / n_action_steps | 50 / 50 |
+| num_inference_steps | 10 |
+| n_obs_steps | 1 |
+| max_state_dim / max_action_dim | 32 / 32 |
+| image_resolution | 224 × 224 |
+| tokenizer_max_length | 200 |
+| normalization | STATE=QUANTILES, ACTION=QUANTILES, VISUAL=IDENTITY |
+| flow time-sampling | beta(α=1.5, β=1.0), scale=0.999, offset=0.001 |
+| period (min/max) | 0.004 / 4.0 |
+
+## Training hyperparameters
+
+| Field | Value |
+|---|---|
+| optimizer | AdamW |
+| learning rate | 2.5e-5 |
+| betas | (0.9, 0.95) |
+| eps | 1e-8 |
+| weight_decay | 0.01 |
+| grad_clip_norm | 1.0 |
+| scheduler | cosine — warmup 1000, decay_steps 30000, decay_lr 2.5e-6 |
+| gradient_checkpointing | false |
+
+### LoRA (`adapter_config.json`)
+
+| Field | Value |
+|---|---|
+| peft_type / version | LORA / 0.19.1 |
+| r / lora_alpha / lora_dropout | 32 / 32 / 0.0 |
+| bias | none |
+| target_modules | `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj` on PaliGemma `language_model` + `gemma_expert` |
+| modules_to_save | `state_proj, action_in_proj, action_out_proj, time_mlp_in, time_mlp_out` |
+| base_model | `lerobot/pi05_base` |
+
+> `model.safetensors` in this repo is the **full merged model** (base + adapter, 7.9 GB);
+> the `adapter_*` files are also included for reference. `from_pretrained` loads the merged
+> weights directly — no separate base download needed.
+
+## Inputs (reproducibility manifest)
+
+- **Dataset:** `TrossenRoboticsCommunity/<TODO>` @ `<revision>` — TODO (candidate:
+  stationary block-transfer / handover-cube lerobot v3).
+- **Base model:** `lerobot/pi05_base`.
+- **Framework:** lerobot-latest (`--policy.type=pi05`).
+- **Batch size:** TODO.
+- **Total steps:** TODO (checkpoints exist at 40k/60k/80k; this record is **40k**).
+- **Training env / hardware:** TODO (local-5090 | cloud-runpod).
+
+## Outputs
+
+- Weights: `model.safetensors` (merged), `adapter_model.safetensors` (LoRA).
+- Config: `config.json`, `adapter_config.json`.
+- Processors: `policy_preprocessor.json` (+ normalizer state), `policy_postprocessor.json` (+ unnormalizer state).
+- TensorBoard logs: **not available** with this checkpoint (RLE DoD item — add if recoverable).
+
+## Evaluation (async inference)
+
+Verified to load and run via the async policy server + Trossen async client. Relative→absolute
+conversion confirmed active. Run from the `lerobot_trossen` workspace:
+
+```bash
+# Terminal A — policy server
+uv run python -m lerobot.async_inference.policy_server \
+  --host=127.0.0.1 --port=8080 --fps=30 --inference_latency=0.033 --obs_queue_timeout=2
+
+# Terminal B — robot client
+uv run lerobot-trossen-async-client \
+  --server_address=127.0.0.1:8080 \
+  --robot.type=bi_widowxai_follower_robot \
+  --robot.left_arm_ip_address=192.168.1.5 --robot.right_arm_ip_address=192.168.1.4 \
+  --robot.id=bimanual_follower \
+  --robot.cameras='{ cam_high: {...}, cam_low: {...}, cam_left_wrist: {...}, cam_right_wrist: {...} }' \
+  --task="Grab and handover the red cube to the other arm" \
+  --policy_type=pi05 \
+  --pretrained_name_or_path=TrossenRoboticsCommunity/pi05-block-transfer-lerobot \
+  --policy_device=cuda \
+  --actions_per_chunk=50 --chunk_size_threshold=0.5 --aggregate_fn_name=weighted_average
+```
+
+The task prompt must match training (π₀.₅ is language-conditioned).
+
+### Eval Results
+
+| Eval Task | Reps | Success | Notes |
+|---|---|---|---|
+| block-transfer | TODO | TODO | policy confirmed working on hardware; formal rep count TODO |
+
+## Links
+
+- **RLE Study (Epic):** TODO — `RLE-NN`
+- **RLE Run (Task):** TODO — `RLE-NN`
+- **Base model:** https://huggingface.co/lerobot/pi05_base

adapter_config.json

@@ -0,0 +1,49 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "PI05Policy",
+    "parent_library": "lerobot.policies.pi05.modeling_pi05"
+  },
+  "base_model_name_or_path": "/home/trossen/.cache/huggingface/hub/models--lerobot--pi05_base/snapshots/9e55186ad36e66b95cda57bc47818d9e6237ae30",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [
+    "state_proj",
+    "action_in_proj",
+    "action_out_proj",
+    "time_mlp_in",
+    "time_mlp_out"
+  ],
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": "model\\.paligemma_with_expert\\.(paligemma\\.model\\.language_model|gemma_expert\\.model)\\..*\\.(q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)$",
+  "target_parameters": null,
+  "task_type": null,
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b20e355e3899370c142c9557801b3cb64b79b30781c172026b4ef4184344f3e
+size 221091648

config.json

@@ -0,0 +1,99 @@
+{
+    "type": "pi05",
+    "n_obs_steps": 1,
+    "input_features": {
+        "observation.state": {
+            "type": "STATE",
+            "shape": [
+                14
+            ]
+        },
+        "observation.images.cam_high": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                480,
+                640
+            ]
+        },
+        "observation.images.cam_low": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                480,
+                640
+            ]
+        },
+        "observation.images.cam_left_wrist": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                480,
+                640
+            ]
+        },
+        "observation.images.cam_right_wrist": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                480,
+                640
+            ]
+        }
+    },
+    "output_features": {
+        "action": {
+            "type": "ACTION",
+            "shape": [
+                14
+            ]
+        }
+    },
+    "device": "cuda",
+    "use_amp": false,
+    "push_to_hub": true,
+    "repo_id": null,
+    "private": null,
+    "tags": null,
+    "license": null,
+    "pretrained_path": null,
+    "paligemma_variant": "gemma_2b",
+    "action_expert_variant": "gemma_300m",
+    "dtype": "bfloat16",
+    "chunk_size": 50,
+    "n_action_steps": 50,
+    "max_state_dim": 32,
+    "max_action_dim": 32,
+    "num_inference_steps": 10,
+    "time_sampling_beta_alpha": 1.5,
+    "time_sampling_beta_beta": 1.0,
+    "time_sampling_scale": 0.999,
+    "time_sampling_offset": 0.001,
+    "min_period": 0.004,
+    "max_period": 4.0,
+    "image_resolution": [
+        224,
+        224
+    ],
+    "empty_cameras": 0,
+    "tokenizer_max_length": 200,
+    "normalization_mapping": {
+        "VISUAL": "IDENTITY",
+        "STATE": "QUANTILES",
+        "ACTION": "QUANTILES"
+    },
+    "gradient_checkpointing": false,
+    "compile_model": false,
+    "compile_mode": "max-autotune",
+    "optimizer_lr": 2.5e-05,
+    "optimizer_betas": [
+        0.9,
+        0.95
+    ],
+    "optimizer_eps": 1e-08,
+    "optimizer_weight_decay": 0.01,
+    "optimizer_grad_clip_norm": 1.0,
+    "scheduler_warmup_steps": 1000,
+    "scheduler_decay_steps": 30000,
+    "scheduler_decay_lr": 2.5e-06
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b8aef81cb02ebf3e88a67c4375d353af6d4fa62f5968bd9e1794ce59f0196a9
+size 8286949888

policy_postprocessor.json

@@ -0,0 +1,38 @@
+{
+  "name": "policy_postprocessor",
+  "steps": [
+    {
+      "registry_name": "unnormalizer_processor",
+      "config": {
+        "eps": 1e-08,
+        "features": {
+          "action": {
+            "type": "ACTION",
+            "shape": [
+              14
+            ]
+          }
+        },
+        "norm_map": {
+          "VISUAL": "IDENTITY",
+          "STATE": "QUANTILES",
+          "ACTION": "QUANTILES"
+        }
+      },
+      "state_file": "policy_postprocessor_step_0_unnormalizer_processor.safetensors"
+    },
+    {
+      "registry_name": "absolute_actions_processor",
+      "config": {
+        "enabled": true
+      }
+    },
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cpu",
+        "float_dtype": null
+      }
+    }
+  ]
+}

policy_postprocessor_step_0_unnormalizer_processor.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6106123fd38231752b33178f74f8fe3440aa533d35ffc2b495a0fb7924313f8
+size 10448

policy_preprocessor.json

@@ -0,0 +1,121 @@
+{
+  "name": "policy_preprocessor",
+  "steps": [
+    {
+      "registry_name": "rename_observations_processor",
+      "config": {
+        "rename_map": {}
+      }
+    },
+    {
+      "registry_name": "to_batch_processor",
+      "config": {}
+    },
+    {
+      "registry_name": "delta_actions_processor",
+      "config": {
+        "enabled": true,
+        "exclude_joints": [
+          "left_carriage_joint",
+          "right_carriage_joint"
+        ],
+        "action_names": [
+          "left_joint_0.pos",
+          "left_joint_1.pos",
+          "left_joint_2.pos",
+          "left_joint_3.pos",
+          "left_joint_4.pos",
+          "left_joint_5.pos",
+          "left_left_carriage_joint.pos",
+          "right_joint_0.pos",
+          "right_joint_1.pos",
+          "right_joint_2.pos",
+          "right_joint_3.pos",
+          "right_joint_4.pos",
+          "right_joint_5.pos",
+          "right_left_carriage_joint.pos"
+        ]
+      }
+    },
+    {
+      "registry_name": "normalizer_processor",
+      "config": {
+        "eps": 1e-08,
+        "features": {
+          "observation.state": {
+            "type": "STATE",
+            "shape": [
+              14
+            ]
+          },
+          "observation.images.cam_high": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              480,
+              640
+            ]
+          },
+          "observation.images.cam_low": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              480,
+              640
+            ]
+          },
+          "observation.images.cam_left_wrist": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              480,
+              640
+            ]
+          },
+          "observation.images.cam_right_wrist": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              480,
+              640
+            ]
+          },
+          "action": {
+            "type": "ACTION",
+            "shape": [
+              14
+            ]
+          }
+        },
+        "norm_map": {
+          "VISUAL": "IDENTITY",
+          "STATE": "QUANTILES",
+          "ACTION": "QUANTILES"
+        }
+      },
+      "state_file": "policy_preprocessor_step_3_normalizer_processor.safetensors"
+    },
+    {
+      "registry_name": "pi05_prepare_state_tokenizer_processor_step",
+      "config": {}
+    },
+    {
+      "registry_name": "tokenizer_processor",
+      "config": {
+        "max_length": 200,
+        "task_key": "task",
+        "padding_side": "right",
+        "padding": "max_length",
+        "truncation": true,
+        "tokenizer_name": "google/paligemma-3b-pt-224"
+      }
+    },
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cuda",
+        "float_dtype": null
+      }
+    }
+  ]
+}

policy_preprocessor_step_3_normalizer_processor.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6106123fd38231752b33178f74f8fe3440aa533d35ffc2b495a0fb7924313f8
+size 10448