---
library_name: lerobot
pipeline_tag: robotics
license: apache-2.0
base_model: lerobot/pi05_base
datasets:
  - TrossenRoboticsCommunity/stationary-block-transfer-lerobot-v3
widget:
  - text: "Grab and handover the red cube to the other arm"
    output:
      url: media/eval_demo.mp4
tags:
  - lerobot
  - pi05
  - robotics
  - lora
  - relative-actions
  - bimanual
  - widowxai
  - model:pi05
  - task:block-transfer
  - framework:lerobot-latest
  - robot:stationary
  - env:local-5090
  - study:RLE-2
---

# pi05-block-transfer-lerobot

π₀.₅ fine-tune for bimanual red-cube handover / block transfer on the Trossen AI
**stationary** (WidowX AI) platform. Trained with **lerobot** (LoRA, **relative actions**).
This Run Record documents the **step-40000** checkpoint.

## Demo — on-robot eval

<Gallery />

Async on-robot evaluation of this checkpoint (step-40000) on the Trossen AI stationary
bimanual platform — block transfer / red-cube handover. See [RLE-34](https://trossenrobotics.atlassian.net/browse/RLE-34).

## TL;DR

- **Policy:** π₀.₅ — PaliGemma `gemma_2b` VLM + `gemma_300m` action expert (flow matching).
- **Framework:** lerobot-latest (`--policy.type=pi05`).
- **Method:** LoRA (r=32, α=32) on attention + MLP projections of both the VLM language
  model and the action expert; base = [`lerobot/pi05_base`](https://huggingface.co/lerobot/pi05_base).
- **Action space:** **relative/delta** on the 12 arm joints, **absolute** on the 2 grippers.
- **Robot:** Trossen AI stationary bimanual, 14-DoF, 4 cameras, 30 fps.
- **Checkpoint:** step 40000.

## How it differs from the openpi fine-tune

| Repo | Framework | Action space | Method |
|---|---|---|---|
| **pi05-block-transfer-lerobot** (this) | lerobot-latest | **relative** (arms) | **LoRA** |
| openpi `pi05_trossen_transfer_block` | openpi | absolute | LoRA |

## How it was trained (relative actions)

Relative actions are implemented as processor steps baked into the saved pipeline (not a
model flag), so evaluation handles them automatically:

- **Preprocessing** — `delta_actions_processor` (`enabled: true`): `action -= state`.
- **Postprocessing** — `absolute_actions_processor` (`enabled: true`): `action += state`.
- `exclude_joints: [left_carriage_joint, right_carriage_joint]` → the two **grippers stay
  absolute**; the 12 arm joints are relative.

The server/loader re-wires the paired steps on load (`_reconnect_relative_absolute_steps`),
so no special flag is needed at inference.

## Model configuration (`config.json`)

| Field | Value |
|---|---|
| type | `pi05` |
| paligemma_variant / action_expert_variant | `gemma_2b` / `gemma_300m` |
| dtype | `bfloat16` |
| input_features | `observation.state` (14); images `cam_high`, `cam_low`, `cam_left_wrist`, `cam_right_wrist` (3×480×640) |
| output_features | `action` (14) |
| chunk_size / n_action_steps | 50 / 50 |
| num_inference_steps | 10 |
| n_obs_steps | 1 |
| max_state_dim / max_action_dim | 32 / 32 |
| image_resolution | 224 × 224 |
| tokenizer_max_length | 200 |
| normalization | STATE=QUANTILES, ACTION=QUANTILES, VISUAL=IDENTITY |
| flow time-sampling | beta(α=1.5, β=1.0), scale=0.999, offset=0.001 |
| period (min/max) | 0.004 / 4.0 |

## Training hyperparameters

| Field | Value |
|---|---|
| optimizer | AdamW |
| learning rate | 2.5e-5 |
| betas | (0.9, 0.95) |
| eps | 1e-8 |
| weight_decay | 0.01 |
| grad_clip_norm | 1.0 |
| scheduler | cosine — warmup 1000, decay_steps 30000, decay_lr 2.5e-6 |
| gradient_checkpointing | false |

### LoRA (`adapter_config.json`)

| Field | Value |
|---|---|
| peft_type / version | LORA / 0.19.1 |
| r / lora_alpha / lora_dropout | 32 / 32 / 0.0 |
| bias | none |
| target_modules | `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj` on PaliGemma `language_model` + `gemma_expert` |
| modules_to_save | `state_proj, action_in_proj, action_out_proj, time_mlp_in, time_mlp_out` |
| base_model | `lerobot/pi05_base` |

> `model.safetensors` in this repo is the **full merged model** (base + adapter, 7.9 GB);
> the `adapter_*` files are also included for reference. `from_pretrained` loads the merged
> weights directly — no separate base download needed.

## Inputs (reproducibility manifest)

- **Dataset:** [`TrossenRoboticsCommunity/stationary-block-transfer-lerobot-v3`](https://huggingface.co/datasets/TrossenRoboticsCommunity/stationary-block-transfer-lerobot-v3) @ `v3.0`.
- **Base model:** `lerobot/pi05_base`.
- **Framework:** lerobot-latest (`--policy.type=pi05`).
- **Batch size:** 8.
- **Total steps:** 40,000 (this is the final checkpoint).
- **Training env / hardware:** local-5090 (run on a local RTX 5090 using the Trossen cloud pipeline).

## Outputs

- Weights: `model.safetensors` (merged), `adapter_model.safetensors` (LoRA).
- Config: `config.json`, `adapter_config.json`.
- Processors: `policy_preprocessor.json` (+ normalizer state), `policy_postprocessor.json` (+ unnormalizer state).
- TensorBoard logs: **not available** with this checkpoint (RLE DoD item — add if recoverable).

## Evaluation (async inference)

Verified to load and run via the async policy server + Trossen async client. Relative→absolute
conversion confirmed active. Run from the `lerobot_trossen` workspace:

```bash
# Terminal A — policy server
uv run python -m lerobot.async_inference.policy_server \
  --host=127.0.0.1 --port=8080 --fps=30 --inference_latency=0.033 --obs_queue_timeout=2

# Terminal B — robot client
uv run lerobot-trossen-async-client \
  --server_address=127.0.0.1:8080 \
  --robot.type=bi_widowxai_follower_robot \
  --robot.left_arm_ip_address=192.168.1.5 --robot.right_arm_ip_address=192.168.1.4 \
  --robot.id=bimanual_follower \
  --robot.cameras='{ cam_high: {...}, cam_low: {...}, cam_left_wrist: {...}, cam_right_wrist: {...} }' \
  --task="Grab and handover the red cube to the other arm" \
  --policy_type=pi05 \
  --pretrained_name_or_path=TrossenRoboticsCommunity/pi05-block-transfer-lerobot \
  --policy_device=cuda \
  --actions_per_chunk=50 --chunk_size_threshold=0.5 --aggregate_fn_name=weighted_average
```

The task prompt must match training (π₀.₅ is language-conditioned).

### Eval Results

| Eval Task | Reps | Success | Notes |
|---|---|---|---|
| block-transfer | TODO | TODO | policy confirmed working on hardware; formal rep count TODO |

## Links

- **RLE Study (Epic):** [RLE-2](https://trossenrobotics.atlassian.net/browse/RLE-2) — Cloud pi0.5 Pipeline Validation
- **RLE Training Run (Task):** RLE-33 (TODO: confirm after creation)
- **RLE Evaluation Run (Task):** RLE-34 (TODO: confirm after creation)
- **Dataset:** https://huggingface.co/datasets/TrossenRoboticsCommunity/stationary-block-transfer-lerobot-v3
- **Base model:** https://huggingface.co/lerobot/pi05_base