---
license: apache-2.0
library_name: lerobot
pipeline_tag: robotics
tags:
- act-policy
- pytorch_model_hub_mixin
- robotics
- IB-Robot
- LeRobot
- so-101
- banana-pick
- knowledge-distillation
- rknn
- ascend
- edge-deployment
---

# Model Card for ACT/BananaPick (Distilled, Multi-Backend)

Action Chunking Transformer Policy (as per [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://arxiv.org/abs/2304.13705)) trained for banana pick-and-place on a 1-arm SO-101 robot. This checkpoint is a **distilled** model obtained via knowledge distillation.

This repo bundles **multiple deployment backends** so the same policy can run on GPU/CPU (PyTorch), Huawei Ascend (OM), and Rockchip RK3588 (RKNN) without per-backend glue code. All backends are described by a single [`inference_manifest.json`](./inference_manifest.json) routing table.

## Repository Structure

```
├── config.json                                                 # torch 权重配置
├── model.safetensors                                           # torch 权重 (340 MB)
├── policy_preprocessor.json
├── policy_postprocessor.json
├── policy_preprocessor_step_3_normalizer_processor.safetensors
├── policy_postprocessor_step_0_unnormalizer_processor.safetensors
├── train_config.json
├── inference_manifest.json                                     # 部署路由表 (权威)
└── artifacts/
    ├── ascend/ascend_310b1/policy-1031c8ca5f1e.om              # Ascend OM (123 MB)
    └── rknn/rk3588/policy-cfcf62db3da4.rknn                    # Rockchip RKNN (146 MB)
```

> The directory layout and every file path inside `inference_manifest.json` must stay in sync. Do not rename or relocate the weight/artifact files — the manifest's sha256 checks and path bindings depend on them.

## Deployment Backends

Read `inference_manifest.json` → `deployments[<target>]` to route to the right backend. Each entry carries the artifact path, runtime, SoC, and full input/output tensor bindings (shape, dtype, semantic name).

| Target | Backend | Runtime | Artifact | Hardware |
|--------|---------|---------|----------|----------|
| `torch-cuda` | torch | PyTorch | `model.safetensors` (in bundle) | NVIDIA GPU |
| `cpu` | torch | PyTorch | `model.safetensors` (in bundle) | CPU |
| `torch-npu` | torch | PyTorch | `model.safetensors` (in bundle) | Huawei NPU |
| `ascend_310b1` | ascend | ACL | `artifacts/ascend/ascend_310b1/policy-*.om` | Ascend 310B1 |
| `rk3588` | rknn | rknn-lite2 | `artifacts/rknn/rk3588/policy-*.rknn` | Rockchip RK3588 |

**Input tensors** (all backends): `observation.state` [1,6] float32, `observation.images.top` [1,3,480,640] NCHW, `observation.images.wrist` [1,3,480,640] NCHW.
**Output tensor**: `action` [1,100,6] float32 (100-step action chunk, 6-dim).

## How to Get Started with the Model

See the [IB-Robot project](https://gitcode.com/openeuler/IB_Robot) (particularly the [inference_service](https://gitcode.com/openeuler/IB_Robot/tree/main/src/inference_service)) for instructions on how to load and deploy this model with ROS 2.

To load the **PyTorch** backend directly in Python (weights are at the repo root):

```python
from lerobot.common.policies.act.modeling_act import ACTPolicy

policy = ACTPolicy.from_pretrained("openEuler/IB_Robot_ACT_banana_pick_distill")
```

For **Ascend (OM)** or **RKNN** deployment, consume `inference_manifest.json` and load the artifact under the matching `deployments[<target>].artifacts.policy.path`.

## Training Details

This model was trained via **knowledge distillation** (`kd: true`) within the IB-Robot framework. The OM/RKNN artifacts were converted from these torch weights (verified same-origin by sha256).

- **Policy:** ACT (Action Chunking with Transformers)
- **Training method:** Knowledge distillation
- **Robot:** 1-arm SO-101
- **Cameras:** top, wrist (480x640)
- **Action dim:** 6
- **Training steps:** 500,000 (configured)
- **Checkpoint:** step_160000
- **Chunk size:** 100
- **Batch size:** 32
- **Vision backbone:** ResNet18
- **Optimizer:** AdamW (lr=1e-5, weight_decay=1e-4)
- **Latent dim (VAE):** 32
- **Dim model:** 1024

## Citation

```bibtex
@software{ib_robot,
  title = {IB-Robot: Intelligence Boom Robot},
  url = {https://gitcode.com/openeuler/IB_Robot},
  license = {Apache-2.0}
}
```