Add model card

README.md

@@ -1,3 +1,239 @@
 ---
-license: gpl-3.0
+library_name: lerobot
+pipeline_tag: robotics
+license: cc-by-sa-4.0
+base_model: lerobot/smolvla_base
+tags:
+  - robotics
+  - lerobot
+  - so101
+  - so-101
+  - vision-language-action
+  - imitation-learning
+  - smolvla
+  - flow-matching
+datasets:
+  - Project-IRA/TPSoSe2026_Dataset_Full_Merged_Final_LeRobot_SO101_V1
 ---
+
+# SmolVLA SO-101 Multi-Task — V6 Full
+
+SmolVLA (~450M) fine-tuned on all four tasks, with the vision encoder unfrozen and image augmentation enabled. The best SmolVLA model in the project — though still meaningfully behind Pi0.5.
+
+
+
+Part of **[Project-IRA](https://huggingface.co/Project-IRA)** — Interactive Robotic Arm.
+Code: https://github.com/Project-IRA/interactive-robotic-arm
+
+| | |
+|---|---|
+| Base model | [`lerobot/smolvla_base`](https://huggingface.co/lerobot/smolvla_base) |
+| Robot | SO-101 follower (6-DOF) |
+| Training data | [`Project-IRA/TPSoSe2026_Dataset_Full_Merged_Final_LeRobot_SO101_V1`](https://huggingface.co/datasets/Project-IRA/TPSoSe2026_Dataset_Full_Merged_Final_LeRobot_SO101_V1) |
+| Recommended checkpoint | around 150000 (**not systematically tested**) |
+| Inputs | `camera1` (wrist) + `camera2` (desk) images — **renamed keys, see Usage** — 6-dim joint state, English instruction |
+| Outputs | 6-dim continuous action chunks |
+
+## Quality
+
+**OK-ish.** Roughly the same quality as the single-task [V2 Lego](https://huggingface.co/Project-IRA/TPSoSe2026_SmolVLA_LeRobot_SO101_Finetuning_V2_Lego) model, but across *most* tasks rather than just Lego — so the multi-task generalisation worked, at the same per-task quality level.
+
+Quality was assessed around checkpoint **150000**, but checkpoints were **not compared systematically** for this run — 150000 is a rough indication, not a validated optimum. Checkpoints exist every 10000 steps to 200000 plus `last`, so it is worth trying several.
+
+For comparison, Pi0.5 on the same dataset works *really well* at checkpoint 8000. SmolVLA at ~450M parameters appears capacity-limited for this four-task set.
+
+## Training
+
+SLURM job 2164957. **Single L40S GPU.**
+
+| Setting | Value |
+|---|---|
+| Base | `lerobot/smolvla_base` |
+| Dataset | 930-episode merged set |
+| Steps | **200000, completed** (`--save_freq=10000`); assessed around 150000 |
+| Batch size | 64 |
+| Vision encoder | **unfrozen** (`--policy.freeze_vision_encoder=false`) |
+| Image augmentation | **on** (`--dataset.image_transforms.enable=true`) |
+| AMP | **on** (`--policy.use_amp=true`) |
+| Parameters | 99,880,992 trainable of 450,046,176 total |
+| Camera keys | **renamed** — `wrist_left`->`camera1`, `desk_view`->`camera2` |
+| SLURM | `--gres=gpu:L40S:1 --cpus-per-task=16 --mem=92G --time=52:00:00` |
+
+```bash
+lerobot-train \
+    --policy.path=lerobot/smolvla_base \
+    --dataset.repo_id=TPSoSe2026_Dataset_Full_Merged_Final_LeRobot_SO101 \
+    --dataset.image_transforms.enable=true \
+    --policy.freeze_vision_encoder=false \
+    --policy.use_amp=true \
+    --batch_size=64 --steps=200000 --save_freq=10000 \
+    --num_workers=16 --tolerance_s=0.01 \
+    --output_dir=outputs/train/smolvla_full_v2 \
+    --job_name=smolvla_full_v2 \
+    --policy.device=cuda --policy.push_to_hub=false --wandb.enable=false \
+    --rename_map='{"observation.images.wrist_left": "observation.images.camera1", "observation.images.desk_view": "observation.images.camera2"}'
+```
+
+### What changed vs V5
+
+V5 used SmolVLA's defaults, where **the vision encoder is frozen** — the model's "eyes"
+stayed locked to their pretrained state and could not adapt to our bricks, lighting and
+camera angles. V6 unfroze it, added image augmentation and AMP, and doubled the step
+count from 100k to 200k.
+
+### A caveat on `freeze_vision_encoder=false`
+
+Unfreezing raised trainable parameters only modestly: the run logs report
+`num_learnable_params=99880992` of `num_total_params=450046176` — the **same ~100M as the
+frozen V5 run**. SmolVLA is a SmolVLM2-500M backbone plus a smaller action expert, and
+most of the backbone stays frozen regardless of this flag. Do not expect this setting
+alone to make all 450M parameters trainable.
+
+## Usage
+
+> [!CAUTION]
+> **This model expects renamed camera keys.** Training used
+> `--rename_map` to remap the dataset's camera features:
+>
+> | Dataset feature | What the policy expects | Physical camera |
+> |---|---|---|
+> | `observation.images.wrist_left` | `observation.images.camera1` | wrist |
+> | `observation.images.desk_view`  | `observation.images.camera2` | desk |
+>
+> If you feed this policy `wrist_left` / `desk_view` it will fail or silently misbehave.
+> Name your cameras `camera1` (wrist) and `camera2` (desk) at inference time, or apply the
+> same `--rename_map` when re-training. **The Pi0.5 models do not do this** — they use the
+> native `wrist_left` / `desk_view` names.
+
+> [!IMPORTANT]
+> **Model files are nested under `outputs_V6/`**, so `from_pretrained` on the repo ID
+> will not work:
+>
+> ```
+> outputs_V6/train/smolvla_full_v2/checkpoints/<step>/pretrained_model/
+> ```
+>
+> **Checkpoints present:** every 10000 steps from `010000` to `200000`, plus `last`.
+> Repo total ~27.7 GB.
+>
+> ```bash
+> hf download Project-IRA/TPSoSe2026_SmolVLA_LeRobot_SO101_Finetuning_V6_Full \
+>   --include 'outputs_V6/train/smolvla_full_v2/checkpoints/150000/pretrained_model/*' \
+>   --local-dir ./smolvla_v6
+> ```
+
+```python
+import torch
+from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy
+
+policy = SmolVLAPolicy.from_pretrained("Project-IRA/TPSoSe2026_SmolVLA_LeRobot_SO101_Finetuning_V6_Full")
+policy = policy.to("cuda").eval()
+```
+
+On-robot rollout:
+
+```bash
+lerobot-record \
+  --robot.type=so101_follower \
+  --robot.port=/dev/ttyACM0 \
+  --robot.id=$ROBOT_ID \
+  --robot.cameras='{
+      camera1: {type: opencv, index_or_path: /dev/v4l/by-path/$WRIST_PATH, width: 640, height: 480, fps: 30},
+      camera2: {type: opencv, index_or_path: /dev/v4l/by-path/$DESK_PATH,  width: 640, height: 480, fps: 30}
+  }' \
+  --policy.path=Project-IRA/TPSoSe2026_SmolVLA_LeRobot_SO101_Finetuning_V6_Full \
+  --dataset.repo_id=$HF_USER/eval_run \
+  --dataset.single_task="Sort the lego by color" \
+  --episodes=10
+```
+
+> Use one of the exact training prompts (see the dataset card) as the task string.
+> Both cameras run at **640x480** at inference time even though `desk_view` was
+> recorded at 800x600.
+
+## Robot setup
+
+| | |
+|---|---|
+| Robot | SO-101 follower arm (6-DOF), `robot_type: so_follower` |
+| Teleoperation | SO-101 leader arm |
+| Control frequency | 30 fps |
+| State / action space | 6-dim: `shoulder_pan.pos`, `shoulder_lift.pos`, `elbow_flex.pos`, `wrist_flex.pos`, `wrist_roll.pos`, `gripper.pos` |
+| Camera `observation.images.desk_view` | 800x600, h264 (recording) |
+| Camera `observation.images.wrist_left` | 640x480, h264 (recording) |
+
+> **Inference note:** both cameras are run at **640x480 during inference**, not at their
+> recording resolutions, to reduce the payload sent to the inference server.
+
+## Environment notes
+
+All training ran on a SLURM cluster with L40S GPUs. Two environment details were required
+and are easy to miss when reproducing:
+
+- **ffmpeg libraries for torchcodec.** A minimal conda env supplies the shared libraries
+  that `torchcodec` discovers at runtime:
+  `export LD_LIBRARY_PATH=$CONDA_PREFIX/envs/ffmpeg_libs_v8/lib:<venv>/lib/python3.12/site-packages/nvidia/npp/lib:$LD_LIBRARY_PATH`
+- **`--tolerance_s=0.01`** on every run, to accommodate timestamp jitter in the recorded
+  episodes.
+
+Multi-GPU runs additionally set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`.
+Datasets and the virtualenv were copied to node-local `/scratch` before training rather
+than read from shared storage.
+
+No Weights & Biases logging was enabled for any run (`--wandb.enable=false`), so there are
+no public training curves — the `job.*.err` SLURM logs are the record.
+
+## Tasks and prompts
+
+The model is conditioned on English natural-language instructions. Prompt phrasing was
+varied roughly every 10 episodes during recording, giving 93 distinct prompts in the
+merged dataset. **Use one of the training prompts verbatim** for best results — the full
+lists are on the [dataset card](https://huggingface.co/datasets/Project-IRA/TPSoSe2026_Dataset_Full_Merged_Final_LeRobot_SO101_V1).
+
+## Limitations
+
+- **Behaviour cloning.** The policy imitates teleoperated demonstrations and has no notion
+  of recovery beyond what was demonstrated. It is susceptible to covariate shift and can
+  fail to recover from states outside the demonstration distribution.
+- **Recovery data is incidental, not systematic.** Recovery behaviour appears in the data
+  only where the operator happened to make and correct a mistake during recording; no
+  recovery episodes were scripted deliberately.
+- **Single environment.** All data comes from one lab desk with one lighting setup, one
+  camera geometry, and one set of physical objects. Expect degradation elsewhere.
+- **Prompt sensitivity.** Language conditioning was trained on a fixed set of phrasings
+  (listed in the dataset card). Prompts far from those phrasings may behave unpredictably.
+- **No formal evaluation.** Quality assessments below are qualitative, from operators
+  observing rollouts on the physical arm. There are no success-rate numbers.
+- **Not safety-rated.** Supervise all physical execution and keep the workspace clear.
+
+## Upstream licensing & attribution
+
+This model is a derivative work of Apache-2.0 licensed components:
+
+| Component | Upstream | License |
+|---|---|---|
+| LeRobot framework | https://github.com/huggingface/lerobot | Apache-2.0 |
+| `lerobot/smolvla_base` | https://huggingface.co/lerobot/smolvla_base | Apache-2.0 |
+
+Apache-2.0 permits relicensing derivative works. We retain the upstream copyright
+notices, license text, and NOTICE files for the incorporated material, as Apache-2.0
+Section 4 requires. The upstream components remain under Apache-2.0 — only this
+project's own contributions (the fine-tuned weights and training configuration) are
+offered under **CC BY-SA 4.0**.
+
+**CC BY-SA 4.0** was chosen because it is share-alike: derivatives must be released under
+the same licence, so this work cannot be taken closed-source. The project's *source code*
+lives in a separate repository under its own licence — see
+https://github.com/Project-IRA/interactive-robotic-arm.
+
+## Citation
+
+```bibtex
+@misc{project_ira_2026,
+  title        = {Project-IRA: Interactive Robotic Arm},
+  author       = {Baten, Cleo and Keppler, Bela and Sapper, Jonas},
+  year         = {2026},
+  howpublished = {\url{https://huggingface.co/Project-IRA}},
+  note         = {Code: \url{https://github.com/Project-IRA/interactive-robotic-arm}}
+}
+```