docs/TESTING.md

Testing Guide

1. Smoke test — defaults (simplest)

source .venv/bin/activate
./run.sh

What to expect:

• Downloads lerobot/smolvla_base model and lerobot/svla_so101_pickplace dataset from HuggingFace (first run only, ~few GB)
• Auto-detects device (MPS on Apple Silicon, CUDA on Linux/Windows, CPU otherwise)
• Processes 8 frames from episode 0
• Saves to outputs/:
  • attention_grid_ep000.png — 3-row grid (original / self-attention heatmap / overlay) x 8 columns
  • Individual frame PNGs (since save_individual: true in defaults)
• Heatmaps should show colored blobs over the image — bright areas = high attention

2. Attention rollout — deeper aggregation

./run.sh --method rollout

What to expect:

• Same output structure, but heatmaps use rollout (multiplies attention across all SigLIP layers accounting for residual connections)
• Heatmaps should look more refined/focused compared to last-layer, since they reflect cumulative information flow

3. Cross-attention — action decoder focus

./run.sh --cross-attention

What to expect:

• Slower — runs the full policy forward pass (not just vision encoder)
• Grid gains 2 extra rows per frame (5 rows total): Row 4 = cross-attention heatmap (hot colormap), Row 5 = dual-color overlay (blue = self-attn, red = cross-attn)
• Cross-attention heatmap shows what the action expert reads from the image — should be tighter/more focused than self-attention if the model is well-trained

4. Per-head grid — head specialization

./run.sh --show-heads

What to expect:

• An additional grid PNG showing attention from each head separately for the first frame
• Look for head specialization: different heads attending to gripper, object, background, etc.

5. Full combo — everything at once

./run.sh --method rollout --cross-attention --show-heads

What to expect:

• All of the above combined
• Slowest run, most comprehensive output
• This is the "everything works" confidence check

6. Quick sanity check with fewer frames

./run.sh --num-frames 2 --episode 0

Takes much less time, good for verifying the pipeline runs end-to-end.

What "working" looks like

• No errors/tracebacks — script runs to completion
• Output files appear in outputs/
• Heatmaps are non-uniform — if every heatmap is perfectly flat/solid, something went wrong with attention capture
• Grid image opens and shows 3 (or 5 with --cross-attention) rows per frame
• Overlay images show colored attention blobs on top of recognizable robot scene images

Red flags

Symptom	Likely cause
FFmpeg/TorchCodec error on dataset load	FFmpeg 6 not installed or DYLD_LIBRARY_PATH not set — use ./run.sh
All heatmaps are uniform gray	Attention hooks didn't capture weights — check for SDPA/Flash fallback warnings
CUDA out of memory	Use --device cpu or --device mps (device is auto-detected by default)
KeyError on image key	Dataset doesn't match expected camera key — check dataset schema