How it figures out why your robot policy fails — and what to fix first
Your VLA model makes wrong actions. Attention maps and GradCAM show where it looks, but not why it fails or what to change.
Automatically analyze the scene, build a multi-signal attribution matrix, form hypotheses, and test them with counterfactual interventions.
Ranked findings with evidence chains, severity levels, and concrete, actionable fixes — not just pretty heatmaps.
Use arrow keys or the buttons below to navigate
Nine phases, from raw image to actionable diagnosis. Each phase feeds into the next.
AI = LLM decides what to do next
Every final recommendation is backed by: signal data → symptom detection → hypothesis → counterfactual test result. No guessing.
Integrated: load model + dataset, run everything.
Post-hoc: analyze saved heatmaps from a previous run (faster, no GPU needed for matrix/symptoms).
The LLM decides which signals to compute, which hypotheses to test, whether to run follow-up experiments, and how to synthesize the narrative. Different evidence leads to different diagnostic paths.
Each phase's output determines which branch executes next.
Signal triage, hypothesis formation, iterative follow-up, and report synthesis.
Detection confidence, SAM → bbox, LLM → rule-based. Never blocks.
Unconfirmed hypothesis → new hypotheses → re-test. Budget: 7 total tests.
Two runs on the same model can follow different diagnostic paths.
The diagnostic system orchestrates six distinct AI/ML techniques.
Regex-based noun chunking: tokenizes instruction, removes stop words, returns noun phrases + implicit "robot gripper" query.
"pick up the white lego block" → white lego block robot gripper
Open-vocabulary ViT. Cross-modal dot products between image patches and text queries → confidence per (query, bbox). Per-class NMS keeps top-3.
No fixed label set — detects any object described in natural language.
Prompted with each OWL-ViT bbox. Produces pixel-level binary masks (H×W). Background = complement of all object masks.
Which patches causally drive actions? Per-patch importance = attention × gradient magnitude. 1024 patches → 32×32 → upsample to 512×512 heatmap.
Attention alone is representational. GradCAM adds gradient weighting for causality.
Which raw pixels matter? Enable gradients on input, backward pass, extract image.grad → L2 norm across channels → per-pixel map.
Full 512×512 resolution (finer than GradCAM's 32×32 but noisier).
Do internal features encode the target? Cosine similarity between patch embeddings and text embeddings.
Key: margin = target_sim - best_non_target_sim. Low margin (<0.08) = object confusion.
Before we diagnose, we need to know what we're diagnosing.
Representational — shows what the encoder focuses on, but not necessarily what drives the action output. Think of it as "what the model looks at."
Closer to causal — the action expert reads from VLM outputs via cross-attention. High cross-attention on a region means the decoder is actively using it.
Causal — gradient-based signals that answer: "if I change these pixels, how much does the action change?" This is the strongest evidence.
The diagnostic agent compares these three signal types against each other. When they disagree, that's informative.
The agent needs to know what's in the scene before it can measure where the model looks.
Extract object names from the language instruction using rule-based noun chunking.
"pick up the white lego block and place it in the stainless steel cup"
↓
white lego block stainless steel cup robot gripper
OWL-ViT v2 — open-vocabulary detector. Searches the image for each parsed object query. Confidence threshold: 0.10.
Fallback: if any query gets zero detections, re-scans at 0.03 confidence to recover task-critical objects.
SAM (Segment Anything) — takes each bounding box and produces a pixel-level mask.
Fallback: if SAM is unavailable, creates rectangular masks from bounding boxes. Background = everything not covered by any object.
Without scene segmentation, we can only say "the model looks at the top-left." With it, we can say "the model puts 78% of its causal attention on the background and only 3% on the target cup."
This is used everywhere downstream — the matrix, the counterfactuals, and the probes all need to know which pixels belong to which semantic region.
The agent collects multiple views of model behavior. No single signal tells the whole story.
| Signal | What It Measures | Type | Cost |
|---|---|---|---|
| Self-Attention | Where the SigLIP encoder focuses (last layer or rollout across layers) | Representational | Cheap |
| GradCAM (SigLIP) | Which patches causally affect the action prediction, via gradients at the encoder output | Causal | Expensive |
| Cross-Attention | Which vision tokens the action decoder reads from during denoising | Near-causal | Cheap |
| Saliency | Which input pixels (not patches) most affect the action, via input gradients | Causal | Expensive |
| GradCAM (Connector) | Information flow through the pixel-shuffle bottleneck (1024 → 64 tokens) | Causal | Expensive |
| Vision vs. State | Fraction of gradient norm from vision pathway vs. proprioceptive state | Scalar | Moderate |
| Per-Action-Dim | Separate GradCAM per action dimension (x, y, z, roll, pitch, yaw, gripper) | Causal | Expensive |
| Language Diff | GradCAM shift when you change the task instruction to something else | Causal | Expensive |
Cheap signals run first. LLM reads their summary and selects up to 4 expensive signals from the registry based on what symptoms it sees.
Loads pre-computed heatmaps from NPZ files. No model needed for matrix — only counterfactuals require loading the model.
The core data structure. It answers: "for each signal, what fraction of its total energy lands on each scene region?"
For each (signal, region) pair:
mass(signal, region) = Σ heatmap[region] / Σ heatmap[all]
Each signal's masses sum to ~1.0 across all regions. This normalizes for the fact that different signals have different magnitudes.
Example matrix row (GradCAM):
| Region | Mass |
|---|---|
| background | 0.78 |
| robot gripper | 0.14 |
| lego block | 0.03 |
| steel cup | 0.05 |
78% of causal signal on background — that's a problem.
Alongside the matrix, the agent computes:
The matrix is computed per-frame, then averaged. This smooths out frame-specific noise and reveals consistent patterns. Frame-to-frame variance is also tracked (used by the "unstable GradCAM" symptom detector).
13 domain-specific detectors run against the matrix. Each checks a known failure pattern.
| Symptom | Severity | Trigger | What It Means |
|---|---|---|---|
| High background attribution | Critical | GradCAM background >75% | Actions driven by background, not task objects |
| Spatial shortcut | Critical | Positional baseline >0.6 + low object attr | Memorized positions, not recognizing objects |
| Low object attribution | Critical | Target object GradCAM <5% | Not using the target object |
| Attention-GradCAM divergence | Warning | Region differs >0.3 attn vs GradCAM | Looks at regions it doesn't use |
| Dead state pathway | Warning | Vision share >99.5% | Proprioceptive state ignored |
| Language insensitivity | Warning | Lang diff max shift <0.05 | Instruction change has no effect |
| Low dataset diversity | Warning | Position std <15px, bg div <0.02 | Training data too uniform |
| Gripper fixation | Warning | Gripper >40% causal signal | Over-reliance on gripper |
| Cross-attention diffuse | Warning | Cross-attn entropy >5.0 | Decoder reads all tokens equally |
| Temporal instability | Warning | Centroid smooth >0.15, tracking <0.3 | Attention jumps erratically |
| Action-attention misalign | Warning | Translation dims → background | Spatial dims use shortcuts |
| Single region dependency | Info | All dims attribute to same region | No per-dimension specialization |
| Unstable GradCAM | Info | GradCAM std/mean >1.0 | Noisy gradient attribution |
Deeper analysis of what the model represents at the patch level, beyond just heatmap attribution.
Measures whether patch representations encode the target object's semantics. Cosine similarity between SigLIP patch embeddings and text embeddings.
Key metrics:
Classifies each attention head as semantic, positional, or mixed. Computes QK attention map per head, then measures:
Classification: advantage = semantic_corr - positional_corr + relocation_bonus
>0.08 → semantic <-0.08 → positional else → mixed
Positional-dominated model likely uses spatial shortcuts.
The LLM doesn't just list problems — it designs experiments to verify them.
System assembles: architecture context, task + scene, diagnostic matrix (markdown), symptom list with values, probe results, and available tests (auto-generated from counterfactual registry).
Returns a JSON array of hypotheses. Robust parser handles markdown fences, trailing commas, partial failures.
No API key? Each symptom maps to a HypothesisTemplate via registry lookup. Same pipeline, no LLM reasoning.
Each of up to 7 hypotheses includes:
description | What might be wrong |
confidence | 0–1, how sure the LLM is |
test_type | Which counterfactual to run |
test_params | Specific parameters (LLM-chosen) |
confirms_on_change | Direction of confirmation |
Hypothesis → Test Result → Surprising? → New Hypothesis
Budget-capped at 7 total tests. Unconfirmed hypotheses get confidence reduced by 60%.
true: Confirmed when action changes. false: Confirmed when action doesn't change.
Eight deterministic primitives the agent digitally applies to the scene. The primitives themselves always do the same thing — but which ones run and with what parameters is decided by the LLM.
Replace background with gray/noise/blur. High delta = model uses background.
Move target by (dx,dy). Tests object tracking vs position memorization.
Shift hue in HSV. Low delta = shape-based, not colour-based recognition.
Cover target with gray/noise. Low delta = spatial shortcut (doesn't need to see object).
Replace instruction with "do nothing". Confirmed when actions don't change (language-blind).
Insert synthetic ellipse/noise. High delta = model distracted by novel objects.
Adjust brightness/contrast. Tests robustness to lighting changes.
Same perturbation across frames. Tests response consistency over time.
Not all counterfactuals are LLM-chosen. Some always run for comparability across diagnostic runs.
These four tests run on every diagnostic, regardless of what the LLM hypothesizes. This ensures you can compare results across models and runs.
| background_substitution | gray fill |
| object_relocation | shift target object |
| task_string_swap | "do nothing" |
| occlusion_targeted | gray fill on target |
The LLM chooses which additional tests to run based on symptoms. The agent also runs iterative follow-ups if initial results are surprising:
Total counterfactual budget is configurable (default: 3 hypothesis-driven + all mandatory).
A counterfactual result is evaluated against the hypothesis:
significant_change = action_delta_L2 > 0.02
If confirms_on_change = true: confirmed when change IS significant (model depends on the feature).
If confirms_on_change = false: confirmed when change is NOT significant (model ignores the feature).
Unconfirmed hypotheses have their confidence reduced by 60% in the final report.
The critical question: is the model recognizing objects, or just memorizing where they usually are?
| object | Model genuinely recognizes and tracks objects |
| spatial | Model relies on memorized positions |
| mixed | Some evidence of both strategies |
| inconclusive | Not enough evidence to tell |
A spatial model works in training but fails when anything moves — new object positions, different camera angles, or real-world deployment.
The fix is different depending on the verdict: spatial bias needs data diversity; weak object grounding needs representation improvements.
The LLM synthesizes all evidence into ranked, actionable findings. It decides severity, writes the narrative, and proposes specific fixes.
severity | critical warning info |
title | Short name |
observation | Cites specific numbers |
test_description | What counterfactual was run |
test_result | What happened |
interpretation | What it means for the model |
fix | Concrete, actionable steps |
expected_impact | Quantified improvement |
evidence_refs | Links to symptoms & hypotheses |
Every finding traces back through the full chain:
This means you can always ask "why does the report say this?" and trace it back to specific numbers in the diagnostic matrix.
The LLM also writes a 2–4 paragraph narrative targeted at robotics engineers (not ML specialists). It answers:
Each counterfactual generates a side-by-side comparison image showing exactly what was changed.
Each primitive specifies which pixels it modified:
| Background sub. | background mask (everything except objects) |
| Object relocation | old position OR new position |
| Object recolor | object mask |
| Occlusion | object mask |
| Distractor | ellipse mask at insertion point |
| Lighting | no mask (whole image changes) |
| Task string swap | per-dimension bar chart (no image change) |
A pick-and-place model trained on 100 episodes with a plain background and a static cup.
Detects: robot gripper lego block steel cup
Segments each with SAM (or bbox fallback).
Cup position std: 4px (static!)
Background diversity: 0.01 (single environment)
→ low_dataset_diversity
Attention: diffuse, mostly on surface
GradCAM: 78% background, 3% lego block
high_background
low_object
spatial_shortcut
low_diversity
33% semantic, 25% positional, 42% mixed.
Top heads are "semantic" but spread attention across background, not the target object.
H1: "Relies on background" → test: background_substitution
H2: "Spatial shortcut for cup" → test: object_relocation
Background sub: delta = 0.30 (confirmed!)
Object relocation: delta = 0.02 (confirmed — model doesn't care where object is)
Verdict: spatial
The model memorized a trajectory. It doesn't visually track objects.
Critical: Background shortcut.
Fix: Add background augmentation during training. Expected: reduce background attribution from 78% to <30%.
Every major component uses a registry pattern. Add a function in one file, and the rest of the pipeline discovers it automatically.
Set scene_model in YAML. Passed as model_id to detect_objects().
agent.py
@register_signal("my_signal", cost_description="~1min", prompt_description="...")
LLM triage prompt auto-generates from registry.
matrix.py
@register_symptom_detector on any function in matrix.py. Matrix, LLM, and report consume symptoms generically.
counterfactual.py
@register_primitive("counterfactual.my_test", prompt_description="...", param_schema="...")
LLM hypothesis prompt auto-generates from registry.
registry.py
register_hypothesis_template("my_symptom", test_type="...", confidence=0.7, ...)
Maps symptom type → counterfactual test automatically.
registry.py
@register_llm_provider("vertex")
on any async function. Agent checks registry before built-in providers.
mandatory_counterfactual_tests: list in diagnostic.yaml. No code changes needed.
Findings are data-driven dataclasses. Any severity, any structure. Nothing to register.
Add a new symptom detector + its hypothesis template + its counterfactual primitive — the LLM prompt, matrix, report, and comparison images all update automatically. Three decorator calls, zero plumbing.
AGENTS.md in the repo root is the onboarding guide for AI coding assistants — it documents the architecture, key invariants, testing patterns, and extension points so that agents (Claude Code, Cursor, etc.) can contribute to the codebase with full context.
48 read-only tools let any MCP-compatible client (Claude Code, Cursor, Windsurf, Continue, Zed, custom agents) query runs, diagnostics, and comparisons directly. No web viewer needed — just ask questions in natural language.
The MCP server reuses the same run_scanner, data_loader, stats_engine, and diagnostic_service modules as the web backend — zero code duplication.
pip install "mcp>=1.0.0"| Discovery | 8 | list, search, tag, validate runs |
| Inspection | 12 | heatmaps, gradients, images |
| Statistics | 3 | viz stats, summary, formatted |
| Diagnostic | 11 | symptoms, hypotheses, findings |
| Comparison | 5 | cross-run diffs, matrix |
| Registry | 7 | primitives, signals, templates |
| Notes | 2 | get/set run notes |
"Compare model v2 vs v3 across bridge and aloha datasets"
"What symptoms did the diagnostic find in the latest run?"
"Show self-attention stats for frame 0"
Compare diagnostic runs side-by-side to track training progress, validate fixes, or detect regressions. No GPU or torch required — works from pre-computed reports.
Comparison reads report.json and weightwatcher.json from each run directory. No model loading, no GPU, no torch import.
Each RunSnapshot loads diagnostic JSON + weight internals. Deltas are computed per-signal, per-test, and per-component. The verdict and recommendations auto-generate from delta patterns.
The DiagnosticReport is the single serializable root. Arrows show ownership; dashed lines show cross-references.