EVALUATION_PROTOCOL.md

# Evaluation Protocol

This file defines how the public Xperience-10M sample episode is turned
into benchmark-style tasks, how the baselines are evaluated, and what the
reported metrics are allowed to mean.

## Protocol At A Glance

| Item | Current protocol |
| --- | --- |
| Source scope | 1 public Xperience-10M sample episode |
| Frames | 5,821 |
| Sliding windows | 1,161 windows, 20 frames each, stride 5 frames |
| Current feature vector | 8,378 dimensions |
| Split | chronological 70/30 train/test by time |
| Baselines | minimal interpretable heads plus compact neural MLP heads |
| Audio | present in MP4 streams and visualized, but not featurized in the current baseline vector |
| Raw data | not redistributed |

## Data Unit

The basic unit is a 20-frame aligned window built from one synchronized
public episode. Feature blocks are documented in
`results/episode_task_suite/feature_manifest.json`; the committed window
table is `results/episode_task_suite/windows.csv`.

## Split Policy

The current suite uses `single_episode_chronological`: The split preserves time order so future episode segments are not mixed randomly into the train set. It is still one episode; cross-episode generalization is evaluated in the multi-episode stage.

This makes some classification metrics intentionally harsh: later test
segments can contain action or subtask labels not present in the train
segment. Those cases are recorded in the task metrics as `unseen_test_classes`.

## Feature And Head Policy

- Input contract: 8,378-dimensional current feature vector.
- Source manifest: `results/episode_task_suite/feature_manifest.json`.
- Normalization: Scalers are fit on train windows only for the baseline heads.
- Audio status: Audio is present in sample MP4 streams and visualized in the atlas, but not extracted into the current 8,378-d feature vector.

Minimal heads are used first because they make task contracts debuggable.
Neural MLP heads reuse the same windows, splits, and feature tensors; they
are not foundation models.

## Task Contracts

| Task | Family | Unit | Input -> target | Primary metric | Minimal | Neural |
| --- | --- | --- | --- | --- | ---: | ---: |
| timeline_action | supervised classification | single window | current 20-frame all-feature window -> current action label | macro_f1 (higher better) | 0.0500 | 0.0263 |
| timeline_subtask | supervised classification | single window | current 20-frame all-feature window -> current subtask label | macro_f1 (higher better) | 0.0495 | 0.0175 |
| transition_detection | temporal diagnostic | single window | current 20-frame all-feature window -> action boundary versus steady | macro_f1 (higher better) | 0.6552 | 0.6485 |
| next_action | short-horizon prediction | single window | current 20-frame all-feature window at time t -> action label at t + 20 frames | macro_f1 (higher better) | 0.0593 | 0.0235 |
| hand_trajectory_forecast | trajectory regression | single window | current all-feature window -> future left/right hand 3D joints for 10 frames | mpjpe (lower better) | 0.8223 | 0.1116 |
| contact_prediction | binary classification | single window | non-contact and non-caption feature blocks -> any body contact | macro_f1 (higher better) | 1.0000 | 1.0000 |
| object_relevance | multi-label classification | single window | non-caption feature blocks -> current relevant object set | micro_f1 (higher better) | 0.1839 | 0.1798 |
| caption_grounding | retrieval | caption query | caption object/interaction query plus candidate sensor windows -> matching time window | mrr (higher better) | 0.0172 | 0.0178 |
| cross_modal_retrieval | retrieval | sensor query | motion, IMU, and camera query features -> matching depth/video window | top5_accuracy (higher better) | 0.3764 | 0.2155 |
| modality_reconstruction | cross-modal regression | single window | motion, IMU, and camera features -> depth/video feature vector | r2 (higher better) | -0.0160 | -0.0102 |
| temporal_order | pairwise diagnostic | adjacent window pair | two adjacent windows -> correct versus reversed order | f1 (higher better) | 0.5487 | 0.8718 |
| misalignment_detection | pairwise diagnostic | paired modality window | motion side plus visual/depth side -> aligned versus shifted by 8 windows | f1 (higher better) | 0.4866 | 0.7335 |

## Leakage Controls

- Use chronological train/test splits instead of random window shuffling.
- Fit scalers and learned projections on train windows only.
- Keep future labels, future mocap, contact labels, object labels, and caption labels on the target side unless a task explicitly treats language as the query.
- For cross-modal tasks, split query-side and candidate-side feature blocks before training and ranking.
- Report unseen test classes when the chronological split exposes labels absent from the train segment.

## Current Limitations

- Cross-episode generalization is evaluated in the later multi-episode stage.
- Feature-vector reconstruction is separate from pixel depth, mesh, NeRF, or Gaussian reconstruction.
- Qwen3-Omni setup artifacts are preparation artifacts until the 32-episode held-out pilot runs.
- Audio-visual learning needs an extracted audio feature block; audio is documented and visualized but not featurized in the current baseline vector.

## Scale-Up Gate

The full Qwen3-Omni fine-tuning pilot requires all of the following before
reporting held-out model metrics:

- at least 32 valid Xperience-10M episodes
- held-out episode split with no train/test episode leakage
- manifest, training metadata, progress logs, metrics, predictions, and run report
- held-out evaluation on test episodes rather than train windows

Current status: prepared but data-gated. Read
`results/omni_finetune/DATA_ACCESS_STATUS.md` and
`results/omni_finetune/MULTI_EPISODE_ACCESS_STATUS.md` before interpreting any
Qwen3-Omni artifact.

## Machine-Readable Copy

The JSON mirror is `docs/data/evaluation_protocol.json`.