PROJECT_BRIEF.md

# Project Brief

This project turns the public Ropedia Xperience-10M sample into a concrete
research task lab for embodied AI. It is designed to answer a practical
question: what can be built, measured, and extended from a richly synchronized
egocentric episode before scaling to held-out multi-episode training?

## Research Intent

The public sample is treated as a small but real research system. The project
does not try to inflate one episode into a final benchmark. Instead, it shows
the full path from data inspection to task design, baseline modeling,
evaluation, artifact packaging, and a guarded scale-up plan. A reader should be
able to trace one model input, understand each task, reproduce the public-sample
results, and see what remains before multi-episode model-quality claims.

## Capability Map

| Capability | Evidence in this project |
| --- | --- |
| Data understanding | `feature_manifest.json`, `available_modalities.json`, modality atlas, episode-window HF viewer |
| Task design | 12 task contracts, task cards, case-study walkthroughs, and four research-direction extension probes |
| Evaluation rigor | chronological split, per-task metrics, predictions, confusion matrices, leakage notes, and generated takeaways |
| Scale-up planning | 128-episode selection/relay plan, Qwen3-Omni path, Cosmos 3 branch, and policy-model candidates after action-space conversion |

## What Exists Now

| Layer | Current artifact |
| --- | --- |
| Data unit | 1 public sample episode, 5,821 frames, 1,161 synchronized 20-frame windows |
| Modalities | Video-derived features, audio, depth, pose/SLAM, mocap, IMU, calibration, and language-derived features |
| Task suite | 12 embodied-AI task contracts with inputs, targets, metrics, predictions, and case-study walkthroughs |
| Models | Minimal linear/ridge/logistic baselines plus compact PyTorch MLP heads for the same 12 tasks |
| Research map | Four Ropedia research directions with direct, proxy, diagnostic, and extension-task coverage |
| Scale-up path | Qwen3-Omni LoRA code path prepared; the gated Xperience-10M dataset is available for a selected 128-episode pilot |

## How To Read It

1. Start with the website or this brief to understand the project shape.
2. Open `RESEARCH_ROADMAP.md` to see how the work scales from the public
   sample to multi-episode modeling.
3. Open `EVALUATION_PROTOCOL.md` before comparing task scores.
4. Use `RESEARCH_TAKEAWAYS.md` for the current metric interpretation.
5. Inspect `results/episode_task_suite/feature_manifest.json` to understand one model input.
6. Use `results/omni_finetune/DATA_ACCESS_STATUS.md` for the multi-episode data status.

## What This Enables

The public sample is enough to build and verify task definitions, feature
contracts, metrics, visualization, and baseline code. It is not enough to
measure final model quality for a general embodied-AI model. The next research
stage is to run the same contracts on held-out episodes, then fine-tune and
evaluate an omni-model with train/test separation at the episode level.

## Best Entry Points

| Entry point | Link |
| --- | --- |
| Visual dashboard | https://chaoyue0307.github.io/ropedia-xperience-10m-task-suite/ |
| Interactive HF Space | https://huggingface.co/spaces/cy0307/ropedia-xperience-10m-task-suite |
| Derived artifacts | https://huggingface.co/datasets/cy0307/ropedia-xperience-10m-task-suite-artifacts |
| Baseline model bundle | https://huggingface.co/cy0307/ropedia-xperience-10m-task-baselines |
| Official Xperience-10M dataset | https://huggingface.co/datasets/ropedia-ai/xperience-10m |