| # Project Brief |
|
|
| This project turns the public Ropedia Xperience-10M sample into a concrete |
| research task lab for embodied AI. It is designed to answer a practical |
| question: what can be built, measured, and extended from a richly synchronized |
| egocentric episode before scaling to held-out multi-episode training? |
|
|
| ## Research Intent |
|
|
| The public sample is treated as a small but real research system. The project |
| does not try to inflate one episode into a final benchmark. Instead, it shows |
| the full path from data inspection to task design, baseline modeling, |
| evaluation, artifact packaging, and a guarded scale-up plan. A reader should be |
| able to trace one model input, understand each task, reproduce the public-sample |
| results, and see what remains before multi-episode model-quality claims. |
|
|
| ## Capability Map |
|
|
| | Capability | Evidence in this project | |
| | --- | --- | |
| | Data understanding | `feature_manifest.json`, `available_modalities.json`, modality atlas, episode-window HF viewer | |
| | Task design | 20 unified task contracts, task cards, case-study walkthroughs, and four research-direction extension probes | |
| | Evaluation rigor | chronological split, per-task metrics, predictions, confusion matrices, leakage notes, and generated takeaways | |
| | Scale-up planning | Final verified 96/16/16 Qwen3-Omni diagnostic result, same-split 128-episode baseline alignment, Cosmos3-Nano compatibility branch, and policy-model candidates after action-space conversion | |
|
|
| ## What Exists Now |
|
|
| | Layer | Current artifact | |
| | --- | --- | |
| | Data unit | 1 public sample episode, 5,821 frames, 1,161 synchronized 20-frame windows | |
| | Modalities | Video-derived features, audio, depth, pose/SLAM, mocap, IMU, calibration, and language-derived features | |
| | Task suite | 20 embodied-AI task contracts with inputs, targets, metrics, predictions, and setup alignment | |
| | Models | Minimal linear/ridge/logistic baselines plus compact PyTorch MLP heads for the unified 20-task public-sample suite | |
| | Research map | Four Ropedia research directions with direct, proxy, diagnostic, and extension-task coverage | |
| | Scale-up path | A selected 96/16/16 Qwen3-Omni LoRA final diagnostic result is verified; strict-JSON validity meets target, while weak action/subtask metrics guide the next error-analysis pass | |
|
|
| ## How To Read It |
|
|
| 1. Start with `PUBLIC_READER_MAP.md` if you need to choose between GitHub, |
| the website, Hugging Face artifacts, baseline weights, model branches, or |
| release-health files. |
| 2. Start with the website or this brief to understand the project shape. |
| 3. Open `RESEARCH_ROADMAP.md` to see how the work scales from the public |
| sample to multi-episode modeling. |
| 4. Open `EVALUATION_PROTOCOL.md` before comparing task scores. |
| 5. Use `RESEARCH_TAKEAWAYS.md` for the current metric interpretation. |
| 6. Inspect `results/episode_task_suite/feature_manifest.json` to understand one model input. |
| 7. Use `TASK_SUITE_20.md` and `docs/data/task_suite_20.json` to read the unified 20-task suite; the historical `docs/data/tier2_task_suite.json` path stores the tasks 13-20 result bundle. |
| 8. Use `docs/data/omni_finetune_verified_result.json` for the current multi-episode Qwen3-Omni pilot result. |
|
|
| ## What This Enables |
|
|
| The public sample is enough to build and verify task definitions, feature |
| contracts, metrics, visualization, and baseline code. It is not enough to |
| measure final model quality for a general embodied-AI model. The first |
| multi-episode Qwen3-Omni diagnostic pilot now verifies the held-out training |
| loop with validation loss recorded; the next research stage is to improve |
| JSON-format reliability and error analysis before larger robustness or |
| alternative backbone claims. |
|
|
| ## Best Entry Points |
|
|
| | Entry point | Link | |
| | --- | --- | |
| | Public reader map | https://github.com/ChaoYue0307/ropedia-xperience-10m-task-suite/blob/main/PUBLIC_READER_MAP.md | |
| | Visual dashboard | https://chaoyue0307.github.io/ropedia-xperience-10m-task-suite/ | |
| | Interactive HF Space | https://huggingface.co/spaces/cy0307/ropedia-xperience-10m-task-suite | |
| | Derived artifacts | https://huggingface.co/datasets/cy0307/ropedia-xperience-10m-task-suite-artifacts | |
| | Baseline model bundle | https://huggingface.co/cy0307/ropedia-xperience-10m-task-baselines | |
| | Official Xperience-10M dataset | https://huggingface.co/datasets/ropedia-ai/xperience-10m | |
|
|
