The public suite has two evidence lines. Line 1 uses one public sample episode to make the 20-task lab inspectable and reproducible. Line 2 uses 128 selected episodes to compare aligned baselines, Qwen3-Omni v6 LoRA, Cosmos3-Super Reasoner, and Cosmos3-Nano Future Window. The public matrix is complete at 180/180 scored method-task records, with six compact-proxy cells explicitly marked.
One public episode becomes aligned windows, task targets, Minimal heads, and Neural MLP heads.
Inspect the sample files, task targets, and local baseline runs.
Selected-128 comparison rows and held-out model behavior.
Seven methods share the selected-episode surface and the same 20 task axes.
Compare same-split baselines, Qwen3-Omni v6, and Cosmos3 rows.
Direct raw-target metrics for the 6 proxy-marked cells.
The full SVG names every task axis, separates the nine methods into readable panels, and keeps source artifacts plus proxy notes attached to the same comparison view.
01 Action Recognition
02 Procedure Step Recognition
03 Action Boundary Detection
04 Next-Action Prediction
05 Hand Trajectory Forecasting
06 Contact State Prediction
07 Object Relevance Prediction
08 Language Grounding
09 Cross-Modal Retrieval
10 Cross-Modal Reconstruction
11 Temporal Order Verification
12 Multimodal Synchronization Detection
13 Long-Horizon Next-Action Forecasting
14 Long-Horizon Next-Subtask Forecasting
15 Interaction Text Prediction
16 Action-Object Relation Prediction
17 Future Object-Set Forecasting
18 IMU-to-Hand Pose Reconstruction
19 Camera-View Synchronization Retrieval
20 Time-to-Next-Transition Regression
Read the suite as two lines. Line 1 proves the task lab is inspectable and reproducible. Line 2 compares selected-128 metadata/raw baselines, Qwen3-Omni v6 LoRA, Cosmos3-Super Reasoner, and Cosmos3-Nano Future Window. Keep the lines separate when interpreting scores.
The mark identifies the shared public package across the GitHub repository, GitHub Pages dashboard, Hugging Face Space, artifact dataset, model mirrors, and social preview. Use this area as the project identity checkpoint before reading the 1-episode and selected-128 evidence lines.
| Line | Data unit | Score statement | Best use | Read separately from | Start here |
|---|---|---|---|---|---|
| 1 sample episode | One public Xperience-10M sample episode; 5,821 frames; 1,161 aligned 20-frame windows; 8,546-dimensional feature contract. | 40/40 direct scores from Minimal and Neural MLP heads. | Raw sample inspection, file organization, task definitions, local reproduction, and controlled Minimal-vs-Neural baseline behavior. | The selected-128 comparison rows and broader held-out model behavior. | Raw browser 1-episode radar JSON result summary JSON line JSON |
| 128 selected episodes | Selected held-out 96/16/16 split; 34,269 exported windows; public-safe metadata/raw-feature artifacts linked to official gated episode paths. | 140/140 selected-128 scores: 134 direct + 6 compact-proxy. | Same-split comparison across metadata/raw baselines, Qwen3-Omni v6 LoRA, Cosmos3-Super, Cosmos3-Nano, and scale-up decisions. | Direct raw-target interpretation for the proxy-marked cells. | 128-episode radar JSON feature index JSON HF selected-128 windows result summary JSON line doc |
| Evidence line | Method block | Methods | Score statement | Read as |
|---|---|---|---|---|
| 1 sample episode | Task-head baselines | Minimal; Neural MLP | 40/40 direct scores. | Task-lab reproducibility and simple-vs-neural behavior. |
| 128 selected episodes | Aligned baseline heads | Metadata simple/NN; raw-feature simple/NN | 80/80 scores: 74 direct + 6 compact-proxy. | Same-split metadata/raw-feature baseline comparison. |
| 128 selected episodes | Qwen3-Omni series | Qwen3-Omni v6 LoRA | 20/20 direct scores from verified selected-128 Qwen3-Omni LoRA and task-specific probes. | Trainable Qwen3-Omni diagnostic baseline on the selected-128 surface. |
| 128 selected episodes | Cosmos3 series | Cosmos3-Super Reasoner; Cosmos3-Nano Future Window | 40/40 direct scores from verified public-safe reasoner and future-window artifacts. | Cosmos3 reasoner and future-window diagnostics on the selected-128 surface. |
Cosmos3-Super Forward-Dynamics LoRA is published as a separate fine-tuned adapter with weights/results; it is not counted as a 20-task matrix method row.
| Qwen run | Purpose | Main change | Eval signal | Use now |
|---|---|---|---|---|
| v1 | Prove the selected-128 LoRA/eval/package loop. | First verified 96/16/16 selected-episode Qwen3-Omni LoRA run. | 448 eval; JSON 0.8750; contact 0.6451. | Lineage only. |
| v2 | Make answers schema-checked. | Structured-JSON contract with full-8-GPU LoRA on the same split. | 448 eval; JSON 0.9978; contact 0.7188. | Structured-output ablation. |
| v3 | Separate prompt/eval effects from training. | Strict-label prompt/eval over the v2 adapter; no new adapter training. | 448 eval; JSON 1.0000; contact 0.7210. | Prompt/eval ablation. |
| v4 | Test longer structured-JSON LoRA training. | New four-epoch full-8-GPU adapter on the same selected split. | 448 eval; JSON 1.0000; contact 0.7299. | Overfit/metric-tradeoff evidence. |
| v5 | Move to denser multiscale evaluation. | Multiscale cap96 export with 4,032 held-out predictions. | 4,032 eval; JSON 1.0000; contact 0.7865. | Pinned prior release; stronger on several non-contact metrics. |
| v6 | Publish the current Qwen 20-task row. | Rank64/lr5e-5 multiscale LoRA plus verified task-specific probes. | 4,032 eval; JSON 0.9990; contact 0.8177. | Current public 20-task Qwen3-Omni row. |
Qwen v1-v6 are run-lineage labels inside the selected-128 evidence line, not project evidence lines. Use v6 for the public 20-task Qwen3-Omni row; keep v5 as the pinned prior multiscale comparator; read v1-v4 as pipeline-hardening and ablation evidence. Full details: qwen3_omni_run_lineage.json and QWEN3_OMNI_RUN_LINEAGE.md.
Use this route if you need the project story, what is public, and how to read each result family.
Read overviewUse the 20-task suite, radar, matrix, and source audit to compare methods without losing metric provenance.
Open task suiteUse scripts, validators, mirrors, and checks when you want to rerun or trust the public package.
Open reproduce pathUse directions and scale-up resources for spatial, world-model, VLA, Qwen3-Omni, and Cosmos3 follow-up work.
Open directionsThe project keeps source code, visual explanation, derived artifacts, model outputs, and release checks on different public surfaces. This map shows what each surface is responsible for before you dive into the full file set.
Start with the brief and status files, then use the dashboard for the visual story.
Use the task contract, protocol, walkthroughs, and radar matrix to follow each scored axis.
Open the sample explorer, raw-file manifest, and feature manifest before reading model scores.
Use the glossary when evidence lines, direct/proxy scores, Qwen v1-v6, Cosmos branches, or HF surfaces are unclear.
Single-episode baselines, 128-episode aligned baselines, Qwen3-Omni v6 LoRA, and Cosmos3-Super/Nano diagnostics stay separated by evidence type.
Spatial intelligence, human-video world modeling, and vision-language-action are documented as trainable directions with task mappings.
Publication checks validate source alignment, package contents, mirror parity, and live URL/hash status.
Xperience-10M is much larger than the public sample. This project focuses on the sample available now, turns it into clear task contracts and baseline artifacts, and keeps the same data contract ready for held-out multi-episode training when more episodes are prepared.
A research-development lab for understanding synchronized egocentric multimodal data, defining embodied-AI tasks, and testing small baselines before omni-model fine-tuning.
The next model-quality stage is stronger action/subtask modeling on the same held-out split, using dense/multiscale windows before requiring more raw episodes.
Maps one public episode into synchronized windows across video, audio, depth, pose/SLAM, mocap, IMU, calibration, and language-derived signals.
Defines embodied-AI inputs, process modules, outputs, metrics, and case-study walkthroughs instead of treating the sample as a generic classification file.
Keeps chronological splits, predictions, confusion matrices, leakage notes, and single-episode limits visible before moving to broader model-quality reads.
Connects the same data contract to 128-episode baselines, a no-new-episode enhancement pack, Qwen3-Omni LoRA, Cosmos-style world modeling, policy/VLA tracks, and the later Xperience-native pretraining goal.
Minimal and Neural MLP baselines over the original public-sample episode, with 40/40 scored method-task records.
Metadata, raw-feature, Qwen3-Omni, and Cosmos3 methods on the aligned 128-episode surface, with all 140 rows scored and proxy/evidence notes kept explicit.
Use this as the front door for the project: it links the unified 20 tasks, four research tracks, current sample evidence, and the multi-episode Qwen3-Omni scale-up path.
One Xperience-10M public sample episode is converted into aligned windows and a documented feature contract.
The unified task suite has minimal and neural baseline evidence across one 20-axis task surface with shared windows, splits, and label discipline.
The public description is aligned to the official gated Xperience-10M dataset card, including modalities, scale, access, and current project coverage. The source snapshot records 31.9 TB on the HF surface, an about-1PB full-scale storage statement, 12,103 episode folders as upstream metadata, not a local data inventory, public sample license cc-by-nc-4.0, HOMIE Toolkit and Rerun 0.29.0 source tooling, and the official limited diversity note. See data/source_alignment_audit.json.
Metrics, figures, walkthroughs, baseline weights, Qwen3-Omni results, and Cosmos3 public-safe packages are staged across GitHub, GitHub Pages, and Hugging Face.
The first selected-episode LoRA pilot is packaged with real held-out predictions and metrics. It proves the pipeline, while the weak scores make it a baseline for error analysis.
Shows how the current selected split can be stressed without more episodes: dense windows, hierarchical labels, raw-feature shards, and `multiscale_20s10_40s20_80s40` as the next export target.
Raw MP4/HDF5/RRD files, private gated Xperience-10M data, and full Qwen weights are excluded from the public repo and HF mirrors.
The project path moves from the current public-sample task lab to the latest verified Qwen3-Omni diagnostic run, same-split 128-episode baseline alignment, a no-new-episode enhancement pack, action/subtask error analysis, robustness runs, world/policy tracks, and the future Xperience Embodied Foundation Model pretraining goal.
One public episode is converted into aligned windows, task contracts, minimal baselines, neural heads, walkthroughs, and figures.
Prepare official gated episodes while preserving episode-level separation and recording missing-view coverage. The first selected split is available for Qwen3-Omni diagnostics.
Train lightweight adapters on selected prepared episodes and evaluate on held-out episodes with committed predictions, metrics, and run reports.
Align simple metadata/text baselines, raw-feature proxies, and neural MLP baselines to the same selected 96/16/16 split and the unified 20-task axes used by the public result matrix.
Use the same selected split, estimate dense/multiscale window exports, define hierarchical action/subtask targets, and prioritize raw-feature shards for tasks that metadata baselines cannot cover.
Keep the 96/16/16 split, tighten JSON decoding or target formatting, and analyze action/subtask failures before presenting stronger model-quality numbers.
Keep Qwen3-Omni as the first trainable held-out pilot, use Cosmos 3 for world modeling and forward-dynamics trainer development, and stage policy candidates after robot-compatible action targets are explicit.
Test whether pilot conclusions survive broader sessions, missing modalities, and stronger ablations.
Extend toward future-window prediction, action-conditioned world modeling, synthetic-data tests, policy-style next action, and affordance reasoning.
Pretrain an Xperience-native domain model over synchronized video, audio, depth, pose, mocap, IMU, and language after smaller scaling stages prove value.
Beyond the current task heads, Qwen3-Omni fine-tuning path, Cosmos/world-model track, and future native pretraining goal, Xperience-10M can support three foundation pipeline tracks plus several concrete research-development tracks.
Train spatial-memory models from multiview RGB, egocentric video, depth, pose, calibration, object/contact cues, and language prompts; evaluate spatial QA, object permanence, counting, retrieval, and pose-aware consistency.
Use windows.csv and shared_windows.npz to slice each 20-frame window, then join six MP4 RGB streams with annotation.hdf5 depth, camera pose, SLAM/calibration, object cues, contacts, and optional language questions.
Build targets such as camera-view match, object relevance, object-set memory, depth/pose reconstruction proxy, caption-grounded retrieval, and spatial QA answers derived from the same public annotation timeline.
Train future-prediction models from observed interaction windows to score next action, next subtask, future object set, contact transition, camera-motion delta, and latent future state, with Qwen-style probes and Cosmos-style dynamics kept separate.
Take the current 20-frame observed window at time t from shared_windows.npz: RGB/audio/sensor summaries, hand/body motion, camera pose, current object/contact state, and current action/subtask context only.
Shift the same episode timeline forward to produce next-action, next-subtask, future object-set, contact-transition, time-to-transition, camera-motion delta, or latent/future-feature targets. Future labels stay out of the input.
Train VLA or policy-compatible heads only after converting egocentric video, captions, hand/body motion, contacts, objects, and procedures into traceable action tokens, chunks, and object-conditioned action targets.
Use egocentric/fisheye video windows, caption/object context from annotation.hdf5, hand/body mocap, contact state, and current subtask text as the observation-language side of each training pair.
For the one-sample suite, output action-token proxies: current/next action, object-conditioned action relation, contact state, interaction-text class, subtask transition, or hand-trajectory/action-chunk proxy. Robot action chunks need a later retargeting converter.
Build an episode atlas, category tags, balance report, and split builder across activities, objects, scenes, sessions, people, and missing modalities.
direction dataVersion train/val/test manifests, task cards, leakage checks, metric scripts, and reference baselines so future model scores are comparable.
direction noteTrain contrastive and masked-prediction encoders over synchronized video, audio, depth, pose, mocap, IMU, and language windows.
JSON planMine action steps, transitions, preconditions, effects, and temporal graphs that connect egocentric perception to planning.
current task mapAdd contact, reachable-object, tool-use, and next-affordance tasks using hands, mocap, objects, contacts, video, and language.
task walkthroughsFuse depth, pose/SLAM, multiview video, and object cues into persistent scene/object maps for spatial reasoning and object permanence.
model tracksTrack timestamp drift, missing streams, calibration consistency, corrupted files, and degraded-mode manifests before large training runs.
evidence contractConvert mocap, hand trajectories, contacts, and object states into action tokens, robot-compatible targets, and imitation-learning examples.
foundation planThe protocol is generated from committed metric artifacts so readers can see the exact data unit, split, task targets, leakage controls, and current limitations before comparing scores.
One 20-frame aligned window from the public sample episode, stride 5 frames, 1,161 windows total, represented by 8,546 synchronized multimodal dimensions.
Single-episode chronological 70/30 train/test split. This avoids random future-window mixing; cross-episode generalization is measured in the later multi-episode pilot.
protocol documentAll 20 tasks list input, target, primary metric, baseline score, and source artifact path in the unified suite file.
task_suite_20.jsonScalers fit on train windows only; future labels, target-side signals, caption/object labels, and contact labels stay on the target side unless explicitly queried.
builder scriptAudio and no-audio variants are evaluated across the walkthrough-backed task contracts under the same chronological split.
audio summaryQwen3-Omni is the first trainable baseline, Cosmos 3 is the world-model track with a camera-pose proxy forward-dynamics contract ready for trainer work, policy models wait for robot-compatible action targets, and Xperience-native pretraining remains a later full-corpus goal.
backbone planThis public-sample run covers single-episode task development. The selected multi-episode Qwen3-Omni final diagnostic result is verified and meets the JSON-validity target; Cosmos3-Nano has a verified future-window compatibility package; and Cosmos3-Super has a verified base-weight JSON-task evaluation plus a fine-tuned forward-dynamics LoRA branch. The next stage is action/subtask error analysis, stronger held-out metrics, and policy-target conversion.
result comparisonBefore adding episodes, the suite should try `multiscale_20s10_40s20_80s40`, hierarchical action/subtask targets, label-normalized scoring, and compact raw-feature shards for unsupported tasks.
task_suite_enhancement_128.jsonFuture Omni, Cosmos, and policy tracks use the same episode split discipline, training metadata, held-out predictions, metrics, run report, and public-safe package gate.
scale-up statusThe project shows the completed public-sample task suite and the first verified multi-episode Qwen3-Omni diagnostic pilot, then lays out the next quality-improvement and model-extension steps.
5,821 frames become 1,161 synchronized 20-frame windows with an 8,546-dimensional representation.
The current release reports nine method families over the unified 20-task axes, with minimal, neural, 128-episode, Qwen3, Cosmos3, and proxy-scored rows kept source-linked.
Audio variants improve the primary metric on 6 walkthrough-backed task contracts in this single-episode setting.
The Ropedia directions are labeled as direct, proxy, or diagnostic coverage, plus one coded extension probe per direction.
Qwen3-Omni stays first for held-out LoRA; Cosmos 3 is the world-model track with camera-pose proxy forward-dynamics targets ready for trainer work; OpenVLA/openpi/GR00T are policy candidates after robot-compatible action conversion; Xperience-native pretraining is the later full-corpus goal.
The selected 96/16/16 episode split now has a verified Qwen3-Omni v6 package with 4,032 held-out test predictions and 99.90% JSON validity. Cosmos3-Nano has 378 held-out future-window predictions, Cosmos3-Super Reasoner has 448 held-out base-weight JSON-task predictions, and Cosmos3-Super Forward-Dynamics LoRA has 448 held-out loss records.
The current 3,808-window export can be expanded through dense/multiscale windows without changing the held-out episode split; the recommended scenario is `multiscale_20s10_40s20_80s40`.
The Qwen3-Omni notes separate earlier diagnostic packages, the final 128-episode LoRA result, and the next action/subtask error-analysis pass.
The website, GitHub repo, Hugging Face Space, artifact dataset, baseline model repo, consolidated weights/results repo, and collection point to the same research project.
The visual set includes the logo, raw-sample stream thumbnails, task-suite figure, unified 20-task model radar, model-architecture figure, provenance baseline chart, and Qwen3-Omni LoRA training-flow figure.
The project logo is used consistently in the website header, favicon, README/HF cards, and social preview.
The public project shares derived task artifacts, figures, reports, and lightweight baseline files. Raw Xperience-10M videos, HDF5 annotations, RRD visualizations, gated data, and full Qwen weights stay outside the repo.
Tabs organize the sample data, 20 tasks, model method, results, research directions, and next-stage resources.
The reproduction guide lists the public sample setup, task-suite rebuild, neural heads, figure generation, and expected outputs.
The project keeps the official Xperience-10M dataset, public sample, dataset website, and HOMIE toolkit visible so readers can trace the data source.
A newcomer should be able to move from the dataset sample to the task design, model baselines, current limitations, and scale-up plan without reading every file first.
Start with the project brief, status, dataset context, task results, roadmap, and Qwen3-Omni scale-up notes. They separate implemented single-episode work from the prepared multi-episode stage.
Use the window table and feature manifest to see the aligned sample unit, modality sources, and leakage controls.
Every task has a small interpretable baseline and a matching neural MLP head over the same feature contract and chronological split.
The multi-episode Qwen3-Omni path now has a final verified diagnostic package and public LoRA adapter. The native-pretraining plan shows how this can grow into a full-corpus research direction after action/subtask improvements and stronger task metrics.
Use the no-new-episode enhancement pack before requesting more storage: it records dense-window estimates, `multiscale_20s10_40s20_80s40`, hierarchical labels, and raw-feature shard priorities.
The official Xperience-10M card describes a gated, large-scale 4D egocentric multimodal dataset. This project records that full upstream scope while focusing the implemented artifacts on one public sample episode. The source-alignment record keeps 31.9 TB, about-1PB, 12,103 episode folders, cc-by-nc-4.0, HOMIE Toolkit, Rerun 0.29.0, not a local data inventory, limited diversity, and data/source_alignment_audit.json visible on the public site.
Xperience-10M is a gated large-scale egocentric multimodal dataset for embodied AI, robotics, spatial intelligence, and world modeling.
The one-episode line builds the inspectable 20-task lab. Use Line 2 for selected-128 held-out comparison.
sample datasetThe raw browser is the canonical place to inspect synchronized video, embedded audio, HDF5 annotation groups, depth, pose/SLAM, mocap, IMU, calibration, and language-derived signals.
open raw browserstream metadataThe selected 128-episode Qwen3-Omni LoRA v6 diagnostic run is verified with 4,032 held-out test predictions and 99.90% JSON validity. Action/subtask metrics are still weak, so this remains a baseline for error analysis.
LoRA adapterv5/v6 comparisonThe Data tab now exposes the official public sample files directly, including playable MP4 video streams and the audio track embedded in fisheye_cam0.mp4.
open raw browserraw manifestRaw MP4, HDF5, RRD files are streamed from the official public sample source when opened here; private gated data and full Qwen weights are not redistributed in this project.
data noticeOne public sample episode, 5,821 frames, 1,161 aligned windows, 8,546-dimensional task inputs, and direct links to the official raw sample files.
task suiteraw manifestAction/subtask labels, next-action prediction, temporal diagnostics, hand trajectory, contact, object relevance, caption grounding, retrieval, reconstruction, misalignment, long-horizon forecasting, interaction text, action-object relation, sensor bridging, camera sync, and transition timing.
summary metricsThis project is for research exploration and excludes identity recognition, surveillance, biometric profiling, sensitive-attribute inference, and safety-critical deployment.
use notesFull audio-visual learning, caption generation, depth-pixel prediction, SLAM estimation, neural rendering, policy learning, cross-episode generalization, held-out Qwen3-Omni evaluation, and future Xperience-native pretraining.
native pretrainingOpen each official Xperience-10M sample file from the project page. Video and audio use compact browser previews derived from the official MP4 files, with direct links beside them for the full raw Hugging Face sources. HDF5 and RRD files are shown with their role, size, organization, and direct source links.
Fisheye camera 0 stream and the public sample audio source. This file can be played as video and as the embedded audio track.
Playing a 12 second fast-start preview derived from the official raw MP4. Use the source link for the complete file.
Video features feed visual tasks; the embedded audio stream feeds audio ablation and acoustic feature blocks.
The official public sample is one episode folder. The task suite reads the HDF5 annotations and six synchronized MP4 streams, then writes 20-frame windows with a 5-frame stride.
xperience-10m-sample/ annotation.hdf5 fisheye_cam0.mp4 fisheye_cam1.mp4 fisheye_cam2.mp4 fisheye_cam3.mp4 stereo_left.mp4 stereo_right.mp4 visualization.rrd
The raw HDF5 is a binary container, so the browser shows its organization rather than loading the whole file into memory.
The source streams are summarized once here, next to the playable files and HDF5 map.
Small derived modality thumbnails remain in modality_atlas.json; raw MP4, HDF5, and RRD files are not redistributed.
The suite connects synchronized multimodal windows to 20 task contracts in one table, one radar surface, and one source-linked result matrix. Historical filenames remain only for stable artifact links.
The unified radar keeps all nine methods in one comparison board, but groups them into small-multiple panels so each method family can be read directly. The split radars separate the 1-episode Minimal/NN baseline comparison from the 128-episode metadata/raw, Qwen3-Omni v6 LoRA, and Cosmos3-Super/Nano comparison.
Higher-is-better metrics are normalized to 0-1; lower-is-better metrics are converted to best/value within the task. The SVG uses sqrt(normalized score) only for visual radius, while raw values, linear normalized scores, status reasons, sources, and compact proxy notes remain in the JSON mirrors.
The matrix has 180/180 scored method-task records: 174 direct scores and 6 compact-proxy scores. The audit records the source artifact, metric key, and proxy reason for each marked cell.
Minimal and Neural MLP are both scored on all 20 public-sample task contracts in one enlarged panel without 128-episode methods competing for attention.
Seven aligned 128-episode methods cover all 20 axes across metadata/text, raw-feature, and foundation-model panels. Proxy axes stay labeled in the SVG and JSON.
Every script works from one data contract: aligned multimodal windows, explicit labels, cached feature extraction, and a manifest that makes omitted modalities visible.
Raw valid episodes move through split validation, parallel export, video/audio/text formatting, sensor-bridge features, LoRA training, and sealed held-out evaluation.
It documents the selected 128-episode final diagnostic result and the action/subtask improvement path for stronger held-out metrics.
It demonstrates the full development loop: reading Xperience-10M sample data, aligning modalities, converting them into model-ready windows, defining meaningful tasks, producing metrics, and packaging artifacts for continued research.
General embodied-intelligence model quality requires many episodes and held-out episode splits; the public sample is the development harness for that next stage.
Read results in this order: choose the line, open the matching radar, inspect the matrix row, then check proxy flags before interpreting totals.
Use 1 episode for the task lab. Use 128 episodes for the selected comparison surface.
Single-episode radar shows Minimal vs Neural MLP. The 128-episode radar shows metadata/raw baselines, Qwen3-Omni v6, Cosmos3-Super, and Cosmos3-Nano.
Each score keeps method, task, metric key, source artifact, and status.
Six selected-128 scores are compact proxies and stay marked in the audit.
Minimal and Neural MLP heads are both scored on all 20 public-sample task contracts. All 40 scores are direct task-target metrics.
A reproducible public task suite and baseline behavior check.
Metadata/raw baselines, Qwen3-Omni v6 LoRA, Cosmos3-Super Reasoner, and Cosmos3-Nano Future Window use the aligned 128-episode surface. It has 134 direct scores plus 6 compact-proxy scores.
A same-split comparison table with explicit source and proxy status.
| Line | Methods | Tasks | Scored records | Direct scores | Proxy scores | Machine-readable source |
|---|---|---|---|---|---|---|
| 1 sample episode | 2 | 20 | 40/40 | 40 | 0 | single-episode radar JSON |
| 128 selected episodes | 7 | 20 | 140/140 | 134 | 6 compact-proxy scores, each source-linked and reasoned. | 128-episode radar JSON |
| Total public matrix | 9 | 20 | 180/180 | 174 | 6 | two-line result summary JSON |
| Line | Block | Methods | Records | Evidence type | Primary artifact |
|---|---|---|---|---|---|
| 1 sample episode | Task-head baselines | Minimal; Neural MLP | 40 direct | Direct target metrics on the public sample windows. | single-episode radar JSON |
| 128 selected episodes | Aligned baseline heads | Metadata simple/NN; raw-feature simple/NN | 74 direct + 6 compact-proxy | Processed-target metrics where available; proxy cells remain source-linked. | score/proxy audit |
| 128 selected episodes | Qwen3-Omni series | Qwen3-Omni v6 LoRA | 20 direct | Verified selected-128 LoRA and task-specific probe artifacts. | model comparison JSON |
| 128 selected episodes | Cosmos3 series | Cosmos3-Super Reasoner; Cosmos3-Nano Future Window | 40 direct | Verified reasoner and future-window public-safe artifacts; forward-dynamics LoRA is a separate adapter artifact outside the 20-task method rows. | model comparison JSON |
Each cell shows the raw metric value to cite, the normalized radar value, the metric key, and a direct/proxy badge. The table is generated from the same task_method_20_result_matrix.json used by the radar, so values stay aligned across GitHub, the website, and Hugging Face mirrors.
| MethodiOne named method family in the matrix, such as Minimal, 128ep Raw NN, Qwen3-Omni v6, or Cosmos3-Super. | LineiA reading lane for a group of results: Line 1 is one public sample episode; Line 2 is selected-128 held-out comparison. | RecordsiOne method evaluated on one task. 9 methods x 20 tasks gives 180 public result records. | DirectiA metric computed against the task target directly. This is the preferred score type in the 20-task matrix. | ProxyiA bounded proxy metric when a direct raw target is not publicly available. It stays explicit so readers do not over-read it. | Scope |
|---|---|---|---|---|---|
| Loading result summary... | |||||
| Method | Loading tasks... |
|---|---|
| Loading 180-result matrix... | |
Best-practice reading rule: compare methods within the same evidence line first, then use the proxy badges before interpreting cross-method totals. Six compact-proxy cells are intentionally visible rather than blended into direct raw-target scores.
The public sample is converted into 5,821 frames, 1,161 aligned 20-frame windows, and an 8,546-dimensional representation for repeatable task evaluation.
All-feature action reaches 0.9829 macro-F1 on its local split, while the chronological action head in the core task suite is 0.0500 macro-F1 with four unseen later action labels.
takeawaysHand MPJPE improves from 0.8647 to 0.1079; temporal-order F1 rises from 0.5400 to 0.8520; misalignment F1 rises from 0.5052 to 0.7153.
metricsRidge/cosine retrieval remains stronger than the neural projection here, and cross-modal feature reconstruction still has negative R2.
retrieval metricsThe next credible model-quality unit is a held-out multi-episode pilot across different sessions, not more adjacent windows from one sample.
scale-up statusMotion-only and current all-feature classifiers use lightweight heads so the comparison stays readable on a laptop and easy to inspect. The neural run keeps the same features and splits, then swaps in PyTorch MLP heads.
The neural baseline uses small PyTorch MLP classifiers/regressors on the same 8,546-dimensional windows, chronological splits, and leakage filters. This isolates the value of a nonlinear head before moving to heavier Qwen/Omni experiments.
Each task is mapped as direct, proxy, or diagnostic evidence for the Ropedia research tracks. The mapping uses two current baselines: minimal interpretable heads and neural MLP heads over the same feature contract.
Direct evidence comes from hand trajectory forecasting and contact prediction; action and object relevance are supporting proxies.
Cross-modal retrieval, modality reconstruction, and misalignment detection check reconstruction prerequisites, not full geometry.
Action, subtask, transition, next-action, object, caption, order, and alignment tasks directly stress egocentric understanding.
Current probes cover task state, object relevance, retrieval, reconstruction, temporal order, and alignment but no persistent map yet.
Softmax, logistic, ridge, and retrieval heads keep every input/output contract readable. They are the first sanity check for whether a task is well-posed.
Small PyTorch MLP classifiers/regressors reuse the same features and splits. They test nonlinear gains before heavier Omni fine-tuning.
All 20 tasks live in the same task table, task-card grid, radar, and 180-record result matrix. Historical result paths are retained only for exact provenance links.
The public task package has one 20-task JSON, per-task metrics, prediction/rank files, Markdown summaries, radar charts, and the 180-record method-task matrix.
Open unified 20-task JSON · Open 180-record matrix · Open unified radar
Every task uses the same 20-frame window unit, 5-frame stride, 8,546-dimensional feature manifest, chronological split discipline, and minimal/neural comparison pattern unless a task-specific leakage rule removes target-side features.
Historical provenance JSON and historical provenance chart remain available for exact source tracing.
Case: classify fast reach/pour windows as high motion and steady holding windows as low motion.
Input: non-mocap video, depth, pose, IMU, SLAM, calibration, and language features.
Output: high_motion or low_motion.
Case: retrieve the synchronized stereo-left window from a fisheye-camera query.
Input: fisheye_cam0 video features against stereo_left candidate features.
Output: ranked synchronized view candidates.
Case: estimate whether a Pour coffee window is near the start, middle, or end of its action segment.
Input: non-caption multimodal features.
Output: 0-to-1 progress inside the current action.
Case: predict how the camera translation changes over the next 20 frames.
Input: current sensors excluding camera translation and captions.
Output: future camera-translation delta vector.
The four research directions now have coded extension probes, prediction/rank CSVs, JSON metrics, a Markdown summary, and a website chart generated from real sample-window features.
A full research result still needs many Xperience-10M episodes, held-out episode splits, stronger encoders, and direction-specific models such as body priors, renderers, or persistent scene graphs.
The diagram separates the shared episode-window representation from the task-specific heads, so the task contracts stay readable before scaling to larger models.
Each task uses a common research name and a concrete case study, then opens into the input, middle modules, output, modality evidence, metric, and current limitation.
Action Recognition
Egocentric Action Recognition
Input: inspect the 20-frame multimodal window before choosing the target.
In the coffee-making sample, a pouring window maps to the current action label.
Metric: macro-F1. Minimal 0.0500; neural MLP 0.0148.
Current limitation: single-episode chronological split.
All 20 task contracts are shown together with readable research names, representative modality thumbnails, explicit input-process-output contracts, and verified minimal versus neural scores. Rich interactive walkthroughs are available for the first 12 task cards; the remaining cards use the same unified task JSON contract.
The overall generated atlas keeps the icon family visible, while each task card below uses its own crisp assigned SVG for reliable loading and public mirrors.
The point is not hidden complexity. Every input group maps back to a source modality and a manifest entry.
The charts make the main lesson visible: within-episode supervised labels are easy under some splits, while retrieval, grounding, forecasting, and alignment remain the useful probes.
Open the single-episode explorer to inspect window-level labels, predictions, modality statistics, object labels, and diagnostic scores. The audio ablation summary records the task-by-task audio contribution.
These are the terms readers most often confuse when moving between the repo, website, Hugging Face mirrors, result matrices, and model-package cards. The full glossary is mirrored as Markdown and JSON.
Evidence lines, public-safe artifacts, and gated upstream data are different objects. The glossary keeps those boundaries visible.
Direct scores, compact-proxy scores, gap audits, and task-method records should not be interpreted as the same kind of evidence.
Minimal/NN heads, metadata/raw baselines, Qwen3-Omni v1-v6, Cosmos3-Super, Cosmos3-Nano, LoRA adapters, and full-parameter gates each mean something specific.
| Term | Meaning here | Use it for | Do not confuse with |
|---|---|---|---|
| Evidence line | A reading lane for a group of results. | Line 1 is the public sample episode; Line 2 is selected-128 held-out comparison. | Qwen v1-v6 run versions. |
| Public sample episode | The one fully inspectable official sample episode. | Raw-file browsing, task construction, single-episode baselines. | The selected-128 comparison rows. |
| Selected 128 episodes | Public-safe derived features linked to official gated episode paths. | Same-split Line 2 baseline/model comparisons. | Redistributed raw MP4/HDF5/RRD files. |
| 20-frame window | A fixed short clip slice used as a model input unit. | Feature rows, labels, tasks, and many baseline heads. | A full episode. |
| Task-method record | One method evaluated on one task. | The 9 x 20 public matrix, now 180 scored records. | A single prediction row. |
| Direct score | A metric computed against the task target directly. | Primary interpretation in the result matrix. | Compact-proxy score. |
| Compact-proxy score | A bounded proxy when the direct raw target is not public. | Explicitly marked cells in the gap audit and matrix. | A direct target measurement. |
| Raw metric value | The original value emitted by the runner or verified package. | The value to cite from the 180-result table. | Normalized radar value. |
| Normalized radar value | A 0-1 plotting value used only for comparable radar polygons. | Visual comparison across metrics with different scales. | The raw metric value to cite. |
| Minimal baseline | A simple non-neural task head; the "minimum" reference row in casual wording. | Single-episode lower-complexity comparison. | Selected-128 Simple baseline rows. |
| Simple baseline | A non-neural selected-128 baseline family. | Metadata/text and raw-feature 128-episode comparisons before NN/foundation rows. | The single-episode Minimal baseline. |
| Qwen3-Omni v6 | The current public Qwen 20-task row. | Qwen3-Omni LoRA plus task-specific probes. | All Qwen v1-v6 experiments. |
| Cosmos3-Super | The larger Cosmos-style branch. | Reasoner diagnostics and a verified forward-dynamics LoRA branch. | Cosmos3-Nano Future Window. |
| LoRA adapter | Lightweight trainable adapter weights. | Public model-branch artifacts when verified. | Full base-model weights. |
| HF artifact dataset | Hugging Face dataset repo for derived evidence. | Reports, metrics, website JSON, sanitized result packages. | The upstream Xperience-10M dataset. |
| Mirror parity | A check that public copies match source files. | Verifying GitHub, website, and HF mirrors. | A model-quality metric. |
Metrics, predictions, manifests, lightweight model weights, and derived window artifacts are organized so the project can be inspected, extended, and scaled before rerunning the full pipeline. Raw Xperience-10M data and Qwen weights are not redistributed.
Open GitHub, HF Space, artifact dataset, baseline models, or consolidated weights/results without guessing.
Open public surfacesUse validators, source alignment, mirror parity, and live URL/hash checks before trusting a number.
Open checksStart from scripts, windows, feature manifests, task contracts, and minimal/neural result outputs.
Open commandsUse Qwen3-Omni v6, Cosmos3-Super/Nano packages, the 128-episode feature index, and foundation-model plans for the next runs.
Open scale-upStart with the files that define the sample windows, modality inputs, task contracts, metrics, walkthroughs, and research-direction mapping.
Every task definition, split detail, feature dimension, and minimal/neural metric in one project output.
Window start/end frames and aligned action/subtask labels for the public sample episode.
window tableSource map for the current modality inputs used by the task suite.
feature inputsPer-task PyTorch MLP metrics, predictions, histories, and checkpoints for the unified task contracts, with historical result-bundle paths retained for provenance.
neural MLP outputsMaps the walkthrough-backed task contracts to the four research tracks: human modeling, 3D/4D reconstruction, egocentric interaction, and world modeling.
research direction outputsFour coded probes, one per research direction, with minimal and neural metrics plus prediction/rank CSVs.
extension probe outputsCase studies for the walkthrough-backed task contracts, including input, middle process modules, output, metric, limitation, and task-player data.
walkthrough outputsAll 72 task/variant rows comparing current audio, no audio, raw audio, replacement, and combined-input settings.
audio ablation outputsInteractive window-level view of labels, predictions, modality statistics, object labels, and diagnostics.
open explorerThe strongest self-supervised signal from the single episode.
retrieval metricsUse these files to navigate the whole project, open the published mirrors, or reproduce the public-sample pipeline.
Single navigation view for GitHub, GitHub Pages, HF Space, artifact dataset, baseline model repo, Qwen3-Omni/Cosmos3 repos, and result-reading lanes.
Definitions for evidence lines, windows, direct/proxy scores, Qwen v1-v6, Cosmos3 branches, adapters, and public mirror terms.
Human-readable map from project scope to data contract, task evidence, platform mirrors, and scale-up status.
Training, visualization, taxonomy, walkthrough, validator, and omni-readiness scripts.
scripts/The dashboard packaged as a public static Space.
HF SpaceStatic dashboard container published to GitHub Container Registry for local browsing with Docker, without raw data or model weights.
GHCR packageMetrics, predictions, docs, and lightweight derived files without raw data redistribution.
artifact collectionMinimal NumPy softmax, ridge baselines, and neural task-head model files.
model repoConsolidated public-safe baseline weights, Qwen3-Omni and Cosmos3 adapters/packages, verified results, analysis files, and manifest.
weights/results repoSpace, artifacts, baseline models, Qwen3-Omni v6 LoRA, Cosmos3-Super, and Cosmos3-Nano repos grouped into one public project collection.
collectionClassifier metrics, predictions, confusion matrix, and model weights.
model metricsCompact route through the project for readers who want the shortest path from scope to results after choosing a surface.
project packetThe multi-episode Qwen3-Omni path is documented, scripted, and verified as a validation-monitored held-out pilot. The next stronger metrics come from structured-output and error-analysis improvements.
Groups Line 1 task-head baselines and Line 2 selected-128 methods: metadata/raw baselines, Qwen3-Omni v6 LoRA, Cosmos3-Nano Future Window, and Cosmos3-Super Reasoner.
Maps every selected official Xperience-10M episode id to its gated source tree and the public-safe processed features: Qwen v6 multiscale windows, dense multiscale rows, and metadata matrices.
Canonical no-new-episode plan for denser supervision: `multiscale_20s10_40s20_80s40`, hierarchical action/subtask labels, stronger scoring slices, and raw-feature shard priorities.
task_suite_enhancement_128.jsonBackbone selection matrix covering Qwen3-Omni, Cosmos 3, GR00T, OpenVLA/openpi, Gemini Robotics, Octo, SmolVLA-style policy candidates, and the future Xperience-native pretraining goal.
foundation model planPublic data-access path, selected 128-episode pilot plan, and preparation requirements.
data accessSeparates the 1-episode sensor-adapter smoke test from Qwen run v1-v6. v6 is the current 20-task matrix row, while v5 remains the pinned prior release.
Qwen v1-v6 lineageQwen groupShows the verified Nano future-window compatibility package, the Super base-weight Reasoner JSON-task evaluation, and the Super fine-tuned forward-dynamics LoRA artifact with separate loss metrics.
Cosmos groupsFuture runs need validation tracking, held-out predictions, quality-target reporting, and the same public-safe package gate.
training requirementsFuture plan for a domain-specific embodied foundation model trained from scratch over full-corpus video, audio, geometry, motion, inertial, and language streams.
pretraining planThis tab now does one job: show the audit files that prove the public pages, mirrors, and package contents are internally consistent.
Checks the expected public package files, docs, figures, data JSONs, and artifact boundaries before a release is trusted.
Validates local website links, referenced data files, asset paths, and generated dashboard dependencies.
website integrityCompares GitHub, HF Space, artifact dataset, baseline model repo, and weights/results mirror snapshots.
mirror parityRecords public-surface readiness, reader-map links, live status pointers, and cross-repo publication checks.
surface QATracks live URLs and hash checks used after publishing to GitHub Pages and Hugging Face surfaces.
live statusSeparates official Xperience-10M source facts from local/public project inventory and derived artifacts.
source alignmentChecks task-count, task-contract, and result-matrix consistency across generated public data files.
task surfaceSummarizes build, validation, mirror, and publication gates that should pass before readers rely on the release.
quality gatesThe selected pilot uses 128 source-balanced episodes across 128 different session UUIDs. The latest v6 held-out package is verified, and its weak metrics define the next structured-output and error-analysis pass.
128 complete episodes selected from 128 unique top-level sessions, balanced across episode-size bands and split 96/16/16 for train/val/test.
source/feature indexDownload raw episodes only from official gated sources, exclude visualization.rrd, validate files, then stage them for training.
The current Qwen3-Omni LoRA artifact is the verified v6 selected 128-episode diagnostic adapter. The v5 row remains pinned as the prior release, and the 1-episode Qwen entry is only a sensor-adapter smoke test.
model groupsThe next suite push does not need more episodes first: use `multiscale_20s10_40s20_80s40`, hierarchical action/subtask targets, and raw-feature shards while keeping the held-out split fixed.
task_suite_enhancement_128.jsonQwen3-Omni uses a separate LoRA model repo; Cosmos3-Nano remains a compatibility package; Cosmos3-Super now has a verified forward-dynamics LoRA artifact with weights in a dedicated model repo.
Cosmos3-Super weightsThe long-term goal is a full-corpus Xperience Embodied Foundation Model trained on synchronized perception, geometry, motion, inertial, audio, and language streams after smaller scaling stages validate the approach.
pretraining planRaw Xperience-10M data is not redistributed here. The reproduction guide states the commands, expected outputs, exact-match reproduction record, and multi-episode requirements.
Human-readable commands, expected artifacts, and current scope for the public single-episode pipeline.
reproducibility guideMachine-readable command matrix covering sample download, baselines, the unified 20-task suite, figures, and validation.
reproducibility matrixThe last metric rebuild reproduced the public-sample outputs from a fresh cache and matched the committed metrics.
reproduction auditThe website organizes the dataset sample, tasks, methods, results, directions, and scale-up path in one tabbed reader flow.
project materialsThe comparison JSON groups selected-128 baselines, Qwen3-Omni v6 LoRA, Cosmos3-Nano Future Window, and Cosmos3-Super Reasoner. Full Qwen v1-v6 detail stays in a separate lineage audit.
comparisonQwen v1-v6Minimal path: install the toolkit dependencies, download the official sample, run the task suite with neural heads, regenerate the historical provenance bundle, build the unified 20-task index, regenerate visualizations, then rebuild the supporting project reports.
git clone https://github.com/Ropedia/HOMIE-toolkit.git
python3.12 -m venv .venv
source .venv/bin/activate
pip install -r HOMIE-toolkit/requirements.txt huggingface_hub hf_xet
git clone https://github.com/ChaoYue0307/ropedia-xperience-10m-task-suite.git
pip install -r ropedia-xperience-10m-task-suite/requirements.txt
pip install torch
hf download ropedia-ai/xperience-10m-sample \
--repo-type dataset \
--local-dir data/sample/xperience-10m-sample
cd ropedia-xperience-10m-task-suite
export WORKSPACE=/path/to/workspace
python scripts/episode_task_suite.py --workspace "$WORKSPACE" --include-neural
python scripts/research_direction_extension_tasks.py
python scripts/tier2_task_suite.py --workspace "$WORKSPACE"
python scripts/build_unified_task_suite.py
python scripts/task_walkthroughs.py
python scripts/build_evaluation_protocol.py
python scripts/generate_visualizations.py
python scripts/render_overview_figures.py
python scripts/render_task_suite_infographic.py
python scripts/export_modality_atlas_assets.py
python scripts/validate_website_integrity.py
python scripts/validate_scope_claims.py
python scripts/build_artifact_index.py
python scripts/validate_mirror_parity.py
python scripts/validate_publication_package.py