public sample episode / multimodal task lab

Ropedia Xperience-10M Task Suite.

A public research surface for physical 4D intelligence: sample-to-task contracts, multimodal baselines, verified multi-episode Qwen3-Omni results, and a clear path toward Xperience-native foundation models. Public raw-data boundaries stay explicit while staged derived artifacts keep the latest model evidence recoverable.

Start guided path See current results Inspect 20 tasks Open explorer
Understand Project in 5 minutes Brief, scope boundary, public sample, and what is complete today. Inspect Tasks and results 20 task contracts, radar views, method rows, and source audits. Reproduce Run the public pipeline Commands, validators, packaged mirrors, and release checks. Extend Train next model branches Spatial intelligence, human-video world models, VLA, and scale-up plans.
5,821frames in sample episode
1,16120-frame windows
8,546feature dimensions
20unified task contracts
home radar comparison 180 method-task records / 180 scored axes / metadata+raw128 complete with documented proxy axes
Unified 20-task model radar with full task-name key, method legend, 20-record counts, scored-axis counts, and raw128 proxy notes

Model comparison uses explicit task names and method contracts.

The full SVG names every axis and keeps 20-result records, scored-axis counts, raw metric sources, and proxy notes attached to the same comparison view.

180method-task records
180scored axes
80/80metadata+raw pass
34,269128ep windows
Minimal + Neural MLPSingle public-sample episode, full 20-task filled polygons.
128ep Metadata + Raw BaselinesEvery method has 20 records; metadata and raw NPZ heads now score all 20 axes, with compact proxy notes attached where the paired raw target is absent.
Qwen3-Omni + CosmosVerified held-out branches carry 20 records; Qwen3, Cosmos3-Super, and Cosmos3-Nano are now represented as 20/20 scored task rows with source/proxy notes attached in the matrix.
01 Action Recognition 05 Hand Trajectory Forecasting 08 Language Grounding 12 Multimodal Sync Detection 15 Interaction Text Prediction 18 IMU-to-Hand Pose Reconstruction 19 Camera-View Sync Retrieval 20 Time-to-Next-Transition Regression
Open full radar Open SVG 1-episode radar 128ep radar Open radar JSON Open 20-result matrix Open gap audit
You are in: Start / Project Overview Use the section pills to move within this layer, or jump to the full reader map. Reader map

Project overview and contributions.

The page is organized like a compact research project: motivation and scope, dataset sample, task suite, method, baselines, research directions, interactive walkthroughs, and resources for continuing the work. The public sample is used as a real but bounded research system, not as a final full-dataset benchmark.

01 understand Start with scope and status

Use this route if you need the project story, what is public, and what claims are safe.

Read overview
02 inspect Follow the task evidence

Use the 20-task suite, radar, matrix, and source audit to compare methods without losing metric provenance.

Open task suite
03 reproduce Run or verify the release

Use scripts, validators, mirrors, and checks when you want to rerun or trust the public package.

Open reproduce path
04 extend Choose the next model branch

Use directions and scale-up resources for spatial, world-model, VLA, Qwen3, and Cosmos follow-up work.

Open directions
Public reader map

Choose the right entry point without losing the evidence trail.

The project keeps source code, visual explanation, derived artifacts, model outputs, and release checks on different public surfaces. This map shows what each surface is responsible for before you dive into the full file set.

overview Understand the project quickly

Start with the brief and status files, then use the dashboard for the visual story.

brief status JSON
benchmark Inspect the 20-task suite

Use the task contract, protocol, walkthroughs, and radar matrix to follow each scored axis.

20 tasks task JSON matrix
sample data Understand one data sample

Open the sample explorer, raw-file manifest, and feature manifest before reading model scores.

explorer files features
results Compare methods cleanly

Single-episode baselines, 128-episode aligned baselines, Qwen3, and Cosmos branches stay separated by evidence type.

takeaways radar data model comparison
directions Read the three foundation pipelines

Spatial intelligence, human-video world modeling, and vision-language-action are documented as trainable directions with task mappings.

directions pipeline JSON
release health Verify public copies

Publication checks validate source alignment, package contents, mirror parity, and live URL/hash status.

reader map JSON surface QA live status
GitHub repoSource of truth for docs, scripts, generated data, validators, and commit history. GitHub PagesVisual dashboard for sample, tasks, results, directions, and resources. HF SpaceHub-hosted dashboard and static app assets. HF artifactsPublic-safe derived reports, metrics, website JSON, and result packages. HF baselinesCompact baseline weights, figures, metrics, and mirrored task artifacts. HF weights + resultsConsolidated baseline weights, adapters, result summaries, analysis, and manifest. HF collectionGrouped project surfaces, baseline repos, and verified model-branch repos.
Open full reader map Open reader-map JSON Open artifact index
Project brief

From one public episode to an extensible embodied-AI task lab.

Xperience-10M is much larger than the public sample. This project focuses on the sample available now, turns it into clear task contracts and baseline artifacts, and keeps the same data contract ready for held-out multi-episode training when more episodes are prepared.

What this is

A research-development lab for understanding synchronized egocentric multimodal data, defining embodied-AI tasks, and testing small baselines before omni-model fine-tuning.

What is implemented
  • 1,161 aligned windows from one public sample episode
  • 20 unified task contracts with minimal and neural evidence
  • Tasks 13-20 aligned to the same setup as tasks 1-12
  • Four research-direction maps and extension probes
What comes next

The next model-quality stage is stronger action/subtask modeling on the same held-out split, using dense/multiscale windows before requiring more raw episodes.

Data understanding

Maps one public episode into synchronized windows across video, audio, depth, pose/SLAM, mocap, IMU, calibration, and language-derived signals.

Task design

Defines embodied-AI inputs, process modules, outputs, metrics, and case-study walkthroughs instead of treating the sample as a generic classification file.

Evaluation discipline

Keeps chronological splits, predictions, confusion matrices, leakage notes, and single-episode limitations explicit before claiming broader model quality.

Scale-up readiness

Connects the same data contract to 128-episode baselines, a no-new-episode enhancement pack, Qwen3-Omni LoRA, Cosmos-style world modeling, policy-model branches, and the later Xperience-native pretraining goal.

Open interactive roadmap Read the brief Project summary Roadmap summary Reader path Current takeaways

1-Episode 20-Task Radar

Minimal and Neural MLP baselines over the original public-sample episode, with 40/40 scored method-task records.

Single-episode 20-task radar comparing Minimal and Neural MLP across all 20 scored task axes
Open SVG Open JSON

128-Episode 20-Task Radar

Metadata, raw-feature, Qwen3-Omni, and Cosmos3 branches on the aligned 128-episode surface, with all 140 rows scored and proxy/evidence notes kept explicit.

128-episode 20-task radar comparing raw-feature baselines, metadata baselines, Qwen3-Omni, and Cosmos3 branches with explicit scored-axis counts
Open SVG Open JSON
featured

Interactive research roadmap

Use this as the front door for the project: it links the unified 20 tasks, four research tracks, current sample evidence, and the multi-episode Qwen3-Omni scale-up path.

tracks 4 tasks 20 tasks 13-20 aligned roadmap phases 5
Open roadmap Roadmap structure
verified

Multimodal episode pipeline

One Xperience-10M public sample episode is converted into aligned windows and a documented feature contract.

frames 5,821 windows 1,161 features 8,546
verified

Task suite and baseline heads

The unified task suite has minimal baseline evidence, and the original task cards plus tasks 13-20 share the same windows, splits, and label discipline.

tasks 20 original neural heads 12 tasks 13-20 8
verified

Dataset source alignment

The public description is aligned to the official gated Xperience-10M dataset card, including modalities, scale, access, and current project coverage. The source snapshot records 31.9 TB on the HF surface, an about-1PB full-scale storage statement, 12,103 episode folders as upstream metadata, not a local data inventory, public sample license cc-by-nc-4.0, HOMIE Toolkit and Rerun 0.29.0 source tooling, and the official limited diversity note. See data/source_alignment_audit.json.

full dataset gated sample scope 1 episode raw data mirrored no
verified

Public research artifacts

Metrics, figures, walkthroughs, baseline weights, Qwen3-Omni results, and Cosmos3 public-safe packages are staged across GitHub, GitHub Pages, and Hugging Face.

tasks 20 baselines minimal + neural reader path tabs
verified diagnostic

Qwen3-Omni held-out pilot

The first selected-episode LoRA pilot is packaged with real held-out predictions and metrics. It proves the pipeline, while the weak scores make it a baseline for error analysis.

split 96 / 16 / 16 test windows 4,032 JSON validity 99.90%
current plan

128-Episode Task Suite Enhancement Pack

Shows how the current selected split can be stressed without more episodes: dense windows, hierarchical labels, raw-feature shards, and `multiscale_20s10_40s20_80s40` as the next export target.

current windows 3,808 multiscale estimate 106,095 data file task_suite_enhancement_128.json
not redistributed

Data governance

Raw MP4/HDF5/RRD files, private gated Xperience-10M data, and full Qwen weights are excluded from the public repo and HF mirrors.

raw Xperience-10M excluded full Qwen weights excluded derived artifacts included
Project brief Project summary Project status Current status Roadmap summary Reader path Project materials

Research roadmap.

The project path moves from the current public-sample task lab to the latest verified Qwen3-Omni diagnostic branch, same-split 128-episode baseline alignment, a no-new-episode enhancement pack, action/subtask error analysis, robustness runs, world/policy branches, and the future Xperience Embodied Foundation Model pretraining goal.

implemented

Public-Sample Task Lab

One public episode is converted into aligned windows, task contracts, minimal baselines, neural heads, walkthroughs, and figures.

Entry

Public Xperience-10M sample episode available.

Evidence

Status, protocol, takeaways, summary metrics, and episode-task outputs.

implemented

Multi-Episode Data Preparation

Prepare official gated episodes while preserving episode-level separation and recording missing-view coverage. The first selected split is available for Qwen3-Omni diagnostics.

Entry

Gated data access and enough storage for selected episodes.

Evidence

Selected-episode plan, data boundary, preparation notes, and verified package summary.

verified latest branch

Qwen3-Omni LoRA Latest Diagnostic Branch

Train lightweight adapters on selected prepared episodes and evaluate on held-out episodes with committed predictions, metrics, and run reports.

Entry

Selected episodes prepared with no train/test episode leakage.

Evidence

Verified result summary, v5/v6 comparison, dataset manifest, training metadata, progress logs, metrics, and predictions.

verified companion result

128-Episode Same-Split Simple/NN Baselines

Align simple metadata/text baselines, raw-feature proxies, and neural MLP baselines to the same selected 96/16/16 split and the unified 20-task axes used by the public result matrix.

Entry

Derived Qwen JSONL export for the selected 96/16/16 split.

Evidence

Baseline alignment report, summary metrics, task metrics, and the 128-task baseline runner.

current no-new-episode plan

128-Episode Task Suite Enhancement Pack

Use the same selected split, estimate dense/multiscale window exports, define hierarchical action/subtask targets, and prioritize raw-feature shards for tasks that metadata baselines cannot cover.

Entry

Current 3,808-window selected 96/16/16 export and verified Qwen v4 metrics.

Evidence

TASK_SUITE_ENHANCEMENT_128.md, task_suite_enhancement_128.json, dense-window CSV, and the enhancement builder script.

active next step

Action/Subtask Error-Analysis Pass

Keep the 96/16/16 split, tighten JSON decoding or target formatting, and analyze action/subtask failures before larger model-quality claims.

Entry

The final diagnostic package is verified, meets strict JSON validity, and exposes weak action/subtask quality.

Evidence

Updated quality-target report, error-analysis tables, held-out metrics, and public-safe package.

current

Foundation-Model Selection Matrix

Keep Qwen3-Omni as the first trainable held-out pilot, use Cosmos 3 for world modeling and forward-dynamics trainer development, and stage policy candidates after robot-compatible action targets are explicit.

Entry

Completed 128-episode preparation or a smaller 3-8 episode preprocessing dry run.

Evidence

Foundation model plan, source links, model-specific entry conditions, and evaluation additions.

partially implemented

64-128 Episode Robustness Run

Test whether pilot conclusions survive broader sessions, missing modalities, and stronger ablations.

Entry

Selected multi-episode pilot trains and evaluates cleanly.

Evidence

Metrics by session, task, modality, ablation, and failure type.

planned

Cosmos 3 and Policy-Model Extensions

Extend toward future-window prediction, action-conditioned world modeling, synthetic-data tests, policy-style next action, and affordance reasoning.

Entry

Enough multi-episode data, compute budget, and model-specific action or world-state targets.

Evidence

Task-specific held-out evaluations, qualitative inspection, and updated model cards.

future

Xperience Embodied Foundation Model Pretraining

Pretrain an Xperience-native domain model over synchronized video, audio, depth, pose, mocap, IMU, and language after smaller scaling stages prove value.

Entry

Full-corpus access, PB-scale storage path, multi-node compute, and positive scaling evidence.

Evidence

Pretraining manifests, scaling curves, held-out evaluations, checkpoint inventory, model card, and data-boundary report.

interactive roadmap roadmap document roadmap stages foundation model plan three foundation pipelines additional directions task_suite_enhancement_128.json pipeline track note native pretraining plan interactive map scale-up status project status

Additional development directions.

Beyond the current task heads, Qwen3-Omni fine-tuning path, Cosmos/world-model branch, and future native pretraining goal, Xperience-10M can support three foundation pipeline tracks plus several concrete research-development tracks.

High-resolution slide diagram showing the Spatial intelligence models direction for Xperience-10M.
High-resolution direction slide

Spatial intelligence models

Train spatial-memory models from multiview RGB, egocentric video, depth, pose, calibration, object/contact cues, and language prompts; evaluate spatial QA, object permanence, counting, retrieval, and pose-aware consistency.

Track JSON Open image
High-resolution slide diagram showing the Human-video world models direction for Xperience-10M.
High-resolution direction slide

Human-video world models

Train future-prediction models from observed interaction windows to score next action, next subtask, future object set, contact transition, camera-motion delta, and latent future state, with Qwen-style probes and Cosmos-style dynamics kept separate.

Track note Open image
High-resolution slide diagram showing the Vision-language-action models direction for Xperience-10M.
High-resolution direction slide

Vision-language-action models

Train VLA or policy-compatible heads only after converting egocentric video, captions, hand/body motion, contacts, objects, and procedures into traceable action tokens, chunks, and object-conditioned action targets.

Track contract Open image

Episode taxonomy and data engine

Build an episode atlas, category tags, balance report, and split builder across activities, objects, scenes, sessions, people, and missing modalities.

direction data

Standardized benchmark protocol

Version train/val/test manifests, task cards, leakage checks, metric scripts, and reference baselines so future model scores are comparable.

direction note

Multimodal representation learning

Train contrastive and masked-prediction encoders over synchronized video, audio, depth, pose, mocap, IMU, and language windows.

JSON plan

Skill and procedure graphs

Mine action steps, transitions, preconditions, effects, and temporal graphs that connect egocentric perception to planning.

current task map

Human-object affordances

Add contact, reachable-object, tool-use, and next-affordance tasks using hands, mocap, objects, contacts, video, and language.

task walkthroughs

3D/4D scene and object memory

Fuse depth, pose/SLAM, multiview video, and object cues into persistent scene/object maps for spatial reasoning and object permanence.

model branches

Quality and sync diagnostics

Track timestamp drift, missing streams, calibration consistency, corrupted files, and degraded-mode manifests before large training runs.

evidence contract

Policy and simulation transfer

Convert mocap, hand trajectories, contacts, and object states into action tokens, robot-compatible targets, and imitation-learning examples.

foundation plan

Evaluation protocol is explicit.

The protocol is generated from committed metric artifacts so readers can see the exact data unit, split, task targets, leakage controls, and current limitations before comparing scores.

Data unit

One 20-frame aligned window from the public sample episode, stride 5 frames, 1,161 windows total, represented by 8,546 synchronized multimodal dimensions.

evaluation protocol

Split policy

Single-episode chronological 70/30 train/test split. This avoids random future-window mixing; cross-episode generalization is measured in the later multi-episode pilot.

protocol document

Metric contract

All 20 tasks list input, target, primary metric, baseline score, and source artifact path in the unified suite file.

task_suite_20.json

Leakage controls

Scalers fit on train windows only; future labels, target-side signals, caption/object labels, and contact labels stay on the target side unless explicitly queried.

builder script

Audio ablation

Audio and no-audio variants are evaluated across the original task contracts under the same chronological split.

audio summary

Foundation branch selection

Qwen3-Omni is the first trainable baseline, Cosmos 3 becomes the world-model branch with a camera-pose proxy forward-dynamics contract ready for trainer work, policy models wait for robot-compatible action targets, and Xperience-native pretraining remains a later full-corpus goal.

backbone plan

Next evaluation stage

This public-sample run covers single-episode task development. The selected multi-episode Qwen3-Omni final diagnostic result is verified and meets the JSON-validity target; Cosmos3-Nano has a verified future-window compatibility package; and Cosmos3-Super has a verified base-weight JSON-task evaluation plus a fine-tuned forward-dynamics LoRA branch. The next stage is action/subtask error analysis, stronger model-quality runs, and policy-target conversion.

result comparison

128-Episode Task Suite Enhancement Pack

Before adding episodes, the suite should try `multiscale_20s10_40s20_80s40`, hierarchical action/subtask targets, label-normalized scoring, and compact raw-feature shards for unsupported tasks.

task_suite_enhancement_128.json

Scale-up requirement

Future Omni, Cosmos, and policy branches use the same episode split discipline, training metadata, held-out predictions, metrics, run report, and public-safe package gate.

scale-up status

Current experiments and next milestones.

The project shows the completed public-sample task suite and the first verified multi-episode Qwen3-Omni diagnostic pilot, then lays out the next quality-improvement and model-extension steps.

verified

Aligned Xperience-10M sample windows

5,821 frames become 1,161 synchronized 20-frame windows with an 8,546-dimensional representation.

task results feature inputs
verified

20 task contracts + 180 public results

The current release reports nine method families over the unified 20-task axes, with minimal, neural, 128-episode, Qwen3, Cosmos3, and proxy-scored rows kept source-linked.

task artifacts neural MLP outputs
verified

Audio contribution is measured task by task

Audio variants improve the primary metric on 6 of the original task contracts in this single-episode setting.

audio summary delta chart audio findings
verified

Four research directions are mapped by evidence type

The Ropedia directions are labeled as direct, proxy, or diagnostic coverage, plus one coded extension probe per direction.

direction map extension probes
current plan

Foundation backbones are separated by role

Qwen3-Omni stays first for held-out LoRA; Cosmos 3 is the world-model branch with camera-pose proxy forward-dynamics targets ready for trainer work; OpenVLA/openpi/GR00T are policy candidates after robot-compatible action conversion; Xperience-native pretraining is the later full-corpus goal.

foundation model plan plan doc pretraining plan
verified diagnostic

Qwen3-Omni and Cosmos3 branches

The selected 96/16/16 episode split now has a verified Qwen3-Omni v6 package with 4,032 held-out test predictions and 99.90% JSON validity. Cosmos3-Nano has 378 held-out future-window predictions, Cosmos3-Super Reasoner has 448 held-out base-weight JSON-task predictions, and Cosmos3-Super Forward-Dynamics LoRA has 448 held-out loss records.

result comparison pilot result verified package
verified

Multi-episode pilot status is explicit

The Qwen3-Omni notes separate earlier diagnostic packages, the final 128-episode LoRA result, and the next action/subtask error-analysis pass.

training status LoRA adapter
verified

Figures are indexed

The visual set includes the logo, modality atlas, task-suite figure, unified 20-task model radar, model-architecture figure, tasks 13-20 chart, and Qwen3-Omni LoRA training-flow figure.

figure guide task-suite figure 20-task radar LoRA figure
verified

Brand assets are packaged consistently

The project logo is used consistently in the website header, favicon, README/HF cards, and social preview.

logo card logo mark
verified

Raw dataset files are not redistributed

The public project shares derived task artifacts, figures, reports, and lightweight baseline files. Raw Xperience-10M videos, HDF5 annotations, RRD visualizations, gated data, and full Qwen weights stay outside the repo.

data notice official dataset
verified

The dashboard is designed as the visual entry point

Tabs organize the sample data, 20 tasks, model method, results, research directions, and next-stage resources.

dataset tasks directions
verified

Reproduction path is documented

The reproduction guide lists the public sample setup, task-suite rebuild, neural heads, figure generation, and expected outputs.

reproducibility latest rebuild
verified

Official dataset source is linked

The project keeps the official Xperience-10M dataset, public sample, dataset website, and HOMIE toolkit visible so readers can trace the data source.

dataset website official dataset HOMIE toolkit

Research reading path.

A newcomer should be able to move from the dataset sample to the task design, model baselines, current limitations, and scale-up plan without reading every file first.

02

Inspect one model input

Use the window table and feature manifest to see the aligned sample unit, modality sources, and leakage controls.

windows features
03

Compare minimal vs neural heads

Every task has a small interpretable baseline and a matching neural MLP head over the same feature contract and chronological split.

summary metrics neural heads
04

Check the scale-up gate

The multi-episode Qwen3-Omni path now has a final verified diagnostic package and public LoRA adapter. The native-pretraining plan shows how this can grow into a full-corpus research direction after action/subtask improvements and stronger task metrics.

scale-up status data access native pretraining reader path
Verified nowOne public episode, 5,821 frames, 1,161 aligned windows, 8,546 dimensions, 20 unified task contracts, 12 original neural heads, and 4 direction-extension probes.
Next: no-new-episode scaleThe selected 128-episode suite should next use dense/multiscale windows, hierarchical labels, and raw-feature shards before adding more episodes.
Next: error analysisThe selected 128-episode Qwen3-Omni LoRA result has a final verified diagnostic package; JSON validity meets target, and the next pass should improve action/subtask quality.
Not redistributedRaw videos, raw annotations, full Qwen weights, and private gated Xperience-10M data are not included in the public repo or HF bundles.

Aligned with the official dataset card.

The official Xperience-10M card describes a gated, large-scale 4D egocentric multimodal dataset. This project records that full upstream scope while focusing the implemented artifacts on one public sample episode. The source-alignment record keeps 31.9 TB, about-1PB, 12,103 episode folders, cc-by-nc-4.0, HOMIE Toolkit, Rerun 0.29.0, not a local data inventory, limited diversity, and data/source_alignment_audit.json visible on the public site.

Official dataset

Xperience-10M is a gated large-scale egocentric multimodal dataset for embodied AI, robotics, spatial intelligence, and world modeling.

official HF dataset

Public sample

The current unified 20-task suite is built from one public sample episode, not from the entire gated dataset.

sample dataset

Modalities

The sample exposes synchronized video, audio, depth, pose/SLAM, motion capture, inertial signals, calibration, and language annotations.

modality atlas

Multi-episode pilot

The selected 128-episode Qwen3-Omni LoRA v6 diagnostic branch is verified with 4,032 held-out test predictions and 99.90% JSON validity. Action/subtask metrics are still weak, so this remains a baseline for error analysis.

LoRA adapterv5/v6 comparison

Raw sample browser

The Data tab now exposes the official public sample files directly, including playable MP4 video streams and the audio track embedded in fisheye_cam0.mp4.

open raw browserraw manifest

Data boundary

Raw MP4, HDF5, RRD files are streamed from the official public sample source when opened here; private gated data and full Qwen weights are not redistributed in this project.

data notice

Current project subset

One public sample episode, 5,821 frames, 1,161 aligned windows, 8,546-dimensional task inputs, plus direct links to the official raw sample files.

modality atlas

Covered now

Action/subtask labels, next-action prediction, temporal diagnostics, hand trajectory, contact, object relevance, caption grounding, retrieval, reconstruction, misalignment, long-horizon forecasting, interaction text, action-object relation, sensor bridging, camera sync, and transition timing.

summary metrics

Responsible use

This project is for research exploration and excludes identity recognition, surveillance, biometric profiling, sensitive-attribute inference, and safety-critical deployment.

use notes

Later milestones

Full audio-visual learning, caption generation, depth-pixel prediction, SLAM estimation, neural rendering, policy learning, cross-episode generalization, held-out Qwen3-Omni evaluation, and future Xperience-native pretraining.

native pretraining

Raw public sample browser.

Open each official Xperience-10M sample file from the project page. Video and audio use compact browser previews derived from the official MP4 files, with direct links beside them for the full raw Hugging Face sources. HDF5 and RRD files are shown with their role, size, organization, and direct source links.

fisheye_cam0.mp4

Fisheye camera 0 stream and the public sample audio source. This file can be played as video and as the embedded audio track.

video + audio

Playing a 12 second fast-start preview derived from the official raw MP4. Use the source link for the complete file.

Embedded audio preview from fisheye_cam0.mp4

Video features feed visual tasks; the embedded audio stream feeds audio ablation and acoustic feature blocks.

open playable preview open full raw source official sample dataset machine-readable manifest

Sample folder organization

The official public sample is one episode folder. The task suite reads the HDF5 annotations and six synchronized MP4 streams, then writes 20-frame windows with a 5-frame stride.

xperience-10m-sample/
  annotation.hdf5
  fisheye_cam0.mp4
  fisheye_cam1.mp4
  fisheye_cam2.mp4
  fisheye_cam3.mp4
  stereo_left.mp4
  stereo_right.mp4
  visualization.rrd

annotation.hdf5 group map

The raw HDF5 is a binary container, so the browser shows its organization rather than loading the whole file into memory.

calibrationCamera intrinsics/extrinsics and static alignment values.
captionJSON text with actions, objects, interactions, segments, and global summary.
depthDepth maps and confidence channels aligned to the episode timeline.
full_body_mocapFull-body joint and contact signals for human motion modeling.
hand_mocapLeft and right hand joint trajectories used by forecast tasks.
imuAccelerometer and gyroscope streams sampled above video rate.
metadataEpisode metadata, frame indexing, and source bookkeeping.
slamCamera trajectory, pose, and sparse SLAM point-cloud information.
videoVideo metadata and per-frame alignment information.

Ropedia Xperience-10M Unified 20-Task Suite.

The suite connects synchronized multimodal windows to 20 task contracts. The large map visualizes the original task families, while tasks 13-20 are listed as the aligned continuation under the same setup.

Infographic showing Ropedia Xperience-10M task families with enlarged full-width modality cards

Unified plus split radars

The unified radar keeps all 9 methods in one view. The two split radars separate the clean 1-episode Minimal/NN baseline comparison from the 128-episode metadata/raw/Qwen/Cosmos comparison.

Metric normalization

Higher-is-better metrics are plotted directly on 0-1 axes. Lower-is-better metrics are converted to best/value within the task, while raw values, status reasons, sources, and the two raw128 compact proxy notes remain in the JSON mirrors.

Score gap audit

The matrix has 180 method-task records and 180 numeric scores. The gap audit remains published as the evidence ledger for which artifacts support each score, including documented proxy axes where raw targets are absent.

Unified 20-task radar comparing Minimal, Neural MLP, 128-episode metadata/raw baselines, Qwen3-Omni, and Cosmos3 with task names, method details, 20-record counts, scored-axis counts, and proxy notes

1-Episode 20-Task Radar

Minimal and Neural MLP are both scored on all 20 public-sample task contracts, shown as two filled polygons without 128-episode overlays.

Single-episode 20-task radar comparing Minimal and Neural MLP across all 20 scored task axes
Open SVG Open JSON

128-Episode 20-Task Radar

Raw128 Simple and Raw128 NN score all 20 axes; metadata, Qwen3, and Cosmos branches keep 20 records but only plot evaluated numeric targets.

128-episode 20-task radar comparing raw-feature baselines, metadata baselines, Qwen3-Omni, and Cosmos3 branches with explicit scored-axis counts
Open SVG Open JSON Gap audit

Readable modality atlas.

Each Xperience-10M stream gets a large thumbnail, a plain sample-content line, and the exact current-baseline use. These are small derived images only; no raw MP4, HDF5, or RRD data is redistributed.

modality atlas
01

Video

visual stream
Public sample fisheye and stereo camera thumbnails
sample contains

6 synchronized camera MP4 streams

current baseline use

RGB/fisheye/stereo frame statistics

02

Audio

acoustic stream
AAC waveform thumbnail from the public sample MP4 stream
sample contains

Audio stream embedded in MP4

current baseline use

Acoustic signal

03

Depth

geometry map
Public sample depth and confidence thumbnails
sample contains

Depth map + confidence channel

current baseline use

Spatial geometry signal

04

Pose / SLAM

camera pose
Public sample camera trajectory and sparse SLAM map thumbnail
sample contains

Trajectory + sparse SLAM map

current baseline use

Position + orientation features

05

Motion Capture

human motion
Public sample body and hand motion capture thumbnail
sample contains

Body + hand joint tracks

current baseline use

3D mocap feature statistics

06

Inertial

wearable sensor
Public sample accelerometer and gyroscope time-series thumbnail
sample contains

Accelerometer + gyroscope

current baseline use

Wearable motion statistics

07

Language

semantic annotation
Public sample object tags and action caption thumbnail
sample contains

Object tags + action captions

current baseline use

Task labels + semantic targets

The atlas redistributes only small derived thumbnails and metadata. Raw MP4, HDF5, and RRD files remain excluded from this repo and the Hugging Face mirrors.

From raw episode to research artifacts.

Every script works from one data contract: aligned multimodal windows, explicit labels, cached feature extraction, and a manifest that makes omitted modalities visible.

Verified Xperience-10M multimodal pipeline diagram

Qwen3-Omni LoRA training flow

Raw valid episodes move through split validation, parallel export, video/audio/text formatting, sensor-bridge features, LoRA training, and sealed held-out evaluation.

What the figure represents

It documents the selected 128-episode final diagnostic result and the action/subtask improvement path needed for stronger model-quality numbers.

Detailed Qwen3-Omni LoRA training pipeline from raw Xperience-10M episodes to adapter outputs, predictions, metrics, and reports

What this project enables

It demonstrates the full development loop: reading Xperience-10M sample data, aligning modalities, converting them into model-ready windows, defining meaningful tasks, producing metrics, and packaging artifacts for continued research.

What still needs more data

General embodied-intelligence model quality requires many episodes and held-out episode splits; the public sample is the development harness for that next stage.

What the current results actually say.

A generated takeaways layer reads the committed metrics, summarizes useful research signals, and identifies what still needs held-out episodes.

One episode becomes a benchmark contract

The public sample is converted into 5,821 frames, 1,161 aligned 20-frame windows, and an 8,546-dimensional representation for repeatable task evaluation.

research takeaways

Chronological split exposes class shift

All-feature action reaches 0.9829 macro-F1 on its local split, while the chronological action head in the core task suite is 0.0500 macro-F1 with four unseen later action labels.

takeaways

Neural heads help dynamics

Hand MPJPE improves from 0.8647 to 0.1079; temporal-order F1 rises from 0.5400 to 0.8520; misalignment F1 rises from 0.5052 to 0.7153.

metrics

Retrieval and reconstruction remain open

Ridge/cosine retrieval remains stronger than the neural projection here, and cross-modal feature reconstruction still has negative R2.

retrieval metrics

Scale means held-out episodes

The next credible model-quality unit is a held-out multi-episode pilot across different sessions, not more adjacent windows from one sample.

scale-up status

Small baselines, no hidden machinery.

Motion-only and current all-feature classifiers use lightweight heads so the comparison stays readable on a laptop and easy to inspect. The neural run keeps the same features and splits, then swaps in PyTorch MLP heads.

Motion-only action

0.9688macro-F1, 18 classes

Current all-feature action

0.9829macro-F1, 8,546 dimensions

Motion-only subtask

0.9528macro-F1, 14 classes

Current all-feature subtask

0.9173macro-F1, chronological caveats
Macro-F1 comparison chart

Neural MLP heads, same task contracts.

The neural baseline uses small PyTorch MLP classifiers/regressors on the same 8,546-dimensional windows, chronological splits, and leakage filters. This isolates the value of a nonlinear head before moving to heavier Qwen/Omni experiments.

Neural hand forecast

0.1079MPJPE, down from 0.8647 minimal

Neural temporal order

0.8520F1, adjacent-window diagnostic

Neural misalignment

0.7153F1, shifted motion/visual/audio pairs

Neural cross-modal retrieval

0.1300MRR; ridge remains stronger here
Neural MLP episode task score chart Minimal versus neural MLP episode task score chart

The original tasks organized into four research directions.

Each task is mapped as direct, proxy, or diagnostic evidence for the Ropedia research tracks. The mapping uses two current baselines: minimal interpretable heads and neural MLP heads over the same feature contract.

partially implemented

A. Human Modeling & Motion Understanding

Direct evidence comes from hand trajectory forecasting and contact prediction; action and object relevance are supporting proxies.

2direct2proxy0diagnostic
proxy tasks only

B. 3D/4D Reconstruction & Neural Rendering

Cross-modal retrieval, modality reconstruction, and misalignment detection check reconstruction prerequisites, not full geometry.

0direct2proxy1diagnostic
strongest implemented

C. Egocentric Vision & Interaction

Action, subtask, transition, next-action, object, caption, order, and alignment tasks directly stress egocentric understanding.

6direct2proxy3diagnostic
early proxy tasks

D. Scene Reconstruction & World Modeling

Current probes cover task state, object relevance, retrieval, reconstruction, temporal order, and alignment but no persistent map yet.

0direct6proxy3diagnostic
Coverage of the original Xperience-10M tasks across four research directions

Baseline 1: minimal heads

Softmax, logistic, ridge, and retrieval heads keep every input/output contract readable. They are the first sanity check for whether a task is well-posed.

Baseline 2: neural MLP heads

Small PyTorch MLP classifiers/regressors reuse the same features and splits. They test nonlinear gains before heavier Omni fine-tuning.

Tasks 13-20 complete the unified 20-task suite.

The original four direction probes remain as focused examples. Tasks 13-20 add eight sample-supported baselines using the same windows, feature manifest, chronological split, and minimal/neural head pattern as tasks 1-12.

Eight Xperience-10M tasks 13-20 with minimal and neural metrics
Task 13 / forecast

Long-Horizon Next-Action Forecasting

Input: current non-caption multimodal window.

Output: action label five seconds later.

0.0750minimal macro-F10.0655neural macro-F1
Task 14 / procedure

Long-Horizon Next-Subtask Forecasting

Input: current non-caption multimodal window.

Output: procedure subtask five seconds later.

0.0455minimal macro-F10.0507neural macro-F1
Task 15 / language

Interaction Text Prediction

Input: current sensor window with caption features removed.

Output: raw annotation interaction phrase.

0.0444minimal macro-F10.0381neural macro-F1
Task 16 / relation

Action-Object Relation Prediction

Input: current sensor window with caption features removed.

Output: joint action plus active object-set label.

0.0000minimal macro-F10.0000neural macro-F1
Task 17 / objects

Future Object-Set Forecasting

Input: current sensor window with caption features removed.

Output: object set active five seconds later.

0.1694minimal micro-F10.1972neural micro-F1
Task 18 / sensor bridge

IMU-to-Hand Pose Reconstruction

Input: IMU acceleration and gyroscope features only.

Output: current left/right hand joint feature blocks.

0.0420minimal MAE0.0426neural MAE
Task 19 / camera sync

Camera-View Synchronization Retrieval

Input: fisheye camera-1 feature query.

Output: synchronized fisheye camera-3 window rank.

0.4943minimal MRR0.2409neural MRR
Task 20 / timing

Time-to-Next-Transition Regression

Input: current non-caption multimodal window.

Output: capped frames until the next action boundary.

10.5374minimal MAE frames10.5545neural MAE frames

Tasks 13-20 artifact package

The eight-task package has JSON metrics, prediction/rank files, a Markdown summary, and a chart generated from the local public-sample annotation and committed shared-window tensor.

Open unified 20-task JSON · tasks 13-20 result JSON

Setup alignment

Tasks 13-20 use the same 20-frame windows, 5-frame stride, 8,546-dimensional feature manifest, chronological split, and minimal/neural comparison pattern as tasks 1-12.

Four Xperience-10M research-direction extension probes with minimal and neural metrics
A / motion

Body and Hand Motion Intensity

Case: classify fast reach/pour windows as high motion and steady holding windows as low motion.

Input: non-mocap video, depth, pose, IMU, SLAM, calibration, and language features.

Output: high_motion or low_motion.

0.7827minimal macro-F10.7986neural macro-F1
B / views

Multi-View Consistency Retrieval

Case: retrieve the synchronized stereo-left window from a fisheye-camera query.

Input: fisheye_cam0 video features against stereo_left candidate features.

Output: ranked synchronized view candidates.

0.5534minimal MRR0.3469neural MRR
C / phase

Action Phase Progress Estimation

Case: estimate whether a Pour coffee window is near the start, middle, or end of its action segment.

Input: non-caption multimodal features.

Output: 0-to-1 progress inside the current action.

0.3416minimal MAE0.3038neural MAE
D / world

Short-Horizon Ego-Motion Forecasting

Case: predict how the camera translation changes over the next 20 frames.

Input: current sensors excluding camera translation and captions.

Output: future camera-translation delta vector.

0.1989minimal MAE0.0989neural MAE

What changed

The four research directions now have coded extension probes, prediction/rank CSVs, JSON metrics, a Markdown summary, and a website chart generated from real sample-window features.

What still needs scale

A full research result still needs many Xperience-10M episodes, held-out episode splits, stronger encoders, and direction-specific models such as body priors, renderers, or persistent scene graphs.

The original task heads share four head families.

The diagram separates the shared episode-window representation from the task-specific heads, so the task contracts stay readable before scaling to larger models.

Verified minimal and neural architecture diagram for Ropedia Xperience-10M task heads

Interactive task walkthrough.

Each task uses a common research name and a concrete case study, then opens into the input, middle modules, output, modality evidence, metric, and current limitation.

Representative sample modality for the selected task
Step 1 / 4 · Input
Action Recognition Egocentric Action Recognition

Input: inspect the 20-frame multimodal window before choosing the target.

01 / 12
supervised multiclass classifier

Action Recognition

In the coffee-making sample, a pouring window maps to the current action label.

    Metric: macro-F1. Minimal 0.0500; neural MLP 0.0148.

    Current limitation: single-episode chronological split.

    Task cards and metrics.

    The original task cards use readable research names, representative modality thumbnails, explicit input-process-output contracts, and verified minimal versus neural scores. The unified 20-task index adds tasks 13-20 in the same suite.

    Every model input has a source.

    The point is not hidden complexity. Every input group maps back to a source modality and a manifest entry.

    All modality source chart

    Diagnostics separate memorization from signal.

    The charts make the main lesson visible: within-episode supervised labels are easy under some splits, while retrieval, grounding, forecasting, and alignment remain the useful probes.

    Episode task suite score chart Cross modal retrieval chart Neural MLP task score chart Minimal versus neural score chart Measured audio delta chart across original task contracts

    Open the single-episode explorer to inspect window-level labels, predictions, modality statistics, object labels, and diagnostic scores. The audio ablation summary records the task-by-task audio contribution.

    Research artifacts for the next experiments.

    Metrics, predictions, manifests, lightweight model weights, and derived window artifacts are organized so the project can be inspected, extended, and scaled before rerunning the full pipeline. Raw Xperience-10M data and Qwen weights are not redistributed.

    download Find the right public surface

    Open GitHub, HF Space, artifact dataset, baseline models, or consolidated weights/results without guessing.

    Open public surfaces
    verify Check claims and parity

    Use validators, source alignment, mirror parity, and live URL/hash checks before trusting a number.

    Open checks
    reproduce Run the task pipeline

    Start from scripts, windows, feature manifests, task contracts, and minimal/neural result outputs.

    Open commands
    scale Continue model work

    Use Qwen3/Cosmos packages, 128-episode feature index, and foundation-model plans for the next runs.

    Open scale-up
    Research artifacts

    From one episode to task heads

    Start with the files that define the sample windows, modality inputs, task contracts, metrics, walkthroughs, and research-direction mapping.

    Task results

    Every task definition, split detail, feature dimension, and minimal/neural metric in one project output.

    task results

    Windows table

    Window start/end frames and aligned action/subtask labels for the public sample episode.

    window table

    Feature inputs

    Source map for the current modality inputs used by the task suite.

    feature inputs

    Neural MLP task results

    Per-task PyTorch MLP metrics, predictions, histories, and checkpoints for the original task contracts, with tasks 13-20 published in the aligned result bundle.

    neural MLP outputs

    Four-direction taxonomy

    Maps the original tasks to the four research tracks: human modeling, 3D/4D reconstruction, egocentric interaction, and world modeling.

    research direction outputs

    Direction extension probes

    Four coded probes, one per research direction, with minimal and neural metrics plus prediction/rank CSVs.

    extension probe outputs

    Task walkthroughs

    Case studies for the original tasks, including input, middle process modules, output, metric, limitation, and task-player data.

    walkthrough outputs

    Audio ablation and raw upgrade

    All 72 task/variant rows comparing current audio, no audio, raw audio, replacement, and combined-input settings.

    audio ablation outputs

    Single-episode explorer

    Interactive window-level view of labels, predictions, modality statistics, object labels, and diagnostics.

    open explorer

    Cross-modal retrieval

    The strongest self-supervised signal from the single episode.

    retrieval metrics

    Qwen3-Omni diagnostic branch is verified.

    The selected pilot uses 128 source-balanced episodes across 128 different session UUIDs. The latest v6 held-out package is verified, and its weak metrics define the next structured-output and error-analysis pass.

    Selection

    128 complete episodes selected from 128 unique top-level sessions, balanced across episode-size bands and split 96/16/16 for train/val/test.

    source/feature index

    Transfer

    Download raw episodes only from official gated sources, exclude visualization.rrd, validate files, then stage them for training.

    Current LoRA artifact

    The current Qwen3-Omni LoRA artifact is the verified v6 selected 128-episode diagnostic adapter. The v5 row remains pinned as the prior release, and the 1-episode Qwen entry is only a sensor-adapter smoke test.

    model groups

    128-Episode Task Suite Enhancement Pack

    The next suite push does not need more episodes first: use `multiscale_20s10_40s20_80s40`, hierarchical action/subtask targets, and raw-feature shards while keeping the held-out split fixed.

    task_suite_enhancement_128.json

    Backbone branches

    Qwen3-Omni uses a separate LoRA model repo; Cosmos3-Nano remains a compatibility package; Cosmos3-Super now has a verified forward-dynamics LoRA branch with weights in a dedicated model repo.

    Cosmos3-Super weights

    Native foundation model

    The long-term goal is a full-corpus Xperience Embodied Foundation Model trained on synchronized perception, geometry, motion, inertial, audio, and language streams after smaller scaling stages validate the approach.

    pretraining plan

    Reproduce the suite.

    Raw Xperience-10M data is not redistributed here. The reproduction guide states the commands, expected outputs, exact-match reproduction record, and multi-episode requirements.

    Reproducibility guide

    Human-readable commands, expected artifacts, and current scope for the public single-episode pipeline.

    reproducibility guide

    Reproducibility matrix

    Machine-readable command matrix covering sample download, baselines, the unified 20-task suite, figures, and validation.

    reproducibility matrix

    Exact-match reproduction record

    The last metric rebuild reproduced the public-sample outputs from a fresh cache and matched the committed metrics.

    reproduction audit

    Project dashboard

    The website organizes the dataset sample, tasks, methods, results, directions, and scale-up path in one tabbed reader flow.

    project materials

    Multi-episode pilot status

    The comparison JSON now supports both the three-version reading and model-family grouping, with Qwen3 v5/v6 detail kept as a separate machine-readable audit.

    comparisonQwen v5/v6

    Minimal path: install the toolkit dependencies, download the official sample, run the task suite with neural heads, regenerate tasks 13-20, build the unified 20-task index, regenerate visualizations, then rebuild the supporting project reports.

    git clone https://github.com/Ropedia/HOMIE-toolkit.git
python3.12 -m venv .venv
source .venv/bin/activate
pip install -r HOMIE-toolkit/requirements.txt huggingface_hub hf_xet
git clone https://github.com/ChaoYue0307/ropedia-xperience-10m-task-suite.git
pip install -r ropedia-xperience-10m-task-suite/requirements.txt
pip install torch

hf download ropedia-ai/xperience-10m-sample \
  --repo-type dataset \
  --local-dir data/sample/xperience-10m-sample

cd ropedia-xperience-10m-task-suite
export WORKSPACE=/path/to/workspace
python scripts/episode_task_suite.py --workspace "$WORKSPACE" --include-neural
python scripts/research_direction_extension_tasks.py
python scripts/tier2_task_suite.py --workspace "$WORKSPACE"
python scripts/build_unified_task_suite.py
python scripts/task_walkthroughs.py
python scripts/build_evaluation_protocol.py
python scripts/generate_visualizations.py
python scripts/render_overview_figures.py
python scripts/render_task_suite_infographic.py
python scripts/export_modality_atlas_assets.py
python scripts/validate_website_integrity.py
python scripts/validate_scope_claims.py
python scripts/build_artifact_index.py
python scripts/validate_mirror_parity.py
python scripts/validate_publication_package.py