GLOSSARY.md

Glossary

This glossary defines project-specific terms and adjacent technical field terms that can be easy to confuse across the GitHub repo, website, Hugging Face Space, artifact dataset, model repos, result matrices, and embodied-AI training discussions. Use it with PUBLIC_READER_MAP.md when choosing what to read first, and with docs/data/glossary.json when a tool needs the same terms in machine-readable form.

How To Read The Terms

Category	What it clarifies
Dataset and scope	Public data boundaries, evidence lines, and how each result family should be read.
Files and features	Raw sample files, windows, feature manifests, and public-safe derivatives.
Multimodal sensing	Video, audio, depth, IMU, motion capture, calibration, and synchronization terms.
Spatial geometry	Camera pose, SLAM, coordinate frames, point clouds, 3D reconstruction, and spatial grounding.
Temporal and world models	Future prediction, rollouts, forward dynamics, long-horizon forecasting, and temporal leakage.
Robotics and VLA	Vision-language-action, policies, action chunks, imitation learning, contact, and dexterity.
Tasks and metrics	Task contracts, scored records, direct scores, compact proxies, and audits.
Training and evaluation	Splits, held-out evaluation, metric types, prompt/schema checks, adapters, and distributed training.
Models and runs	Baseline families, Qwen3-Omni, Cosmos3, LoRA adapters, and full-parameter gates.
Public surfaces	GitHub, website, Hugging Face repos, parity checks, and package validation.

Core And Field Terms

Dataset and scope

Term	Plain meaning	In this project	Do not confuse with
Evidence line	A reading lane for a group of results.	Line 1 is one public sample episode; Line 2 is selected-128 held-out comparison.	Qwen run versions v1-v6, which are model-run lineage.
Official gated data	Upstream files that require official dataset access.	Raw Xperience-10M MP4/HDF5/RRD files and full source directories remain outside the public repo.	Public-safe metrics, derived features, figures, and manifests.
Public sample episode	One officially available sample episode.	The fully inspectable Line 1 unit used for raw-file browsing, 20-frame windows, task construction, and single-episode baselines.	The selected-128 comparison rows.
Selected 128 episodes	A public-safe selected subset of official gated episode paths.	Line 2 uses derived windows/features and keeps links back to official episode ids and gated source paths.	Redistributed raw MP4/HDF5/RRD data.
Xperience-10M	The upstream embodied human-interaction dataset.	Source dataset behind the public sample, selected-128 features, task suite, and model diagnostics.	This repo, which only redistributes public-safe derived artifacts.

Files and features

Term	Plain meaning	In this project	Do not confuse with
20-frame window	A fixed short clip slice.	The sample episode is converted into aligned 20-frame units for features, labels, and many task heads.	A full episode or arbitrary video segment.
annotation.hdf5	Upstream annotation container for the sample.	Contains original labels/metadata; some public derived files expose processed features instead of every raw text field.	Task result summaries.
Episode	One recorded interaction sequence.	The basic source unit behind windows, labels, and train/val/test splits.	A 20-frame window.
Feature manifest	A map from model-input columns to source modalities.	Explains feature groups and dimensions for the sample task suite.	The raw annotation file.
Interaction text	Natural-language interaction/caption content.	Used by task 15 and some derived text features; public matrices record direct or compact-proxy status.	Numeric action ids or subtask ids.
Modality	A type of signal.	Video, audio, depth, pose/SLAM, motion capture, inertial, calibration, and language-derived signals.	A task target.
Raw sample file map	A human-readable inventory of the sample episode files.	Explains videos, annotations, calibration, motion, and derived previews.	A training manifest.
visualization.rrd	Rerun viewer recording for visual inspection.	Can be downloaded from the official sample dataset and opened in Rerun 0.29.0 to inspect the sample episode. It is not used for published training or metric rows.	MP4 video streams or model inputs.
Window stride	The frame step between neighboring windows.	Creates overlapping examples while preserving chronological order and leakage controls.	Video frame rate.

Multimodal sensing

Term	Plain meaning	In this project	Do not confuse with
Audio waveform	A time-series pressure signal from sound.	The audio ablation measures whether embedded audio helps selected task contracts.	Language captions or text labels.
Calibration	Parameters that relate sensors to each other and to physical space.	Needed to interpret camera streams, depth, pose, and synchronized multimodal features together.	A model training hyperparameter.
Camera extrinsics	A camera position and orientation relative to another coordinate frame.	Connects different camera streams and world coordinates.	Camera intrinsics.
Camera intrinsics	Internal camera parameters such as focal length and distortion.	Explain how image pixels project to rays for geometry tasks.	Camera extrinsics.
Depth map	A per-pixel estimate of distance from the camera.	Depth-derived signals support spatial and geometry-oriented tasks.	RGB brightness or semantic segmentation.
Egocentric video	Video captured from a first-person or body-mounted viewpoint.	The sample streams are egocentric views of human interaction and are the visual basis for many tasks.	Third-person robot-camera footage.
Fisheye camera	A wide-angle camera with strong lens distortion.	Multiple fisheye MP4 streams give broad room coverage but need calibration-aware interpretation.	A rectilinear pinhole camera image.
IMU	An inertial measurement unit with accelerometer and gyroscope signals.	Supports motion, temporal, and sensor-bridging tasks.	Motion capture skeleton data.
Metric depth	Depth expressed in physical units rather than arbitrary relative scale.	Useful for distance-sensitive spatial reasoning and reconstruction targets.	Relative monocular depth.
Motion capture	A system that records body or hand motion over time.	Provides hand/body motion evidence when exposed through public-safe derived features.	Video-only pose estimation.
RGB frame	A color image frame from a video stream.	Used for visual statistics, previews, and many model inputs.	Depth values or point-cloud coordinates.
Sensor alignment	Putting different sensor streams into a shared temporal or spatial reference.	Used to make video, audio, pose, depth, IMU, and mocap usable in the same task input.	Model ensembling.
Stereo camera	A paired-camera setup that supports depth or geometry estimation.	The sample browser exposes stereo streams as part of the visual modality set.	Single-view RGB video.
Timestamp synchronization	Aligning sensor samples by time.	The task suite assumes aligned windows across modalities so labels and features refer to the same moment.	Randomly joining files with similar names.

Spatial geometry

Term	Plain meaning	In this project	Do not confuse with
3D reconstruction	Recovering 3D scene structure from sensor data.	One core spatial-intelligence direction for Xperience-style data.	Next-action classification.
Affordance	An action possibility offered by an object or scene.	Relevant when moving from observed human interaction to robot-action or VLA tasks.	A detected object category alone.
Camera pose	The camera position and orientation at a time step.	Supports spatial-intelligence tasks, view synchronization, and geometry diagnostics.	The human body pose.
Coordinate frame	A reference system for positions and orientations.	Needed when comparing camera, body, object, and world measurements.	A video frame.
Object-centric representation	A representation organized around objects and their relations.	Useful for object relevance, object-set forecast, and action-object relation tasks.	A flat feature vector without object identity.
Odometry	Motion estimated from sensor changes over time.	A relevant spatial term for ego-motion and camera-pose reasoning.	Ground-truth motion capture.
Point cloud	A set of 3D points representing scene structure.	A likely target or intermediate representation for spatial-intelligence extensions.	A 2D image grid.
SLAM	Simultaneous localization and mapping.	A field term for estimating camera motion and scene structure from sensor observations.	A task label or action class.
Spatial grounding	Linking language or labels to locations, objects, or geometry.	Connects language grounding tasks with 3D/spatial reasoning.	General text classification.
Trajectory	A sequence of positions over time.	Used for hand motion, camera motion, and future-path tasks.	A single coordinate or label.

Temporal and world models

Term	Plain meaning	In this project	Do not confuse with
Action forecasting	Predicting a future action before it happens.	Covered by next-action and long-horizon task contracts.	Recognizing the current action only.
Autoregressive prediction	Generating each future token, state, or frame conditioned on prior outputs.	Relevant for model branches that produce structured JSON or temporal predictions.	A one-shot classifier.
Forward dynamics	Predicting the next state from the current state and action/context.	The Cosmos3-Super LoRA branch uses a forward-dynamics-style diagnostic contract.	Reverse inference from result back to cause.
Latent state	A hidden representation that summarizes observed context.	Useful for future foundation-model and world-model training plans.	A visible annotation column.
Long-horizon prediction	Predicting outcomes several seconds or steps ahead.	Tasks 13 and 14 test longer temporal context beyond immediate recognition.	Single-frame classification.
Next-frame prediction	Predicting future visual frames from past frames.	A field-level world-model objective related to the human-video world-model direction.	Next-action prediction.
Object persistence	Tracking that an object remains present over time even when view or interaction changes.	Relevant for object-set forecast and long-video reasoning.	A single-frame object detection.
Rollout	Repeatedly predicting future steps from a model state.	Important for judging world models beyond one-step prediction.	A held-out static test row.
Subtask forecasting	Predicting the next higher-level step in an activity.	Used in the future-task probe line for Qwen3-Omni.	Frame-level action classification.
Teacher forcing	Training a sequence model using ground-truth previous outputs.	A likely training option for future sequence/world-model baselines.	Free-running rollout evaluation.
Temporal leakage	Using future information that would not be available at prediction time.	Avoided by chronological splits and target-side feature controls.	A low model score.
Transition timing	Estimating when the next state or action transition happens.	Task 20 turns temporal change into a regression target.	Classifying the transition type only.

Robotics and VLA

Term	Plain meaning	In this project	Do not confuse with
Action chunk	A short sequence of low-level actions predicted together.	The VLA figure and plan use action chunks as the policy-output concept.	A natural-language action label.
Behavior cloning	A supervised imitation-learning method for predicting demonstrated actions.	A plausible baseline once action targets are converted.	Generative video modeling.
Contact event	A moment when a hand, body, or tool touches an object or surface.	Used in contact-related tasks and action-quality interpretation.	Visual co-occurrence without touch.
Dexterity	Fine-grained physical manipulation ability.	Relevant to hand-object interaction, contact, and VLA/policy directions.	High text-generation accuracy.
End effector	The robot part that acts on the world, such as a gripper or hand.	A key target frame for future manipulation-policy conversion.	A camera or global scene coordinate.
Hand-object interaction	A physical interaction between hands and objects.	A central signal family behind action, contact, object relevance, and interaction-text tasks.	Object detection without action.
Imitation learning	Training a policy to imitate demonstrated behavior.	Relevant when converting human video/motion into action supervision.	Reinforcement learning from online robot trials.
Language grounding	Connecting text to observed objects, actions, or spatial context.	Task 8 and VLA directions use language as grounded supervision rather than standalone text.	Caption fluency alone.
Policy	A mapping from observations to actions.	A future target for robot-compatible Xperience-derived action data.	A benchmark metric.
Robot-compatible action target	An action representation a robot policy can execute or imitate.	Needed before OpenVLA/openpi/GR00T-style policy training is meaningful here.	Human-only caption text.
Vision-language-action model	A model that maps visual context and language into actions.	The VLA direction is a future path after action targets are converted into robot-compatible chunks.	A vision-language model that only answers text.

Tasks and metrics

Term	Plain meaning	In this project	Do not confuse with
Compact-proxy score	A bounded proxy metric when a direct raw target is not publicly available.	Kept explicit in the matrix and gap audit so readers do not over-read it.	A direct target measurement.
Direct score	A metric computed against the task target directly.	The preferred score type in the 20-task matrix.	Compact-proxy score.
Gap audit	A coverage and source-status audit.	Explains scored, proxy, and unsupported cells.	A performance leaderboard.
Leakage control	A split or feature rule that prevents using target information unfairly.	Chronological splits, held-out splits, and source audits protect task interpretation.	Lower training accuracy.
Normalized radar value	A 0-1 plotting value used only to draw comparable radar polygons.	Helps visualize metrics with different scales and directions.	The raw metric value to cite.
Raw metric value	The original metric value emitted by the runner or verified result package.	This is the value to cite from the 180-result table.	The normalized radar value.
Task contract	The definition of one benchmark task.	Includes input, target/output, metric, split, source artifact, and limitation.	A model architecture.
Task-method record	One method evaluated on one task.	9 methods x 20 tasks gives 180 public result records.	A single prediction row.
Unified 20-task suite	The current task surface.	All 20 task contracts are presented together and scored across methods where real artifacts exist.	Historical tier2_task_suite filenames, which are provenance paths rather than a second suite.

Training and evaluation

Term	Plain meaning	In this project	Do not confuse with
Adapter checkpoint	Saved adapter weights from a fine-tuning run.	The public model branches publish adapters when validated and public-safe.	Full base-model checkpoint.
Balanced accuracy	Accuracy averaged across classes to reduce majority-class dominance.	Useful for imbalanced task labels.	Overall accuracy.
Chronological split	A split ordered by time.	Used for the single-episode baselines to reduce future-window leakage.	A random row split.
Confusion matrix	A table of predicted classes versus true classes.	Helps inspect which task labels a method confuses.	A scalar leaderboard score.
FSDP	Fully Sharded Data Parallel, a distributed training strategy.	Appears in full-parameter feasibility and multi-GPU training notes.	A model architecture.
Held-out evaluation	Testing on examples not used for training.	Required before promoting Qwen/Cosmos results to public evidence.	Training-set loss.
JSON validity	Whether model output parses as the required JSON schema.	A key diagnostic for Qwen3-Omni structured-output runs.	Task correctness after parsing.
Macro F1	The average F1 score across classes, usually treating classes equally.	Used when class imbalance matters in classification tasks.	Accuracy dominated by frequent classes.
Mean absolute error	The average absolute difference between predicted and true numeric values.	Used for regression-style task rows such as timing or trajectory targets.	A classification F1 score.
Overfit check	A small training test that verifies a model can learn a tiny subset.	Useful for catching data/model wiring bugs before full training.	Evidence of generalization.
Parameter-efficient fine-tuning	Updating a small number of added or selected parameters.	LoRA is the current parameter-efficient path for Qwen/Cosmos branches.	Full-parameter fine-tuning.
Schema compliance	Whether an output follows the expected field names and value types.	Needed for structured task probes and public package validation.	High semantic accuracy.
Smoke run	A short run that checks whether a pipeline can start and execute key steps.	Used for feasibility gates before expensive full runs.	A complete benchmark result.
Top-k accuracy	A score that counts a prediction correct if the target is among the k highest-ranked outputs.	Useful for large-label or retrieval-style tasks.	Top-1 exact accuracy.
Train/validation/test split	A partition that separates model fitting, tuning, and final evaluation examples.	The selected-128 setup uses a held-out split discipline for model branches.	A random shuffle without temporal or episode boundaries.

Models and runs

Term	Plain meaning	In this project	Do not confuse with
Cosmos3-Nano	A smaller Cosmos3 compatibility/future-window branch.	Used for the Nano Future Window row and related diagnostics.	Cosmos3-Super fine-tuned adapter.
Cosmos3-Super	The larger Cosmos3-style branch tracked in this project.	Published as Reasoner diagnostics and a separate forward-dynamics LoRA adapter/result branch when verified.	Cosmos3-Nano.
Foundation pipeline	A high-level training direction.	Spatial intelligence, human-video world modeling, and vision-language-action are documented as trainable directions with task mappings.	A completed public result row.
Full-parameter fine-tuning	Updating the whole model rather than only adapters.	This project records feasibility gates and short pilots, but does not publish full checkpoints.	LoRA adapter publication.
Human-video world model	Learning future frames, actions, and interaction dynamics from human video.	Uses temporal prediction, next-action, transition, and object-forecast tasks.	Robot policy execution.
LoRA adapter	A lightweight set of trainable adapter weights.	Published only when the package is verified and public-safe.	Full base-model weights.
Metadata baseline	A selected-128 baseline using metadata or text-derived public-safe features.	Compares simple and neural heads on the held-out split.	Raw video, depth, or audio feature baselines.
Minimal baseline	A simple non-neural task head; the "minimum" reference row in casual wording.	Provides a reproducible lower-complexity comparison for task feasibility.	Metadata-only selected-128 baseline family.
Neural MLP	A compact neural task head.	Used for single-episode and selected-128 baseline comparisons.	Foundation-model fine-tuning.
Qwen v1-v6	The Qwen3-Omni run lineage.	v1-v4 are earlier pipeline/ablation evidence, v5 is the prior pinned release, and v6 is the current public 20-task row.	Six different evidence lines.
Qwen3-Omni	The multimodal foundation-model family used for the Qwen branch.	The current public 20-task Qwen row is Qwen3-Omni v6 LoRA plus task-specific probes.	Cosmos3 or single-episode task-head baselines.
Raw-feature baseline	A selected-128 baseline using exported public-safe raw-feature groups.	Tracks what non-foundation heads can do with richer processed inputs.	Raw gated media redistribution.
Simple baseline	A non-neural baseline family for the selected-128 rows.	Used for metadata/text and raw-feature 128-episode comparisons before NN/foundation-model rows.	The single-episode Minimal baseline.
Spatial intelligence	Learning geometry and spatial reasoning from egocentric data.	Uses video, depth, camera pose, and language tasks to target 3D/space reasoning.	World-model future prediction.
Vision-language-action	Mapping perception and language to action chunks.	A future policy/VLA direction that needs action-target conversion and stronger policy packaging.	Qwen3-Omni diagnostic scoring.

Public surfaces

Term	Plain meaning	In this project	Do not confuse with
HF artifact dataset	Hugging Face dataset repo for derived evidence.	Stores public-safe reports, metrics, website JSON, and sanitized result packages.	Original Xperience-10M dataset.
HF baseline model repo	Hugging Face model repo for lightweight baseline artifacts.	Mirrors baseline weights, figures, metrics, and task artifacts.	Qwen/Cosmos adapter-specific repos.
HF Space	Hugging Face-hosted app/site surface.	Mirrors the dashboard and static website assets.	HF artifact dataset or model repo.
HF weights/results repo	A consolidated public-safe model-result bundle.	Groups baseline weights, verified model artifacts, analysis files, and manifests.	The upstream raw dataset.
Mirror parity	A check that public copies match the source files.	Records whether GitHub, website, and HF mirrors agree.	A model-quality metric.
Public-safe artifact	A file that can be mirrored publicly without raw gated content.	Metrics, JSON summaries, model cards, figures, derived manifests, and approved lightweight weights/adapters.	Raw dataset redistribution.
Publication audit	A public-package validation report.	Confirms required files exist and forbidden raw/private assets are not included.	Scientific peer review.
Verified package	A result or artifact bundle that passed local/public validators.	Only verified packages are promoted to README, website, and HF surfaces as public evidence.	A running or exploratory experiment.

File Entry Points

Need	Open
Reader navigation	PUBLIC_READER_MAP.md, docs/data/public_reader_map.json
Task definitions	TASK_SUITE_20.md, docs/data/task_suite_20.json
Result matrix	TASK_METHOD_20_RESULT_MATRIX.md, docs/data/task_method_20_result_matrix.json
Direct/proxy status	TASK_METHOD_20_GAP_AUDIT.md, docs/data/task_method_20_gap_audit.json
Qwen lineage	QWEN3_OMNI_RUN_LINEAGE.md, docs/data/qwen3_omni_run_lineage.json
128-episode source/features	XPERIENCE10M_128_EPISODE_FEATURE_INDEX.md, docs/data/xperience10m_128_episode_feature_index.json
Public mirrors	PUBLIC_SURFACE_QA.md, docs/data/mirror_parity.json, docs/data/live_publication_status.json