README.md

---
license: openmdw-1.1
base_model:
  - nvidia/Cosmos3-Nano
datasets:
  - chrisvoncsefalvay/openembodiment-dvrk-subset
  - nvidia/PhysicalAI-Robotics-Open-H-Embodiment
tags:
  - cosmos3
  - surgical-robotics
  - dvrk
  - surgical-kinematics
  - motion-prediction
  - inverse-dynamics
  - forward-dynamics
  - action-conditioned
  - world-model
  - lerobot
  - robotics
library_name: cosmos3
private: true
---
# Cosmos 3 Nano-H surgical dVRK

![image](https://cdn-uploads.huggingface.co/production/uploads/6515c41e4256270d6fa09259/6A4LQihXfQD2bbsF7_tO8.gif)




Cosmos 3 Nano-0H Surgical dVRK is a domain-specific post-trained Cosmos 3 Nano
checkpoint for short-horizon surgical kinematics prediction and dVRK motion
simulation. It adapts the Nano-scale Cosmos world model to dVRK/da Vinci
endoscopic scenes and uses a native dVRK action representation that is
compatible with GR00T-H-style dual-arm end-effector control.

The model is intended as a research world model for surgical robotics:
predicting and comparing near-future tool motion, recovering action-like
kinematic structure from short video clips, validating action interfaces and
studying low-latency motion simulation behaviour in visually constrained robotic
manipulation settings.

Video is a secondary diagnostic channel in this limited NanoH run, not the main
product. The video objective remains much harder than the action/kinematic
objectives: generated video quality is substantially weaker than the
kinematics outputs even when the rollout is conditioned. Use this checkpoint as
a compact surgical motion model, not as a surgical video generator.

It is not intended for clinical decision-making, autonomous surgery, medical
diagnosis or patient-facing deployment.

## Model details

- **Base model:** `nvidia/Cosmos3-Nano`
- **Adaptation:** supervised post-training on dVRK surgical windows
- **Primary task:** surgical kinematics prediction and short-horizon dVRK motion simulation
- **Secondary task:** action-conditioned visual rollout for context and diagnostics
- **Prediction horizon:** short-horizon 16-frame chunks
- **Visual input:** single endoscopic camera stream
- **Action input:** dual-PSM dVRK tensor, relative EEF motion with rot6D
- **Training objective:** mixed forward dynamics, inverse dynamics and policy
- **Precision:** bfloat16 training
- **Distributed strategy:** single-node FSDP
- **Checkpoint format:** Cosmos DCP checkpoints plus optional HF export

## Action space

The model registers dVRK as a first-class Cosmos embodiment rather than mapping
it onto an existing robot id.

```text
embodiment: dvrk
domain_id: 22
raw_action_dim: 20
chunk_length: 16
```

Native dVRK tensor:

```text
actions_dvrk: (16, 20)

left_xyz_rel(3), left_rot6d_rel(6), left_jaw_abs(1),
right_xyz_rel(3), right_rot6d_rel(6), right_jaw_abs(1)
```

Surgical simulator bridge:

```text
actions_unified44: (16, 44)
actions_unified44[..., :20] == actions_dvrk
actions_unified44[..., 20:44] == 0
```

Cosmos action head:

```text
actions_cosmos3: (16, 64)
raw_action_dim: 20
```

The 20D native tensor is padded to the Cosmos 3 64D action head while preserving
the raw dimension for action-channel masking. A 44D zero-padded bridge is
included for surgical simulator interoperability.

## Training data

Training uses
[`chrisvoncsefalvay/openembodiment-dvrk-subset`](https://huggingface.co/datasets/chrisvoncsefalvay/openembodiment-dvrk-subset),
a convenience subset of
[`nvidia/PhysicalAI-Robotics-Open-H-Embodiment`](https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-Open-H-Embodiment).

The subset mirrors dVRK/da Vinci records and
associated metadata so training machines can stream from one stable
HF/Xet-backed repository. The underlying source data and credit remain with
NVIDIA and the Open-H Embodiment contributors.

## Training recipe

The release recipe is designed for a single 8xH100 node:

- FSDP over 8 GPUs
- bfloat16 precision
- activation checkpointing
- 16-frame action chunks
- single-camera video stream
- all-objective sampling across forward dynamics, inverse dynamics and policy
- checkpoint save interval of 100 iterations
- W&B metrics and qualitative visual logging
- interrupt-tolerant DCP checkpoints
- final artefact upload to a private Hugging Face model repository

The recipe demonstrates that useful domain adaptation of Cosmos 3 Nano does not
require enterprise-scale cluster choreography. Larger runs remain valuable, but
the single-node path is already strong enough to reveal meaningful surgical
domain improvement.

## Evaluation

Evaluation emphasises short-horizon kinematic consistency and action-space
validity. Video metrics are retained as diagnostics, but kinematic and toolpath
checks are the primary success criteria for this run:

- tensor validation for dVRK, 44D simulator bridge and Cosmos 64D padded action
- held-out episode splits
- FDS-L1 over short rollouts
- toolpath and end-effector trajectory comparison against ground truth
- inverse-kinematics consistency on held-out motion snippets
- action-space normalization and scale audits
- qualitative W&B panels with context, prediction, actual, error strip and
  action norms
- GR00T-H interop samples in native, 44D and 64D action formats
- PSNR, lightweight global SSIM and temporal frame-difference L1 as secondary
  video diagnostics

Latency and smoothness are first-class evaluation concerns. A short rollout that
keeps motion coherent and fast is more valuable for this use case than a long
rollout that looks impressive only until the simulator has to respond.

## Intended use

Appropriate use:

- research on surgical robotics world models
- short-horizon dVRK kinematics and toolpath prediction
- action-space validation and interop work
- simulation prototyping and qualitative model comparison
- non-clinical robotics research demos

Out-of-scope use:

- clinical decision support
- autonomous surgical control
- diagnosis, treatment planning or patient triage
- safety-critical deployment
- replacing a validated simulator or physical robot test

## Limitations

- The model is specialised to dVRK-like endoscopic surgical scenes.
- It is optimised for short-horizon consistency, not indefinite rollout length.
- It may hallucinate tool geometry, tissue response, lighting or contact
  effects.
- Video rollouts are weaker than the kinematics outputs and should be treated
  as qualitative diagnostics rather than photorealistic surgical video.
- It should not be treated as a calibrated dynamics oracle.
- Single-camera training limits viewpoint disentanglement.
- Dataset coverage reflects the dVRK subset available in Open-H Embodiment.

## Safety

This checkpoint is a research artefact. Predicted toolpaths, actions and
generated video should be interpreted as learned simulation outputs, not as
measured surgical reality. Any downstream robotics use requires independent
validation, simulator checks, hardware safety systems and domain-expert review.

## Citation

Please cite the upstream Open-H Embodiment dataset and NVIDIA Cosmos materials
when using this model or the associated dataset subset. Cite this checkpoint
repository for the dVRK action mapping, kinematics-focused training recipe and
released weights.

```bibtex
@misc{voncsefalvay2026cosmos3nanohsurgicaldvrk,
  title = {Cosmos 3 NanoH surgical dVRK},
  author = {von Csefalvay, Chris},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face model repository},
  url = {https://huggingface.co/chrisvoncsefalvay/cosmos3-nanoh-surgical-dvrk}
}
```

## Artefacts

Expected repository contents after release:

```text
model/                 exported HF checkpoint
checkpoints/           resumable Cosmos DCP checkpoints
eval/                  quantitative reports and qualitative references
interop/               dVRK, 44D simulator and 64D Cosmos action samples
README.md              this model card
```

## License

Released model materials are licensed under OpenMDW-1.1. Model code and adapter
utilities in the source repository remain Apache-2.0. Dataset files retain the
terms of the upstream Open-H Embodiment release. Base model usage is subject to
the terms of the upstream Cosmos 3 Nano release.