Instructions to use nvidia/Cosmos3-Super-Text2Image-4Step with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Cosmos
How to use nvidia/Cosmos3-Super-Text2Image-4Step with Cosmos:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Commit ·
0573a4b
1
Parent(s): 41ed7c9
Add Diffusers usage section for the distilled 4-step T2I pipeline (#17)
Browse files- Add Diffusers usage section for the distilled 4-step T2I pipeline (4f396a09a7b98d7627f78bb216b00abc2d596a9b)
Co-authored-by: Yuliya Zhautouskaya <yzhautouskay@users.noreply.huggingface.co>
- .gitattributes +1 -0
- README.md +66 -1
- assets/example_output_diffusers.png +3 -0
.gitattributes
CHANGED
|
@@ -38,3 +38,4 @@ assets/example_image.png filter=lfs diff=lfs merge=lfs -text
|
|
| 38 |
images/aa-t2i-all-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
|
| 39 |
images/aa-t2i-open-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
|
| 40 |
assets/benchmark-text2image.png filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 38 |
images/aa-t2i-all-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
|
| 39 |
images/aa-t2i-open-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
|
| 40 |
assets/benchmark-text2image.png filter=lfs diff=lfs merge=lfs -text
|
| 41 |
+
assets/example_output_diffusers.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -196,6 +196,7 @@ Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated sys
|
|
| 196 |
|
| 197 |
- [PyTorch](https://github.com/nvidia/cosmos3)
|
| 198 |
- [vLLM-Omni](https://github.com/vllm-project/vllm-omni)
|
|
|
|
| 199 |
|
| 200 |
**Supported Hardware Microarchitecture Compatibility:**
|
| 201 |
|
|
@@ -494,6 +495,70 @@ Example output generated from `assets/example_caption.json`:
|
|
| 494 |
|
| 495 |

|
| 496 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 497 |
## Limitations
|
| 498 |
|
| 499 |
Cosmos3 may produce imperfect outputs in challenging scenarios. Generation artifacts include temporal inconsistency, unstable camera or object motion, imprecise physical interactions, inaccurate audio-video synchronization, and action-state drift — especially in long-horizon or high-resolution outputs. Reasoning may also be incorrect: object states, causal relationships, spatial geometry, temporal ordering, agent intent, and future outcomes can be misinferred, and complex or long-context inputs may yield hallucinated entities, inconsistent interpretations, or implausible predictions. Because the model lacks an explicit physics simulator, 3D geometry, 4D space-time evolution, object permanence, contact dynamics, and physical laws are only approximated — producing artifacts such as disappearing or morphing objects, unrealistic collisions, and physically implausible motions. Quality further degrades in out-of-distribution environments, safety-critical edge cases, and domains underrepresented in training.
|
|
@@ -502,7 +567,7 @@ Cosmos3 outputs should not be treated as physically accurate simulation, reliabl
|
|
| 502 |
|
| 503 |
## Inference
|
| 504 |
|
| 505 |
-
**Acceleration Engine:** [PyTorch](https://pytorch.org/), [vLLM-Omni](https://github.com/vllm-project/vllm-omni)
|
| 506 |
|
| 507 |
**Test Hardware:** RTX PRO 6000, H20, H100, H200, B200
|
| 508 |
|
|
|
|
| 196 |
|
| 197 |
- [PyTorch](https://github.com/nvidia/cosmos3)
|
| 198 |
- [vLLM-Omni](https://github.com/vllm-project/vllm-omni)
|
| 199 |
+
- [Hugging Face Diffusers](https://github.com/huggingface/diffusers)
|
| 200 |
|
| 201 |
**Supported Hardware Microarchitecture Compatibility:**
|
| 202 |
|
|
|
|
| 495 |
|
| 496 |

|
| 497 |
|
| 498 |
+
## Usage: Run Inference with Diffusers
|
| 499 |
+
|
| 500 |
+
`Cosmos3-Super-Text2Image-4Step` is supported by the Hugging Face Diffusers **modular** pipeline. The distilled checkpoint is not compatible with `Cosmos3OmniPipeline` or `Cosmos3OmniModularPipeline`; it must be loaded with `Cosmos3DistilledModularPipeline`.
|
| 501 |
+
|
| 502 |
+
### Install
|
| 503 |
+
|
| 504 |
+
Install Diffusers from the main branch until a release containing `Cosmos3DistilledModularPipeline` is available:
|
| 505 |
+
|
| 506 |
+
```bash
|
| 507 |
+
uv venv --python 3.13 --seed --managed-python
|
| 508 |
+
source .venv/bin/activate
|
| 509 |
+
uv pip install \
|
| 510 |
+
"diffusers @ git+https://github.com/huggingface/diffusers.git" \
|
| 511 |
+
accelerate \
|
| 512 |
+
av \
|
| 513 |
+
cosmos_guardrail \
|
| 514 |
+
huggingface_hub \
|
| 515 |
+
imageio \
|
| 516 |
+
imageio-ffmpeg \
|
| 517 |
+
torch \
|
| 518 |
+
torchvision \
|
| 519 |
+
transformers
|
| 520 |
+
```
|
| 521 |
+
|
| 522 |
+
The safety checker requires access to the gated [nvidia/Cosmos-1.0-Guardrail](https://huggingface.co/nvidia/Cosmos-1.0-Guardrail) repository.
|
| 523 |
+
|
| 524 |
+
### Example: Text to Image Generation
|
| 525 |
+
|
| 526 |
+
```python
|
| 527 |
+
import json
|
| 528 |
+
|
| 529 |
+
import torch
|
| 530 |
+
from diffusers import Cosmos3DistilledModularPipeline
|
| 531 |
+
|
| 532 |
+
json_prompt = json.load(open("assets/example_caption.json"))
|
| 533 |
+
json_prompt["resolution"] = {"H": 768, "W": 768}
|
| 534 |
+
json_prompt["aspect_ratio"] = "1,1"
|
| 535 |
+
|
| 536 |
+
pipe = Cosmos3DistilledModularPipeline.from_pretrained("nvidia/Cosmos3-Super-Text2Image-4Step")
|
| 537 |
+
pipe.load_components(torch_dtype=torch.bfloat16)
|
| 538 |
+
pipe.enable_safety_checker()
|
| 539 |
+
pipe.to("cuda")
|
| 540 |
+
|
| 541 |
+
images = pipe(
|
| 542 |
+
prompt=json.dumps(json_prompt),
|
| 543 |
+
num_frames=1,
|
| 544 |
+
height=768,
|
| 545 |
+
width=768,
|
| 546 |
+
add_resolution_template=False,
|
| 547 |
+
generator=torch.Generator(device="cuda").manual_seed(1143),
|
| 548 |
+
output="videos",
|
| 549 |
+
)
|
| 550 |
+
images[0].save("cosmos3_t2i_4step_diffusers.png")
|
| 551 |
+
print("Saved image to cosmos3_t2i_4step_diffusers.png")
|
| 552 |
+
```
|
| 553 |
+
|
| 554 |
+
Single-frame generation uses `num_frames=1`, and the frame comes back through the pipeline's `videos` output.
|
| 555 |
+
|
| 556 |
+
The 4-step schedule and the guidance scale come from the checkpoint: `num_inference_steps` is fixed to the length of the distilled sigma schedule and `guidance_scale` is forced to `1.0`, so passing any other value raises an error. `negative_prompt` is ignored, and unlike the base Cosmos3 models there is no scheduler to swap in and no `flow_shift` to set.
|
| 557 |
+
|
| 558 |
+
Example output generated from `assets/example_caption.json`:
|
| 559 |
+
|
| 560 |
+

|
| 561 |
+
|
| 562 |
## Limitations
|
| 563 |
|
| 564 |
Cosmos3 may produce imperfect outputs in challenging scenarios. Generation artifacts include temporal inconsistency, unstable camera or object motion, imprecise physical interactions, inaccurate audio-video synchronization, and action-state drift — especially in long-horizon or high-resolution outputs. Reasoning may also be incorrect: object states, causal relationships, spatial geometry, temporal ordering, agent intent, and future outcomes can be misinferred, and complex or long-context inputs may yield hallucinated entities, inconsistent interpretations, or implausible predictions. Because the model lacks an explicit physics simulator, 3D geometry, 4D space-time evolution, object permanence, contact dynamics, and physical laws are only approximated — producing artifacts such as disappearing or morphing objects, unrealistic collisions, and physically implausible motions. Quality further degrades in out-of-distribution environments, safety-critical edge cases, and domains underrepresented in training.
|
|
|
|
| 567 |
|
| 568 |
## Inference
|
| 569 |
|
| 570 |
+
**Acceleration Engine:** [PyTorch](https://pytorch.org/), [vLLM-Omni](https://github.com/vllm-project/vllm-omni), [Hugging Face Diffusers](https://github.com/huggingface/diffusers)
|
| 571 |
|
| 572 |
**Test Hardware:** RTX PRO 6000, H20, H100, H200, B200
|
| 573 |
|
assets/example_output_diffusers.png
ADDED
|
Git LFS Details
|