tangyue0820 yzhautouskay commited on
Commit
0573a4b
·
1 Parent(s): 41ed7c9

Add Diffusers usage section for the distilled 4-step T2I pipeline (#17)

Browse files

- Add Diffusers usage section for the distilled 4-step T2I pipeline (4f396a09a7b98d7627f78bb216b00abc2d596a9b)


Co-authored-by: Yuliya Zhautouskaya <yzhautouskay@users.noreply.huggingface.co>

.gitattributes CHANGED
@@ -38,3 +38,4 @@ assets/example_image.png filter=lfs diff=lfs merge=lfs -text
38
  images/aa-t2i-all-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
39
  images/aa-t2i-open-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
40
  assets/benchmark-text2image.png filter=lfs diff=lfs merge=lfs -text
 
 
38
  images/aa-t2i-all-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
39
  images/aa-t2i-open-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
40
  assets/benchmark-text2image.png filter=lfs diff=lfs merge=lfs -text
41
+ assets/example_output_diffusers.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -196,6 +196,7 @@ Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated sys
196
 
197
  - [PyTorch](https://github.com/nvidia/cosmos3)
198
  - [vLLM-Omni](https://github.com/vllm-project/vllm-omni)
 
199
 
200
  **Supported Hardware Microarchitecture Compatibility:**
201
 
@@ -494,6 +495,70 @@ Example output generated from `assets/example_caption.json`:
494
 
495
  ![example_image](assets/example_image.png)
496
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
497
  ## Limitations
498
 
499
  Cosmos3 may produce imperfect outputs in challenging scenarios. Generation artifacts include temporal inconsistency, unstable camera or object motion, imprecise physical interactions, inaccurate audio-video synchronization, and action-state drift — especially in long-horizon or high-resolution outputs. Reasoning may also be incorrect: object states, causal relationships, spatial geometry, temporal ordering, agent intent, and future outcomes can be misinferred, and complex or long-context inputs may yield hallucinated entities, inconsistent interpretations, or implausible predictions. Because the model lacks an explicit physics simulator, 3D geometry, 4D space-time evolution, object permanence, contact dynamics, and physical laws are only approximated — producing artifacts such as disappearing or morphing objects, unrealistic collisions, and physically implausible motions. Quality further degrades in out-of-distribution environments, safety-critical edge cases, and domains underrepresented in training.
@@ -502,7 +567,7 @@ Cosmos3 outputs should not be treated as physically accurate simulation, reliabl
502
 
503
  ## Inference
504
 
505
- **Acceleration Engine:** [PyTorch](https://pytorch.org/), [vLLM-Omni](https://github.com/vllm-project/vllm-omni)
506
 
507
  **Test Hardware:** RTX PRO 6000, H20, H100, H200, B200
508
 
 
196
 
197
  - [PyTorch](https://github.com/nvidia/cosmos3)
198
  - [vLLM-Omni](https://github.com/vllm-project/vllm-omni)
199
+ - [Hugging Face Diffusers](https://github.com/huggingface/diffusers)
200
 
201
  **Supported Hardware Microarchitecture Compatibility:**
202
 
 
495
 
496
  ![example_image](assets/example_image.png)
497
 
498
+ ## Usage: Run Inference with Diffusers
499
+
500
+ `Cosmos3-Super-Text2Image-4Step` is supported by the Hugging Face Diffusers **modular** pipeline. The distilled checkpoint is not compatible with `Cosmos3OmniPipeline` or `Cosmos3OmniModularPipeline`; it must be loaded with `Cosmos3DistilledModularPipeline`.
501
+
502
+ ### Install
503
+
504
+ Install Diffusers from the main branch until a release containing `Cosmos3DistilledModularPipeline` is available:
505
+
506
+ ```bash
507
+ uv venv --python 3.13 --seed --managed-python
508
+ source .venv/bin/activate
509
+ uv pip install \
510
+ "diffusers @ git+https://github.com/huggingface/diffusers.git" \
511
+ accelerate \
512
+ av \
513
+ cosmos_guardrail \
514
+ huggingface_hub \
515
+ imageio \
516
+ imageio-ffmpeg \
517
+ torch \
518
+ torchvision \
519
+ transformers
520
+ ```
521
+
522
+ The safety checker requires access to the gated [nvidia/Cosmos-1.0-Guardrail](https://huggingface.co/nvidia/Cosmos-1.0-Guardrail) repository.
523
+
524
+ ### Example: Text to Image Generation
525
+
526
+ ```python
527
+ import json
528
+
529
+ import torch
530
+ from diffusers import Cosmos3DistilledModularPipeline
531
+
532
+ json_prompt = json.load(open("assets/example_caption.json"))
533
+ json_prompt["resolution"] = {"H": 768, "W": 768}
534
+ json_prompt["aspect_ratio"] = "1,1"
535
+
536
+ pipe = Cosmos3DistilledModularPipeline.from_pretrained("nvidia/Cosmos3-Super-Text2Image-4Step")
537
+ pipe.load_components(torch_dtype=torch.bfloat16)
538
+ pipe.enable_safety_checker()
539
+ pipe.to("cuda")
540
+
541
+ images = pipe(
542
+ prompt=json.dumps(json_prompt),
543
+ num_frames=1,
544
+ height=768,
545
+ width=768,
546
+ add_resolution_template=False,
547
+ generator=torch.Generator(device="cuda").manual_seed(1143),
548
+ output="videos",
549
+ )
550
+ images[0].save("cosmos3_t2i_4step_diffusers.png")
551
+ print("Saved image to cosmos3_t2i_4step_diffusers.png")
552
+ ```
553
+
554
+ Single-frame generation uses `num_frames=1`, and the frame comes back through the pipeline's `videos` output.
555
+
556
+ The 4-step schedule and the guidance scale come from the checkpoint: `num_inference_steps` is fixed to the length of the distilled sigma schedule and `guidance_scale` is forced to `1.0`, so passing any other value raises an error. `negative_prompt` is ignored, and unlike the base Cosmos3 models there is no scheduler to swap in and no `flow_shift` to set.
557
+
558
+ Example output generated from `assets/example_caption.json`:
559
+
560
+ ![example_image](https://huggingface.co/nvidia/Cosmos3-Super-Text2Image-4Step/resolve/main/assets/example_output_diffusers.png)
561
+
562
  ## Limitations
563
 
564
  Cosmos3 may produce imperfect outputs in challenging scenarios. Generation artifacts include temporal inconsistency, unstable camera or object motion, imprecise physical interactions, inaccurate audio-video synchronization, and action-state drift — especially in long-horizon or high-resolution outputs. Reasoning may also be incorrect: object states, causal relationships, spatial geometry, temporal ordering, agent intent, and future outcomes can be misinferred, and complex or long-context inputs may yield hallucinated entities, inconsistent interpretations, or implausible predictions. Because the model lacks an explicit physics simulator, 3D geometry, 4D space-time evolution, object permanence, contact dynamics, and physical laws are only approximated — producing artifacts such as disappearing or morphing objects, unrealistic collisions, and physically implausible motions. Quality further degrades in out-of-distribution environments, safety-critical edge cases, and domains underrepresented in training.
 
567
 
568
  ## Inference
569
 
570
+ **Acceleration Engine:** [PyTorch](https://pytorch.org/), [vLLM-Omni](https://github.com/vllm-project/vllm-omni), [Hugging Face Diffusers](https://github.com/huggingface/diffusers)
571
 
572
  **Test Hardware:** RTX PRO 6000, H20, H100, H200, B200
573
 
assets/example_output_diffusers.png ADDED

Git LFS Details

  • SHA256: 9e8a7b0392fbbbb1bd6686ed2627bb8062b61fe8ba295a441602e26f1a2a8f77
  • Pointer size: 131 Bytes
  • Size of remote file: 719 kB