---
pipeline_tag: text-to-video
license: apache-2.0
tags:
- dit
- text-to-video
- text-to-image
- video-generation
- image-generation
---
## Towards Stabilized and Efficient Diffusion Transformers through Long-Skip-Connections with Spectral Constraints
(a) Feature similarities between standard and cache-accelerated outputs in vanilla DiT caching with FORA and Faster-Diff and Skip-DiT. Skip-DiT presents consistently higher feature similarity, demonstrating superior stability after caching. (b) Illustration of Skip-DiT that modifies vanilla DiT models using long-skip-connection to connect shallow and deep DiT blocks. Dashed arrows indicate paths where computation can be skipped in cached inference. (c) Comparison of video generation quality (PNSR) and inference speedup of different DiT caching methods. Skip-DiT maintains higher generation quality even at greater speedup factors.
### 🎉🎉🎉 About
This repository contains the official PyTorch implementation of the paper: **[Towards Stabilized and Efficient Diffusion Transformers through Long-Skip-Connections with Spectral Constraints](https://huggingface.co/papers/2411.17616)**.
Diffusion Transformers (DiT) have emerged as a powerful architecture for image and video generation, offering superior quality and scalability. However, their practical application suffers from inherent dynamic feature instability, leading to error amplification during cached inference. Through systematic analysis, the authors identify the absence of long-range feature preservation mechanisms as the root cause of unstable feature propagation. Skip-DiT is an image and video generative DiT variant enhanced with Long-Skip-Connections (LSCs) which stabilize feature dynamics and enable an efficient statical caching mechanism.
### 🌟 Feature Stability of Skip-DiT
Visualization of the feature stability of Skip-DiT compared with vanilla DiT. Skip-DiT also shows superior training efficiency.
### 🛒 Released Models
| Model | Task | Training Data | Backbone | Size(G) | Skip-Cache |
|:--:|:--:|:--:|:--:|:--:|:--:|
| [Latte-skip](https://huggingface.co/GuanjieChen/Skip-DiT/blob/main/DiT-XL-2-skip.pt) | text-to-video |Vimeo|Latte|8.76| ✅ |
| [DiT-XL/2-skip](https://huggingface.co/GuanjieChen/Skip-DiT/blob/main/Latte-skip.pt) | class-to-image |ImageNet|DiT-XL/2|11.40|✅ |
| [ucf101-skip](https://huggingface.co/GuanjieChen/Skip-DiT/blob/main/ucf101-skip.pt) | class-to-video|UCF101|Latte|2.77|✅ |
| [taichi-skip](https://huggingface.co/GuanjieChen/Skip-DiT/blob/main/taichi-skip.pt) | class-to-video|Taichi-HD|Latte|2.77|✅ |
| [skytimelapse-skip](https://huggingface.co/GuanjieChen/Skip-DiT/blob/main/skylapse-skip.pt) | class-to-video|SkyTimelapse|Latte|2.77|✅ |
| [ffs-skip](https://huggingface.co/GuanjieChen/Skip-DiT/blob/main/ffs-skip.pt) | class-to-video|FaceForensics|Latte|2.77|✅ |
Pretrained text-to-image Model of [HunYuan-DiT](https://github.com/Tencent/HunyuanDiT) can be found in [Huggingface](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT-v1.2/tree/main/t2i/model).
### 🚀 Quick Start
#### Text-to-video Inference
To generate videos with Latte-skip, you just need 3 steps:
```shell
# 1. Prepare your conda environments
cd text-to-video ; conda env create -f environment.yaml ; conda activate latte
# 2. Download checkpoints of Latte and Latte-skip
python download.py
# 3. Generate videos with only one command line!
python sample/sample_t2v.py --config ./configs/t2v/t2v_sample_skip.yaml
# 4. (Optional) To accelerate generation with skip-cache, run following command
python sample/sample_t2v.py --config ./configs/t2v/t2v_sample_skip_cache.yaml --cache N2-700-50
```
#### Text-to-image Inference
To generate images with Hunyuan-DiT:
```shell
# 1. Prepare your conda environments
cd text-to-image ; conda env create -f environment.yaml ; conda activate HunyuanDiT
# 2. Download checkpoints of Hunyuan-DiT
mkdir ckpts ; huggingface-cli download Tencent-Hunyuan/HunyuanDiT-v1.2 --local-dir ./ckpts
# 3. Generate images with only one command line!
python sample_t2i.py --prompt "渔舟唱晚" --no-enhance --infer-steps 100 --image-size 1024 1024
# 4. (Optional) To accelerate generation with skip-cache, run the following command
python sample_t2i.py --prompt "渔舟唱晚" --no-enhance --infer-steps 100 --image-size 1024 1024 --cache --cache-step 2
```
### 🏃 Training
The training code of Latte-skip is available in the repository. To train the text-to-video model:
1. Prepare your text-video datasets and implement the `text-to-video/datasets/t2v_joint_dataset.py`
2. Run the two-stage training strategy:
1. Freeze all the parameters except skip-branches. Set `freeze=True` in `text-to-video/configs/train_t2v.yaml`. And then run the training scripts at `text-to-video/train_scripts/t2v_joint_train_skip.sh`.
2. Overall training. Set `freeze=False` in `text-to-video/configs/train_t2v.yaml`. And then run the training scripts.
### 🌺 Acknowledgement
Skip-DiT has been greatly inspired by the following works and teams: [DeepCache](https://arxiv.org/abs/2312.00858), [Latte](https://github.com/Vchitect/Latte), [DiT](https://github.com/facebookresearch/DiT), and [HunYuan-DiT](https://github.com/Tencent/HunyuanDiT).
### License
The code and model weights are licensed under the Apache License 2.0.
### Citation
```bibtex
@article{chen2024towards,
title={Towards Stabilized and Efficient Diffusion Transformers through Long-Skip-Connections with Spectral Constraints},
author={Chen, Guanjie and Zhao, Xinyu and Zhou, Yucheng and Chen, Tianlong and Yu, Cheng},
journal={arXiv preprint arXiv:2411.17616},
year={2024}
}
```
