Text-to-Video
Diffusers
Safetensors
Transformers
English
Chinese
image-to-video
video-continuation
Eval Results
Instructions to use meituan-longcat/LongCat-Video with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use meituan-longcat/LongCat-Video with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("meituan-longcat/LongCat-Video", dtype=torch.bfloat16, device_map="cuda") prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" image = pipe(prompt).images[0] - Transformers
How to use meituan-longcat/LongCat-Video with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("meituan-longcat/LongCat-Video", dtype="auto") - Inference
- Notebooks
- Google Colab
- Kaggle
Add files using upload-large-folder tool
Browse files- README.md +29 -13
- assets/longcat-logo.svg +78 -0
README.md
CHANGED
|
@@ -1,13 +1,13 @@
|
|
| 1 |
# LongCat-Video
|
| 2 |
|
| 3 |
<div align="center">
|
| 4 |
-
<img src="assets/
|
| 5 |
</div>
|
| 6 |
<hr>
|
| 7 |
|
| 8 |
<div align="center" style="line-height: 1;">
|
| 9 |
<a href='https://meituan-longcat.github.io/longcatvideo/'><img src='https://img.shields.io/badge/Project-Page-green'></a>
|
| 10 |
-
<a href='https://
|
| 11 |
<a href='https://huggingface.co/meituan-longcat/LongCat-Video'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue'></a>
|
| 12 |
</div>
|
| 13 |
|
|
@@ -29,7 +29,7 @@ We introduce LongCat-Video, a foundational video generation model with 13.6B par
|
|
| 29 |
- ๐ **Efficient inference**: LongCat-Video generates $720p$, $30fps$ videos within minutes by employing a coarse-to-fine generation strategy along both the temporal and spatial axes. Block Sparse Attention further enhances efficiency, particularly at high resolutions
|
| 30 |
- ๐ **Strong performance with multi-reward RLHF**: Powered by multi-reward Group Relative Policy Optimization (GRPO), comprehensive evaluations on both internal and public benchmarks demonstrate that LongCat-Video achieves performance comparable to leading open-source video generation models as well as the latest commercial solutions.
|
| 31 |
|
| 32 |
-
For more detail, please refer to the comprehensive [***LongCat-Video Technical Report***](https://github.com/meituan-longcat/LongCat-
|
| 33 |
|
| 34 |
## Quick Start
|
| 35 |
|
|
@@ -48,58 +48,69 @@ Install dependencies:
|
|
| 48 |
pip install -r requirements.txt
|
| 49 |
```
|
| 50 |
|
|
|
|
|
|
|
| 51 |
### Model Download
|
| 52 |
|
| 53 |
| Models | Download Link |
|
| 54 |
| --- | --- |
|
| 55 |
-
| LongCat-Video | ๐ค [Huggingface](https://huggingface.co/
|
| 56 |
|
| 57 |
Download models using huggingface-cli:
|
| 58 |
```shell
|
| 59 |
pip install "huggingface_hub[cli]"
|
| 60 |
-
huggingface-cli download meituan-longcat/LongCat-Video --local-dir ./
|
| 61 |
```
|
| 62 |
|
| 63 |
### Run Text-to-Video
|
| 64 |
|
| 65 |
```shell
|
| 66 |
# Single-GPU inference
|
| 67 |
-
torchrun run_demo_image_to_video.py --checkpoint_dir=./
|
| 68 |
|
| 69 |
# Multi-GPU inference
|
| 70 |
-
torchrun --nproc_per_node=2 run_demo_image_to_video.py --context_parallel_size=2 --checkpoint_dir=./
|
| 71 |
```
|
| 72 |
|
| 73 |
### Run Image-to-Video
|
| 74 |
|
| 75 |
```shell
|
| 76 |
# Single-GPU inference
|
| 77 |
-
torchrun run_demo_image_to_video.py --checkpoint_dir=./
|
| 78 |
|
| 79 |
# Multi-GPU inference
|
| 80 |
-
torchrun --nproc_per_node=2 run_demo_image_to_video.py --context_parallel_size=2 --checkpoint_dir=./
|
| 81 |
```
|
| 82 |
|
| 83 |
### Run Video-Continuation
|
| 84 |
|
| 85 |
```shell
|
| 86 |
# Single-GPU inference
|
| 87 |
-
torchrun run_demo_video_continuation.py --checkpoint_dir=./
|
| 88 |
|
| 89 |
# Multi-GPU inference
|
| 90 |
-
torchrun --nproc_per_node=2 run_demo_video_continuation.py --context_parallel_size=2 --checkpoint_dir=./
|
| 91 |
```
|
| 92 |
|
| 93 |
### Run Long-Video Generation
|
| 94 |
|
| 95 |
```shell
|
| 96 |
# Single-GPU inference
|
| 97 |
-
torchrun run_demo_long_video.py --checkpoint_dir=./
|
| 98 |
|
| 99 |
# Multi-GPU inference
|
| 100 |
-
torchrun --nproc_per_node=2 run_demo_long_video.py --context_parallel_size=2 --checkpoint_dir=./
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 101 |
```
|
| 102 |
|
|
|
|
|
|
|
| 103 |
## Evaluation Results
|
| 104 |
|
| 105 |
### Text-to-Video
|
|
@@ -162,5 +173,10 @@ We kindly encourage citation of our work if you find it useful.
|
|
| 162 |
}
|
| 163 |
```
|
| 164 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 165 |
## Contact
|
| 166 |
Please contact us at <a href="mailto:longcat-team@meituan.com">longcat-team@meituan.com</a> or join our WeChat Group if you have any questions.
|
|
|
|
| 1 |
# LongCat-Video
|
| 2 |
|
| 3 |
<div align="center">
|
| 4 |
+
<img src="assets/longcat-video_logo.svg" width="45%" alt="LongCat-Video" />
|
| 5 |
</div>
|
| 6 |
<hr>
|
| 7 |
|
| 8 |
<div align="center" style="line-height: 1;">
|
| 9 |
<a href='https://meituan-longcat.github.io/longcatvideo/'><img src='https://img.shields.io/badge/Project-Page-green'></a>
|
| 10 |
+
<a href='https://github.com/meituan-longcat/LongCat-Video/blob/main/longcatvideo_tech_report.pdf'><img src='https://img.shields.io/badge/Technique-Report-red'></a>
|
| 11 |
<a href='https://huggingface.co/meituan-longcat/LongCat-Video'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue'></a>
|
| 12 |
</div>
|
| 13 |
|
|
|
|
| 29 |
- ๐ **Efficient inference**: LongCat-Video generates $720p$, $30fps$ videos within minutes by employing a coarse-to-fine generation strategy along both the temporal and spatial axes. Block Sparse Attention further enhances efficiency, particularly at high resolutions
|
| 30 |
- ๐ **Strong performance with multi-reward RLHF**: Powered by multi-reward Group Relative Policy Optimization (GRPO), comprehensive evaluations on both internal and public benchmarks demonstrate that LongCat-Video achieves performance comparable to leading open-source video generation models as well as the latest commercial solutions.
|
| 31 |
|
| 32 |
+
For more detail, please refer to the comprehensive [***LongCat-Video Technical Report***](https://github.com/meituan-longcat/LongCat-Video/blob/main/longcatvideo_tech_report.pdf).
|
| 33 |
|
| 34 |
## Quick Start
|
| 35 |
|
|
|
|
| 48 |
pip install -r requirements.txt
|
| 49 |
```
|
| 50 |
|
| 51 |
+
FlashAttention-2 is enabled in the model config by default; you can also change the model config to use FlashAttention-3 or xformers.
|
| 52 |
+
|
| 53 |
### Model Download
|
| 54 |
|
| 55 |
| Models | Download Link |
|
| 56 |
| --- | --- |
|
| 57 |
+
| LongCat-Video | ๐ค [Huggingface](https://huggingface.co/meituan-longcat/LongCat-Video) |
|
| 58 |
|
| 59 |
Download models using huggingface-cli:
|
| 60 |
```shell
|
| 61 |
pip install "huggingface_hub[cli]"
|
| 62 |
+
huggingface-cli download meituan-longcat/LongCat-Video --local-dir ./weights/LongCat-Video
|
| 63 |
```
|
| 64 |
|
| 65 |
### Run Text-to-Video
|
| 66 |
|
| 67 |
```shell
|
| 68 |
# Single-GPU inference
|
| 69 |
+
torchrun run_demo_image_to_video.py --checkpoint_dir=./weights/LongCat-Video --enable_compile
|
| 70 |
|
| 71 |
# Multi-GPU inference
|
| 72 |
+
torchrun --nproc_per_node=2 run_demo_image_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video --enable_compile
|
| 73 |
```
|
| 74 |
|
| 75 |
### Run Image-to-Video
|
| 76 |
|
| 77 |
```shell
|
| 78 |
# Single-GPU inference
|
| 79 |
+
torchrun run_demo_image_to_video.py --checkpoint_dir=./weights/LongCat-Video --enable_compile
|
| 80 |
|
| 81 |
# Multi-GPU inference
|
| 82 |
+
torchrun --nproc_per_node=2 run_demo_image_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video --enable_compile
|
| 83 |
```
|
| 84 |
|
| 85 |
### Run Video-Continuation
|
| 86 |
|
| 87 |
```shell
|
| 88 |
# Single-GPU inference
|
| 89 |
+
torchrun run_demo_video_continuation.py --checkpoint_dir=./weights/LongCat-Video --enable_compile
|
| 90 |
|
| 91 |
# Multi-GPU inference
|
| 92 |
+
torchrun --nproc_per_node=2 run_demo_video_continuation.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video --enable_compile
|
| 93 |
```
|
| 94 |
|
| 95 |
### Run Long-Video Generation
|
| 96 |
|
| 97 |
```shell
|
| 98 |
# Single-GPU inference
|
| 99 |
+
torchrun run_demo_long_video.py --checkpoint_dir=./weights/LongCat-Video --enable_compile
|
| 100 |
|
| 101 |
# Multi-GPU inference
|
| 102 |
+
torchrun --nproc_per_node=2 run_demo_long_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video --enable_compile
|
| 103 |
+
```
|
| 104 |
+
|
| 105 |
+
### Run Streamlit
|
| 106 |
+
|
| 107 |
+
```shell
|
| 108 |
+
# Single-GPU inference
|
| 109 |
+
streamlit run ./run_streamlit.py --server.fileWatcherType none --server.headless=false
|
| 110 |
```
|
| 111 |
|
| 112 |
+
|
| 113 |
+
|
| 114 |
## Evaluation Results
|
| 115 |
|
| 116 |
### Text-to-Video
|
|
|
|
| 173 |
}
|
| 174 |
```
|
| 175 |
|
| 176 |
+
## Acknowledgements
|
| 177 |
+
|
| 178 |
+
We would like to thank the contributors to the [Wan](https://huggingface.co/Wan-AI), [UMT5-XXL](https://huggingface.co/google/umt5-xxl), [Diffusers](https://github.com/huggingface/diffusers) and [HuggingFace](https://huggingface.co) repositories, for their open research.
|
| 179 |
+
|
| 180 |
+
|
| 181 |
## Contact
|
| 182 |
Please contact us at <a href="mailto:longcat-team@meituan.com">longcat-team@meituan.com</a> or join our WeChat Group if you have any questions.
|
assets/longcat-logo.svg
ADDED
|
|