Add license, base model, and full author list to model card
#1
by nielsr HF Staff - opened
README.md
CHANGED
|
@@ -1,4 +1,9 @@
|
|
| 1 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
tags:
|
| 3 |
- audio-visual
|
| 4 |
- world-model
|
|
@@ -6,9 +11,6 @@ tags:
|
|
| 6 |
- real-time
|
| 7 |
- streaming-generation
|
| 8 |
- video-generation
|
| 9 |
-
language:
|
| 10 |
-
- en
|
| 11 |
-
pipeline_tag: any-to-any
|
| 12 |
---
|
| 13 |
|
| 14 |
# MaineCoon: Pursuing A Real-Time Audio-Visual Social World Model
|
|
@@ -17,91 +19,59 @@ pipeline_tag: any-to-any
|
|
| 17 |
|
| 18 |
| | |
|
| 19 |
|---|---|
|
| 20 |
-
| 🌐 Project | https://mainecoon.tech/ |
|
| 21 |
-
| 🕹️ Experience | https://mainecoon.tech/experience-platform |
|
| 22 |
-
| 📄 Paper (arXiv) | https://arxiv.org/abs/2606.17800 |
|
| 23 |
-
| 📝 Blog | https://mainecoon.tech/blogs |
|
| 24 |
-
| 💻 GitHub | https://github.com/catnip-ai-tech/MaineCoon |
|
| 25 |
|
| 26 |
## Abstract
|
| 27 |
|
| 28 |
-
As
|
| 29 |
-
|
| 30 |
-
## Highlights
|
| 31 |
|
| 32 |
-
|
| 33 |
-
- **🌍 A new paradigm: social world models.** MaineCoon positions and serves as the first generative core for *social world models*, a technical foundation for next-generation AI-native social platforms.
|
| 34 |
-
- **🎓 Forcing-free streaming training.** A multi-stage training paradigm — **self-resampling**, **cross-modal representation alignment**, **domain-aware preference optimization**, and **reinforced online-policy distillation (ROPD)** — that enables native, efficient streaming audio-visual training at 22B scale.
|
| 35 |
-
- **🧠 Agentic streaming inference.** An agentic inference framework that supports **thousand-second-scale** generation while mitigating drift through agentic cache management, chunk commitment, long-context rollout, and prompt planning.
|
| 36 |
-
- **📊 SocialVideo-Bench.** A new benchmark focused on audio-visual social-video generation, with 9 representative metrics covering visual quality, motion, audio quality, audio-visual alignment, and social-video harmony. MaineCoon outperforms 7 representative open audio-visual models while achieving the fastest generation speed — a new state of the art for real-time social video generation.
|
| 37 |
|
| 38 |
-
##
|
| 39 |
-
|
| 40 |
-
Hand-picked MaineCoon generations (audio-visual, with sound) play directly in the **[GitHub repository](https://github.com/catnip-ai-tech/MaineCoon)**.
|
| 41 |
|
| 42 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 43 |
|
| 44 |
## Benchmark — SocialVideo-Bench
|
| 45 |
|
| 46 |
-
**
|
| 47 |
|
| 48 |
-
| Type | Model | Vis↑ | Mot↑ | Aud↑ |
|
| 49 |
-
|:--|:--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
|
| 50 |
-
| Bidirectional T2AV |
|
| 51 |
-
|
|
| 52 |
-
|
|
| 53 |
-
| | MoVA | _4.66_ | 1.68 | 3.69 | 0.133 | 0.105 | 0.258 | _0.359_ | 0.245 | 0.216 | 0.842 |
|
| 54 |
-
| | LTX-2.3 | 4.10 | 0.99 | 4.06 | 0.132 | 0.111 | 0.311 | 0.334 | 0.287 | _0.247_ | 0.848 |
|
| 55 |
-
| Streaming TA2V | LiveAvatar | 4.60 | 1.46 | _4.13_ | 0.131 | 0.120 | _0.316_ | 0.326 | _0.291_ | 0.246 | 0.892 |
|
| 56 |
-
| | SoulX-FlashTalk | 4.65 | _1.99_ | 4.07 | 0.128 | _0.120_ | 0.307 | 0.279 | 0.283 | 0.238 | _0.895_ |
|
| 57 |
-
| **Streaming T2AV** | 🐱 **MaineCoon (Ours)** | **4.71** | 1.62 | **4.35** | 0.127 | **0.130** | **0.318** | 0.334 | **0.308** | **0.272** | **0.934** 🥇 |
|
| 58 |
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
**Table 3. Latency and model size comparison.** Sampling throughput (FPS) is measured for 480P 20-second generation on a single H100 GPU. 🐱 **MaineCoon (Ours)** has the **largest model yet by far the fastest** speed — up to **7× faster** than other streaming audio-visual generators, and faster even than a 1.3B streaming video model.
|
| 62 |
|
| 63 |
| Type | Model | Params | FPS↑ |
|
| 64 |
|:--|:--|:--:|:--:|
|
| 65 |
-
| Bidirectional T2AV |
|
| 66 |
-
| | Ovi | 11B | 0.58 |
|
| 67 |
-
| | JoyAI-Echo | 23B | 18.0 |
|
| 68 |
-
| | MoVA | 32B | 0.26 |
|
| 69 |
-
| | LTX-2.3 | 22B | 1.40 |
|
| 70 |
-
| | LTX-2.3-Distilled | 22B | _20.7_ |
|
| 71 |
| Streaming T2V | Causal-Forcing | 1.3B | 19.1 |
|
| 72 |
-
| | Helios-Distilled | 14B | 18.2 |
|
| 73 |
-
| | Krea | 14B | 6.1 |
|
| 74 |
-
| Streaming TA2V | LiveAvatar | 14B | 6.7 |
|
| 75 |
-
| | SoulX-FlashTalk | 14B | 6.6 |
|
| 76 |
| **Streaming T2AV** | 🐱 **MaineCoon (Ours)** | **22B** | **47.5** 🥇 |
|
| 77 |
|
| 78 |
-
<sub>🐱 = our method · **bold** = best, _italic_ = second best. FPS for 480P-20s on a single H100.</sub>
|
| 79 |
-
|
| 80 |
-
## Paper
|
| 81 |
-
|
| 82 |
-
The full paper is available on **[arXiv:2606.17800](https://arxiv.org/abs/2606.17800)**. A PDF copy is also included in this repository: [`MaineCoon_Technical_Report.pdf`](./MaineCoon_Technical_Report.pdf). It covers the social-video data infrastructure, the native streaming autoregressive training recipe, the agentic streaming inference framework, SocialVideo-Bench, and a position/outlook on social world models.
|
| 83 |
-
|
| 84 |
## Acknowledgements
|
| 85 |
|
| 86 |
MaineCoon stands on the shoulders of the open-source community. We are especially grateful to:
|
| 87 |
|
| 88 |
-
- **🎬 LTX-2.3 & the LTX series — [Lightricks](https://github.com/Lightricks).** MaineCoon's audio-visual backbone builds on the excellent open **LTX-2.3** model.
|
| 89 |
-
|
| 90 |
-
- **LTX-Video**: https://github.com/Lightricks/LTX-Video
|
| 91 |
-
- **⚡ DMD series & the distribution-matching distillation community.** Our reinforced online-policy distillation (ROPD) builds on the **Distribution Matching Distillation (DMD / DMD2)** line of work and the wider few-step / real-time distillation community.
|
| 92 |
-
- **DMD2**: https://github.com/tianweiy/DMD2
|
| 93 |
-
- **DMD** (project page): https://tianweiy.github.io/dmd/
|
| 94 |
-
|
| 95 |
-
We thank these projects and their communities for advancing real-time, few-step, and streaming video generation.
|
| 96 |
|
| 97 |
## Citation
|
| 98 |
|
| 99 |
```bibtex
|
| 100 |
@article{catnip2026mainecoon,
|
| 101 |
title = {MaineCoon: Pursuing A Real-Time Audio-Visual Social World Model},
|
| 102 |
-
author = {
|
| 103 |
year = {2026},
|
| 104 |
journal = {arXiv preprint arXiv:2606.17800},
|
| 105 |
url = {https://arxiv.org/abs/2606.17800}
|
| 106 |
}
|
| 107 |
-
```
|
|
|
|
| 1 |
---
|
| 2 |
+
language:
|
| 3 |
+
- en
|
| 4 |
+
license: apache-2.0
|
| 5 |
+
pipeline_tag: any-to-any
|
| 6 |
+
base_model: Lightricks/LTX-Video
|
| 7 |
tags:
|
| 8 |
- audio-visual
|
| 9 |
- world-model
|
|
|
|
| 11 |
- real-time
|
| 12 |
- streaming-generation
|
| 13 |
- video-generation
|
|
|
|
|
|
|
|
|
|
| 14 |
---
|
| 15 |
|
| 16 |
# MaineCoon: Pursuing A Real-Time Audio-Visual Social World Model
|
|
|
|
| 19 |
|
| 20 |
| | |
|
| 21 |
|---|---|
|
| 22 |
+
| 🌐 Project | [mainecoon.tech](https://mainecoon.tech/) |
|
| 23 |
+
| 🕹️ Experience | [Try it live](https://mainecoon.tech/experience-platform) |
|
| 24 |
+
| 📄 Paper (arXiv) | [arXiv:2606.17800](https://arxiv.org/abs/2606.17800) |
|
| 25 |
+
| 📝 Blog | [Technical Blog](https://mainecoon.tech/blogs) |
|
| 26 |
+
| 💻 GitHub | [catnip-ai-tech/MaineCoon](https://github.com/catnip-ai-tech/MaineCoon) |
|
| 27 |
|
| 28 |
## Abstract
|
| 29 |
|
| 30 |
+
As video content is increasingly consumed on social platforms, video generation models built for social worlds are important but largely overlooked. We present **MaineCoon**, the first real-time audio-visual autoregressive model. With **22B parameters**, it is capable of real-time streaming generation and sub-second interaction, achieving a record-breaking frame rate of **up to 47.5 FPS on a single GPU**.
|
|
|
|
|
|
|
| 31 |
|
| 32 |
+
MaineCoon is optimized for social-interactive applications using several novel techniques: self-resampling, cross-modal representation alignment, domain-aware preference optimization, and reinforced online-policy distillation (ROPD). We also introduce an agentic streaming inference framework that supports thousand-second-scale generation while mitigating drift.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
|
| 34 |
+
## Highlights
|
|
|
|
|
|
|
| 35 |
|
| 36 |
+
- **⚡ Real-time on a single GPU.** Capable of streaming generation and sub-second interaction on a single H100.
|
| 37 |
+
- **🌍 A new paradigm: social world models.** Serves as the first generative core for social world models, a foundation for next-generation AI-native social platforms.
|
| 38 |
+
- **🎓 Forcing-free streaming training.** Multi-stage training enabling native, efficient streaming audio-visual training at 22B scale.
|
| 39 |
+
- **🧠 Agentic streaming inference.** Supports long-horizon generation through agentic cache management and prompt planning.
|
| 40 |
+
- **📊 SocialVideo-Bench.** A new benchmark where MaineCoon outperforms 7 representative open audio-visual models while achieving the fastest generation speed.
|
| 41 |
|
| 42 |
## Benchmark — SocialVideo-Bench
|
| 43 |
|
| 44 |
+
**Main quantitative results on SocialVideo-Bench.** 🐱 **MaineCoon (Ours)** achieves the best average score and wins most metrics, including Audio-Visual Harmony (AVH) and Joint Audio-Visual Integrated Score (JAVIS).
|
| 45 |
|
| 46 |
+
| Type | Model | Vis↑ | Mot↑ | Aud↑ | AV-Al↑ | AVH↑ | JAVIS↑ | Average↑ |
|
| 47 |
+
|:--|:--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
|
| 48 |
+
| Bidirectional T2AV | LTX-2.3 | 4.10 | 0.99 | 4.06 | 0.334 | 0.287 | 0.247 | 0.848 |
|
| 49 |
+
| Streaming TA2V | SoulX-FlashTalk | 4.65 | 1.99 | 4.07 | 0.279 | 0.283 | 0.238 | 0.895 |
|
| 50 |
+
| **Streaming T2AV** | 🐱 **MaineCoon (Ours)** | **4.71** | 1.62 | **4.35** | 0.334 | **0.308** | **0.272** | **0.934** 🥇 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 51 |
|
| 52 |
+
**Latency and model size comparison.** Sampling throughput (FPS) measured for 480P 20-second generation on a single H100 GPU.
|
|
|
|
|
|
|
| 53 |
|
| 54 |
| Type | Model | Params | FPS↑ |
|
| 55 |
|:--|:--|:--:|:--:|
|
| 56 |
+
| Bidirectional T2AV | LTX-2.3-Distilled | 22B | 20.7 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 57 |
| Streaming T2V | Causal-Forcing | 1.3B | 19.1 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
| **Streaming T2AV** | 🐱 **MaineCoon (Ours)** | **22B** | **47.5** 🥇 |
|
| 59 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
## Acknowledgements
|
| 61 |
|
| 62 |
MaineCoon stands on the shoulders of the open-source community. We are especially grateful to:
|
| 63 |
|
| 64 |
+
- **🎬 LTX-2.3 & the LTX series — [Lightricks](https://github.com/Lightricks).** MaineCoon's audio-visual backbone builds on the excellent open **LTX-2.3** model.
|
| 65 |
+
- **⚡ DMD series.** Our reinforced online-policy distillation (ROPD) builds on the **Distribution Matching Distillation (DMD / DMD2)** line of work.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 66 |
|
| 67 |
## Citation
|
| 68 |
|
| 69 |
```bibtex
|
| 70 |
@article{catnip2026mainecoon,
|
| 71 |
title = {MaineCoon: Pursuing A Real-Time Audio-Visual Social World Model},
|
| 72 |
+
author = {Lichen Bai and Tianhao Zhang and Shitong Shao and Dingwei Tan and Qiyu Zhong and Zhengpeng Xie and Haopeng Li and Qinghao Huang and Dandan Shen and Tengjiao Ji and Wei Wang and Peicheng Wu and Yuxuan Zhao and Xiangyu Zhu and Welly Luo and Shurui Yang and Zeke Xie},
|
| 73 |
year = {2026},
|
| 74 |
journal = {arXiv preprint arXiv:2606.17800},
|
| 75 |
url = {https://arxiv.org/abs/2606.17800}
|
| 76 |
}
|
| 77 |
+
```
|