---
title: PrismAudio
emoji: 🎵
colorFrom: purple
colorTo: blue
sdk: gradio
sdk_version: "5.42.0"
python_version: "3.10"
app_file: app.py
pinned: false
---
PrismAudio
If you find this project useful,
a star ⭐ on GitHub would be greatly appreciated!
---
**PrismAudio** is the first framework to integrate Reinforcement Learning into Video-to-Audio (V2A) generation with specialized Chain-of-Thought (CoT) planning. Building upon [ThinkSound](https://arxiv.org/pdf/2506.21448)'s pioneering CoT-based V2A framework, PrismAudio further decomposes monolithic reasoning into four specialized CoT modules (Semantic, Temporal, Aesthetic, and Spatial), each paired with targeted reward functions, enabling multi-dimensional RL optimization that jointly improves reasoning across all perceptual dimensions.
---
## 📰 News
- **2026.03.22** 🔥 We have released **PrismAudio**, our next-generation video-to-audio generation model! Model weights are available on [Hugging Face](https://huggingface.co/FunAudioLLM/PrismAudio) and [ModelScope](https://www.modelscope.cn/models/iic/PrismAudio). For more details, please refer to the [`prismaudio`](https://github.com/liuhuadai/ThinkSound/tree/prismaudio) branch!
- **2026.01.26** 🎉 PrismAudio has been accepted to the **ICLR 2026 Main Conference**!
- **2025.11.25** 🔥 [PrismAudio Online Demo](http://prismaudio-project.github.io/) is live!
- **2025.11.25** 🔥 [PrismAudio paper](https://arxiv.org/pdf/2511.18833) released on arXiv!
- **2025.09.19** 🎉 ThinkSound has been accepted to the **NeurIPS 2025 Main Conference**!
- **2025.09.01** AudioCoT dataset is now open-sourced on [Hugging Face](https://huggingface.co/datasets/liuhuadai/AudioCoT)!
- **2025.07.17** 🧠 Finetuning enabled: training and finetuning code is now publicly available!
- **2025.07.15** 📦 Simplified installation with Windows `.bat` scripts for one-click setup!
- **2025.07.08** 🔧 Major update: model lightweighted, optimized memory and GPU usage, supports large-scale high-throughput audio generation!
- **2025.07.01** Online demo on [Hugging Face Spaces](https://huggingface.co/spaces/FunAudioLLM/ThinkSound) and [ModelScope](https://modelscope.cn/studios/iic/ThinkSound)!
- **2025.07.01** Released inference scripts and web interface!
- **2025.06** [ThinkSound paper](https://arxiv.org/pdf/2506.21448) released on arXiv!
- **2025.06** [Online Demo](http://thinksound-project.github.io/) is live!
---
## ⚡ Quick Start
For detailed training and inference code, please refer to [ThinkSound (prismaudio branch)](https://github.com/FunAudioLLM/ThinkSound/tree/prismaudio).
---
## 🚀 Features
- **V2A SOTA**: Achieves state-of-the-art results across all four perceptual dimensions on both VGGSound and AudioCanvas benchmarks.
- **Decomposed CoT Reasoning**: Four specialized CoT modules (Semantic, Temporal, Aesthetic, Spatial) each providing focused, interpretable reasoning for its corresponding perceptual dimension.
- **Multi-dimensional RL**: Fast-GRPO enables efficient multi-dimensional reward optimization without compromising generation quality.
- **New Benchmark AudioCanvas**: A rigorous V2A benchmark with 300 single-event classes and 501 multi-event samples covering diverse and challenging scenarios.
- **Efficient**: 518M parameters with faster inference than prior SOTAs.
---
## ✨ Method Overview
PrismAudio consists of three main components:
1. **CoT-Aware Audio Foundation Model**: Built on a Multimodal Diffusion Transformer with flow matching, enhanced with VideoPrism for video understanding and T5-Gemma for structured CoT text encoding.
2. **Decomposed Multi-Dimensional CoT Reasoning**: Four specialized CoT modules — Semantic, Temporal, Aesthetic, and Spatial — each providing targeted reasoning for its corresponding perceptual dimension.
3. **Fast-GRPO Multi-Dimensional RL Framework**: A hybrid ODE-SDE sampling strategy that dramatically reduces training overhead while enabling multi-dimensional reward optimization across all perceptual dimensions.
---
## 📄 License
This project is released under the Apache 2.0 License.
> **Note:**
> The code, models, and dataset are **for research and educational purposes only**.
> **Commercial use is NOT permitted.**
> For commercial licensing, please contact the authors.
**📦 Third-Party Components**
- **Stable Audio Open VAE** (by Stability AI): Licensed under the [Stability AI Community License](./third_party/LICENSE_StabilityAI.md). **Commercial use and redistribution require prior permission from Stability AI.**
- 📘 **All other code and models** are released under the Apache License 2.0.
---
## Acknowledgements
Many thanks to:
- **stable-audio-tools** (by Stability AI): For providing an easy-to-use framework for audio generation, as well as the VAE module and weights.
---
## 📖 Citation
If you find PrismAudio useful in your research or work, please cite our paper:
```bibtex
@misc{liu2025prismaudiodecomposedchainofthoughtsmultidimensional,
title={PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generation},
author={Huadai Liu and Kaicheng Luo and Wen Wang and Qian Chen and Peiwen Sun and Rongjie Huang and Xiangang Li and Jieping Ye and Wei Xue},
year={2025},
eprint={2511.18833},
archivePrefix={arXiv},
primaryClass={cs.SD},
url={https://arxiv.org/abs/2511.18833},
}
```
📬 Contact
✨ Feel free to open an issue or contact us via email (huadai.liu@connect.ust.hk) if you have any questions or suggestions!