Spaces:
Runtime error
title: PrismAudio
emoji: π΅
colorFrom: purple
colorTo: blue
sdk: gradio
sdk_version: 5.42.0
python_version: '3.10'
app_file: app.py
pinned: false
PrismAudio
If you find this project useful,
a star β on GitHub would be greatly appreciated!
PrismAudio is the first framework to integrate Reinforcement Learning into Video-to-Audio (V2A) generation with specialized Chain-of-Thought (CoT) planning. Building upon ThinkSound's pioneering CoT-based V2A framework, PrismAudio further decomposes monolithic reasoning into four specialized CoT modules (Semantic, Temporal, Aesthetic, and Spatial), each paired with targeted reward functions, enabling multi-dimensional RL optimization that jointly improves reasoning across all perceptual dimensions.
π° News
- 2026.03.22 π₯ We have released PrismAudio, our next-generation video-to-audio generation model! Model weights are available on Hugging Face and ModelScope. For more details, please refer to the
prismaudiobranch! - 2026.01.26 π PrismAudio has been accepted to the ICLR 2026 Main Conference!
- 2025.11.25 π₯ PrismAudio Online Demo is live!
- 2025.11.25 π₯ PrismAudio paper released on arXiv!
- 2025.09.19 π ThinkSound has been accepted to the NeurIPS 2025 Main Conference!
- 2025.09.01 AudioCoT dataset is now open-sourced on Hugging Face!
- 2025.07.17 π§ Finetuning enabled: training and finetuning code is now publicly available!
- 2025.07.15 π¦ Simplified installation with Windows
.batscripts for one-click setup! - 2025.07.08 π§ Major update: model lightweighted, optimized memory and GPU usage, supports large-scale high-throughput audio generation!
- 2025.07.01 Online demo on Hugging Face Spaces and ModelScope!
- 2025.07.01 Released inference scripts and web interface!
- 2025.06 ThinkSound paper released on arXiv!
- 2025.06 Online Demo is live!
β‘ Quick Start
For detailed training and inference code, please refer to ThinkSound (prismaudio branch).
π Features
- V2A SOTA: Achieves state-of-the-art results across all four perceptual dimensions on both VGGSound and AudioCanvas benchmarks.
- Decomposed CoT Reasoning: Four specialized CoT modules (Semantic, Temporal, Aesthetic, Spatial) each providing focused, interpretable reasoning for its corresponding perceptual dimension.
- Multi-dimensional RL: Fast-GRPO enables efficient multi-dimensional reward optimization without compromising generation quality.
- New Benchmark AudioCanvas: A rigorous V2A benchmark with 300 single-event classes and 501 multi-event samples covering diverse and challenging scenarios.
- Efficient: 518M parameters with faster inference than prior SOTAs.
β¨ Method Overview
PrismAudio consists of three main components:
- CoT-Aware Audio Foundation Model: Built on a Multimodal Diffusion Transformer with flow matching, enhanced with VideoPrism for video understanding and T5-Gemma for structured CoT text encoding.
- Decomposed Multi-Dimensional CoT Reasoning: Four specialized CoT modules β Semantic, Temporal, Aesthetic, and Spatial β each providing targeted reasoning for its corresponding perceptual dimension.
- Fast-GRPO Multi-Dimensional RL Framework: A hybrid ODE-SDE sampling strategy that dramatically reduces training overhead while enabling multi-dimensional reward optimization across all perceptual dimensions.
π License
This project is released under the Apache 2.0 License.
Note: The code, models, and dataset are for research and educational purposes only. Commercial use is NOT permitted. For commercial licensing, please contact the authors.
π¦ Third-Party Components
- Stable Audio Open VAE (by Stability AI): Licensed under the Stability AI Community License. Commercial use and redistribution require prior permission from Stability AI.
- π All other code and models are released under the Apache License 2.0.
Acknowledgements
Many thanks to:
- stable-audio-tools (by Stability AI): For providing an easy-to-use framework for audio generation, as well as the VAE module and weights.
π Citation
If you find PrismAudio useful in your research or work, please cite our paper:
@misc{liu2025prismaudiodecomposedchainofthoughtsmultidimensional,
title={PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generation},
author={Huadai Liu and Kaicheng Luo and Wen Wang and Qian Chen and Peiwen Sun and Rongjie Huang and Xiangang Li and Jieping Ye and Wei Xue},
year={2025},
eprint={2511.18833},
archivePrefix={arXiv},
primaryClass={cs.SD},
url={https://arxiv.org/abs/2511.18833},
}
π¬ Contact β¨ Feel free to open an issue or contact us via email (huadai.liu@connect.ust.hk) if you have any questions or suggestions!