Spaces:
Running on Zero
Running on Zero
File size: 3,270 Bytes
5471346 e08bfc9 5471346 e08bfc9 5471346 e08bfc9 dbff0a2 e08bfc9 dbff0a2 e08bfc9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | ---
title: OmniVAE T2AV
emoji: 🎬
colorFrom: yellow
colorTo: gray
sdk: gradio
sdk_version: 6.24.0
app_file: app.py
short_description: Text to synchronized audio + video with OmniVAE
python_version: "3.12"
startup_duration_timeout: 1h
models:
- OpenMOSS-Team/OmniVAE
tags:
- text-to-video
- text-to-audio
- audio-video
- omnivae
license: apache-2.0
---
# OmniVAE — Text to Audio + Video
Generate a short clip with **natively synchronized sound** from a single text prompt.
- Model: [`OpenMOSS-Team/OmniVAE`](https://huggingface.co/OpenMOSS-Team/OmniVAE) (Apache-2.0)
- Code: [OpenMOSS/OmniVAE](https://github.com/OpenMOSS/OmniVAE) (Apache-2.0)
OmniVAE is a unified audio-video tokenizer. This Space runs the released
`t2av_recon_distill_avclip` joint text-to-audio-video model built on top of it: two
Z-Image diffusion-transformer branches (video + audio) coupled by bridge cross-attention,
so the picture and the soundtrack are denoised together rather than dubbed afterwards.
## What it produces
The checkpoint's native shape: **121 frames · 256×256 · 24 fps · 48 kHz audio (~5 s)**,
muxed into a single MP4. A default 50-step run takes ~97 s on ZeroGPU (measured:
~1.9 s/step plus ~3 s for text encoding, VAE decoding and muxing).
## Faithfulness to the reference implementation
The app is a thin wrapper around the upstream inference path
(`generation/infer/t2av/t2av_pipeline.py` → `load_joint_av_pipeline` / `generate_one_av`)
with the defaults from the release smoke test in `generation/docs/inference.md`
(`validate_checkpoints.sh --cfg 4`):
| setting | value |
| --- | --- |
| mode | `joint_av` |
| CFG | BridgeDiT dual CFG (NFE = 3) |
| steps | 50 |
| guidance (text + cross-modal, both branches) | 4.0 |
| shape | 121 frames, 256×256, 24 fps, 5.04 s audio |
| prompt wrapping | `t2av` task prefix on, duration suffix off |
Deployment-only deltas from the upstream tree, all of which leave the numerics untouched:
- `omnivae_generation/trainer/eval/guided_diffusion.py` — the upstream repo imports three
small checkpoint/run-dir helpers from an evaluation module it does not publish; they are
re-implemented here.
- `omnivae_generation/trainer/vae/_audiotools_compat.py` — stand-ins for the two
`descript-audiotools` symbols the DAC audio VAE imports (`ml.BaseModel`, `AudioSignal`).
The real package pins `protobuf<3.20`; neither symbol carries behaviour on the
continuous-VAE inference path.
- `joint_av/__init__.py` — the training-only `torchcodec`-backed dataset import is made
optional.
- `joint_av/validation.py` — the denoising loop is wrapped in `tqdm` so the UI can show
progress.
- `torchaudio.save` is routed through `soundfile` in `app.py`: torchaudio ≥ 2.10 delegates
`save` to `torchcodec`, whose wheels are bound to a specific FFmpeg ABI. Both write the
same 48 kHz PCM WAV.
- Components are loaded on CPU and then moved with `.to("cuda")`; upstream's
`device="cuda"` path uses `device_map={"": "cuda:0"}` + `torch.cuda.set_device`, which
ZeroGPU does not support.
## Example prompts
Taken from the authors' own material: `generation/examples/prompts/t2av_valid.jsonl` and
the authored prompt list in the released
`models/dit/t2av/t2av_recon_distill_avclip/resolved_config.json`.
|