---
title: OmniVAE T2AV
emoji: 🎬
colorFrom: yellow
colorTo: gray
sdk: gradio
sdk_version: 6.24.0
app_file: app.py
short_description: Text to synchronized audio + video with OmniVAE
python_version: "3.12"
startup_duration_timeout: 1h
models:
  - OpenMOSS-Team/OmniVAE
tags:
  - text-to-video
  - text-to-audio
  - audio-video
  - omnivae
license: apache-2.0
---

# OmniVAE — Text to Audio + Video

Generate a short clip with **natively synchronized sound** from a single text prompt.

- Model: [`OpenMOSS-Team/OmniVAE`](https://huggingface.co/OpenMOSS-Team/OmniVAE) (Apache-2.0)
- Code: [OpenMOSS/OmniVAE](https://github.com/OpenMOSS/OmniVAE) (Apache-2.0)

OmniVAE is a unified audio-video tokenizer. This Space runs the released
`t2av_recon_distill_avclip` joint text-to-audio-video model built on top of it: two
Z-Image diffusion-transformer branches (video + audio) coupled by bridge cross-attention,
so the picture and the soundtrack are denoised together rather than dubbed afterwards.

## What it produces

The checkpoint's native shape: **121 frames · 256×256 · 24 fps · 48 kHz audio (~5 s)**,
muxed into a single MP4. A default 50-step run takes ~97 s on ZeroGPU (measured:
~1.9 s/step plus ~3 s for text encoding, VAE decoding and muxing).

## Faithfulness to the reference implementation

The app is a thin wrapper around the upstream inference path
(`generation/infer/t2av/t2av_pipeline.py` → `load_joint_av_pipeline` / `generate_one_av`)
with the defaults from the release smoke test in `generation/docs/inference.md`
(`validate_checkpoints.sh --cfg 4`):

| setting | value |
| --- | --- |
| mode | `joint_av` |
| CFG | BridgeDiT dual CFG (NFE = 3) |
| steps | 50 |
| guidance (text + cross-modal, both branches) | 4.0 |
| shape | 121 frames, 256×256, 24 fps, 5.04 s audio |
| prompt wrapping | `t2av` task prefix on, duration suffix off |

Deployment-only deltas from the upstream tree, all of which leave the numerics untouched:

- `omnivae_generation/trainer/eval/guided_diffusion.py` — the upstream repo imports three
  small checkpoint/run-dir helpers from an evaluation module it does not publish; they are
  re-implemented here.
- `omnivae_generation/trainer/vae/_audiotools_compat.py` — stand-ins for the two
  `descript-audiotools` symbols the DAC audio VAE imports (`ml.BaseModel`, `AudioSignal`).
  The real package pins `protobuf<3.20`; neither symbol carries behaviour on the
  continuous-VAE inference path.
- `joint_av/__init__.py` — the training-only `torchcodec`-backed dataset import is made
  optional.
- `joint_av/validation.py` — the denoising loop is wrapped in `tqdm` so the UI can show
  progress.
- `torchaudio.save` is routed through `soundfile` in `app.py`: torchaudio ≥ 2.10 delegates
  `save` to `torchcodec`, whose wheels are bound to a specific FFmpeg ABI. Both write the
  same 48 kHz PCM WAV.
- Components are loaded on CPU and then moved with `.to("cuda")`; upstream's
  `device="cuda"` path uses `device_map={"": "cuda:0"}` + `torch.cuda.set_device`, which
  ZeroGPU does not support.

## Example prompts

Taken from the authors' own material: `generation/examples/prompts/t2av_valid.jsonl` and
the authored prompt list in the released
`models/dit/t2av/t2av_recon_distill_avclip/resolved_config.json`.