README.md

metadata

title: OmniVAE T2AV
emoji: 🎬
colorFrom: yellow
colorTo: gray
sdk: gradio
sdk_version: 6.24.0
app_file: app.py
short_description: Text to synchronized audio + video with OmniVAE
python_version: '3.12'
startup_duration_timeout: 1h
models:
  - OpenMOSS-Team/OmniVAE
tags:
  - text-to-video
  - text-to-audio
  - audio-video
  - omnivae
license: apache-2.0

OmniVAE — Text to Audio + Video

Generate a short clip with natively synchronized sound from a single text prompt.

• Model: OpenMOSS-Team/OmniVAE (Apache-2.0)
• Code: OpenMOSS/OmniVAE (Apache-2.0)

OmniVAE is a unified audio-video tokenizer. This Space runs the released t2av_recon_distill_avclip joint text-to-audio-video model built on top of it: two Z-Image diffusion-transformer branches (video + audio) coupled by bridge cross-attention, so the picture and the soundtrack are denoised together rather than dubbed afterwards.

What it produces

The checkpoint's native shape: 121 frames · 256×256 · 24 fps · 48 kHz audio (~5 s), muxed into a single MP4. A default 50-step run takes ~97 s on ZeroGPU (measured: ~1.9 s/step plus ~3 s for text encoding, VAE decoding and muxing).

Faithfulness to the reference implementation

The app is a thin wrapper around the upstream inference path (generation/infer/t2av/t2av_pipeline.py → load_joint_av_pipeline / generate_one_av) with the defaults from the release smoke test in generation/docs/inference.md (validate_checkpoints.sh --cfg 4):

setting	value
mode	joint_av
CFG	BridgeDiT dual CFG (NFE = 3)
steps	50
guidance (text + cross-modal, both branches)	4.0
shape	121 frames, 256×256, 24 fps, 5.04 s audio
prompt wrapping	t2av task prefix on, duration suffix off

Deployment-only deltas from the upstream tree, all of which leave the numerics untouched:

• omnivae_generation/trainer/eval/guided_diffusion.py — the upstream repo imports three small checkpoint/run-dir helpers from an evaluation module it does not publish; they are re-implemented here.
• omnivae_generation/trainer/vae/_audiotools_compat.py — stand-ins for the two descript-audiotools symbols the DAC audio VAE imports (ml.BaseModel, AudioSignal). The real package pins protobuf<3.20; neither symbol carries behaviour on the continuous-VAE inference path.
• joint_av/__init__.py — the training-only torchcodec-backed dataset import is made optional.
• joint_av/validation.py — the denoising loop is wrapped in tqdm so the UI can show progress.
• torchaudio.save is routed through soundfile in app.py: torchaudio ≥ 2.10 delegates save to torchcodec, whose wheels are bound to a specific FFmpeg ABI. Both write the same 48 kHz PCM WAV.
• Components are loaded on CPU and then moved with .to("cuda"); upstream's device="cuda" path uses device_map={"": "cuda:0"} + torch.cuda.set_device, which ZeroGPU does not support.

Example prompts

Taken from the authors' own material: generation/examples/prompts/t2av_valid.jsonl and the authored prompt list in the released models/dit/t2av/t2av_recon_distill_avclip/resolved_config.json.