You need to agree to share your contact information to access this model

By clicking "Agree and Access" you acknowledge the Privacy Policy and consent to receive offers and updates. You can unsubscribe at any time.

LTX-2.3 Foley V2A LoRA

Video-to-audio Foley LoRA for LTX-2.3-22B: generates synchronized, music-free sound effects from silent video (and works for text-to-audio Foley prompts).

It is based on the LTX-2.3 foundation model.

Example Outputs

Prompt: A heavy steel hammer repeatedly strikes glowing iron against a solid steel anvil, each blow landing as a hard metallic clang with a short resonant ring, brief spark spits, and faint forge-room ambience. No speech is present. No music is present.

Prompt: Fingers type continuously on low-profile plastic laptop keys, producing rapid light clicks, muted key taps, and occasional soft contact on the aluminum palm rest in a quiet indoor room. No speech is present. No music is present.

Prompt: A fighter delivers quick punches and forceful kicks into focus pads, producing tight padded smacks and deeper leather thuds in sequence, with fast foot pivots and soft scuffs on the gym floor. No speech is present. No music is present.

Prompt: Shoes walk along weathered wooden boardwalk planks, producing firm hollow wooden footfalls, subtle plank creaks, and light sole scuffs, receding into a quiet forest. No speech is present. No music is present.

Prompt: Heavy boots step through thin snow and wet icy ground, producing soft snow compression, damp slush squelches, small ice-crust crunches, and muddy sole suction at close range. No speech is present. No music is present.

Prompt: Motorcycles ride quickly along smooth mountain asphalt, their engines producing strong throaty revs and rising mechanical roars with tire hiss and steady wind rushing past. No speech is present. No music is present.

The gallery above shows muted → foley comparison examples generated with this LoRA on IP-free stock clips (each panel plays the original silent clip against the same clip with generated Foley).

Model Files

ltx-2.3-22b-lora-foley-v2a-1.0.safetensors

Recommended checkpoint: step 500 (selected by validation loudness; later steps tend to over-suppress volume).

Model Details

Base Model: LTX-2.3-22B Video
Training Type: LoRA
Modality: video-to-audio (primary), text-to-audio
Recommended checkpoint: step 500 (ltx-2.3-22b-lora-foley-v2a-1.0.safetensors)
Audio: Trained for synchronized Foley / SFX audio generation (no speech, no music bed)

Intended Use & Out-of-Scope

Intended use: Add or regenerate Foley for silent or muted clips — footsteps, impacts, materials, ambience, tools, kitchen, weather — with prompts that describe on-screen action. Research and creative tooling on top of LTX-2.3.

Out of scope: Dialogue / speech synthesis, music generation, lip-sync performance, or replacing a full production mix. Not a substitute for licensed commercial SFX libraries when rights matter.

How It Works

The adapter targets the audio attention / FFN blocks and video→audio cross-attention, so the model keeps the input video and synthesizes a matching audio track. Training used a flexible V2A strategy (video fixed, audio generated) on cleaned Foley clips with strong music/speech suppression in captions and negatives. Prefer mid-training checkpoints (here step 500): longer training improved music rejection but often crushed loudness.

Usage

ComfyUI

Copy ltx-2.3-22b-lora-foley-v2a-1.0.safetensors into models/loras.
Load the LTX-2.3-22B base model.
Add the Foley LoRA.
For text-to-audio, use the T2A workflow from ComfyUI-LTXVideo.
For video-to-audio (primary), use the dedicated Foley V2A workflow: it encodes and freezes the input video and denoises only the audio latent, so the output is the original video frames plus generated Foley. It requires the ComfyUI-LTXVideo-Internal audio nodes (LTXVConcatAVLatent, LTXVSetAudioVideoMaskByTime, MultimodalGuider, LTXVAudioVAEDecode, …) plus ComfyUI-VideoHelperSuite. See Pipeline Details.
Start with LoRA strength 0.8–1.0 and adjust per clip.

Pipeline Details

Primary task is V2A Foley: mute or strip the source audio, feed the video as conditioning, generate audio with this LoRA.

Recommended inference recipe used for the demos:

Guidance scale 6.0 (guidance ~4 often collapses to near-silence on some seeds)
30 inference steps
STG: scale 1.0, block [29], mode stg_av
Resolution / length bucket used in training & demos: 960×544, 89 frames @ 24 fps (~3.7 s)
If a result is silent or sounds materially incorrect, try 2–3 generations and choose the one that best matches the visible material and timing—not simply the loudest.

Recommended Settings

LoRA strength / weight: 0.8–1.0 (demos used full strength)
Inference steps: 30
Guidance scale: 6.0 (prefer 6–8 over 4 for audible Foley)
Resolution & length: Use 960×544 video at 24 fps. The main training length is approximately 3.7 seconds, and the provided workflow supports clips up to approximately 7 seconds.
Spatial / audio guidance (STG): scale 1.0, blocks [29], mode stg_av
Prompting: Describe the visible sound source, action, contact material, timing, and physical sound character. Avoid using the same generic crackle/crinkle wording for unrelated materials. End with No speech is present. No music is present. when you want clean SFX-only audio.
Multi-seed fallback: If output is silent or sounds materially wrong, change seed and try 2–3 generations, choosing the one that best matches the visible material and timing—not simply the loudest.
Checkpoint choice: Prefer step 500 over later steps for volume; later steps can over-suppress.

Example positive prompt:

A sledgehammer smashes a glass bottle: a heavy whoosh, a sharp glass shatter and tinkling shards. No speech is present. No music is present.

Example negative prompt:

music, melody, song, singing, vocals, score, soundtrack, beat, rhythm bed, instrumental backing, speech, dialogue, talking, narration, tinny, thin, harsh, clipped, distorted, low bitrate

References

Code: GitHub Repository
ComfyUI: ComfyUI-LTXVideo

Tips & Troubleshooting

Near-silent output: Raise guidance to 6+, try a new seed, and confirm you are on the step-500 weights (not a late over-suppressed checkpoint).
Music bleed: Keep the music/speech negative prompt; reinforce No music is present / No speech is present in the positive prompt.
Weak ambience (rain, fire, pottery, typing): These are seed-sensitive — multi-seed and pick the best; slightly more assertive “loud / close-mic / prominent” wording helps.
Sync: Prompt the action you can see (impacts, footsteps, materials). Mismatched prompts produce plausible but unsynced SFX.

Dataset

~5,374 short clips (~3.7 s, 960×544 @ 24 fps):

Curated / cleaned Foley rebuild set (Demucs + HPSS filtering, silence/music rejection, captions)
FoleyBench — 5k non-speech / non-music clips (CC BY-NC-SA; academic / non-commercial terms apply to that portion)

Captions emphasize on-screen Foley and explicitly exclude speech and music.

Training

Technique: LoRA on audio self-attn, audio cross-attn, video-to-audio attn, and audio FFN; flexible V2A strategy (video latents fixed, audio generated). Warm-started from an earlier Foley LoRA checkpoint.
Hyperparameters: rank 32, alpha 32, lr 1e-5, AdamW, linear schedule (start_factor 0.1 → end_factor 0.05), batch size 1, grad accumulation 2, max_grad_norm 0.5, bf16, gradient checkpointing
Steps: 3000 (best validation loudness at step 500)
Infrastructure: LTX-2 Community Trainer (single-GPU run after multi-GPU NaN instability)