data_recipe.md

How the "Helium" Dataset Was Built

This is the companion to the model card; the actual recipe used here and how to rebuild it. The why (why pitch, why a bare tone, the seesaw, the correlation hole) is all in the README; this is just the how. It's not perfect, but hopefully provides a way for others to make better ones.

Where It Comes From

Real talking-head footage, so the target side has no synthetic-to-real gap: the CelebVHQ and TalkVid preprocessed sets. Both are 512×512, 25 fps, ~5-second (121-frame) clips; I re-render the targets at 256×256 due to consumer grade GPU limitations (and video wasn't the target goal).

How Each Pair Is Made

For a clip, I measure its natural speaking pitch, then pick a target pitch that's a bounded shift off it (at most ±7 semitones, ~3.6 on average), and pitch shift the speech to that target while keeping the timing so the lips still line up. The video stays untouched. The reference paired with it is a synthesized voiced tone at the same target pitch: 2 seconds, 16 kHz, mono, with the timbre varied across four flavors that are decorrelated from pitch so the model can't read timbre as a pitch shortcut. Every pair gets the same caption, "a person speaking", and no init frame, so the tone is the only non-caption input.

Then a quality gate: keep a pair only if the shifted speech re-measures within 30 Hz of its target pitch. That drops the clips where the pitch-shift fell apart and leaves 292 clean pairs out of 300, spanning 76 to 353 Hz. Two correlations worth knowing: reference timbre vs. pitch comes out near zero (~0.03), which is what I wanted; target pitch vs. the clip's own natural pitch is ~0.75, because the target is a bounded shift off it, which is the known weak spot (the README explains why it matters). The fix for the next run is to pick the target pitch at random, independent of the clip, with a formant-preserving shift.

What the Dataset Looks Like on Disk

This is the layout of the dataset itself, what the trainer reads, not of this HF download. The dataset isn't shipped here (it's derived from the CelebVHQ/TalkVid sets above, and it's large); this release contains only the IC-LoRA checkpoints, the training config, and this recipe (and a link to the github trainer fork). The structure below is what you'd produce by rebuilding it.

pitch_ref_gate_v1/
  clips/clip_NNNN.mp4            target AV: shifted audio, untouched video, 256x256
  references/ref_NNNN.wav        the 2.0s voiced reference tone at the target pitch
  captions.json                 the constant caption
  manifest.jsonl                all 300 generated rows (pre-filter)
  manifest_train.jsonl          the 292 clean rows after the quality gate
  precomputed/
    latents/                    video latents
    audio_latents/              target audio latents
    conditions/                 text conditions
    reference_audio_latents/    the reference-tone latents (the IC-LoRA reference channel)

Each row in manifest_train.jsonl carries the video and reference paths, the caption, the natural / target / actual (re-measured) pitches, the shift in semitones, a timbre id, and a train/heldout split label. One heads-up if you're reproducing: the trainer finds samples by globbing the precomputed .pt files and doesn't read that split label, so it trains on everything; the label is informational unless you wire it up yourself.

Rebuilding It

The scripts and the audio_reference strategy live in the trainer fork, fblissjr/LTX-2 @ audio-guidance-iclora-vtv, under packages/ltx-trainer/. Clone it and set up the env (it uses uv), then run these from packages/ltx-trainer/ (paths below are relative to it):

# 1. encode the targets: video + audio latents + text conditions
uv run python scripts/process_videos.py --with-audio \
  --dataset-dir <dataset_dir> \
  --model-path <ltx2_distilled_checkpoint> \
  --text-encoder-path <text_encoder>

# 2. encode the reference tones into the reference_audio_latents channel
uv run python scripts/precompute_reference_audio.py \
  --dataset-dir <dataset_dir> \
  --reference-key reference \
  --output-dir <dataset_dir>/precomputed/reference_audio_latents \
  --model-path <ltx2_distilled_checkpoint>

# 3. train (config ships alongside this file)
uv run python scripts/train.py ltx2_audio_reference.yaml

The reference encode runs the audio VAE in float32 to match the target-audio encode; the LoRA itself is bf16. Target modules, rank, and the rest of the setup are in ltx2_audio_reference.yaml and the README.

What I'd Change Next Time

The organizing idea first: a clean reference at negative RoPE positions suits time-invariant attributes (an average pitch, a timbre, a voice). That is my read, not something I tested, but it shapes the list.

• Fix the 0.75 by construction, not luck. Reuse each clip at several target pitches, each with its own reference tone, caption held constant. The same face then appears across many pitches, so identity can't predict the answer and the tone is the only thing that co-varies, and it multiplies a too-small dataset for free. Cheap partial version: pick the target pitch at random, independent of the clip, with a formant-preserving shift.
• Voice transfer is the real goal: a reference of someone speaking, and the generated voice adopts it. Voice identity is time-invariant, so it fits. Harder than pitch, for the same reason the 0.75 hole exists: the generated face carries a voice prior the audio reference has to override. The clean test is two variants on the same voice data, audio-only reference versus image+audio. Pitch is the leading indicator: if pure-audio pitch doesn't move, pure-audio voice almost certainly won't.
• Don't chase a time-varying contour. A negative-RoPE reference sits in a disjoint region with no alignment to the target timeline, so there is nothing to learn a moment-to-moment mapping from. Time-varying control (prosody, rhythm) wants an aligned driving channel at positive positions, the way LipDub drives lips frame by frame. Different experiment.
• If the data is fine but the adapter is just too weak, the lever is the model side: add the cross-modal bridges (audio_to_video_attn, video_to_audio_attn), not more data.