thewh1teagle/vocos-mel-24khz-causal

Vocos (mel, 24 kHz) with causal streaming support

Causal fine-tune of charactr/vocos-mel-24khz for real-time streaming inference. Mel frames go in chunk by chunk, audio comes out with ~85 ms algorithmic latency (4-frame lookahead + ISTFT overlap-add), numerically identical to the offline forward pass.

Code, streaming inference (VocosStreamer), and training setup: https://github.com/thewh1teagle/vocos

Model details

• Backbone: VocosBackbone (dim 512, intermediate 1536, 8 ConvNeXt layers) with causal=True, lookahead_frames=4
• Head: ISTFTHead, n_fft 1024, hop 256, padding "same"
• Features: 100-bin mel, 24 kHz
• Initialized from the pretrained non-causal charactr/vocos-mel-24khz weights (weight-compatible)

Training data

Phase	Data	Steps
1	LibriTTS-R (train-clean, 148,694 files, ~460 h)	0 → 235k
2	LibriTTS-R + 20% Hebrew podcast speech (37,174 segments, ~76 h, quality-filtered, upsampled 16→24 kHz)	235k → current

Trained with the standard Vocos GAN objective (multi-period + multi-resolution discriminators, mel + adversarial + feature-matching losses), batch 16, 16384-sample crops. This checkpoint is a full PyTorch Lightning checkpoint (~epoch 14, step ~277k) including generator, discriminators, and optimizer states — usable both for inference (filter the feature_extractor./backbone./head. keys) and to resume training.

Usage

See examples/streaming_infer_ckpt.py for loading this checkpoint and streaming audio chunk by chunk.