Upload folder using huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
- he
|
| 6 |
+
tags:
|
| 7 |
+
- vocoder
|
| 8 |
+
- audio
|
| 9 |
+
- speech
|
| 10 |
+
- streaming
|
| 11 |
+
- vocos
|
| 12 |
+
base_model: charactr/vocos-mel-24khz
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
# Vocos (mel, 24 kHz) with causal streaming support
|
| 16 |
+
|
| 17 |
+
Causal fine-tune of [charactr/vocos-mel-24khz](https://huggingface.co/charactr/vocos-mel-24khz) for real-time streaming inference.
|
| 18 |
+
Mel frames go in chunk by chunk, audio comes out with ~85 ms algorithmic latency (4-frame lookahead + ISTFT overlap-add),
|
| 19 |
+
numerically identical to the offline forward pass.
|
| 20 |
+
|
| 21 |
+
Code, streaming inference (`VocosStreamer`), and training setup: **https://github.com/thewh1teagle/vocos**
|
| 22 |
+
|
| 23 |
+
## Model details
|
| 24 |
+
|
| 25 |
+
- Backbone: `VocosBackbone` (dim 512, intermediate 1536, 8 ConvNeXt layers) with `causal=True`, `lookahead_frames=4`
|
| 26 |
+
- Head: `ISTFTHead`, n_fft 1024, hop 256, padding "same"
|
| 27 |
+
- Features: 100-bin mel, 24 kHz
|
| 28 |
+
- Initialized from the pretrained non-causal `charactr/vocos-mel-24khz` weights (weight-compatible)
|
| 29 |
+
|
| 30 |
+
## Training data
|
| 31 |
+
|
| 32 |
+
| Phase | Data | Steps |
|
| 33 |
+
|---|---|---|
|
| 34 |
+
| 1 | LibriTTS-R (train-clean, 148,694 files, ~460 h) | 0 → 235k |
|
| 35 |
+
| 2 | LibriTTS-R + 20% Hebrew podcast speech (37,174 segments, ~76 h, quality-filtered, upsampled 16→24 kHz) | 235k → current |
|
| 36 |
+
|
| 37 |
+
Trained with the standard Vocos GAN objective (multi-period + multi-resolution discriminators, mel + adversarial + feature-matching losses),
|
| 38 |
+
batch 16, 16384-sample crops. This checkpoint is a full PyTorch Lightning checkpoint (~epoch 14, step ~277k) including generator,
|
| 39 |
+
discriminators, and optimizer states — usable both for inference (filter the `feature_extractor.`/`backbone.`/`head.` keys) and to
|
| 40 |
+
resume training.
|
| 41 |
+
|
| 42 |
+
## Usage
|
| 43 |
+
|
| 44 |
+
See [examples/streaming_infer_ckpt.py](https://github.com/thewh1teagle/vocos/blob/main/examples/streaming_infer_ckpt.py)
|
| 45 |
+
for loading this checkpoint and streaming audio chunk by chunk.
|
last.ckpt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ccd8bfedd6511cb38278f620e4d441ec62808de65e3106c924c3c5a05fd67088
|
| 3 |
+
size 673556924
|