Instructions to use TuKoResearch/WavCochCausalV8192-vocoder-causal-centered with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use TuKoResearch/WavCochCausalV8192-vocoder-causal-centered with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="TuKoResearch/WavCochCausalV8192-vocoder-causal-centered", trust_remote_code=True)# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("TuKoResearch/WavCochCausalV8192-vocoder-causal-centered", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Restore geometric centered padding (421, 500)
Browse files- README.md +18 -13
- config.json +2 -2
- configuration_wavcoch.py +1 -6
- modeling_wavcoch.py +1 -1
README.md
CHANGED
|
@@ -42,8 +42,9 @@ wavcoch = AutoModel.from_pretrained(
|
|
| 42 |
causal_codes = wavcoch.quantize(waveform_tensor)
|
| 43 |
assert torch.equal(causal_codes, wavcoch.quantize(waveform_tensor, mode="causal"))
|
| 44 |
|
| 45 |
-
# Centered mode preserves the number of frames
|
| 46 |
-
#
|
|
|
|
| 47 |
centered_codes = wavcoch.quantize(waveform_tensor, mode="centered")
|
| 48 |
|
| 49 |
codes = causal_codes
|
|
@@ -62,12 +63,11 @@ audio = wavcoch.decode_audio(codes)
|
|
| 62 |
This repo includes a bundled vocoder and supports `decode_audio(...)` for end-to-end waveform synthesis.
|
| 63 |
|
| 64 |
`mode="causal"` is the default and uses the original 921 samples of left
|
| 65 |
-
padding. `mode="centered"` redistributes the same total padding as
|
| 66 |
-
on the left and
|
| 67 |
-
while
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
encoding uses future waveform context and is therefore not streaming-causal.
|
| 71 |
|
| 72 |
The mode argument is accepted by `forward(...)`, `quantize(...)`, and
|
| 73 |
`wav2coch(...)`. Decoding methods operate on codes and do not take a mode.
|
|
@@ -87,17 +87,22 @@ Validation used float32 with TF32 disabled:
|
|
| 87 |
divisible by the 80-sample hop.
|
| 88 |
- Six synthetic beep configurations produced the same 200 frames for causal
|
| 89 |
and centered modes from one second of audio.
|
| 90 |
-
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
peaks to 0 ms on the 5 ms output grid.
|
| 94 |
- AuriStream7BDeep-40Pred float32 head-1 and head-40 surprisals were bit-exact
|
| 95 |
between the base model's codes and this model's causal codes.
|
| 96 |
|
| 97 |
Centered mode changes the encoded token sequence and uses approximately
|
| 98 |
-
|
| 99 |
on causal WavCoch codes should therefore be evaluated explicitly before
|
| 100 |
centered codes are used for likelihood estimates or generation.
|
| 101 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 102 |
When called with `output_hidden_states=True`, WavCoch exposes a single hidden-state layer:
|
| 103 |
the post-FSQ projected embedding sequence used for direct probing.
|
|
|
|
| 42 |
causal_codes = wavcoch.quantize(waveform_tensor)
|
| 43 |
assert torch.equal(causal_codes, wavcoch.quantize(waveform_tensor, mode="causal"))
|
| 44 |
|
| 45 |
+
# Centered mode preserves the number of frames while geometrically centering
|
| 46 |
+
# the 1001-sample analysis window on token endpoints. It uses 421 left and
|
| 47 |
+
# 500 right zeros.
|
| 48 |
centered_codes = wavcoch.quantize(waveform_tensor, mode="centered")
|
| 49 |
|
| 50 |
codes = causal_codes
|
|
|
|
| 63 |
This repo includes a bundled vocoder and supports `decode_audio(...)` for end-to-end waveform synthesis.
|
| 64 |
|
| 65 |
`mode="causal"` is the default and uses the original 921 samples of left
|
| 66 |
+
padding. `mode="centered"` redistributes the same total padding as 421 samples
|
| 67 |
+
on the left and 500 samples on the right. This keeps frame counts unchanged
|
| 68 |
+
while moving the analysis-window center forward by 500 samples (31.25 ms at
|
| 69 |
+
16 kHz). Centered encoding uses future waveform context and is therefore not
|
| 70 |
+
streaming-causal.
|
|
|
|
| 71 |
|
| 72 |
The mode argument is accepted by `forward(...)`, `quantize(...)`, and
|
| 73 |
`wav2coch(...)`. Decoding methods operate on codes and do not take a mode.
|
|
|
|
| 87 |
divisible by the 80-sample hop.
|
| 88 |
- Six synthetic beep configurations produced the same 200 frames for causal
|
| 89 |
and centered modes from one second of audio.
|
| 90 |
+
- Centered decoded-cochleagram power advanced by 6 frames (30 ms, the nearest
|
| 91 |
+
5 ms token-grid value to 31.25 ms), with shifted-curve Pearson correlations
|
| 92 |
+
from 0.986 to 0.999.
|
|
|
|
| 93 |
- AuriStream7BDeep-40Pred float32 head-1 and head-40 surprisals were bit-exact
|
| 94 |
between the base model's codes and this model's causal codes.
|
| 95 |
|
| 96 |
Centered mode changes the encoded token sequence and uses approximately
|
| 97 |
+
31.25 ms of future waveform context relative to token endpoints. Models trained
|
| 98 |
on causal WavCoch codes should therefore be evaluated explicitly before
|
| 99 |
centered codes are used for likelihood estimates or generation.
|
| 100 |
|
| 101 |
+
An end-to-end impulse diagnostic found that the learned causal decoder can move
|
| 102 |
+
decoded power peaks about 5 ms later than the geometrically centered front end.
|
| 103 |
+
The experimental 341-left/580-right compensation is intentionally not exposed
|
| 104 |
+
as a named mode in this checkpoint; `centered` means geometric endpoint
|
| 105 |
+
centering with 421-left/500-right padding.
|
| 106 |
+
|
| 107 |
When called with `output_hidden_states=True`, WavCoch exposes a single hidden-state layer:
|
| 108 |
the post-FSQ projected embedding sequence used for direct probing.
|
config.json
CHANGED
|
@@ -56,8 +56,8 @@
|
|
| 56 |
"causal",
|
| 57 |
"centered"
|
| 58 |
],
|
| 59 |
-
"centered_left_padding":
|
| 60 |
-
"centered_right_padding":
|
| 61 |
"hop_length": 80,
|
| 62 |
"causal_convs": true,
|
| 63 |
"encoder_layers": 8,
|
|
|
|
| 56 |
"causal",
|
| 57 |
"centered"
|
| 58 |
],
|
| 59 |
+
"centered_left_padding": 421,
|
| 60 |
+
"centered_right_padding": 500,
|
| 61 |
"hop_length": 80,
|
| 62 |
"causal_convs": true,
|
| 63 |
"encoder_layers": 8,
|
configuration_wavcoch.py
CHANGED
|
@@ -48,13 +48,8 @@ class WavCochConfig(PretrainedConfig):
|
|
| 48 |
self.window_size = int(window_size)
|
| 49 |
self.window_padding = int(window_padding)
|
| 50 |
half_window = (self.window_size - 1) // 2
|
| 51 |
-
# Advance one additional hop beyond geometric endpoint centering to
|
| 52 |
-
# compensate the measured causal encoder/decoder response delay.
|
| 53 |
-
default_centered_right_padding = half_window + int(hop_length)
|
| 54 |
self.centered_right_padding = int(
|
| 55 |
-
|
| 56 |
-
if centered_right_padding is None
|
| 57 |
-
else centered_right_padding
|
| 58 |
)
|
| 59 |
expected_left_padding = self.window_padding - self.centered_right_padding
|
| 60 |
self.centered_left_padding = int(
|
|
|
|
| 48 |
self.window_size = int(window_size)
|
| 49 |
self.window_padding = int(window_padding)
|
| 50 |
half_window = (self.window_size - 1) // 2
|
|
|
|
|
|
|
|
|
|
| 51 |
self.centered_right_padding = int(
|
| 52 |
+
half_window if centered_right_padding is None else centered_right_padding
|
|
|
|
|
|
|
| 53 |
)
|
| 54 |
expected_left_padding = self.window_padding - self.centered_right_padding
|
| 55 |
self.centered_left_padding = int(
|
modeling_wavcoch.py
CHANGED
|
@@ -399,7 +399,7 @@ class WavCoch(PreTrainedModel):
|
|
| 399 |
self.hop_length = int(config.hop_length)
|
| 400 |
self.window_padding = int(getattr(config, "window_padding", self.N - self.hop_length))
|
| 401 |
self.centered_right_padding = int(
|
| 402 |
-
getattr(config, "centered_right_padding", (self.N - 1) // 2
|
| 403 |
)
|
| 404 |
self.centered_left_padding = int(
|
| 405 |
getattr(config, "centered_left_padding", self.window_padding - self.centered_right_padding)
|
|
|
|
| 399 |
self.hop_length = int(config.hop_length)
|
| 400 |
self.window_padding = int(getattr(config, "window_padding", self.N - self.hop_length))
|
| 401 |
self.centered_right_padding = int(
|
| 402 |
+
getattr(config, "centered_right_padding", (self.N - 1) // 2)
|
| 403 |
)
|
| 404 |
self.centered_left_padding = int(
|
| 405 |
getattr(config, "centered_left_padding", self.window_padding - self.centered_right_padding)
|