tamarher commited on
Commit
c993ab9
·
verified ·
1 Parent(s): 7c53c40

Add model card

Browse files
Files changed (1) hide show
  1. README.md +104 -0
README.md ADDED
@@ -0,0 +1,104 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: mlx
3
+ pipeline_tag: audio-to-audio
4
+ language:
5
+ - en
6
+ license: other
7
+ license_name: nvidia-source-code-license-nc
8
+ license_link: https://huggingface.co/nvidia/RE-USE
9
+ base_model:
10
+ - nvidia/RE-USE
11
+ tags:
12
+ - mlx
13
+ - semamba
14
+ - speech-enhancement
15
+ - denoising
16
+ - mamba
17
+ - ssm
18
+ - dramabox
19
+ - apple-silicon
20
+ - non-commercial
21
+ ---
22
+
23
+ # RE-USE SEMamba Speech Enhancement (MLX)
24
+
25
+ [![GitHub](https://img.shields.io/badge/GitHub-mlx--speech-181717?logo=github&logoColor=white)](https://github.com/appautomaton/mlx-speech)
26
+ [![App Automaton](https://img.shields.io/badge/App%20Automaton-project-1f6feb)](https://appautomaton.github.io)
27
+ [![DramaBox TTS](https://img.shields.io/badge/%F0%9F%A4%97%20model-DramaBox%20TTS-yellow)](https://huggingface.co/appautomaton/dramabox-tts-3.3b-bf16-mlx)
28
+
29
+ Pure-MLX conversion of [NVIDIA RE-USE](https://huggingface.co/nvidia/RE-USE), a
30
+ ~9.6M-parameter SEMamba universal speech-enhancement model. In
31
+ [mlx-speech](https://github.com/appautomaton/mlx-speech) it cleans a voice
32
+ reference before VAE conditioning when [DramaBox TTS](https://huggingface.co/appautomaton/dramabox-tts-3.3b-bf16-mlx)
33
+ runs with `denoise_ref=True`, giving the cloning model a clean speaker anchor.
34
+
35
+ > **Non-commercial weights.** These weights derive from NVIDIA RE-USE, licensed
36
+ > under the NVIDIA Source Code License (non-commercial). See the License section.
37
+
38
+ ## Model Details
39
+
40
+ - Developed by: [App Automaton](https://appautomaton.github.io)
41
+ - Upstream model: [`nvidia/RE-USE`](https://huggingface.co/nvidia/RE-USE) (SEMamba, bidirectional Mamba over STFT magnitude + phase)
42
+ - Role: input-side voice-reference denoiser for DramaBox `denoise_ref=True`. Optional, off by default.
43
+ - Conversion: format-only port of the fp32 weights to MLX `.safetensors` (1416 keys, ~9.6M params). No quantization, no architecture change.
44
+ - Runtime: pure MLX on Apple Silicon. The selective scan mirrors the `mamba_ssm` `selective_scan_ref` reference math, so no CUDA kernels (`mamba-ssm` / `causal-conv1d`) are required.
45
+ - Parity: the MLX port matches the torch reference at amplitude-weighted complex correlation 0.9998 (model) and 0.9997 (end-to-end waveform on real speech).
46
+
47
+ ## Contents
48
+
49
+ | File | Component | Format | Size |
50
+ | --- | --- | --- | --- |
51
+ | `model.safetensors` | SEMamba enhancer | fp32 | ~38 MB |
52
+ | `config.json` | Model + STFT config | JSON | n/a |
53
+
54
+ ## How to Get Started
55
+
56
+ Used automatically by DramaBox when you opt in:
57
+
58
+ ```python
59
+ import mlx_speech
60
+
61
+ tts = mlx_speech.tts.load("dramabox")
62
+ result = tts.generate(
63
+ "Voice cloning from a noisy reference.",
64
+ reference_audio="noisy_speaker.wav",
65
+ denoise_ref=True, # cleans the reference with this model first
66
+ )
67
+ ```
68
+
69
+ `tts.load("dramabox")` resolves these weights automatically. To run the enhancer
70
+ directly:
71
+
72
+ ```bash
73
+ hf download appautomaton/re-use-semamba-mlx --local-dir models/reuse/mlx
74
+ ```
75
+
76
+ ```python
77
+ from pathlib import Path
78
+ from mlx_speech.generation.reuse import REUSEEnhancer
79
+
80
+ enhancer = REUSEEnhancer.from_dir(Path("models/reuse/mlx"))
81
+ clean = enhancer.enhance(noisy_waveform, in_sr=16000) # mono in, mono out
82
+ ```
83
+
84
+ ## Intended Use
85
+
86
+ Denoising a short voice-reference clip before voice cloning, so the model
87
+ conditions on a clean speaker/style anchor rather than the recording's noise.
88
+ The enhancer runs on the reference input, never on generated output, so the
89
+ TTS model's paralinguistic events (breaths, laughs) are preserved.
90
+
91
+ ## Links
92
+
93
+ - Source code: [`appautomaton/mlx-speech`](https://github.com/appautomaton/mlx-speech)
94
+ - Paired model: [`appautomaton/dramabox-tts-3.3b-bf16-mlx`](https://huggingface.co/appautomaton/dramabox-tts-3.3b-bf16-mlx)
95
+ - More from App Automaton: [GitHub](https://github.com/appautomaton) · [Hugging Face](https://huggingface.co/appautomaton)
96
+
97
+ ## License
98
+
99
+ NVIDIA Source Code License (non-commercial). These weights are a format
100
+ conversion of [`nvidia/RE-USE`](https://huggingface.co/nvidia/RE-USE) and remain
101
+ governed by NVIDIA's license terms; by downloading or using them you agree to
102
+ those terms. They may not be used commercially. Set `denoise_ref=False` (the
103
+ default) to run DramaBox voice cloning without this model. The mlx-speech
104
+ runtime code is MIT.