Instructions to use Dvirile/chatterbox-turbo-lora with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Chatterbox
How to use Dvirile/chatterbox-turbo-lora with Chatterbox:
# pip install chatterbox-tts import torchaudio as ta from chatterbox.tts import ChatterboxTTS model = ChatterboxTTS.from_pretrained(device="cuda") text = "Ezreal and Jinx teamed up with Ahri, Yasuo, and Teemo to take down the enemy's Nexus in an epic late-game pentakill." wav = model.generate(text) ta.save("test-1.wav", wav, model.sr) # If you want to synthesize with a different voice, specify the audio prompt AUDIO_PROMPT_PATH="YOUR_FILE.wav" wav = model.generate(text, audio_prompt_path=AUDIO_PROMPT_PATH) ta.save("test-2.wav", wav, model.sr) - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -12,12 +12,15 @@ base_model: ResembleAI/chatterbox
|
|
| 12 |
|
| 13 |
# chatterbox-turbo-lora
|
| 14 |
|
| 15 |
-
LoRA-aware ONNX export of the Chatterbox-Turbo
|
| 16 |
quantized to **q4f16** for on-device inference (Android / iOS via ORT).
|
| 17 |
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
|
|
|
|
|
|
|
|
|
| 21 |
|
| 22 |
## What's new in v2.0.0
|
| 23 |
|
|
@@ -27,9 +30,7 @@ projection inside the transformer (`attn.c_attn`, `attn.c_proj`,
|
|
| 27 |
`mlp.c_fc`, `mlp.c_proj`) in a `LoraConv1D` that exposes two extra
|
| 28 |
input tensors per layer:
|
| 29 |
|
| 30 |
-
```
|
| 31 |
lora.layers.{N}.{attn_c_attn|attn_c_proj|mlp_c_fc|mlp_c_proj}.{A|B}
|
| 32 |
-
```
|
| 33 |
|
| 34 |
→ **24 layers × 4 projections × 2 tensors = 192 extra graph inputs.**
|
| 35 |
|
|
@@ -41,29 +42,45 @@ synthesizes as-is.
|
|
| 41 |
|
| 42 |
## Files
|
| 43 |
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
|
| 49 |
-
|
| 50 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 51 |
|
| 52 |
## Quantization details
|
| 53 |
|
| 54 |
- **Format:** q4f16 (4-bit asymmetric weights + fp16 scales)
|
| 55 |
- **Block size:** 32 (matches Chatterbox's original 1.0.0 layout)
|
| 56 |
- **Quantizer:** `onnxruntime.quantization.MatMulNBitsQuantizer`
|
| 57 |
-
- **Excluded from quantization
|
| 58 |
-
(kept fp32 so adapter weights can be hot-swapped
|
| 59 |
-
scales)
|
| 60 |
-
- The 96 GPT-2 projection layers are first rewritten
|
| 61 |
-
`MatMul + Add` so they fall inside the quantizer's
|
| 62 |
-
this step the file would stay at ~1.2 GB.
|
| 63 |
|
| 64 |
Embedding tables (`wte`, `wpe`) remain fp32 because they are read via
|
| 65 |
`Gather`, not `MatMul`, and `MatMulNBitsQuantizer` does not touch them.
|
| 66 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 67 |
## License
|
| 68 |
|
| 69 |
MIT, inherited from the upstream Chatterbox project.
|
|
|
|
| 12 |
|
| 13 |
# chatterbox-turbo-lora
|
| 14 |
|
| 15 |
+
LoRA-aware ONNX export of the **complete Chatterbox-Turbo TTS pipeline**,
|
| 16 |
quantized to **q4f16** for on-device inference (Android / iOS via ORT).
|
| 17 |
|
| 18 |
+
The `language_model` subgraph is a custom fork that exposes 192 extra
|
| 19 |
+
input ports so a PEFT-trained LoRA adapter can be bound at runtime. The
|
| 20 |
+
other three subgraphs (`speech_encoder`, `embed_tokens`,
|
| 21 |
+
`conditional_decoder`) are byte-identical mirrors of the corresponding
|
| 22 |
+
files from `ResembleAI/chatterbox-turbo-ONNX/onnx/`, hosted here so an
|
| 23 |
+
on-device app has a single-repo source for the full pipeline.
|
| 24 |
|
| 25 |
## What's new in v2.0.0
|
| 26 |
|
|
|
|
| 30 |
`mlp.c_fc`, `mlp.c_proj`) in a `LoraConv1D` that exposes two extra
|
| 31 |
input tensors per layer:
|
| 32 |
|
|
|
|
| 33 |
lora.layers.{N}.{attn_c_attn|attn_c_proj|mlp_c_fc|mlp_c_proj}.{A|B}
|
|
|
|
| 34 |
|
| 35 |
→ **24 layers × 4 projections × 2 tensors = 192 extra graph inputs.**
|
| 36 |
|
|
|
|
| 42 |
|
| 43 |
## Files
|
| 44 |
|
| 45 |
+
Four subgraphs, each a `.onnx` protobuf + a `.onnx_data` external weight
|
| 46 |
+
blob. Every `.onnx` file references its `.onnx_data` companion by
|
| 47 |
+
relative filename, so both files of a pair must be downloaded and kept
|
| 48 |
+
next to each other.
|
| 49 |
|
| 50 |
+
| File | Size | Purpose | Source |
|
| 51 |
+
|---|---|---|---|
|
| 52 |
+
| `speech_encoder_q4f16.onnx` | ~1.2 MB | Extracts speaker embedding + audio tokens from a voiceprint clip | Mirrored from upstream |
|
| 53 |
+
| `speech_encoder_q4f16.onnx_data` | ~169 MB | External weights | Mirrored from upstream |
|
| 54 |
+
| `embed_tokens_q4f16.onnx` | ~2.5 KB | Token embedding lookup | Mirrored from upstream |
|
| 55 |
+
| `embed_tokens_q4f16.onnx_data` | ~32 MB | External weights | Mirrored from upstream |
|
| 56 |
+
| `language_model_q4f16.onnx` | ~0.8 MB | Autoregressive LM with LoRA input ports (see above) | Custom (v2.0.0) |
|
| 57 |
+
| `language_model_q4f16.onnx_data` | ~222 MB | External weights | Custom (v2.0.0) |
|
| 58 |
+
| `conditional_decoder_q4f16.onnx` | ~2.3 MB | Speech-token → waveform vocoder | Mirrored from upstream |
|
| 59 |
+
| `conditional_decoder_q4f16.onnx_data` | ~155 MB | External weights | Mirrored from upstream |
|
| 60 |
+
|
| 61 |
+
Total: ~581 MB.
|
| 62 |
|
| 63 |
## Quantization details
|
| 64 |
|
| 65 |
- **Format:** q4f16 (4-bit asymmetric weights + fp16 scales)
|
| 66 |
- **Block size:** 32 (matches Chatterbox's original 1.0.0 layout)
|
| 67 |
- **Quantizer:** `onnxruntime.quantization.MatMulNBitsQuantizer`
|
| 68 |
+
- **Excluded from quantization** (language_model only): all 192 LoRA-path
|
| 69 |
+
`MatMul`/`Add` nodes (kept fp32 so adapter weights can be hot-swapped
|
| 70 |
+
without retraining scales)
|
| 71 |
+
- The 96 GPT-2 projection layers in `language_model` are first rewritten
|
| 72 |
+
from `Gemm` → `MatMul + Add` so they fall inside the quantizer's
|
| 73 |
+
coverage; without this step the file would stay at ~1.2 GB.
|
| 74 |
|
| 75 |
Embedding tables (`wte`, `wpe`) remain fp32 because they are read via
|
| 76 |
`Gather`, not `MatMul`, and `MatMulNBitsQuantizer` does not touch them.
|
| 77 |
|
| 78 |
+
The three mirrored subgraphs (`speech_encoder`, `embed_tokens`,
|
| 79 |
+
`conditional_decoder`) use whatever quantization the upstream ResembleAI
|
| 80 |
+
build applied. They are bit-for-bit identical to
|
| 81 |
+
`ResembleAI/chatterbox-turbo-ONNX/onnx/*_q4f16.onnx{,_data}` — you can
|
| 82 |
+
verify with sha256 against the upstream repo.
|
| 83 |
+
|
| 84 |
## License
|
| 85 |
|
| 86 |
MIT, inherited from the upstream Chatterbox project.
|