Dvirile commited on
Commit
372da42
·
verified ·
1 Parent(s): 064a350

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +35 -18
README.md CHANGED
@@ -12,12 +12,15 @@ base_model: ResembleAI/chatterbox
12
 
13
  # chatterbox-turbo-lora
14
 
15
- LoRA-aware ONNX export of the Chatterbox-Turbo language-model subgraph,
16
  quantized to **q4f16** for on-device inference (Android / iOS via ORT).
17
 
18
- This is the base LM only the full Chatterbox-Turbo TTS pipeline still
19
- needs the conditional-encoder, speech-token, and S3-Gen subgraphs from
20
- the upstream Chatterbox repo.
 
 
 
21
 
22
  ## What's new in v2.0.0
23
 
@@ -27,9 +30,7 @@ projection inside the transformer (`attn.c_attn`, `attn.c_proj`,
27
  `mlp.c_fc`, `mlp.c_proj`) in a `LoraConv1D` that exposes two extra
28
  input tensors per layer:
29
 
30
- ```
31
  lora.layers.{N}.{attn_c_attn|attn_c_proj|mlp_c_fc|mlp_c_proj}.{A|B}
32
- ```
33
 
34
  → **24 layers × 4 projections × 2 tensors = 192 extra graph inputs.**
35
 
@@ -41,29 +42,45 @@ synthesizes as-is.
41
 
42
  ## Files
43
 
44
- | File | Size | Purpose |
45
- |---|---|---|
46
- | `language_model_q4f16.onnx` | ~0.8 MB | Graph protobuf (no weights) |
47
- | `language_model_q4f16.onnx_data` | ~222 MB | External weight data (q4f16) |
48
 
49
- Both files must be downloaded together. The `.onnx` file references the
50
- `.onnx_data` companion by relative filename.
 
 
 
 
 
 
 
 
 
 
51
 
52
  ## Quantization details
53
 
54
  - **Format:** q4f16 (4-bit asymmetric weights + fp16 scales)
55
  - **Block size:** 32 (matches Chatterbox's original 1.0.0 layout)
56
  - **Quantizer:** `onnxruntime.quantization.MatMulNBitsQuantizer`
57
- - **Excluded from quantization:** all 192 LoRA-path `MatMul`/`Add` nodes
58
- (kept fp32 so adapter weights can be hot-swapped without retraining
59
- scales)
60
- - The 96 GPT-2 projection layers are first rewritten from `Gemm` →
61
- `MatMul + Add` so they fall inside the quantizer's coverage; without
62
- this step the file would stay at ~1.2 GB.
63
 
64
  Embedding tables (`wte`, `wpe`) remain fp32 because they are read via
65
  `Gather`, not `MatMul`, and `MatMulNBitsQuantizer` does not touch them.
66
 
 
 
 
 
 
 
67
  ## License
68
 
69
  MIT, inherited from the upstream Chatterbox project.
 
12
 
13
  # chatterbox-turbo-lora
14
 
15
+ LoRA-aware ONNX export of the **complete Chatterbox-Turbo TTS pipeline**,
16
  quantized to **q4f16** for on-device inference (Android / iOS via ORT).
17
 
18
+ The `language_model` subgraph is a custom fork that exposes 192 extra
19
+ input ports so a PEFT-trained LoRA adapter can be bound at runtime. The
20
+ other three subgraphs (`speech_encoder`, `embed_tokens`,
21
+ `conditional_decoder`) are byte-identical mirrors of the corresponding
22
+ files from `ResembleAI/chatterbox-turbo-ONNX/onnx/`, hosted here so an
23
+ on-device app has a single-repo source for the full pipeline.
24
 
25
  ## What's new in v2.0.0
26
 
 
30
  `mlp.c_fc`, `mlp.c_proj`) in a `LoraConv1D` that exposes two extra
31
  input tensors per layer:
32
 
 
33
  lora.layers.{N}.{attn_c_attn|attn_c_proj|mlp_c_fc|mlp_c_proj}.{A|B}
 
34
 
35
  → **24 layers × 4 projections × 2 tensors = 192 extra graph inputs.**
36
 
 
42
 
43
  ## Files
44
 
45
+ Four subgraphs, each a `.onnx` protobuf + a `.onnx_data` external weight
46
+ blob. Every `.onnx` file references its `.onnx_data` companion by
47
+ relative filename, so both files of a pair must be downloaded and kept
48
+ next to each other.
49
 
50
+ | File | Size | Purpose | Source |
51
+ |---|---|---|---|
52
+ | `speech_encoder_q4f16.onnx` | ~1.2 MB | Extracts speaker embedding + audio tokens from a voiceprint clip | Mirrored from upstream |
53
+ | `speech_encoder_q4f16.onnx_data` | ~169 MB | External weights | Mirrored from upstream |
54
+ | `embed_tokens_q4f16.onnx` | ~2.5 KB | Token embedding lookup | Mirrored from upstream |
55
+ | `embed_tokens_q4f16.onnx_data` | ~32 MB | External weights | Mirrored from upstream |
56
+ | `language_model_q4f16.onnx` | ~0.8 MB | Autoregressive LM with LoRA input ports (see above) | Custom (v2.0.0) |
57
+ | `language_model_q4f16.onnx_data` | ~222 MB | External weights | Custom (v2.0.0) |
58
+ | `conditional_decoder_q4f16.onnx` | ~2.3 MB | Speech-token → waveform vocoder | Mirrored from upstream |
59
+ | `conditional_decoder_q4f16.onnx_data` | ~155 MB | External weights | Mirrored from upstream |
60
+
61
+ Total: ~581 MB.
62
 
63
  ## Quantization details
64
 
65
  - **Format:** q4f16 (4-bit asymmetric weights + fp16 scales)
66
  - **Block size:** 32 (matches Chatterbox's original 1.0.0 layout)
67
  - **Quantizer:** `onnxruntime.quantization.MatMulNBitsQuantizer`
68
+ - **Excluded from quantization** (language_model only): all 192 LoRA-path
69
+ `MatMul`/`Add` nodes (kept fp32 so adapter weights can be hot-swapped
70
+ without retraining scales)
71
+ - The 96 GPT-2 projection layers in `language_model` are first rewritten
72
+ from `Gemm` → `MatMul + Add` so they fall inside the quantizer's
73
+ coverage; without this step the file would stay at ~1.2 GB.
74
 
75
  Embedding tables (`wte`, `wpe`) remain fp32 because they are read via
76
  `Gather`, not `MatMul`, and `MatMulNBitsQuantizer` does not touch them.
77
 
78
+ The three mirrored subgraphs (`speech_encoder`, `embed_tokens`,
79
+ `conditional_decoder`) use whatever quantization the upstream ResembleAI
80
+ build applied. They are bit-for-bit identical to
81
+ `ResembleAI/chatterbox-turbo-ONNX/onnx/*_q4f16.onnx{,_data}` — you can
82
+ verify with sha256 against the upstream repo.
83
+
84
  ## License
85
 
86
  MIT, inherited from the upstream Chatterbox project.