# ZipVoice.AXERA

[ZipVoice]((https://github.com/k2-fsa/ZipVoice)) AXERA 板端 & AXCL 算力卡推理 demo。

## 功能

- 支持中文和英文语音生成。
- 支持语音克隆。
- 支持 ZipVoice、ZipVoice Distill
- 支持 C++ 和 Python 推理
- 支持 AXERA 和 AXCL platform

## 模型说明

ZipVoice Distill 是 ZipVoice 的蒸馏版本，主要优势是在较小性能损失下提升推理速度。

## 性能对比

### Distill 模型 (num_step=4)

| 场景 | 音频时长 | Python AX650 (耗时/RTF) | C++ AX650 (耗时/RTF) | C++ AXCL (耗时/RTF) |
|------|---------|------------------------|---------------------|---------------------|
| 中文句子 | 6.41s | 1.992s / 0.311 | 1.057s / 0.165 | 0.918s / **0.143** |
| 中文段落 | 44.97s | 13.457s / 0.301 | 7.357s / 0.164 | 6.443s / **0.143** |
| 英文句子 | 6.41s | 2.045s / 0.319 | 1.067s / 0.166 | 0.919s / **0.143** |
| 英文段落 | 59.16s | 19.715s / 0.305 | 10.529s / 0.178 | 9.207s / **0.156** |

### 普通模型 (num_step=10)

| 场景 | 音频时长 | Python AX650 (耗时/RTF) | C++ AX650 (耗时/RTF) | C++ AXCL (耗时/RTF) |
|------|---------|------------------------|---------------------|---------------------|
| 中文句子 | 6.41s | 5.781s / 0.902 | 5.709s / 0.891 | 5.452s / **0.850** |
| 中文段落 | 44.97s | 40.292s / 0.901 | 39.714s / 0.883 | 38.138s / **0.848** |
| 英文句子 | 6.41s | 5.711s / 0.891 | 5.693s / 0.888 | 5.447s / **0.850** |
| 英文段落 | 59.16s | 62.161s / 0.960 | 56.759s / 0.957 | 54.451s / **0.920** |


## 模型转换

模型量化参考:

- [Pulsar2 Docs — How to Convert ONNX to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/pulsar2/introduction.html)
- [export / quantization Project](https://github.com/AXERA-TECH/ZipVoice.AXERA)

## 支持平台

- AX650
  - AX650 demo 板
  - [M4N-Dock（爱芯派Pro）](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html)
  - [M.2 Accelerator Card](https://docs.m5stack.com/zh_CN/ai_hardware/LLM-8850_Card)


## 目录结构

```text
ZipVoice.AXERA_AXCL/
├── assets/
│   ├── moss_prompts/
│   └── paragraphs/
├── models/
│   ├── zipvoice_ax650/
│   ├── zipvoice_distill_ax650/
│   └── zipvoice_distill_ax630C/
├── resources/
│   ├── vocos-mel-24khz/
│   └── zipvoice_hf/
├── cpp/                         # C++ 源码 (AXERA & AXCL)
├── scripts/
├── infer_zipvoice_axera.py      # Python 推理入口
├── run_ax650.sh                 # AXERA 板端一键运行
├── run_axcl.sh                  # AXCL 算力卡一键运行
├── requirements.txt
└── README.md
```

## Python

### 环境

安装 pyaxengine：
[pyaxengine Releases](https://github.com/AXERA-TECH/pyaxengine/releases/latest) 下载对应版本安装：
```bash
pip3 install axengine-x.x.x-py3-none-any.whl
```

安装依赖：

```bash
conda create -n ZipVoice python=3.10
conda activate ZipVoice
pip3 install -r requirements.txt
```

### 推理命令

进入目录：

```bash
cd ZipVoice.AXERA
```

#### AX650 ZipVoice

中文句子：

```bash
python3 infer_zipvoice_axera.py \
  --model-name zipvoice_ax650 \
  --text "今天午后天气很好，我打开窗户，听见远处有人聊天，水杯也轻轻晃了一下。" \
  --prompt-text "不管怎么样我和汤姆还是要感谢贝尔卡金的援手" \
  --prompt-wav assets/moss_prompts/zh_1_4p5s.wav \
  --output-wav outputs/zh_sentence_ax650.wav \
  --seed 42
```

推理结果：
```text
推理耗时: 5.781s
生成语音时长: 6.411s
RTF: 0.9018
```

音频：[outputs/zh_sentence_ax650.wav](outputs/zh_sentence_ax650.wav)  
提示音：[assets/moss_prompts/zh_1_4p5s.wav](assets/moss_prompts/zh_1_4p5s.wav)

英文句子：

```bash
python3 infer_zipvoice_axera.py \
  --model-name zipvoice_ax650 \
  --text "This morning, a small train left the station, carrying sleepy passengers toward a bright coastal town." \
  --prompt-text "This is almost twice the current industry production level per train." \
  --prompt-wav assets/moss_prompts/en_4_4p5s.wav \
  --output-wav outputs/en_sentence_ax650.wav \
  --seed 42
```

推理结果：
```text
推理耗时: 5.711s
生成语音时长: 6.411s
RTF: 0.8909
```

音频：[outputs/en_sentence_ax650.wav](outputs/en_sentence_ax650.wav)  
提示音：[assets/moss_prompts/en_4_4p5s.wav](assets/moss_prompts/en_4_4p5s.wav)

中文段落：

```bash
python3 infer_zipvoice_axera.py \
  --model-name zipvoice_ax650 \
  --text-file assets/paragraphs/zh_ginkgo.txt \
  --prompt-text "不管怎么样我和汤姆还是要感谢贝尔卡金的援手" \
  --prompt-wav assets/moss_prompts/zh_1_4p5s.wav \
  --output-wav outputs/zh_long_paragraph_ax650.wav \
  --seed 42
```

推理结果：
```text
推理耗时: 40.292s
生成语音时长: 44.744s
RTF: 0.9005
```

音频：[outputs/zh_long_paragraph_ax650.wav](outputs/zh_long_paragraph_ax650.wav)  
提示音：[assets/moss_prompts/zh_1_4p5s.wav](assets/moss_prompts/zh_1_4p5s.wav)

英文段落：

```bash
python3 infer_zipvoice_axera.py \
  --model-name zipvoice_ax650 \
  --text-file assets/paragraphs/en_scavenger.txt \
  --prompt-text "This is almost twice the current industry production level per train." \
  --prompt-wav assets/moss_prompts/en_4_4p5s.wav \
  --output-wav outputs/en_long_paragraph_ax650.wav \
  --seed 42
```

推理结果：
```text
推理耗时: 62.161s
生成语音时长: 64.749s
RTF: 0.9600
```

音频：[outputs/en_long_paragraph_ax650.wav](outputs/en_long_paragraph_ax650.wav)  
提示音：[assets/moss_prompts/en_4_4p5s.wav](assets/moss_prompts/en_4_4p5s.wav)

#### AX650 ZipVoice Distill

中文句子：

```bash
python3 infer_zipvoice_axera.py \
  --model-name zipvoice_distill_ax650 \
  --text "今天午后天气很好，我打开窗户，听见远处有人聊天，水杯也轻轻晃了一下。" \
  --prompt-text "不管怎么样我和汤姆还是要感谢贝尔卡金的援手" \
  --prompt-wav assets/moss_prompts/zh_1_4p5s.wav \
  --output-wav outputs/zh_sentence_distill_ax650.wav \
  --seed 42
```

推理结果：
```text
推理耗时: 1.992s
生成语音时长: 6.411s
RTF: 0.3107
```

音频：[outputs/zh_sentence_distill_ax650.wav](outputs/zh_sentence_distill_ax650.wav)  
提示音：[assets/moss_prompts/zh_1_4p5s.wav](assets/moss_prompts/zh_1_4p5s.wav)

英文句子：

```bash
python3 infer_zipvoice_axera.py \
  --model-name zipvoice_distill_ax650 \
  --text "This morning, a small train left the station, carrying sleepy passengers toward a bright coastal town." \
  --prompt-text "This is almost twice the current industry production level per train." \
  --prompt-wav assets/moss_prompts/en_4_4p5s.wav \
  --output-wav outputs/en_sentence_distill_ax650.wav \
  --seed 42
```

推理结果：
```text
推理耗时: 2.045s
生成语音时长: 6.411s
RTF: 0.3189
```

音频：[outputs/en_sentence_distill_ax650.wav](outputs/en_sentence_distill_ax650.wav)  
提示音：[assets/moss_prompts/en_4_4p5s.wav](assets/moss_prompts/en_4_4p5s.wav)

中文段落：

```bash
python3 infer_zipvoice_axera.py \
  --model-name zipvoice_distill_ax650 \
  --text-file assets/paragraphs/zh_ginkgo.txt \
  --prompt-text "不管怎么样我和汤姆还是要感谢贝尔卡金的援手" \
  --prompt-wav assets/moss_prompts/zh_1_4p5s.wav \
  --output-wav outputs/zh_long_paragraph_distill_ax650.wav \
  --seed 42
```

推理结果：
```text
推理耗时: 13.457s
生成语音时长: 44.744s
RTF: 0.3008
```

音频：[outputs/zh_long_paragraph_distill_ax650.wav](outputs/zh_long_paragraph_distill_ax650.wav)  
提示音：[assets/moss_prompts/zh_1_4p5s.wav](assets/moss_prompts/zh_1_4p5s.wav)

英文段落：

```bash
python3 infer_zipvoice_axera.py \
  --model-name zipvoice_distill_ax650 \
  --text-file assets/paragraphs/en_scavenger.txt \
  --prompt-text "This is almost twice the current industry production level per train." \
  --prompt-wav assets/moss_prompts/en_4_4p5s.wav \
  --output-wav outputs/en_long_paragraph_distill_ax650.wav \
  --seed 42
```

推理结果：
```text
推理耗时: 19.715s
生成语音时长: 64.749s
RTF: 0.3045
```

音频：[outputs/en_long_paragraph_distill_ax650.wav](outputs/en_long_paragraph_distill_ax650.wav)  
提示音：[assets/moss_prompts/en_4_4p5s.wav](assets/moss_prompts/en_4_4p5s.wav)


### 参数说明

- `--model-name`：选择模型目录。可选 `zipvoice_ax650`、`zipvoice_distill_ax650`、`zipvoice_distill_ax630C`。
- `--prompt-wav`：参考音频，用于控制音色，建议 3-5s。
- `--prompt-text`：参考音频对应文本，必须尽量和 `prompt-wav` 内容一致。
- `--num-step`：采样步数。默认从模型目录的 `runtime_config.json` 读取。
- `--max-feat-len`：decoder 固定 feature 长度，当前模型均为 1024。

## C++

C++ 推理实现。详见 [cpp/README.md](cpp/README.md)。


## 参考

- [ZipVoice](https://github.com/k2-fsa/ZipVoice)
- [Pulsar2 Docs](https://pulsar2-docs.readthedocs.io/en/latest/pulsar2/introduction.html)
- [模型量化工程](https://github.com/AXERA-TECH/ZipVoice.AXERA)