| --- |
| license: apache-2.0 |
| language: |
| - zh |
| - en |
| library_name: rapidspeech |
| tags: |
| - automatic-speech-recognition |
| - streaming |
| - zipformer2 |
| - transducer |
| - ggml |
| - gguf |
| - code-switching |
| base_model: |
| - GilgameshWind/X-ASR-zh-en |
| pipeline_tag: automatic-speech-recognition |
| --- |
| |
| # X-ASR zh-en streaming zipformer2 transducer — GGUF |
|
|
| GGUF conversions of the [X-ASR zh-en](https://huggingface.co/GilgameshWind/X-ASR-zh-en) |
| streaming **zipformer2 transducer** (k2-fsa / sherpa-onnx export), for use with |
| [RapidSpeech.cpp](https://github.com/vieenrose/RapidSpeech.cpp) (ggml backend, |
| CPU + CUDA). Mandarin–English code-switching ASR with punctuation. |
|
|
| Converted with `tools/convert_xasr_to_gguf.py`. The encoder/decoder/joiner are |
| fused into a single GGUF per chunk variant; streaming uses per-layer recurrent |
| caches mirroring the ONNX state contract. |
|
|
| ## Variants |
|
|
| All four chunk variants share the **same architecture** (6 stacks / 19 layers, |
| dims 192·256·512·768·512·256, vocab 5000); they differ only in the streaming |
| chunk size (latency vs. accuracy trade-off). |
|
|
| | Folder | Chunk shift | Encoder T | Latency | f16 | Q4_K | |
| |-----------|-------------|-----------|---------|--------|--------| |
| | `160ms/` | 16 frames | 29 | lowest | ~292 MB| ~85 MB | |
| | `480ms/` | 48 frames | 61 | low | ~292 MB| ~85 MB | |
| | `960ms/` | 96 frames | 109 | medium | ~292 MB| ~85 MB | |
| | `1920ms/` | 192 frames | 205 | highest accuracy | ~292 MB| ~86 MB | |
| |
| Each folder contains `*-f16.gguf`, `*-q8_0.gguf`, `*-q4_k.gguf`, `*-q3_k.gguf` |
| (imatrix-calibrated), the `imatrix-*.dat` calibration file, and `tokens.txt`. |
| Convolution kernels are always kept at f16 (quantizing them hurts accuracy). |
| |
| ### Which weight format to use |
| |
| Measured on the 960 ms variant (RapidSpeech CPU); accuracy is token edit-distance |
| vs the f16 reference on a zh-en code-switch clip (0 = token-exact): |
| |
| | Format | Size | Accuracy (edit-dist) | Notes | |
| |--------|-----:|:--------------------:|-------| |
| | **q8_0** | 158 MB | **0 — lossless** (all test clips) | best quality, ~1.2× faster than f16 on CPU | |
| | q4_k | 86 MB | 3 | near-lossless (minor casing) | |
| | **q3_k (imatrix)** | **69 MB** | **0 on calib clip, ~1–2 tok on held-out** | **smaller *and* more accurate than q4_k** | |
| | q3_k (no imatrix) | 69 MB | 4 | "Monday"→"MD"; use the imatrix build instead | |
| | q2_k / iq2 (imatrix) | 51–57 MB | 3–6 | degraded even with imatrix — q3_k is the floor | |
| |
| The `q3_k.gguf` files here are **imatrix-calibrated** (activation-aware, AWQ): |
| an importance matrix collected over calibration audio protects the most important |
| weight channels, which recovers the accuracy 3-bit quantization normally loses. |
| Generate your own with `xasr-dev-test imatrix` + `rs-quantize --imatrix`. |
|
|
| **Recommendation: `q8_0` for lossless quality, or `q3_k` (imatrix) for the |
| smallest near-lossless footprint (69 MB).** Quantizing the matmul weights also |
| *speeds up* ggml CPU inference (less memory traffic + tuned vec-dot kernels). |
| Below 3-bit, accuracy degrades even with an imatrix. |
|
|
| ## Parity |
|
|
| RapidSpeech.cpp (CPU, f16) is **token-exact with sherpa-onnx (onnxruntime CPU, |
| fp32)** on the reference audio for all four variants. Q4_K matches to within |
| occasional capitalization. |
| |
| Example (10 s zh-en code-switching clip): |
| > 昨天是 Monday，today is 礼拜二，the day after tomorrow 是星期三 |
| |
| ## Benchmark (streaming, steady-state ms/chunk, warm-up excluded) |
| |
| Measured on an NVIDIA GB10 host (the original Jetson Nano gen1 target was |
| unavailable). RapidSpeech CUDA uses the FP32 non-tensor path (emulating the |
| Nano's tensor-core-less sm_53). Numbers are relative, not Nano wall-clock. |
|
|
| | Variant | sherpa-onnx CPU | RapidSpeech CPU | RapidSpeech CUDA | |
| |---------|----------------:|----------------:|-----------------:| |
| | 160 ms | 16.0 | 27.7 | 26.9 | |
| | 480 ms | 23.1 | 48.0 | 38.4 | |
| | 960 ms | 31.1 | 83.8 | 53.3 | |
| | 1920 ms | 40.9 | 169.7 | 83.0 | |
|
|
| All configurations run faster than real time. CUDA's speedup over CPU grows with |
| chunk size (1.0× → 2.0×) as larger GEMMs amortize per-chunk kernel-launch cost. |
|
|
| ## Usage |
|
|
| ```bash |
| # RapidSpeech.cpp WebSocket streaming server |
| rs-xasr-ws-server -m 960ms/x-asr-zh-en-960ms-f16.gguf --port 6006 |
| ``` |
|
|
| See [RapidSpeech.cpp](https://github.com/vieenrose/RapidSpeech.cpp) for build |
| instructions (incl. the CUDA-10.2 / sm_53 Jetson Nano path). |
| |
| ## License |
| |
| Apache-2.0, following the upstream X-ASR model. |
| |
