---
license: apache-2.0
language:
- zh
- en
library_name: rapidspeech
tags:
- automatic-speech-recognition
- streaming
- zipformer2
- transducer
- ggml
- gguf
- code-switching
base_model:
- GilgameshWind/X-ASR-zh-en
pipeline_tag: automatic-speech-recognition
---

# X-ASR zh-en streaming zipformer2 transducer — GGUF

GGUF conversions of the [X-ASR zh-en](https://huggingface.co/GilgameshWind/X-ASR-zh-en)
streaming **zipformer2 transducer** (k2-fsa / sherpa-onnx export), for use with
[RapidSpeech.cpp](https://github.com/vieenrose/RapidSpeech.cpp) (ggml backend,
CPU + CUDA). Mandarin–English code-switching ASR with punctuation.

Converted with `tools/convert_xasr_to_gguf.py`. The encoder/decoder/joiner are
fused into a single GGUF per chunk variant; streaming uses per-layer recurrent
caches mirroring the ONNX state contract.

## Variants

All four chunk variants share the **same architecture** (6 stacks / 19 layers,
dims 192·256·512·768·512·256, vocab 5000); they differ only in the streaming
chunk size (latency vs. accuracy trade-off).

| Folder    | Chunk shift | Encoder T | Latency | f16    | Q4_K   |
|-----------|-------------|-----------|---------|--------|--------|
| `160ms/`  | 16 frames   | 29        | lowest  | ~292 MB| ~85 MB |
| `480ms/`  | 48 frames   | 61        | low     | ~292 MB| ~85 MB |
| `960ms/`  | 96 frames   | 109       | medium  | ~292 MB| ~85 MB |
| `1920ms/` | 192 frames  | 205       | highest accuracy | ~292 MB| ~86 MB |

Each folder contains `*-f16.gguf`, `*-q8_0.gguf`, `*-q4_k.gguf`, `*-q3_k.gguf`
(imatrix-calibrated), the `imatrix-*.dat` calibration file, and `tokens.txt`.
The **960 ms** folder additionally ships `*-iq4_xs.gguf` (a lossless 4-bit IQ
build — the 960 ms variant was used for the full quant sweep below).
Convolution kernels are always kept at f16 (quantizing them hurts accuracy).

### Which weight format to use

Per-weight accuracy, measured on the **960 ms** variant. Accuracy is the
**token edit-distance vs the f16 reference** on a zh-en code-switch clip
(`0` = token-exact). Sizes are the actual GGUF bytes.

| Format | Size | Edit-dist | Published | Notes |
|--------|-----:|:---------:|:---------:|-------|
| **f16** | 307 MB | 0 (ref) | ✅ | reference |
| **q8_0** | 165 MB | **0 — lossless** | ✅ | best quality; ~1.2× faster than f16 on CPU |
| **iq4_xs** | 87 MB | **0 — lossless** | ✅ | lossless 4-bit IQ (960 ms only) |
| q4_k | 90 MB | 3 | ✅ | near-lossless (minor casing: `Monday`→`monday`) |
| **q3_k (imatrix)** | **72 MB** | **0 — lossless** | ✅ | **smallest lossless** build |

The `q3_k.gguf` files here are **imatrix-calibrated** (activation-aware, AWQ):
an importance matrix collected over calibration audio protects the most important
weight channels, recovering the accuracy 3-bit quantization normally loses
(without it, q3_k scores edit-dist 4 — `Monday`→`MD`). Generate your own with
`xasr-dev-test imatrix` + `rs-quantize --imatrix`.

**Recommendation: `q8_0` for lossless quality, or `q3_k` (imatrix) for the
smallest lossless footprint (72 MB).** Quantizing the matmul weights also
*speeds up* ggml CPU inference (less memory traffic + tuned vec-dot kernels).

#### Sub-3-bit was evaluated but is *not* published

A full sweep below 3-bit was run on the 960 ms variant and **deliberately
excluded** — none are useful:

| Format | Size | Edit-dist | Why excluded |
|--------|-----:|:---------:|--------------|
| q2_k (imatrix) | 59 MB | 3 | degraded — below the 3-bit floor |
| iq2_s | 58 MB | 3–4 | degraded |
| iq3_s | 86 MB | 4 | dominated (bigger than q3_k-im *and* worse) |
| iq2_xxs (imatrix) | 53 MB | 6 | degraded |
| iq1_s, iq2_xxs (no imatrix) | 90 MB | 3 | **fake** — fell back to q4_k size, not real low-bit |
| iq1_m | 45 MB | 25–58 | **broken** — garbage output |

**3-bit + imatrix is the accuracy floor.** Below it, accuracy degrades (edit-dist
3–6) and 1-bit collapses entirely.

## Parity

RapidSpeech.cpp (CPU, f16) is **token-exact with sherpa-onnx (onnxruntime CPU,
fp32)** on the reference audio for all four variants. Q4_K matches to within
occasional capitalization.

Example (10 s zh-en code-switching clip):
> 昨天是 Monday，today is 礼拜二，the day after tomorrow 是星期三

## Benchmark (streaming, steady-state ms/chunk, warm-up excluded)

Measured on an NVIDIA GB10 host (the original Jetson Nano gen1 target was
unavailable). RapidSpeech CUDA uses the FP32 non-tensor path (emulating the
Nano's tensor-core-less sm_53). Numbers are relative, not Nano wall-clock.

| Variant | sherpa-onnx CPU | RapidSpeech CPU | RapidSpeech CUDA |
|---------|----------------:|----------------:|-----------------:|
| 160 ms  | 16.0 | 27.7 | 26.9 |
| 480 ms  | 23.1 | 48.0 | 38.4 |
| 960 ms  | 31.1 | 83.8 | 53.3 |
| 1920 ms | 40.9 | 169.7 | 83.0 |

All configurations run faster than real time. CUDA's speedup over CPU grows with
chunk size (1.0× → 2.0×) as larger GEMMs amortize per-chunk kernel-launch cost.

### On-device Jetson Nano gen1 (sm_53) — measured

Measured on a **real Jetson Nano gen1** (Tegra X1 / GM20B, sm_53, L4T R32.5.1,
CUDA 10.2, MAXN, clocks unpinned → ~10% run-to-run noise), 960 ms variant,
**encoder ms/chunk** (the fair cross-engine metric), 4 CPU threads. Full data
and the CPU-thread / CUDA-core sweeps are in
[BENCHMARKS.md](https://github.com/vieenrose/x-asr-rapidspeech/blob/main/BENCHMARKS.md).

| Engine / weights | CPU (4 thr) | CUDA (best) | Correct? |
|------------------|------------:|------------:|:--------:|
| sherpa-onnx fp32 | 396.9 | — *not runnable* | ✓ |
| **sherpa-onnx int8** | **329.8** | — *not runnable* | ✓ |
| RapidSpeech f16 | 639.1 | **386.5** (`RS_GEMM_FP16`) | ✓ |
| RapidSpeech q8_0 | 459.2 | **355.3** (FP32) | ✓ |
| RapidSpeech q3_k-im | 498.5 | 445.4 | ✓ |
| RapidSpeech iq4_xs | 452.2 | 451.8 | ✓ |

- **sherpa-onnx CUDA cannot run on the Nano** (onnxruntime is CPU-only here; no
  aarch64 GPU wheel, and it would OOM 4 GB). RapidSpeech.cpp is the only way to use
  the Nano GPU for this model — that is the point of the port.
- **`RS_GEMM_FP16=1` gives RapidSpeech CUDA a ~1.6–1.75× encoder speedup on f16**
  (sm_53 has native 2× FP16 throughput); it reaches parity with sherpa-onnx CPU
  while leaving the 4 A57 cores free. On already-quantized weights the lever is
  neutral — quantization and FP16 are *substitute* bandwidth levers.
- **sherpa-onnx CPU is the fastest engine** (onnxruntime/MLAS: tuned ARM GEMM, op
  fusion, int8) and **saturates at 3 threads**; RapidSpeech CPU scales ~linearly to
  4 threads but starts behind. **CUDA latency is independent of CPU threads.**
- All correct except q4_k (lowercases "monday", drops a comma).

## Usage

```bash
# RapidSpeech.cpp WebSocket streaming server
rs-xasr-ws-server -m 960ms/x-asr-zh-en-960ms-f16.gguf --port 6006
```

See [RapidSpeech.cpp](https://github.com/vieenrose/RapidSpeech.cpp) for build
instructions (incl. the CUDA-10.2 / sm_53 Jetson Nano path).

## License

Apache-2.0, following the upstream X-ASR model.