Luigi/qwen3-asr-0.6b-rapidspeech-gguf

Qwen3-ASR-0.6B — GGUF (Jetson Nano gen1 optimized)

Single-file GGUF builds of Qwen/Qwen3-ASR-0.6B (audio encoder + Qwen3-0.6B LLM decoder) that run with CUDA on the original Jetson Nano gen1 (Tegra X1, sm_53, CUDA 10.2). Native Traditional Chinese + punctuation, zh/en code-switch.

Variants

file	size	accuracy	recommended for
qwen3-asr-0.6b-f16.gguf	1.88 GB	full (native Traditional)	reference / max accuracy
qwen3-asr-0.6b-q8_0.gguf	0.96 GB	full (native Traditional)	Jetson Nano gen1 (recommended)
qwen3-asr-0.6b-q4_0.gguf	0.52 GB	warning: collapses to Simplified	Ampere+ GPUs only, Simplified-OK

Why q8_0, not q4_0, on the Jetson Nano (Maxwell sm_53)

Maxwell sm_53 has no dp4a (int8 dot-product), so ggml dequantizes low-bit weights and the extra ALU work makes q4_0 slower than q8_0/f16 on this GPU (measured ~1.7x slower). q4_0 also loses Traditional-character precision on this 0.6B model (output collapses to Simplified). So q8_0 is the sweet spot: half the size of f16, same speed and accuracy on Maxwell. q4_0 is only worth it on Ampere+ (fast low-bit GEMM) where Simplified output is acceptable.

Engines (both Jetson-Nano-gen1 optimized)

Two open engines run this GGUF on the Nano gen1; both share the cuFFT batched mel and the CUDA-10.2/sm_53 build fixes:

engine	RTF (13.5s clip, q8_0)	notes
vieenrose/qwen3-asr.cpp (standalone)	~1.1	dedicated 2-engine design: encoder on GPU + autoregressive decode on CPU (~195 ms/tok); fastest on the Nano
vieenrose/RapidSpeech.cpp (integrated)	~1.3	Qwen3-ASR inside the multi-model RapidSpeech framework (all weights on GPU); ships the converter scripts/convert_qwen3_asr_to_gguf.py

The standalone reaches lower RTF because it keeps encoder weights on the GPU and LLM weights on the CPU simultaneously (batch-1 decode is faster on the A57 than on Maxwell); RapidSpeech's unified single-weight-buffer keeps everything on GPU.

Usage

Standalone:

qwen3-asr-cli -m qwen3-asr-0.6b-q8_0.gguf -f clip_16k.wav -t 4

RapidSpeech:

rs-asr-offline -m qwen3-asr-0.6b-q8_0.gguf -w clip_16k.wav

Convert your own

From a local Qwen/Qwen3-ASR-0.6B checkpoint, using the RapidSpeech fork:

python scripts/convert_qwen3_asr_to_gguf.py --hf-dir /path/to/Qwen3-ASR-0.6B \
    --output qwen3-asr-0.6b-q8_0.gguf --quant q8_0