---
license: cc-by-4.0
language: [en]
tags: [automatic-speech-recognition, onnx, onnx-asr, speech-llm, salm, fastconformer, canary, qwen3]
base_model: nvidia/canary-qwen-2.5b
library_name: onnx-asr
---

# Canary-Qwen-2.5B — ONNX (speech-llm)

ONNX export of [nvidia/canary-qwen-2.5b](https://huggingface.co/nvidia/canary-qwen-2.5b) for
[onnx-asr](https://github.com/istupakov/onnx-asr) with the `speech-llm` model type.

The source model is a NeMo SALM: a FastConformer encoder (32 layers, `d_model` 1024, 8x
subsampling) with a linear projection into the embedding space of an unmodified Qwen3-1.7B
decoder. It tops the Open ASR Leaderboard for English. Model and weights are by NVIDIA,
released under CC-BY-4.0.

## Requirements

This export needs the `speech-llm` model family, which is not in upstream onnx-asr yet:

* branch [`feat/speech-llm-canary`](https://github.com/TigreGotico/onnx-asr/tree/feat/speech-llm-canary)
  of the TigreGotico fork. The stock code of
  [PR #3](https://github.com/TigreGotico/onnx-asr/pull/3) is **not** enough: the NeMo encoder
  graph takes the feature length and returns the embedding length, which PR #3 does not pass.

```sh
pip install "onnx-asr @ git+https://github.com/TigreGotico/onnx-asr@feat/speech-llm-canary"
```

## Usage

```python
import onnx_asr

model = onnx_asr.load_model("speech-llm", "path/to/this/repo")  # or quantization="int8"
print(model.recognize("audio_16khz.wav"))
```

## Graph layout

| Graph | Inputs | Outputs |
| --- | --- | --- |
| `encoder.onnx` | `input_features` `(1, 128, T)`, `input_features_lens` `(1,)` | `audio_embeds` `(1, L, 2048)`, `audio_embeds_lens` `(1,)` |
| `embed_tokens.onnx` | `input_ids` `(1, S)` | `inputs_embeds` `(1, S, 2048)` |
| `decoder.onnx` | `inputs_embeds`, `attn_bias`, `position_ids`, `past_key_values.{0..27}.{key,value}` `(1, 8, P, 128)` | `logits`, `present.{0..27}.{key,value}` |

`decoder.onnx` is a merged prefill and decode graph; a zero-length past is valid, so there is no
`use_cache_branch`. The chat-template prompt token ids (`Transcribe the following: <audio>`,
Qwen chat template) are baked into `config.json`, so no tokenizer encoder is needed at runtime.
The audio front end is the standard NeMo log-mel featurizer, so `config.json` selects the
existing `nemo128` preprocessor of onnx-asr; no new preprocessor code was added.

## Files

| File | Size |
| --- | --- |
| `encoder.onnx` + `encoder.onnx_data` | 42.4 MB + 3.25 GB |
| `encoder_int8.onnx` | 859.3 MB |
| `embed_tokens.onnx` | 1.24 GB |
| `embed_tokens_int8.onnx` | 311.2 MB |
| `decoder.onnx` + `decoder.onnx_data` | 1.6 MB + 6.88 GB |
| `decoder_int8.onnx` + `decoder_int8.onnx_data` | 1.7 GB |
| `vocab.json` | 3.1 MB |

Total fp32 about 11.4 GB, total int8 about 2.9 GB. int8 is dynamic weight quantization
(`QInt8`, `MatMulConstBOnly`).

## Accuracy

Four Google FLEURS validation clips, two `en_us` and two `pt_br`. The model is English-only;
the Portuguese clips are robustness probes, not a quality claim.

ONNX fp32 matches native NeMo `SALM.generate` **character for character on all four clips**,
including the two Portuguese ones where both produce nonsense.

int8 keeps the English clips correct, with two trivial differences on the longer one
("Archipelago" instead of "archipelago", "chilies" instead of "chillies"). On the Portuguese
clips int8 collapses into repetition loops. Use int8 for English only.

| Clip | native NeMo fp32 | ONNX fp32 | ONNX int8 |
| --- | --- | --- | --- |
| en_1 | When you call someone who is thousands of miles away you are using a satellite | identical | identical |
| en_2 | Now widely available throughout the archipelago, Japanese cuisine features an array of simply seasoned dishes, ... | identical | "Archipelago", "chilies" |
| pt_1 | (French-sounding nonsense) | identical | repetition loop |
| pt_2 | (Spanish-sounding nonsense) | identical | repetition loop |

## Speed

AMD Ryzen 5 7600 (6 cores, 12 threads), onnxruntime 1.28.0 CPU provider, default thread count,
greedy decoding, batch 1.

| Clip | audio | native NeMo | ONNX fp32 | ONNX int8 |
| --- | --- | --- | --- | --- |
| en_1 | 6.5 s | 4.8 s (RTFx 1.4) | 23.7 s (RTFx 0.28) | 1.9 s (RTFx 3.5) |
| en_2 | 16.4 s | 12.8 s (RTFx 1.3) | 37.4 s (RTFx 0.44) | 4.7 s (RTFx 3.5) |
| pt_1 | 11.8 s | 10.5 s (RTFx 1.1) | 14.2 s (RTFx 0.83) | 4.0 s (RTFx 3.0) |
| pt_2 | 14.6 s | 12.6 s (RTFx 1.2) | 13.4 s (RTFx 1.1) | 37.2 s (loop) |

Model load (cold, weights on a 7200 rpm HDD): 787 s fp32, 119 s int8. On an SSD this is
dominated by the file read, not by onnxruntime.

fp32 on CPU is slower than PyTorch because the 11 GB of weights do not fit in cache and each
decode step is a separate graph run. int8 is the useful CPU configuration.

## Limitations

* English only. The encoder saw German, French and Spanish during pretraining, so other
  languages produce plausible-looking nonsense rather than an error.
* Batch 1 graphs. `recognize_batch` loops over the waveforms.
* Greedy decoding only. No beam search, no timestamps.
* Trained on audio up to 40 s and 1024 total tokens. Longer audio may degrade.
* Only the baked ASR prompt. The LLM mode of the source model (summarize, answer questions
  about a transcript) is not exported.

## License and attribution

CC-BY-4.0, inherited from the source model. Model and weights: NVIDIA
([nvidia/canary-qwen-2.5b](https://huggingface.co/nvidia/canary-qwen-2.5b)), built on
`nvidia/canary-1b-flash` and `Qwen/Qwen3-1.7B`. This repository only adds the ONNX export.