# Canary-Qwen-2.5B ONNX export for onnx-asr

ONNX export of `nvidia/canary-qwen-2.5b` (NeMo SALM) into the `speech-llm` family of the
TigreGotico onnx-asr fork. It is the second model of that family, after Qwen3-ASR-0.6B.

## Source model

* Repository: `nvidia/canary-qwen-2.5b`, CC-BY-4.0, NVIDIA. English only.
* Architecture (from its `config.json`): NeMo `AudioPerceptionModule` = NeMo log-mel
  featurizer (128 mel, `n_fft` 512, window 25 ms, hop 10 ms, per-feature normalization) ->
  `ConformerEncoder` (32 layers, `d_model` 1024, `dw_striding` subsampling factor 8, full
  `rel_pos` attention) -> `IdentityConnector` -> `Linear(1024, 2048)`. The LLM is an
  unmodified `Qwen/Qwen3-1.7B` (28 layers, hidden 2048, 8 KV heads, head dim 128, vocab
  151936, tied embeddings) with LoRA `r=128` on `q_proj` and `v_proj`.
* The export merges the LoRA adapters into the base weights (`merge_and_unload`), so the
  decoder graph is a plain Qwen3 causal LM.

## Graph contract

| Graph | Inputs | Outputs |
| --- | --- | --- |
| `encoder.onnx` | `input_features` `(1, 128, T)` f32, `input_features_lens` `(1,)` i64 | `audio_embeds` `(1, L, 2048)` f32, `audio_embeds_lens` `(1,)` i64 |
| `embed_tokens.onnx` | `input_ids` `(1, S)` i64 | `inputs_embeds` `(1, S, 2048)` f32 |
| `decoder.onnx` | `inputs_embeds` `(1, S, 2048)`, `attn_bias` `(1, 1, S, P+S)`, `position_ids` `(1, S)`, `past_key_values.{0..27}.{key,value}` `(1, 8, P, 128)` | `logits` `(1, S, 151936)`, `present.{0..27}.{key,value}` `(1, 8, P+S, 128)` |

`decoder.onnx` is a merged prefill and decode graph; a zero-length past is valid.

Unlike the Qwen3-ASR encoder, this encoder needs no `valid_indices` and no `attn_bias`: the
FastConformer uses full attention and builds its own padding mask from the length, so the
length goes in and the valid embedding length comes out.

### Preprocessor

The model's featurizer is the standard NeMo `AudioToMelSpectrogramPreprocessor`, which is what
onnx-asr already implements as `nemo128`. `config.json` sets `"preprocessor": "nemo128"`; no
preprocessor code was added. Two known small differences remain in onnx-asr's `nemo128`, both
pre-existing and shared with every other NeMo model in onnx-asr: it zero-pads the STFT instead
of reflect-padding it, and it returns one mel frame less than NeMo. Neither changed the output
on any test clip.

### Prompt

`config.json` carries the Qwen chat-template ids that NeMo's `PromptFormatter.resolve("qwen")`
produces for the turn `{"role": "user", "content": "Transcribe the following: <|audioplaceholder|>"}`,
split at the audio placeholder id (151669):

* `prompt_prefix_ids` = `[151644, 872, 198, 3167, 3114, 279, 2701, 25, 220]`
* `prompt_suffix_ids` = `[151645, 198, 151644, 77091, 198]`

No tokenizer encoder is needed at runtime. Detokenization uses `vocab.json` with the GPT-2 byte
table, the same path as Qwen3-ASR. `eos_token_ids` are 151643 and 151645. There is no preamble
token, so `text_start_token_id` is unset.

## Runtime change

Branch `feat/speech-llm-canary` (from `feat/speech-llm-qwen3-asr`), commit `e913ce6`, one file,
13 added lines in `src/onnx_asr/models/speech_llm.py`:

1. `SpeechLlm._encode` gains a third encoder shape. When the graph declares
   `input_features_lens`, the runtime passes the feature length and trims the returned
   `audio_embeds` to `audio_embeds_lens`. The two existing shapes (Whisper-style fixed length,
   Qwen3-ASR packed windows) are untouched.
2. The feature length is clamped to the number of frames the preprocessor actually produced.
   The NeMo preprocessor returns `floor(samples / hop)` frames while the runtime computed
   `ceil(samples / hop)`, which would have declared one frame more than the array holds. This
   is a no-op for the Whisper-based models, whose preprocessor always returns 3000 frames.

No new preprocessor, no new model class, no loader or resolver change.

## Files

| File | Size |
| --- | --- |
| `encoder.onnx` + `encoder.onnx_data` | 42.4 MB + 3252.1 MB |
| `encoder_int8.onnx` | 859.3 MB |
| `embed_tokens.onnx` | 1244.7 MB |
| `embed_tokens_int8.onnx` | 311.2 MB |
| `decoder.onnx` + `decoder.onnx_data` | 1.6 MB + 6882.3 MB |
| `decoder_int8.onnx` | 1723.3 MB |
| `vocab.json` | 3.1 MB |
| `config.json` | 617 B |

Total fp32 about 11.4 GB, total int8 about 2.9 GB. int8 is dynamic weight quantization
(`QInt8`, `MatMulConstBOnly`), with `use_external_data_format=True` for the encoder and the
decoder (both over 1.5 GB fp32).

## Verification

Clips: Google FLEURS validation, two `en_us` and two `pt_br`, the same four files used for the
Qwen3-ASR export. The model is English-only, so the Portuguese clips are robustness probes.
Native reference: `nemo.collections.speechlm2.models.SALM.generate` in fp32 on CPU.
ONNX: `onnx_asr.load_model("speech-llm", ...)` on CPU from the branch above.

| Clip | native NeMo fp32 | ONNX fp32 | ONNX int8 |
| --- | --- | --- | --- |
| en_1 | When you call someone who is thousands of miles away you are using a satellite | identical | identical |
| en_2 | Now widely available throughout the archipelago, Japanese cuisine features an array of simply seasoned dishes, the predominant flavorings the Japanese favor being peanuts, chillies, sugar, especially Japanese coconut sugar, and various aromatic spices. | identical | "Archipelago" and "chilies" instead of "archipelago" and "chillies" |
| pt_1 | Objection principale de cette analyse est entierement fonctionnelle a travers des methodes scientifiques et des methodes de courant d'interpretations scientifiques | identical | repetition loop ("I'm not sure how to say it ...") |
| pt_2 | Esquiar y hacer viaje importante y viaje para muchos entusiastas ocasionalmente coincidir con ski bums, trepanaciones, ferias interas o para esquiar en local especifico | identical | repetition loop, hits the 448-token cap |

ONNX fp32 matches native character for character on **all four** clips, including the two
Portuguese ones where the model produces nonsense. Both nonsense outputs are reproduced exactly,
which is a stronger parity check than the English clips alone.

int8 is safe on English and unusable on out-of-domain audio.

`native.json`, `onnx_int8.json`, `onnx_fp32.log` and `onnx_int8.log` hold the exact strings.

## Speed

AMD Ryzen 5 7600 (6 cores, 12 threads), onnxruntime 1.28.0 CPU execution provider, default
intra-op thread count (12), greedy decoding, batch 1. Weights on a 7200 rpm HDD.

| Clip | audio | native NeMo | ONNX fp32 | ONNX int8 |
| --- | --- | --- | --- | --- |
| en_1 | 6.54 s | 4.79 s (RTFx 1.4) | 23.7 s (RTFx 0.28) | 1.87 s (RTFx 3.5) |
| en_2 | 16.38 s | 12.75 s (RTFx 1.3) | 37.4 s (RTFx 0.44) | 4.71 s (RTFx 3.5) |
| pt_1 | 11.82 s | 10.53 s (RTFx 1.1) | 14.2 s (RTFx 0.83) | 3.97 s (RTFx 3.0) |
| pt_2 | 14.58 s | 12.61 s (RTFx 1.2) | 13.4 s (RTFx 1.1) | 37.2 s (repetition loop) |

Model load, cold: 787 s fp32, 119 s int8. Both are disk reads of 11.4 GB and 2.9 GB from a
spinning disk, not onnxruntime work.

fp32 ONNX is slower than PyTorch here: 11 GB of weights, one graph run per decoded token, and
no fused attention kernel. int8 is the configuration worth using on CPU.

## Limitations

* English only, by the source model.
* Batch 1 graphs. `recognize_batch` loops over the waveforms.
* Greedy decoding, no beam search, no timestamps.
* The source model's LLM mode (summarize or answer questions about a transcript, with the LoRA
  adapters disabled) is not exported. Only the ASR prompt is baked in.
* Trained on audio up to 40 s and 1024 total tokens.

## Reproducing

On a box with 32 GB of RAM and 30 GB of free disk, Python 3.12:

```sh
uv venv --python 3.12 venv && source venv/bin/activate
uv pip install torch --index-url https://download.pytorch.org/whl/cpu
uv pip install "nemo_toolkit[asr]==2.7.3" "numba>=0.61" "numpy<2.3" onnx onnxruntime soundfile
python export_canary_qwen.py          # graphs, vocab.json, config.json, int8
python verify_native.py               # native.json
python verify_onnx.py fp32 int8       # onnx.json
```

## Published

`OpenVoiceOS/canary-qwen-2.5b-onnx`, in collection
`OpenVoiceOS/stt-asr-onnx-699321e8732462509c642fbe`.