---
license: cc-by-4.0
base_model: nvidia/canary-qwen-2.5b
base_model_relation: quantized
pipeline_tag: automatic-speech-recognition
tags:
- automatic-speech-recognition
- speech-to-text
- canary-qwen
- nvidia
- qwen
- optimum-quanto
- qint8
- int8
- open-webui
- openai-compatible
- rtx-5090
- quanto
- quantized
language:
- en
---

# Canary-Qwen-2.5B Quanto QInt8 OpenWebUI STT

A practical local deployment package for running `nvidia/canary-qwen-2.5b` as an OpenAI-compatible speech-to-text endpoint for Open WebUI.

This repository includes service code, systemd configuration, smoke-test scripts, environment notes, and optional Optimum Quanto qint8 artifact scripts for the SALM LLM component.

## Included components

- `canary_qwen_stt_service.py`
  - FastAPI OpenAI-compatible `/v1/audio/transcriptions` endpoint
  - Loads `nvidia/canary-qwen-2.5b`
  - Quantizes the LLM component with Optimum Quanto qint8
  - Normalizes uploaded audio through `ffmpeg` to 16 kHz mono PCM WAV before Lhotse/Canary-Qwen processing

- `canary_qwen_stt_service_8001.py`
  - Runs the same FastAPI app on port `8001`

- `systemd/stt-compat-8001.service`
  - Systemd service for Open WebUI-compatible deployment on port `8001`

- `scripts/`
  - Host curl test
  - Open WebUI container-path test
  - Systemd install helper
  - qint8 export script
  - qint8 reload smoke-test script

- `docs/`
  - Environment capture
  - pip freeze
  - validation summary

- `quantized_artifacts/`
  - Intended location for optional exported qint8 LLM-component artifacts

## What this is

This is a reproducible local deployment package and integration path for Canary-Qwen speech-to-text behind Open WebUI.

It provides:

- OpenAI-compatible STT endpoint
- Open WebUI browser microphone compatibility
- WebM/Opus upload handling through ffmpeg normalization
- Runtime Optimum Quanto qint8 quantization
- Optional exported qint8 artifact workflow

## What this is not

- This is not a newly trained base model.
- This does not claim ownership of NVIDIA's Canary-Qwen weights.
- This is not advertised as a generic `AutoModel.from_pretrained()` drop-in model.
- The qint8 artifact path applies to the SALM LLM component used by this wrapper.

## Validated local behavior

Validated on a local RTX 5090 system:

- Host health check on port `8001`: OK
- Host WAV transcription request on port `8001`: OK
- Open WebUI Docker container request to port `8001`: OK
- Real Open WebUI browser microphone WebM/Opus upload: OK
- Earlier duplicate port `8022` service stopped to avoid double-loading the model

## Architecture

```text
Open WebUI browser mic
  -> Open WebUI audio router
  -> http://<host>:8001/v1/audio/transcriptions
  -> FastAPI Canary-Qwen STT service
  -> ffmpeg normalize input to 16 kHz mono PCM WAV
  -> Canary-Qwen SALM generate()
  -> OpenAI-style JSON response