---
license: cc-by-4.0
base_model: nvidia/canary-qwen-2.5b
base_model_relation: quantized
pipeline_tag: automatic-speech-recognition
tags:
- automatic-speech-recognition
- speech-to-text
- canary-qwen
- nvidia
- qwen
- optimum-quanto
- qint8
- int8
- open-webui
- openai-compatible
- rtx-5090
- quanto
- quantized
- openwebui
- 1.7gb
- audio-encoder-unquantized
- llm-only-quant
- asr-preserved
- systemd
- fastapi
- webm-opus
- browser-mic
- quanto-qint8
- qint8-safetensors
- 8-bit
- 8bit
- safetensors
language:
- en
---

# Canary-Qwen-2.5B Quanto 8-bit Safetensors STT


A smaller local speech-to-text deployment package for `nvidia/canary-qwen-2.5b`, built to avoid the BF16 VRAM footprint while preserving the speech/audio recognition path. It includes an OpenAI-compatible STT service and an 8-bit safetensors LLM-component artifact.

## Quick facts

- **Artifact size:** `llm_qint8.safetensors` is approximately **1.7 GB**, much smaller than the upstream BF16 model weights
- **Observed loaded STT process size:** approximately **6.5 GB GPU memory** on the tested RTX 5090 stack, not a 20GB+ VRAM hog
- **What was reduced:** only the **SALM LLM component** is 8-bit quantized
- **What was preserved:** the speech/audio perception side is **not quantized**, so the part most directly tied to ASR quality is left intact
- **Open WebUI support:** browser microphone WebM/Opus uploads are normalized through `ffmpeg` before transcription

A smaller local speech-to-text deployment package for `nvidia/canary-qwen-2.5b`, built to avoid the BF16 VRAM footprint while preserving the speech/audio recognition path. It includes an OpenAI-compatible STT service and an 8-bit safetensors LLM-component artifact.

This repository includes service code, systemd configuration, smoke-test scripts, environment notes, and optional Optimum Quanto qint8 artifact scripts for the SALM LLM component.


## Included qint8 safetensors artifact

This repo includes an exported Optimum Quanto qint8 safetensors artifact for the SALM LLM component:

```text
quantized_artifacts/llm_qint8.safetensors      approximately 1.7 GB
quantized_artifacts/llm_quantization_map.json  required for requantize
quantized_artifacts/artifact_manifest.json
```

This is not a generic `AutoModel.from_pretrained()` root model file. It is a component artifact intended for the included Canary-Qwen STT wrapper and reload path.

## Included components

- `canary_qwen_stt_service.py`
  - FastAPI OpenAI-compatible `/v1/audio/transcriptions` endpoint
  - Loads `nvidia/canary-qwen-2.5b`
  - Quantizes the LLM component with Optimum Quanto qint8
  - Normalizes uploaded audio through `ffmpeg` to 16 kHz mono PCM WAV before Lhotse/Canary-Qwen processing

- `canary_qwen_stt_service_8001.py`
  - Runs the same FastAPI app on port `8001`

- `systemd/stt-compat-8001.service`
  - Systemd service for Open WebUI-compatible deployment on port `8001`

- `scripts/`
  - Host curl test
  - Open WebUI container-path test
  - Systemd install helper
  - qint8 export script
  - qint8 reload smoke-test script

- `docs/`
  - Environment capture
  - pip freeze
  - validation summary

- `quantized_artifacts/`
  - Intended location for optional exported qint8 LLM-component artifacts

## What this is

This is a reproducible local deployment package and integration path for Canary-Qwen speech-to-text behind Open WebUI.

It provides:

- OpenAI-compatible STT endpoint
- Open WebUI browser microphone compatibility
- WebM/Opus upload handling through ffmpeg normalization
- Runtime Optimum Quanto qint8 quantization
- Optional exported qint8 artifact workflow

## What this is not

- This is not a newly trained base model.
- This does not claim ownership of NVIDIA's Canary-Qwen weights.
- This is not advertised as a generic `AutoModel.from_pretrained()` drop-in model.
- The qint8 artifact path applies to the SALM LLM component used by this wrapper.

## Validated local behavior

Validated on a local RTX 5090 system:

- Host health check on port `8001`: OK
- Host WAV transcription request on port `8001`: OK
- Open WebUI Docker container request to port `8001`: OK
- Real Open WebUI browser microphone WebM/Opus upload: OK
- Earlier duplicate port `8022` service stopped to avoid double-loading the model


## Observed local memory

On the tested Sera5090 RTX 5090 stack after restart:

```text
NVIDIA-SMI total: 32607 MiB
NVIDIA-SMI used: 29417 MiB
Canary-Qwen STT process: 6500 MiB
service health vram_free_gib: 2.631
service health vram_total_gib: 31.358
```

These are observed full-system loaded-state figures, not isolated model-only VRAM measurements.

## Architecture

```text
Open WebUI browser mic
  -> Open WebUI audio router
  -> http://<host>:8001/v1/audio/transcriptions
  -> FastAPI Canary-Qwen STT service
  -> ffmpeg normalize input to 16 kHz mono PCM WAV
  -> Canary-Qwen SALM generate()
  -> OpenAI-style JSON response