SLLLC/canary-qwen-2.5b-quanto-8bit-safetensors-stt

Canary-Qwen-2.5B Quanto 8-bit Safetensors STT

A smaller local speech-to-text deployment package for nvidia/canary-qwen-2.5b, built to avoid the BF16 VRAM footprint while preserving the speech/audio recognition path. It includes an OpenAI-compatible STT service and an 8-bit safetensors LLM-component artifact.

Quick facts

• Artifact size: llm_qint8.safetensors is approximately 1.7 GB, much smaller than the upstream BF16 model weights
• Observed loaded STT process size: approximately 6.5 GB GPU memory on the tested RTX 5090 stack, not a 20GB+ VRAM hog
• What was reduced: only the SALM LLM component is 8-bit quantized
• What was preserved: the speech/audio perception side is not quantized, so the part most directly tied to ASR quality is left intact
• Open WebUI support: browser microphone WebM/Opus uploads are normalized through ffmpeg before transcription

A smaller local speech-to-text deployment package for nvidia/canary-qwen-2.5b, built to avoid the BF16 VRAM footprint while preserving the speech/audio recognition path. It includes an OpenAI-compatible STT service and an 8-bit safetensors LLM-component artifact.

This repository includes service code, systemd configuration, smoke-test scripts, environment notes, and optional Optimum Quanto qint8 artifact scripts for the SALM LLM component.

Included qint8 safetensors artifact

This repo includes an exported Optimum Quanto qint8 safetensors artifact for the SALM LLM component:

quantized_artifacts/llm_qint8.safetensors      approximately 1.7 GB
quantized_artifacts/llm_quantization_map.json  required for requantize
quantized_artifacts/artifact_manifest.json

This is not a generic AutoModel.from_pretrained() root model file. It is a component artifact intended for the included Canary-Qwen STT wrapper and reload path.

Included components

• canary_qwen_stt_service.py

  • FastAPI OpenAI-compatible /v1/audio/transcriptions endpoint
  • Loads nvidia/canary-qwen-2.5b
  • Quantizes the LLM component with Optimum Quanto qint8
  • Normalizes uploaded audio through ffmpeg to 16 kHz mono PCM WAV before Lhotse/Canary-Qwen processing

• canary_qwen_stt_service_8001.py

  • Runs the same FastAPI app on port 8001

• systemd/stt-compat-8001.service

  • Systemd service for Open WebUI-compatible deployment on port 8001

• scripts/

  • Host curl test
  • Open WebUI container-path test
  • Systemd install helper
  • qint8 export script
  • qint8 reload smoke-test script

• docs/

  • Environment capture
  • pip freeze
  • validation summary

• quantized_artifacts/

  • Intended location for optional exported qint8 LLM-component artifacts

What this is

This is a reproducible local deployment package and integration path for Canary-Qwen speech-to-text behind Open WebUI.

It provides:

• OpenAI-compatible STT endpoint
• Open WebUI browser microphone compatibility
• WebM/Opus upload handling through ffmpeg normalization
• Runtime Optimum Quanto qint8 quantization
• Optional exported qint8 artifact workflow

What this is not

• This is not a newly trained base model.
• This does not claim ownership of NVIDIA's Canary-Qwen weights.
• This is not advertised as a generic AutoModel.from_pretrained() drop-in model.
• The qint8 artifact path applies to the SALM LLM component used by this wrapper.

Validated local behavior

Validated on a local RTX 5090 system:

• Host health check on port 8001: OK
• Host WAV transcription request on port 8001: OK
• Open WebUI Docker container request to port 8001: OK
• Real Open WebUI browser microphone WebM/Opus upload: OK
• Earlier duplicate port 8022 service stopped to avoid double-loading the model

Observed local memory

On the tested Sera5090 RTX 5090 stack after restart:

NVIDIA-SMI total: 32607 MiB
NVIDIA-SMI used: 29417 MiB
Canary-Qwen STT process: 6500 MiB
service health vram_free_gib: 2.631
service health vram_total_gib: 31.358

These are observed full-system loaded-state figures, not isolated model-only VRAM measurements.

Architecture

Open WebUI browser mic
  -> Open WebUI audio router
  -> http://<host>:8001/v1/audio/transcriptions
  -> FastAPI Canary-Qwen STT service
  -> ffmpeg normalize input to 16 kHz mono PCM WAV
  -> Canary-Qwen SALM generate()
  -> OpenAI-style JSON response