Add int8 dynamic quantization (encoder + merged decoder)

by Masterx - opened 7 days ago

base: refs/heads/main

←

from: refs/pr/2

Discussion Files changed

+14

-0

Masterx

7 days ago

8-bit dynamic weight quantization (ONNX Runtime quantize_dynamic, QInt8) of the fp32 encoder and merged decoder. ~2.1 GB total. Verified: greedy decode matches fp32 on an English clip (en/en and unklang/unklang both correct). Complements the existing fp16/q4/q4f16 with an accuracy tier between q4 and fp32.

Add int8 dynamic quantization (encoder + merged decoder)378c058b

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Ready to merge

This branch is ready to get merged automatically.

· Sign up or log in to comment