Masterx/cohere-transcribe-arabic-07-2026-ONNX

Cohere Transcribe Arabic 07-2026 — DirectML-optimized ONNX

ONNX export of CohereLabs/cohere-transcribe-arabic-07-2026 (2B, Arabic + English) restructured to run fast on the ONNX Runtime DirectML EP. Same architecture as cohere-transcribe-03-2026, hand-exported from the reference PyTorch weights (stock optimum can't export cohere_asr).

What changed (tools/onnx/cohere_decompose_attention.py in WinSTT)

1. Loop-invariant cross-KV hoisted into the encoder — cross-attention K/V (which depend only on the audio) are computed once per utterance in encoder_model*.onnx, not per decoded token.
2. Static self-KV decode — the self-attention KV cache is a fixed-length buffer updated by a masked write, so the DirectML EP compiles the decoder once per utterance instead of re-fusing its graph on every autoregressive step. Measured 3.5 ms/token on DirectML (RTX 3080 Ti) vs ~16 ms/token on the CPU EP — **4.6× faster**, entirely on the GPU.

Two decoders per precision (both fed by the hoisted encoder, sharing the decoder sidecar): decoder_model_merged*.onnx (static self-KV, marker winstt_static_kv, fastest on DirectML) and decoder_model_merged*_dyn.onnx (growing self-KV, faster on the CPU EP). WinSTT picks per device.

Numerically verified against the source on CPU (autoregressive logits parity: bit-exact fp32/q4).

Notes vs the multilingual export: this torch-traced graph uses a (B, S, nh, hd) head layout, so the cross-attention is left dynamic (not padded to a fixed bucket) — perfectly fast for the ≤10 s segments a dictation VAD produces; only unrealistically long single clips would benefit from bucketing. The int8 decoder ships dynamic-only (decoder_model_merged_int8.onnx is the growing-self graph): its DynamicQuantizeLinear scales interact badly with the fixed-KV write, so it runs the hybrid path (encoder-DML / decoder-CPU). fp32 and q4 get the full static decode.

Files

• onnx/encoder_model[_int8|_q4].onnx (+ external data) — hoisted encoders (cross-KV outputs).
• onnx/decoder_model_merged[_q4].onnx (+ _dyn) — static / dynamic decoders (fp32, q4).
• onnx/decoder_model_merged_int8.onnx — dynamic-only (int8).
• Tokenizer / configs — unchanged.