Add files using upload-large-folder tool

.gitattributes

@@ -50,3 +50,4 @@ onnx/encoder_model_fp16.onnx_data filter=lfs diff=lfs merge=lfs -text
 onnx/encoder_model_q4f16.onnx_data filter=lfs diff=lfs merge=lfs -text
 onnx/encoder_model.onnx_data_3 filter=lfs diff=lfs merge=lfs -text
 onnx/decoder_model_merged_q4f16.onnx_data filter=lfs diff=lfs merge=lfs -text
+onnx/encoder_model_int8.onnx_data_1 filter=lfs diff=lfs merge=lfs -text

README.md

@@ -26,67 +26,54 @@ base_model:
 - CohereLabs/cohere-transcribe-03-2026
 ---
 
-# Cohere Transcribe 03-2026 — DirectML-safe, decode-optimized ONNX
+# Cohere Transcribe 03-2026 — DirectML-safe, fully-static ONNX
 
 ONNX export of [CohereLabs/cohere-transcribe-03-2026](https://huggingface.co/CohereLabs/cohere-transcribe-03-2026)
-restructured so the model runs **correctly and fast on the ONNX Runtime DirectML EP** (and every other
-EP). Weight bytes are identical to
+restructured to run **correctly and fast on the ONNX Runtime DirectML EP** (and every other EP).
+Weight bytes are identical to
 [onnx-community/cohere-transcribe-03-2026-ONNX](https://huggingface.co/onnx-community/cohere-transcribe-03-2026-ONNX);
-only the graph protos differ (the encoder proto additionally embeds the ~67 MB of cross-attention
-projection weights hoisted out of the decoder — see below).
+the encoder proto additionally embeds the cross-attention projection weights hoisted out of the
+decoder (see below).
 
-## Why the stock export can't run on DirectML
+## Why the stock export can't run well on DirectML
 
-The onnx-community decoders fuse attention into two `com.microsoft` contrib ops that the DirectML EP
-mishandles (ORT ≤ 1.24, unfixed — the DML EP is in maintenance mode):
+- The fused decoder attention (`com.microsoft.MultiHeadAttention` cross-attn + `GroupQueryAttention`
+  self-attn) **crashes / mis-computes** on the DML EP (ORT ≤ 1.24, unfixed).
+- Even after decomposing to plain ops, the DML EP **re-fuses its graph on every autoregressive
+  shape change** (~34 ms/token fixed), so naïve GPU decode loses to CPU.
 
-- **`MultiHeadAttention`** (cross-attention, 3-D query + 4-D precomputed K/V): the DML kernel routes
-  the 4-D K/V into its PastKey/PastValue slots and builds a `DML_MULTIHEAD_ATTENTION_OPERATOR_DESC`
-  with null Key/Value tensors, which DirectML rejects → `E_INVALIDARG` crash.
-- **`GroupQueryAttention`** (self-attention): the DML kernel **silently ignores the `attention_bias`
-  input** and never reads `past_key/past_value` (it assumes shared max-length buffers), so a
-  concat-style KV cache decodes garbage with no error.
-- Both are gated by two `If` nodes; control-flow ops have no DML kernel (CPU-only), forcing a
-  GPU↔CPU round-trip of all 16 cross-KV tensors every decoded token.
+## What changed (three passes, `tools/onnx/cohere_decompose_attention.py`)
 
-## What changed
+1. **Attention decomposed** to Reshape/Transpose/MatMul/Softmax/… with an explicit causal mask; the
+   two cross-attn `If` nodes flattened away.
+2. **Loop-invariant cross-KV hoisted into the encoder** — the cross-attention K/V (which depend only
+   on the audio) are computed once per utterance in `encoder_model*.onnx` (extra `cross_attn.*`
+   outputs) instead of on every decoded token.
+3. **Fully static decode** — the self-KV cache is a fixed-length buffer updated by a masked write,
+   the cross-KV are padded to a fixed length with a `cross_bias` mask, and every per-step shape is
+   constant. The DML EP now compiles the decoder **once** and reuses it, so per-token cost is
+   **constant ~4 ms regardless of audio length**, entirely on the GPU.
 
-1. **Attention decomposed** (`decoder_model_merged*.onnx`, all precisions): every
-   `MultiHeadAttention` / `GroupQueryAttention` node rewritten to plain
-   Reshape/Transpose/MatMul/Mul/Add/Softmax/Concat with an explicit causal mask (GQA applies it
-   internally; the exported `attention_bias` input is only the padding mask). The two cross-attn
-   `If` nodes are flattened away.
-2. **Loop-invariant cross-KV hoisted into the encoder**: the cross-attention K/V projections (and
-   the pre-transposed keys) depend only on the encoder output, so they now run ONCE per utterance
-   inside `encoder_model*.onnx` (extra `cross_attn.*` outputs) instead of on EVERY decoded token
-   inside the decoder (which cost ~100× the decoder's real per-token compute). The decoder consumes
-   them as same-named `cross_attn.*` inputs; `present.*.encoder.*` outputs are gone. Decoder proto
-   metadata carries `winstt_hoisted_cross_kv=1`.
+Two decoders are shipped per precision:
 
-Verified numerically against the fused graphs on CPU (autoregressive logits parity: bit-exact for
-fp32/int8/q4; fp16-rounding-level for fp16/q4f16).
+- `decoder_model_merged*.onnx` — **fully static** (fixed self- + cross-KV). Fastest on DirectML
+  (metadata `winstt_static_kv`).
+- `decoder_model_merged*_dyn.onnx` — **growing self-KV** (cross still fixed). Faster on the CPU EP,
+  which doesn't benefit from static shapes. WinSTT loads this one when the decoder is CPU-bound.
 
-Measured on a 66 s clip (fp32, ORT 1.24, RTX 3080 Ti, identical transcripts):
+Both are numerically identical to the fused graph on CPU (autoregressive logits parity: bit-exact
+for fp32/int8/q4; fp16-rounding-level for fp16/q4f16). One padded encoder feeds both.
 
-| export | DirectML | CPU |
-|---|---|---|
-| fused (onnx-community) | crash | 21–33 s |
-| decomposed only | 25.2 s | 20.8 s |
-| decomposed + branchless | 14.0 s | 28.7 s |
-| **this repo (hoisted)** | **2.7 s**¹ | ~7.4 s² |
+Measured decode per-token (fp32, ORT 1.24, RTX 3080 Ti), 3.4–4.4× faster than the prior hybrid:
 
-¹ hybrid placement: encoder on DirectML, decoder on the CPU EP (the DML EP re-fuses its graph on
-every autoregressive shape change, ~34 ms/token fixed; the CPU EP decodes the hoisted decoder at
-~12 ms/token). This is how [WinSTT](https://github.com/dahshury/WinSTT) runs it.
-² all-CPU: encoder 6.1 s + decode 1.3 s.
+| | DirectML (static) | CPU (dynamic) |
+|---|---|---|
+| any length (5 s … 66 s) | **~4 ms/token** | ~12 ms/token |
 
-Produced by WinSTT's `tools/onnx/cohere_decompose_attention.py`.
+Produced by [WinSTT](https://github.com/dahshury/WinSTT).
 
 ## Files
 
-- `onnx/encoder_model[_fp16|_int8|_q4|_q4f16|_quantized].onnx` (+ external data) — onnx-community
-  weights + the hoisted cross-KV tail (extra `cross_attn.*` outputs; projection weights embedded in
-  the proto).
-- `onnx/decoder_model_merged[_fp16|_int8|_q4|_q4f16|_quantized].onnx` (+ external data) —
-  decomposed, branchless, cross-KV-free decoders (consume `cross_attn.*` inputs).
+- `onnx/encoder_model[_fp16|_int8|_q4|_q4f16].onnx` (+ external data) — hoisted, cross-padded encoder.
+- `onnx/decoder_model_merged[_fp16|_int8|_q4|_q4f16].onnx` (+ `_dyn`) — static / dynamic decoders.
 - Tokenizer / configs — unchanged.

onnx/decoder_model_merged.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9d2bc6c61625ad2e96641e76a7056440e0f70db17b7982371215ce4317385e8e
-size 126143
+oid sha256:27b1d14ada637e195cf342b2697cf1577e7e45ac89fa158beac7e7ae37f7ebb8
+size 86993

onnx/decoder_model_merged_dyn.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:636bdfd6df17145167f7ef485396eb89be04bced97193c92a2f1ba300b8df052
+size 92153

onnx/decoder_model_merged_fp16.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:84746ed4964cd92913159df8a29c49609ccdcf5f6db5fa1ab3f1a623a88b322a
-size 127116
+oid sha256:54ed4cbbeb3c62fb4c0372a4c0ebe75d3fffbdf38492f1e2682afd584eed183a
+size 87988

onnx/decoder_model_merged_fp16_dyn.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9b80a9edb9eb7b5859c6bc6dabdcfb280581df0f2de7150cc36153a2ab966808
+size 93137

onnx/decoder_model_merged_int8.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:daecf1938f01ff7adcc2611123cccc26a4a06615dbef80519be88b1d12094ee2
-size 206757
+oid sha256:f8bc38b9dfb63a604a92c89011c36aae772144cdd74386df3766a2a80f14537c
+size 124678

onnx/decoder_model_merged_int8.onnx_data

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a325e8660c93fd1edca7ef546875c19d10f3fa4f9089c1cad480e1ad0c8158b3
-size 219856896
+oid sha256:84d949b4b89d2b2b4e9ba201b28344f2c517fe03bd50a6b92be1756942f9d40b
+size 195903488

onnx/decoder_model_merged_int8_dyn.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:733a29d122ee18e096d7eff9f6daebe532f5e6f01c021ecd403a9877bf7c309f
+size 129838

onnx/decoder_model_merged_q4.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:165e4be4ee266782ff8baae525d42c14ef54587caa05c38720010d527972a2d4
-size 162847
+oid sha256:f37f499b15f11d47415bcde5d8f98acb7dffcc759614b74f958f8d6ea81c7964
+size 123697

onnx/decoder_model_merged_q4_dyn.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0357b92b2b4134c1a47db267e237f2f5920fe330f7cc164ddb996b03a35660ea
+size 128857

onnx/decoder_model_merged_q4f16.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:48c802f4d0b82056d8764628e21a58f6159b589e9f48dd6ef44d94e1a8dcadab
-size 163843
+oid sha256:42ae84f0b5248a1144fa7a1ee92013997f43ba39fce6a33b964d02dd76a0c694
+size 124715

onnx/decoder_model_merged_q4f16_dyn.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d5201d34ad353b0ca1fe1718cdd243bd71691c6a7628319d4fdb8c397cec8cd
+size 129864

onnx/encoder_model.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:328ed95c4e99a022fc2bff2784b67c43d5513ebbc056120489e58c52d342e7fb
-size 68375619
+oid sha256:90ae8ddbc00da09f30f7114fe130e71005cacef0d136fb22f7183b2c3eca3dc5
+size 68033797

onnx/encoder_model_fp16.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b85245dd273d1df75fddd8098a16067cf73595785664ac19629beac40118863c
-size 34800172
+oid sha256:6133d2ad4bf50e0576872d18526e76564dd61a1712a23bbc4bd6c5fd3ef4b89d
+size 34458099

onnx/encoder_model_int8.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e74076d7ff0f33df1b8f00761fa522a4ad5e2b41466448d9abf4ba7e0aebbdf6
-size 69268785
+oid sha256:f6af2d163fecc7b36ca3a2050c66238c5865befceea4a1f2ca0937eb5b9d4445
+size 20554104

onnx/encoder_model_int8.onnx_data

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2e8746b53d8f170d5d909470f28b2dfbf2d47d8659a003b7ec20e184929b4ab8
-size 1902071040
+oid sha256:8733a14ed4e365cacaa8f1af8764e31b7f728fc4db79d8e8d4836e59d396ccea
+size 2096573952

onnx/encoder_model_int8.onnx_data_1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a86b005d5c77583389f0f0dcf558ebe4ed615fad96f3473303eaaf7d9990a7ae
+size 774430720

onnx/encoder_model_q4.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4620d38ba212bca72ebef5148ec30ad37e102175b20661d03d819d86c0961cb6
-size 12238833
+oid sha256:e6e433e9587d761ccd47eb210f417ba0c315d2b80fd851348d07688d5bb0b993
+size 11897011

onnx/encoder_model_q4f16.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:96a3669a92755fde0f8642f1030e405ba11c33f7fdf70dd5d52b57a943688ec0
-size 11168545
+oid sha256:07fe72cabe3d23f59f79ffb12f5340e345db6cfec033e6070efe840a08a95030
+size 10826472