Add files using upload-large-folder tool

README.md

@@ -26,53 +26,67 @@ base_model:
 - CohereLabs/cohere-transcribe-03-2026
 ---
 
-# Cohere Transcribe 03-2026 — DirectML-safe ONNX
+# Cohere Transcribe 03-2026 — DirectML-safe, decode-optimized ONNX
 
 ONNX export of [CohereLabs/cohere-transcribe-03-2026](https://huggingface.co/CohereLabs/cohere-transcribe-03-2026)
-with the **decoder attention decomposed to plain ONNX primitives**, so the model runs **correctly and
-fast on the ONNX Runtime DirectML EP** (and every other EP). Weights are byte-identical to
+restructured so the model runs **correctly and fast on the ONNX Runtime DirectML EP** (and every other
+EP). Weight bytes are identical to
 [onnx-community/cohere-transcribe-03-2026-ONNX](https://huggingface.co/onnx-community/cohere-transcribe-03-2026-ONNX);
-only the decoder graph protos differ.
+only the graph protos differ (the encoder proto additionally embeds the ~67 MB of cross-attention
+projection weights hoisted out of the decoder — see below).
 
-## Why
+## Why the stock export can't run on DirectML
 
 The onnx-community decoders fuse attention into two `com.microsoft` contrib ops that the DirectML EP
 mishandles (ORT ≤ 1.24, unfixed — the DML EP is in maintenance mode):
 
 - **`MultiHeadAttention`** (cross-attention, 3-D query + 4-D precomputed K/V): the DML kernel routes
   the 4-D K/V into its PastKey/PastValue slots and builds a `DML_MULTIHEAD_ATTENTION_OPERATOR_DESC`
-  with null Key/Value tensors, which DirectML rejects → `E_INVALIDARG` crash at
-  `/model/layers.0/encoder_attn/MultiHeadAttention`.
+  with null Key/Value tensors, which DirectML rejects → `E_INVALIDARG` crash.
 - **`GroupQueryAttention`** (self-attention): the DML kernel **silently ignores the `attention_bias`
-  input** and never reads `past_key/past_value` (it assumes shared max-length past/present buffers),
-  so a concat-style KV cache decodes garbage with no error.
-- Both are gated by two `If` nodes; control-flow ops have no DML kernel and always run on CPU, forcing
-  a GPU↔CPU round-trip of all 16 cross-KV tensors **every decoded token**.
+  input** and never reads `past_key/past_value` (it assumes shared max-length buffers), so a
+  concat-style KV cache decodes garbage with no error.
+- Both are gated by two `If` nodes; control-flow ops have no DML kernel (CPU-only), forcing a
+  GPU↔CPU round-trip of all 16 cross-KV tensors every decoded token.
 
 ## What changed
 
-Decoder-only graph rewrite (`decoder_model_merged*.onnx`, all precisions):
+1. **Attention decomposed** (`decoder_model_merged*.onnx`, all precisions): every
+   `MultiHeadAttention` / `GroupQueryAttention` node rewritten to plain
+   Reshape/Transpose/MatMul/Mul/Add/Softmax/Concat with an explicit causal mask (GQA applies it
+   internally; the exported `attention_bias` input is only the padding mask). The two cross-attn
+   `If` nodes are flattened away.
+2. **Loop-invariant cross-KV hoisted into the encoder**: the cross-attention K/V projections (and
+   the pre-transposed keys) depend only on the encoder output, so they now run ONCE per utterance
+   inside `encoder_model*.onnx` (extra `cross_attn.*` outputs) instead of on EVERY decoded token
+   inside the decoder (which cost ~100× the decoder's real per-token compute). The decoder consumes
+   them as same-named `cross_attn.*` inputs; `present.*.encoder.*` outputs are gone. Decoder proto
+   metadata carries `winstt_hoisted_cross_kv=1`.
 
-1. The two cross-attn `If` nodes are flattened: cross-attention K/V are recomputed from
-   `encoder_hidden_states` every step (branchless), and echoed as `present.*.encoder.*`.
-2. Every `MultiHeadAttention` / `GroupQueryAttention` node is decomposed into
-   Reshape/Transpose/MatMul/Mul/Add/Softmax/Concat, with an explicit causal mask (GQA applies it
-   internally; the exported `attention_bias` input is only the padding mask).
+Verified numerically against the fused graphs on CPU (autoregressive logits parity: bit-exact for
+fp32/int8/q4; fp16-rounding-level for fp16/q4f16).
 
-Verified numerically identical to the fused graphs on CPU (autoregressive logits parity: bit-exact for
-fp32/int8/q4; fp16-rounding-level for fp16/q4f16). Encoders are untouched (their self-attention MHA
-form runs correctly on DirectML).
+Measured on a 66 s clip (fp32, ORT 1.24, RTX 3080 Ti, identical transcripts):
 
-Measured on a 66 s clip (fp32, ORT 1.24 DirectML vs CPU EP, same transcripts):
-fused decoder = **crash on DML** / 21 s CPU · this export = **14 s DML** / 29 s CPU.
+| export | DirectML | CPU |
+|---|---|---|
+| fused (onnx-community) | crash | 21–33 s |
+| decomposed only | 25.2 s | 20.8 s |
+| decomposed + branchless | 14.0 s | 28.7 s |
+| **this repo (hoisted)** | **2.7 s**¹ | ~7.4 s² |
 
-Produced by [WinSTT](https://github.com/dahshury/WinSTT)'s
-`tools/onnx/cohere_decompose_attention.py`.
+¹ hybrid placement: encoder on DirectML, decoder on the CPU EP (the DML EP re-fuses its graph on
+every autoregressive shape change, ~34 ms/token fixed; the CPU EP decodes the hoisted decoder at
+~12 ms/token). This is how [WinSTT](https://github.com/dahshury/WinSTT) runs it.
+² all-CPU: encoder 6.1 s + decode 1.3 s.
+
+Produced by WinSTT's `tools/onnx/cohere_decompose_attention.py`.
 
 ## Files
 
-- `onnx/encoder_model[_fp16|_int8|_q4|_q4f16|_quantized].onnx` (+ external data) — unchanged from
-  onnx-community (int8 encoder: `quantize_dynamic` of the fp32, from the previous revision of this repo).
-- `onnx/decoder_model_merged[_fp16|_int8|_q4|_q4f16|_quantized].onnx` (+ external data) — DirectML-safe
-  rewrites as above.
+- `onnx/encoder_model[_fp16|_int8|_q4|_q4f16|_quantized].onnx` (+ external data) — onnx-community
+  weights + the hoisted cross-KV tail (extra `cross_attn.*` outputs; projection weights embedded in
+  the proto).
+- `onnx/decoder_model_merged[_fp16|_int8|_q4|_q4f16|_quantized].onnx` (+ external data) —
+  decomposed, branchless, cross-KV-free decoders (consume `cross_attn.*` inputs).
 - Tokenizer / configs — unchanged.

onnx/decoder_model_merged.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:71fc3a0f4a7baa707e9aaf20120ddacd454033f5bcc807795f28989b7538b68a
-size 140697
+oid sha256:9d2bc6c61625ad2e96641e76a7056440e0f70db17b7982371215ce4317385e8e
+size 126143

onnx/decoder_model_merged_fp16.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:60f7f8f458c30bb2b1ee35d3adad588107c84e2c9cbedfc4ece580ec145a79b2
-size 142475
+oid sha256:84746ed4964cd92913159df8a29c49609ccdcf5f6db5fa1ab3f1a623a88b322a
+size 127116

onnx/decoder_model_merged_int8.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c3e625649c7feb1f1facb6bf3e243dc49e9e69f656fc417bee45fc8b1aeab423
-size 221311
+oid sha256:daecf1938f01ff7adcc2611123cccc26a4a06615dbef80519be88b1d12094ee2
+size 206757

onnx/decoder_model_merged_q4.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0cf74a4dc7c284c0fffef0644aa0ca42cc5a4c741749d7bc8675548e1b0cbf6b
-size 180313
+oid sha256:165e4be4ee266782ff8baae525d42c14ef54587caa05c38720010d527972a2d4
+size 162847

onnx/decoder_model_merged_q4f16.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cab0f156cb48de1f0754c8f6bc8fdd2fd8da913839eb9ca3cfda76aa82ccbaed
-size 182114
+oid sha256:48c802f4d0b82056d8764628e21a58f6159b589e9f48dd6ef44d94e1a8dcadab
+size 163843

onnx/decoder_model_merged_quantized.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0e8240480882a161820ad1bd99c49f8546e3ad077cf3653a08a5c4b7b5b032e4
-size 183253
+oid sha256:03487477a68eac454194afd14aacc520169807048abc288e32295ef0915d8b99
+size 165787

onnx/encoder_model.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:503c1100e62d26fe25864621df54d7e440f07f4aef698836c503bbdcbe024686
-size 1181690
+oid sha256:328ed95c4e99a022fc2bff2784b67c43d5513ebbc056120489e58c52d342e7fb
+size 68375619

onnx/encoder_model_fp16.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:155046e82c022bb37a43ca13c0d7837bc0fd046b152d578d8312fc1e10567e8a
-size 1192583
+oid sha256:b85245dd273d1df75fddd8098a16067cf73595785664ac19629beac40118863c
+size 34800172

onnx/encoder_model_int8.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8568fde562bbfba56028c886690c8a443087abbe2f87dc355a3323b2de86bb05
-size 2074856
+oid sha256:e74076d7ff0f33df1b8f00761fa522a4ad5e2b41466448d9abf4ba7e0aebbdf6
+size 69268785

onnx/encoder_model_q4.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:de0f7e2c5f4c2e46e3704704a3cb41153ed45f5af07530b4b1d34f895c36db4b
-size 1400792
+oid sha256:4620d38ba212bca72ebef5148ec30ad37e102175b20661d03d819d86c0961cb6
+size 12238833

onnx/encoder_model_q4f16.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:81c3369197348c87f28048fcd0cc3fa286b4f63cd950fa52c9756f2ea1c6eefa
-size 1410988
+oid sha256:96a3669a92755fde0f8642f1030e405ba11c33f7fdf70dd5d52b57a943688ec0
+size 11168545

onnx/encoder_model_quantized.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9e7e59a14821e5a08c71b0d849a956b6375007a5f0be795c448499b0328d7923
-size 1420203
+oid sha256:d13b2ea09d118bee13b4e5fdf0a130be0bd6a949a52d948c9a7a3a920c5e8b9b
+size 20908996