Mirror of mlboydaisuke/Streaming-Sortformer-Diar-CoreAI

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+sortformer_float16.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+sortformer_float16.h18p.aimodelc/main-h18p-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/original_model_0.mpsgraph filter=lfs diff=lfs merge=lfs -text
+sortformer_float16.h18p.aimodelc/main-h18p-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/specialized_model_1.mpsgraph filter=lfs diff=lfs merge=lfs -text
+sortformer_mel_filters_128x257.f32 filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,59 @@
+---
+license: cc-by-4.0
+library_name: coreai
+pipeline_tag: voice-activity-detection
+base_model: nvidia/diar_streaming_sortformer_4spk-v2
+tags: [core-ai, coreaikit, sortformer, speaker-diarization, diarization, streaming, on-device, apple]
+---
+
+> **Mirror** of [`mlboydaisuke/Streaming-Sortformer-Diar-CoreAI`](https://huggingface.co/mlboydaisuke/Streaming-Sortformer-Diar-CoreAI) — the canonical repo ([CoreAI Model Zoo](https://github.com/john-rocky/coreai-model-zoo)). Updates land there first.
+
+
+# Streaming Sortformer 4-spk v2 — Core AI
+
+[`nvidia/diar_streaming_sortformer_4spk-v2`](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2)
+(cc-by-4.0, 117M) converted to **Apple Core AI** — streaming **speaker diarization** ("who spoke
+when", up to 4 speakers) running fully on-device via the
+[zoo](https://github.com/john-rocky/coreai-model-zoo). Only the neural core is a graph; the NeMo
+128-mel frontend, the streaming chunk loop, and the AOSC speaker-cache compression run in the Swift
+host — a 1:1 port of NeMo `sortformer_modules.py` (inference path).
+
+⚠️ Use the **streaming v2** checkpoint (cc-by-4.0). The offline `diar_sortformer_4spk-v1` is CC-BY-**NC**.
+
+## Files
+
+- `sortformer_float16.aimodel` — the static `forward_for_export` core, fp16 (~237 MB, macOS GPU).
+- `sortformer_float16.h18p.aimodelc` — the same graph AOT-compiled for iPhone (h18p, ~450 MB).
+- `sortformer_mel_filters_128x257.f32` — librosa-slaney mel filterbank (host log-mel frontend).
+- `metadata.json` — streaming params + the fixed-buffer graph contract.
+
+## Fixed-buffer graph contract
+
+```
+inputs:  chunk_mel [1,1520,128]   host zero-pads each mel chunk
+         spkcache  [1,188,512]     host-maintained speaker cache
+         valid     [1,378]         1 = real frame / 0 = pad (spkcache block [0:len], chunk block [188:188+pe_len])
+outputs: preds     [1,378,4]       sigmoid speaker activity
+         chunk_pe  [1,190,512]     pre-encode embeddings (host appends them to the speaker cache)
+```
+Host: NeMo 128-mel (preemph 0.97 → STFT n_fft=512/win=400/hop=160 → slaney mel → log, normalize=NA)
+→ chunk the mel (188·8 frames, ±1 subsample ctx) → run the graph → slice chunk preds →
+`streaming_update` + `compress_spkcache` (AOSC) → threshold 0.5/frame/speaker (frame = 80 ms) → turns.
+
+## Verification
+
+Byte-gated vs NeMo `forward_streaming` at **100.00 % speaker-activity agreement** (@0.5) on a 21.5 s
+and a 64.5 s clip (the latter exercises the AOSC cache compression ~4×), in Python, in Swift on
+**Mac GPU**, and on **iPhone 17 Pro** (A19 Pro, AOT h18p) — all driving this exported fp16 graph.
+
+## Use
+
+Ships in the **coreai-audio** app (Transcribe tab, "Diarize — who said what"): the diarizer segments
+each speaker turn, then the on-device ASR (Whisper / Qwen3-ASR / Parakeet / Nemotron) transcribes it
+into a **diarized transcript** — `Speaker 1 [0.3–4.1s]: …`. Speaker diarization already ships
+on-device elsewhere (e.g. CoreML/ANE); this is speed parity, offered as a diarized transcript wired to
+the zoo's own ASR. Conversion + Swift host loop: see
+[`conversion/sortformer_diar`](https://github.com/john-rocky/coreai-model-zoo/tree/main/conversion/sortformer_diar).
+
+Derived from NVIDIA's `diar_streaming_sortformer_4spk-v2` (CC-BY-4.0); this Core AI conversion is
+released under the same license.

metadata.json

@@ -0,0 +1,48 @@
+{
+  "model": "Streaming Sortformer 4-spk v2 (NVIDIA)",
+  "license": "cc-by-4.0",
+  "sample_rate": 16000,
+  "n_mels": 128,
+  "n_fft": 512,
+  "win_length": 400,
+  "hop": 160,
+  "preemph": 0.97,
+  "normalize": "NA",
+  "mag_power": 2.0,
+  "log_guard": 5.960464477539063e-08,
+  "pad_to": 16,
+  "subsampling_factor": 8,
+  "frame_hop_ms": 80,
+  "n_spk": 4,
+  "spkcache_len": 188,
+  "fifo_len": 0,
+  "chunk_len": 188,
+  "chunk_left_context": 1,
+  "chunk_right_context": 1,
+  "spkcache_update_period": 188,
+  "spkcache_sil_frames_per_spk": 3,
+  "scores_boost_latest": 0.05,
+  "sil_threshold": 0.2,
+  "pred_score_threshold": 0.25,
+  "strong_rate": 0.75,
+  "weak_rate": 1.5,
+  "min_pos_rate": 0.5,
+  "max_index": 99999,
+  "tf_max": 1520,
+  "spk": 188,
+  "pe_max": 190,
+  "t": 378,
+  "graph_io": "chunk_mel[1,1520,128],spkcache[1,188,512],valid[1,378] -> preds[1,378,4],chunk_pe[1,190,512]",
+  "golden": {
+    "mel_shape": [
+      128,
+      2160
+    ],
+    "mel_len": 2150,
+    "total_preds_shape": [
+      270,
+      4
+    ]
+  },
+  "note": "host: NeMo 128-mel (preemph->stft512/win400/hop160->slaney mel->log, normalize=NA) -> feat_chunks (188*8, L/R ctx 1) -> per chunk zero-pad mel to 1520, build spkcache(188)+valid, run graph, slice chunk preds, streaming_update + AOSC compress_spkcache; threshold 0.5/frame/spk (frame=80ms) -> speaker turns."
+}

sortformer_float16.aimodel/main.hash

@@ -0,0 +1 @@
+U&:2-��Kl�����Sx��\���^3�Ƈ

sortformer_float16.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:55263a17322def93ef1e4b6cddd510aefde3895378f3d55cfceacb5e33b3c687
+size 236655368

sortformer_float16.aimodel/metadata.json

@@ -0,0 +1,4 @@
+{
+  "assetVersion" : "2.0",
+  "license" : "cc-by-4.0"
+}

sortformer_float16.h18p.aimodelc/main-h18p-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/original_model_0.mpsgraph

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ba1201c93a7fad90c4b66613b065c367b705e4e07d2f6ae97c20724912bacf9
+size 388870

sortformer_float16.h18p.aimodelc/main-h18p-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/resources.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f1f7e8d2af18e3628c29b8013a1f425d2e608035360469e39f361d406599f2a7
+size 470978768

sortformer_float16.h18p.aimodelc/main-h18p-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/specialized_model_1.mpsgraph

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a2837dd6e8f9da9aaa165ae4b096e9fc2258e710dba2b56367c9fc57617cb4d
+size 628062

sortformer_float16.h18p.aimodelc/main.hash

@@ -0,0 +1 @@
+Ԧ',��!s�{�$������rv�ny@��V5

sortformer_float16.h18p.aimodelc/metadata.json

@@ -0,0 +1,7 @@
+{
+  "producer" : "coreai-build-3600.67.5.8.1",
+  "sourceHash" : "55263A17322DEF93EF1E4B6CDDD510AEFDE3895378F3D55CFCEACB5E33B3C687",
+  "creationDate" : "20260706T035105Z",
+  "assetVersion" : "2.0",
+  "license" : "cc-by-4.0"
+}

sortformer_float16.h18p.aimodelc/stats.json

@@ -0,0 +1 @@
+{"computeTypes":["Float16","Float32","Float64","Int32","UInt32","UInt64"],"operationDistribution":[{"name":"constant","count":4708},{"count":1039,"name":"reshape"},{"name":"broadcast_in_dims","count":730},{"count":608,"name":"add"},{"count":569,"name":"broadcasting_add"},{"count":492,"name":"transpose"},{"name":"batch_matmul","count":352},{"name":"broadcasting_batch_matmul","count":352},{"name":"slice","count":280},{"count":220,"name":"broadcast_to"},{"count":213,"name":"concat"},{"count":162,"name":"broadcasting_divide"},{"count":162,"name":"divide"},{"name":"invoke","count":138},{"name":"mul","count":135},{"name":"broadcasting_mul","count":131},{"name":"exp","count":104},{"count":89,"name":"cast"},{"name":"broadcasting_sub","count":74},{"count":74,"name":"sub"},{"count":56,"name":"conv2d"},{"name":"silu","count":51},{"count":43,"name":"reduce"},{"count":39,"name":"reduce_sum"},{"name":"pad","count":37},{"name":"softmax","count":35},{"count":23,"name":"broadcasting_maximum"},{"name":"maximum","count":23},{"name":"relu","count":23},{"count":18,"name":"sigmoid"},{"count":4,"name":"broadcast_shapes"},{"name":"gather_nd","count":4},{"count":4,"name":"get_shape"},{"count":4,"name":"reduce_mean"},{"count":4,"name":"reduce_product"},{"count":2,"name":"rsqrt"},{"count":1,"name":"sqrt"}],"storageTypes":[{"name":"Float16","count":118096523},{"name":"Int32","count":4245},{"count":2302,"name":"UInt32"},{"name":"Float64","count":39},{"count":36,"name":"Float32"},{"name":"UInt64","count":4}]}

sortformer_mel_filters_128x257.f32

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bce5ec5f194a5913f6508cee5a85512e7bad2352db8fc28f5c6ff75af8b09137
+size 131584