---
license: cc-by-4.0
language:
  - en
tags:
  - speaker-diarization
  - coreml
  - apple-silicon
  - neural-engine
  - sortformer
  - streaming
datasets:
  - voxconverse
base_model: nvidia/diar_streaming_sortformer_4spk-v2.1
library_name: coreml
pipeline_tag: audio-classification
---

# Sortformer Diarization (CoreML)

CoreML port of [NVIDIA Sortformer](https://arxiv.org/abs/2409.06656) for end-to-end speaker diarization on Apple Silicon. Runs on the Neural Engine. No separate embedding extraction or clustering — the model directly predicts per-frame speaker activity for up to 4 speakers, with streaming state (speaker cache + FIFO) carried across calls.

## Model

| | |
|---|---|
| Architecture | Sortformer (Sort Loss + 17-layer FastConformer + 18-layer Transformer) |
| Base model | [`nvidia/diar_streaming_sortformer_4spk-v2.1`](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1) (117M params) |
| Precision | FP16 compute, FP32 I/O boundaries |
| Sample rate | 16 kHz, 128 mel bins (n_fft 400, hop 160) |
| Max speakers | 4 (arrival-order slots) |
| Speaker cache / FIFO | 188 / 40 encoder frames |
| Frame duration | 80 ms per prediction frame |

Three variants ship from the same checkpoint, differing only in chunk shape:

| Variant | New audio per call | Mel input | Use case |
|---|---|---|---|
| `Sortformer` (default) | ~27 s | `[1, 3048, 128]` | offline batch, highest throughput |
| `Sortformer_balanced` | ~8 s | `[1, 968, 128]` | ~3× faster first-load compile |
| `Sortformer_streaming` | 480 ms | `[1, 112, 128]` | incremental / realtime |

## Files

| File | Size | Description |
|---|---|---|
| `Sortformer.mlmodelc` / `.mlpackage` | ~230 MB | default variant (compiled / source) |
| `Sortformer_balanced.mlmodelc` / `.mlpackage` | ~230 MB | balanced variant |
| `Sortformer_streaming.mlmodelc` / `.mlpackage` | ~230 MB | streaming variant |
| `config.json`, `config_balanced.json`, `config_streaming.json` | <1 KB | chunk-shape hyperparameters per variant |

## Performance

Measured on M-series Apple Silicon (Neural Engine, warm):

| Variant | Per-call latency | Realtime factor |
|---|---|---|
| default | one call per ~27 s of audio | ~125–750× |
| balanced | one call per ~8 s | hundreds of × |
| streaming | **8.1 ms median per 480 ms step** | ~59× per step; ~39× end-to-end incremental |

The streaming export is parity-gated: driven by NeMo's own streaming feature
loader and cache-update algorithm, it matches NeMo's native
`forward_streaming` loop at **100% frame-decision agreement (MAE 0.0005)**.
On a five-file VoxConverse-dev pilot, the incremental session reaches 8.1%
DER (collar 0.25) with correct speaker counts on all files.

## Streaming interface

The streaming variant carries state through the CoreML interface; the
speaker-cache update runs host-side between calls.

**Inputs:** `chunk [1,112,128]`, `chunk_lengths [1]`, `spkcache [1,188,512]`,
`spkcache_lengths [1]`, `fifo [1,40,512]`, `fifo_lengths [1]`

**Outputs:** `speaker_preds_out [1,242,4]`,
`chunk_pre_encoder_embs_out [1,14,512]`, `chunk_pre_encoder_lengths_out [1]`

The pipeline contains two sub-models: **PreEncoder** (mel pre-encode +
state concat) and **Head** (FastConformer + Transformer + sigmoid heads).

## Usage

```swift
// Whole-buffer diarization
let diarizer = try await SortformerDiarizer.fromPretrained()
let result = diarizer.diarize(audio: samples, sampleRate: 16000)

// Incremental streaming: push PCM in any size, stable speaker slots
let session = try await SortformerStreamingSession.fromPretrained()
let snapshot = try session.push(audio: samples)
let final = try session.finish()
```

```bash
speech diarize meeting.wav --engine sortformer
```

## Source

- Upstream model: [nvidia/diar_streaming_sortformer_4spk-v2.1](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1) · [Sortformer paper](https://arxiv.org/abs/2409.06656) · [Streaming Sortformer paper](https://arxiv.org/abs/2507.18446)
- Upstream license: NVIDIA Open Model License (weights); this conversion is published under CC-BY-4.0 with attribution to NVIDIA.

## Links

- [speech-swift](https://github.com/soniqo/speech-swift) — Apple SDK
- [Docs](https://soniqo.audio/getting-started) — install and CLI docs
- [Guide](https://soniqo.audio/guides/diarize) — speaker diarization guide
- [soniqo.audio](https://soniqo.audio) — website
- [blog](https://soniqo.audio/blog) — blog