README.md

---
license: apache-2.0
language:
- en
tags:
- audio
- keyword-spotting
- wake-word
- kws
- zipformer
- transducer
- coreml
- on-device
base_model: pkufool/keyword-spotting-models
library_name: coreml
pipeline_tag: audio-classification
---

# KWS Zipformer 3M — CoreML INT8

Streaming, zero-shot, open-vocabulary keyword spotting for iOS / macOS / visionOS. Exported from icefall's KWS-finetuned Zipformer transducer (gigaspeech, 3.49M parameters) to CoreML with INT8 palettized weights, FP16 compute, and an iOS 17+ minimum deployment target.

Given an arbitrary list of English keywords at runtime (no retraining required), the model emits a match when it hears one. Runs ~26× real-time on Apple Silicon CPU + Neural Engine.

## Model

| | |
|---|---|
| Architecture | Zipformer2 encoder + stateless transducer decoder + joiner |
| Parameters | 3.49M |
| Quantization | INT8 k-means palettization (encoder + joiner); decoder FP16 |
| Compute precision | FP16 |
| Format | `.mlmodelc` (pre-compiled, ship-ready) |
| Min deployment target | iOS 17 / macOS 14 / visionOS 1 |
| Sample rate | 16 kHz |
| Feature | 80-dim Kaldi fbank, 25 / 10 ms |
| Chunk size | 320 ms (16 output frames × 20 ms each) |
| Left context | 64 subsampled frames (~2.5 s) |
| Vocab | 500 BPE tokens |

## Files

| file | size | description |
|---|---|---|
| `encoder.mlmodelc` | 3.3 MB | Zipformer2 streaming encoder (45 × 80 mel → 8 × 320 joiner-space) with 36 layer cache tensors + ConvNeXt pad + processed-lens state |
| `decoder.mlmodelc` | 525 KB | Stateless 2-token-context predictor + `decoder_proj` |
| `joiner.mlmodelc` | 160 KB | `output_linear(tanh(enc + dec))` → 500 logits |
| `bpe.model` | 239 KB | SentencePiece BPE-500 tokenizer (icefall gigaspeech) |
| `tokens.txt` | 4.9 KB | Token id → subword map |
| `commands_small.txt` | 0.2 KB | Example keyword list (20 short commands) |
| `commands_large.txt` | 5.9 KB | Example keyword list (248 commands) |
| `config.json` | 3.8 KB | Fbank params, encoder cache shapes, default KWS thresholds |

## Performance

Measured on Apple Silicon (CPU + Neural Engine), tuned defaults `ac_threshold=0.15 context_score=0.5 num_trailing_blanks=1`.

### Latency (per call)

| component | latency |
|---|---|
| Encoder (45 × 80 mel → 8 × 320 joiner-space) | 5.4 ms / 320 ms chunk |
| Decoder (1 step, 2-token context) | 0.08 ms |
| Joiner (1 frame → 500 logits) | 0.13 ms |
| **RTF** (encoder only, streaming) | **0.038** (~26× real-time) |

### Accuracy (LibriSpeech test-clean, 12 keywords, 158 positive + 60 negative utterances)

| keyword | N | recall |
|---|---|---|
| WHICH | 27 | 1.00 |
| LITTLE | 17 | 0.94 |
| BEFORE | 18 | 1.00 |
| GREAT | 19 | 0.95 |
| LIGHT | 15 | 1.00 |
| YOUNG | 15 | 1.00 |
| THROUGH | 15 | 0.93 |
| WORLD | 15 | 0.93 |
| ALWAYS | 15 | 0.93 |
| PEOPLE | 15 | 1.00 |
| THOUGHT | 15 | 0.87 |
| MISTER | 17 | 0.00 |
| **TOTAL** | **203** | **0.88** |

False positive rate: 0.27 / utterance on 60 random negative utterances. CoreML INT8 output agrees with the PyTorch FP32 reference on 99% of utterances (FP16 palettization drift is not systematic).

> **Note on "MISTER"**: SentencePiece tokenizes it as `[▁MI, S, TER]` (3 tokens). Stateless transducers rarely lock onto 3-token sequences in beam search. For production wake words, prefer single-token keywords.

### Memory

Loaded all three `.mlmodelc` models: peak RSS delta ≈ **63 MB** on macOS. After 218-utterance streaming workload: +127 MB total.

## Usage

### Python (coremltools)

```python
import json
from pathlib import Path

import coremltools as ct
import numpy as np

model_dir = Path("./KWS-Zipformer-3M-CoreML-INT8")
config = json.loads((model_dir / "config.json").read_text())

encoder = ct.models.CompiledMLModel(str(model_dir / "encoder.mlmodelc"))
decoder = ct.models.CompiledMLModel(str(model_dir / "decoder.mlmodelc"))
joiner = ct.models.CompiledMLModel(str(model_dir / "joiner.mlmodelc"))

# Build zero state from config
state = {}
for name, shape in zip(config["encoder"]["layerStateNames"],
                       config["encoder"]["layerStateShapes"]):
    state[name] = np.zeros(shape, dtype=np.float32)
state["cached_embed_left_pad"] = np.zeros(
    config["encoder"]["cachedEmbedLeftPadShape"], dtype=np.float32)
state["processed_lens"] = np.zeros((1,), dtype=np.int32)

# Feed 45 mel frames (80-dim fbank) per chunk
x = np.zeros((1, 45, 80), dtype=np.float32)  # replace with real fbank
out = encoder.predict({"x": x, **state})
encoder_out = out["encoder_out"]                  # (1, 8, 320) in joiner space
# Stream: feed encoder_out[0, t] into decoder/joiner + beam search
```

### Swift (speech-swift)

```swift
import SpeechSwift

let model = try await KWSZipformerModel.fromPretrained(
    "aufklarer/KWS-Zipformer-3M-CoreML-INT8"
)
let stream = model.streamingSession(keywords: ["HEY SONIQO", "STOP", "LIGHTS ON"])
try stream.feed(audio: pcm16k)
for match in stream.emissions {
    print("matched: \(match.phrase) at \(match.timestamp)")
}
```

### Reference Python decoder

The upstream Aho-Corasick ContextGraph + boost-cancellation beam search algorithm is available as a dependency-free pure-Python reference for porting to other runtimes.

## Source

Exported from **icefall-kws-zipformer-gigaspeech-20240219** (Apache-2.0), specifically the `exp-finetune/pretrained.pt` KWS-finetuned checkpoint, published in [pkufool/keyword-spotting-models v0.11](https://github.com/pkufool/keyword-spotting-models/releases/tag/v0.11).

The reference ONNX export is [`csukuangfj/sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01`](https://github.com/k2-fsa/sherpa-onnx/releases/download/kws-models/sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01.tar.bz2); this CoreML bundle matches its encoder / decoder / joiner outputs within FP16 tolerance (≤1e-3 on encoder, ≤1e-4 on decoder, ≤1e-3 on joiner).

## Links

- [speech-swift](https://github.com/soniqo/speech-swift) — Apple SDK
- [soniqo.audio](https://soniqo.audio) — website
- [blog](https://soniqo.audio/blog)