---
license: mit
library_name: unity-sentis
tags:
  - unity
  - sentis
  - on-device
  - automatic-speech-recognition
  - whisper
---

# sentis-whisper

Whisper speech-to-text converted to **Unity Inference Engine (Sentis)** FP16. Both `tiny` and `base`
sizes are included.

## Files

```
logmel_spectrogram_fp16.sentis        # audio -> log-mel front end
vocab.json                            # token vocabulary for de-tokenization
tiny/   base/
  encoder_model_fp16.sentis           # mel -> encoder_hidden_states
  decoder_model_fp16.sentis           # first pass: tokens (+ encoder states) -> logits + present KV
  decoder_with_past_model_fp16.sentis # subsequent steps: next token + past KV -> logits + new KV
WhisperRecognizer.cs                  # self-contained Unity Sentis inference
```

## Pipeline

This is a four-model greedy-decode pipeline (encoder KV is computed once; the decoder loop carries its
own self-attention KV so cost is linear in tokens):

1. **Log-mel** — audio (16 kHz mono) → log-mel spectrogram tensor.
2. **Encoder** — mel → `encoder_hidden_states` (run once per utterance).
3. **Decoder (prefill)** — `decoder_model` over the prompt tokens
   (`<|startoftranscript|>`, language, `<|transcribe|>`, `<|notimestamps|>`) → first-token logits +
   `present.*` KV.
4. **Decoder loop** — `decoder_with_past_model` with the last token + `past_key_values.*` → next-token
   logits + updated KV. Argmax each step (greedy), append, repeat until `<|endoftext|>`.
5. De-tokenize the collected ids with `vocab.json`.

A complete self-contained implementation lives in [`WhisperRecognizer.cs`](WhisperRecognizer.cs)
(log-mel → encoder → incremental KV greedy decode, language auto-detect + lock, LocalAgreement-2
streaming partials, byte-level BPE de-tokenization, GPU warmup). Minimal usage:

```csharp
var recognizer = new WhisperRecognizer(WhisperRecognizer.ModelSize.Tiny, BackendType.GPUCompute);
recognizer.Load(modelRoot); // folder holding vocab.json + logmel + tiny/ and base/ subfolders
recognizer.FinalTranscript += text => Debug.Log(text);
recognizer.BeginUtterance();
recognizer.PushSamples(samples, samples.Length, startSequence); // 16 kHz mono, monotonic sequence
recognizer.EndUtterance();
```

## License & attribution

MIT. Converted from [openai/whisper](https://github.com/openai/whisper) (MIT). The weights are
derivatives of the upstream Whisper checkpoints; see the upstream license for the original terms.