litert-community

/

whisper-tiny

like 2

Follow

LiteRT Community (FKA TFLite) 8.12k

This PR adds ACFT short-window variants of whisper-tiny: fixed 5 s / 10 s / 30 s window TFLite graphs converted from the audio-context fine-tuned checkpoint futo-org/acft-whisper-tiny (Apache-2.0), using the same encode/decode-split graph interface as the existing exports in this repo.

Why short windows

Whisper always encodes a 30 s window; for sub-5 s voice commands most encoder work is spent on padding. The encoder conv stack is length-agnostic and the encoder positional table is a fixed sinusoid, so slicing the first N rows of embed_positions yields a shorter window — exactly what whisper.cpp's audio_ctx parameter does at runtime. Stock Whisper decoders destabilize when the window shrinks (repetition loops: stock base at a 5 s window collapsed to CER 10.4 on a sentence clip in our evaluation); the futo-org ACFT (audio-context fine-tuning) checkpoints are fine-tuned via self-distillation (L2 on hidden states) to tolerate exactly this. Method and training: futo-org/whisper-acft — full credit to FUTO for the checkpoints; this PR only converts them (no retraining).

Files

Conversion: transformers TFWhisperForConditionalGeneration → two-signature graph (same pipeline as the 30 s exports in this repo), converter-time dynamic-range quantization (int8 weights, fp32 activations, tf.lite.Optimize.DEFAULT).

Tokenizer/vocab unchanged (51865) — the stock whisper-tiny tokenizer.json works as-is.

REQUIRED integration note

These are fixed short-window graphs. The encode signature input is [1, 80, frames] with frames = 500 / 1000 / 3000. The runtime must size the mel window from the encode signature's input shape (compute/pad the log-mel to exactly that frame count) instead of assuming 3000 frames. The decode signature input order of these exports is (mask, audio, tokens) — opposite of the stock export in this repo — so bind decode tensors by shape/name rather than position. Reference implementation: the JNI bridge in LiteRT-LM-Unity, which auto-detects 500/1000/3000-frame windows from the signature on one code path.

Validation

Desktop probe of these files (LiteRT interpreter, XNNPACK CPU, greedy decode; CER vs punctuation-normalized references, whitespace removed):

The short windows track the checkpoint's own 30 s-window behavior (same clips right/wrong at every window; the residual errors are tiny's model-capacity errors, present at 30 s too) at ~14x encoder speedup for the 5 s window (0.013–0.015 s vs 0.206–0.214 s per encode). In the same evaluation on whisper-base, the ACFT 5 s f32 graph reproduced the stock 30 s transcripts clip-for-clip at ~8x encoder speedup, ~20x with drq weights; short-window graphs are also device-validated on Android arm64 (Snapdragon 865, 0.5–0.8 s end-to-end per command clip for tiny/base 5 s).

Caveats

• The 5 s/10 s graphs hard-truncate longer audio — pick the window that matches your audio length; keep a 30 s graph for dictation.
• tiny retains its stock model-capacity limits on short noisy/quiet clips; base is the recommended minimum for non-English voice commands.

License: Apache-2.0 (this repo's license; the source ACFT checkpoint is also Apache-2.0).

Converted by the LiteRT-LM-Unity project (https://github.com/Leuconoe/LiteRT-LM-Unity).