litert-community

/

whisper-base

like 2

Follow

LiteRT Community (FKA TFLite) 8.12k

This PR adds ACFT short-window variants of whisper-base: fixed 5 s / 10 s / 30 s window TFLite graphs converted from the audio-context fine-tuned checkpoint futo-org/acft-whisper-base (Apache-2.0), using the same encode/decode-split graph interface as the existing exports in this repo.

Why short windows

Whisper always encodes a 30 s window; for sub-5 s voice commands most encoder work is spent on padding. The encoder conv stack is length-agnostic and the encoder positional table is a fixed sinusoid, so slicing the first N rows of embed_positions yields a shorter window — exactly what whisper.cpp's audio_ctx parameter does at runtime. Stock Whisper decoders destabilize when the window shrinks (repetition loops: stock whisper-base at a 5 s window collapsed to CER 10.4 on a sentence clip in our evaluation); the futo-org ACFT (audio-context fine-tuning) checkpoints are fine-tuned via self-distillation (L2 on hidden states) to tolerate exactly this. Method and training: futo-org/whisper-acft — full credit to FUTO for the checkpoints; this PR only converts them (no retraining).

Files

Conversion: transformers TFWhisperForConditionalGeneration → two-signature graph (same pipeline as the 30 s exports in this repo), converter-time dynamic-range quantization (int8 weights, fp32 activations, tf.lite.Optimize.DEFAULT).

Tokenizer/vocab unchanged (51865) — the stock whisper-base tokenizer.json works as-is.

REQUIRED integration note

These are fixed short-window graphs. The encode signature input is [1, 80, frames] with frames = 500 / 1000 / 3000. The runtime must size the mel window from the encode signature's input shape (compute/pad the log-mel to exactly that frame count) instead of assuming 3000 frames. The decode signature input order of these exports is (mask, audio, tokens) — opposite of the stock export in this repo — so bind decode tensors by shape/name rather than position. Reference implementation: the JNI bridge in LiteRT-LM-Unity, which auto-detects 500/1000/3000-frame windows from the signature on one code path.

Validation

Desktop evaluation (LiteRT interpreter, XNNPACK CPU, greedy decode, 7 Korean/English clips, CER vs punctuation-normalized references):

• The ACFT 5 s f32 graph reproduces the stock 30 s export's transcripts clip-for-clip (identical CER per clip) at ~8x encoder speedup (0.076–0.084 s vs 0.62–0.65 s per encode).
• The drq (int8) tier pushes encode to 0.03 s (20x vs stock 30 s f32) with unchanged transcripts on the probe set.
• Stock whisper-base at the same 5 s window (no ACFT) collapses on sentence-length clips (repetition loop, CER 10.4 on one clip) — the failure mode ACFT fixes.

Desktop probe of these drq files:

(The single 5 s/10 s miss — "and soul" on the English clip — is the stock whisper-base 30 s f32 export's own transcript for that clip; the 5 s window is ~12x faster per encode than the 30 s window.)

On device (Snapdragon 865, arm64 CPU, LiteRT, fresh process incl. compile): the 5 s drq graph reads a 3.8 s Korean report sentence exact in 1.33 s total and short commands in ~0.76 s total (encode 0.047–0.06 s — ~12x the deployed stock base-30s encoder on the same device; ~3.5x faster end-to-end).

Caveats

• The 5 s/10 s graphs hard-truncate longer audio — pick the window that matches your audio length; keep a 30 s graph for dictation.
• Very quiet sub-second clips remain sensitive (capture-side AGC/loudness normalization recommended) — a base-model capacity trait, unchanged by ACFT.

License: Apache-2.0 (this repo's license; the source ACFT checkpoint is also Apache-2.0).

Converted by the LiteRT-LM-Unity project (https://github.com/Leuconoe/LiteRT-LM-Unity).