File size: 6,373 Bytes
49fd202 c2ceda0 49fd202 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 | ---
license: apache-2.0
library_name: litert
pipeline_tag: video-classification
tags:
- litert
- tflite
- android
- on-device
- gpu
- video-action-recognition
- kinetics-600
- movinet
- streaming
---
# MoViNet-A0 Stream — LiteRT (on-device video action recognition, GPU)
On-device **streaming video action recognition**: recognises human actions across a
stream of camera frames — one frame at a time, constant memory, real-time — running
**fully on the LiteRT `CompiledModel` GPU delegate** (no CPU fallback).
- **Architecture:** [MoViNet-A0](https://arxiv.org/abs/2103.11511) streaming variant
(Google Research) — a causal 2+1D CNN.
- **Task:** [Kinetics-600](https://github.com/cvdfoundation/kinetics-dataset) — 600 action classes.
- **Weights:** ported PyTorch checkpoint from [Atze00/MoViNet-pytorch](https://github.com/Atze00/MoViNet-pytorch).
- **Size:** 15 MB · ~3.75 M params · input frame `172×172`.
## How the streaming graph works
MoViNet's temporal convolutions and global-average-pools each keep a small buffer of
the recent past, so the network can be fed **one frame at a time** and its prediction
sharpens as more frames of the same action arrive. The stock streaming graph carries
that history in **5D** state tensors `[1, T, H, W, C]`, which a GPU delegate cannot
compile (all tensors must be ≤ 4D). This model is re-authored as a **single-frame,
4D-only functional forward** (**47 inputs / 28 outputs**) with the recurrent state
threaded explicitly through the graph I/O:
| I/O slot | count | shape | meaning |
|-----------------|-------|-----------------|-------------------------------------------|
| `input[0]` | 1 | `[1,3,172,172]` | current RGB frame (NCHW, 0..1) |
| `input[1..28]` | 28 | `[1,C,H,W]` | temporal-conv stream buffers (11 convs) |
| `input[29..44]` | 16 | `[1,C,1,1]` | streaming avg-pool running sums (15 SE + head) |
| `input[45]` | 1 | `[1,1,1,1]` | `inv_count` = 1 / current frame number |
| `input[46]` | 1 | `[1,1,1,1]` | constant `1.0` (Mali output decoupler) |
| `output[0]` | 1 | `[1,600]` | Kinetics-600 logits |
| `output[1..11]` | 11 | `[1,C,H,W]` | current per-temporal-conv frame |
| `output[12..27]`| 16 | `[1,C,1,1]` | fresh per-frame spatial means |
The **stream-buffer shift register and pool running-sum accumulation are done
host-side**: each frame you run once, shift each stream buffer (drop oldest, append
the emitted current frame), accumulate `running_sum += emitted_mean`, and feed both
back as inputs. The converted graph is **all float32, 0 tensors of rank > 4, 0
GPU-incompatible ops** and matches the original PyTorch model bit-for-bit
(correlation 0.99999999999, top-5 identical; device GPU on a Pixel 8a locks onto
"jumping jacks" within a few frames). Keeping the state in-graph tripped three silent
Mali `CompiledModel` bugs, which is why the state plumbing is host-side.
## Minimal usage
### Python (LiteRT / ai-edge-litert, frame-by-frame)
```python
from ai_edge_litert.interpreter import Interpreter
import numpy as np
it = Interpreter(model_path="movinet_a0_stream.tflite"); it.allocate_tensors()
inp, out = it.get_input_details(), it.get_output_details()
DIMS = [2, 2, 2, 4, 2, 2, 4, 2, 2, 2, 4] # temporal-conv buffer depths
offs, o = [], 0
for d in DIMS: offs.append(o); o += d
hist = [[np.zeros(inp[1 + offs[c] + i]["shape"], np.float32) for i in range(DIMS[c])]
for c in range(11)] # host-side shift registers
psum = [np.zeros(inp[29 + i]["shape"], np.float32) for i in range(16)] # running sums
for n, frame in enumerate(video_frames, start=1): # frame: [1,3,172,172], RGB, 0..1
it.set_tensor(inp[0]["index"], frame.astype(np.float32))
for c in range(11):
for i in range(DIMS[c]): it.set_tensor(inp[1 + offs[c] + i]["index"], hist[c][i])
for i in range(16): it.set_tensor(inp[29 + i]["index"], psum[i])
it.set_tensor(inp[45]["index"], np.full((1, 1, 1, 1), 1.0 / n, np.float32)) # inv_count
it.set_tensor(inp[46]["index"], np.ones((1, 1, 1, 1), np.float32)) # decoupler
it.invoke()
logits = it.get_tensor(out[0]["index"])[0] # [600]
for c in range(11): # shift: drop oldest, append current
hist[c] = hist[c][1:] + [it.get_tensor(out[1 + c]["index"]).copy()]
for i in range(16): # accumulate running sum
psum[i] = psum[i] + it.get_tensor(out[12 + i]["index"])
print("top-1:", int(logits.argmax()))
```
### Kotlin (Android, LiteRT CompiledModel GPU)
```kotlin
val options = CompiledModel.Options(Accelerator.GPU)
val model = CompiledModel.create(context.assets, "movinet_a0_stream.tflite", options, null)
val inBufs = model.createInputBuffers() // [0]=frame, [1..28]=stream, [29..44]=pool sums, [45]=inv_count, [46]=1.0
val outBufs = model.createOutputBuffers() // [0]=logits, [1..11]=current frames, [12..27]=fresh means
inBufs[46].writeFloat(floatArrayOf(1f)) // constant decoupler
// reset recurrent state (zeros) at the start of a clip; keep host-side stream buffers + pool sums
for ((n, frameNCHW) in videoFrames.withIndex()) { // frame: [1,3,172,172], RGB, 0..1
inBufs[0].writeFloat(frameNCHW)
inBufs[45].writeFloat(floatArrayOf(1f / (n + 1))) // inv_count
// (stream inputs 1..28 and pool inputs 29..44 already staged from the previous frame)
model.run(inBufs, outBufs)
val logits = outBufs[0].readFloat() // [600] Kinetics-600
// host-side: shift each stream buffer with the emitted current frame (outBufs[1..11]),
// and accumulate poolSum[i] += outBufs[12+i], then write both back to inBufs for the next frame.
}
```
A full implementation (camera → per-frame → top-5, with the host-side shift register and pool
accumulation) is in the sample app's `ActionRecognizer.kt`.
## Conversion
Re-authored and converted with **litert-torch**. See the sample app and build script:
`build_movinet.py` + `stream_model.py`.
## License
Apache-2.0 (MoViNet / Atze00/MoViNet-pytorch). Kinetics-600 label taxonomy from the
DeepMind Kinetics dataset.
|