Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,107 @@
+---
+license: apache-2.0
+library_name: litert
+pipeline_tag: video-classification
+tags:
+  - litert
+  - tflite
+  - android
+  - on-device
+  - gpu
+  - video-action-recognition
+  - kinetics-600
+  - movinet
+  - streaming
+---
+
+# MoViNet-A0 Stream — LiteRT (on-device video action recognition, GPU)
+
+On-device **streaming video action recognition**: recognises human actions across a
+stream of camera frames — one frame at a time, constant memory, real-time — running
+**fully on the LiteRT `CompiledModel` GPU delegate** (no CPU fallback).
+
+- **Architecture:** [MoViNet-A0](https://arxiv.org/abs/2103.11511) streaming variant
+  (Google Research) — a causal 2+1D CNN.
+- **Task:** [Kinetics-600](https://github.com/cvdfoundation/kinetics-dataset) — 600 action classes.
+- **Weights:** ported PyTorch checkpoint from [Atze00/MoViNet-pytorch](https://github.com/Atze00/MoViNet-pytorch).
+- **Size:** 15 MB · ~3.75 M params · input frame `172×172`.
+
+![MoViNet-A0 streaming action recognition](hero.png)
+
+## How the streaming graph works
+
+MoViNet's temporal convolutions and global-average-pools each keep a small buffer of
+the recent past, so the network can be fed **one frame at a time** and its prediction
+sharpens as more frames of the same action arrive. The stock streaming graph carries
+that history in **5D** state tensors `[1, T, H, W, C]`, which a GPU delegate cannot
+compile (all tensors must be ≤ 4D). This model is re-authored as a **single-frame,
+4D-only functional forward** (**47 inputs / 28 outputs**) with the recurrent state
+threaded explicitly through the graph I/O:
+
+| I/O slot        | count | shape           | meaning                                   |
+|-----------------|-------|-----------------|-------------------------------------------|
+| `input[0]`      | 1     | `[1,3,172,172]` | current RGB frame (NCHW, 0..1)            |
+| `input[1..28]`  | 28    | `[1,C,H,W]`     | temporal-conv stream buffers (11 convs)   |
+| `input[29..44]` | 16    | `[1,C,1,1]`     | streaming avg-pool running sums (15 SE + head) |
+| `input[45]`     | 1     | `[1,1,1,1]`     | `inv_count` = 1 / current frame number    |
+| `input[46]`     | 1     | `[1,1,1,1]`     | constant `1.0` (Mali output decoupler)    |
+| `output[0]`     | 1     | `[1,600]`       | Kinetics-600 logits                       |
+| `output[1..11]` | 11    | `[1,C,H,W]`     | current per-temporal-conv frame           |
+| `output[12..27]`| 16    | `[1,C,1,1]`     | fresh per-frame spatial means             |
+
+The **stream-buffer shift register and pool running-sum accumulation are done
+host-side**: each frame you run once, shift each stream buffer (drop oldest, append
+the emitted current frame), accumulate `running_sum += emitted_mean`, and feed both
+back as inputs. The converted graph is **all float32, 0 tensors of rank > 4, 0
+GPU-incompatible ops** and matches the original PyTorch model bit-for-bit
+(correlation 0.99999999999, top-5 identical; device GPU on a Pixel 8a locks onto
+"jumping jacks" within a few frames). Keeping the state in-graph tripped three silent
+Mali `CompiledModel` bugs, which is why the state plumbing is host-side.
+
+## Minimal usage
+
+### Python (LiteRT / ai-edge-litert, frame-by-frame)
+
+```python
+from ai_edge_litert.interpreter import Interpreter
+import numpy as np
+
+it = Interpreter(model_path="movinet_a0_stream.tflite"); it.allocate_tensors()
+inp, out = it.get_input_details(), it.get_output_details()
+
+DIMS = [2, 2, 2, 4, 2, 2, 4, 2, 2, 2, 4]        # temporal-conv buffer depths
+offs, o = [], 0
+for d in DIMS: offs.append(o); o += d
+hist = [[np.zeros(inp[1 + offs[c] + i]["shape"], np.float32) for i in range(DIMS[c])]
+        for c in range(11)]                      # host-side shift registers
+psum = [np.zeros(inp[29 + i]["shape"], np.float32) for i in range(16)]  # running sums
+
+for n, frame in enumerate(video_frames, start=1):   # frame: [1,3,172,172], RGB, 0..1
+    it.set_tensor(inp[0]["index"], frame.astype(np.float32))
+    for c in range(11):
+        for i in range(DIMS[c]): it.set_tensor(inp[1 + offs[c] + i]["index"], hist[c][i])
+    for i in range(16): it.set_tensor(inp[29 + i]["index"], psum[i])
+    it.set_tensor(inp[45]["index"], np.full((1, 1, 1, 1), 1.0 / n, np.float32))  # inv_count
+    it.set_tensor(inp[46]["index"], np.ones((1, 1, 1, 1), np.float32))           # decoupler
+    it.invoke()
+    logits = it.get_tensor(out[0]["index"])[0]      # [600]
+    for c in range(11):                             # shift: drop oldest, append current
+        hist[c] = hist[c][1:] + [it.get_tensor(out[1 + c]["index"]).copy()]
+    for i in range(16):                             # accumulate running sum
+        psum[i] = psum[i] + it.get_tensor(out[12 + i]["index"])
+
+print("top-1:", int(logits.argmax()))
+```
+
+A full Kotlin implementation (camera → per-frame → top-5) is in the sample app's
+`ActionRecognizer.kt`.
+
+## Conversion
+
+Re-authored and converted with **litert-torch**. See the sample app and build script:
+`build_movinet.py` + `stream_model.py`.
+
+## License
+
+Apache-2.0 (MoViNet / Atze00/MoViNet-pytorch). Kinetics-600 label taxonomy from the
+DeepMind Kinetics dataset.