File size: 3,127 Bytes
7fd63ba | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 | ---
license: mit
library_name: litert
pipeline_tag: image-segmentation
tags:
- litert
- tflite
- android
- on-device
- gpu
- instance-segmentation
- yolact
- coco
- real-time
---
# YOLACT-ResNet50 — LiteRT (real-time instance segmentation, GPU)
On-device **real-time instance segmentation** running **fully on the LiteRT
`CompiledModel` GPU** delegate (no CPU fallback). [YOLACT](https://arxiv.org/abs/1904.02689)
(ICCV 2019) predicts per-instance COCO masks. The network (ResNet50 + FPN +
protonet + heads) runs on the GPU; the lightweight decode (NMS + linear-combination
masks) runs host-side. ~41 ms/graph on a Pixel 8a.
- **Architecture:** YOLACT-ResNet50 (base, no deformable conv) — pure CNN.
- **Weights:** [dbolya/yolact](https://github.com/dbolya/yolact) (`yolact_resnet50_54_800000`) · MIT.
- **Size:** 125 MB.
## Files
- `yolact.tflite` — the GPU graph (input `[1,3,550,550]` NCHW).
- `priors.bin` — 19248 SSD priors `[cx,cy,w,h]` (float32) used by the host-side box decode.
## I/O
- **Input:** `[1, 3, 550, 550]` NCHW, **BGR**, normalized `(x - [103.94,116.78,123.68]) / [57.38,57.12,58.40]`
(no /255).
- **Raw outputs:** `loc [1,19248,4]`, `conf [1,19248,81]` (softmax, incl. background),
`mask [1,19248,32]` (coefficients), `proto [1,138,138,32]` (prototype masks).
## Host-side decode
1. **Boxes:** SSD `decode(loc, priors, variances=[0.1,0.2])`.
2. **NMS:** per-class, score-threshold ~0.3, IoU 0.5, top-k.
3. **Masks (lincomb):** for each kept detection, `mask = sigmoid(proto @ coeff)` → crop
to the box → threshold 0.5 → upscale.
## GPU conversion
Base YOLACT is a pure CNN, so the graph converts fully GPU-compatible (**138/138
nodes on the delegate, 1 partition**; device corr 0.99999–1.0 vs PyTorch on all four
raw outputs) with **one patch**: the ResNet50 stem `MaxPool2d(padding=1)` lowers to a
`-inf` PADV2 (rejected by Mali), replaced by a 0-pad + unpadded maxpool (exact
post-ReLU). The scripted FPN is made traceable by disabling YOLACT's JIT
(`use_jit=False`). CPU-exact vs PyTorch (corr 1.0).
## Minimal usage (Python, LiteRT / ai-edge-litert)
```python
import numpy as np
from ai_edge_litert.interpreter import Interpreter
it = Interpreter(model_path="yolact.tflite"); it.allocate_tensors()
inp, out = it.get_input_details(), it.get_output_details()
it.set_tensor(inp[0]["index"], x) # [1,3,550,550] BGR, normalized (see above)
it.invoke()
outs = {tuple(o["shape"][1:]): it.get_tensor(o["index"])[0] for o in out}
loc = outs[(19248, 4)]; conf = outs[(19248, 81)]
mask = outs[(19248, 32)]; proto = outs[(138, 138, 32)]
priors = np.fromfile("priors.bin", np.float32).reshape(-1, 4)
cxy = priors[:, :2] + loc[:, :2] * 0.1 * priors[:, 2:]
wh = priors[:, 2:] * np.exp(loc[:, 2:] * 0.2)
boxes = np.concatenate([cxy - wh / 2, cxy + wh / 2], 1) # x1y1x2y2 (0..1)
# then per-class NMS on conf, and mask_i = sigmoid(proto @ mask[i]) cropped to boxes[i]
```
A full Android decode is in the sample app (`YolactSegmenter.kt`).
## License
MIT (YOLACT / dbolya/yolact). COCO class taxonomy.
|