Mask Generation
LiteRT
LiteRT
sam2
segment-anything
mask-decoder
interactive-segmentation
on-device
gpu
Instructions to use litert-community/SAM2.1-Hiera-Tiny-Mask-Decoder with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT
How to use litert-community/SAM2.1-Hiera-Tiny-Mask-Decoder with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- sam2
How to use litert-community/SAM2.1-Hiera-Tiny-Mask-Decoder with sam2:
# Use SAM2 with images import torch from sam2.sam2_image_predictor import SAM2ImagePredictor predictor = SAM2ImagePredictor.from_pretrained(litert-community/SAM2.1-Hiera-Tiny-Mask-Decoder) with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16): predictor.set_image(<your_image>) masks, _, _ = predictor.predict(<input_prompts>)# Use SAM2 with videos import torch from sam2.sam2_video_predictor import SAM2VideoPredictor predictor = SAM2VideoPredictor.from_pretrained(litert-community/SAM2.1-Hiera-Tiny-Mask-Decoder) with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16): state = predictor.init_state(<your_video>) # add new prompts and instantly get the output on the same frame frame_idx, object_ids, masks = predictor.add_new_points(state, <your_prompts>): # propagate the prompts to get masklets throughout the video for frame_idx, object_ids, masks in predictor.propagate_in_video(state): ... - Notebooks
- Google Colab
- Kaggle
Add minimal usage snippets and hero image
Browse files
README.md
CHANGED
|
@@ -20,6 +20,8 @@ On-device **LiteRT / TFLite** conversion of the **prompt-conditioned mask decode
|
|
| 20 |
[**SAM 2.1 Hiera-Tiny**](https://huggingface.co/facebook/sam2.1-hiera-tiny) (Meta, Apache-2.0),
|
| 21 |
running **fully on the mobile GPU** via the LiteRT `CompiledModel` API (ML Drift / `LITERT_CL` delegate).
|
| 22 |
|
|
|
|
|
|
|
| 23 |
This is the lightweight, per-click half of the SAM 2 image path. Pair it with the
|
| 24 |
[**SAM 2.1 Hiera-Tiny image encoder**](https://huggingface.co/litert-community/SAM2.1-Hiera-Tiny-Image-Encoder)
|
| 25 |
(run **once** per image, ~7 ms): the encoder produces the multi-scale feature pyramid, and this decoder
|
|
@@ -102,15 +104,54 @@ Eager re-authoring is **numerically exact** (`cos = 1.000`). End-to-end through
|
|
| 102 |
The deepest `64×64` image embedding drifts slightly on the GPU (true-fp16 deep attention; see the encoder
|
| 103 |
card). Mask boundaries are carried by the near-exact high-resolution features, so mask quality holds.
|
| 104 |
|
| 105 |
-
##
|
|
|
|
|
|
|
| 106 |
|
| 107 |
```kotlin
|
| 108 |
-
// once per image
|
| 109 |
-
val enc = CompiledModel.create(assets, "
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 114 |
```
|
| 115 |
|
| 116 |
## Training data & PII
|
|
|
|
| 20 |
[**SAM 2.1 Hiera-Tiny**](https://huggingface.co/facebook/sam2.1-hiera-tiny) (Meta, Apache-2.0),
|
| 21 |
running **fully on the mobile GPU** via the LiteRT `CompiledModel` API (ML Drift / `LITERT_CL` delegate).
|
| 22 |
|
| 23 |
+

|
| 24 |
+
|
| 25 |
This is the lightweight, per-click half of the SAM 2 image path. Pair it with the
|
| 26 |
[**SAM 2.1 Hiera-Tiny image encoder**](https://huggingface.co/litert-community/SAM2.1-Hiera-Tiny-Image-Encoder)
|
| 27 |
(run **once** per image, ~7 ms): the encoder produces the multi-scale feature pyramid, and this decoder
|
|
|
|
| 104 |
The deepest `64×64` image embedding drifts slightly on the GPU (true-fp16 deep attention; see the encoder
|
| 105 |
card). Mask boundaries are carried by the near-exact high-resolution features, so mask quality holds.
|
| 106 |
|
| 107 |
+
## Minimal usage
|
| 108 |
+
|
| 109 |
+
**Android (Kotlin, CompiledModel GPU)**
|
| 110 |
|
| 111 |
```kotlin
|
| 112 |
+
// once per image - encoder on GPU (decoder-ready v2 variant from the companion repo)
|
| 113 |
+
val enc = CompiledModel.create(context.assets, "sam2_tiny_image_encoder_v2_fp16.tflite",
|
| 114 |
+
CompiledModel.Options(Accelerator.GPU), null)
|
| 115 |
+
// per tap - decoder on CPU (GPU-resident but fp16-incorrect on device; see the note above)
|
| 116 |
+
val dec = CompiledModel.create(context.assets, "sam2_tiny_mask_decoder_fp16.tflite",
|
| 117 |
+
CompiledModel.Options(Accelerator.CPU), null)
|
| 118 |
+
// dec inputs: 0 image_embeddings[1,256,64,64], 1 sparse[1,2,256],
|
| 119 |
+
// 2 feat_s1[1,64,128,128], 3 feat_s0[1,32,256,256]
|
| 120 |
+
// dec outputs: pred_masks[1,3,256,256] logits, iou_scores[1,3] -> argmax(iou), threshold 0
|
| 121 |
+
```
|
| 122 |
+
|
| 123 |
+
**Python (desktop verification)**
|
| 124 |
+
|
| 125 |
+
```python
|
| 126 |
+
MEAN = np.array([0.485, 0.456, 0.406], np.float32)
|
| 127 |
+
STD = np.array([0.229, 0.224, 0.225], np.float32)
|
| 128 |
+
import numpy as np
|
| 129 |
+
from PIL import Image
|
| 130 |
+
from ai_edge_litert.interpreter import Interpreter
|
| 131 |
+
|
| 132 |
+
# 1) encode once (decoder-ready v2 encoder from the companion encoder repo)
|
| 133 |
+
img = Image.open("photo.jpg").convert("RGB").resize((1024, 1024))
|
| 134 |
+
x = ((np.asarray(img, np.float32) / 255 - MEAN) / STD).transpose(2, 0, 1)[None]
|
| 135 |
+
enc = Interpreter(model_path="sam2_tiny_image_encoder_v2_fp16.tflite"); enc.allocate_tensors()
|
| 136 |
+
enc.set_tensor(enc.get_input_details()[0]["index"], x); enc.invoke()
|
| 137 |
+
eo = {tuple(d["shape"]): enc.get_tensor(d["index"]) for d in enc.get_output_details()}
|
| 138 |
+
|
| 139 |
+
# 2) host prompt-encode one positive tap (px, py) in 1024-space (constants: this repo)
|
| 140 |
+
px, py = 512, 384
|
| 141 |
+
posmat, pe1, nap = np.split(np.fromfile("prompt_encode_const.bin", np.float32), [256, 512])
|
| 142 |
+
coord = 2 * np.pi * ((2 * (np.array([px, py], np.float32) + 0.5) / 1024 - 1) @ posmat.reshape(2, 128))
|
| 143 |
+
tok0 = np.concatenate([np.sin(coord), np.cos(coord)]) + pe1
|
| 144 |
+
sparse = np.stack([tok0, nap])[None].astype(np.float32) # [1,2,256]
|
| 145 |
+
|
| 146 |
+
# 3) decode masks (CPU)
|
| 147 |
+
dec = Interpreter(model_path="sam2_tiny_mask_decoder_fp16.tflite"); dec.allocate_tensors()
|
| 148 |
+
feed = {(1,2,256): sparse}; feed.update(eo) # match inputs by shape
|
| 149 |
+
for d in dec.get_input_details(): dec.set_tensor(d["index"], feed[tuple(d["shape"])])
|
| 150 |
+
dec.invoke()
|
| 151 |
+
o = {len(d["shape"]): dec.get_tensor(d["index"]) for d in dec.get_output_details()}
|
| 152 |
+
masks, iou = o[4], o[2] # [1,3,256,256], [1,3]
|
| 153 |
+
best = masks[0, iou[0].argmax()] > 0 # [256,256] binary mask
|
| 154 |
+
Image.fromarray(best.astype(np.uint8) * 255).resize(Image.open("photo.jpg").size).save("mask.png")
|
| 155 |
```
|
| 156 |
|
| 157 |
## Training data & PII
|