mlboydaisuke commited on
Commit
e058a71
·
verified ·
1 Parent(s): 8542eec

Add minimal usage snippets and hero image

Browse files
Files changed (1) hide show
  1. README.md +48 -7
README.md CHANGED
@@ -20,6 +20,8 @@ On-device **LiteRT / TFLite** conversion of the **prompt-conditioned mask decode
20
  [**SAM 2.1 Hiera-Tiny**](https://huggingface.co/facebook/sam2.1-hiera-tiny) (Meta, Apache-2.0),
21
  running **fully on the mobile GPU** via the LiteRT `CompiledModel` API (ML Drift / `LITERT_CL` delegate).
22
 
 
 
23
  This is the lightweight, per-click half of the SAM 2 image path. Pair it with the
24
  [**SAM 2.1 Hiera-Tiny image encoder**](https://huggingface.co/litert-community/SAM2.1-Hiera-Tiny-Image-Encoder)
25
  (run **once** per image, ~7 ms): the encoder produces the multi-scale feature pyramid, and this decoder
@@ -102,15 +104,54 @@ Eager re-authoring is **numerically exact** (`cos = 1.000`). End-to-end through
102
  The deepest `64×64` image embedding drifts slightly on the GPU (true-fp16 deep attention; see the encoder
103
  card). Mask boundaries are carried by the near-exact high-resolution features, so mask quality holds.
104
 
105
- ## Usage (Android / LiteRT CompiledModel)
 
 
106
 
107
  ```kotlin
108
- // once per image encoder on GPU
109
- val enc = CompiledModel.create(assets, "sam2_image_encoder_v2_fp16.tflite", Options(Accelerator.GPU), null)
110
- // per tap — decoder on CPU (GPU-resident but fp16-incorrect on device; see the residency note above)
111
- val dec = CompiledModel.create(assets, "sam2_mask_decoder_fp16.tflite", Options(Accelerator.CPU), null)
112
- // dec inputs (by index): 0 image_embeddings[1,256,64,64], 1 sparse[1,2,256], 2 feat_s1[1,64,128,128], 3 feat_s0[1,32,256,256]
113
- // dec outputs: pred_masks[1,3,256,256] logits, iou_scores[1,3] -> pick argmax(iou), upsample, threshold 0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
114
  ```
115
 
116
  ## Training data & PII
 
20
  [**SAM 2.1 Hiera-Tiny**](https://huggingface.co/facebook/sam2.1-hiera-tiny) (Meta, Apache-2.0),
21
  running **fully on the mobile GPU** via the LiteRT `CompiledModel` API (ML Drift / `LITERT_CL` delegate).
22
 
23
+ ![SAM 2.1 tap-to-segment running on-device (LiteRT)](demo.gif)
24
+
25
  This is the lightweight, per-click half of the SAM 2 image path. Pair it with the
26
  [**SAM 2.1 Hiera-Tiny image encoder**](https://huggingface.co/litert-community/SAM2.1-Hiera-Tiny-Image-Encoder)
27
  (run **once** per image, ~7 ms): the encoder produces the multi-scale feature pyramid, and this decoder
 
104
  The deepest `64×64` image embedding drifts slightly on the GPU (true-fp16 deep attention; see the encoder
105
  card). Mask boundaries are carried by the near-exact high-resolution features, so mask quality holds.
106
 
107
+ ## Minimal usage
108
+
109
+ **Android (Kotlin, CompiledModel GPU)**
110
 
111
  ```kotlin
112
+ // once per image - encoder on GPU (decoder-ready v2 variant from the companion repo)
113
+ val enc = CompiledModel.create(context.assets, "sam2_tiny_image_encoder_v2_fp16.tflite",
114
+ CompiledModel.Options(Accelerator.GPU), null)
115
+ // per tap - decoder on CPU (GPU-resident but fp16-incorrect on device; see the note above)
116
+ val dec = CompiledModel.create(context.assets, "sam2_tiny_mask_decoder_fp16.tflite",
117
+ CompiledModel.Options(Accelerator.CPU), null)
118
+ // dec inputs: 0 image_embeddings[1,256,64,64], 1 sparse[1,2,256],
119
+ // 2 feat_s1[1,64,128,128], 3 feat_s0[1,32,256,256]
120
+ // dec outputs: pred_masks[1,3,256,256] logits, iou_scores[1,3] -> argmax(iou), threshold 0
121
+ ```
122
+
123
+ **Python (desktop verification)**
124
+
125
+ ```python
126
+ MEAN = np.array([0.485, 0.456, 0.406], np.float32)
127
+ STD = np.array([0.229, 0.224, 0.225], np.float32)
128
+ import numpy as np
129
+ from PIL import Image
130
+ from ai_edge_litert.interpreter import Interpreter
131
+
132
+ # 1) encode once (decoder-ready v2 encoder from the companion encoder repo)
133
+ img = Image.open("photo.jpg").convert("RGB").resize((1024, 1024))
134
+ x = ((np.asarray(img, np.float32) / 255 - MEAN) / STD).transpose(2, 0, 1)[None]
135
+ enc = Interpreter(model_path="sam2_tiny_image_encoder_v2_fp16.tflite"); enc.allocate_tensors()
136
+ enc.set_tensor(enc.get_input_details()[0]["index"], x); enc.invoke()
137
+ eo = {tuple(d["shape"]): enc.get_tensor(d["index"]) for d in enc.get_output_details()}
138
+
139
+ # 2) host prompt-encode one positive tap (px, py) in 1024-space (constants: this repo)
140
+ px, py = 512, 384
141
+ posmat, pe1, nap = np.split(np.fromfile("prompt_encode_const.bin", np.float32), [256, 512])
142
+ coord = 2 * np.pi * ((2 * (np.array([px, py], np.float32) + 0.5) / 1024 - 1) @ posmat.reshape(2, 128))
143
+ tok0 = np.concatenate([np.sin(coord), np.cos(coord)]) + pe1
144
+ sparse = np.stack([tok0, nap])[None].astype(np.float32) # [1,2,256]
145
+
146
+ # 3) decode masks (CPU)
147
+ dec = Interpreter(model_path="sam2_tiny_mask_decoder_fp16.tflite"); dec.allocate_tensors()
148
+ feed = {(1,2,256): sparse}; feed.update(eo) # match inputs by shape
149
+ for d in dec.get_input_details(): dec.set_tensor(d["index"], feed[tuple(d["shape"])])
150
+ dec.invoke()
151
+ o = {len(d["shape"]): dec.get_tensor(d["index"]) for d in dec.get_output_details()}
152
+ masks, iou = o[4], o[2] # [1,3,256,256], [1,3]
153
+ best = masks[0, iou[0].argmax()] > 0 # [256,256] binary mask
154
+ Image.fromarray(best.astype(np.uint8) * 255).resize(Image.open("photo.jpg").size).save("mask.png")
155
  ```
156
 
157
  ## Training data & PII