rgthelen commited on
Commit
2522c09
·
verified ·
1 Parent(s): cbcafde

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +69 -0
README.md CHANGED
@@ -123,3 +123,72 @@ backbone (matching across time/cameras) and v2 as the query encoder
123
 
124
  Full session report:
125
  `gs://thunderdome-tracking-lab/test-runs/CLIP_DATASET_REPORT.md`
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
123
 
124
  Full session report:
125
  `gs://thunderdome-tracking-lab/test-runs/CLIP_DATASET_REPORT.md`
126
+
127
+ ## Edge Export (Hailo-10H)
128
+
129
+ This release includes deployment artifacts for the full path from training to chip:
130
+
131
+ | Artifact | Size | Purpose |
132
+ |---|---|---|
133
+ | `best.pt` / `last.pt` | 598 MB | PyTorch fine-tuned weights (full precision) |
134
+ | `clip-aerial-vit-b16-v3-224.onnx` | 345 MB | Clean ONNX (x86 / ARM CPU / GPU runtime) |
135
+ | `clip-aerial-vit-b16-v3-surgical.onnx` | 344 MB | Hailo-DFC-compatible ONNX (weights grafted into HMZ ref ONNX) |
136
+ | `clip-aerial-vit-b16-v3-224-hailo10h.hef` | 79.7 MB | **Compiled HEF for Hailo-10H edge accelerator** |
137
+ | `quant_report.json` | — | INT8 quantization report from DFC compile |
138
+ | `validate_report.json` | — | PT-fp32 vs HEF (SDK_QUANTIZED) cosine + overlap@5 |
139
+
140
+ ### HEF compilation pipeline
141
+
142
+ ```bash
143
+ thunderdome clip onnx-surgery \
144
+ --our-pt best.pt \
145
+ --output clip-aerial-vit-b16-v3-surgical.onnx \
146
+ --ref-onnx /tmp/hailo_ref/clip_vit_b_16.onnx # ViT-B/16 surgery (DFC parser bug workaround)
147
+
148
+ thunderdome clip compile-hef \
149
+ --onnx clip-aerial-vit-b16-v3-surgical.onnx \
150
+ --alls configs/hailo/clip_vit_b_16_image_encoder_hailomz.alls \
151
+ --calib-dir <crops_dir> --calib-count 500 \
152
+ --target hailo10h --output-dir hef/ \
153
+ --net-name clip_aerial_vit_b_16 --imgsz 224
154
+ ```
155
+
156
+ ### Surgery + quantization fidelity
157
+
158
+ - Surgical ONNX vs PT (fp32 cosine, single pass): **0.9798**
159
+ - HEF emulator (SDK_QUANTIZED) vs PT (20-image mean cosine): **0.9551**
160
+ - HEF emulator vs PT overlap@5 (ranking agreement): **0.820**
161
+
162
+ The drop from 0.98 → 0.96 → 0.82 is INT8 quantization noise + the
163
+ documented QuickGELU↔GELU activation mismatch in Hailo's reference
164
+ ONNX. Retrieval rankings remain in agreement, which is what matters
165
+ for downstream ReID / open-vocab search.
166
+
167
+ ### Runtime preprocessing
168
+
169
+ The HEF has the OpenAI CLIP normalization layer **baked into the
170
+ compiled graph** (via `norm1` in the alls config). On the chip,
171
+ pass **letterbox-padded raw uint8** images (114-gray fill, 224×224):
172
+
173
+ ```python
174
+ from PIL import Image
175
+ import numpy as np
176
+
177
+ def letterbox(img, sz=224, fill=(114, 114, 114)):
178
+ w, h = img.size
179
+ s = sz / max(w, h)
180
+ nw, nh = int(round(w*s)), int(round(h*s))
181
+ img = img.resize((nw, nh), Image.BILINEAR)
182
+ canvas = Image.new("RGB", (sz, sz), fill)
183
+ canvas.paste(img, ((sz-nw)//2, (sz-nh)//2))
184
+ return canvas
185
+
186
+ # Pass to HEF as np.uint8 [1, 224, 224, 3] (NHWC).
187
+ ```
188
+
189
+ ### Architecture preserved
190
+
191
+ - Same `ViT-B-16` body as v2 (no quantization-aware training)
192
+ - 512-dim embedding (drop-in for arc-uas / any ViT-B/16 consumer)
193
+ - Text encoder frozen during fine-tune → text embeddings can come
194
+ from any standard openai-pretrained ViT-B/16