Zero-Shot Image Classification
OpenCLIP
ONNX
English
clip
vision
aerial
drone
tracking
re-identification
Instructions to use llama-farm/llama-thunderdome-clip-aerial-vit-b16-v3-drone-fleet with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- OpenCLIP
How to use llama-farm/llama-thunderdome-clip-aerial-vit-b16-v3-drone-fleet with OpenCLIP:
import open_clip model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('hf-hub:llama-farm/llama-thunderdome-clip-aerial-vit-b16-v3-drone-fleet') tokenizer = open_clip.get_tokenizer('hf-hub:llama-farm/llama-thunderdome-clip-aerial-vit-b16-v3-drone-fleet') - Notebooks
- Google Colab
- Kaggle
Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -123,3 +123,72 @@ backbone (matching across time/cameras) and v2 as the query encoder
|
|
| 123 |
|
| 124 |
Full session report:
|
| 125 |
`gs://thunderdome-tracking-lab/test-runs/CLIP_DATASET_REPORT.md`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 123 |
|
| 124 |
Full session report:
|
| 125 |
`gs://thunderdome-tracking-lab/test-runs/CLIP_DATASET_REPORT.md`
|
| 126 |
+
|
| 127 |
+
## Edge Export (Hailo-10H)
|
| 128 |
+
|
| 129 |
+
This release includes deployment artifacts for the full path from training to chip:
|
| 130 |
+
|
| 131 |
+
| Artifact | Size | Purpose |
|
| 132 |
+
|---|---|---|
|
| 133 |
+
| `best.pt` / `last.pt` | 598 MB | PyTorch fine-tuned weights (full precision) |
|
| 134 |
+
| `clip-aerial-vit-b16-v3-224.onnx` | 345 MB | Clean ONNX (x86 / ARM CPU / GPU runtime) |
|
| 135 |
+
| `clip-aerial-vit-b16-v3-surgical.onnx` | 344 MB | Hailo-DFC-compatible ONNX (weights grafted into HMZ ref ONNX) |
|
| 136 |
+
| `clip-aerial-vit-b16-v3-224-hailo10h.hef` | 79.7 MB | **Compiled HEF for Hailo-10H edge accelerator** |
|
| 137 |
+
| `quant_report.json` | — | INT8 quantization report from DFC compile |
|
| 138 |
+
| `validate_report.json` | — | PT-fp32 vs HEF (SDK_QUANTIZED) cosine + overlap@5 |
|
| 139 |
+
|
| 140 |
+
### HEF compilation pipeline
|
| 141 |
+
|
| 142 |
+
```bash
|
| 143 |
+
thunderdome clip onnx-surgery \
|
| 144 |
+
--our-pt best.pt \
|
| 145 |
+
--output clip-aerial-vit-b16-v3-surgical.onnx \
|
| 146 |
+
--ref-onnx /tmp/hailo_ref/clip_vit_b_16.onnx # ViT-B/16 surgery (DFC parser bug workaround)
|
| 147 |
+
|
| 148 |
+
thunderdome clip compile-hef \
|
| 149 |
+
--onnx clip-aerial-vit-b16-v3-surgical.onnx \
|
| 150 |
+
--alls configs/hailo/clip_vit_b_16_image_encoder_hailomz.alls \
|
| 151 |
+
--calib-dir <crops_dir> --calib-count 500 \
|
| 152 |
+
--target hailo10h --output-dir hef/ \
|
| 153 |
+
--net-name clip_aerial_vit_b_16 --imgsz 224
|
| 154 |
+
```
|
| 155 |
+
|
| 156 |
+
### Surgery + quantization fidelity
|
| 157 |
+
|
| 158 |
+
- Surgical ONNX vs PT (fp32 cosine, single pass): **0.9798**
|
| 159 |
+
- HEF emulator (SDK_QUANTIZED) vs PT (20-image mean cosine): **0.9551**
|
| 160 |
+
- HEF emulator vs PT overlap@5 (ranking agreement): **0.820**
|
| 161 |
+
|
| 162 |
+
The drop from 0.98 → 0.96 → 0.82 is INT8 quantization noise + the
|
| 163 |
+
documented QuickGELU↔GELU activation mismatch in Hailo's reference
|
| 164 |
+
ONNX. Retrieval rankings remain in agreement, which is what matters
|
| 165 |
+
for downstream ReID / open-vocab search.
|
| 166 |
+
|
| 167 |
+
### Runtime preprocessing
|
| 168 |
+
|
| 169 |
+
The HEF has the OpenAI CLIP normalization layer **baked into the
|
| 170 |
+
compiled graph** (via `norm1` in the alls config). On the chip,
|
| 171 |
+
pass **letterbox-padded raw uint8** images (114-gray fill, 224×224):
|
| 172 |
+
|
| 173 |
+
```python
|
| 174 |
+
from PIL import Image
|
| 175 |
+
import numpy as np
|
| 176 |
+
|
| 177 |
+
def letterbox(img, sz=224, fill=(114, 114, 114)):
|
| 178 |
+
w, h = img.size
|
| 179 |
+
s = sz / max(w, h)
|
| 180 |
+
nw, nh = int(round(w*s)), int(round(h*s))
|
| 181 |
+
img = img.resize((nw, nh), Image.BILINEAR)
|
| 182 |
+
canvas = Image.new("RGB", (sz, sz), fill)
|
| 183 |
+
canvas.paste(img, ((sz-nw)//2, (sz-nh)//2))
|
| 184 |
+
return canvas
|
| 185 |
+
|
| 186 |
+
# Pass to HEF as np.uint8 [1, 224, 224, 3] (NHWC).
|
| 187 |
+
```
|
| 188 |
+
|
| 189 |
+
### Architecture preserved
|
| 190 |
+
|
| 191 |
+
- Same `ViT-B-16` body as v2 (no quantization-aware training)
|
| 192 |
+
- 512-dim embedding (drop-in for arc-uas / any ViT-B/16 consumer)
|
| 193 |
+
- Text encoder frozen during fine-tune → text embeddings can come
|
| 194 |
+
from any standard openai-pretrained ViT-B/16
|