---
language:
  - en
license: apache-2.0
library_name: open_clip
tags:
  - clip
  - vision
  - aerial
  - drone
  - tracking
  - re-identification
pipeline_tag: zero-shot-image-classification
base_model: openai/clip-vit-base-patch16
---

# llama-thunderdome-clip-aerial-vit-b16-v3-drone-fleet

A drone-fleet-specialized fine-tune of OpenCLIP **ViT-B/16** for
aerial vehicle + person retrieval and re-identification.

## What this is

`v3-drone-fleet` is the **third** member of the
`llama-thunderdome-clip-aerial-*` family. It complements `v2`:

* `v2` — fine-tuned with **subtype-aware captions** (sedan/SUV/pickup/
  school_bus/etc) on curated aerial imagery. Best for **specific
  vehicle subtype** queries.
* **`v3-drone-fleet`** (this model) — fine-tuned on **real drone-fleet
  footage** (2 drones, 12 videos, 4,575 Gemini-captioned crops). Best
  for **drone-fleet domain adaptation** — improves discrimination on
  the actual camera profile + altitude of operational drones.

## Training data

| Source | Crops | Captioning |
|---|---:|---|
| Real drone footage (2 drone profiles, 12 videos) | 4,575 | Gemini-2.5-flash-lite verdict + subtype + color/clothing |

Class distribution: person 3,130 · car 1,089 · truck 356.

Caption style: `overhead drone view of <color> <subtype>` for
vehicles, `person in <clothing_color> clothing` for persons.

## Training recipe

```
base_model:    ViT-B-16
pretrained:    openai
text_encoder:  frozen
epochs:        15  (best at epoch 4)
batch_size:    64
learning_rate: 1e-5
warmup:        100 steps
val_split:     0.1
```

## Eval results

| Metric | Baseline (openai) | v3 best (epoch 4) | Delta |
|---|---:|---:|---:|
| `val_loss` | 2.83 | **2.74** | -3.2% |
| `val_recall@1` (image→text) | 0.169 | **0.212** | +25.4% (rel) |

## How to use

### As a regular open_clip model

```python
import open_clip
from huggingface_hub import hf_hub_download
import torch

ckpt_path = hf_hub_download(
    repo_id="llama-farm/llama-thunderdome-clip-aerial-vit-b16-v3-drone-fleet", filename="best.pt"
)
model, _, preprocess = open_clip.create_model_and_transforms(
    "ViT-B-16", pretrained="openai"
)
state = torch.load(ckpt_path, map_location="cpu")
model.load_state_dict(state["model"] if "model" in state else state)
model.eval()
```

### As a ReID backbone inside `llama-thunderdome` (BotSORT / DeepOcSort)

```python
from thunderdome.tracking.clip_reid_adapter import CLIPReidAdapter
adapter = CLIPReidAdapter(
    checkpoint="best.pt", base_model="ViT-B-16"
)
features = adapter.get_features(xyxys, img)   # [N, 512] L2-normalized
```

`thunderdome tracking-lab` auto-detects CLIP checkpoints and routes
them through the adapter inside the `TrackerWrapper` — no custom code
needed at the operator level.

## When to pick v3 vs v2

* **v3 (this model)** -> re-identification across video frames from
  YOUR drones. Better at "is this the same person/vehicle 5 seconds
  later?" because it's tuned to the specific camera profile.
* **v2** -> subtype queries ("a yellow pickup truck", "a school bus").
  Better at semantic disambiguation between vehicle classes.

For most drone tracking workloads, **use both** — v3 as the ReID
backbone (matching across time/cameras) and v2 as the query encoder
(matching text -> object).

## Limitations

* Crops smaller than ~32x32 px (people-as-dots at altitude > 80m) do
  not produce discriminative embeddings — all map to similar vectors.
  Spatial-temporal clustering via `thunderdome tracking-lab
  count-unique` is more appropriate at that scale.
* Trained on 2 drone profiles. Cross-fleet generalization unverified.

## Generated

2026-05-26T14:36:33.904905 via the llama-thunderdome agent loop.

Full session report:
`gs://thunderdome-tracking-lab/test-runs/CLIP_DATASET_REPORT.md`

## Edge Export (Hailo-10H)

This release includes deployment artifacts for the full path from training to chip:

| Artifact | Size | Purpose |
|---|---|---|
| `best.pt` / `last.pt` | 598 MB | PyTorch fine-tuned weights (full precision) |
| `clip-aerial-vit-b16-v3-224.onnx` | 345 MB | Clean ONNX (x86 / ARM CPU / GPU runtime) |
| `clip-aerial-vit-b16-v3-surgical.onnx` | 344 MB | Hailo-DFC-compatible ONNX (weights grafted into HMZ ref ONNX) |
| `clip-aerial-vit-b16-v3-224-hailo10h.hef` | 79.7 MB | **Compiled HEF for Hailo-10H edge accelerator** |
| `quant_report.json` | — | INT8 quantization report from DFC compile |
| `validate_report.json` | — | PT-fp32 vs HEF (SDK_QUANTIZED) cosine + overlap@5 |

### HEF compilation pipeline

```bash
thunderdome clip onnx-surgery \
    --our-pt best.pt \
    --output clip-aerial-vit-b16-v3-surgical.onnx \
    --ref-onnx /tmp/hailo_ref/clip_vit_b_16.onnx          # ViT-B/16 surgery (DFC parser bug workaround)

thunderdome clip compile-hef \
    --onnx clip-aerial-vit-b16-v3-surgical.onnx \
    --alls configs/hailo/clip_vit_b_16_image_encoder_hailomz.alls \
    --calib-dir <crops_dir> --calib-count 500 \
    --target hailo10h --output-dir hef/ \
    --net-name clip_aerial_vit_b_16 --imgsz 224
```

### Surgery + quantization fidelity

- Surgical ONNX vs PT (fp32 cosine, single pass): **0.9798**
- HEF emulator (SDK_QUANTIZED) vs PT (20-image mean cosine): **0.9551**
- HEF emulator vs PT overlap@5 (ranking agreement): **0.820**

The drop from 0.98 → 0.96 → 0.82 is INT8 quantization noise + the
documented QuickGELU↔GELU activation mismatch in Hailo's reference
ONNX. Retrieval rankings remain in agreement, which is what matters
for downstream ReID / open-vocab search.

### Runtime preprocessing

The HEF has the OpenAI CLIP normalization layer **baked into the
compiled graph** (via `norm1` in the alls config). On the chip,
pass **letterbox-padded raw uint8** images (114-gray fill, 224×224):

```python
from PIL import Image
import numpy as np

def letterbox(img, sz=224, fill=(114, 114, 114)):
    w, h = img.size
    s = sz / max(w, h)
    nw, nh = int(round(w*s)), int(round(h*s))
    img = img.resize((nw, nh), Image.BILINEAR)
    canvas = Image.new("RGB", (sz, sz), fill)
    canvas.paste(img, ((sz-nw)//2, (sz-nh)//2))
    return canvas

# Pass to HEF as np.uint8 [1, 224, 224, 3] (NHWC).
```

### Architecture preserved

- Same `ViT-B-16` body as v2 (no quantization-aware training)
- 512-dim embedding (drop-in for arc-uas / any ViT-B/16 consumer)
- Text encoder frozen during fine-tune → text embeddings can come
  from any standard openai-pretrained ViT-B/16