llama-thunderdome-clip-aerial-vit-b16-v3-drone-fleet

A drone-fleet-specialized fine-tune of OpenCLIP ViT-B/16 for aerial vehicle + person retrieval and re-identification.

What this is

v3-drone-fleet is the third member of the llama-thunderdome-clip-aerial-* family. It complements v2:

• v2 — fine-tuned with subtype-aware captions (sedan/SUV/pickup/ school_bus/etc) on curated aerial imagery. Best for specific vehicle subtype queries.
• v3-drone-fleet (this model) — fine-tuned on real drone-fleet footage (2 drones, 12 videos, 4,575 Gemini-captioned crops). Best for drone-fleet domain adaptation — improves discrimination on the actual camera profile + altitude of operational drones.

Training data

Source	Crops	Captioning
Real drone footage (2 drone profiles, 12 videos)	4,575	Gemini-2.5-flash-lite verdict + subtype + color/clothing

Class distribution: person 3,130 · car 1,089 · truck 356.

Caption style: overhead drone view of <color> <subtype> for vehicles, person in <clothing_color> clothing for persons.

Training recipe

base_model:    ViT-B-16
pretrained:    openai
text_encoder:  frozen
epochs:        15  (best at epoch 4)
batch_size:    64
learning_rate: 1e-5
warmup:        100 steps
val_split:     0.1

Eval results

Metric	Baseline (openai)	v3 best (epoch 4)	Delta
val_loss	2.83	2.74	-3.2%
val_recall@1 (image→text)	0.169	0.212	+25.4% (rel)

How to use

As a regular open_clip model

import open_clip
from huggingface_hub import hf_hub_download
import torch

ckpt_path = hf_hub_download(
    repo_id="llama-farm/llama-thunderdome-clip-aerial-vit-b16-v3-drone-fleet", filename="best.pt"
)
model, _, preprocess = open_clip.create_model_and_transforms(
    "ViT-B-16", pretrained="openai"
)
state = torch.load(ckpt_path, map_location="cpu")
model.load_state_dict(state["model"] if "model" in state else state)
model.eval()

As a ReID backbone inside llama-thunderdome (BotSORT / DeepOcSort)

from thunderdome.tracking.clip_reid_adapter import CLIPReidAdapter
adapter = CLIPReidAdapter(
    checkpoint="best.pt", base_model="ViT-B-16"
)
features = adapter.get_features(xyxys, img)   # [N, 512] L2-normalized

thunderdome tracking-lab auto-detects CLIP checkpoints and routes them through the adapter inside the TrackerWrapper — no custom code needed at the operator level.

When to pick v3 vs v2

• v3 (this model) -> re-identification across video frames from YOUR drones. Better at "is this the same person/vehicle 5 seconds later?" because it's tuned to the specific camera profile.
• v2 -> subtype queries ("a yellow pickup truck", "a school bus"). Better at semantic disambiguation between vehicle classes.

For most drone tracking workloads, use both — v3 as the ReID backbone (matching across time/cameras) and v2 as the query encoder (matching text -> object).

Limitations

• Crops smaller than ~32x32 px (people-as-dots at altitude > 80m) do not produce discriminative embeddings — all map to similar vectors. Spatial-temporal clustering via thunderdome tracking-lab count-unique is more appropriate at that scale.
• Trained on 2 drone profiles. Cross-fleet generalization unverified.

Generated

2026-05-26T14:36:33.904905 via the llama-thunderdome agent loop.

Full session report: gs://thunderdome-tracking-lab/test-runs/CLIP_DATASET_REPORT.md

Edge Export (Hailo-10H)

This release includes deployment artifacts for the full path from training to chip:

Artifact	Size	Purpose
best.pt / last.pt	598 MB	PyTorch fine-tuned weights (full precision)
clip-aerial-vit-b16-v3-224.onnx	345 MB	Clean ONNX (x86 / ARM CPU / GPU runtime)
clip-aerial-vit-b16-v3-surgical.onnx	344 MB	Hailo-DFC-compatible ONNX (weights grafted into HMZ ref ONNX)
clip-aerial-vit-b16-v3-224-hailo10h.hef	79.7 MB	Compiled HEF for Hailo-10H edge accelerator
quant_report.json	—	INT8 quantization report from DFC compile
validate_report.json	—	PT-fp32 vs HEF (SDK_QUANTIZED) cosine + overlap@5

HEF compilation pipeline

thunderdome clip onnx-surgery \
    --our-pt best.pt \
    --output clip-aerial-vit-b16-v3-surgical.onnx \
    --ref-onnx /tmp/hailo_ref/clip_vit_b_16.onnx          # ViT-B/16 surgery (DFC parser bug workaround)

thunderdome clip compile-hef \
    --onnx clip-aerial-vit-b16-v3-surgical.onnx \
    --alls configs/hailo/clip_vit_b_16_image_encoder_hailomz.alls \
    --calib-dir <crops_dir> --calib-count 500 \
    --target hailo10h --output-dir hef/ \
    --net-name clip_aerial_vit_b_16 --imgsz 224

Surgery + quantization fidelity

• Surgical ONNX vs PT (fp32 cosine, single pass): 0.9798
• HEF emulator (SDK_QUANTIZED) vs PT (20-image mean cosine): 0.9551
• HEF emulator vs PT overlap@5 (ranking agreement): 0.820

The drop from 0.98 → 0.96 → 0.82 is INT8 quantization noise + the documented QuickGELU↔GELU activation mismatch in Hailo's reference ONNX. Retrieval rankings remain in agreement, which is what matters for downstream ReID / open-vocab search.

Runtime preprocessing

The HEF has the OpenAI CLIP normalization layer baked into the compiled graph (via norm1 in the alls config). On the chip, pass letterbox-padded raw uint8 images (114-gray fill, 224×224):

from PIL import Image
import numpy as np

def letterbox(img, sz=224, fill=(114, 114, 114)):
    w, h = img.size
    s = sz / max(w, h)
    nw, nh = int(round(w*s)), int(round(h*s))
    img = img.resize((nw, nh), Image.BILINEAR)
    canvas = Image.new("RGB", (sz, sz), fill)
    canvas.paste(img, ((sz-nw)//2, (sz-nh)//2))
    return canvas

# Pass to HEF as np.uint8 [1, 224, 224, 3] (NHWC).

Architecture preserved

• Same ViT-B-16 body as v2 (no quantization-aware training)
• 512-dim embedding (drop-in for arc-uas / any ViT-B/16 consumer)
• Text encoder frozen during fine-tune → text embeddings can come from any standard openai-pretrained ViT-B/16