rollenso/siglip-synthetic-hq-retrieval-v1

SigLIP: Fine-Tuned for Detailed Image Retrieval

This repository contains a fine-tuned version of Google's SigLIP, optimized for high-precision semantic search and detailed image description alignment. The model is specifically engineered to understand complex visual compositions, premium aesthetics, and intricate details that standard CLIP/SigLIP models often miss.

github repo

🚀 Model Details

• Base Model: google/siglip-base-patch16-224
• Architecture: Contrastive Image-Text Encoder
• Parameter Count: ~200M (bfloat16)
• Primary Use Case: High-fidelity image retrieval, aesthetic-aware search engines, and vector databases (e.g., using pgvector or Milvus).

🧠 Training: The Power of Rich Captions

Unlike standard models trained on brief web alt-texts, this model was fine-tuned using a High-Density Synthetic Dataset.

• Data Source: ~32,000 image-text pairs refined by a Vision Language Model (VLM).
• Descriptive Richness: Instead of simple tags, the model learned from multi-sentence descriptions generated by Qwen-VL. These captions cover lighting, texture (e.g., Glassmorphism, Liquid Metal), professional composition, and specific object relationships.
• Optimization: Fine-tuned via LoRA (Low-Rank Adaptation) to preserve base knowledge while injecting deep domain expertise in aesthetics.

⚠️ Search Strategy: Detailed Queries vs. Tags

Due to its training on VLM-generated descriptions, this model excels when provided with descriptive, natural language queries. It is designed to match the "richness" of the visual data.

For optimal results, use detailed prompts that describe the scene:

❌ Basic Tag (Standard Performance): texts = ["modern office"]

✅ Rich Description (Superior Performance): texts = ["A minimalist modern office with natural sunlight, wooden furniture, and a clean glass desk against a neutral wall."]

Pro Tip: For production environments, we recommend using a small LLM proxy (like Qwen 1.5B or Llama 3) to expand simple user keywords into descriptive search sentences before generating embeddings.

💻 Usage (Inference)

import torch
import torch.nn.functional as F
from transformers import AutoProcessor, AutoModel
from PIL import Image

device = torch.device("mps" if torch.backends.mps.is_available() else ("cuda" if torch.cuda.is_available() else "cpu"))
model_path = "rollenso/siglip-synthetic-hq-retrieval-v1"

processor = AutoProcessor.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, dtype=torch.bfloat16).to(device)
model.eval()

texts = [
    "A product photography shot of a black mirrorless camera on a blue metallic tripod.",
    "A glowing liquid glass user interface dashboard."
]
image = Image.open("path_to_your_image.jpg").convert("RGB")

inputs = processor(
    text=texts, 
    images=image, 
    padding="max_length", 
    max_length=64, 
    truncation=True,
    return_tensors="pt"
).to(device)

with torch.no_grad():
    outputs = model(**inputs)
    image_embeds = F.normalize(outputs.image_embeds, p=2, dim=-1)
    text_embeds = F.normalize(outputs.text_embeds, p=2, dim=-1)
    similarity = torch.matmul(image_embeds, text_embeds.t())

scores = similarity[0].cpu().tolist()
for text, score in zip(texts, scores):
    print(f"[{score:.4f}] {text}")