# CLAUDE.md — instructions for Claude Code

## Project
Cross-platform visual search for sneakers & streetwear. User uploads a photo;
the system returns visually similar listings across resale marketplaces, ranked
by visual similarity, with size/budget filters and affiliate buy links.

**First MVP:** StockX + GOAT only. Single category (sneakers/streetwear), so NO
category detection yet. New + used inventory. Goal: deployed web app tested by
10 friends. Broader green-zone platforms come later.

## Environment
- Python 3.13 on Windows. Use `py` (not `python`) and `py -m pip`.
- Use Windows backslash paths in commands.
- Install editable: `py -m pip install -e ".[dev]"`.

## Architecture
- **MVP (now): live recall, no vector DB.**
  upload -> Grounding DINO detect the sneaker(s) (open-vocab detection prompted "shoe.
  sneaker."; needed so Lens matches the shoe, not other garments, on busy screenshots;
  no sneaker found -> ask for a clearer photo, no full-image fallback). The detector finds
  EVERY shoe in frame (subject + bystanders), so we do NOT union them all (that swallowed the
  pants + other people's feet -> Lens matched the wrong shoe); instead detect.detect_shoes
  groups each person's pair into one cropped option, ranked by prominence (area x centrality x
  confidence). 1 option -> auto-search it; 2+ -> the GotYa frontend shows a "Which shoe?"
  picker (POST /api/detect returns per-shoe tokens + thumbnails; the chosen crop_token drives
  POST /api/search). search() crops+searches the single best shoe (auto path, Streamlit/scripts);
  search_image() searches an already-chosen crop. ADR-0011 -> SerpAPI Google
  Lens recall on the
  CROPPED image (the query image is made public for Lens by SELF-HOSTING it on our own app at
  /q/{token}.png — random token + short TTL — NOT a third-party temp host: that upload was the
  single biggest search cost (~65% of wall-clock) and put the user's photo on a public host;
  recall falls back to the temp host only when the app's public URL is unknown, e.g. local
  scripts/Streamlit; config.query_image_self_host defaults on, query_host.py + app/api.py. One
  upload, two text-hinted queries: image + "stockx", then "goat"; keep
  StockX/GOAT product URLs) -> cap candidates to config.max_candidates_total (15), drawn
  round-robin per platform so StockX/GOAT stay balanced -> build listings from SerpAPI data
  (title/image/price/link), NO scraping (Cloudflare); adapters classify_url + canonicalize +
  build_affiliate_url -> SECOND StockX source for coverage: query the StockX official catalog
  API ONCE (rate limit 1 req/s; product name mined from the Lens responses, preserving the
  colorway when agreed — most specific color-bearing related_searches, else longest shared
  color-bearing title fragment, else base model — so the search narrows to the right color
  instead of all colorways; visual rerank stays the primary color discriminator; cached by
  query text in data/stockx_cache), build listings
  (listing_url stockx.com/{urlKey}, source=stockx_api, price=None because retailPrice is MSRP,
  not market price), dedupe vs Lens by canonical urlKey, reconstruct each product's CDN image
  from BOTH its title and its urlKey slug (catalog search carries no image; the two miss on
  different renamed products — together 10/10 image-bearing on a live Mexico 66 search vs 8/10
  for title alone, styleId never matches; concurrent HEAD probe) so image-bearing ones join
  the rerank and the rest become image-less coverage; skip silently if STOCKX_REFRESH_TOKEN
  is unset so the app runs
  without API config (ingest/stockx_recall.py, ADR-0010). If BOTH hinted Lens responses are
  textless (no query to mine), recall fires ONE extra unhinted Lens call (no q) on the same
  crop and folds it in two ways: its StockX/GOAT URLs are candidates and its
  related_searches/titles feed the mined query (+1 SerpAPI call only when hinted is textless;
  cached at the (image, "") key) -> Marqo embed cropped query +
  candidate thumbnails (candidate
  embeddings cached on disk by thumbnail URL in embedding_cache.py -> repeat candidates skip
  download+embed) -> in-memory cosine rerank -> verify links alive -> results with affiliate
  links. recall_candidates returns a RecallResult{candidates, more_on_stockx} and search() a
  SearchResponse{results, more_on_stockx}; the image-less StockX API listings render below the
  ranked grid as a "More matches on StockX" section with NO similarity score. The query is
  cropped ONCE and reused for recall + embedding. Each search records per-stage timings
  (timing.py: crop/upload/lens/stockx_api/downloads/embed/total) for the log + a UI caption;
  the auto search() path records the crop stage, while the API picker path omits it because
  detection ran in the earlier /api/detect request (search_image takes the chosen crop). And
  search() emits progress events to an on_progress callback for the live UI.
  Size filter + new/used badge deferred to the API phase (ADR-0007); still new+used. StockX
  market price is also deferred to the market-data endpoint (catalog search has MSRP only).
- **Target (later): pre-indexed search.** Once the StockX official API (and a
  GOAT data source) are available, pull catalogs -> index product images in
  Qdrant (embeddings via Modal) -> query the index. See docs/architecture.md.
- The data source is a pluggable module (ingest/). Swapping Google Lens for the
  API later changes one module, not the pipeline.

## Models (do not substitute without an ADR)
- Embedding model: Marqo-FashionSigLIP (Marqo/marqo-fashionSigLIP, fine-tuned SigLIP
  ViT-B/16), chosen in the Phase 1 sneaker benchmark (ADR-0001). Load via open_clip
  (create_model_and_transforms("hf-hub:Marqo/marqo-fashionSigLIP")); embed with
  model.encode_image on the model's own preprocess_val, then L2-normalize. Output dim
  is SigLIP's native width — detect at runtime via embedding.embedding_dim(), do not
  hardcode it. See src/sneaker_search/embedding.py.
- GR-Lite (srpone/gr-lite): the Phase 1 benchmark baseline only (it lost), kept in
  scripts/benchmark_models.py. It is a transformers PreTrainedModel loaded from remote
  code + model.safetensors (NOT torch.load(gr_lite.pt), which is a state dict; and it
  has NO .search() method). Loading details: docs/adr/0006-gr-lite-loading.md.
- Grounding DINO (IDEA-Research/grounding-dino-tiny): open-vocabulary object detector,
  standard transformers (AutoProcessor + AutoModelForZeroShotObjectDetection), prompted
  "shoe. sneaker.". Replaced SegFormer-clothes, which missed sneakers in real social-media
  screenshots (ADR-0009). Detects the sneaker(s) at the FRONT of the pipeline (before recall):
  keep boxes clearing the threshold, merge same-foot duplicates (IoU), cluster each person's
  pair, pad ~12%, crop each cluster, rank by prominence (detect_shoes). Do NOT union all boxes
  (ADR-0011: that merged bystanders' shoes + pants -> wrong match); the user picks among
  options when there are several. If nothing clears the threshold, detect_shoes returns []
  (crop_to_sneaker None) -> NoSneakerDetectedError (no full-image fallback, so the UI asks for a
  clearer photo). Grouping knobs: config.shoe_merge_iou / shoe_pair_gap_factor / max_shoe_options. For speed, detection runs on a downscaled copy
  (config.detect_max_side, longest side ~800px); the box is scaled back and the crop is
  taken from the full-res image, so crop quality is unchanged. post_process kwarg is
  `threshold` (renamed from box_threshold) on transformers 5.x. See src/sneaker_search/detect.py.

## Calibrated thresholds (re-measure before changing — see config.py)
- Recalibrated at the Phase 3 gate for Marqo-FashionSigLIP. SigLIP cosine runs much
  higher than GR-Lite's: good matches ~0.8-0.98 same-domain, off-target items ~0.7+.
- Good match: cosine >= 0.70. Similar style: 0.60-0.70. Below ~0.60: weak.
- These are LABELS, not a hard filter: recall + the SegFormer shoe-crop decide the
  candidate set, and the MVP shows the top-N ranked results (see config.py).
- Caveat: the good cut is from same-domain pairs; re-confirm on cropped cross-domain
  photos at the 10-photo eyeball. The old GR-Lite 0.40/0.25 are retired.

## Development workflow (follow this)
- Work ONE story at a time from docs/workflow-status.md. Do not start the next
  until the current one meets its acceptance criteria.
- If a request conflicts with docs/architecture.md, HALT and ask — do not invent
  a divergent workaround.
- main must always run. Do risky work on a feature branch.
- After a story: update docs/workflow-status.md, then say it is ready to commit.
  Use conventional commit messages (feat/fix/docs/refactor/test/chore).
- Keep CLAUDE.md and docs/ current when decisions change.

## Code standards
- Every function: type hints + a one-line docstring.
- One responsibility per function; split if over ~40 lines.
- All config via src/sneaker_search/config.py reading .env — never hardcode keys,
  URLs, or thresholds.
- Errors fail loudly with a clear message; no silent except: pass.
- Format/lint with ruff before committing (ruff format . && ruff check --fix .).
- Tests in tests/; cover critical paths (embedding shape, each adapter's parse).

## Things NOT to do
- Do not replace visual search with LLM/text attribute matching.
- Do not add a vector DB (Qdrant) for the MVP — it is live recall; Qdrant is for
  the pre-indexed target once API/feed data exists.
- Do not use AutoModel.from_pretrained for GR-Lite: it loads the weights but then
  crashes on transformers >=5 (model targets 4.49). Instantiate GRLiteModel from the
  remote code and load_state_dict(model.safetensors) instead — see ADR-0006.
- Do not add category detection yet (single category in the MVP).
- Do not scrape StockX/GOAT product pages (Cloudflare); build listings from SerpAPI
  recall data instead (ADR-0007).
- Do not commit .env, data/, or model weights (see .gitignore).