samer k64 colpali

This repository contains the adapter-only SaMer checkpoint for vidore/colpali-v1.3-hf. It does not redistribute the base model weights. The repository stores only the trained projection layer and custom trust_remote_code wrapper needed to apply SaMer feature-spatial object-aware token merging at inference time.

Usage

import torch
from transformers import AutoModel, AutoProcessor

model = AutoModel.from_pretrained(
    "dmis-lab/samer-k64-colpali",
    trust_remote_code=True,
).to("cuda").eval()

processor = AutoProcessor.from_pretrained(
    model.config.base_model_name_or_path,
    trust_remote_code=True,
)

# Build processor inputs with the same convention as the base model.
# image_inputs = processor(images=[image], return_tensors="pt").to("cuda")
# query_inputs = processor(text=["a query"], return_tensors="pt", padding=True).to("cuda")

# image_tokens = model.encode_image(image_inputs)          # [B, 64, D]
# query_tokens, query_mask = model.encode_query(query_inputs, return_mask=True)
# scores = model.score(query_tokens, image_tokens, query_mask=query_mask)

SaMer Configuration

num_regions: 64
cluster_iters: 3
spatial_weight: 0.1
assignment_temperature: 0.07
training objective: retrieval loss only
inference: bbox-free feature-spatial soft centroid merging

Base Model

The base model is loaded from vidore/colpali-v1.3-hf at runtime. Please follow the base model license and usage terms.

Citation

@misc{park2026visualtokensmatterequally,
      title={Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval},
      author={Suhyeong Park and Junha Jung and Jungwoo Park and Jaewoo Kang},
      year={2026},
      eprint={2607.04605},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2607.04605},
}