This version of ColPali should be loaded with the transformers 🤗 release, not with colpali-engine. It was converted using the convert_colpali_weights_to_hf.py script from the vidore/colpali-v1.3-merged checkpoint.

ColPali: Visual Retriever based on PaliGemma-3B with ColBERT strategy

ColPali is a model based on a novel model architecture and training strategy based on Vision Language Models (VLMs) to efficiently index documents from their visual features. It is a PaliGemma-3B extension that generates ColBERT- style multi-vector representations of text and images. It was introduced in the paper ColPali: Efficient Document Retrieval with Vision Language Models and first released in this repository

The HuggingFace transformers 🤗 implementation was contributed by Tony Wu (@tonywu71) and Yoni Gozlan (@yonigozlan).

Model Description

Read the transformers 🤗 model card: https://huggingface.co/docs/transformers/en/model_doc/colpali.

Model Training

Dataset

Our training dataset of 127,460 query-page pairs is comprised of train sets of openly available academic datasets (63%) and a synthetic dataset made up of pages from web-crawled PDF documents and augmented with VLM-generated (Claude-3 Sonnet) pseudo-questions (37%). Our training set is fully English by design, enabling us to study zero-shot generalization to non-English languages. We explicitly verify no multi-page PDF document is used both ViDoRe and in the train set to prevent evaluation contamination. A validation set is created with 2% of the samples to tune hyperparameters.

Note: Multilingual data is present in the pretraining corpus of the language model (Gemma-2B) and potentially occurs during PaliGemma-3B's multimodal training.

Parameters

All models are trained for 1 epoch on the train set. Unless specified otherwise, we train models in bfloat16 format, use low-rank adapters (LoRA) with alpha=32 and r=32 on the transformer layers from the language model, as well as the final randomly initialized projection layer, and use a paged_adamw_8bit optimizer. We train on an 8 GPU setup with data parallelism, a learning rate of 5e-5 with linear decay with 2.5% warmup steps, and a batch size of 32.

Usage

This checkpoint was trained with the query prefix "Query: ", but the repository ships no processor_config.json, so transformers.ColPaliProcessor falls back to its class default of "Question: ". Set processor.query_prefix = "Query: " after loading the processor to reproduce the trained format. colpali-engine no longer sends this prefix either, as of 0.3.13 (illuin-tech/colpali#339). The Sentence Transformers configuration below restores it automatically.

Using Sentence Transformers

Install Sentence Transformers:

pip install "sentence_transformers[image]"

from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("vidore/colpali-v1.3-hf")

queries = [
    "What is the variable represented on the y-axis of the graph?",
    "Total outlay is maximum in which year?",
]
documents = [
    f"https://huggingface.co/vidore/colpali-v1.3-hf/resolve/main/assets/doc{i}.jpg" for i in range(1, 5)
]

query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (28, 128) (1030, 128)

similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[22.3635, 19.9078, 19.7110, 19.0631],
#         [ 5.8969, 13.2868,  6.1814,  6.7820]])

similarity uses MaxSim (ColBERT late interaction), so each row scores one query against every page. Embeddings are ragged: one (num_tokens, 128) array per query or page.

Documents may be file paths, URLs or PIL images. Prefer paths and URLs: Sentence Transformers loads those with transformers.image_utils.load_image, which converts to RGB. PIL images are passed to the processor as-is, and transformers 5.x no longer converts them, so a grayscale page in a batch with colour pages raises RuntimeError: stack expects each tensor to be equal size. Call .convert("RGB") yourself when passing PIL images.

Setting keep_only_token_ids on the MultiVectorMask module to [model[0].processor.image_token_id] drops the 6 non-image tokens from each page, mirroring colpali-engine's mask_non_image_embeddings. It is off by default, since 1024 of the 1030 page tokens are image patches and dropping the rest changes the scores.

Using transformers

import torch
from PIL import Image

from transformers import ColPaliForRetrieval, ColPaliProcessor

model_name = "vidore/colpali-v1.3-hf"

model = ColPaliForRetrieval.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="cuda:0",  # or "mps" if on Apple Silicon
).eval()

processor = ColPaliProcessor.from_pretrained(model_name)
processor.query_prefix = "Query: "  # the prefix this checkpoint was trained with, see the note above

# Your inputs
images = [
    Image.new("RGB", (32, 32), color="white"),
    Image.new("RGB", (16, 16), color="black"),
]
queries = [
    "What is the organizational structure for our R&D department?",
    "Can you provide a breakdown of last year’s financial performance?",
]

# Process the inputs
batch_images = processor(images=images).to(model.device)
batch_queries = processor(text=queries).to(model.device)

# Forward pass
with torch.no_grad():
    image_embeddings = model(**batch_images)
    query_embeddings = model(**batch_queries)

# Score the queries against the images
scores = processor.score_retrieval(query_embeddings.embeddings, image_embeddings.embeddings)

Resources

The ColPali arXiv paper can be found here. 📄
The official blog post detailing ColPali can be found here. 📝
The original model implementation code for the ColPali model and for the colpali-engine package can be found here. 🌎
Cookbooks for learning to use the transformers-native version of ColPali, fine-tuning, and similarity maps generation can be found here. 📚

Limitations

Focus: The model primarily focuses on PDF-type documents and high-ressources languages, potentially limiting its generalization to other document types or less represented languages.
Support: The model relies on multi-vector retreiving derived from the ColBERT late interaction mechanism, which may require engineering efforts to adapt to widely used vector retrieval frameworks that lack native multi-vector support.

License

ColPali's vision language backbone model (PaliGemma) is under gemma license as specified in its model card. ColPali inherits from this gemma license.

Contact

Manuel Faysse: manuel.faysse@illuin.tech
Hugues Sibille: hugues.sibille@illuin.tech
Tony Wu: tony.wu@illuin.tech

Citation

If you use any datasets or models from this organization in your research, please cite the original dataset as follows:

@misc{faysse2024colpaliefficientdocumentretrieval,
  title={ColPali: Efficient Document Retrieval with Vision Language Models}, 
  author={Manuel Faysse and Hugues Sibille and Tony Wu and Bilel Omrani and Gautier Viaud and Céline Hudelot and Pierre Colombo},
  year={2024},
  eprint={2407.01449},
  archivePrefix={arXiv},
  primaryClass={cs.IR},
  url={https://arxiv.org/abs/2407.01449}, 
}