Add model card

README.md

@@ -0,0 +1,214 @@
+---
+license: apache-2.0
+language:
+- en
+- fr
+base_model:
+- Qwen/Qwen3.5-2B
+pipeline_tag: image-text-to-text
+library_name: transformers
+tags:
+- reranker
+- cross-encoder
+- multimodal
+- text-ranking
+- document-reranking
+- vidore
+- beir
+---
+
+<div align="center">
+<img src="https://cdn-uploads.huggingface.co/production/uploads/609bbe2f4932693ca2009d6a/kbQOAarw0eaApow3M9HIl.png" alt="LightOn" width="512">
+
+[![Website](https://img.shields.io/badge/LightOn-Website-blue?logo=google-chrome)](https://lighton.ai)
+[![LinkedIn](https://img.shields.io/badge/LightOn-LinkedIn-0A66C2?logo=linkedin)](https://www.linkedin.com/company/lighton/)
+[![X](https://img.shields.io/badge/@LightOnIO-X-black?logo=x)](https://x.com/LightOnIO)
+
+📝 Blog post: *coming soon* <!-- TODO: add blog post link -->
+
+</div>
+
+<h1 align="center">LightOn-rerank-PW-2B</h1>
+<h3 align="center">Unified Text + Visual Document Reranker by LightOn</h3>
+
+<p align="center">
+<a href="https://huggingface.co/lightonai/LightOn-rerank-PW-0.8B">PW-0.8B</a> |
+<a href="https://huggingface.co/lightonai/LightOn-rerank-LW-0.8B">LW-0.8B</a> |
+<a href="https://huggingface.co/lightonai/LightOn-rerank-PW-2B">PW-2B</a> |
+<a href="https://huggingface.co/lightonai/LightOn-rerank-LW-2B">LW-2B</a> |
+<a href="https://huggingface.co/lightonai/LightOn-rerank-PW-4B">PW-4B</a> |
+<a href="https://huggingface.co/lightonai/LightOn-rerank-LW-4B">LW-4B</a>
+</p>
+
+---
+
+## About the LightOn-rerank family
+
+Production retrieval pipelines usually need two rerankers: one for text passages and one for visual documents (PDF pages, slides, scans). **LightOn-rerank** models are *unified* cross-encoder rerankers: a single model scores both text passages and document page images against a query, on top of any first-stage retriever (BM25, dense embeddings, or ColPali-family late-interaction models).
+
+The models are built on Qwen3.5 vision-language backbones (hybrid linear + full attention) and jointly fine-tuned on text and visual reranking data with mixed-modality batches (LoRA, merged into the released weights). Training data is English-only; French performance transfers zero-shot from the multilingual backbone.
+
+The family comes in two scoring flavours × three sizes (0.8B / 2B / 4B):
+
+- **PW (pointwise)** — each candidate is scored independently: the model judges whether the document answers the query, and the score is `logit("Yes") − logit("No")`. One forward pass per candidate, no generation — simple to serve (vLLM-compatible) and embarrassingly parallel.
+- **LW (listwise)** — generative listwise ranking: 4 candidates are placed in a single prompt and the model generates a permutation (`[2] > [4] > [1] > [3]`). Larger candidate pools are ranked with a sliding window (window 4, stride 2, bottom-to-top). Cross-document attention makes LW markedly stronger on hard visual reranking, and unlike pointwise scoring it keeps improving with backbone size.
+
+**LightOn-rerank-PW-2B** is the pointwise flagship of the family: a single 2B model that is simultaneously competitive with dedicated text rerankers on BEIR and with vision-specialised rerankers on ViDoRe V2/V3 — with none of the usual trade-off between the two modalities, and the serving simplicity of independent per-candidate scoring.
+
+## Results
+
+**ViDoRe V3** (visual document reranking, 8 domains × EN/FR queries), NDCG@10, ColQwen2.5-v0.2 first stage, retrieve 100 / rerank 100. All models — including baselines — were re-evaluated under this same two-stage protocol, so numbers are mutually comparable but not comparable to vendor-reported end-to-end results.
+
+| Model | Params | Scoring | ViDoRe V3 overall @10 |
+|---|---|---|---|
+| [LightOn-rerank-LW-4B](https://huggingface.co/lightonai/LightOn-rerank-LW-4B) | 4.5B | listwise | 0.6469 |
+| *Qwen3-VL-Reranker-8B* | *8B* | *pointwise (pooling)* | *0.6423* |
+| [LightOn-rerank-LW-2B](https://huggingface.co/lightonai/LightOn-rerank-LW-2B) | 2.2B | listwise | 0.6266 |
+| **LightOn-rerank-PW-2B (this model)** | 2.2B | pointwise | **0.5987** |
+| [LightOn-rerank-PW-4B](https://huggingface.co/lightonai/LightOn-rerank-PW-4B) | 4.5B | pointwise | 0.5980 |
+| *jina-reranker-m0* | *2.4B* | *pointwise* | *0.5939* |
+| *Qwen3-VL-Reranker-2B* | *2B* | *pointwise (pooling)* | *0.5918* |
+| [LightOn-rerank-LW-0.8B](https://huggingface.co/lightonai/LightOn-rerank-LW-0.8B) | 0.85B | listwise | 0.5825 |
+| [LightOn-rerank-PW-0.8B](https://huggingface.co/lightonai/LightOn-rerank-PW-0.8B) | 0.85B | pointwise | 0.4820 |
+
+### ViDoRe V3 detail (NDCG, ColQwen2.5 first stage, rerank-100)
+
+| Domain | EN @5 | EN @10 | FR @5 | FR @10 |
+|---|---|---|---|---|
+| finance_en | 0.5893 | 0.6113 | 0.5585 | 0.5716 |
+| finance_fr | 0.4643 | 0.4934 | 0.4947 | 0.5260 |
+| computer_science | 0.7239 | 0.7398 | 0.6814 | 0.7047 |
+| hr | 0.6570 | 0.6665 | 0.5996 | 0.6139 |
+| energy | 0.6634 | 0.6921 | 0.6735 | 0.7013 |
+| industrial | 0.5420 | 0.5520 | 0.4808 | 0.4864 |
+| pharmaceuticals | 0.6298 | 0.6385 | 0.5958 | 0.6039 |
+| physics | 0.4551 | 0.4851 | 0.4568 | 0.4927 |
+| **mean** | **0.5906** | **0.6098** | **0.5676** | **0.5876** |
+
+Overall @10: **0.5987** (EN 0.6098 / FR 0.5876).
+
+### Other benchmarks (same fixed-first-stage protocol)
+
+| Benchmark | LightOn-rerank-PW-2B | Best 2B-class baseline |
+|---|---|---|
+| ViDoRe V2, mean NDCG@10 (ColQwen2.5 first stage) | 0.8558 | jina-reranker-m0: 0.8596; MonoQwen2-VL-v0.1: 0.8454; Qwen3-VL-Reranker-2B: 0.8428 |
+| BEIR 15 datasets, clean mean NDCG@10 (jina-embeddings-v3 first stage) | 0.5227 | jina-reranker-m0: 0.5677; Qwen3-VL-Reranker-2B: 0.5025 |
+| BEIR 15 datasets, clean mean NDCG@10 (BM25 first stage) | 0.4968 | jina-reranker-m0: 0.5397; Qwen3-VL-Reranker-2B: 0.4815 |
+
+Clean mean excludes the two contaminated datasets (NQ, MSMARCO). On text it sits between the two strongest 2B baselines; on visual reranking it matches or beats every 2B-class model — jina-reranker-m0 is the only baseline that is close on both, and it trails on ViDoRe V3 (0.5939 vs 0.5987).
+
+## Model Details
+
+- **Model type:** multimodal cross-encoder reranker — **pointwise**: each candidate is scored independently as `logit("Yes") − logit("No")`
+- **Base model:** [Qwen/Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) (Qwen3.5 hybrid linear + full attention VLM)
+- **Parameters:** ≈2.2B (bfloat16, 4.4 GB)
+- **Inputs:** query (text) + candidate document(s) — text passage **or** page image
+- **Fine-tuning:** joint text+vision LoRA (r=32, α=32, rsLoRA — merged into the released weights), mixed-modality batches (2 text + 2 vision groups per micro-batch), vision loss weight 1.3, lr 1e-4, 1 epoch (465 steps)
+- **Data:** ~120k EN text queries (NQ, TriviaQA, MSMARCO) with mined hard negatives + ~118k visual query–page pairs with ColPali-family hard negatives (`dedup_120k_en_v3` + `ColPali_hard_neg_v2`)
+- **Languages:** English (training), French (zero-shot transfer)
+- **Requirements:** `transformers >= 5.4.0` (`qwen3_5` architecture)
+- **Internal experiment ID:** `exp30`
+
+## Usage — pointwise reranking
+
+Each candidate is scored independently as `logit("Yes") − logit("No")` at the first generated position; sort candidates by descending score. The model was trained with a fixed system prompt and user template (`RERANKER_SYSTEM_V2` / `RERANKER_TEXT_V2` / `RERANKER_VISION_V2`) — these will be published alongside the blog post and **must** be used verbatim for best results.
+
+```python
+import torch
+from transformers import AutoModelForImageTextToText, AutoProcessor
+
+model_id = "lightonai/LightOn-rerank-PW-2B"
+model = AutoModelForImageTextToText.from_pretrained(
+    model_id,
+    dtype=torch.bfloat16,
+    attn_implementation="flash_attention_2",
+    device_map="cuda",
+).eval()
+processor = AutoProcessor.from_pretrained(model_id)
+processor.tokenizer.padding_side = "left"  # scores are read at the last position
+
+YES_TOKEN_ID = 9175  # "Yes"
+NO_TOKEN_ID = 2665   # "No"
+
+SYSTEM_PROMPT = ...   # RERANKER_SYSTEM_V2 — released with the blog post
+USER_TEMPLATE = ...   # RERANKER_TEXT_V2, with {query} and {doc} fields
+
+query = "What is late interaction in neural information retrieval?"
+documents = [
+    "ColBERT computes token-level query-document interactions at search time...",
+    "The Eiffel Tower is located on the Champ de Mars in Paris.",
+]
+
+texts = [
+    processor.apply_chat_template(
+        [
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {"role": "user", "content": USER_TEMPLATE.format(query=query, doc=doc)},
+        ],
+        tokenize=False,
+        add_generation_prompt=True,
+    )
+    for doc in documents
+]
+inputs = processor(
+    text=texts, return_tensors="pt", padding=True, truncation=True, max_length=2048
+).to(model.device)
+with torch.inference_mode():
+    logits = model(**inputs).logits[:, -1]
+scores = (logits[:, YES_TOKEN_ID] - logits[:, NO_TOKEN_ID]).tolist()
+
+ranked = sorted(zip(scores, documents), reverse=True)
+```
+
+To score a **page image** instead of a text passage, replace the user message with:
+
+```python
+{"role": "user", "content": [
+    {"type": "image", "image": page_image},  # PIL.Image
+    {"type": "text", "text": VISION_TEMPLATE.format(query=query)},  # RERANKER_VISION_V2
+]}
+```
+
+and pass `images=[page_image, ...]` to the processor call.
+
+### Serving with vLLM
+
+```bash
+vllm serve lightonai/LightOn-rerank-PW-2B --trust-remote-code --max-model-len 16384
+```
+
+```python
+resp = client.chat.completions.create(
+    model="lightonai/LightOn-rerank-PW-2B",
+    messages=messages,          # same system + user messages as above
+    max_tokens=1,
+    logprobs=True,
+    top_logprobs=20,
+    temperature=0.0,
+)
+top = resp.choices[0].logprobs.content[0].top_logprobs
+lp = {t.token: t.logprob for t in top}
+score = lp.get("Yes", -100.0) - lp.get("No", -100.0)
+```
+
+Full-page document images can exceed 8k tokens — keep `--max-model-len` at 16384 or higher when reranking page images.
+
+## Notes & limitations
+
+- Pointwise scoring does not improve with backbone scale (the 4B pointwise model ties this one) — if you have the compute budget, [LightOn-rerank-LW-2B](https://huggingface.co/lightonai/LightOn-rerank-LW-2B) is +0.03 NDCG@10 on ViDoRe V3 at the same size.
+- Training data is English-only. French works zero-shot (the backbone is multilingual) but is slightly behind English on average.
+- Vision hard negatives were mined with ColPali-family retrievers; the model pairs best with a ColPali-family first stage (e.g. ColQwen2.5) for visual reranking.
+- BEIR contamination flag: NQ and MSMARCO are part of the text training data; headline text figures use clean means that exclude them.
+
+## The LightOn-rerank family
+
+| Model | Backbone | Scoring | ViDoRe V3 @10 |
+|---|---|---|---|
+| [LightOn-rerank-PW-0.8B](https://huggingface.co/lightonai/LightOn-rerank-PW-0.8B) | Qwen3.5-0.8B | pointwise | 0.4820 |
+| [LightOn-rerank-LW-0.8B](https://huggingface.co/lightonai/LightOn-rerank-LW-0.8B) | Qwen3.5-0.8B | listwise | 0.5825 |
+| [LightOn-rerank-PW-2B](https://huggingface.co/lightonai/LightOn-rerank-PW-2B) | Qwen3.5-2B | pointwise | 0.5987 |
+| [LightOn-rerank-LW-2B](https://huggingface.co/lightonai/LightOn-rerank-LW-2B) | Qwen3.5-2B | listwise | 0.6266 |
+| [LightOn-rerank-PW-4B](https://huggingface.co/lightonai/LightOn-rerank-PW-4B) | Qwen3.5-4B | pointwise | 0.5980 |
+| [LightOn-rerank-LW-4B](https://huggingface.co/lightonai/LightOn-rerank-LW-4B) | Qwen3.5-4B | listwise | **0.6469** |
+
+Rule of thumb: **LW** models are stronger at every size (and the gap grows with size); **PW** models are cheaper to serve and score candidates independently. For the best quality pick [LW-4B](https://huggingface.co/lightonai/LightOn-rerank-LW-4B); for the best quality/cost trade-off pick [LW-2B](https://huggingface.co/lightonai/LightOn-rerank-LW-2B); for maximum throughput on text-heavy workloads pick a PW model.