Instructions to use lightonai/LightOn-rerank-PW-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use lightonai/LightOn-rerank-PW-4B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="lightonai/LightOn-rerank-PW-4B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("lightonai/LightOn-rerank-PW-4B")
model = AutoModelForMultimodalLM.from_pretrained("lightonai/LightOn-rerank-PW-4B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

sentence-transformers

How to use lightonai/LightOn-rerank-PW-4B with sentence-transformers:

from sentence_transformers import CrossEncoder

model = CrossEncoder("lightonai/LightOn-rerank-PW-4B")

query = "Which planet is known as the Red Planet?"
passages = [
	"Venus is often called Earth's twin because of its similar size and proximity.",
	"Mars, known for its reddish appearance, is often referred to as the Red Planet.",
	"Jupiter, the largest planet in our solar system, has a prominent red spot.",
	"Saturn, famous for its rings, is sometimes mistaken for the Red Planet."
]

scores = model.predict([(query, passage) for passage in passages])
print(scores)

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use lightonai/LightOn-rerank-PW-4B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "lightonai/LightOn-rerank-PW-4B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lightonai/LightOn-rerank-PW-4B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/lightonai/LightOn-rerank-PW-4B

SGLang

How to use lightonai/LightOn-rerank-PW-4B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "lightonai/LightOn-rerank-PW-4B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lightonai/LightOn-rerank-PW-4B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "lightonai/LightOn-rerank-PW-4B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lightonai/LightOn-rerank-PW-4B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use lightonai/LightOn-rerank-PW-4B with Docker Model Runner:
```
docker model run hf.co/lightonai/LightOn-rerank-PW-4B
```

Integrate with Sentence Transformers

by tomaarsen HF Staff - opened 6 days ago

base: refs/heads/main

←

from: refs/pr/1

Discussion Files changed

+210

-2

tomaarsen

6 days ago

•

edited 6 days ago

Hello!

Preface

This is the same integration as proposed for LightOn-rerank-PW-0.8B, applied to PW-4B. All explanations there (module pipeline, chat template handling, system prompt overriding, dtype behaviour) apply verbatim, so this description only lists the differences.

Heads up, this PR was AI-generated and human-reviewed.

Pull Request overview

Integrate this model with Sentence Transformers (v5.4.0+) as a CrossEncoder

Details

The PW siblings share the same tokenizer, chat template, and Yes/No scoring format, so the five added integration files (modules.json, sentence_bert_config.json, config_sentence_transformers.json, 1_LogitScore/config.json, additional_chat_templates/reranker.jinja) are byte-identical to the ones in the 0.8B PR. No weights were changed. The differences with that PR:

Qwen3.5-4B enables thinking by default in its chat template (the 0.8B and 2B backbones default to non-thinking). This changes nothing in the integration files: the bundled reranker chat template already hardcodes the non-thinking generation prompt, so CrossEncoder users get correctly positioned Yes/No logits without ever passing enable_thinking=False. The "Thinking must be disabled" note in the README now mentions this.
README.md: the same sentence-transformers tag and "Using Sentence Transformers" section, but with this model's expected outputs, plus an expected-output comment (# [-3.0, -8.0625]) in the existing transformers snippet.
Verification was re-run against this checkpoint's own plain-transformers baseline (with enable_thinking=False) in fp32 (text-only, image-only, and mixed text+image batches): max |diff| is about 4e-5, on the released sentence-transformers 5.4.0 and 5.6.0 with transformers 5.4.0 and 5.13.1.

from sentence_transformers import CrossEncoder

model = CrossEncoder("lightonai/LightOn-rerank-PW-4B")

query = "What is late interaction in neural information retrieval?"
documents = [
    "ColBERT computes token-level query-document interactions at search time...",
    "The Eiffel Tower is located on the Champ de Mars in Paris.",
]

pairs = [(query, doc) for doc in documents]
scores = model.predict(pairs)
print(scores)
# [-3.     -8.0625]

rankings = model.rank(query, documents)
print(rankings)
# [{'corpus_id': 0, 'score': -3.0}, {'corpus_id': 1, 'score': -8.0625}]

As before, page images can be passed directly as documents (PIL.Image, URL, or file path), text and image candidates can be mixed in one predict call, and none of the existing transformers/vLLM usage is affected.

Happy to tweak anything you'd like changed. Please let me know if you have any questions or feedback!

Tom Aarsen

Integrate with Sentence Transformers95c9c83d

tomaarsen changed pull request status to open 6 days ago

NohTow changed pull request status to merged 6 days ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment