README.md

---
license: other
license_name: nvidia-open-model-license
license_link: >-
  https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/
tags:
  - reranker
  - cross-encoder
  - visual-document-retrieval
  - question-answering retrieval
  - multimodal reranking
  - semantic-search
  - rag
  - transformers
  - sentence-transformers
pipeline_tag: text-ranking
language:
  - multilingual
library_name: transformers
---


## **Model Overview**


### Description:

The NVIDIA `llama-nemotron-rerank-vl-1b-v2-fp8` is the quantized version of the [NVIDIA llama-nemotron-rerank-vl-1b-v2](https://huggingface.co/nvidia/llama-nemotron-rerank-vl-1b-v2/tree/9c20c4ae) which is developed for multimodal question-answering retrieval. For more information, please check [here](https://huggingface.co/nvidia/llama-nemotron-rerank-vl-1b-v2/tree/9c20c4ae). The NVIDIA `llama-nemotron-rerank-vl-1b-v2-fp8` model is quantized with [TensorRT Model Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer).

For this quantization checkpoint we quantize only the language portion of the model, llama-3-1B. All other parts including vision encoder (SigLIP 400M) and the binary classification head are not quantized and are left in default bf-16 or fp-32 precision.

This model is ready for commercial use. <br>

### License/Terms of Use:

The use of this model is governed by the [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/) and the use of the post-processing scripts are licensed under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0.txt). Additional Information: [Llama 3.2 Community Model License Agreement](https://www.llama.com/llama3_2/license/). Built with Llama. |

### Deployment Geography:
Global <br>

### Use Case: <br>
The llama-nemotron-rerank-vl-1b-v2-fp8 is suitable for users who want to build a multimodal question-and-answer application over a large corpus, leveraging the latest dense retrieval technology.  

The input of the model is a query combined with a text or image document and the output is a float representing the relevance score (logit) reflecting how strongly it predicts that the passage or document contains the information relevant to the query. Multiple documents could be passed with a single query for batched processing, in this case the output contains a logit per document.

The rerank model is a cross-encoder that supports context in textual format (e.g. the query or the OCR text of a page or a section of a document) or the image of a document page.  

Typically, the embedding model is used first to embed (vectorize) the whole corpus (document images or text chunks), and embeddings are stored in a vector database associated to its raw content (image or text). Then at inference time, the embedding model is used to embed the query and pick closest documents based on the cosine similarity of document and query embeddings. Finally, the rerank model is used on the query and top selections from the embedding model to improve the selection document relevance scores with better context information.

### Release Date
Hugging Face 07/28/2026 via https://huggingface.co/nvidia/llama-nemotron-rerank-vl-1b-v2-fp8

## Model Architecture:

**Architecture Type:** Transformer<br>
**Network Architecture:** : Eagle VLM architecture with SigLIP 2 400M vision encoder and [llama-nemotron-rerank-1b-v2](https://huggingface.co/nvidia/llama-nemotron-rerank-1b-v2) model as language model. <br>

The llama-nemotron-rerank-vl-1b-v2 is a cross-encoder model with approximately 1.7B parameters. It is a fine-tuned version of an NVIDIA Eagle-family model, which consists of the SigLIP 2 400M vision encoder and the Llama 3.2 1B language model. The final embedding output by the decoder is aggregated using a mean pooling strategy, and a binary classification head is fine-tuned for the ranking task. The CrossEntropy loss is used to maximize the likelihood of (visual) documents containing information to answer the question and minimize the likelihood for (negative) documents that do not contain information to answer the question.

The vision-language model reranker incorporates key innovations from NVIDIA, including [Eagle 2 work](https://arxiv.org/abs/2501.14818) which uses a tiling-based VLM architecture, and [nemoretriever-parse](https://build.nvidia.com/nvidia/nemoretriever-parse). The Eagle 2 architecture, available on [Hugging Face](https://huggingface.co/collections/nvidia/eagle-2-6764ba887fa1ef387f7df067), significantly enhances multimodal understanding through its dynamic tiling and mixture of vision encoders design. It particularly improves performance on tasks that involve high-resolution images and complex visual content.

**Number of model parameters:**  
- Llama 3.2 1B language model: 1.23 B (Transformer parameters: 973 M, Token embedding parameters: 262 M)  
- SigLip 2 image encoder: 428.77 M

## Input(s): <br>
**Input Type(s):** Image, Text<br>

**Input Format(s):** <br>
- Image: Red, Green, Blue (RGB) <br>
- Text: String <br>

**Input Parameters:** 
- Image: Two-Dimensional (2D) <br>
- Text: One-Dimensional (1D) <br>

**Other Properties Related to Input:** <br>

The model was fine-tuned exclusively on image data, using max_input_tiles = 4 and the maximum context length of 2048 tokens. For evaluation, it was tested on image-only, image+text, and text-only inputs, with max_input_tiles = 6 and the maximum context length of 10240 tokens. Inputs exceeding the maximum length are truncated. <br>


## Output(s)

**Output Type(s):** Floats <br>

**Output Format(s):** List of Floats <br>

**Output Parameters:** One-Dimensional (1D) <br>

**Other Properties Related to Output:** Each value corresponds to a raw logit. Users can choose to apply a Sigmoid activation function to the logits to convert them into probabilities during model usage.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions. <br> 


## vLLM Usage

The model can be served with [vLLM](https://github.com/vllm-project/vllm) for high-throughput reranking. A chat template **must** be provided via `--chat-template` to correctly apply the `question:` / `passage:` prefixes based on the message role — without it, the prompt template is not applied and results will be incorrect.

See the [vLLM documentation](https://docs.vllm.ai/en/stable/models/pooling_models/specific_models/#llama-nemotron-multimodal) for full details.

### Online Serving

Create the score template file and start the server:
```bash
cat > nemotron-rerank-vl.jinja << 'JINJA'
{%- set query_msg = (messages | selectattr("role", "equalto", "query") | list | first) -%} 
{%- set doc_msg   = (messages | selectattr("role", "equalto", "document") | list | first) -%} 
{%- set q = query_msg["content"] -%} 
{%- set d = doc_msg["content"] -%} 
{# If the doc contains <image> anywhere, hoist a single <image> to the front #} 
{%- set has_image = ("<image>" in d) -%} 
{%- set d_clean = d | replace("<image>", "") -%} 
{%- set q_clean = q | replace("<image>", "") -%} 
{%- if has_image -%}<image>{{ " " }}{%- endif -%} 
question:{{ q_clean }}{{ " " }} 
{{ " " }} 
{{ " " }}passage:{{ d_clean }} 
JINJA

vllm serve nvidia/llama-nemotron-rerank-vl-1b-v2-fp8 \
  --runner pooling \
  --trust-remote-code \
  --max-model-len 10240 \
  --chat-template nemotron-rerank-vl.jinja
```

Note: Use `--max-model-len 10240` to support all modalities including image+text. A smaller value like `2048` can be used if only processing image-only or text-only inputs.

The chat template uses the message `role` to combine the correct prefixes: set `role` to `"query"` for queries (prepends `question:`) and `"document"` for passages (prepends `passage:`).

Send a rerank request:
```python
import requests

url = "http://localhost:8000/v1/rerank"

# Text-only reranking
response = requests.post(url, json={
    "model": "nvidia/llama-nemotron-rerank-vl-1b-v2-fp8",
    "query": "How is AI improving the intelligence and capabilities of robots?",
    "documents": [
        "AI enables robots to perceive, plan, and act autonomously.",
        "A biological foundation model designed to analyze DNA, RNA, and protein sequences.",
    ],
})
print(response.json())

# Image document reranking
response = requests.post(url, json={
    "model": "nvidia/llama-nemotron-rerank-vl-1b-v2-fp8",
    "query": "How is AI improving the intelligence and capabilities of robots?",
    "documents": [
        {"content": [{"type": "image_url", "image_url": {"url": "https://example.com/page.png"}}]},
        {"content": [
            {"type": "text", "text": "AI enables robots to perceive, plan, and act autonomously."},
            {"type": "image_url", "image_url": {"url": "https://example.com/page2.png"}},
        ]},
    ],
})
print(response.json())
```

### Offline / In-Process

```python
from vllm import LLM

SCORE_TEMPLATE = """\
{%- set query_msg = (messages | selectattr('role', 'equalto', 'query') | list | first) -%}
{%- set doc_msg   = (messages | selectattr('role', 'equalto', 'document') | list | first) -%}
{%- set q = query_msg['content'] -%}
{%- set d = doc_msg['content'] -%}
{%- set has_image = ("<image>" in d) -%}
{%- set d_clean = d | replace("<image>", "") -%}
{%- set q_clean = q | replace("<image>", "") -%}
{%- if has_image -%}<image>{{ " " }}{%- endif -%}
question:{{ q_clean }}{{ " " }}
{{ " " }}
{{ " " }}passage:{{ d_clean }}"""

llm = LLM(
    model="nvidia/llama-nemotron-rerank-vl-1b-v2-fp8",
    runner="pooling",
    max_model_len=10240,
    trust_remote_code=True,
)

query = "How is AI improving the intelligence and capabilities of robots?"
documents = [
    "AI enables robots to perceive, plan, and act autonomously.",
    "A biological foundation model designed to analyze DNA, RNA, and protein sequences.",
]

# Text-only scoring
outputs = llm.score(query, documents, chat_template=SCORE_TEMPLATE)
for doc, output in zip(documents, outputs):
    print(f"Score: {output.outputs.score:.4f} | {doc}")

# Image document scoring
outputs = llm.score(
    query,
    [
        {"content": [{"type": "image_url", "image_url": {"url": "https://example.com/page.png"}}]},
        {"content": [
            {"type": "text", "text": "AI enables robots to perceive, plan, and act autonomously."},
            {"type": "image_url", "image_url": {"url": "https://example.com/page2.png"}},
        ]},
    ],
    chat_template=SCORE_TEMPLATE,
)
for output in outputs:
    print(f"Score: {output.outputs.score:.4f}")
```


## Software Integration:

**Runtime Engine(s)**: vLLM<br>
**Supported Hardware Microarchitecture Compatibility**: NVIDIA Blackwell, NVIDIA Hopper, NVIDIA Lovelace <br>
**Preferred/Supported Operating System(s):** Linux<br>

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment. <br>


## Model Version(s):
`llama-nemotron-rerank-vl-1b-v2-fp8`<br>
The model is quantized with nvidia-modelopt v0.43.0

## Post Training Quantization:

This model was obtained by quantizing the weights and activations of [nvidia/llama-nemotron-rerank-vl-1b-v2](https://huggingface.co/nvidia/llama-nemotron-rerank-vl-1b-v2/tree/9c20c4ae) to FP8 data type, ready for inference with vLLM

## Training, Testing, and Evaluation Datasets:

### Dataset Overview

This checkpoint is an FP8 post-training-quantized derivative of [nvidia/llama-nemotron-rerank-vl-1b-v2](https://huggingface.co/nvidia/llama-nemotron-rerank-vl-1b-v2/tree/9c20c4ae). No additional supervised training or fine-tuning data was used to create this FP8 checkpoint.

For this FP8 release, data was used for two purposes: post-training quantization calibration and evaluation. The [cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail) dataset was used for FP8 quantization calibration.

The FP8 checkpoint was produced by applying post-training quantization to the BF16 parent model using NVIDIA TensorRT Model Optimizer. Calibration samples were converted into model inputs and used to estimate quantization parameters for weights and activations. This calibration process did not involve supervised training, fine-tuning, or modification of the model objective. 

## Calibration dataset

* Link: [cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail) 
**Data Collection Method by dataset** <br>
* Automated <br>

**Labeling Method by dataset** <br>
* Automated <br>

**Properties:** <br>
* The cnn_dailymail dataset is an English-language dataset containing just over 300k unique news articles as written by journalists at CNN and the Daily Mail. 512 samples from the train split were used for calibration. <br>

## Training Dataset

### Data Modality:

* Image <br>

### Training Data Size:

**Image Training Data Size** <br>

* Undisclosed <br>

**Data Collection Method by dataset** <br>
* Not Applicable <br>

**Labeling Method by dataset** <br>
* Not Applicable <br>

**Properties:** <br>
* Not Applicable <br>

## Evaluation Dataset

In this section, we compare the performance of quantized model **llama-nemotron-rerank-vl-1b-v2-fp8** with baseline implementation **llama-nemotron-rerank-vl-1b-v2**. We applied the ranking model to the candidates retrieved from the `llama-nemotron-embed-vl-1b-v2` model.

ViDoRe V3, KoViDoRe, and ZhViDoRe (an internal Chinese visual document) retrieval benchmark were used to evaluate ranking quality relative to the BF16 baseline model. Evaluation inputs were formatted as text-only, image-only, and image-plus-text examples to compare retrieval accuracy between the FP8 checkpoint and the BF16 baseline.

We evaluate the embedding + reranking pipeline on a set of evaluation benchmarks. Both the baseline BF16 and the quantized FP8 checkpoints were provided with the same candidates retrieved from `llama-nemotron-embed-vl-1b-v2` BF16 model. Top 20 candidates were selected based on cosine similarity of embeddings, then rerank model was used to score relevancy of those top-20 documents and compute recall of top 5 scoring documents.

In each configuration, the embedding modality matched the reranking modality (e.g., text-only reranking used text-only embeddings, image-only reranking used image-only embeddings, and image-plus-text reranking used image-plus-text embeddings).

<table>
<caption><b><em>FP8 model accuracy relative to baseline BF16 model, on Visual Document Retrieval benchmarks. Chinese: ZhViDoRe (internal); Korean: KoViDoRe; English/French: ViDoRe V3.</em></b></caption>    
<thead>
  <tr>
    <th></th>
    <th colspan="3">Dataset</th>
  </tr></thead>
<tbody>
  <tr>
    <td>Modality</td>
    <td>All</td>
    <td>Chinese/Korean</td>
    <td>English/French</td>
  </tr>
  <tr>
    <td>image+text</td>
    <td>99.53%</td>
    <td>100.28%</td>
    <td>99.34%</td>
  </tr>
  <tr>
    <td>image</td>
    <td>99.68%</td>
    <td> - </td>
    <td>99.68%</td>
  </tr>
  <tr>
    <td>text</td>
    <td>99.03%</td>
    <td>98.88%</td>
    <td>99.07%</td>
  </tr>
</tbody>
</table>

Note that the model does not have Chinese/Korean language support for image-only inputs.

**Data Collection Method by dataset** <br> 
* Hybrid: Automated, Human, Synthetic <br>

**Labeling Method by dataset:** <br>
* Hybrid: Automated, Human, Synthetic <br>

**Properties** <br>
* ZhViDoRe comprises 922 queries, KoViDoRe comprises 706 queries, and ViDoRe V3 comprises 14,514 queries. More details on ViDoRe benchmarks can be found on  their Hugging Face [page](https://huggingface.co/vidore).

## Inference

**Acceleration Engine:** [vLLM](https://vllm.ai/) <br>
**Test Hardware:** <br>
* NVIDIA Hopper (H100 SXM) <br>


## **Ethical Considerations**

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>

Please make sure you have proper rights and permissions for all input image content; if image includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included. <br>

For more detailed information on ethical considerations for this model, please see the Explainability, Bias, Safety & Security, and Privacy sections.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).

## Citation
```
@inproceedings{moreira2025_nvretriever,
author = {Moreira, Gabriel de Souza P. and Osmulski, Radek and Xu, Mengyao and Ak, Ronay and Schifferer, Benedikt and Oldridge, Even},
title = {Improving Text Embedding Models with Positive-aware Hard-negative Mining},
year = {2025},
isbn = {9798400720406},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3746252.3761254},
doi = {10.1145/3746252.3761254},
pages = {2169–2178},
numpages = {10},
keywords = {contrastive learning, distillation, embedding models, hard-negative mining, rag, text retrieval, transformers},
location = {Seoul, Republic of Korea},
series = {CIKM '25}
}
```

## Bias

| Field | Response |
| ----- | ----- |
| Participation considerations from adversely impacted groups [protected classes](https://www.senate.ca.gov/content/protected-classes) in model design and testing | None |
| Measures taken to mitigate against unwanted bias | None |
| Bias Metric (If Measured) | None |

## Explainability

| Field | Response |
| ----- | ----- |
| Intended Application & Domain: | Passage or document ranking for question and answer retrieval. |
| Model Type: | Transformer cross-encoder. |
| Intended User: | Generative AI developers working with conversational AI models. This is suitable for users building question-answering applications over large multimodal corpora and aiming to improve retrieval performance by reranking a set of candidate documents for a given question. The corpus may include visually rich document images (e.g., pages with text, figures, tables, charts, or infographics) and/or text extracted from documents. |
| Output: | List of Floats (Score/Logit indicating if a passage/document relevant to a question). |
| Describe how the model works: | The model outputs a relevance score (logit) reflecting how strongly it predicts that the passage or document contains the information required to answer the question. |
| Technical Limitations: | The model's max sequence length is 10240. Longer text inputs should be truncated. |
| Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of: | N/A |
| Verified to have met prescribed NVIDIA quality standards: | Yes |
| Performance Metrics: | Accuracy, Throughput, and Latency. |
| Potential Known Risks: | This model does not always guarantee to provide a meaningful ranking of document(s) for a given query. |
| Licensing & Terms of Use: | The use of this model is governed by the [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/) and the use of the post-processing scripts are licensed under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0.txt). Additional Information: [Llama 3.2 Community Model License Agreement](https://www.llama.com/llama3_2/license/). Built with Llama. |

## Privacy

| Field | Response |
| ----- | ----- |
| Generatable or reverse engineerable personal data? | No |
| Personal data used to create this model? | None Known |
| Was consent obtained for any personal data used? | Not Applicable |
| How often is dataset reviewed? | Dataset is initially reviewed upon addition, and subsequent reviews are conducted as needed or upon request for changes. |
| Is there provenance for all datasets used in training? | Yes |
| Does data labeling (annotation, metadata) comply with privacy laws? | Yes |
| Is data compliant with data subject requests for data correction or removal, if such a request was made? | No, not possible with externally-sourced data. |
| Was data from user interactions with the AI model (e.g. user input and prompts) used to train the model? | No |
| Was consent obtained for any personal data used? | Not Applicable |
| Applicable Privacy Policy | https://www.nvidia.com/en-us/about-nvidia/privacy-policy/ |

## Safety & Security

| Field | Response |
| ----- | ----- |
| Model Application(s): | Document Reranking for Retrieval. User queries can be text and documents can be text, document page images, charts, tables, and infographics. |
| Describe the physical safety impact (if present). | Not Applicable |
| Use Case Restrictions: | The use of this model is governed by the [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/) and the use of the post-processing scripts are licensed under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0.txt). Additional Information: [Llama 3.2 Community Model License Agreement](https://www.llama.com/llama3_2/license/). Built with Llama. |
| Model and dataset restrictions: | The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training, and dataset license constraints adhered to. |