Model Overview

Description:

The NVIDIA llama-nemotron-rerank-vl-1b-v2-fp8 is the quantized version of the NVIDIA llama-nemotron-rerank-vl-1b-v2 which is developed for multimodal question-answering retrieval. For more information, please check here. The NVIDIA llama-nemotron-rerank-vl-1b-v2-fp8 model is quantized with TensorRT Model Optimizer.

For this quantization checkpoint we quantize only the language portion of the model, llama-3-1B. All other parts including vision encoder (SigLIP 400M) and the binary classification head are not quantized and are left in default bf-16 or fp-32 precision.

This model is ready for commercial use.

License/Terms of Use:

The use of this model is governed by the NVIDIA Open Model License Agreement and the use of the post-processing scripts are licensed under Apache 2.0. Additional Information: Llama 3.2 Community Model License Agreement. Built with Llama. |

Deployment Geography:

Global

Use Case:

The llama-nemotron-rerank-vl-1b-v2-fp8 is suitable for users who want to build a multimodal question-and-answer application over a large corpus, leveraging the latest dense retrieval technology.

The input of the model is a query combined with a text or image document and the output is a float representing the relevance score (logit) reflecting how strongly it predicts that the passage or document contains the information relevant to the query. Multiple documents could be passed with a single query for batched processing, in this case the output contains a logit per document.

The rerank model is a cross-encoder that supports context in textual format (e.g. the query or the OCR text of a page or a section of a document) or the image of a document page.

Typically, the embedding model is used first to embed (vectorize) the whole corpus (document images or text chunks), and embeddings are stored in a vector database associated to its raw content (image or text). Then at inference time, the embedding model is used to embed the query and pick closest documents based on the cosine similarity of document and query embeddings. Finally, the rerank model is used on the query and top selections from the embedding model to improve the selection document relevance scores with better context information.

Release Date

Hugging Face 07/28/2026 via https://huggingface.co/nvidia/llama-nemotron-rerank-vl-1b-v2-fp8

Model Architecture:

Architecture Type: Transformer
Network Architecture: : Eagle VLM architecture with SigLIP 2 400M vision encoder and llama-nemotron-rerank-1b-v2 model as language model.

The llama-nemotron-rerank-vl-1b-v2 is a cross-encoder model with approximately 1.7B parameters. It is a fine-tuned version of an NVIDIA Eagle-family model, which consists of the SigLIP 2 400M vision encoder and the Llama 3.2 1B language model. The final embedding output by the decoder is aggregated using a mean pooling strategy, and a binary classification head is fine-tuned for the ranking task. The CrossEntropy loss is used to maximize the likelihood of (visual) documents containing information to answer the question and minimize the likelihood for (negative) documents that do not contain information to answer the question.

The vision-language model reranker incorporates key innovations from NVIDIA, including Eagle 2 work which uses a tiling-based VLM architecture, and nemoretriever-parse. The Eagle 2 architecture, available on Hugging Face, significantly enhances multimodal understanding through its dynamic tiling and mixture of vision encoders design. It particularly improves performance on tasks that involve high-resolution images and complex visual content.

Number of model parameters:

Llama 3.2 1B language model: 1.23 B (Transformer parameters: 973 M, Token embedding parameters: 262 M)
SigLip 2 image encoder: 428.77 M

Input(s):

Input Type(s): Image, Text

Input Format(s):

Image: Red, Green, Blue (RGB)
Text: String

Input Parameters:

Image: Two-Dimensional (2D)
Text: One-Dimensional (1D)

Other Properties Related to Input:

The model was fine-tuned exclusively on image data, using max_input_tiles = 4 and the maximum context length of 2048 tokens. For evaluation, it was tested on image-only, image+text, and text-only inputs, with max_input_tiles = 6 and the maximum context length of 10240 tokens. Inputs exceeding the maximum length are truncated.

Output(s)

Output Type(s): Floats

Output Format(s): List of Floats

Output Parameters: One-Dimensional (1D)

Other Properties Related to Output: Each value corresponds to a raw logit. Users can choose to apply a Sigmoid activation function to the logits to convert them into probabilities during model usage.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

vLLM Usage

The model can be served with vLLM for high-throughput reranking. A chat template must be provided via --chat-template to correctly apply the question: / passage: prefixes based on the message role — without it, the prompt template is not applied and results will be incorrect.

See the vLLM documentation for full details.

Online Serving

Create the score template file and start the server:

cat > nemotron-rerank-vl.jinja << 'JINJA'
{%- set query_msg = (messages | selectattr("role", "equalto", "query") | list | first) -%} 
{%- set doc_msg   = (messages | selectattr("role", "equalto", "document") | list | first) -%} 
{%- set q = query_msg["content"] -%} 
{%- set d = doc_msg["content"] -%} 
{# If the doc contains <image> anywhere, hoist a single <image> to the front #} 
{%- set has_image = ("<image>" in d) -%} 
{%- set d_clean = d | replace("<image>", "") -%} 
{%- set q_clean = q | replace("<image>", "") -%} 
{%- if has_image -%}<image>{{ " " }}{%- endif -%} 
question:{{ q_clean }}{{ " " }} 
{{ " " }} 
{{ " " }}passage:{{ d_clean }} 
JINJA

vllm serve nvidia/llama-nemotron-rerank-vl-1b-v2-fp8 \
  --runner pooling \
  --trust-remote-code \
  --max-model-len 10240 \
  --chat-template nemotron-rerank-vl.jinja

Note: Use --max-model-len 10240 to support all modalities including image+text. A smaller value like 2048 can be used if only processing image-only or text-only inputs.

The chat template uses the message role to combine the correct prefixes: set role to "query" for queries (prepends question:) and "document" for passages (prepends passage:).

Send a rerank request:

import requests

url = "http://localhost:8000/v1/rerank"

# Text-only reranking
response = requests.post(url, json={
    "model": "nvidia/llama-nemotron-rerank-vl-1b-v2-fp8",
    "query": "How is AI improving the intelligence and capabilities of robots?",
    "documents": [
        "AI enables robots to perceive, plan, and act autonomously.",
        "A biological foundation model designed to analyze DNA, RNA, and protein sequences.",
    ],
})
print(response.json())

# Image document reranking
response = requests.post(url, json={
    "model": "nvidia/llama-nemotron-rerank-vl-1b-v2-fp8",
    "query": "How is AI improving the intelligence and capabilities of robots?",
    "documents": [
        {"content": [{"type": "image_url", "image_url": {"url": "https://example.com/page.png"}}]},
        {"content": [
            {"type": "text", "text": "AI enables robots to perceive, plan, and act autonomously."},
            {"type": "image_url", "image_url": {"url": "https://example.com/page2.png"}},
        ]},
    ],
})
print(response.json())

Offline / In-Process

from vllm import LLM

SCORE_TEMPLATE = """\
{%- set query_msg = (messages | selectattr('role', 'equalto', 'query') | list | first) -%}
{%- set doc_msg   = (messages | selectattr('role', 'equalto', 'document') | list | first) -%}
{%- set q = query_msg['content'] -%}
{%- set d = doc_msg['content'] -%}
{%- set has_image = ("<image>" in d) -%}
{%- set d_clean = d | replace("<image>", "") -%}
{%- set q_clean = q | replace("<image>", "") -%}
{%- if has_image -%}<image>{{ " " }}{%- endif -%}
question:{{ q_clean }}{{ " " }}
{{ " " }}
{{ " " }}passage:{{ d_clean }}"""

llm = LLM(
    model="nvidia/llama-nemotron-rerank-vl-1b-v2-fp8",
    runner="pooling",
    max_model_len=10240,
    trust_remote_code=True,
)

query = "How is AI improving the intelligence and capabilities of robots?"
documents = [
    "AI enables robots to perceive, plan, and act autonomously.",
    "A biological foundation model designed to analyze DNA, RNA, and protein sequences.",
]

# Text-only scoring
outputs = llm.score(query, documents, chat_template=SCORE_TEMPLATE)
for doc, output in zip(documents, outputs):
    print(f"Score: {output.outputs.score:.4f} | {doc}")

# Image document scoring
outputs = llm.score(
    query,
    [
        {"content": [{"type": "image_url", "image_url": {"url": "https://example.com/page.png"}}]},
        {"content": [
            {"type": "text", "text": "AI enables robots to perceive, plan, and act autonomously."},
            {"type": "image_url", "image_url": {"url": "https://example.com/page2.png"}},
        ]},
    ],
    chat_template=SCORE_TEMPLATE,
)
for output in outputs:
    print(f"Score: {output.outputs.score:.4f}")

Software Integration:

Runtime Engine(s): vLLM
Supported Hardware Microarchitecture Compatibility: NVIDIA Blackwell, NVIDIA Hopper, NVIDIA Lovelace
Preferred/Supported Operating System(s): Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s):

llama-nemotron-rerank-vl-1b-v2-fp8
The model is quantized with nvidia-modelopt v0.43.0

Post Training Quantization:

This model was obtained by quantizing the weights and activations of nvidia/llama-nemotron-rerank-vl-1b-v2 to FP8 data type, ready for inference with vLLM

Training, Testing, and Evaluation Datasets:

Dataset Overview

This checkpoint is an FP8 post-training-quantized derivative of nvidia/llama-nemotron-rerank-vl-1b-v2. No additional supervised training or fine-tuning data was used to create this FP8 checkpoint.

For this FP8 release, data was used for two purposes: post-training quantization calibration and evaluation. The cnn_dailymail dataset was used for FP8 quantization calibration.

The FP8 checkpoint was produced by applying post-training quantization to the BF16 parent model using NVIDIA TensorRT Model Optimizer. Calibration samples were converted into model inputs and used to estimate quantization parameters for weights and activations. This calibration process did not involve supervised training, fine-tuning, or modification of the model objective.

Calibration dataset

Link: cnn_dailymail Data Collection Method by dataset
Automated

Labeling Method by dataset

Automated

Properties:

The cnn_dailymail dataset is an English-language dataset containing just over 300k unique news articles as written by journalists at CNN and the Daily Mail. 512 samples from the train split were used for calibration.

Training Dataset

Data Modality:

Image

Training Data Size:

Image Training Data Size

Undisclosed

Data Collection Method by dataset

Not Applicable

Labeling Method by dataset

Not Applicable

Properties:

Not Applicable

Evaluation Dataset

In this section, we compare the performance of quantized model llama-nemotron-rerank-vl-1b-v2-fp8 with baseline implementation llama-nemotron-rerank-vl-1b-v2. We applied the ranking model to the candidates retrieved from the llama-nemotron-embed-vl-1b-v2 model.

ViDoRe V3, KoViDoRe, and ZhViDoRe (an internal Chinese visual document) retrieval benchmark were used to evaluate ranking quality relative to the BF16 baseline model. Evaluation inputs were formatted as text-only, image-only, and image-plus-text examples to compare retrieval accuracy between the FP8 checkpoint and the BF16 baseline.

We evaluate the embedding + reranking pipeline on a set of evaluation benchmarks. Both the baseline BF16 and the quantized FP8 checkpoints were provided with the same candidates retrieved from llama-nemotron-embed-vl-1b-v2 BF16 model. Top 20 candidates were selected based on cosine similarity of embeddings, then rerank model was used to score relevancy of those top-20 documents and compute recall of top 5 scoring documents.

In each configuration, the embedding modality matched the reranking modality (e.g., text-only reranking used text-only embeddings, image-only reranking used image-only embeddings, and image-plus-text reranking used image-plus-text embeddings).

***FP8 model accuracy relative to baseline BF16 model, on Visual Document Retrieval benchmarks. Chinese: ZhViDoRe (internal); Korean: KoViDoRe; English/French: ViDoRe V3.***
	Dataset
Modality	All	Chinese/Korean	English/French
image+text	99.53%	100.28%	99.34%
image	99.68%	-	99.68%
text	99.03%	98.88%	99.07%

Note that the model does not have Chinese/Korean language support for image-only inputs.

Data Collection Method by dataset

Hybrid: Automated, Human, Synthetic

Labeling Method by dataset:

Hybrid: Automated, Human, Synthetic

Properties

ZhViDoRe comprises 922 queries, KoViDoRe comprises 706 queries, and ViDoRe V3 comprises 14,514 queries. More details on ViDoRe benchmarks can be found on their Hugging Face page.

Inference

Acceleration Engine: vLLM
Test Hardware:

NVIDIA Hopper (H100 SXM)

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please make sure you have proper rights and permissions for all input image content; if image includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.

For more detailed information on ethical considerations for this model, please see the Explainability, Bias, Safety & Security, and Privacy sections.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.

Citation

@inproceedings{moreira2025_nvretriever,
author = {Moreira, Gabriel de Souza P. and Osmulski, Radek and Xu, Mengyao and Ak, Ronay and Schifferer, Benedikt and Oldridge, Even},
title = {Improving Text Embedding Models with Positive-aware Hard-negative Mining},
year = {2025},
isbn = {9798400720406},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3746252.3761254},
doi = {10.1145/3746252.3761254},
pages = {2169–2178},
numpages = {10},
keywords = {contrastive learning, distillation, embedding models, hard-negative mining, rag, text retrieval, transformers},
location = {Seoul, Republic of Korea},
series = {CIKM '25}
}

Bias

Field	Response
Participation considerations from adversely impacted groups protected classes in model design and testing	None
Measures taken to mitigate against unwanted bias	None
Bias Metric (If Measured)	None

Explainability

Field	Response
Intended Application & Domain:	Passage or document ranking for question and answer retrieval.
Model Type:	Transformer cross-encoder.
Intended User:	Generative AI developers working with conversational AI models. This is suitable for users building question-answering applications over large multimodal corpora and aiming to improve retrieval performance by reranking a set of candidate documents for a given question. The corpus may include visually rich document images (e.g., pages with text, figures, tables, charts, or infographics) and/or text extracted from documents.
Output:	List of Floats (Score/Logit indicating if a passage/document relevant to a question).
Describe how the model works:	The model outputs a relevance score (logit) reflecting how strongly it predicts that the passage or document contains the information required to answer the question.
Technical Limitations:	The model's max sequence length is 10240. Longer text inputs should be truncated.
Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of:	N/A
Verified to have met prescribed NVIDIA quality standards:	Yes
Performance Metrics:	Accuracy, Throughput, and Latency.
Potential Known Risks:	This model does not always guarantee to provide a meaningful ranking of document(s) for a given query.
Licensing & Terms of Use:	The use of this model is governed by the NVIDIA Open Model License Agreement and the use of the post-processing scripts are licensed under Apache 2.0. Additional Information: Llama 3.2 Community Model License Agreement. Built with Llama.

Privacy

Field	Response
Generatable or reverse engineerable personal data?	No
Personal data used to create this model?	None Known
Was consent obtained for any personal data used?	Not Applicable
How often is dataset reviewed?	Dataset is initially reviewed upon addition, and subsequent reviews are conducted as needed or upon request for changes.
Is there provenance for all datasets used in training?	Yes
Does data labeling (annotation, metadata) comply with privacy laws?	Yes
Is data compliant with data subject requests for data correction or removal, if such a request was made?	No, not possible with externally-sourced data.
Was data from user interactions with the AI model (e.g. user input and prompts) used to train the model?	No
Was consent obtained for any personal data used?	Not Applicable
Applicable Privacy Policy	https://www.nvidia.com/en-us/about-nvidia/privacy-policy/

Safety & Security

Field	Response
Model Application(s):	Document Reranking for Retrieval. User queries can be text and documents can be text, document page images, charts, tables, and infographics.
Describe the physical safety impact (if present).	Not Applicable
Use Case Restrictions:	The use of this model is governed by the NVIDIA Open Model License Agreement and the use of the post-processing scripts are licensed under Apache 2.0. Additional Information: Llama 3.2 Community Model License Agreement. Built with Llama.
Model and dataset restrictions:	The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training, and dataset license constraints adhered to.