README.md

metadata

library_name: transformers
tags:
  - deberta-v3
  - regression
  - text-evaluation
  - multilingual
  - trust-remote-code

OmniScore DeBERTa-v3

QCRI/OmniScore-deberta-v3 is a multi-output regression model for automatic text quality evaluation. It predicts four scalar scores in the range [1, 5]:

• informativeness
• clarity
• plausibility
• faithfulness

The model is built on top of microsoft/deberta-v3-base and published with custom model code (AutoModel + trust_remote_code=True).

Model Details

• Base model: microsoft/deberta-v3-base
• Architecture: ScorePredictorModel (custom transformers model)
• Model type: encoder-only text regression
• Max sequence length: 512
• Number of outputs: 4
• Output range: [1, 5] (sigmoid-scaled in model head)
• Backbone hidden size: 768
• Saved dtype: float32

Quick Access

Model page: https://huggingface.co/QCRI/OmniScore-deberta-v3

from transformers import AutoTokenizer, AutoModel

repo_id = "QCRI/OmniScore-deberta-v3"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)

What Input To Provide

The model takes a single text string and returns four quality scores. For best results, keep a consistent prompt/input format during inference.

Recommended flat format:

Task: <task_name>
Source: <source text, if available>
Reference: <reference text, if available>
Candidate: <model output being evaluated>

Chat-style input can be flattened as:

System: ...
User: ...
Assistant: ...

Usage Examples

Install dependencies:

pip install -U torch transformers sentencepiece

1) Single Text Example

import torch
from transformers import AutoTokenizer, AutoModel

repo_id = "QCRI/OmniScore-deberta-v3"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True).eval()

text = """Task: headline_evaluation
Source: Full article text goes here.
Candidate: Microsoft releases detailed model documentation."""

inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    outputs = model(**inputs)

scores = {
    name: float(outputs.predictions[0, i])
    for i, name in enumerate(model.config.score_names)
}
print(scores)

2) Batch Example (GPU/CPU)

import torch
from transformers import AutoTokenizer, AutoModel

repo_id = "QCRI/OmniScore-deberta-v3"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True).to(device).eval()

texts = [
    "Task: summarization\nSource: ...\nCandidate: ...",
    "Task: translation_evaluation\nSource: ...\nReference: ...\nCandidate: ...",
]

batch = tokenizer(texts, return_tensors="pt", truncation=True, padding=True, max_length=512)
batch = {k: v.to(device) for k, v in batch.items()}

with torch.no_grad():
    pred = model(**batch).predictions

results = []
for row in pred.cpu():
    results.append({name: float(row[i]) for i, name in enumerate(model.config.score_names)})

print(results)

3) Chat Messages Helper

from transformers import AutoTokenizer, AutoModel
import torch

repo_id = "QCRI/OmniScore-deberta-v3"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True).eval()

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Write a concise summary of this article."},
    {"role": "assistant", "content": "Here is a short summary..."},
]

flat_text = " ".join([f"{m['role'].capitalize()}: {m['content']}" for m in messages])
inputs = tokenizer(flat_text, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    outputs = model(**inputs)

print(dict((n, float(outputs.predictions[0, i])) for i, n in enumerate(model.config.score_names)))

Programmatic Download (Optional)

from huggingface_hub import snapshot_download

local_dir = snapshot_download("QCRI/OmniScore-deberta-v3")
print(local_dir)

Evaluation Summary

Metrics below are from metrics_final.json on a held-out test set (num_samples = 17175).

Dimension	RMSE	MAE	Pearson r	Spearman rho	Acc@0.5	Acc@1.0
overall	1.1581	0.8992	0.1098	0.0636	0.3543	0.7164
informativeness	1.5689	1.2259	0.0120	0.0140	0.3001	0.4880
clarity	1.1084	0.8495	0.1285	0.0789	0.3566	0.7711
plausibility	1.0431	0.7889	0.1183	0.0289	0.3750	0.8298
faithfulness	0.9120	0.7326	0.1803	0.1327	0.3856	0.7766

Additional values:

• Label range: [1.0, 5.0]
• Prediction range: [1.2022, 4.9631]
• Exact match (all four scores): 0.0293

Data and Task Coverage

This checkpoint is for multi-task text quality scoring and is evaluated on a mixed test set covering:

• Chat evaluation
• Headline evaluation
• Paraphrase evaluation
• QA evaluation
• Summarization evaluation
• Translation evaluation

The underlying project data is multilingual and multi-domain.

Intended Use

Use this model to score generated text quality (or response quality) as a supporting signal in:

• evaluation dashboards
• ranking experiments
• offline model comparison
• human-in-the-loop workflows

Not intended as a sole decision maker for high-stakes or safety-critical settings.

Limitations

• Scores are continuous estimates and should not be treated as absolute truth.
• Performance differs by task, language, and domain.
• The model can inherit annotation noise and dataset biases.
• Long inputs beyond 512 tokens are truncated.
• Low correlation metrics on some dimensions indicate that rank ordering can be weak for certain subsets.

Responsible Use

Recommended:

• Use as a decision-support signal, not as a sole decision maker.
• Calibrate thresholds on your own validation set before production use.
• Monitor by language/task slices for fairness and reliability.

Not recommended:

• High-stakes automated decisions without human oversight.
• Out-of-domain deployment without re-validation.

Reproducibility Notes

Published artifacts include:

• model.safetensors
• config.json
• configuration_score_predictor.py
• modeling_score_predictor.py
• tokenizer files
• metrics_final.json
• predictions.jsonl

Load with trust_remote_code=True because the architecture is custom.

Citation

If you use this model, please cite the project/repository and this model URL:

@misc{qcri_omniscore_deberta_v3,
  title        = {OmniScore DeBERTa-v3},
  author       = {QCRI},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/QCRI/OmniScore-deberta-v3}}
}