Update model card with clearer usage and eval details

README.md

@@ -31,20 +31,33 @@ The model is built on top of `microsoft/deberta-v3-base` and published with cust
 - Backbone hidden size: 768
 - Saved dtype: `float32`
 
-## What This Model Expects As Input
+## Quick Access
 
-The model takes plain text and returns four quality scores.
+Model page: <https://huggingface.co/QCRI/OmniScore-deberta-v3>
 
-For best behavior, format inputs consistently with your evaluation setup, for example:
+```python
+from transformers import AutoTokenizer, AutoModel
+
+repo_id = "QCRI/OmniScore-deberta-v3"
+tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
+model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)
+```
+
+## What Input To Provide
+
+The model takes a single text string and returns four quality scores.
+For best results, keep a consistent prompt/input format during inference.
+
+Recommended flat format:
 
 ```text
-Task: headline_evaluation
-Input: Headline: ...
-Context: Full source text...
-Candidate: Generated headline...
+Task: <task_name>
+Source: <source text, if available>
+Reference: <reference text, if available>
+Candidate: <model output being evaluated>
 ```
 
-If your data is chat-style, you can flatten messages as:
+Chat-style input can be flattened as:
 
 ```text
 System: ...
@@ -52,7 +65,7 @@ User: ...
 Assistant: ...
 ```
 
-## Quickstart
+## Usage Examples
 
 Install dependencies:
 
@@ -60,7 +73,7 @@ Install dependencies:
 pip install -U torch transformers sentencepiece
 ```
 
-Load directly from Hugging Face:
+### 1) Single Text Example
 
 ```python
 import torch
@@ -69,20 +82,13 @@ from transformers import AutoTokenizer, AutoModel
 repo_id = "QCRI/OmniScore-deberta-v3"
 
 tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
-model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)
-model.eval()
+model = AutoModel.from_pretrained(repo_id, trust_remote_code=True).eval()
 
 text = """Task: headline_evaluation
-Input: Headline: Microsoft announces new model card.
-Context: Full article text goes here.
+Source: Full article text goes here.
 Candidate: Microsoft releases detailed model documentation."""
 
-inputs = tokenizer(
-    text,
-    return_tensors="pt",
-    truncation=True,
-    max_length=512,
-)
+inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
 
 with torch.no_grad():
     outputs = model(**inputs)
@@ -91,12 +97,10 @@ scores = {
     name: float(outputs.predictions[0, i])
     for i, name in enumerate(model.config.score_names)
 }
-
 print(scores)
-# {'informativeness': ..., 'clarity': ..., 'plausibility': ..., 'faithfulness': ...}
 ```
 
-### Batch Inference Example
+### 2) Batch Example (GPU/CPU)
 
 ```python
 import torch
@@ -109,31 +113,48 @@ tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
 model = AutoModel.from_pretrained(repo_id, trust_remote_code=True).to(device).eval()
 
 texts = [
-    "Task: summarization\\nInput: ...\\nCandidate: ...",
-    "Task: translation_evaluation\\nInput: ...\\nCandidate: ...",
+    "Task: summarization\nSource: ...\nCandidate: ...",
+    "Task: translation_evaluation\nSource: ...\nReference: ...\nCandidate: ...",
 ]
 
-batch = tokenizer(
-    texts,
-    return_tensors="pt",
-    truncation=True,
-    padding=True,
-    max_length=512,
-).to(device)
+batch = tokenizer(texts, return_tensors="pt", truncation=True, padding=True, max_length=512)
+batch = {k: v.to(device) for k, v in batch.items()}
 
 with torch.no_grad():
     pred = model(**batch).predictions
 
 results = []
 for row in pred.cpu():
-    results.append({
-        name: float(row[i])
-        for i, name in enumerate(model.config.score_names)
-    })
+    results.append({name: float(row[i]) for i, name in enumerate(model.config.score_names)})
 
 print(results)
 ```
 
+### 3) Chat Messages Helper
+
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch
+
+repo_id = "QCRI/OmniScore-deberta-v3"
+tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
+model = AutoModel.from_pretrained(repo_id, trust_remote_code=True).eval()
+
+messages = [
+    {"role": "system", "content": "You are a helpful assistant."},
+    {"role": "user", "content": "Write a concise summary of this article."},
+    {"role": "assistant", "content": "Here is a short summary..."},
+]
+
+flat_text = " ".join([f"{m['role'].capitalize()}: {m['content']}" for m in messages])
+inputs = tokenizer(flat_text, return_tensors="pt", truncation=True, max_length=512)
+
+with torch.no_grad():
+    outputs = model(**inputs)
+
+print(dict((n, float(outputs.predictions[0, i])) for i, n in enumerate(model.config.score_names)))
+```
+
 ### Programmatic Download (Optional)
 
 ```python
@@ -145,7 +166,7 @@ print(local_dir)
 
 ## Evaluation Summary
 
-Metrics below are from `metrics_final.json` on a held-out test set (`num_samples = 17,175`).
+Metrics below are from `metrics_final.json` on a held-out test set (`num_samples = 17175`).
 
 | Dimension | RMSE | MAE | Pearson r | Spearman rho | Acc@0.5 | Acc@1.0 |
 |---|---:|---:|---:|---:|---:|---:|
@@ -155,15 +176,16 @@ Metrics below are from `metrics_final.json` on a held-out test set (`num_samples
 | plausibility | 1.0431 | 0.7889 | 0.1183 | 0.0289 | 0.3750 | 0.8298 |
 | faithfulness | 0.9120 | 0.7326 | 0.1803 | 0.1327 | 0.3856 | 0.7766 |
 
-Additional notes from evaluation artifacts:
+Additional values:
 
 - Label range: `[1.0, 5.0]`
 - Prediction range: `[1.2022, 4.9631]`
-- Exact match on all four scores: `0.0293`
+- Exact match (all four scores): `0.0293`
+
 
 ## Data and Task Coverage
 
-This checkpoint is for multi-task text quality scoring and is evaluated on a mixed test set of 17,175 examples covering:
+This checkpoint is for multi-task text quality scoring and is evaluated on a mixed test set covering:
 
 - Chat evaluation
 - Headline evaluation
@@ -174,6 +196,17 @@ This checkpoint is for multi-task text quality scoring and is evaluated on a mix
 
 The underlying project data is multilingual and multi-domain.
 
+## Intended Use
+
+Use this model to score generated text quality (or response quality) as a supporting signal in:
+
+- evaluation dashboards
+- ranking experiments
+- offline model comparison
+- human-in-the-loop workflows
+
+Not intended as a sole decision maker for high-stakes or safety-critical settings.
+
 ## Limitations
 
 - Scores are continuous estimates and should not be treated as absolute truth.