Add complete model card with usage examples

README.md

@@ -1,3 +1,223 @@
----
-license: cc-by-sa-4.0
----
+---
+library_name: transformers
+tags:
+- deberta-v3
+- regression
+- text-evaluation
+- multilingual
+- trust-remote-code
+---
+
+# OmniScore DeBERTa-v3
+
+`QCRI/OmniScore-deberta-v3` is a multi-output regression model for automatic text quality evaluation.
+It predicts four scalar scores in the range `[1, 5]`:
+
+- `informativeness`
+- `clarity`
+- `plausibility`
+- `faithfulness`
+
+The model is built on top of `microsoft/deberta-v3-base` and published with custom model code (`AutoModel` + `trust_remote_code=True`).
+
+## Model Details
+
+- Base model: `microsoft/deberta-v3-base`
+- Architecture: `ScorePredictorModel` (custom `transformers` model)
+- Model type: encoder-only text regression
+- Max sequence length: 512
+- Number of outputs: 4
+- Output range: `[1, 5]` (sigmoid-scaled in model head)
+- Backbone hidden size: 768
+- Saved dtype: `float32`
+
+## What This Model Expects As Input
+
+The model takes plain text and returns four quality scores.
+
+For best behavior, format inputs consistently with your evaluation setup, for example:
+
+```text
+Task: headline_evaluation
+Input: Headline: ...
+Context: Full source text...
+Candidate: Generated headline...
+```
+
+If your data is chat-style, you can flatten messages as:
+
+```text
+System: ...
+User: ...
+Assistant: ...
+```
+
+## Quickstart
+
+Install dependencies:
+
+```bash
+pip install -U torch transformers sentencepiece
+```
+
+Load directly from Hugging Face:
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModel
+
+repo_id = "QCRI/OmniScore-deberta-v3"
+
+tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
+model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)
+model.eval()
+
+text = """Task: headline_evaluation
+Input: Headline: Microsoft announces new model card.
+Context: Full article text goes here.
+Candidate: Microsoft releases detailed model documentation."""
+
+inputs = tokenizer(
+    text,
+    return_tensors="pt",
+    truncation=True,
+    max_length=512,
+)
+
+with torch.no_grad():
+    outputs = model(**inputs)
+
+scores = {
+    name: float(outputs.predictions[0, i])
+    for i, name in enumerate(model.config.score_names)
+}
+
+print(scores)
+# {'informativeness': ..., 'clarity': ..., 'plausibility': ..., 'faithfulness': ...}
+```
+
+### Batch Inference Example
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModel
+
+repo_id = "QCRI/OmniScore-deberta-v3"
+device = "cuda" if torch.cuda.is_available() else "cpu"
+
+tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
+model = AutoModel.from_pretrained(repo_id, trust_remote_code=True).to(device).eval()
+
+texts = [
+    "Task: summarization\\nInput: ...\\nCandidate: ...",
+    "Task: translation_evaluation\\nInput: ...\\nCandidate: ...",
+]
+
+batch = tokenizer(
+    texts,
+    return_tensors="pt",
+    truncation=True,
+    padding=True,
+    max_length=512,
+).to(device)
+
+with torch.no_grad():
+    pred = model(**batch).predictions
+
+results = []
+for row in pred.cpu():
+    results.append({
+        name: float(row[i])
+        for i, name in enumerate(model.config.score_names)
+    })
+
+print(results)
+```
+
+### Programmatic Download (Optional)
+
+```python
+from huggingface_hub import snapshot_download
+
+local_dir = snapshot_download("QCRI/OmniScore-deberta-v3")
+print(local_dir)
+```
+
+## Evaluation Summary
+
+Metrics below are from `metrics_final.json` on a held-out test set (`num_samples = 17,175`).
+
+| Dimension | RMSE | MAE | Pearson r | Spearman rho | Acc@0.5 | Acc@1.0 |
+|---|---:|---:|---:|---:|---:|---:|
+| overall | 1.1581 | 0.8992 | 0.1098 | 0.0636 | 0.3543 | 0.7164 |
+| informativeness | 1.5689 | 1.2259 | 0.0120 | 0.0140 | 0.3001 | 0.4880 |
+| clarity | 1.1084 | 0.8495 | 0.1285 | 0.0789 | 0.3566 | 0.7711 |
+| plausibility | 1.0431 | 0.7889 | 0.1183 | 0.0289 | 0.3750 | 0.8298 |
+| faithfulness | 0.9120 | 0.7326 | 0.1803 | 0.1327 | 0.3856 | 0.7766 |
+
+Additional notes from evaluation artifacts:
+
+- Label range: `[1.0, 5.0]`
+- Prediction range: `[1.2022, 4.9631]`
+- Exact match on all four scores: `0.0293`
+
+## Data and Task Coverage
+
+This checkpoint is for multi-task text quality scoring and is evaluated on a mixed test set of 17,175 examples covering:
+
+- Chat evaluation
+- Headline evaluation
+- Paraphrase evaluation
+- QA evaluation
+- Summarization evaluation
+- Translation evaluation
+
+The underlying project data is multilingual and multi-domain.
+
+## Limitations
+
+- Scores are continuous estimates and should not be treated as absolute truth.
+- Performance differs by task, language, and domain.
+- The model can inherit annotation noise and dataset biases.
+- Long inputs beyond 512 tokens are truncated.
+- Low correlation metrics on some dimensions indicate that rank ordering can be weak for certain subsets.
+
+## Responsible Use
+
+Recommended:
+
+- Use as a decision-support signal, not as a sole decision maker.
+- Calibrate thresholds on your own validation set before production use.
+- Monitor by language/task slices for fairness and reliability.
+
+Not recommended:
+
+- High-stakes automated decisions without human oversight.
+- Out-of-domain deployment without re-validation.
+
+## Reproducibility Notes
+
+Published artifacts include:
+
+- `model.safetensors`
+- `config.json`
+- `configuration_score_predictor.py`
+- `modeling_score_predictor.py`
+- tokenizer files
+- `metrics_final.json`
+- `predictions.jsonl`
+
+Load with `trust_remote_code=True` because the architecture is custom.
+
+## Citation
+
+If you use this model, please cite the project/repository and this model URL:
+
+```bibtex
+@misc{qcri_omniscore_deberta_v3,
+  title        = {OmniScore DeBERTa-v3},
+  author       = {QCRI},
+  year         = {2026},
+  howpublished = {\url{https://huggingface.co/QCRI/OmniScore-deberta-v3}}
+}
+```