Improve Pyrrho v2 model card

README.md

@@ -3,32 +3,126 @@ license: cc-by-nc-4.0
 base_model: answerdotai/ModernBERT-base
 library_name: transformers
 pipeline_tag: text-classification
+language:
+  - en
 tags:
   - rag
   - governance
+  - hallucination-detection
+  - evidence-verification
   - pyrrho
   - fitz-gov-v2
   - modernbert
   - multi-label-classification
+datasets:
+  - yafitzdev/fitz-gov-v2
+metrics:
+  - accuracy
+  - f1
 ---
 
 # pyrrho-v2-nano-g1
 
-`pyrrho-v2-nano-g1` is the Pyrrho v2 ModernBERT-base classifier for Fitz RAG
-governance. It emits four native v2 heads:
+`pyrrho-v2-nano-g1` is a small local RAG governance co-processor. It reads a user
+question plus retrieved source passages, then returns whether the evidence is
+`SUFFICIENT`, `DISPUTED`, or `INSUFFICIENT` before an answer is generated.
 
-- `evidence_verdict`: `INSUFFICIENT`, `DISPUTED`, `SUFFICIENT`
-- `failure_mode`: `none`, `unresolved_conflict`, `missing_or_incomplete_evidence`, `wrong_scope_or_version`, `ambiguous_request`
-- `retrieval_intents`: multi-label `needs_lookup`, `needs_temporal_resolution`, `needs_comparison_or_set`, `needs_broad_coverage`
-- `evidence_kinds`: multi-label `needs_text`, `needs_table_or_record`, `needs_code_or_symbol`, `needs_config_or_setting`, `needs_log_or_run_result`, `needs_document_layout`
+It is not an answer generator, not a retriever, and not an open-world fact
+checker. It sits between retrieval and generation, or beside a retrieval
+pipeline as a fast evidence-quality layer, so downstream systems can answer,
+show a dispute, retry retrieval, or ask for missing evidence.
 
-The model is intended for local governance in `fitz-sage`: decide whether
-retrieved evidence is sufficient, insufficient, or disputed, and expose
-actionable retrieval/failure metadata.
+Compared with the older Pyrrho v1 line, v2 exposes a smaller native head shape:
+one evidence verdict, one failure reason, and two compact multi-label metadata
+heads. The goal is a cleaner governance contract for `fitz-sage` and other RAG
+systems.
 
-## Inputs
+## Native V2 Heads
 
-Governance input:
+| Head | Labels / values | Intended use |
+|---|---|---|
+| `evidence_verdict` | `SUFFICIENT`, `DISPUTED`, `INSUFFICIENT` | Post-retrieval evidence sufficiency and conflict decision. |
+| `failure_mode` | `none`, `unresolved_conflict`, `missing_or_incomplete_evidence`, `wrong_scope_or_version`, `ambiguous_request` | Actionable reason when evidence is disputed or insufficient. |
+| `retrieval_intents` | `needs_lookup`, `needs_temporal_resolution`, `needs_comparison_or_set`, `needs_broad_coverage` | Query/evidence task metadata for retry and retrieval policy. |
+| `evidence_kinds` | `needs_text`, `needs_table_or_record`, `needs_code_or_symbol`, `needs_config_or_setting`, `needs_log_or_run_result`, `needs_document_layout` | Evidence-surface metadata for routing, audit, and missing-source hints. |
+
+## Output Contract
+
+The raw Hugging Face model output is an 18-logit vector. It is not one flat
+softmax. Decode it by head:
+
+| Logit slice | Head | Decoding |
+|---|---|---|
+| `0:3` | `evidence_verdict` | softmax over `INSUFFICIENT`, `DISPUTED`, `SUFFICIENT` |
+| `3:8` | `failure_mode` | softmax over the five failure labels |
+| `8:12` | `retrieval_intents` | sigmoid multi-label scores |
+| `12:18` | `evidence_kinds` | sigmoid multi-label scores |
+
+Most integrations should expose a structured decision object derived from those
+logits:
+
+| Field | Meaning |
+|---|---|
+| `evidence_verdict.final_label` | Final v2 verdict: `SUFFICIENT`, `DISPUTED`, or `INSUFFICIENT`. |
+| `evidence_verdict.probabilities` | Softmax probability distribution over the three verdict labels. |
+| `failure_mode.final_label` | Most likely failure reason, or `none` for sufficient evidence. |
+| `retrieval_intents.final_labels` | Intent labels above the configured sigmoid threshold. |
+| `evidence_kinds.final_labels` | Evidence-kind labels above the configured sigmoid threshold. |
+| `confidence` | Probability or score assigned to the selected label. |
+
+Example normalized output:
+
+```json
+{
+  "schema_version": "pyrrho_v2_prediction",
+  "evidence_verdict": {
+    "final_label": "DISPUTED",
+    "confidence": 0.86,
+    "probabilities": {
+      "INSUFFICIENT": 0.08,
+      "DISPUTED": 0.86,
+      "SUFFICIENT": 0.06
+    }
+  },
+  "failure_mode": {
+    "final_label": "unresolved_conflict",
+    "confidence": 0.81
+  },
+  "retrieval_intents": {
+    "final_labels": ["needs_comparison_or_set"],
+    "scores": {
+      "needs_comparison_or_set": 0.77
+    }
+  },
+  "evidence_kinds": {
+    "final_labels": ["needs_text", "needs_table_or_record"],
+    "scores": {
+      "needs_text": 0.91,
+      "needs_table_or_record": 0.63
+    }
+  }
+}
+```
+
+The model does not generate answers, citations, source spans, retrieval results,
+or natural-language explanations. It classifies and scores the `(query,
+retrieved_contexts)` evidence state.
+
+## Intended Use
+
+Use this model when a RAG or retrieval system needs fast local signals about:
+
+- whether retrieved evidence is enough to answer,
+- whether retrieved evidence contains an unresolved conflict,
+- why evidence is insufficient or disputed,
+- whether another retrieval pass should focus on lookup, time, comparison, or broad coverage,
+- which source surface appears relevant or missing,
+- how to log governance decisions for later audit.
+
+This model is not intended to verify facts outside the provided sources, replace
+a retriever, write answers, or replace human review in high-stakes settings.
+
+## Input Format
 
 ```text
 Question: <user query>
@@ -38,38 +132,29 @@ Sources:
 [2] <retrieved source text>
 ```
 
-## Output Decoding
-
-The 18 logits are not one flat softmax. Decode them by group:
+## Quick Start
 
 ```python
 import torch
-from transformers import AutoModelForSequenceClassification, PreTrainedTokenizerFast
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
 
-model_id = "yafitzdev/pyrrho-v2-nano-g1"
-tokenizer = PreTrainedTokenizerFast.from_pretrained(model_id)
-model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()
+MODEL_ID = "yafitzdev/pyrrho-v2-nano-g1"
 
-text = "Question: What is the capital of France?\n\nSources:\n[1] Paris is the capital of France."
-encoded = tokenizer(text, return_tensors="pt", truncation=True, max_length=2048)
-with torch.no_grad():
-    logits = model(**encoded).logits[0]
-
-verdict_labels = ["INSUFFICIENT", "DISPUTED", "SUFFICIENT"]
-failure_labels = [
+VERDICT_LABELS = ["INSUFFICIENT", "DISPUTED", "SUFFICIENT"]
+FAILURE_LABELS = [
     "none",
     "unresolved_conflict",
     "missing_or_incomplete_evidence",
     "wrong_scope_or_version",
     "ambiguous_request",
 ]
-intent_labels = [
+INTENT_LABELS = [
     "needs_lookup",
     "needs_temporal_resolution",
     "needs_comparison_or_set",
     "needs_broad_coverage",
 ]
-kind_labels = [
+KIND_LABELS = [
     "needs_text",
     "needs_table_or_record",
     "needs_code_or_symbol",
@@ -78,32 +163,61 @@ kind_labels = [
     "needs_document_layout",
 ]
 
-verdict = verdict_labels[int(torch.softmax(logits[0:3], dim=-1).argmax())]
-failure = failure_labels[int(torch.softmax(logits[3:8], dim=-1).argmax())]
+query = "Has the company achieved profitability?"
+contexts = [
+    "The company posted net income of $4 million in Q2.",
+    "The company recorded a quarterly loss of $12 million in Q3.",
+]
+text = "Question: " + query + "\n\nSources:\n" + "\n".join(
+    f"[{i}] {context}" for i, context in enumerate(contexts, start=1)
+)
+
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
+model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID).eval()
+
+inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=2048)
+with torch.no_grad():
+    logits = model(**inputs).logits[0]
+
+verdict_probs = torch.softmax(logits[0:3], dim=-1)
+failure_probs = torch.softmax(logits[3:8], dim=-1)
+intent_scores = torch.sigmoid(logits[8:12])
+kind_scores = torch.sigmoid(logits[12:18])
+
+verdict = VERDICT_LABELS[int(verdict_probs.argmax())]
+failure = FAILURE_LABELS[int(failure_probs.argmax())]
 intents = [
-    label for label, score in zip(intent_labels, torch.sigmoid(logits[8:12]))
+    label for label, score in zip(INTENT_LABELS, intent_scores)
     if float(score) >= 0.5
 ]
 kinds = [
-    label for label, score in zip(kind_labels, torch.sigmoid(logits[12:18]))
+    label for label, score in zip(KIND_LABELS, kind_scores)
     if float(score) >= 0.5
 ]
+
+print(verdict)
+print(failure)
+print(intents)
+print(kinds)
 ```
 
-## Training Snapshot
+## CPU ONNX
+
+The repository includes both FP32 and INT8 ONNX exports:
 
-- Dataset: `fitz-gov-v2`
-- Clean active training rows: 41,358
-- Training source pointer: `fitz_gov_v2_41358_20260703`
-- Base model: `answerdotai/ModernBERT-base`
-- Seed: 42
+- `model.onnx`
+- `model_quantized.onnx`
 
-## Local Evaluation
+For CPU inference, load `model_quantized.onnx` through `onnxruntime` or an
+Optimum ONNX runtime wrapper. Decode the resulting 18 logits using the same
+slices shown above.
+
+## Evaluation
 
 Held-out training eval from `outputs/modernbert_base_v2_alpha_41358_active_20260704_seed42`:
 
 | Metric | Value |
-| --- | ---: |
+|---|---:|
 | overall score | 0.9497 |
 | verdict accuracy | 0.9727 |
 | false sufficient rate | 0.0455 |
@@ -113,16 +227,26 @@ Held-out training eval from `outputs/modernbert_base_v2_alpha_41358_active_20260
 | evidence-kind exact match | 0.9809 |
 | evidence-kind macro F1 | 0.9950 |
 
-Fitz-sage benchmark check for this release candidate:
+Fitz-sage release-candidate checks:
 
 | Benchmark | Result |
-| --- | ---: |
+|---|---:|
 | balanced fixed-evidence governance sanity suite | 120/120 |
 | live fitz-sage benchmark | 86/120 |
 
-The live benchmark result is the practical integration target; the fixed-evidence
+The live benchmark result is the practical integration target. The fixed-evidence
 suite is a minimal sanity check for the governance head.
 
+## Training Data
+
+| Field | Value |
+|---|---|
+| Dataset | [`fitz-gov-v2`](https://huggingface.co/datasets/yafitzdev/fitz-gov-v2) |
+| Clean active training rows | 41,358 |
+| Training source pointer | `fitz_gov_v2_41358_20260703` |
+| Base model | `answerdotai/ModernBERT-base` |
+| Seed | 42 |
+
 ## Artifacts
 
 This repository contains:
@@ -133,6 +257,20 @@ This repository contains:
 - tokenizer/config files
 - `manifest.json`: release metadata
 
+## Limitations
+
+1. **Evidence-bounded judgment.** Pyrrho judges only the retrieved evidence it
+   is given. It does not retrieve new evidence or verify claims against outside
+   knowledge.
+2. **English synthetic training data.** The v2 dataset is English synthetic RAG
+   governance data. Multilingual behavior is not established.
+3. **Metadata heads are policy signals, not formal proof.** `retrieval_intents`
+   and `evidence_kinds` are useful routing and audit hints. They do not prove
+   SQL correctness, code execution behavior, or complete corpus coverage.
+4. **RAG integration still matters.** Bad retrieval can produce bad evidence
+   packs. Pyrrho can flag insufficiency or conflict, but it cannot recover
+   source material that was never retrieved.
+
 ## License
 
 CC BY-NC 4.0. Free for research, evaluation, and personal use; commercial use