File size: 4,022 Bytes
d482f6b 55a0162 d482f6b 55a0162 d482f6b 55a0162 d482f6b 55a0162 d482f6b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 | ---
license: cc-by-nc-4.0
base_model: answerdotai/ModernBERT-base
library_name: transformers
pipeline_tag: text-classification
tags:
- rag
- governance
- pyrrho
- fitz-gov-v2
- modernbert
- multi-label-classification
---
# pyrrho-v2-nano-g1
`pyrrho-v2-nano-g1` is the Pyrrho v2 ModernBERT-base classifier for Fitz RAG
governance. It emits four native v2 heads:
- `evidence_verdict`: `INSUFFICIENT`, `DISPUTED`, `SUFFICIENT`
- `failure_mode`: `none`, `unresolved_conflict`, `missing_or_incomplete_evidence`, `wrong_scope_or_version`, `ambiguous_request`
- `retrieval_intents`: multi-label `needs_lookup`, `needs_temporal_resolution`, `needs_comparison_or_set`, `needs_broad_coverage`
- `evidence_kinds`: multi-label `needs_text`, `needs_table_or_record`, `needs_code_or_symbol`, `needs_config_or_setting`, `needs_log_or_run_result`, `needs_document_layout`
The model is intended for local governance in `fitz-sage`: decide whether
retrieved evidence is sufficient, insufficient, or disputed, and expose
actionable retrieval/failure metadata.
## Inputs
Governance input:
```text
Question: <user query>
Sources:
[1] <retrieved source text>
[2] <retrieved source text>
```
## Output Decoding
The 18 logits are not one flat softmax. Decode them by group:
```python
import torch
from transformers import AutoModelForSequenceClassification, PreTrainedTokenizerFast
model_id = "yafitzdev/pyrrho-v2-nano-g1"
tokenizer = PreTrainedTokenizerFast.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()
text = "Question: What is the capital of France?\n\nSources:\n[1] Paris is the capital of France."
encoded = tokenizer(text, return_tensors="pt", truncation=True, max_length=2048)
with torch.no_grad():
logits = model(**encoded).logits[0]
verdict_labels = ["INSUFFICIENT", "DISPUTED", "SUFFICIENT"]
failure_labels = [
"none",
"unresolved_conflict",
"missing_or_incomplete_evidence",
"wrong_scope_or_version",
"ambiguous_request",
]
intent_labels = [
"needs_lookup",
"needs_temporal_resolution",
"needs_comparison_or_set",
"needs_broad_coverage",
]
kind_labels = [
"needs_text",
"needs_table_or_record",
"needs_code_or_symbol",
"needs_config_or_setting",
"needs_log_or_run_result",
"needs_document_layout",
]
verdict = verdict_labels[int(torch.softmax(logits[0:3], dim=-1).argmax())]
failure = failure_labels[int(torch.softmax(logits[3:8], dim=-1).argmax())]
intents = [
label for label, score in zip(intent_labels, torch.sigmoid(logits[8:12]))
if float(score) >= 0.5
]
kinds = [
label for label, score in zip(kind_labels, torch.sigmoid(logits[12:18]))
if float(score) >= 0.5
]
```
## Training Snapshot
- Dataset: `fitz-gov-v2`
- Clean active training rows: 41,358
- Training source pointer: `fitz_gov_v2_41358_20260703`
- Base model: `answerdotai/ModernBERT-base`
- Seed: 42
## Local Evaluation
Held-out training eval from `outputs/modernbert_base_v2_alpha_41358_active_20260704_seed42`:
| Metric | Value |
| --- | ---: |
| overall score | 0.9497 |
| verdict accuracy | 0.9727 |
| false sufficient rate | 0.0455 |
| failure accuracy | 0.9601 |
| retrieval exact match | 0.8335 |
| retrieval macro F1 | 0.9300 |
| evidence-kind exact match | 0.9809 |
| evidence-kind macro F1 | 0.9950 |
Fitz-sage benchmark check for this release candidate:
| Benchmark | Result |
| --- | ---: |
| balanced fixed-evidence governance sanity suite | 120/120 |
| live fitz-sage benchmark | 86/120 |
The live benchmark result is the practical integration target; the fixed-evidence
suite is a minimal sanity check for the governance head.
## Artifacts
This repository contains:
- `model.safetensors`: Transformers checkpoint
- `model.onnx`: FP32 ONNX export
- `model_quantized.onnx`: INT8 dynamic ONNX export
- tokenizer/config files
- `manifest.json`: release metadata
## License
CC BY-NC 4.0. Free for research, evaluation, and personal use; commercial use
requires a separate license.
|