yafitzdev commited on
Commit
948f050
·
verified ·
1 Parent(s): 64cd34f

Release dual PRE/POST pyrrho-v2-nano-g1

Browse files
Files changed (5) hide show
  1. README.md +47 -21
  2. manifest.json +49 -17
  3. model.onnx +2 -2
  4. model.safetensors +1 -1
  5. model_quantized.onnx +1 -1
README.md CHANGED
@@ -23,8 +23,9 @@ metrics:
23
 
24
  # pyrrho-v2-nano-g1
25
 
26
- `pyrrho-v2-nano-g1` is a small local RAG governance co-processor. It reads a user
27
- question plus retrieved source passages, then returns whether the evidence is
 
28
  `SUFFICIENT`, `DISPUTED`, or `INSUFFICIENT` before an answer is generated.
29
 
30
  It is not an answer generator, not a retriever, and not an open-world fact
@@ -34,8 +35,8 @@ show a dispute, retry retrieval, or ask for missing evidence.
34
 
35
  Compared with the older Pyrrho v1 line, v2 exposes a smaller native head shape:
36
  one evidence verdict, one failure reason, and two compact multi-label metadata
37
- heads. The goal is a cleaner governance contract for `fitz-sage` and other RAG
38
- systems.
39
 
40
  ## Native V2 Heads
41
 
@@ -43,7 +44,7 @@ systems.
43
  |---|---|---|
44
  | `evidence_verdict` | `SUFFICIENT`, `DISPUTED`, `INSUFFICIENT` | Post-retrieval evidence sufficiency and conflict decision. |
45
  | `failure_mode` | `none`, `unresolved_conflict`, `missing_or_incomplete_evidence`, `wrong_scope_or_version`, `ambiguous_request` | Actionable reason when evidence is disputed or insufficient. |
46
- | `retrieval_intents` | `needs_lookup`, `needs_temporal_resolution`, `needs_comparison_or_set`, `needs_broad_coverage` | Query/evidence task metadata for retry and retrieval policy. |
47
  | `evidence_kinds` | `needs_text`, `needs_table_or_record`, `needs_code_or_symbol`, `needs_config_or_setting`, `needs_log_or_run_result`, `needs_document_layout` | Evidence-surface metadata for routing, audit, and missing-source hints. |
48
 
49
  ## Output Contract
@@ -58,8 +59,9 @@ softmax. Decode it by head:
58
  | `8:12` | `retrieval_intents` | sigmoid multi-label scores |
59
  | `12:18` | `evidence_kinds` | sigmoid multi-label scores |
60
 
61
- Most integrations should expose a structured decision object derived from those
62
- logits:
 
63
 
64
  | Field | Meaning |
65
  |---|---|
@@ -105,8 +107,8 @@ Example normalized output:
105
  ```
106
 
107
  The model does not generate answers, citations, source spans, retrieval results,
108
- or natural-language explanations. It classifies and scores the `(query,
109
- retrieved_contexts)` evidence state.
110
 
111
  ## Intended Use
112
 
@@ -124,7 +126,17 @@ a retriever, write answers, or replace human review in high-stakes settings.
124
 
125
  ## Input Format
126
 
 
 
 
 
 
 
 
 
 
127
  ```text
 
128
  Question: <user query>
129
 
130
  Sources:
@@ -168,7 +180,7 @@ contexts = [
168
  "The company posted net income of $4 million in Q2.",
169
  "The company recorded a quarterly loss of $12 million in Q3.",
170
  ]
171
- text = "Question: " + query + "\n\nSources:\n" + "\n".join(
172
  f"[{i}] {context}" for i, context in enumerate(contexts, start=1)
173
  )
174
 
@@ -208,31 +220,45 @@ The repository includes both FP32 and INT8 ONNX exports:
208
  - `model.onnx`
209
  - `model_quantized.onnx`
210
 
211
- For CPU inference, load `model_quantized.onnx` through `onnxruntime` or an
212
- Optimum ONNX runtime wrapper. Decode the resulting 18 logits using the same
213
- slices shown above.
 
214
 
215
  ## Evaluation
216
 
217
- Held-out training eval from `outputs/modernbert_base_v2_alpha_41358_active_20260704_seed42`:
 
218
 
219
  | Metric | Value |
220
  |---|---:|
221
- | overall score | 0.9497 |
222
- | verdict accuracy | 0.9727 |
223
- | false sufficient rate | 0.0455 |
224
- | failure accuracy | 0.9601 |
225
- | retrieval exact match | 0.8335 |
226
- | retrieval macro F1 | 0.9300 |
227
  | evidence-kind exact match | 0.9809 |
228
  | evidence-kind macro F1 | 0.9950 |
229
 
 
 
 
 
 
 
 
 
 
230
  Fitz-sage release-candidate checks:
231
 
232
  | Benchmark | Result |
233
  |---|---:|
234
  | balanced fixed-evidence governance sanity suite | 120/120 |
235
- | live fitz-sage benchmark | 86/120 |
 
 
 
236
 
237
  The live benchmark result is the practical integration target. The fixed-evidence
238
  suite is a minimal sanity check for the governance head.
 
23
 
24
  # pyrrho-v2-nano-g1
25
 
26
+ `pyrrho-v2-nano-g1` is a small local RAG planning and governance co-processor. It can
27
+ run before retrieval on a user query and after retrieval on the query plus
28
+ source passages. The post-retrieval pass returns whether the evidence is
29
  `SUFFICIENT`, `DISPUTED`, or `INSUFFICIENT` before an answer is generated.
30
 
31
  It is not an answer generator, not a retriever, and not an open-world fact
 
35
 
36
  Compared with the older Pyrrho v1 line, v2 exposes a smaller native head shape:
37
  one evidence verdict, one failure reason, and two compact multi-label metadata
38
+ heads. The same model owns the `fitz-sage` pre-retrieval query-planning pass and
39
+ the post-retrieval evidence-governance pass.
40
 
41
  ## Native V2 Heads
42
 
 
44
  |---|---|---|
45
  | `evidence_verdict` | `SUFFICIENT`, `DISPUTED`, `INSUFFICIENT` | Post-retrieval evidence sufficiency and conflict decision. |
46
  | `failure_mode` | `none`, `unresolved_conflict`, `missing_or_incomplete_evidence`, `wrong_scope_or_version`, `ambiguous_request` | Actionable reason when evidence is disputed or insufficient. |
47
+ | `retrieval_intents` | `needs_lookup`, `needs_temporal_resolution`, `needs_comparison_or_set`, `needs_broad_coverage` | Pre-retrieval planning and post-retrieval retry metadata. |
48
  | `evidence_kinds` | `needs_text`, `needs_table_or_record`, `needs_code_or_symbol`, `needs_config_or_setting`, `needs_log_or_run_result`, `needs_document_layout` | Evidence-surface metadata for routing, audit, and missing-source hints. |
49
 
50
  ## Output Contract
 
59
  | `8:12` | `retrieval_intents` | sigmoid multi-label scores |
60
  | `12:18` | `evidence_kinds` | sigmoid multi-label scores |
61
 
62
+ Most integrations should expose structured objects derived from those logits.
63
+ For `[PYRRHO_PRE]`, use only `retrieval_intents` and `evidence_kinds`. For
64
+ `[PYRRHO_POST]`, use all four heads.
65
 
66
  | Field | Meaning |
67
  |---|---|
 
107
  ```
108
 
109
  The model does not generate answers, citations, source spans, retrieval results,
110
+ or natural-language explanations. It classifies and scores query intent before
111
+ retrieval and the `(query, retrieved_contexts)` evidence state after retrieval.
112
 
113
  ## Intended Use
114
 
 
126
 
127
  ## Input Format
128
 
129
+ Pre-retrieval query planning:
130
+
131
+ ```text
132
+ [PYRRHO_PRE]
133
+ Question: <user query>
134
+ ```
135
+
136
+ Post-retrieval evidence governance:
137
+
138
  ```text
139
+ [PYRRHO_POST]
140
  Question: <user query>
141
 
142
  Sources:
 
180
  "The company posted net income of $4 million in Q2.",
181
  "The company recorded a quarterly loss of $12 million in Q3.",
182
  ]
183
+ text = "[PYRRHO_POST]\nQuestion: " + query + "\n\nSources:\n" + "\n".join(
184
  f"[{i}] {context}" for i, context in enumerate(contexts, start=1)
185
  )
186
 
 
220
  - `model.onnx`
221
  - `model_quantized.onnx`
222
 
223
+ `fitz-sage` loads `model.onnx` by default for governance accuracy. The quantized
224
+ graph is included for integrations that explicitly trade some accuracy margin
225
+ for smaller artifacts. Decode the resulting 18 logits using the same slices
226
+ shown above.
227
 
228
  ## Evaluation
229
 
230
+ Held-out post-retrieval eval from
231
+ `outputs/modernbert_base_v2_dual_from_g1_41358_active_20260704_seed42`:
232
 
233
  | Metric | Value |
234
  |---|---:|
235
+ | overall score | 0.9471 |
236
+ | verdict accuracy | 0.9703 |
237
+ | false sufficient rate | 0.0484 |
238
+ | failure accuracy | 0.9567 |
239
+ | retrieval exact match | 0.8308 |
240
+ | retrieval macro F1 | 0.9277 |
241
  | evidence-kind exact match | 0.9809 |
242
  | evidence-kind macro F1 | 0.9950 |
243
 
244
+ Held-out pre-retrieval query eval:
245
+
246
+ | Metric | Value |
247
+ |---|---:|
248
+ | retrieval exact match | 0.8248 |
249
+ | retrieval macro F1 | 0.9266 |
250
+ | evidence-kind exact match | 0.9637 |
251
+ | evidence-kind macro F1 | 0.9873 |
252
+
253
  Fitz-sage release-candidate checks:
254
 
255
  | Benchmark | Result |
256
  |---|---:|
257
  | balanced fixed-evidence governance sanity suite | 120/120 |
258
+ | live fitz-sage benchmark | 97/120 |
259
+ | core | 19/20 |
260
+ | holdout | 43/50 |
261
+ | holdout2 | 35/50 |
262
 
263
  The live benchmark result is the practical integration target. The fixed-evidence
264
  suite is a minimal sanity check for the governance head.
manifest.json CHANGED
@@ -1,18 +1,34 @@
1
  {
2
- "schema_version": "pyrrho-v2-release-manifest-1.0",
3
  "release": {
4
  "model_id": "yafitzdev/pyrrho-v2-nano-g1",
5
  "release_name": "pyrrho-v2-nano-g1",
6
- "created_date": "2026-07-06",
7
  "base_model": "answerdotai/ModernBERT-base",
8
  "architecture": "ModernBertForSequenceClassification",
9
  "num_logits": 18,
10
  "max_length": 2048,
 
11
  "trustworthy_threshold": null
12
  },
 
 
 
 
 
 
 
 
 
 
 
 
 
 
13
  "source_checkpoint": {
14
- "path": "outputs/modernbert_base_v2_alpha_41358_active_20260704_seed42/best_model",
15
- "run_id": "modernbert_base_v2_alpha_41358_active_20260704_seed42",
 
16
  "seed": 42,
17
  "training_rows": 41358,
18
  "dataset_pointer": "fitz_gov_v2_41358_20260703"
@@ -51,21 +67,37 @@
51
  ]
52
  },
53
  "eval": {
54
- "eval_overall_score": 0.949743230482894,
55
- "eval_verdict_accuracy": 0.9726855209088712,
56
- "eval_false_sufficient_rate": 0.04551920341394026,
57
- "eval_failure_accuracy": 0.9601160261058739,
58
- "eval_retrieval_exact_match": 0.8334541938602852,
59
- "eval_retrieval_macro_f1": 0.9299832758138473,
60
- "eval_evidence_kind_exact_match": 0.9809040367416002,
61
- "eval_evidence_kind_macro_f1": 0.9949836611151947
 
 
 
 
 
 
 
 
 
 
62
  },
63
  "fitz_sage_benchmark": {
64
- "balanced_fixed_evidence_toy_cases": "120/120",
65
- "live_total": "86/120",
66
- "live_core": "15/20",
67
- "live_holdout": "38/50",
68
- "live_holdout2": "33/50"
 
 
 
 
 
 
69
  },
70
  "artifacts": {
71
  "transformers": [
 
1
  {
2
+ "schema_version": "pyrrho-v2-release-manifest-1.1",
3
  "release": {
4
  "model_id": "yafitzdev/pyrrho-v2-nano-g1",
5
  "release_name": "pyrrho-v2-nano-g1",
6
+ "created_date": "2026-07-08",
7
  "base_model": "answerdotai/ModernBERT-base",
8
  "architecture": "ModernBertForSequenceClassification",
9
  "num_logits": 18,
10
  "max_length": 2048,
11
+ "input_mode": "dual",
12
  "trustworthy_threshold": null
13
  },
14
+ "input_contract": {
15
+ "pre_tag": "[PYRRHO_PRE]",
16
+ "post_tag": "[PYRRHO_POST]",
17
+ "pre_trains_heads": [
18
+ "retrieval_intents",
19
+ "evidence_kinds"
20
+ ],
21
+ "post_trains_heads": [
22
+ "evidence_verdict",
23
+ "failure_mode",
24
+ "retrieval_intents",
25
+ "evidence_kinds"
26
+ ]
27
+ },
28
  "source_checkpoint": {
29
+ "path": "outputs/modernbert_base_v2_dual_from_g1_41358_active_20260704_seed42/best_model",
30
+ "run_id": "modernbert_base_v2_dual_from_g1_41358_active_20260704_seed42",
31
+ "continued_from": "models/pyrrho-v2-nano-g1",
32
  "seed": 42,
33
  "training_rows": 41358,
34
  "dataset_pointer": "fitz_gov_v2_41358_20260703"
 
67
  ]
68
  },
69
  "eval": {
70
+ "post": {
71
+ "eval_overall_score": 0.9470664187704583,
72
+ "eval_verdict_accuracy": 0.9702683103698332,
73
+ "eval_verdict_macro_f1": 0.9591237257007766,
74
+ "eval_false_sufficient_rate": 0.04836415362731152,
75
+ "eval_failure_accuracy": 0.9567319313512207,
76
+ "eval_failure_macro_f1": 0.9064147136375201,
77
+ "eval_retrieval_exact_match": 0.8307952622673435,
78
+ "eval_retrieval_macro_f1": 0.9277362299701075,
79
+ "eval_evidence_kind_exact_match": 0.9809040367416002,
80
+ "eval_evidence_kind_macro_f1": 0.9949910057734287
81
+ },
82
+ "pre": {
83
+ "pre_eval_retrieval_exact_match": 0.8247522359197486,
84
+ "pre_eval_retrieval_macro_f1": 0.9266146369602828,
85
+ "pre_eval_evidence_kind_exact_match": 0.9637418419144308,
86
+ "pre_eval_evidence_kind_macro_f1": 0.9873183319207018
87
+ }
88
  },
89
  "fitz_sage_benchmark": {
90
+ "balanced_fixed_evidence_cases": "120/120",
91
+ "live_total": "97/120",
92
+ "live_core": "19/20",
93
+ "live_holdout": "43/50",
94
+ "live_holdout2": "35/50",
95
+ "results": {
96
+ "balanced": "benchmarks/results/dual_candidate_governance_balanced_v2.json",
97
+ "core": "benchmarks/results/dual_candidate_core_v2.json",
98
+ "holdout": "benchmarks/results/dual_candidate_holdout_v2.json",
99
+ "holdout2": "benchmarks/results/dual_candidate_holdout2_v2.json"
100
+ }
101
  },
102
  "artifacts": {
103
  "transformers": [
model.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:f376b9ccead222fc056ae664fe98279bd783f0902f5965747e52c4f2603b7417
3
- size 599049587
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:63982dc593fd03792177df26d7f15c9db5c41062da2f88325e63935638ab1a20
3
+ size 599049588
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:a214c3d960dd2e8438e9df9ed7073cbcd2cf04c89d3e1b0ddcbba1c0f920d812
3
  size 598489008
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fd9b63199e997f94f2e8d334030b8412b9456434dda4836c9858a2cea6e0c3a6
3
  size 598489008
model_quantized.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:9c2db516e0babcefed860082ca3c2d94b2467d5554356dd24d9dbd00cb155ce6
3
  size 150664051
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4041907531982651cc1b613101427ca82c849a58e67252343237e870cb46becc
3
  size 150664051