Hugging Face's logo

Etolith
/
perseval-task-completion

Text Classification
ONNX
Rust
perseval
agent-evaluation
task-completion
Model card Files
xet
 Community
sachaarbonel commited on
Commit
861e002
·
verified ·
1 Parent(s): 71e53bc

Publish development task-completion runtime candidate

Browse files
Files changed (5) hide show
  1. README.md +82 -0
  2. manifest.json +62 -0
  3. model.onnx +3 -0
  4. parity.json +543 -0
  5. tokenizer.json +0 -0
README.md ADDED
@@ -0,0 +1,82 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: onnx
4
+ pipeline_tag: text-classification
5
+ tags:
6
+ - perseval
7
+ - agent-evaluation
8
+ - task-completion
9
+ - onnx
10
+ - rust
11
+ ---
12
+
13
+ # Perseval Task Completion — development runtime candidate
14
+
15
+ This repository contains the compact ONNX artifact used to exercise Perseval's
16
+ local Task Completion evaluator and model-management flow.
17
+
18
+ ## Status
19
+
20
+ This is a **development candidate**, not a release-certified evaluator. It is
21
+ published so the macOS application can install, verify, update, and execute a
22
+ real local model end to end. Do not interpret its scores as production-quality
23
+ ground truth.
24
+
25
+ The frozen private comparison holdout remains sealed. A later release will be
26
+ marked certified only after it passes the documented holdout gates.
27
+
28
+ ## Files
29
+
30
+ - `manifest.json` — versioned model, projector, feature, calibration, and file
31
+ integrity contract.
32
+ - `model.onnx` — compact binary Task Completion head.
33
+ - `tokenizer.json` — pinned tokenizer reference.
34
+ - `parity.json` — Python/ONNX/Rust parity fixtures.
35
+ - `catalog.json` — mutable release pointer whose artifact revision is immutable.
36
+
37
+ Perseval downloads files from the immutable revision named in `catalog.json`,
38
+ checks every size and SHA-256 digest, then validates the internal manifest
39
+ before making the installation active.
40
+
41
+ ## Runtime contract
42
+
43
+ - Projector: `traceeval.compact-task-completion-projector.v1`
44
+ - Structured feature schema:
45
+ `traceeval.task_completion_structured_evidence.v2`
46
+ - Inputs: 39 structured evidence features
47
+ - Outputs: binary completion logits
48
+ - Calibration threshold: `0.745`
49
+
50
+ The application keeps trace projection, privacy enforcement, revision checks,
51
+ hash verification, inference, evidence resolution, and threshold application
52
+ in Rust.
53
+
54
+ ## Intended use
55
+
56
+ Use this candidate to:
57
+
58
+ - test Perseval's local learned-assessment runtime;
59
+ - validate model installation and updates without manual folder entry;
60
+ - run development benchmarks and Python/ONNX/Rust parity checks.
61
+
62
+ Do not use it as an automated decision-maker or as a substitute for human
63
+ review in high-impact settings.
64
+
65
+ ## Training data and limitations
66
+
67
+ The development pipeline combines source-balanced agent traces, structured
68
+ proof-bearing evidence, and group-safe splits. Synthetic, redistributable
69
+ examples are published separately at
70
+ [`Etolith/perseval-agent-evaluator-synthetic`](https://huggingface.co/datasets/Etolith/perseval-agent-evaluator-synthetic).
71
+
72
+ Known limitations include domain shift, unsupported-success claims, sparse
73
+ external-side-effect evidence, and incomplete calibration across long,
74
+ multi-agent traces. The compact head consumes a projected representation, not
75
+ raw screenshots or arbitrary full-context traces.
76
+
77
+ ## Reproducibility
78
+
79
+ The manifest pins the model, tokenizer, parity fixtures, projector, feature
80
+ schema, training run, and calibration versions. The public application verifies
81
+ those contracts before loading the artifact. Private traces, credentials,
82
+ teacher prompts, and unreleased checkpoints are not included.
manifest.json ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema_version": "perseval.task_completion_model_manifest.v1",
3
+ "model_id": "perseval-task-completion-runtime-candidate@f5d54444a518ac4a",
4
+ "model_file": {
5
+ "path": "model.onnx",
6
+ "sha256": "sha256:fbba6fdde8e0acf1f614396e5a430397bc1d12e09343b416f5d54444a518ac4a",
7
+ "size_bytes": 7273
8
+ },
9
+ "tokenizer_file": {
10
+ "path": "tokenizer.json",
11
+ "sha256": "sha256:4084ce5afffa730ba5697e3e81ec5707e14b13ad1bace0cc43bb0bdb411cfdcf",
12
+ "size_bytes": 711662
13
+ },
14
+ "parity_file": {
15
+ "path": "parity.json",
16
+ "sha256": "sha256:9428a085c4f969ba42eb6acf17cffc485f56e0db11edabf6fb2932199fab28a9",
17
+ "size_bytes": 11508
18
+ },
19
+ "parity_absolute_tolerance": 0.0001,
20
+ "inputs": [
21
+ {
22
+ "name": "structured_features",
23
+ "element_type": "f32",
24
+ "dimensions": [
25
+ null,
26
+ 39
27
+ ]
28
+ }
29
+ ],
30
+ "outputs": [
31
+ {
32
+ "name": "logits",
33
+ "element_type": "f32",
34
+ "dimensions": [
35
+ null,
36
+ 2
37
+ ]
38
+ }
39
+ ],
40
+ "decision_output": {
41
+ "output_name": "logits",
42
+ "incomplete_logit_index": 0,
43
+ "complete_logit_index": 1
44
+ },
45
+ "calibration": {
46
+ "version": "development-group-threshold-v1",
47
+ "threshold_complete": 0.7449999999999999,
48
+ "temperature": 1.0
49
+ },
50
+ "lineage": {
51
+ "base_model_id": "perseval-structured-evidence-mlp",
52
+ "base_model_revision": "source-balanced-seed-29-split-29",
53
+ "projector_schema_version": "traceeval.compact_task_completion_projection.v1",
54
+ "projector_version": "traceeval.compact-task-completion-projector.v1",
55
+ "training_record_schema_version": "traceeval.task_completion_training_record.v1",
56
+ "feature_set_version": "traceeval.task_completion_structured_evidence.v2",
57
+ "dataset_version": "audited-adjudicable-baseline-v1-projector-v3",
58
+ "dataset_sha256": "sha256:e7a0afc62fa603f3b09fe79b56a60f2db79a06a7f1d1e274572d78e4e7181916",
59
+ "training_version": "runtime-compatibility-candidate-v1",
60
+ "calibration_version": "development-group-threshold-v1"
61
+ }
62
+ }
model.onnx ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fbba6fdde8e0acf1f614396e5a430397bc1d12e09343b416f5d54444a518ac4a
3
+ size 7273
parity.json ADDED
@@ -0,0 +1,543 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema_version": "perseval.task_completion_onnx_parity.v1",
3
+ "model_id": "perseval-task-completion-runtime-candidate@f5d54444a518ac4a",
4
+ "absolute_tolerance": 0.0001,
5
+ "cases": [
6
+ {
7
+ "case_id": "development-1",
8
+ "inputs": [
9
+ {
10
+ "name": "structured_features",
11
+ "element_type": "f32",
12
+ "shape": [
13
+ 1,
14
+ 39
15
+ ],
16
+ "f32_values": [
17
+ 1.7917594909667969,
18
+ 0.6931471824645996,
19
+ 0.8333333134651184,
20
+ 0.1409505158662796,
21
+ 0.47426068782806396,
22
+ 0.0,
23
+ 0.0,
24
+ 0.0,
25
+ 0.0,
26
+ 0.20000000298023224,
27
+ 0.800000011920929,
28
+ 0.0,
29
+ 0.0,
30
+ 0.0,
31
+ 0.0,
32
+ 0.0,
33
+ 0.0,
34
+ 0.0,
35
+ 0.0,
36
+ 0.0,
37
+ 0.0,
38
+ 0.0,
39
+ 1.6094379425048828,
40
+ 0.0,
41
+ 0.6931471824645996,
42
+ 0.0,
43
+ 0.0,
44
+ 0.0,
45
+ 0.0,
46
+ 0.0,
47
+ 1.0986123085021973,
48
+ 0.4000000059604645,
49
+ 0.6000000238418579,
50
+ 0.012702078558504581,
51
+ 0.012702078558504581,
52
+ 0.0,
53
+ 0.0,
54
+ 0.0,
55
+ 0.0
56
+ ]
57
+ }
58
+ ],
59
+ "expected_outputs": [
60
+ {
61
+ "name": "logits",
62
+ "shape": [
63
+ 1,
64
+ 2
65
+ ],
66
+ "f32_values": [
67
+ -1.7064403295516968,
68
+ 1.2582646608352661
69
+ ]
70
+ }
71
+ ]
72
+ },
73
+ {
74
+ "case_id": "development-2",
75
+ "inputs": [
76
+ {
77
+ "name": "structured_features",
78
+ "element_type": "f32",
79
+ "shape": [
80
+ 1,
81
+ 39
82
+ ],
83
+ "f32_values": [
84
+ 1.3862943649291992,
85
+ 3.1354942321777344,
86
+ 0.11999999731779099,
87
+ 0.09619140625,
88
+ 0.10016949474811554,
89
+ 0.0,
90
+ 0.0,
91
+ 0.0,
92
+ 0.0,
93
+ 0.3333333432674408,
94
+ 0.6666666865348816,
95
+ 0.0,
96
+ 0.0,
97
+ 0.0,
98
+ 0.0,
99
+ 0.0,
100
+ 0.0,
101
+ 0.0,
102
+ 0.0,
103
+ 0.0,
104
+ 0.0,
105
+ 0.0,
106
+ 1.0986123085021973,
107
+ 0.0,
108
+ 0.6931471824645996,
109
+ 0.0,
110
+ 0.0,
111
+ 0.0,
112
+ 0.0,
113
+ 0.0,
114
+ 1.0986123085021973,
115
+ 0.6666666865348816,
116
+ 0.3333333432674408,
117
+ 0.018612520769238472,
118
+ 0.018612520769238472,
119
+ 0.0,
120
+ 0.0,
121
+ 0.0,
122
+ 0.0
123
+ ]
124
+ }
125
+ ],
126
+ "expected_outputs": [
127
+ {
128
+ "name": "logits",
129
+ "shape": [
130
+ 1,
131
+ 2
132
+ ],
133
+ "f32_values": [
134
+ 7.030359268188477,
135
+ -7.286364555358887
136
+ ]
137
+ }
138
+ ]
139
+ },
140
+ {
141
+ "case_id": "development-3",
142
+ "inputs": [
143
+ {
144
+ "name": "structured_features",
145
+ "element_type": "f32",
146
+ "shape": [
147
+ 1,
148
+ 39
149
+ ],
150
+ "f32_values": [
151
+ 3.044522523880005,
152
+ 3.4965076446533203,
153
+ 0.38461539149284363,
154
+ 0.3160807192325592,
155
+ 0.21864445507526398,
156
+ 0.0,
157
+ 0.0,
158
+ 0.0,
159
+ 0.0,
160
+ 0.05000000074505806,
161
+ 0.949999988079071,
162
+ 0.0,
163
+ 0.0,
164
+ 0.0,
165
+ 0.0,
166
+ 0.0,
167
+ 0.0,
168
+ 0.0,
169
+ 0.0,
170
+ 0.0,
171
+ 0.0,
172
+ 0.0,
173
+ 2.995732307434082,
174
+ 0.0,
175
+ 0.6931471824645996,
176
+ 0.0,
177
+ 0.0,
178
+ 0.0,
179
+ 0.0,
180
+ 0.0,
181
+ 1.0986123085021973,
182
+ 0.10000000149011612,
183
+ 0.8999999761581421,
184
+ 0.005664263851940632,
185
+ 0.005664263851940632,
186
+ 0.0,
187
+ 0.0,
188
+ 0.0,
189
+ 0.0
190
+ ]
191
+ }
192
+ ],
193
+ "expected_outputs": [
194
+ {
195
+ "name": "logits",
196
+ "shape": [
197
+ 1,
198
+ 2
199
+ ],
200
+ "f32_values": [
201
+ -3.417558193206787,
202
+ 2.5924134254455566
203
+ ]
204
+ }
205
+ ]
206
+ },
207
+ {
208
+ "case_id": "development-4",
209
+ "inputs": [
210
+ {
211
+ "name": "structured_features",
212
+ "element_type": "f32",
213
+ "shape": [
214
+ 1,
215
+ 39
216
+ ],
217
+ "f32_values": [
218
+ 2.6390573978424072,
219
+ 3.6888794898986816,
220
+ 0.25,
221
+ 0.2177734375,
222
+ 0.1523571014404297,
223
+ 0.0,
224
+ 0.0,
225
+ 0.0,
226
+ 0.0,
227
+ 0.07692307978868484,
228
+ 0.9230769276618958,
229
+ 0.0,
230
+ 0.0,
231
+ 0.0,
232
+ 0.0,
233
+ 0.0,
234
+ 0.0,
235
+ 0.0,
236
+ 0.0,
237
+ 0.0,
238
+ 0.0,
239
+ 0.0,
240
+ 2.5649492740631104,
241
+ 0.0,
242
+ 0.6931471824645996,
243
+ 0.0,
244
+ 0.0,
245
+ 0.0,
246
+ 0.0,
247
+ 0.0,
248
+ 1.0986123085021973,
249
+ 0.1538461595773697,
250
+ 0.8461538553237915,
251
+ 0.008221225813031197,
252
+ 0.008221225813031197,
253
+ 0.0,
254
+ 0.0,
255
+ 0.0,
256
+ 0.0
257
+ ]
258
+ }
259
+ ],
260
+ "expected_outputs": [
261
+ {
262
+ "name": "logits",
263
+ "shape": [
264
+ 1,
265
+ 2
266
+ ],
267
+ "f32_values": [
268
+ 2.6683249473571777,
269
+ -3.4865827560424805
270
+ ]
271
+ }
272
+ ]
273
+ },
274
+ {
275
+ "case_id": "development-5",
276
+ "inputs": [
277
+ {
278
+ "name": "structured_features",
279
+ "element_type": "f32",
280
+ "shape": [
281
+ 1,
282
+ 39
283
+ ],
284
+ "f32_values": [
285
+ 3.828641414642334,
286
+ 0.6931471824645996,
287
+ 0.97826087474823,
288
+ 0.85400390625,
289
+ 0.23878219723701477,
290
+ 0.0,
291
+ 0.6931471824645996,
292
+ 1.0,
293
+ 0.02222222276031971,
294
+ 0.9777777791023254,
295
+ 0.0,
296
+ 1.0986123085021973,
297
+ 0.0,
298
+ 0.0,
299
+ 0.6931471824645996,
300
+ 0.6931471824645996,
301
+ 1.945910096168518,
302
+ 0.0,
303
+ 3.5553481578826904,
304
+ 0.0,
305
+ 0.0,
306
+ 0.0,
307
+ 0.0,
308
+ 0.6931471824645996,
309
+ 3.8066625595092773,
310
+ 0.0,
311
+ 1.0,
312
+ 0.5777778029441833,
313
+ 2.995732307434082,
314
+ 0.0,
315
+ 2.397895336151123,
316
+ 0.2666666805744171,
317
+ 0.7333333492279053,
318
+ 0.0020964359864592552,
319
+ 0.0045740422792732716,
320
+ 0.0,
321
+ 0.0,
322
+ 0.0,
323
+ 0.0
324
+ ]
325
+ }
326
+ ],
327
+ "expected_outputs": [
328
+ {
329
+ "name": "logits",
330
+ "shape": [
331
+ 1,
332
+ 2
333
+ ],
334
+ "f32_values": [
335
+ 10.38206672668457,
336
+ -7.639095783233643
337
+ ]
338
+ }
339
+ ]
340
+ },
341
+ {
342
+ "case_id": "development-6",
343
+ "inputs": [
344
+ {
345
+ "name": "structured_features",
346
+ "element_type": "f32",
347
+ "shape": [
348
+ 1,
349
+ 39
350
+ ],
351
+ "f32_values": [
352
+ 3.828641414642334,
353
+ 0.6931471824645996,
354
+ 0.97826087474823,
355
+ 0.97509765625,
356
+ 0.2621193528175354,
357
+ 0.0,
358
+ 0.0,
359
+ 0.0,
360
+ 0.02222222276031971,
361
+ 0.9777777791023254,
362
+ 0.0,
363
+ 0.6931471824645996,
364
+ 0.0,
365
+ 0.0,
366
+ 0.6931471824645996,
367
+ 0.0,
368
+ 0.0,
369
+ 0.0,
370
+ 3.7376697063446045,
371
+ 0.6931471824645996,
372
+ 0.0,
373
+ 0.0,
374
+ 0.0,
375
+ 0.6931471824645996,
376
+ 3.8066625595092773,
377
+ 0.0,
378
+ 1.0,
379
+ 0.02222222276031971,
380
+ 3.7841897010803223,
381
+ 0.0,
382
+ 2.3025851249694824,
383
+ 0.1111111119389534,
384
+ 0.8888888955116272,
385
+ 0.0018360874382779002,
386
+ 0.0018360874382779002,
387
+ 0.0,
388
+ 0.0,
389
+ 0.0,
390
+ 0.0
391
+ ]
392
+ }
393
+ ],
394
+ "expected_outputs": [
395
+ {
396
+ "name": "logits",
397
+ "shape": [
398
+ 1,
399
+ 2
400
+ ],
401
+ "f32_values": [
402
+ 6.084724426269531,
403
+ -2.748540163040161
404
+ ]
405
+ }
406
+ ]
407
+ },
408
+ {
409
+ "case_id": "development-7",
410
+ "inputs": [
411
+ {
412
+ "name": "structured_features",
413
+ "element_type": "f32",
414
+ "shape": [
415
+ 1,
416
+ 39
417
+ ],
418
+ "f32_values": [
419
+ 3.4965076446533203,
420
+ 2.70805025100708,
421
+ 0.695652186870575,
422
+ 0.9895833134651184,
423
+ 0.24384374916553497,
424
+ 0.0,
425
+ 0.0,
426
+ 0.0,
427
+ 0.0,
428
+ 1.0,
429
+ 0.0,
430
+ 1.0986123085021973,
431
+ 0.0,
432
+ 0.0,
433
+ 2.1972246170043945,
434
+ 0.0,
435
+ 0.0,
436
+ 0.0,
437
+ 3.0910425186157227,
438
+ 0.0,
439
+ 0.0,
440
+ 0.0,
441
+ 0.0,
442
+ 0.0,
443
+ 3.4965076446533203,
444
+ 0.0,
445
+ 1.0,
446
+ 0.0,
447
+ 0.0,
448
+ 0.0,
449
+ 2.079441547393799,
450
+ 0.375,
451
+ 0.625,
452
+ 0.001809210516512394,
453
+ 0.001809210516512394,
454
+ 0.0,
455
+ 0.0,
456
+ 0.0,
457
+ 0.0
458
+ ]
459
+ }
460
+ ],
461
+ "expected_outputs": [
462
+ {
463
+ "name": "logits",
464
+ "shape": [
465
+ 1,
466
+ 2
467
+ ],
468
+ "f32_values": [
469
+ -6.550144195556641,
470
+ 6.765974521636963
471
+ ]
472
+ }
473
+ ]
474
+ },
475
+ {
476
+ "case_id": "development-8",
477
+ "inputs": [
478
+ {
479
+ "name": "structured_features",
480
+ "element_type": "f32",
481
+ "shape": [
482
+ 1,
483
+ 39
484
+ ],
485
+ "f32_values": [
486
+ 3.5835189819335938,
487
+ 2.4849066734313965,
488
+ 0.760869562625885,
489
+ 0.9913737177848816,
490
+ 0.21843284368515015,
491
+ 0.0,
492
+ 0.0,
493
+ 0.0,
494
+ 0.0,
495
+ 1.0,
496
+ 0.0,
497
+ 0.6931471824645996,
498
+ 0.0,
499
+ 0.0,
500
+ 1.945910096168518,
501
+ 0.0,
502
+ 0.6931471824645996,
503
+ 0.0,
504
+ 3.295836925506592,
505
+ 0.0,
506
+ 0.0,
507
+ 0.0,
508
+ 0.0,
509
+ 0.0,
510
+ 3.5835189819335938,
511
+ 0.0,
512
+ 1.0,
513
+ 0.0,
514
+ 0.0,
515
+ 0.0,
516
+ 2.3025851249694824,
517
+ 0.2857142984867096,
518
+ 0.7142857313156128,
519
+ 0.0018059432040899992,
520
+ 0.0018059432040899992,
521
+ 0.0,
522
+ 0.0,
523
+ 0.0,
524
+ 0.0
525
+ ]
526
+ }
527
+ ],
528
+ "expected_outputs": [
529
+ {
530
+ "name": "logits",
531
+ "shape": [
532
+ 1,
533
+ 2
534
+ ],
535
+ "f32_values": [
536
+ -0.5859965682029724,
537
+ 1.4117785692214966
538
+ ]
539
+ }
540
+ ]
541
+ }
542
+ ]
543
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff