Spaces:
Running
Running
deploy(hf): sync szl-holdings/a11oy@main derived COPY set
Browse filesReusable Dockerfile-COPY-derived deploy from szl-holdings/a11oy main.
Files: 1131 Pruned: 0
Derived from Dockerfile COPY sources (NO hand-maintained allowlist).
Signed-off-by: SZL Holdings <noreply@szlholdings.ai>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- model_release/receipt-agent/MODEL_CARD_DRAFT.md +2 -2
- model_release/receipt-agent/README.md +2 -2
- model_release/receipt-agent/admission-manifest.json +1 -1
- model_release/receipt-agent/release-manifest.json +1 -1
- model_release/receipt-agent/release-manifest.schema.json +1 -1
- model_release/szl-forge-family.json +10 -4
- model_release/szl-khipu-second-brain.json +38 -9
- schemas/quant-claims/benchmark-run.v1.schema.json +217 -0
- schemas/quant-claims/claim.v1.schema.json +118 -0
model_release/receipt-agent/MODEL_CARD_DRAFT.md
CHANGED
|
@@ -80,12 +80,12 @@ The existing adapter showed a directional loss improvement on a 16-row observati
|
|
| 80 |
|---|---:|---:|---|
|
| 81 |
| Existing SFT dataset | 167 rows | 0 approved for reuse | Rights/privacy/contamination review incomplete |
|
| 82 |
| A11oy Brain | 9,464 raw nodes | 0 | All raw nodes training-quarantined |
|
| 83 |
-
| Formula
|
| 84 |
| ORPO candidate | 12 qualification checks | 0 | 0/12 passed; quarantined |
|
| 85 |
|
| 86 |
The Brain and formula system can make the released model more useful **without being copied into the weights**. They remain external retrieval, holdout, contradiction, freshness, and proof-status infrastructure. A future training curriculum must be newly assembled from owned or explicitly licensed rows with item-level provenance, deduplication, contamination analysis, and immutable split receipts.
|
| 87 |
|
| 88 |
-
No card may claim training on "all 200 formulas," the Brain, Lean, mathlib, GitHub, publications, or Hugging Face until row-level admission evidence proves the exact statement. The current repository evidence supports
|
| 89 |
|
| 90 |
## Two-stage output contract
|
| 91 |
|
|
|
|
| 80 |
|---|---:|---:|---|
|
| 81 |
| Existing SFT dataset | 167 rows | 0 approved for reuse | Rights/privacy/contamination review incomplete |
|
| 82 |
| A11oy Brain | 9,464 raw nodes | 0 | All raw nodes training-quarantined |
|
| 83 |
+
| Formula admission | 146 formula crosswalk records + 2 SZL-Lake evidence records = 148 holdout rows | 0 | Frozen holdout; namespace/status evaluation only |
|
| 84 |
| ORPO candidate | 12 qualification checks | 0 | 0/12 passed; quarantined |
|
| 85 |
|
| 86 |
The Brain and formula system can make the released model more useful **without being copied into the weights**. They remain external retrieval, holdout, contradiction, freshness, and proof-status infrastructure. A future training curriculum must be newly assembled from owned or explicitly licensed rows with item-level provenance, deduplication, contamination analysis, and immutable split receipts.
|
| 87 |
|
| 88 |
+
No card may claim training on "all 200 formulas," the Brain, Lean, mathlib, GitHub, publications, or Hugging Face until row-level admission evidence proves the exact statement. The current repository evidence supports 146 formula crosswalk records plus 2 SZL-Lake evidence records in a 148-row holdout tranche, not a verified 200-formula training set.
|
| 89 |
|
| 90 |
## Two-stage output contract
|
| 91 |
|
model_release/receipt-agent/README.md
CHANGED
|
@@ -27,7 +27,7 @@ turn retrieved text into a mathematical proof.
|
|
| 27 |
| Existing evaluation | Incomplete | A 16-row loss direction exists, but general quality is not established. |
|
| 28 |
| ORPO candidate | 0/12 qualification checks | Quarantined; cannot influence release. |
|
| 29 |
| Brain | 9,464 raw nodes | All 9,464 are training-quarantined; zero raw-node text rows are admitted to train. |
|
| 30 |
-
| Formula
|
| 31 |
| Proposed model | No ReceiptAgent weight artifact | `NOT_PROMOTED`, `NOT_ESTABLISHED`, and unavailable for inference. |
|
| 32 |
|
| 33 |
The current Brain and formula corpus are valuable as a governed retrieval and
|
|
@@ -63,7 +63,7 @@ and permits zero catastrophic errors. Current blockers are:
|
|
| 63 |
|
| 64 |
- incomplete row-level governance for the existing 167-row dataset;
|
| 65 |
- 9,464/9,464 raw Brain nodes quarantined from training;
|
| 66 |
-
-
|
| 67 |
- the ORPO candidate failing all 12 qualification checks;
|
| 68 |
- no ReceiptAgent-specific training receipt or weights;
|
| 69 |
- no frozen three-way evaluation;
|
|
|
|
| 27 |
| Existing evaluation | Incomplete | A 16-row loss direction exists, but general quality is not established. |
|
| 28 |
| ORPO candidate | 0/12 qualification checks | Quarantined; cannot influence release. |
|
| 29 |
| Brain | 9,464 raw nodes | All 9,464 are training-quarantined; zero raw-node text rows are admitted to train. |
|
| 30 |
+
| Formula admission | 146 formula crosswalk records + 2 SZL-Lake evidence records = 148 holdout tranche rows | All 148 are holdout; zero are admitted to train. Formula IDs are namespace-scoped. |
|
| 31 |
| Proposed model | No ReceiptAgent weight artifact | `NOT_PROMOTED`, `NOT_ESTABLISHED`, and unavailable for inference. |
|
| 32 |
|
| 33 |
The current Brain and formula corpus are valuable as a governed retrieval and
|
|
|
|
| 63 |
|
| 64 |
- incomplete row-level governance for the existing 167-row dataset;
|
| 65 |
- 9,464/9,464 raw Brain nodes quarantined from training;
|
| 66 |
+
- 146 formula records plus 2 SZL-Lake evidence records reserved as 148/148 holdout rows, with F1-F23 namespace collisions;
|
| 67 |
- the ORPO candidate failing all 12 qualification checks;
|
| 68 |
- no ReceiptAgent-specific training receipt or weights;
|
| 69 |
- no frozen three-way evaluation;
|
model_release/receipt-agent/admission-manifest.json
CHANGED
|
@@ -59,7 +59,7 @@
|
|
| 59 |
"state": "PASS",
|
| 60 |
"required_for_promotion": true,
|
| 61 |
"exact_requirement": "Formula IDs must be namespace-scoped, proof transfer must require an explicit semantic binding, and no holdout formula may enter training.",
|
| 62 |
-
"observed": "The current crosswalk has 148 holdout
|
| 63 |
"evidence": [
|
| 64 |
"research/formula-training-admission/admission-manifest.json",
|
| 65 |
"research/formula-training-admission/formula-id-crosswalk.json"
|
|
|
|
| 59 |
"state": "PASS",
|
| 60 |
"required_for_promotion": true,
|
| 61 |
"exact_requirement": "Formula IDs must be namespace-scoped, proof transfer must require an explicit semantic binding, and no holdout formula may enter training.",
|
| 62 |
+
"observed": "The current crosswalk has 146 formula records; two SZL-Lake evidence records produce a 148-row holdout tranche with zero train rows. F1-F23 namespace collisions block implicit proof transfer.",
|
| 63 |
"evidence": [
|
| 64 |
"research/formula-training-admission/admission-manifest.json",
|
| 65 |
"research/formula-training-admission/formula-id-crosswalk.json"
|
model_release/receipt-agent/release-manifest.json
CHANGED
|
@@ -17,7 +17,7 @@
|
|
| 17 |
"brain_raw_nodes": 9464,
|
| 18 |
"brain_raw_nodes_training_quarantined": 9464,
|
| 19 |
"brain_raw_rows_admitted_to_train": 0,
|
| 20 |
-
"formula_crosswalk_rows":
|
| 21 |
"formula_holdout_rows": 148,
|
| 22 |
"formula_train_rows": 0,
|
| 23 |
"orpo_checks_passed": 0,
|
|
|
|
| 17 |
"brain_raw_nodes": 9464,
|
| 18 |
"brain_raw_nodes_training_quarantined": 9464,
|
| 19 |
"brain_raw_rows_admitted_to_train": 0,
|
| 20 |
+
"formula_crosswalk_rows": 146,
|
| 21 |
"formula_holdout_rows": 148,
|
| 22 |
"formula_train_rows": 0,
|
| 23 |
"orpo_checks_passed": 0,
|
model_release/receipt-agent/release-manifest.schema.json
CHANGED
|
@@ -95,7 +95,7 @@
|
|
| 95 |
"const": 0
|
| 96 |
},
|
| 97 |
"formula_crosswalk_rows": {
|
| 98 |
-
"const":
|
| 99 |
},
|
| 100 |
"formula_holdout_rows": {
|
| 101 |
"const": 148
|
|
|
|
| 95 |
"const": 0
|
| 96 |
},
|
| 97 |
"formula_crosswalk_rows": {
|
| 98 |
+
"const": 146
|
| 99 |
},
|
| 100 |
"formula_holdout_rows": {
|
| 101 |
"const": 148
|
model_release/szl-forge-family.json
CHANGED
|
@@ -114,12 +114,18 @@
|
|
| 114 |
"raw_nodes_admitted_to_gradients": 0,
|
| 115 |
"current_state": "ROW_LEVEL_ADMISSION_ENGINE_IMPLEMENTED_CURRENT_RAW_ROWS_QUARANTINED",
|
| 116 |
"admission_engine": "szl_brain_training_admission.py",
|
| 117 |
-
"admission_contract": "szl.brain-training-admission-report.
|
| 118 |
-
"admission_requirement": "Each gradient row requires Ed25519-signed
|
| 119 |
"required_signed_inputs": [
|
| 120 |
"protected_eval_content_sha256_list",
|
| 121 |
-
"
|
| 122 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 123 |
]
|
| 124 |
},
|
| 125 |
"formula_policy": {
|
|
|
|
| 114 |
"raw_nodes_admitted_to_gradients": 0,
|
| 115 |
"current_state": "ROW_LEVEL_ADMISSION_ENGINE_IMPLEMENTED_CURRENT_RAW_ROWS_QUARANTINED",
|
| 116 |
"admission_engine": "szl_brain_training_admission.py",
|
| 117 |
+
"admission_contract": "szl.brain-training-admission-report.v2",
|
| 118 |
+
"admission_requirement": "Each gradient row requires Ed25519-signed, purpose-scoped evidence for source identity, authorship/rightsholder permission, privacy/PII clearance, immutable revision, freshness, deduplication, held-out contamination analysis, reviewer-owned approval, and a frozen cross-run split assignment rooted in a signed policy bundle; train admission is disabled by default and terminal artifacts require a distinct signing key.",
|
| 119 |
"required_signed_inputs": [
|
| 120 |
"protected_eval_content_sha256_list",
|
| 121 |
+
"purpose_scoped_evidence_trust_store",
|
| 122 |
+
"policy_root_signer",
|
| 123 |
+
"root_signed_policy_bundle",
|
| 124 |
+
"signed_prior_split_ledger_descriptor",
|
| 125 |
+
"exact_split_ledger_head_sha256",
|
| 126 |
+
"reviewer_allowlist",
|
| 127 |
+
"artifact_signing_key",
|
| 128 |
+
"explicit_train_admission_switch"
|
| 129 |
]
|
| 130 |
},
|
| 131 |
"formula_policy": {
|
model_release/szl-khipu-second-brain.json
CHANGED
|
@@ -50,21 +50,32 @@
|
|
| 50 |
"current_state": "ROW_LEVEL_ADMISSION_ENGINE_IMPLEMENTED_CURRENT_RAW_ROWS_QUARANTINED",
|
| 51 |
"admission_engine": "szl_brain_training_admission.py",
|
| 52 |
"admission_test": "tests/test_brain_training_admission.py",
|
| 53 |
-
"admission_contract": "szl.brain-training-admission-report.
|
| 54 |
"admission_evidence_security": {
|
| 55 |
"signature": "ED25519",
|
| 56 |
-
"trust_model": "
|
| 57 |
-
"content_binding": "
|
| 58 |
-
"cross_run_isolation": "
|
|
|
|
|
|
|
| 59 |
"self_attestation_allowed": false
|
| 60 |
},
|
| 61 |
"required_admission_inputs": [
|
| 62 |
"protected_eval_content_sha256_list",
|
| 63 |
-
"
|
| 64 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
],
|
| 66 |
"current_inventory_obligations": {
|
| 67 |
"license_or_rights_unknown": 9464,
|
|
|
|
|
|
|
|
|
|
| 68 |
"immutable_source_revision_unknown": 9464,
|
| 69 |
"freshness_unknown": 9430,
|
| 70 |
"duplicate_candidates": 3747,
|
|
@@ -85,24 +96,37 @@
|
|
| 85 |
"file_sha256": "6f5e0d9ecb60d8aa52fc229a42ad37d62fc7fa7406993a0b8352b2b574fb40a4"
|
| 86 |
}
|
| 87 |
},
|
| 88 |
-
"candidate_schema": "szl.brain-training-candidate.
|
| 89 |
-
"decision_schema": "szl.brain-training-admission-decision.
|
| 90 |
-
"report_schema": "szl.brain-training-admission-report.
|
| 91 |
"candidate_required_inputs": [
|
| 92 |
"node_id",
|
| 93 |
"content",
|
| 94 |
"content_sha256",
|
| 95 |
"source.uri",
|
|
|
|
| 96 |
"source.revision",
|
| 97 |
"source.timestamp_utc",
|
| 98 |
"source.evidence",
|
|
|
|
|
|
|
| 99 |
"rights.basis",
|
| 100 |
"rights.license",
|
|
|
|
| 101 |
"rights.evidence",
|
|
|
|
|
|
|
|
|
|
|
|
|
| 102 |
"contamination.result",
|
| 103 |
"contamination.method",
|
| 104 |
"contamination.checked_against",
|
| 105 |
"contamination.evidence",
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
"split"
|
| 107 |
],
|
| 108 |
"derived_decision_fields": [
|
|
@@ -112,7 +136,9 @@
|
|
| 112 |
"freshness.state",
|
| 113 |
"source.status",
|
| 114 |
"rights.status",
|
|
|
|
| 115 |
"contamination.observed_result",
|
|
|
|
| 116 |
"training_eligible",
|
| 117 |
"evaluation_eligible",
|
| 118 |
"reason_codes",
|
|
@@ -123,6 +149,9 @@
|
|
| 123 |
"COPY_EVALUATION_QRELS_INTO_TRAINING",
|
| 124 |
"TREAT_RETRIEVAL_AS_TRAINING",
|
| 125 |
"LABEL_INDEX_NODES_AS_PARAMETERS",
|
|
|
|
|
|
|
|
|
|
| 126 |
"WEAKEN_GATES_TO_REACH_A_TARGET_ROW_COUNT"
|
| 127 |
]
|
| 128 |
},
|
|
|
|
| 50 |
"current_state": "ROW_LEVEL_ADMISSION_ENGINE_IMPLEMENTED_CURRENT_RAW_ROWS_QUARANTINED",
|
| 51 |
"admission_engine": "szl_brain_training_admission.py",
|
| 52 |
"admission_test": "tests/test_brain_training_admission.py",
|
| 53 |
+
"admission_contract": "szl.brain-training-admission-report.v2",
|
| 54 |
"admission_evidence_security": {
|
| 55 |
"signature": "ED25519",
|
| 56 |
+
"trust_model": "ROOT_SIGNED_PURPOSE_SCOPED_ISSUER_TOOL_KEY",
|
| 57 |
+
"content_binding": "EXACT_CONTENT_SOURCE_IDENTITY_REVISION_AUTHOR_RIGHTSHOLDER_PERMISSION_PRIVACY_REVIEW_AND_CONTAMINATION",
|
| 58 |
+
"cross_run_isolation": "POLICY_PINNED_SIGNED_SPLIT_LEDGER_HEAD",
|
| 59 |
+
"review_signer_binding": "REVIEWER_OWNED_KEY",
|
| 60 |
+
"terminal_artifact": "ED25519_SIGNED_MANIFEST",
|
| 61 |
"self_attestation_allowed": false
|
| 62 |
},
|
| 63 |
"required_admission_inputs": [
|
| 64 |
"protected_eval_content_sha256_list",
|
| 65 |
+
"purpose_scoped_evidence_trust_store",
|
| 66 |
+
"policy_root_signer",
|
| 67 |
+
"root_signed_policy_bundle",
|
| 68 |
+
"signed_prior_split_ledger_descriptor",
|
| 69 |
+
"exact_split_ledger_head_sha256",
|
| 70 |
+
"reviewer_allowlist",
|
| 71 |
+
"artifact_signing_key",
|
| 72 |
+
"explicit_train_admission_switch"
|
| 73 |
],
|
| 74 |
"current_inventory_obligations": {
|
| 75 |
"license_or_rights_unknown": 9464,
|
| 76 |
+
"author_or_rightsholder_permission_unknown": 9464,
|
| 77 |
+
"privacy_or_pii_clearance_unknown": 9464,
|
| 78 |
+
"allowlisted_review_unknown": 9464,
|
| 79 |
"immutable_source_revision_unknown": 9464,
|
| 80 |
"freshness_unknown": 9430,
|
| 81 |
"duplicate_candidates": 3747,
|
|
|
|
| 96 |
"file_sha256": "6f5e0d9ecb60d8aa52fc229a42ad37d62fc7fa7406993a0b8352b2b574fb40a4"
|
| 97 |
}
|
| 98 |
},
|
| 99 |
+
"candidate_schema": "szl.brain-training-candidate.v2",
|
| 100 |
+
"decision_schema": "szl.brain-training-admission-decision.v2",
|
| 101 |
+
"report_schema": "szl.brain-training-admission-report.v2",
|
| 102 |
"candidate_required_inputs": [
|
| 103 |
"node_id",
|
| 104 |
"content",
|
| 105 |
"content_sha256",
|
| 106 |
"source.uri",
|
| 107 |
+
"source.identity",
|
| 108 |
"source.revision",
|
| 109 |
"source.timestamp_utc",
|
| 110 |
"source.evidence",
|
| 111 |
+
"rights.author",
|
| 112 |
+
"rights.rightsholder",
|
| 113 |
"rights.basis",
|
| 114 |
"rights.license",
|
| 115 |
+
"rights.permission_scope",
|
| 116 |
"rights.evidence",
|
| 117 |
+
"privacy.classification",
|
| 118 |
+
"privacy.pii_result",
|
| 119 |
+
"privacy.method",
|
| 120 |
+
"privacy.evidence",
|
| 121 |
"contamination.result",
|
| 122 |
"contamination.method",
|
| 123 |
"contamination.checked_against",
|
| 124 |
"contamination.evidence",
|
| 125 |
+
"review.state",
|
| 126 |
+
"review.reviewer",
|
| 127 |
+
"review.reviewed_at_utc",
|
| 128 |
+
"review.reasons",
|
| 129 |
+
"review.evidence",
|
| 130 |
"split"
|
| 131 |
],
|
| 132 |
"derived_decision_fields": [
|
|
|
|
| 136 |
"freshness.state",
|
| 137 |
"source.status",
|
| 138 |
"rights.status",
|
| 139 |
+
"privacy.status",
|
| 140 |
"contamination.observed_result",
|
| 141 |
+
"review.status",
|
| 142 |
"training_eligible",
|
| 143 |
"evaluation_eligible",
|
| 144 |
"reason_codes",
|
|
|
|
| 149 |
"COPY_EVALUATION_QRELS_INTO_TRAINING",
|
| 150 |
"TREAT_RETRIEVAL_AS_TRAINING",
|
| 151 |
"LABEL_INDEX_NODES_AS_PARAMETERS",
|
| 152 |
+
"AUTO_APPROVE_UNKNOWN_PRIVACY_OR_RIGHTS",
|
| 153 |
+
"SELF_REVIEW_WITHOUT_ALLOWLISTED_SIGNED_EVIDENCE",
|
| 154 |
+
"ENABLE_TRAIN_ADMISSION_BY_DEFAULT",
|
| 155 |
"WEAKEN_GATES_TO_REACH_A_TARGET_ROW_COUNT"
|
| 156 |
]
|
| 157 |
},
|
schemas/quant-claims/benchmark-run.v1.schema.json
ADDED
|
@@ -0,0 +1,217 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
| 3 |
+
"$id": "https://a-11-oy.com/schemas/quant-claims/benchmark-run.v1.schema.json",
|
| 4 |
+
"title": "SZL Quant Benchmark Run v1",
|
| 5 |
+
"type": "object",
|
| 6 |
+
"additionalProperties": false,
|
| 7 |
+
"required": [
|
| 8 |
+
"schema_version", "run_id", "claim_id", "protocol", "subject", "baseline",
|
| 9 |
+
"dataset", "hardware", "software", "execution", "raw_trials", "measurement",
|
| 10 |
+
"correctness", "energy", "freshness", "review", "artifacts"
|
| 11 |
+
],
|
| 12 |
+
"properties": {
|
| 13 |
+
"schema_version": {"const": "szl.quant.benchmark-run/v1"},
|
| 14 |
+
"run_id": {"type": "string", "pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]{2,127}$"},
|
| 15 |
+
"claim_id": {"type": "string", "pattern": "^[a-z0-9][a-z0-9-]{2,95}$"},
|
| 16 |
+
"protocol": {
|
| 17 |
+
"type": "object", "additionalProperties": false,
|
| 18 |
+
"required": ["id", "version", "sha256", "preregistered_at", "manifest"],
|
| 19 |
+
"properties": {
|
| 20 |
+
"id": {"type": "string"}, "version": {"type": "string"},
|
| 21 |
+
"sha256": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"},
|
| 22 |
+
"preregistered_at": {"type": "string", "format": "date-time"},
|
| 23 |
+
"manifest": {
|
| 24 |
+
"type": "object", "additionalProperties": false,
|
| 25 |
+
"required": [
|
| 26 |
+
"id", "version", "metric", "unit", "preregistered_at", "minimum_trials",
|
| 27 |
+
"hardware_class", "requires_energy", "dataset_manifest_sha256"
|
| 28 |
+
],
|
| 29 |
+
"properties": {
|
| 30 |
+
"id": {"type": "string"}, "version": {"type": "string"},
|
| 31 |
+
"metric": {"type": "string"}, "unit": {"type": "string"},
|
| 32 |
+
"preregistered_at": {"type": "string", "format": "date-time"},
|
| 33 |
+
"minimum_trials": {"type": "integer", "minimum": 3},
|
| 34 |
+
"hardware_class": {"type": "string"},
|
| 35 |
+
"requires_energy": {"type": "boolean"},
|
| 36 |
+
"dataset_manifest_sha256": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"}
|
| 37 |
+
}
|
| 38 |
+
}
|
| 39 |
+
}
|
| 40 |
+
},
|
| 41 |
+
"subject": {"$ref": "#/$defs/artifactSubject"},
|
| 42 |
+
"baseline": {"$ref": "#/$defs/artifactSubject"},
|
| 43 |
+
"dataset": {
|
| 44 |
+
"type": "object", "additionalProperties": false,
|
| 45 |
+
"required": ["manifest_sha256", "split", "seeds", "rights_admitted"],
|
| 46 |
+
"properties": {
|
| 47 |
+
"manifest_sha256": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"},
|
| 48 |
+
"split": {"type": "string"},
|
| 49 |
+
"seeds": {"type": "array", "minItems": 1, "items": {"type": "integer"}},
|
| 50 |
+
"rights_admitted": {"const": true}
|
| 51 |
+
}
|
| 52 |
+
},
|
| 53 |
+
"hardware": {
|
| 54 |
+
"type": "object", "additionalProperties": false,
|
| 55 |
+
"required": ["class", "devices", "driver", "cuda"],
|
| 56 |
+
"properties": {
|
| 57 |
+
"class": {"type": "string"},
|
| 58 |
+
"devices": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/device"}},
|
| 59 |
+
"driver": {"type": "string"}, "cuda": {"type": "string"}
|
| 60 |
+
}
|
| 61 |
+
},
|
| 62 |
+
"software": {
|
| 63 |
+
"type": "object", "additionalProperties": false,
|
| 64 |
+
"required": ["git_commit", "container_digest", "lock_sha256"],
|
| 65 |
+
"properties": {
|
| 66 |
+
"git_commit": {"type": "string", "pattern": "^[0-9a-f]{40}$"},
|
| 67 |
+
"container_digest": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"},
|
| 68 |
+
"lock_sha256": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"}
|
| 69 |
+
}
|
| 70 |
+
},
|
| 71 |
+
"execution": {
|
| 72 |
+
"type": "object", "additionalProperties": false,
|
| 73 |
+
"required": ["started_at", "completed_at", "warmup_trials", "measured_trials", "config"],
|
| 74 |
+
"properties": {
|
| 75 |
+
"started_at": {"type": "string", "format": "date-time"},
|
| 76 |
+
"completed_at": {"type": "string", "format": "date-time"},
|
| 77 |
+
"warmup_trials": {"type": "integer", "minimum": 0},
|
| 78 |
+
"measured_trials": {"type": "integer", "minimum": 3},
|
| 79 |
+
"config": {"type": "object"}
|
| 80 |
+
}
|
| 81 |
+
},
|
| 82 |
+
"raw_trials": {
|
| 83 |
+
"type": "array", "minItems": 3,
|
| 84 |
+
"items": {"$ref": "#/$defs/rawTrial"}
|
| 85 |
+
},
|
| 86 |
+
"measurement": {"$ref": "#/$defs/measurement"},
|
| 87 |
+
"correctness": {"$ref": "#/$defs/correctness"},
|
| 88 |
+
"energy": {"$ref": "#/$defs/energy"},
|
| 89 |
+
"freshness": {
|
| 90 |
+
"type": "object", "additionalProperties": false,
|
| 91 |
+
"required": ["fresh_until"],
|
| 92 |
+
"properties": {"fresh_until": {"type": "string", "format": "date-time"}}
|
| 93 |
+
},
|
| 94 |
+
"review": {
|
| 95 |
+
"type": "object", "additionalProperties": false,
|
| 96 |
+
"required": ["status", "reviewer", "reviewed_at"],
|
| 97 |
+
"properties": {
|
| 98 |
+
"status": {"const": "APPROVED"}, "reviewer": {"type": "string", "minLength": 1},
|
| 99 |
+
"reviewed_at": {"type": "string", "format": "date-time"}
|
| 100 |
+
}
|
| 101 |
+
},
|
| 102 |
+
"artifacts": {
|
| 103 |
+
"type": "object", "additionalProperties": false,
|
| 104 |
+
"required": ["result_sha256", "result_bundle"],
|
| 105 |
+
"properties": {
|
| 106 |
+
"result_sha256": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"},
|
| 107 |
+
"result_bundle": {
|
| 108 |
+
"type": "object", "additionalProperties": false,
|
| 109 |
+
"required": ["raw_trials", "measurement", "correctness", "energy"],
|
| 110 |
+
"properties": {
|
| 111 |
+
"raw_trials": {"type": "array", "minItems": 3, "items": {"$ref": "#/$defs/rawTrial"}},
|
| 112 |
+
"measurement": {"$ref": "#/$defs/measurement"},
|
| 113 |
+
"correctness": {"$ref": "#/$defs/correctness"},
|
| 114 |
+
"energy": {"$ref": "#/$defs/energy"}
|
| 115 |
+
}
|
| 116 |
+
}
|
| 117 |
+
}
|
| 118 |
+
}
|
| 119 |
+
},
|
| 120 |
+
"$defs": {
|
| 121 |
+
"artifactSubject": {
|
| 122 |
+
"type": "object", "additionalProperties": false,
|
| 123 |
+
"required": ["artifact_id", "revision", "sha256", "license"],
|
| 124 |
+
"properties": {
|
| 125 |
+
"artifact_id": {"type": "string"}, "revision": {"type": "string"},
|
| 126 |
+
"sha256": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"},
|
| 127 |
+
"license": {"type": "string", "minLength": 1}
|
| 128 |
+
}
|
| 129 |
+
},
|
| 130 |
+
"device": {
|
| 131 |
+
"type": "object", "additionalProperties": false,
|
| 132 |
+
"required": ["kind", "model", "count"],
|
| 133 |
+
"properties": {
|
| 134 |
+
"kind": {"enum": ["CPU", "GPU", "REMOTE_ACCELERATOR"]},
|
| 135 |
+
"model": {"type": "string", "minLength": 1},
|
| 136 |
+
"count": {"type": "integer", "minimum": 1}
|
| 137 |
+
}
|
| 138 |
+
},
|
| 139 |
+
"rawTrial": {
|
| 140 |
+
"type": "object", "additionalProperties": false,
|
| 141 |
+
"required": ["trial", "value", "unit"],
|
| 142 |
+
"properties": {
|
| 143 |
+
"trial": {"type": "integer", "minimum": 1},
|
| 144 |
+
"value": {"type": "number"},
|
| 145 |
+
"unit": {"type": "string", "minLength": 1}
|
| 146 |
+
}
|
| 147 |
+
},
|
| 148 |
+
"measurement": {
|
| 149 |
+
"type": "object", "additionalProperties": false,
|
| 150 |
+
"required": ["metric", "value", "unit", "estimator", "confidence_interval", "summary_tolerance"],
|
| 151 |
+
"properties": {
|
| 152 |
+
"metric": {"type": "string", "minLength": 1},
|
| 153 |
+
"value": {"type": "number"},
|
| 154 |
+
"unit": {"type": "string", "minLength": 1},
|
| 155 |
+
"estimator": {"enum": ["mean", "median"]},
|
| 156 |
+
"summary_tolerance": {"type": "number", "minimum": 0},
|
| 157 |
+
"confidence_interval": {
|
| 158 |
+
"oneOf": [
|
| 159 |
+
{"type": "null"},
|
| 160 |
+
{"type": "array", "prefixItems": [{"type": "number"}, {"type": "number"}], "items": false, "minItems": 2, "maxItems": 2}
|
| 161 |
+
]
|
| 162 |
+
}
|
| 163 |
+
}
|
| 164 |
+
},
|
| 165 |
+
"correctness": {
|
| 166 |
+
"type": "object", "additionalProperties": false,
|
| 167 |
+
"required": ["passed", "checks"],
|
| 168 |
+
"properties": {
|
| 169 |
+
"passed": {"const": true},
|
| 170 |
+
"checks": {
|
| 171 |
+
"type": "array", "minItems": 1,
|
| 172 |
+
"items": {
|
| 173 |
+
"type": "object", "additionalProperties": false,
|
| 174 |
+
"required": ["name", "passed", "tolerance"],
|
| 175 |
+
"properties": {
|
| 176 |
+
"name": {"type": "string", "minLength": 1},
|
| 177 |
+
"passed": {"const": true},
|
| 178 |
+
"tolerance": {"type": "string", "minLength": 1}
|
| 179 |
+
}
|
| 180 |
+
}
|
| 181 |
+
}
|
| 182 |
+
}
|
| 183 |
+
},
|
| 184 |
+
"energy": {
|
| 185 |
+
"oneOf": [
|
| 186 |
+
{
|
| 187 |
+
"type": "object", "additionalProperties": false,
|
| 188 |
+
"required": ["status", "blocker"],
|
| 189 |
+
"properties": {
|
| 190 |
+
"status": {"const": "NOT_OBSERVABLE"},
|
| 191 |
+
"blocker": {"type": "string", "minLength": 1}
|
| 192 |
+
}
|
| 193 |
+
},
|
| 194 |
+
{
|
| 195 |
+
"type": "object", "additionalProperties": false,
|
| 196 |
+
"required": [
|
| 197 |
+
"status", "exporter_id", "sampled_at_start", "sampled_at_end",
|
| 198 |
+
"joules_before", "joules_after", "work_units_before", "work_units_after",
|
| 199 |
+
"work_unit", "fresh"
|
| 200 |
+
],
|
| 201 |
+
"properties": {
|
| 202 |
+
"status": {"const": "MEASURED_DELTA"},
|
| 203 |
+
"exporter_id": {"type": "string", "minLength": 1},
|
| 204 |
+
"sampled_at_start": {"type": "string", "format": "date-time"},
|
| 205 |
+
"sampled_at_end": {"type": "string", "format": "date-time"},
|
| 206 |
+
"joules_before": {"type": "number", "minimum": 0},
|
| 207 |
+
"joules_after": {"type": "number", "minimum": 0},
|
| 208 |
+
"work_units_before": {"type": "number", "minimum": 0},
|
| 209 |
+
"work_units_after": {"type": "number", "minimum": 0},
|
| 210 |
+
"work_unit": {"type": "string", "minLength": 1},
|
| 211 |
+
"fresh": {"const": true}
|
| 212 |
+
}
|
| 213 |
+
}
|
| 214 |
+
]
|
| 215 |
+
}
|
| 216 |
+
}
|
| 217 |
+
}
|
schemas/quant-claims/claim.v1.schema.json
ADDED
|
@@ -0,0 +1,118 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
| 3 |
+
"$id": "https://a-11-oy.com/schemas/quant-claims/claim.v1.schema.json",
|
| 4 |
+
"title": "SZL Quant External Claim Resolution v1",
|
| 5 |
+
"type": "object",
|
| 6 |
+
"additionalProperties": false,
|
| 7 |
+
"required": [
|
| 8 |
+
"schema_version",
|
| 9 |
+
"claim_id",
|
| 10 |
+
"claim",
|
| 11 |
+
"external_report",
|
| 12 |
+
"protocol",
|
| 13 |
+
"required_hardware_class",
|
| 14 |
+
"measurement_receipt_id",
|
| 15 |
+
"blocked_by",
|
| 16 |
+
"szl_label",
|
| 17 |
+
"szl_measured",
|
| 18 |
+
"measurement_gate"
|
| 19 |
+
],
|
| 20 |
+
"properties": {
|
| 21 |
+
"schema_version": {"const": "szl.quant.claim/v1"},
|
| 22 |
+
"claim_id": {"type": "string", "pattern": "^[a-z0-9][a-z0-9-]{2,95}$"},
|
| 23 |
+
"claim": {"type": "string", "minLength": 1, "maxLength": 240},
|
| 24 |
+
"external_report": {
|
| 25 |
+
"type": "object",
|
| 26 |
+
"additionalProperties": false,
|
| 27 |
+
"required": ["value", "units", "scope", "source"],
|
| 28 |
+
"properties": {
|
| 29 |
+
"value": {"type": "string", "minLength": 1, "maxLength": 120},
|
| 30 |
+
"units": {"type": "string", "minLength": 1, "maxLength": 80},
|
| 31 |
+
"scope": {"type": "string", "minLength": 1, "maxLength": 500},
|
| 32 |
+
"source": {
|
| 33 |
+
"type": "object",
|
| 34 |
+
"additionalProperties": false,
|
| 35 |
+
"required": ["organization", "title", "url", "kind"],
|
| 36 |
+
"properties": {
|
| 37 |
+
"organization": {"type": "string", "minLength": 1, "maxLength": 120},
|
| 38 |
+
"title": {"type": "string", "minLength": 1, "maxLength": 240},
|
| 39 |
+
"url": {"type": "string", "pattern": "^https://"},
|
| 40 |
+
"kind": {
|
| 41 |
+
"enum": ["OFFICIAL_VENDOR_REPORT", "OFFICIAL_RESEARCH", "OFFICIAL_DOCS", "OFFICIAL_MODEL_CARD", "ACADEMIC_PAPER"]
|
| 42 |
+
}
|
| 43 |
+
}
|
| 44 |
+
}
|
| 45 |
+
}
|
| 46 |
+
},
|
| 47 |
+
"protocol": {
|
| 48 |
+
"type": "object",
|
| 49 |
+
"additionalProperties": false,
|
| 50 |
+
"required": ["id", "metric", "unit", "description", "requires_energy"],
|
| 51 |
+
"properties": {
|
| 52 |
+
"id": {"type": "string", "pattern": "^[a-z0-9][a-z0-9-]{2,95}$"},
|
| 53 |
+
"metric": {"type": "string", "pattern": "^[a-z][a-z0-9_]{2,95}$"},
|
| 54 |
+
"unit": {"type": "string", "minLength": 1, "maxLength": 80},
|
| 55 |
+
"description": {"type": "string", "minLength": 1, "maxLength": 500},
|
| 56 |
+
"requires_energy": {"type": "boolean"}
|
| 57 |
+
}
|
| 58 |
+
},
|
| 59 |
+
"required_hardware_class": {"type": "string", "pattern": "^[A-Z][A-Z0-9_]{2,95}$"},
|
| 60 |
+
"measurement_receipt_id": {
|
| 61 |
+
"oneOf": [
|
| 62 |
+
{"type": "null"},
|
| 63 |
+
{"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"}
|
| 64 |
+
]
|
| 65 |
+
},
|
| 66 |
+
"blocked_by": {
|
| 67 |
+
"type": "array",
|
| 68 |
+
"minItems": 1,
|
| 69 |
+
"items": {"type": "string", "minLength": 1, "maxLength": 240},
|
| 70 |
+
"uniqueItems": true
|
| 71 |
+
},
|
| 72 |
+
"szl_label": {"enum": ["ROADMAP", "MEASURED", "UNVERIFIED", "CONFLICT"]},
|
| 73 |
+
"szl_measured": {
|
| 74 |
+
"oneOf": [
|
| 75 |
+
{"type": "null"},
|
| 76 |
+
{
|
| 77 |
+
"type": "object",
|
| 78 |
+
"additionalProperties": false,
|
| 79 |
+
"required": ["metric", "value", "unit", "estimator", "confidence_interval", "summary_tolerance"],
|
| 80 |
+
"properties": {
|
| 81 |
+
"metric": {"type": "string"},
|
| 82 |
+
"value": {"type": "number"},
|
| 83 |
+
"unit": {"type": "string"},
|
| 84 |
+
"estimator": {"enum": ["mean", "median"]},
|
| 85 |
+
"summary_tolerance": {"type": "number", "minimum": 0},
|
| 86 |
+
"confidence_interval": {
|
| 87 |
+
"oneOf": [
|
| 88 |
+
{"type": "null"},
|
| 89 |
+
{
|
| 90 |
+
"type": "array",
|
| 91 |
+
"prefixItems": [{"type": "number"}, {"type": "number"}],
|
| 92 |
+
"items": false,
|
| 93 |
+
"minItems": 2,
|
| 94 |
+
"maxItems": 2
|
| 95 |
+
}
|
| 96 |
+
]
|
| 97 |
+
}
|
| 98 |
+
}
|
| 99 |
+
}
|
| 100 |
+
]
|
| 101 |
+
},
|
| 102 |
+
"measurement_gate": {
|
| 103 |
+
"type": "object",
|
| 104 |
+
"additionalProperties": false,
|
| 105 |
+
"required": ["promoted", "reason", "receipt_id"],
|
| 106 |
+
"properties": {
|
| 107 |
+
"promoted": {"type": "boolean"},
|
| 108 |
+
"reason": {"type": "string", "minLength": 1, "maxLength": 240},
|
| 109 |
+
"receipt_id": {
|
| 110 |
+
"oneOf": [
|
| 111 |
+
{"type": "null"},
|
| 112 |
+
{"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"}
|
| 113 |
+
]
|
| 114 |
+
}
|
| 115 |
+
}
|
| 116 |
+
}
|
| 117 |
+
}
|
| 118 |
+
}
|