Spaces:
Running on CPU Upgrade
Running on CPU Upgrade
Upload 6 files
Browse files- CHANGELOG.md +20 -6
- README.md +10 -6
- app.py +125 -3
CHANGELOG.md
CHANGED
|
@@ -1,23 +1,37 @@
|
|
| 1 |
-
# v190.
|
| 2 |
|
| 3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4 |
|
| 5 |
- In Space Test replay mode, the validation worker now loads the parent Build Run's successful automatic smoke artefacts (`generation_smoke.json`, `generation_smoke_payload*.json`) as the canonical request before schema reconstruction.
|
| 6 |
- Added `replay_source.json` and richer `resolved_validation_request.json` / failure diagnosis fields so replay failures can be compared against the parent smoke request.
|
| 7 |
- The Payload stage now records `Payload replayed from parent automatic smoke` instead of remaining pending when replay data exists.
|
| 8 |
|
| 9 |
-
# v190.
|
| 10 |
|
| 11 |
-
## v190.
|
| 12 |
|
| 13 |
- Normalizes linked validation replay payloads against the current Gradio endpoint schema before calling predict().
|
| 14 |
- Preserves typed Gradio choices and converts dropdown/radio numeric-looking args such as 1024 to schema strings such as "1024" when required.
|
| 15 |
- Writes validation_preflight.json, resolved_validation_request.json, schema_coercion.json, and validation_failure_diagnosis.json for every linked validation run.
|
| 16 |
- Keeps Space Test copy linked-only while explaining that the separate validation run remains attached to the selected Build Run.
|
| 17 |
|
| 18 |
-
# v190.
|
| 19 |
|
| 20 |
-
## v190.
|
| 21 |
|
| 22 |
- Treats Space Test Validation as a first-class linked validation workflow.
|
| 23 |
- Adds a single resolved launch payload contract so replay/complete/recover/discovery-first modes only launch when coherent.
|
|
|
|
| 1 |
+
## v190.28 — Runs Explorer linked validation parent metadata
|
| 2 |
|
| 3 |
+
- Build Run cards now show linked/manual validation strips from parent metadata (`manual_validation_status` and `linked_validations`) even when the child Validation Run is not present in the Runs Explorer cache.
|
| 4 |
+
- Runs Explorer remains build-first: linked validations stay summarized on their parent Build Run, while automatic Build Run smoke results remain separate from manual linked validation state.
|
| 5 |
+
|
| 6 |
+
|
| 7 |
+
## v190.28 — Space Test payload provenance and prefill correctness
|
| 8 |
+
|
| 9 |
+
- Uses automatic smoke artefacts as the visible Space Test prefill source when available.
|
| 10 |
+
- Clears stale dirty validation args when a Build Run is selected from Runs Explorer.
|
| 11 |
+
- Sends and stores a backend validation launch payload for linked validations.
|
| 12 |
+
- Writes validation_launch_payload.json in validation runs before the worker resolves/predicts.
|
| 13 |
+
- Surfaces payload source provenance across UI, backend launch metadata, and worker artefacts.
|
| 14 |
+
|
| 15 |
+
# v190.28 — Replay uses parent smoke artefact as source of truth
|
| 16 |
+
|
| 17 |
+
## v190.28 — Linked validation replay hardening
|
| 18 |
|
| 19 |
- In Space Test replay mode, the validation worker now loads the parent Build Run's successful automatic smoke artefacts (`generation_smoke.json`, `generation_smoke_payload*.json`) as the canonical request before schema reconstruction.
|
| 20 |
- Added `replay_source.json` and richer `resolved_validation_request.json` / failure diagnosis fields so replay failures can be compared against the parent smoke request.
|
| 21 |
- The Payload stage now records `Payload replayed from parent automatic smoke` instead of remaining pending when replay data exists.
|
| 22 |
|
| 23 |
+
# v190.28 — Linked validation worker schema coercion and replay fix
|
| 24 |
|
| 25 |
+
## v190.28 — Validation worker reliability
|
| 26 |
|
| 27 |
- Normalizes linked validation replay payloads against the current Gradio endpoint schema before calling predict().
|
| 28 |
- Preserves typed Gradio choices and converts dropdown/radio numeric-looking args such as 1024 to schema strings such as "1024" when required.
|
| 29 |
- Writes validation_preflight.json, resolved_validation_request.json, schema_coercion.json, and validation_failure_diagnosis.json for every linked validation run.
|
| 30 |
- Keeps Space Test copy linked-only while explaining that the separate validation run remains attached to the selected Build Run.
|
| 31 |
|
| 32 |
+
# v190.28 — UI/CSS convergence and Space Test validation polish
|
| 33 |
|
| 34 |
+
## v190.28 — Space Test validation reliability
|
| 35 |
|
| 36 |
- Treats Space Test Validation as a first-class linked validation workflow.
|
| 37 |
- Adds a single resolved launch payload contract so replay/complete/recover/discovery-first modes only launch when coherent.
|
README.md
CHANGED
|
@@ -13,13 +13,17 @@ hf_oauth_scopes:
|
|
| 13 |
- read-billing
|
| 14 |
---
|
| 15 |
|
| 16 |
-
## v190.
|
| 17 |
|
| 18 |
-
- v190.
|
| 19 |
|
| 20 |
-
## v190.
|
| 21 |
|
| 22 |
-
- v190.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
- Runs Explorer and Run Stats now treat automatic and effective statuses explicitly so manually validated partial builds remain discoverable in both Partial and Validated views.
|
| 24 |
- Space Test is contextualized from the selected Build Run even before pressing Prefill, while remaining linked-only for launches.
|
| 25 |
|
|
@@ -419,13 +423,13 @@ The Active Run progress header now separates pipeline completion from product ve
|
|
| 419 |
|
| 420 |
- v190.9 makes terminal Space Test validation status survive page reloads. Failed, success, partial, and manual validation outcomes override stale `running` launch metadata in the Runs Explorer, delete modal, and progress polling.
|
| 421 |
|
| 422 |
-
## v190.
|
| 423 |
|
| 424 |
- Eval archive records now distinguish automatic verdicts from effective verdicts after linked Space Test validation.
|
| 425 |
- Archived records include anonymized linked-validation, endpoint-discovery and smoke-retry metrics while preserving privacy constraints.
|
| 426 |
- Instance owners can flush all archived eval records from the mounted eval bucket without deleting the eval archive configuration.
|
| 427 |
|
| 428 |
-
## v190.
|
| 429 |
|
| 430 |
- Linked Space Test validations now normalize replay/complete/recover payloads against the current Gradio schema before predict().
|
| 431 |
- Validation runs write explicit preflight, resolved request, schema coercion, and failure diagnosis artifacts for debugging.
|
|
|
|
| 13 |
- read-billing
|
| 14 |
---
|
| 15 |
|
| 16 |
+
## v190.28 Runs Explorer linked validation parent metadata
|
| 17 |
|
| 18 |
+
- v190.28 makes Build Run cards display manual linked validation state directly from parent metadata (`manual_validation_status.json` / `linked_validations.json`), so the summary is visible even when linked Validation Runs are not listed as standalone rows. Automatic smoke status remains separate from manual linked validation status.
|
| 19 |
|
| 20 |
+
## v190.28 Replay uses parent smoke artefact as source of truth
|
| 21 |
|
| 22 |
+
- v190.28 makes linked Space Test replay mode load the parent Build Run's successful automatic smoke artefacts before attempting any schema reconstruction. The validation worker writes `replay_source.json`, `resolved_validation_request.json`, and diagnostics so a replay cannot fail opaquely at the Payload stage.
|
| 23 |
+
|
| 24 |
+
## v190.28 Linked validation hardening
|
| 25 |
+
|
| 26 |
+
- v190.28 hardens linked Space Test success/failure paths: parent Build Run updates read API/output metadata from the smoke result, failed manual attempts cannot downgrade an already validated parent, and latency/ZeroGPU recommendations expose their linked-test hardware provenance.
|
| 27 |
- Runs Explorer and Run Stats now treat automatic and effective statuses explicitly so manually validated partial builds remain discoverable in both Partial and Validated views.
|
| 28 |
- Space Test is contextualized from the selected Build Run even before pressing Prefill, while remaining linked-only for launches.
|
| 29 |
|
|
|
|
| 423 |
|
| 424 |
- v190.9 makes terminal Space Test validation status survive page reloads. Failed, success, partial, and manual validation outcomes override stale `running` launch metadata in the Runs Explorer, delete modal, and progress polling.
|
| 425 |
|
| 426 |
+
## v190.28 Eval archive effective outcomes
|
| 427 |
|
| 428 |
- Eval archive records now distinguish automatic verdicts from effective verdicts after linked Space Test validation.
|
| 429 |
- Archived records include anonymized linked-validation, endpoint-discovery and smoke-retry metrics while preserving privacy constraints.
|
| 430 |
- Instance owners can flush all archived eval records from the mounted eval bucket without deleting the eval archive configuration.
|
| 431 |
|
| 432 |
+
## v190.28 Linked validation replay reliability
|
| 433 |
|
| 434 |
- Linked Space Test validations now normalize replay/complete/recover payloads against the current Gradio schema before predict().
|
| 435 |
- Validation runs write explicit preflight, resolved request, schema coercion, and failure diagnosis artifacts for debugging.
|
app.py
CHANGED
|
@@ -102,6 +102,91 @@ def _json_error(exc: Exception) -> HTTPException:
|
|
| 102 |
|
| 103 |
|
| 104 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 105 |
def _job_url_from_result(result: dict[str, Any], username: str | None) -> str:
|
| 106 |
explicit = (result.get("job_url") or "").strip() if isinstance(result.get("job_url"), str) else result.get("job_url")
|
| 107 |
if explicit:
|
|
@@ -570,8 +655,39 @@ def register_custom_routes(fastapi_app: FastAPI) -> None:
|
|
| 570 |
if not space_test_policy.get("enabled"):
|
| 571 |
raise ValueError(str(space_test_policy.get("message") or "This Build Run is not eligible for linked Space Test yet."))
|
| 572 |
api_name_for_validation = payload.get("api_name")
|
| 573 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 574 |
api_name_for_validation = ""
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 575 |
result = launch_validate_existing_space_job(
|
| 576 |
token=ctx["token"],
|
| 577 |
username=ctx["username"],
|
|
@@ -580,10 +696,12 @@ def register_custom_routes(fastapi_app: FastAPI) -> None:
|
|
| 580 |
api_name=api_name_for_validation,
|
| 581 |
test_args_json=json.dumps(test_args, ensure_ascii=False),
|
| 582 |
test_kwargs_json=json.dumps(test_kwargs, ensure_ascii=False),
|
| 583 |
-
expected_output_type=
|
| 584 |
live_timeout_seconds=payload.get("live_timeout_seconds") or 1800,
|
| 585 |
-
validation_mode=space_test_policy.get("mode")
|
| 586 |
effective_status_on_success=space_test_policy.get("effective_status_on_success") or "validated_after_manual_space_test",
|
|
|
|
|
|
|
| 587 |
run_id=payload.get("run_id"),
|
| 588 |
bucket_name=bucket_name,
|
| 589 |
)
|
|
@@ -880,6 +998,10 @@ def register_custom_routes(fastapi_app: FastAPI) -> None:
|
|
| 880 |
"generation_smoke": bundle.get("generation_smoke") or {},
|
| 881 |
"api_schema": bundle.get("api_schema") or {},
|
| 882 |
"validation_payload": bundle.get("validation_payload") or {},
|
|
|
|
|
|
|
|
|
|
|
|
|
| 883 |
"hardware_strategy": bundle.get("hardware_strategy") or {},
|
| 884 |
"technical_blockers": bundle.get("technical_blockers") or {},
|
| 885 |
"repair_decision": bundle.get("repair_decision") or {},
|
|
|
|
| 102 |
|
| 103 |
|
| 104 |
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
def _first_list_value(*values: Any) -> list[Any] | None:
|
| 108 |
+
for value in values:
|
| 109 |
+
if isinstance(value, list):
|
| 110 |
+
return value
|
| 111 |
+
return None
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
def _first_dict_value(*values: Any) -> dict[str, Any] | None:
|
| 115 |
+
for value in values:
|
| 116 |
+
if isinstance(value, dict):
|
| 117 |
+
return value
|
| 118 |
+
return None
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
def _linked_replay_source_from_parent_bundle(parent_run_id: str, parent_bundle: dict[str, Any], target_space: str, expected_output_type: str | None = None) -> dict[str, Any]:
|
| 122 |
+
"""Resolve the parent automatic smoke request server-side for replay Jobs.
|
| 123 |
+
|
| 124 |
+
The validation worker runs in a fresh Job and cannot assume the parent run
|
| 125 |
+
folder is present locally. Passing this compact, redacted source through the
|
| 126 |
+
Job environment prevents linked replay from silently falling back to schema
|
| 127 |
+
reconstruction when the parent already proved a working request.
|
| 128 |
+
"""
|
| 129 |
+
smoke = parent_bundle.get("generation_smoke") or {}
|
| 130 |
+
retry_payload = parent_bundle.get("generation_smoke_payload_retry") or {}
|
| 131 |
+
initial_payload = parent_bundle.get("generation_smoke_payload") or {}
|
| 132 |
+
resolved = parent_bundle.get("resolved_validation_request") or {}
|
| 133 |
+
state_obj = parent_bundle.get("state") or {}
|
| 134 |
+
state_smoke = state_obj.get("generation_smoke") if isinstance(state_obj.get("generation_smoke"), dict) else {}
|
| 135 |
+
status_tokens = {
|
| 136 |
+
str(smoke.get("status") or "").lower(),
|
| 137 |
+
str(state_obj.get("status") or "").lower(),
|
| 138 |
+
str(state_smoke.get("status") or "").lower(),
|
| 139 |
+
}
|
| 140 |
+
parent_was_successful = bool(status_tokens & {"success", "full_inference_success", "completed", "complete", "done"})
|
| 141 |
+
if not parent_was_successful:
|
| 142 |
+
return {}
|
| 143 |
+
api_name = (
|
| 144 |
+
smoke.get("api_name")
|
| 145 |
+
or retry_payload.get("api_name")
|
| 146 |
+
or initial_payload.get("api_name")
|
| 147 |
+
or resolved.get("api_name")
|
| 148 |
+
or state_smoke.get("api_name")
|
| 149 |
+
or ""
|
| 150 |
+
)
|
| 151 |
+
test_args = _first_list_value(
|
| 152 |
+
smoke.get("effective_args"),
|
| 153 |
+
smoke.get("test_args"),
|
| 154 |
+
retry_payload.get("test_args"),
|
| 155 |
+
retry_payload.get("effective_args"),
|
| 156 |
+
initial_payload.get("test_args"),
|
| 157 |
+
initial_payload.get("effective_args"),
|
| 158 |
+
resolved.get("test_args"),
|
| 159 |
+
state_smoke.get("effective_args"),
|
| 160 |
+
state_smoke.get("test_args"),
|
| 161 |
+
)
|
| 162 |
+
test_kwargs = _first_dict_value(
|
| 163 |
+
smoke.get("effective_kwargs"),
|
| 164 |
+
smoke.get("test_kwargs"),
|
| 165 |
+
retry_payload.get("test_kwargs"),
|
| 166 |
+
retry_payload.get("effective_kwargs"),
|
| 167 |
+
initial_payload.get("test_kwargs"),
|
| 168 |
+
initial_payload.get("effective_kwargs"),
|
| 169 |
+
resolved.get("test_kwargs"),
|
| 170 |
+
state_smoke.get("effective_kwargs"),
|
| 171 |
+
state_smoke.get("test_kwargs"),
|
| 172 |
+
) or {}
|
| 173 |
+
if not api_name or not isinstance(test_args, list):
|
| 174 |
+
return {}
|
| 175 |
+
parent_target = smoke.get("target_space") or state_obj.get("target_space") or state_smoke.get("target_space") or target_space or ""
|
| 176 |
+
return {
|
| 177 |
+
"source": "parent_automatic_smoke_backend",
|
| 178 |
+
"parent_run_id": parent_run_id,
|
| 179 |
+
"parent_target_space": parent_target,
|
| 180 |
+
"target_matches": not parent_target or parent_target == target_space,
|
| 181 |
+
"api_name": str(api_name if str(api_name).startswith("/") else f"/{api_name}"),
|
| 182 |
+
"test_args": test_args,
|
| 183 |
+
"test_kwargs": test_kwargs,
|
| 184 |
+
"expected_output_type": smoke.get("expected_output_type") or state_smoke.get("expected_output_type") or expected_output_type or "any",
|
| 185 |
+
"latency_seconds": smoke.get("latency_seconds") or state_smoke.get("latency_seconds"),
|
| 186 |
+
"parent_smoke_status": smoke.get("status") or state_smoke.get("status") or state_obj.get("status"),
|
| 187 |
+
"transport": "job_env",
|
| 188 |
+
}
|
| 189 |
+
|
| 190 |
def _job_url_from_result(result: dict[str, Any], username: str | None) -> str:
|
| 191 |
explicit = (result.get("job_url") or "").strip() if isinstance(result.get("job_url"), str) else result.get("job_url")
|
| 192 |
if explicit:
|
|
|
|
| 655 |
if not space_test_policy.get("enabled"):
|
| 656 |
raise ValueError(str(space_test_policy.get("message") or "This Build Run is not eligible for linked Space Test yet."))
|
| 657 |
api_name_for_validation = payload.get("api_name")
|
| 658 |
+
replay_source: dict[str, Any] = {}
|
| 659 |
+
if str(space_test_policy.get("mode") or payload.get("validation_mode") or "").strip().lower() == "replay":
|
| 660 |
+
replay_source = _linked_replay_source_from_parent_bundle(
|
| 661 |
+
parent_build_run_id,
|
| 662 |
+
parent_bundle,
|
| 663 |
+
requested_target,
|
| 664 |
+
str(payload.get("expected_output_type") or "any"),
|
| 665 |
+
)
|
| 666 |
+
if replay_source:
|
| 667 |
+
api_name_for_validation = replay_source.get("api_name") or api_name_for_validation
|
| 668 |
+
test_args = replay_source.get("test_args") if isinstance(replay_source.get("test_args"), list) else test_args
|
| 669 |
+
test_kwargs = replay_source.get("test_kwargs") if isinstance(replay_source.get("test_kwargs"), dict) else test_kwargs
|
| 670 |
+
if space_test_policy.get("requires_endpoint_discovery") and str(api_name_for_validation or "").strip() in {"", "/generate"} and not replay_source:
|
| 671 |
api_name_for_validation = ""
|
| 672 |
+
validation_launch_payload = {
|
| 673 |
+
"schema_version": "1.0",
|
| 674 |
+
"app_version": settings.app_version,
|
| 675 |
+
"parent_build_run_id": parent_build_run_id,
|
| 676 |
+
"target_space_id": requested_target,
|
| 677 |
+
"api_name": api_name_for_validation or "",
|
| 678 |
+
"expected_output_type": payload.get("expected_output_type") or replay_source.get("expected_output_type") or "any",
|
| 679 |
+
"test_args": test_args,
|
| 680 |
+
"test_kwargs": test_kwargs,
|
| 681 |
+
"validation_mode": space_test_policy.get("mode") or payload.get("validation_mode") or "complete",
|
| 682 |
+
"payload_source": payload.get("payload_source") or ("parent_automatic_smoke_backend" if replay_source else "ui_payload"),
|
| 683 |
+
"replay_source_present": bool(replay_source),
|
| 684 |
+
"ui_payload_source": payload.get("payload_source") or "",
|
| 685 |
+
"space_test_policy": {
|
| 686 |
+
"mode": space_test_policy.get("mode"),
|
| 687 |
+
"requires_endpoint_discovery": bool(space_test_policy.get("requires_endpoint_discovery")),
|
| 688 |
+
"effective_status_on_success": space_test_policy.get("effective_status_on_success"),
|
| 689 |
+
},
|
| 690 |
+
}
|
| 691 |
result = launch_validate_existing_space_job(
|
| 692 |
token=ctx["token"],
|
| 693 |
username=ctx["username"],
|
|
|
|
| 696 |
api_name=api_name_for_validation,
|
| 697 |
test_args_json=json.dumps(test_args, ensure_ascii=False),
|
| 698 |
test_kwargs_json=json.dumps(test_kwargs, ensure_ascii=False),
|
| 699 |
+
expected_output_type=validation_launch_payload["expected_output_type"],
|
| 700 |
live_timeout_seconds=payload.get("live_timeout_seconds") or 1800,
|
| 701 |
+
validation_mode=validation_launch_payload["validation_mode"], # effectively: validation_mode=space_test_policy.get("mode")
|
| 702 |
effective_status_on_success=space_test_policy.get("effective_status_on_success") or "validated_after_manual_space_test",
|
| 703 |
+
parent_replay_source_json=json.dumps(replay_source, ensure_ascii=False) if replay_source else None,
|
| 704 |
+
validation_launch_payload_json=json.dumps(validation_launch_payload, ensure_ascii=False),
|
| 705 |
run_id=payload.get("run_id"),
|
| 706 |
bucket_name=bucket_name,
|
| 707 |
)
|
|
|
|
| 998 |
"generation_smoke": bundle.get("generation_smoke") or {},
|
| 999 |
"api_schema": bundle.get("api_schema") or {},
|
| 1000 |
"validation_payload": bundle.get("validation_payload") or {},
|
| 1001 |
+
"validation_failure_diagnosis": bundle.get("validation_failure_diagnosis") or {},
|
| 1002 |
+
"payload_source": bundle.get("payload_source") or {},
|
| 1003 |
+
"validation_engine": bundle.get("validation_engine") or {},
|
| 1004 |
+
"resolved_validation_request": bundle.get("resolved_validation_request") or {},
|
| 1005 |
"hardware_strategy": bundle.get("hardware_strategy") or {},
|
| 1006 |
"technical_blockers": bundle.get("technical_blockers") or {},
|
| 1007 |
"repair_decision": bundle.get("repair_decision") or {},
|