Upload 6 files

CHANGELOG.md

@@ -1,5 +1,44 @@
+# v190.19 — Eval archive effective outcomes + owner flush
 
-## v190.14 — Build-first explorer and linked validation stats polish
+- Eval records now preserve automatic outcome and add effective outcome metadata for linked Space Test validations.
+- Backend archive publishing enriches records with linked validation summaries, endpoint discovery metrics, smoke retry metrics and stricter privacy flags without storing prompts, args, target Space IDs or generated code.
+- Eval publish statuses now use schema v1.3 and republish when a parent Build Run gains new effective validation state.
+- Instance owners / configured eval admins can flush archived eval records from the mounted eval bucket while preserving archive configuration.
+
+# v190.19 — Build completion matrix + endpoint discovery-first Space Test
+
+- Added canonical `space_test_policy` for replay/complete/recover/blocked/unavailable Build Run states.
+- Linked Space Test can now discover Gradio endpoints before validation when a Build Run has no known endpoint schema.
+- Space Test UI consumes policy labels/messages and shows discovery-first validation state.
+- Runs Explorer/Stats can surface recoverable and needs-validation states without treating Space Test as standalone.
+
+## v190.19 — Linked validation hardening and effective status UX
+
+- Fixed linked Space Test parent update after smoke success by sourcing `api_name` and `expected_output_type` from the smoke payload/environment instead of out-of-scope variables.
+- Prevented failed linked validation attempts from downgrading a parent Build Run that already has a successful linked validation.
+- Clarified effective status filters/stats, Eval Archive effective hints, Space Test context, and latency/ZeroGPU recommendation provenance.
+
+## v190.16 — Cascade delete linked Space Tests
+
+- Deleting a Build Run now cascades to its linked Space Test validation runs so no hidden validation artifacts are orphaned.
+- Deleting a linked validation run updates the parent Build Run linked validation metadata and manual validation status.
+- The delete modal now reports linked validation counts and the frontend removes parent/child cache entries together.
+
+## v190.16 — Build-first explorer compatibility cleanup
+
+- Keeps Runs Explorer simple after the linked-only Space Test change: Build Runs remain the primary list items, linked validations are summarized on their parent, and no standalone/legacy validation section is introduced.
+- Hardens the run cache, stats, pagination, and selection paths against null, partial, or legacy Build Run records that do not contain manual validation metadata.
+- Fixes linked validation parent selection without duplicating variables in the snapshot path.
+- Keeps the v190.12 generic Gradio schema-choice smoke retry intact.
+
+
+## v190.16 hotfix — Runs Explorer null-safe linked validation cache
+
+- Hardened Runs Explorer against null or malformed run summaries returned by stale bucket/list responses.
+- Preserved build-first linked validation grouping while filtering invalid cache entries safely.
+- Added regression coverage for null-safe linked validation parent lookup.
+
+## v190.16 — Build-first explorer and linked validation stats polish
 
 - Treat linked validation runs as children of Build Runs in Runs Explorer by default.
 - Rework Run Stats around Build Runs, linked tests, automatic success, and effective success.
@@ -15,9 +54,9 @@
 - Active Run snapshots and run summaries can now surface manual validation status, effective verdict, latency, and ZeroGPU duration measurements sourced from linked Space Test.
 - Runs Explorer labels linked validation runs, shows manual validation passed on Build cards, and opens the parent Build Run when a linked validation is selected.
 
-# Agentic Space Factory v190.14
+# Agentic Space Factory v190.16
 
-## v190.14 smoke retry and final summary hardening
+## v190.16 smoke retry and final summary hardening
 
 - Generation smoke payloads now preserve Gradio schema choice value types before calling predict.
 - If Gradio rejects a smoke payload with a choice type mismatch, the worker records the first error, coerces args to schema choices, and retries once.

README.md

@@ -12,10 +12,29 @@ hf_oauth_scopes:
   - jobs
   - read-billing
 ---
-## v190.14 smoke retry and live validation telemetry
 
-- v190.14 keeps Space Test prefill disabled until a build has reached a final Space state that can be tested manually.
-- v190.14 retries generation smoke once when Gradio reports schema-choice type mismatches, and writes terminal summary.json plus live_status.json for cleaner audits.
+## v190.19 Linked validation hardening
+
+- v190.19 hardens linked Space Test success/failure paths: parent Build Run updates read API/output metadata from the smoke result, failed manual attempts cannot downgrade an already validated parent, and latency/ZeroGPU recommendations expose their linked-test hardware provenance.
+- Runs Explorer and Run Stats now treat automatic and effective statuses explicitly so manually validated partial builds remain discoverable in both Partial and Validated views.
+- Space Test is contextualized from the selected Build Run even before pressing Prefill, while remaining linked-only for launches.
+
+
+## v190.16 Cascade delete linked Space Tests
+
+- Build Run deletion now removes linked Space Test validation runs from the run bucket as part of the same cleanup.
+- Linked validation deletion keeps the parent Build Run but refreshes its manual validation metadata.
+- Runs Explorer remains Build-first and clears parent/child rows from cache immediately after deletion.
+
+## v190.16 Build-first explorer compatibility
+
+- v190.16 keeps Runs Explorer simple: Build Runs are the primary objects, linked Space Tests are summarized on their parent, and legacy/incomplete Build Runs remain visible without requiring validation metadata.
+- It hardens the run cache against null or partial records and keeps validation runs out of the primary explorer by default.
+
+## v190.16 smoke retry and live validation telemetry
+
+- v190.16 keeps Space Test prefill disabled until a build has reached a final Space state that can be tested manually.
+- v190.16 retries generation smoke once when Gradio reports schema-choice type mismatches, and writes terminal summary.json plus live_status.json for cleaner audits.
 - The canonical run snapshot now includes a `live_validation` telemetry model so the Live test phase can show Space build/runtime, health, Gradio endpoint discovery and generation smoke state instead of a generic running message.
 - Active polling runs at a faster visible cadence for smoother event updates.
 
@@ -395,3 +414,9 @@ The Active Run progress header now separates pipeline completion from product ve
 ## v190.9 — Space Test canonical validation persistence
 
 - v190.9 makes terminal Space Test validation status survive page reloads. Failed, success, partial, and manual validation outcomes override stale `running` launch metadata in the Runs Explorer, delete modal, and progress polling.
+
+## v190.19 Eval archive effective outcomes
+
+- Eval archive records now distinguish automatic verdicts from effective verdicts after linked Space Test validation.
+- Archived records include anonymized linked-validation, endpoint-discovery and smoke-retry metrics while preserving privacy constraints.
+- Instance owners can flush all archived eval records from the mounted eval bucket without deleting the eval archive configuration.

app.py

@@ -12,7 +12,7 @@ from huggingface_hub import HfApi, attach_huggingface_oauth
 
 from src.bucket import RunPaths, check_user_bucket, create_user_bucket, delete_run_folder, read_run_bundle, list_recent_runs, write_json, write_launch_metadata
 from src.config import settings, user_bucket_source
-from src.eval_config import activate_eval_archive_config, disable_eval_archive_config, public_eval_config
+from src.eval_config import activate_eval_archive_config, disable_eval_archive_config, flush_eval_archive_records, public_eval_config
 from src.eval_archive import maybe_publish_eval_record
 from src.auth import extract_oauth_context, public_oauth_context, oauth_warning_messages, verify_token_identity
 from src.jobs import (
@@ -420,6 +420,19 @@ def register_custom_routes(fastapi_app: FastAPI) -> None:
             raise HTTPException(status_code=400, detail=str(exc)) from exc
         return JSONResponse(cfg)
 
+    @fastapi_app.post("/api/eval-archive/flush")
+    async def api_eval_archive_flush(request: Request):  # type: ignore[no-untyped-def]
+        ctx = _oauth_context_from_request(request)
+        try:
+            cfg = flush_eval_archive_records(username=ctx["username"])
+        except PermissionError as exc:
+            raise HTTPException(status_code=403, detail=str(exc)) from exc
+        except FileNotFoundError as exc:
+            raise HTTPException(status_code=409, detail=str(exc)) from exc
+        except Exception as exc:  # noqa: BLE001
+            raise HTTPException(status_code=400, detail=str(exc)) from exc
+        return JSONResponse(cfg)
+
     @fastapi_app.get("/api/oauth/diagnostics")
     async def api_oauth_diagnostics(request: Request):  # type: ignore[no-untyped-def]
         ctx = extract_oauth_context(request)
@@ -544,7 +557,7 @@ def register_custom_routes(fastapi_app: FastAPI) -> None:
             if not isinstance(test_kwargs, dict):
                 raise ValueError("test_kwargs_json must be a JSON object; it is passed as keyword args to gradio_client.predict().")
             parent_build_run_id = validate_run_id(str(payload.get("parent_build_run_id") or ""))
-            parent_bundle = read_run_bundle(parent_build_run_id, bucket_source=bucket_status.get("bucket_source") or user_bucket_source(username=ctx["username"], bucket_name=bucket_name), token=ctx["token"], include_heavy=False)
+            parent_bundle = read_run_bundle(parent_build_run_id, bucket_source=bucket_status.get("bucket_source") or user_bucket_source(username=ctx["username"], bucket_name=bucket_name), token=ctx["token"], include_heavy=True)
             parent_summary = parent_bundle.get("summary") or {}
             parent_target = parent_summary.get("target_space") or (parent_bundle.get("state") or {}).get("target_space") or (parent_bundle.get("launch") or {}).get("target_space") or ""
             requested_target = str(payload.get("target_space_id") or "").strip()
@@ -552,16 +565,25 @@ def register_custom_routes(fastapi_app: FastAPI) -> None:
                 raise ValueError("Linked Space Test requires a parent Build Run with a generated target Space.")
             if requested_target != parent_target:
                 raise ValueError("Space Test must remain linked to the selected Build Run target Space. Standalone validation is not supported in this app.")
+            parent_view = build_run_view_model(parent_build_run_id, parent_bundle, bucket_source=bucket_status.get("bucket_source") or user_bucket_source(username=ctx["username"], bucket_name=bucket_name))
+            space_test_policy = parent_view.get("space_test_policy") or parent_view.get("space_test", {}).get("policy") or {}
+            if not space_test_policy.get("enabled"):
+                raise ValueError(str(space_test_policy.get("message") or "This Build Run is not eligible for linked Space Test yet."))
+            api_name_for_validation = payload.get("api_name")
+            if space_test_policy.get("requires_endpoint_discovery") and str(api_name_for_validation or "").strip() in {"", "/generate"}:
+                api_name_for_validation = ""
             result = launch_validate_existing_space_job(
                 token=ctx["token"],
                 username=ctx["username"],
                 target_space_id=requested_target,
                 parent_build_run_id=parent_build_run_id,
-                api_name=payload.get("api_name"),
+                api_name=api_name_for_validation,
                 test_args_json=json.dumps(test_args, ensure_ascii=False),
                 test_kwargs_json=json.dumps(test_kwargs, ensure_ascii=False),
                 expected_output_type=payload.get("expected_output_type"),
                 live_timeout_seconds=payload.get("live_timeout_seconds") or 1800,
+                validation_mode=space_test_policy.get("mode"),
+                effective_status_on_success=space_test_policy.get("effective_status_on_success") or "validated_after_manual_space_test",
                 run_id=payload.get("run_id"),
                 bucket_name=bucket_name,
             )
@@ -723,26 +745,73 @@ def register_custom_routes(fastapi_app: FastAPI) -> None:
                 body = {}
         except Exception:
             body = {}
+
         delete_space = bool(body.get("delete_space"))
         bundle: dict[str, Any] = {}
+        try:
+            bundle = read_run_bundle(run_id, bucket_source=bucket_source, token=ctx["token"], include_heavy=False)
+        except Exception:
+            bundle = {}
+
+        is_validation = _run_is_validation_like({**bundle, "run_id": run_id})
+        linked_children: list[dict[str, Any]] = []
+        if not is_validation:
+            linked_children = _linked_validation_summaries(run_id, bucket_source=bucket_source, token=ctx["token"])
+
         space_report: dict[str, Any] = {"space_delete_requested": delete_space, "space_deleted": False}
         if delete_space:
             try:
-                bundle = read_run_bundle(run_id, bucket_source=bucket_source, token=ctx["token"], include_heavy=False)
-                associated_space = _associated_space_from_bundle(bundle)
-                kind = str((bundle.get("summary") or {}).get("kind") or (bundle.get("state") or {}).get("kind") or (bundle.get("launch") or {}).get("kind") or "").lower()
-                if "validation" in kind or "space_test" in kind:
+                if is_validation:
                     raise PermissionError("Associated Space deletion is not available for validation runs.")
+                associated_space = _associated_space_from_bundle(bundle)
                 space_report = _delete_associated_space(associated_space, username=ctx["username"], token=ctx["token"])
             except Exception as exc:  # noqa: BLE001
                 space_report = {"space_delete_requested": True, "space_deleted": False, "space_delete_error": redact(str(exc))}
+
+        deleted_linked_validations: list[str] = []
+        linked_delete_errors: list[str] = []
+        for child in linked_children:
+            child_id = str(child.get("run_id") or "").strip()
+            if not child_id or child_id == run_id:
+                continue
+            try:
+                delete_run_folder(validate_run_id(child_id), bucket_source=bucket_source, token=ctx["token"])
+                deleted_linked_validations.append(child_id)
+            except FileNotFoundError:
+                deleted_linked_validations.append(child_id)
+            except Exception as exc:  # noqa: BLE001
+                linked_delete_errors.append(f"{child_id}: {redact(str(exc))}")
+
+        parent_update: dict[str, Any] = {}
+        parent_build_run_id = _linked_validation_parent_id({**bundle, "run_id": run_id}) if is_validation else ""
         try:
             delete_report = delete_run_folder(run_id, bucket_source=bucket_source, token=ctx["token"])
         except FileNotFoundError:
             delete_report = {"matched_count": 0, "deleted_count": 0, "remaining_count": 0}
         except Exception as exc:  # noqa: BLE001
             raise HTTPException(status_code=400, detail=redact(str(exc))) from exc
-        return JSONResponse({"ok": True, "run_id": run_id, "deleted": True, **delete_report, **space_report})
+
+        if is_validation and parent_build_run_id:
+            remaining = [row for row in _linked_validation_summaries(parent_build_run_id, bucket_source=bucket_source, token=ctx["token"]) if str(row.get("run_id") or "") != run_id]
+            try:
+                parent_update = _write_parent_linked_validation_state(parent_build_run_id, remaining, bucket_source=bucket_source, token=ctx["token"])
+            except Exception as exc:  # noqa: BLE001
+                parent_update = {"parent_update_error": redact(str(exc))}
+
+        response = {
+            "ok": True,
+            "run_id": run_id,
+            "deleted": True,
+            **delete_report,
+            **space_report,
+            "deleted_linked_validations": deleted_linked_validations,
+            "linked_delete_errors": linked_delete_errors,
+            "parent_build_run_id": parent_build_run_id,
+            "parent_update": parent_update,
+        }
+        if linked_delete_errors:
+            response["partial_delete"] = True
+        return JSONResponse(response)
 
     @fastapi_app.get("/api/runs/{run_id}/progress")
     async def api_run_progress(request: Request, run_id: str, bucket_name: str = settings.bucket_name, include_job_logs: bool = False):  # type: ignore[no-untyped-def]
@@ -1298,6 +1367,113 @@ def _delete_associated_space(space_id: str, *, username: str, token: str) -> dic
             return {"space_delete_requested": True, "space_deleted": False, "space_already_missing": True, "space_id": cleaned, "space_delete_error": message}
         return {"space_delete_requested": True, "space_deleted": False, "space_id": cleaned, "space_delete_error": message}
 
+
+def _run_kind_from_bundle(bundle: dict[str, Any]) -> str:
+    summary = bundle.get("summary") or bundle.get("summary_file") or {}
+    state = bundle.get("state") or {}
+    launch = bundle.get("launch") or {}
+    return str(summary.get("kind") or state.get("kind") or launch.get("kind") or bundle.get("kind") or "").lower()
+
+
+def _run_is_validation_like(bundle_or_summary: dict[str, Any]) -> bool:
+    kind = str(
+        bundle_or_summary.get("kind")
+        or bundle_or_summary.get("run_type")
+        or (bundle_or_summary.get("summary") or {}).get("kind")
+        or (bundle_or_summary.get("state") or {}).get("kind")
+        or (bundle_or_summary.get("launch") or {}).get("kind")
+        or ""
+    ).lower()
+    run_id = str(bundle_or_summary.get("run_id") or (bundle_or_summary.get("summary") or {}).get("run_id") or "").lower()
+    return "validation" in kind or "space_test" in kind or "validate" in kind or run_id.startswith("validate-")
+
+
+def _linked_validation_parent_id(run: dict[str, Any]) -> str:
+    summary = run.get("summary") or {}
+    state = run.get("state") or {}
+    launch = run.get("launch") or {}
+    return str(
+        run.get("parent_build_run_id")
+        or run.get("parentBuildRunId")
+        or summary.get("parent_build_run_id")
+        or state.get("parent_build_run_id")
+        or launch.get("parent_build_run_id")
+        or ""
+    ).strip()
+
+
+def _linked_validation_summaries(parent_run_id: str, *, bucket_source: str, token: str | None = None) -> list[dict[str, Any]]:
+    if not parent_run_id:
+        return []
+    try:
+        rows = list_recent_runs(bucket_source=bucket_source, token=token, limit=200)
+    except Exception:
+        return []
+    linked: list[dict[str, Any]] = []
+    for row in rows:
+        if not isinstance(row, dict):
+            continue
+        if not _run_is_validation_like(row):
+            continue
+        if _linked_validation_parent_id(row) == parent_run_id:
+            linked.append(row)
+    linked.sort(key=lambda r: str(r.get("updated_at") or r.get("created_at") or r.get("run_id") or ""), reverse=True)
+    return linked
+
+
+def _manual_validation_status_from_linked(parent_run_id: str, linked: list[dict[str, Any]]) -> dict[str, Any]:
+    successes = [row for row in linked if str(row.get("status") or row.get("effective_status") or "").lower() in {"full_inference_success", "success", "succeeded", "passed", "validated_after_manual_space_test"}]
+    failures = [row for row in linked if "failed" in str(row.get("status") or "").lower() or "error" in str(row.get("status") or "").lower()]
+    source = successes[0] if successes else (failures[0] if failures else {})
+    if not source:
+        return {"status": "none", "parent_build_run_id": parent_run_id, "validation_run_id": "", "updated_at": utc_now_iso()}
+    status = "success" if successes else "failed"
+    payload: dict[str, Any] = {
+        "status": status,
+        "parent_build_run_id": parent_run_id,
+        "validation_run_id": source.get("run_id") or "",
+        "target_space": source.get("target_space") or source.get("target_space_id") or "",
+        "api_name": source.get("api_name") or "",
+        "updated_at": utc_now_iso(),
+    }
+    if status == "success":
+        payload["effective_status"] = "validated_after_manual_space_test"
+        for key in ["latency_seconds", "observed_latency_seconds", "recommended_zero_gpu_duration_seconds", "recommended_zerogpu_duration_seconds", "recommendation_source", "recommendation_hardware", "hardware_used_for_validation"]:
+            if source.get(key) is not None:
+                payload[key] = source.get(key)
+        payload.setdefault("recommendation_source", "linked_space_test")
+    return payload
+
+
+def _write_parent_linked_validation_state(parent_run_id: str, linked: list[dict[str, Any]], *, bucket_source: str, token: str | None = None) -> dict[str, Any]:
+    paths = RunPaths(parent_run_id, bucket_source=bucket_source)
+    compact = []
+    for row in linked:
+        compact.append({
+            "validation_run_id": row.get("run_id") or "",
+            "status": row.get("status") or "unknown",
+            "target_space": row.get("target_space") or row.get("target_space_id") or "",
+            "api_name": row.get("api_name") or "",
+            "latency_seconds": row.get("latency_seconds") or row.get("observed_latency_seconds"),
+            "observed_latency_seconds": row.get("observed_latency_seconds") or row.get("latency_seconds"),
+            "recommended_zero_gpu_duration_seconds": row.get("recommended_zero_gpu_duration_seconds") or row.get("recommended_zerogpu_duration_seconds"),
+            "recommendation_source": row.get("recommendation_source") or "",
+            "recommendation_hardware": row.get("recommendation_hardware") or row.get("hardware_used_for_validation") or "",
+            "hardware_used_for_validation": row.get("hardware_used_for_validation") or row.get("recommendation_hardware") or "",
+            "updated_at": row.get("updated_at") or row.get("created_at") or "",
+        })
+    linked_payload = {
+        "parent_build_run_id": parent_run_id,
+        "validations": compact,
+        "updated_at": utc_now_iso(),
+    }
+    manual_status = _manual_validation_status_from_linked(parent_run_id, linked)
+    if manual_status.get("status") == "success":
+        linked_payload["effective_status"] = "validated_after_manual_space_test"
+    write_json(f"{paths.root}/linked_validations.json", linked_payload, token=token)
+    write_json(f"{paths.root}/manual_validation_status.json", manual_status, token=token)
+    return {"linked_validations": linked_payload, "manual_validation_status": manual_status}
+
 def create_app() -> FastAPI:
     """Create the product FastAPI app.