## v124 — Timeline and Run Explorer stabilization - Fixed the compact Overall Progress timeline so dynamically visible repair/failure groups cannot wrap dots onto a second row. - Removed smooth timeline auto-scroll that could cause visible flicker during polling. - Simplified Run Explorer type/status badges to neutral type labels and short status labels such as Success, Failed, Running and Manual. - Added regression checks for timeline single-row behavior and compact Run Explorer labels. ## v123 — Critical Job launch + failure timeline fix - Fixed a production-critical HF Job startup failure where the large embedded worker script could make Python fail before startup with `argument list too long`. - The worker script is now persisted as `runs//_worker.py` in the mounted run bucket before launch, and the Job receives only `WORKER_SCRIPT_PATH`. - Kept a small `WORKER_SCRIPT_B64` fallback only for compatibility/manual launch paths; normal app launches no longer put the worker source in env/argv. - Added a failure-log fallback so pre-worker Job failures such as `argument list too long` still create an explicit failed timeline event and red failure point. ## v122 — Stable UI rollback after latency cleanup regression - Reverted the risky v121 client-side run-detail cache/polling optimizations after they caused build/run UI instability and flicker in real usage. - Restored the proven v120 Active Run / Space Test selection and polling behavior while keeping the v118/v119/v120 Pi recovery core intact. - Kept the Agent recovery panel, unified agent traces, compact Run Explorer, and blockage protocol UI from the stable v120 line. - Removed the v121 rendering-signature shortcuts and progress-payload cache writes that could leave panels in stale or inconsistent states. - This release prioritizes correctness and stable polling over speculative latency optimization. ## v120 — Agent recovery UI for blockage protocol - Added an **Agent recovery** card to the Active Run panel so the v118/v119 core recovery protocol is visible to users. - The card summarizes Pi diagnosis decisions, selected action, logs quality/failure category/confidence, evidence, next step, and the diagnose → decide → act → revalidate ladder. - Exposed `repair/REPAIR_DECISION.json` and `repair/BLOCKAGE.json` through the backend bundle, progress payload, and Run traces dock. - Kept the existing unified `agent_trace.jsonl` as the canonical raw/redacted narrative across initial build, diagnosis, and repair. - Added regression tests for the recovery panel, decision/blockage artifact links, progress payload exposure, and docs. # Changelog ## v119 — unified agentic Pi traces - Added a run-level `traces/raw/agent_trace.jsonl` and `traces/redacted/agent_trace.jsonl` journal that ties together the initial Pi build, Pi blockage diagnosis, repair decision, and optional repair patch into one auditable agent narrative. - Kept specialized logs (`logs/pi_output.txt`, `logs/pi_diagnosis_output.txt`, `logs/pi_repair_output.txt`) as detailed artifacts, but made the curated `Pi RAW` / `Pi redacted` dock point to the unified trace when available. - Diagnosis and repair phases now add explicit trace records for command start/end, output artifacts, repair decisions, and repair plan/summary artifacts so they no longer look like disconnected conversations. ## v118 — Pi diagnosis blockage protocol - Added a Pi **blockage diagnosis** step before any code repair. Failures now produce `repair/INCIDENT_BRIEF.md`, `repair/PI_DIAGNOSIS_GOAL.md`, `repair/REPAIR_DECISION.json`, and `logs/pi_diagnosis_output.txt`. - Pi/Qwen now chooses a bounded action instead of being asked to patch immediately: `wait_for_logs`, `inspect_more_logs`, `factory_rebuild_same_code`, `patch_code`, `request_manual_hardware`, or `declare_technical_blocker`. - Added Factory guardrails: no code patch on empty/no-reason logs, no patch for auth/quota/hardware blockers, and strict one-action budgets for wait, same-code rebuild, and patch repair. - Added same-code factory rebuild recovery for HF build/runtime flakes where logs are empty or non-actionable. - Patch repair now runs only after Pi diagnosis chooses `patch_code` and the Factory validates the decision. - Added blockage artifacts and timeline events: `failure_detected`, `failure_diagnosis`, `pi_diagnosis`, `repair_decision`, `wait_for_logs`, `factory_rebuild`, `technical_blocker`, and `manual_hardware_required`. - Updated tests to protect the diagnosis-before-patch contract and the failure timeline mapping. ## v117 — Structured repair mode and repair timeline - Reworked Pi repair from a loose log-based retry into a structured repair method. - Added failure classification before repair (`dependency_error`, `import_error`, `gradio_api_mismatch`, `cuda_oom`, `model_loading_error`, `hf_auth_error`, `space_boot_timeout`, `wrong_output_type`, `zero_gpu_duration_error`, or `unknown_runtime_error`). - The worker now writes `repair/REPAIR_BRIEF.md`, requests `REPAIR_PLAN.md` before patching, and stores `REPAIR_SUMMARY.md` after patching. - Added strict repair constraints: patch the current workspace, do not rebuild unnecessarily, preserve real inference, do not fake outputs, keep the health endpoint, and preserve the validation contract. - Added repair sanity checks before redeploying a patched workspace. - Added repair-specific events: `repair_diagnosis`, `repair_plan`, `repair_patch`, `repair_upload`, and `repair_validation`. - Updated the progress timeline so repair phases are visible and terminal failures always have a red failure point. - Added a Repair document shortcut in the Active Run traces dock when repair artifacts exist. ## v116 — Compact Run Explorer and billing note relocation - Reworked the Run Explorer from large cards into a compact premium navigation list with one primary line, one context line, status pill, relative time and a discreet delete action. - Kept build and validation rows distinct while making the business entity more prominent than the raw run ID. - Moved the Inference Providers / paid credits / restricted A100-H200 fallback note out of New Build and into Billing & compute. - Added regression tests for the compact list and billing-note placement. ## v114 — Premium vector activity feed icons - Replaced text-coded activity markers such as `PI`, `API`, and `GPU` with inline SVG icons rendered by a dedicated `renderEventIcon(...)` helper. - Mapped event types to product-style vector icons for upload, hardware, Space creation, generation smoke tests, traces, reports, model analysis, auth, running and error states. - Added CSS states for vector activity icons, including status colors and a lightweight running animation for pending/running events. - Kept the event feed layout compact and readable while avoiding emoji or text badges as the primary iconography. ## v112 — Verification, cleanup, and documentation refresh - Added one-time JS binding helpers for navigation, action buttons, filters and form controls to prevent accidental listener duplication if initialization is re-run. - Switched run pagination to delegated events so rerendering the controls does not attach fresh listeners on every page update. - Added a final CSS cohesion layer for narrow panels, model pre-scan content, endpoint chips, Run Stats, run cards and activity feeds. - Updated README and docs to cover the calibrated model pre-scan, `Tongyi-MAI/Z-Image-Turbo` default, endpoint picker behavior, UI reliability checks, and current production-preview guidance. ## v111 — Calibrated model-card pre-scan default - Set the default model example to `Tongyi-MAI/Z-Image-Turbo`, the known-good default test model. - Calibrated the fast pre-scan to treat a clear Diffusers model card with safetensors and runnable usage code as a strong positive signal. - Added model-card guidance signals for Diffusers examples, pipeline class and runtime hints. ## v110 — Model pre-scan gate - Added a fast Model pre-scan card in New Build. - New backend endpoint: `POST /api/models/pre-scan`. - The scan checks Hub metadata, file names, safetensors, gated/private state, config/model_index and custom-code signals without downloading model weights. - Build launch is gated until the current model has been scanned. `unsupported` blocks launch; `risky` requires explicit user acknowledgement. ## v109 — Cohesive implementation mode labels - Centralized implementation-mode copy in shared UI helpers. - Kept compact dropdown labels and reused the same user-facing labels in Active Run and Run Explorer metadata. - Added a Build goal field in Active Run with hover help, without exposing internal mode ids. ## v108 — Compact implementation mode labels - Shortened the Implementation mode dropdown labels to keep the left column clean. - Added a dynamic helper below the field explaining the selected mode without changing internal values. ## v107 — Clear build goals and safer automatic hardware fallbacks - Renamed implementation-mode labels to user-facing build goals while keeping the same internal values. - Removed A100 and H200 from automatic fallback UI choices. - Added backend and worker-side sanitization so old clients cannot request high/restricted hardware as automatic fallback. - Updated docs to clarify that A100/H200-class hardware should be selected manually only when available to the account or organization. ## v106 — Documentation and production-readiness pass - Rewrote README to match the current v105 product UI: left storage/build/billing column, center Active Run/Space Test tabs, right Run Explorer/Run Stats column. - Updated architecture docs to describe the Pi gist instruction flow, per-user Bucket layout, validation-run separation, trace dock, run discovery, latency, and ZeroGPU duration recommendation. - Updated public-preview docs with the current demo path and production-readiness note. - Updated security docs with token, Bucket, raw/redacted trace, billing, and generated-Space review guidance. - Updated next steps to focus on production hardening, cost guardrails, trace safety, repair workflow, validation presets, and retention controls. ## v100 — Compact storage and billing cockpit - Reworked the left-column Run storage block into a compact status card with clearer checking/creating/ready feedback. - Added a Billing & compute card below New Build showing account, payment-readiness signal, inference-credit tier, and links to Hugging Face Billing, Jobs pricing, and Inference Providers pricing. - Added `/api/billing/status` to expose only the billing signals available through OAuth, without pretending to mirror numeric usage totals from the HF Billing dashboard. - Kept the build and validation workflows unchanged. ## v47 — Simple spec reset - Reset the custom UI to three top-level tabs only: Build, Runs, and Space Test. - Removed the run explorer from the Build tab so launching a new build stays focused. - Made Runs the single place to inspect current and historical runs, progress, events, reports, diagnostics, and artifacts. - Kept run selection in-place with no automatic tab switch. - Preserved backend endpoints, run view models, recovery, bucket gating, and Space Test validation. - Added responsive overrides so the simple layout stays readable on mobile. ## v46 — Run tab selection + mobile framing fixes - Selecting a run no longer forces a navigation switch; it updates the visible run detail panels in place. - Improved mobile responsive rules to prevent narrow grid columns from rendering words letter-by-letter. - Added regression tests for run selection navigation and mobile text wrapping safeguards. ## V40 — Tabbed product UI and link/progress polish - Replaced the left sidebar with a compact top tab workflow: **Run new build**, **Live progress**, and **Test Space**. - Kept the Run Explorer inside the Live progress cockpit, so previous runs are inspectable without confusing the build form. - Added a visible Logout action in the header. - Made the Job link more reliable for historical runs by reconstructing it from `job_id` when the stored URL is missing. - Reworked the live progress timeline to a compact wrapped step grid instead of a scrolling panel. - Automatically switches to Live progress after launching a build or validation. - Preserved bucket gating, per-user buckets, OAuth checks, hardware strategy, and validation Jobs. ## V36 - Docker OAuth dependency fix - Added `huggingface_hub[oauth]>=1.0.0` to the runtime requirements so FastAPI OAuth via `attach_huggingface_oauth(...)` can initialize inside the Docker Space. - Keeps the V35 Docker/FastAPI custom UI startup model unchanged: one `uvicorn` server, custom UI at `/`, `/custom` alias. ## v103 - Aggregated Run Stats - Replaced the ambiguous Run Health panel with an aggregate-only Run Stats panel below the Run Explorer. - Removed latest/current/issue rows that could mix multiple jobs and warnings. - Added indexed run counters, build success rate, validation pass rate, validation latency median/p95, and aggregated issue counts. - Kept Run Explorer as the navigation surface and central tabs as the detail surfaces. ## v102 - UI polish and fixed Bucket document links - Fixed Bucket document links to use `/tree/runs/...` file URLs instead of broken `/blob/runs/...` URLs. - Clarified the Billing & compute panel: OAuth can show payment readiness, while live quotas/spend remain in the Hugging Face Billing dashboard. - Reworked Run Health rows with explicit labels, titles, status and metadata. - Made Run Explorer cards narrow-safe so status/actions no longer overlap metadata. - Replaced Space Test validation columns with a compact horizontal point timeline. ## V35 — Docker FastAPI root custom UI startup - Switches the public product Space to `sdk: docker`. - Adds a Dockerfile running exactly one uvicorn process: `uvicorn app:app --host 0.0.0.0 --port 7860`. - Attaches Hugging Face OAuth directly to FastAPI with `attach_huggingface_oauth`. - Removes the Gradio-mounted debug app from the normal startup path. - Updates login/logout links to `/oauth/huggingface/login` and `/oauth/huggingface/logout`. - Keeps the custom dashboard at `/` and `/custom`. ## V34 — Space startup fix - Removed the explicit `uvicorn.run(app, host="0.0.0.0", port=7860)` launcher from `app.py`. - Kept the custom FastAPI/ASGI `app` object exposed at module level so Spaces can run it without binding port 7860 twice. - Added a regression test to prevent reintroducing a second server start. ## v32 — Mockup-aligned custom dashboard - Reworked the custom Build page to match the validated dashboard mockup. - Restored the Run Explorer as a first-class right-side panel on the main Build view. - Moved live progress into the main cockpit with compact progress metadata and a vertical timeline. - Added right-rail Run details, quick links, report preview, and “How it works” panels. - Kept the full Runs page for deeper run inspection while making recent runs visible from the home dashboard. - Preserved the V31 functional feedback, OAuth, bucket gating, and polling behavior. ## v103 - Aggregated Run Stats - Replaced the ambiguous Run Health panel with an aggregate-only Run Stats panel below the Run Explorer. - Removed latest/current/issue rows that could mix multiple jobs and warnings. - Added indexed run counters, build success rate, validation pass rate, validation latency median/p95, and aggregated issue counts. - Kept Run Explorer as the navigation surface and central tabs as the detail surfaces. ## V33 — Root custom dashboard - Serve the custom dashboard directly at `/` and `/custom`. - Move the legacy Gradio controls behind `/_gradio` for OAuth/debug only. - Remove the product-facing Gradio iframe/preview experience. - Keep OAuth-backed API routes and custom dashboard behavior intact. - Add `uvicorn` as an explicit runtime dependency. ## v31 — Functional feedback verification - Added stronger Custom UI feedback states so bucket checks, build launch, validation launch, and polling always show visible status. - Added client-side validation for required build fields and JSON validation payloads before launching Jobs. - Improved API error parsing so browser messages show clean backend errors instead of raw JSON blobs. - Added backend bucket preflight checks for `/api/build` and `/api/validate` to prevent Jobs from launching against a missing user bucket. - Added terminal-status handling so polling stops cleanly on `manual_hardware_required`, `technical_blocker`, `full_inference_candidate_health_passed`, and other final states. - Added last-polled visibility and non-blocking retry feedback for live progress. ## v30 — Custom UI robustness cleanup - Fixed the custom UI flex layout to avoid horizontal overflow on laptop-width screens. - Removed duplicate auth-panel CSS and cleaned responsive behavior. - Made the Run Explorer horizontally scrollable on medium screens and card-like on mobile. - Restyled JSON textareas as monospace code inputs. - Made the ZeroGPU-first checkbox user-controllable. - Added build gating: users must check or create their private run bucket before launching a build from the custom UI. - Removed leftover mockup-only readiness behavior and replaced it with a real bucket status pill. ## v25 — Custom UI OAuth bridge - Wired the `/custom` frontend to real OAuth-backed API endpoints. - Added custom UI actions for bucket check/create, build launch, validate launch, run progress polling, and run explorer. - Added `/api/me`, `/api/bucket/status`, `/api/bucket/create`, `/api/build`, `/api/validate`, `/api/runs/{run_id}`, and `/api/runs/{run_id}/progress`. - Added bucket run listing support for the Run Explorer. - Kept the Gradio UI as a fallback while the custom frontend is validated. ## v24 — Custom UI shell + progress API skeleton - Added a custom product UI shell served at `/custom` with dedicated HTML/CSS/JS assets. - Added a live job progress visualization model based on `events.jsonl`. - Added `/api/app-info`, `/api/progress/from-events`, and a placeholder `/api/runs` endpoint for the upcoming custom frontend auth bridge. - Added a Custom UI preview tab while preserving the functional Gradio workflows as the source of truth. - Added tests for the custom shell assets and progress mapping. ## V23 — Hardware-at-creation strategy - Requests Space hardware as early as possible via `create_repo(..., space_hardware=...)`. - Falls back through ZeroGPU → preferred hardware → fixed GPU fallback before creating a default CPU Space. - Writes `hardware_strategy.json` alongside `hardware_attempts.json`. - Stops treating OAuth hardware errors as transient retries; if automatic hardware is not authorized, the run is marked `manual_hardware_required` and the user can validate after manual hardware selection. ## V21 — Per-user bucket onboarding - Added Run storage controls to check or create the signed-in user's private run bucket from the UI. - Jobs now preflight the per-user bucket before launch and show a clear error if it is missing or inaccessible. - The app no longer requires users to leave the workflow to manually create `space-factory-runs` before first use. ## v20 — Per-user Bucket isolation - Reverted to the V17 product UI baseline. - Replaced the hardcoded shared bucket source with a per-user bucket policy. - Jobs now mount `/` dynamically. - Refresh/readback and artifacts links now use the signed-in user's bucket. - Added a Run Bucket name field to Build and Validate workflows. - Updated docs to require each user to create their own private `space-factory-runs` bucket. ## v28 — Custom UI live polish - Added Run Explorer status filters and a manual refresh action. - Persisted the active run in browser localStorage so progress resumes after refresh. - Added a non-blocking polling warning when live progress cannot be refreshed. - Highlighted the selected run row in the explorer. ## v17-product-button-links - Replaced custom HTML/Markdown quick links with native Gradio `gr.Button(link=..., link_target="_blank")` controls. - Quick link buttons are available in both public workflows: - Build from model card - Validate existing Space - Kept the product workflow unchanged: ZeroGPU-first, fixed GPU fallback, manual hardware fallback, and existing-Space validation. ## v16-product-links - Added quick links after launching jobs: - open HF Job in a new tab - open target Space in a new tab - open Space settings in a new tab - open run artifacts in the configured Bucket - Added the same link panel to both Build and Validate workflows. ## v15-product-verified - Removed legacy phase launchers from product code. - Kept only the public workflows: - Build from model card - Validate existing Space - Cleaned documentation to match the public-preview product workflow. - Removed duplicate UI component in the build events tab. - Rechecked previously observed failure modes: - Gradio Client auth parameter compatibility - endpoint discovery instead of hardcoded `/predict` - generic README metadata sanitation - ZeroGPU-first with fixed-GPU/manual fallback - separate validation/smoke-test workflow ## v14-product - Public product UI without experimental tabs. - Universal model-card builder and existing-Space validator. ## V22 — dependency compatibility and build repair - Removed the obsolete generated-Space constraint `huggingface_hub<1.0.0`. - Generated Spaces now normalize Hub requirements to a modern compatible range (`huggingface_hub>=0.34.0,<2.0.0`, or `>=1.5.0,<2.0.0` when Transformers 5.x is detected). - Added one automatic Pi repair pass when the generated Space fails build/runtime validation. - The repair pass collects Space build/runtime logs, asks Pi for a minimal patch, reuploads the workspace, and retries validation once. - Repair artifacts are stored under `runs//repair/` and logs under `logs/pi_repair_output.txt`. ## v27 — Custom run actions workflow - Adds selected-run actions in the custom Run Explorer. - Adds a manual hardware action panel in live progress. - Lets users prepare the Validate existing Space form directly from a selected run. - Keeps the Gradio fallback UI and backend workflow unchanged. ## v26-custom-run-explorer - Expanded the custom UI Run Explorer from a simple list into a usable run-detail view. - Added per-run detail rendering for status, model, target Space, hardware, validation result, artifacts, and report preview. - Enhanced `/api/runs/{run_id}` and `/api/runs/{run_id}/progress` to expose `inference_gate`, `generation_smoke`, `hardware_strategy`, `technical_blockers`, file indexes, and normalized run summaries. - Added frontend rendering for validation metrics sourced from live progress and smoke-test artifacts. - Kept the Gradio fallback UI and core backend workflows unchanged. ## v37 — Premium simplified custom UI - Simplified the Custom UI navigation to the two real product workflows: **Run build** and **Test Space**. - Kept the Run Explorer visible on the build dashboard instead of exposing it as a redundant tab. - Reworked the dashboard layout to feel closer to the validated premium mockup while reducing confusion. - Replaced ambiguous timeline dots with explicit step states: completed / active / pending / failed. - Fixed run artifact links to consistently target the run folder: `runs/` inside the connected bucket. - Reduced persistent banners: success/info messages now auto-clear and the OAuth OK state is reflected in the header instead of a permanent banner. - Preserved the Docker/FastAPI Custom UI architecture, OAuth flow, per-user buckets, build Jobs, validation Jobs, and live polling. ## V38 — Build/run separation and faster Run Explorer - Added an explicit **Prepare new build** action to separate the fresh build form from selected past runs. - Added a build mode notice explaining that selecting a run only changes progress/details and never mutates the build form. - Run Explorer filters and search now operate locally from a cached run list for a much snappier interaction. - Refresh now explicitly reloads the connected bucket; ordinary filter/search changes no longer re-query the bucket. - Selecting a run highlights it, switches the detail/progress context, and keeps the build form independent. ## V42 — Product tabs and run recovery hardening - Renamed the custom UI top-level workflow tabs to the intended product model: **New Build**, **Live Progress**, and **Space Test**. - Kept the build form isolated from selected historical runs; selecting a run only updates Live Progress and details. - Hardened run discovery from the connected bucket by combining `ls` and targeted `glob` patterns so partial/in-progress runs are recovered more reliably. - Persisted `launch.json`, `summary.json`, and a minimal `state.json` immediately after a Job launch, so running Jobs appear in the Run Explorer before the worker writes final artifacts. - Progress polling now falls back to launch metadata when `state.json`/`events.jsonl` are not ready yet. - Job links are preserved or reconstructed from launch metadata for active and historical runs. ## v43 — Canonical run view model + mockup-oriented dashboard - Added a canonical Run View Model that normalizes heterogeneous worker/bucket statuses into stable product states. - Added `/api/runs/resumable` and `/api/runs/{run_id}/view` endpoints for bucket-led recovery and UI rendering. - Enriched `/api/runs/{run_id}/progress` with the new `view` payload while preserving the previous progress contract. - Reworked the custom UI toward the product mockup: sidebar, run summary header, product pipeline stepper, agent activity feed, diagnostics panel, and Space Test preview. - Added explicit tests for running, manual hardware, success, stale, and resumable run selection states. ## v115 — remove obsolete resume-latest header action - Removed the obsolete top-bar **Resume latest run** button now that the Run Explorer and cached run selection are the canonical recovery/navigation workflow. - Removed the related frontend click handlers and hidden support element so no dead header action remains. - Updated regression tests to assert the button stays absent while preserving the backend resumable endpoint for compatibility. ## v113 — clean UI fixes, faster run reloads, and clearer validation endpoints - Fixed the build gate after a successful model pre-scan and made the target Space name optional; empty names now rely on backend auto-generation from the run ID. - Simplified the Billing & compute card to a compact payment/dashboard reminder instead of a dense pseudo-quota panel. - Reworked primary run events to use compact coded icons (`PI`, `API`, `TR`, `GPU`, etc.) instead of emoji. - Improved Gradio endpoint rendering with endpoint input/output details, clearer parameter chips, and a clearer `Set API name` action. - Added an in-memory run-detail cache so selecting a previously loaded run immediately rehydrates the correct central panel while optional network refresh continues. - Polished Runs Explorer cards with a cleaner premium layout, explicit metadata rows, isolated delete action, and stronger truncation behavior. ## v127 — Billing usefulness + UI event/timeline audit - Reworked the Billing & compute card into an actionable compute checklist: payment readiness, Pi Inference Providers, build Job cost, generated Space hardware and high-tier GPU guardrails. - Kept exact spend/quotas delegated to the Hugging Face Billing dashboard instead of inventing usage totals. - Audited JS event bindings for one-shot/delegated listeners across navigation, run rows and timeline scroll. - Reduced timeline auto-scroll churn: identical timeline renders no longer force keep-visible scrolling after user interaction. - Confirmed run selection keeps the immediate cached/optimistic render path before network hydration.