# Architecture Agentic Space Factory is a Hugging Face-native productization of the workflow described in Victor's ZeroGPU Spaces article: an agent receives a model-card build goal, follows a shared Pi instruction gist, creates a private Gradio Space, validates it, and records the full run. ```text User → Docker/FastAPI orchestrator Space with Hugging Face OAuth → per-user private Storage Bucket → ephemeral Hugging Face Job → Pi coding agent + HF Inference Providers model → generated private target Space → live build progress, traces, reports, validation results ``` ## Core components ### Orchestrator Space The public product UI is a custom Docker/FastAPI dashboard served at `/`. It uses Hugging Face OAuth and does not store a global admin token. The UI is organized around three responsibilities: - **Left column** — storage setup, new build launch, and billing/compute readiness links. - **Center workspace** — tabbed detail area with `Active Run` for build runs and `Space Test` for validation runs. - **Right column** — `Runs Explorer` for compact navigation and `Run Stats` for aggregate history metrics. The Run Explorer uses delegated row events, cached rerender signatures, relative freshness labels, and a compact two-line row layout so polling does not repeatedly attach listeners or redraw unchanged lists. ### Build workflow A build starts from a Hugging Face model ID or model-card URL. Before launching a paid Job, the orchestrator runs a metadata-only model pre-scan. The pre-scan reads Hub metadata and small repo files such as the model card, `config.json`, and `model_index.json`; it does not download weights. It looks for runnable examples, Diffusers compatibility, safetensors, unsupported tasks, gated/private access, and custom-code risk. After an acceptable verdict, the orchestrator launches a Hugging Face Job that: 1. reads the model card and model metadata; 2. creates a private target Space under the signed-in user's namespace; 3. launches Pi with a generated `GOAL.md`; 4. instructs Pi to read the configured GitHub gist before coding; 5. uploads generated Space files; 6. attempts hardware assignment on a best-effort basis; 7. runs health/API checks and, when possible, live generation smoke validation; 8. writes events, traces, report, and structured artifacts to the user's Storage Bucket. ### Pi instruction model The worker injects the operational gist into the goal sent to Pi. Pi is expected to follow the gist's workflow for Spaces iteration: inspect logs, patch generated code, upload only relevant files, wait for rebuilds, and verify the generated app through Gradio/API calls. The local worker adds stricter product requirements around that method: - generated Spaces stay private; - placeholders must not be reported as full inference; - blockers must be written explicitly when full inference is impossible; - `/health` and generation endpoints should be added where possible; - ZeroGPU duration should be informed by measured inference latency when applicable. ### Storage Bucket Runs are stored in the signed-in user's Bucket, normally: ```text https://huggingface.co/buckets//space-factory-runs ``` Typical run layout: ```text runs//launch.json runs//state.json runs//summary.json runs//events.jsonl runs//report.md runs//generated/ runs//tests/ runs//artifacts/ runs//traces/raw/ runs//traces/raw/agent_trace.jsonl runs//traces/redacted/ runs//traces/redacted/agent_trace.jsonl runs//logs/ ``` The UI exposes the full run folder through the `Artifacts` button and highlights a small curated trace dock in `Active Run`: raw Pi traces, redacted traces, report, smoke result, and blockers when relevant. ### Validation workflow Validation runs are separate from build runs. Selecting a validation run in the Run Explorer opens the `Space Test` tab, not `Active Run`. A validation job: 1. waits for the target Space to be reachable; 2. discovers Gradio API endpoints when possible; 3. calls the selected endpoint with configured args/kwargs; 4. checks the returned output type; 5. measures latency; 6. stores returned output artifacts; 7. writes a validation timeline and summary. The `Space Test` panel displays validation status, latency, Gradio endpoints, selected endpoint parameters, a point-based progress line, and the validation run actions. ### Run discovery and polling The Run Explorer reads indexed run summaries from the Bucket. It can recover in-progress or partial runs from `launch.json`, `summary.json`, and `state.json`, so a run can appear before all final artifacts exist. The frontend avoids duplicate `/api/runs` refreshes, uses delegated event listeners for dynamic lists, and keeps the active run panel responsive by rendering lightweight summaries before heavier run details finish loading. ## Result statuses - `full_inference_success` — live generation returned the expected output type. - `manual_hardware_required` — the Space exists, but hardware must be configured manually before validation. - `full_inference_candidate_health_passed` — the Space boots and appears inference-capable, but live generation was not fully validated. - `health_only` — the Space boots, but no real inference path was validated. - `technical_blocker` — concrete blockers prevent reliable full inference. - `failed` — build, runtime, or validation failed. ## Known limits - Automatic paid hardware assignment is best-effort and may fail because of OAuth, quota, billing, namespace limits, or restricted hardware availability. The app avoids A100/H200 as automatic fallbacks; if those are needed, select them manually in Space Settings only when available for the account or organization. - Numeric spend/quota totals are not mirrored in the app; users should use the Hugging Face Billing dashboard for live usage. The Billing & compute card also centralizes the Inference Providers / paid credits / restricted fallback warning so the New Build form stays focused on build configuration. - Raw traces can contain sensitive operational details. Redaction is best-effort, not a formal data-loss-prevention guarantee. - Some models require Docker, multi-GPU, custom CUDA/FlashAttention, external services, gated licenses, or manual engineering. ## Model pre-scan The custom API exposes `POST /api/models/pre-scan`, which performs a fast metadata-only scan of the selected Hugging Face model before `/api/build` is enabled in the browser. The scan uses Hub metadata and small files such as `config.json`, `model_index.json`, and `README.md`; it avoids downloading weights. This keeps the feature cheap and prevents accidental compute spend on obviously risky or unsupported model cards. ## Frontend stability pass The production-preview UI avoids listener accumulation by binding navigation, action buttons, run rows, pagination, and endpoint chips with one-time or delegated event handlers. Run list refreshes are deduped through a shared in-flight promise and skip DOM rerenders when the visible signature is unchanged. CSS is organized around the current three-column layout and keeps narrow panels safe through explicit `min-width: 0`, truncation, and responsive fallbacks for run cards, endpoint chips, activity rows, stats, and pre-scan content. ### v114 activity feed iconography The main `Latest events` feed now renders event-type icons as inline SVG, not emoji and not two-letter text badges. The icon selection is deterministic and local to the frontend: steps such as `upload_files`, `hardware`, `create_space`, `generation_smoke`, `traces`, `report_write`, `auth`, and `model_analysis` map to compact vector symbols. Status classes still control color and motion, so running, success, warning, hardware fallback, and error events stay visually distinct without changing the event payload contract. ### v113 UI and caching notes Run selection now uses a small client-side detail cache. Previously loaded run details are rendered immediately when the user switches between runs, and the app only fetches missing or stale light/heavy details. This keeps the central `Active Run` / `Space Test` panels responsive without changing the backend run bundle API. The `New Build` form no longer requires a target Space slug. The backend's `normalize_target_space()` remains the source of truth and generates a `space-factory-` slug when the user leaves the field empty. ## Navigation note The top bar intentionally does not include a resume-latest action. Run recovery and historical navigation are handled by the Runs Explorer, with cached run-detail hydration for fast repeat selection. ## Pi diagnosis and structured recovery workflow The recovery path is diagnosis-first. It runs only after live Space validation fails, and it separates **deciding what to do** from **patching code**. 1. Collect HF build logs, runtime logs, Space status, validation errors, available Gradio API information and the current workspace inventory. 2. Write `repair/INCIDENT_BRIEF.md` and `repair/PI_DIAGNOSIS_GOAL.md`. 3. Run Pi/Qwen Coder in blockage diagnosis mode. Pi must not edit files in this step. It writes `repair/REPAIR_DECISION.json`. 4. The Factory validates the decision against budgets and safety rules. Allowed actions are `wait_for_logs`, `inspect_more_logs`, `factory_rebuild_same_code`, `patch_code`, `request_manual_hardware`, and `declare_technical_blocker`. 5. Empty or no-reason HF failures cannot go directly to code patching. They are routed through wait/rebuild/blocker decisions unless Pi cites concrete actionable evidence. 6. A same-code factory rebuild re-uploads the same workspace and revalidates the live Space without changing files. This covers common HF build/runtime flakes where logs are delayed or missing. 7. A patch repair is allowed only after `patch_code` is accepted. Pi then writes `REPAIR_BRIEF.md`, `REPAIR_PLAN.md`, patches minimally, writes `REPAIR_SUMMARY.md`, and the Factory runs sanity checks before re-uploading. 8. Recovery is bounded: one wait, one same-code rebuild and one patch repair by default. A final failure produces `TECHNICAL_BLOCKERS.json` / `repair/BLOCKAGE.json` rather than an unbounded loop. The progress timeline groups these stages under the repair/recovery ring while preserving individual events such as `pi_diagnosis`, `repair_decision`, `factory_rebuild`, `repair_patch`, and `repair_validation`. Terminal failures still create an explicit red failure point. ## Unified agentic trace journal The worker keeps Pi session files and stdout logs, but the canonical product trace is the run-level `agent_trace.jsonl` pair. Each record has a `phase` field (`initial_build`, `diagnosis`, `repair_patch`), an `event`, a status, optional artifacts, and redacted data. This lets the UI and bucket artifacts present the whole autonomous loop — build, failure observation, Pi diagnosis, repair decision, optional patch, and revalidation — as one continuous agent run. The redacted version is generated with the same token redaction used elsewhere and is still best-effort, not a formal DLP guarantee. ## Agent recovery UI The v120 UI makes the core blockage protocol visible in the Active Run panel. When the worker writes `repair/REPAIR_DECISION.json`, `repair/BLOCKAGE.json`, or any recovery events, the **Agent recovery** card appears and summarizes: - the Pi diagnosis action, such as `wait_for_logs`, `factory_rebuild_same_code`, or `patch_code`; - failure category, logs quality, and confidence when available; - the recovery ladder: diagnose → decide → act → revalidate; - evidence and the next step. This is intentionally separate from the activity feed. The activity feed remains an event stream, while Agent recovery explains the current agentic recovery strategy. The Run traces dock exposes the unified raw/redacted `agent_trace.jsonl` plus `REPAIR_DECISION.json`, the repair folder, and `BLOCKAGE.json` when present. ## v122 stability note The v122 line intentionally rolls back the experimental v121 client-side caching/polling optimizations after real UI testing exposed flicker and broken build/run behavior. The app keeps the v120 Agent recovery UI and the v118/v119 core blockage protocol, but returns to the proven polling and run-selection model for production stability. ## v123 — Critical Job launch + failure timeline fix - Fixed a production-critical HF Job startup failure where the large embedded worker script could make Python fail before startup with `argument list too long`. - The worker script is now persisted as `runs//_worker.py` in the mounted run bucket before launch, and the Job receives only `WORKER_SCRIPT_PATH`. - Kept a small `WORKER_SCRIPT_B64` fallback only for compatibility/manual launch paths; normal app launches no longer put the worker source in env/argv. - Added a failure-log fallback so pre-worker Job failures such as `argument list too long` still create an explicit failed timeline event and red failure point.