CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project overview

FormScout is a Gradio app (Hugging Face Space) that scores Functional Movement Screen (FMS) videos 0–3 per test with a written rationale and an annotated overlay. It is a **screening aid** — not a diagnosis, not an injury predictor. Built for the Build Small Hackathon (Backyard AI track). Full product spec is in `docs/FormScout-FMS-Spec.md`; the engineering contract is in `docs/plans/FormScout-Build-Prompt.md`.

**Current status:** Phase 2 complete. All 7 FMS test rubric scorers, JudgeAgent, MovementClassifierAgent, ReportAgent, PoseVisualizer (overlay video), and a user-selectable pose-model registry are implemented and tested (86/87 passing). Phase 3 is next (ST-GCN fine-tune + RAG retrieval).

## Common commands

```bash
# Run the Gradio app locally
python3 app.py

# Headless pipeline test (no Gradio)
python3 -m formscout.run sample.mp4

# Run all tests
pytest tests/

# Run a single test file or test
pytest tests/test_phase2.py
pytest tests/test_biomechanics.py::TestBiomechanicsAgent::test_deep_squat_score

# Lint / format
ruff check . && ruff format .

# Start the local VLM judge server (llama.cpp, port 8080)
./scripts/serve_judge.sh

# Push source tree to the HF model repo + Space (PRs; message from last commit)
./scripts/hf_upload.sh

# Run Svelte component tests (when frontend work is added)
npx vitest run
```

## Architecture

The pipeline is a sequence of **typed specialist agents**. Each agent accepts and returns a frozen dataclass from `formscout/types.py`. The Director in `formscout/pipeline.py` orchestrates them as a deterministic state machine (not an LLM).

### Agent pipeline

```
IngestAgent → Pose2DAgent → [Body3DAgent — optional]
→ MovementClassifierAgent → BiomechanicsAgent
→ rubric/score_test() → JudgeAgent → ReportAgent
```

The **Director** (`pipeline.py`) owns the flow. `app.py` creates one `Director()` instance and calls `director.run(video_path, test_name, side, model_key)` per submission. The Gradio UI passes `test_name` directly (from dropdown), bypassing the classifier; `model_key` selects the pose backend from `config.POSE_MODELS`.

`PoseVisualizer` (`formscout/agents/visualizer.py`) renders the annotated overlay video (skeleton, trails, velocity arrows) from `IngestResult` + `Pose2DResult`. It is called from `app.py` after the pipeline run — it is a UI-layer component, not a Director stage. It returns `None` on failure, never raises.

### The tiering rule (most important invariant)

**The 2D path is the default and must stand alone as a complete, functional pipeline.** `Body3DAgent` is only activated when `config.ENABLE_3D == True` AND the checkpoint loads successfully. If 3D is off or fails, `Body3DResult(used=False, ...)` is returned — this is a normal success path, not an error. `BiomechFeatures.view` is `"2d"` or `"3d"` so the `JudgeAgent` can caveat its rationale appropriately. Never put `Body3DAgent` on the critical path.

### Feature flags in `config.py` and their current state

| Flag | Default | Meaning |
|------|---------|---------|
| `ENABLE_JUDGE` | `True` | Judge/Classifier call Qwen3-VL via llama-server; graceful rubric fallback when the server is down |
| `ENABLE_3D` | `False` | When False, Body3DAgent returns `used=False` immediately |
| `ENABLE_STGCN` | `False` | Phase 3 — ST-GCN learned scoring head |
| `ENABLE_RAG` | `False` | Phase 3 — RetrievalAgent exemplar lookup |

All model IDs, thresholds, k-values, and feature flags live in `config.py` — never scattered literals.

### Judge backend selection (local vs Space)

`config.resolve_judge_backend()` picks the VLM backend via `FORMSCOUT_JUDGE_BACKEND` (`llama_cpp` | `transformers` | `auto`). `auto` (default) uses **llama-server locally** and the **in-process transformers backend on a Space** (detected via `SPACE_ID`). `JudgeAgent` gets its client from `serving.get_vlm_client()`.

- **`llama_cpp`** — `LlamaCppClient` → llama-server at `127.0.0.1:8080` (start with `scripts/serve_judge.sh`). The local path; works perfectly.
- **`transformers`** — `TransformersVLMClient` loads Qwen3-VL-8B via transformers, GPU-wrapped with `spaces.GPU` (ZeroGPU). Lazy model load, cached per process. On any load/inference failure it returns `{"fallback": True}` and the Judge falls back to the rubric. **Needs validation on real ZeroGPU hardware** — not exercised in CPU tests.

### Fallback chain (important for local dev and Spaces)

1. `ENABLE_JUDGE=False` → JudgeAgent returns rubric score wrapped as JudgeResult (no VLM needed)
2. `ENABLE_JUDGE=True` + selected backend unavailable / transformers load fails → same rubric fallback, logs a warning
3. `ENABLE_JUDGE=True` + backend available → calls Qwen3-VL-8B-Instruct (llama-server locally, transformers/ZeroGPU on a Space)

Start the VLM server with `scripts/serve_judge.sh` (downloads live in `checkpoints/qwen3-vl/`, gitignored). To use a fine-tuned GGUF, set `FORMSCOUT_JUDGE_GGUF` (and `FORMSCOUT_JUDGE_MMPROJ` if it ships its own projector) — no code change needed. Multimodal requests go through the OpenAI-compatible `/v1/chat/completions` endpoint (the legacy `/completion` + `image_data` path does not work with modern llama-server).

This means the app is **fully functional without any GPU or llama.cpp** — rubric scoring is pure Python.

### Rubric scorers

Each FMS test has a pure-function scorer in `formscout/rubric/`:

```
score_deep_squat / score_hurdle_step / score_inline_lunge /
score_shoulder_mobility / score_active_slr /
score_trunk_stability_pushup / score_rotary_stability
```

All accept `BiomechFeatures` and return `ScoreResult`. Dispatch via `rubric.score_test(features)`. **Rubric functions must remain pure** — no model calls, no I/O.

### Bilateral tests

`hurdle_step`, `inline_lunge`, `shoulder_mobility`, `active_slr` are bilateral. `ReportAgent` groups them by test name, takes the **lower** score, and always emits the asymmetry delta even when scores are equal. `composite` is `None` when any test is unscored.

### Types contract

Every agent I/O is a frozen dataclass from `formscout/types.py`. Key types:

- `IngestResult` — decoded frames (np.ndarray list), fps, duration, dimensions
- `Pose2DResult` — per-frame keypoints as `dict[int, {x, y, conf}]` (COCO 17 joints)
- `Body3DResult` — optional 3D joints, always has `used: bool`
- `MovementResult` — `test_name` (validated enum), `side` ("left"|"right"|"na")
- `BiomechFeatures` — `angles: dict`, `alignments: dict`, `view: "2d"|"3d"`, `symmetry_delta`
- `ScoreResult` — `score: int` (0–3), `rationale`, `needs_human`
- `JudgeResult` — same as ScoreResult + `compensation_tags`, `corrective_hint`; `score=None` when `needs_human=True`
- `PipelineState` — mutable accumulator threaded through the Director

`MovementResult` and `JudgeResult` validate their fields in `__post_init__` — passing invalid values raises immediately.

### Pose model selection and checkpoints

`config.POSE_MODELS` is a registry of pose backends: MediaPipe (CPU-friendly), five YOLO26 sizes (n/s/m/l/x), and Sapiens2 variants (Phase 3, need the custom `sapiens` repo installed). `config.DEFAULT_POSE_MODEL` is YOLO26n. The Gradio UI exposes a dropdown built from `config.available_pose_models()` (filters to checkpoints actually present) and passes the chosen `model_key` through `Director.run` to `Pose2DAgent`. `config.YOLO_POSE_MODEL` is a backward-compat alias only.

Checkpoints are **not** committed (`checkpoints/` is gitignored). `formscout/startup.py:ensure_checkpoints()` downloads missing YOLO26/MediaPipe files from the `silas-therapy/formscout-checkpoints` HF repo once at app startup. Models load once per process and are cached — never inside the inference hot path.

### llama.cpp serving

`formscout/serving/llama_cpp.py` provides `LlamaCppClient` (VLM, port 8080) and `EmbeddingClient` (embeddings, port 8081). Both check `/health` before use and return safe error dicts when unavailable. Only active when the corresponding `ENABLE_*` flag is True.

### Deploying to Hugging Face

The repo deploys to both `silas-therapy/small-functional-movement-screening` (model repo) and the Space of the same name (README frontmatter is the Space config). Use `./scripts/hf_upload.sh` — never raw `hf upload .`: the `hf` CLI does **not** read `.hfignore`, so a raw upload hashes the entire `.venv` (~44k files) and pushes torch binaries. The script parses `.hfignore` into `--exclude` globs, preflights the file count, creates PRs on both repos, and auto-switches to `hf upload-large-folder` (resumable, but no PR / no commit message) above 500 files.

## Key constraints and invariants

- **No cloud model APIs.** All inference runs on-Space (ZeroGPU). No OpenAI/Anthropic/Gemini calls.
- **Pain is never auto-scored.** Any clearing test or visible distress sets `needs_human=True` — enforced in rubric functions and JudgeAgent. `JudgeResult.score` must be `None` when `needs_human=True`.
- **Quality gates (Director, never silently skip):**
  - Any agent `confidence < config.MIN_CONFIDENCE` (0.6) → warn or stop
  - `|rubric.score - judge.score| >= 1` → flag disagreement
  - `MovementResult.test_name == "unknown"` → stop pipeline, surface manual override
  - `JudgeAgent.needs_human == True` → no numeric score emitted
- **Composite is null** when any test is unscored. Never show a partial 0–21 as complete.
- **Pipeline runs headless.** No Gradio imports in any agent file.
- **Safety banner** ("Screening aid — not a diagnosis…") must always be visible in the UI — appears at top and bottom of `app.py`.

## Engineering standards

- Every agent: one public entrypoint, typed dataclass I/O from `types.py`, `confidence: float` and `notes: str` on every result.
- Models load once at module/instance init — never inside the inference hot path.
- Every agent module docstring states: purpose, inputs, outputs, failure behavior, model param count, license, and gated status.
- `tracing.py` records structured per-agent I/O for any run; one full run gets exported to the Hub.
- Every agent ships with a pytest in `tests/` that runs without model downloads and asserts the typed contract.

## Model stack (~17.6B total — stay under 32B)

| Component | Model | Params | Status |
|---|---|---|---|
| 2D pose (primary) | YOLO26-Pose n/s/m/l/x (default: n) | 0.0007–0.058B | Ready (auto-downloaded at startup) |
| 2D pose (CPU alt) | MediaPipe Pose Landmarker (full) | ~0.004B | Ready (auto-downloaded at startup) |
| 2D pose (HQ alt) | `facebook/sapiens2-pose-0.4b/0.8b/1b/5b` | 0.4–5B | Phase 3 — needs custom `sapiens` repo |
| Segmentation | SAM 3.1 base | ~0.85B | Access accepted |
| 3D biomechanics | `facebook/sam-3d-body-dinov3` | ~0.84B | **Access ACCEPTED Jun 4 2026** |
| Learned scoring | ST-GCN (pyskl) | ~0.03B | Phase 3 |
| Judge + Classifier | Qwen3-VL-8B-Instruct (llama.cpp) | 8B | **Online** — `scripts/serve_judge.sh`, ENABLE_JUDGE=True |
| Retrieval | Qwen3-VL-Embedding-8B (llama.cpp) | 8B | Phase 3 |

Track the running sum in `MODEL_BUDGET.md`. The two Qwen3-VL-8B models share a backbone.

## Gradio + Svelte UI guidance

The UI uses **Gradio `gr.Blocks`** with custom CSS/theme (`formscout/ui/theme.py`). Custom Svelte components for score dial, asymmetry bars, rubric drawer are planned for Phase 4. Use `gradio-svelte-expert` agent for Svelte component work.

- ZeroGPU: `app.py`'s `process_video` (the Start Analysis handler) is decorated with `@spaces.GPU` (via the `gpu_task` shim, no-op off-Space) so one GPU window wraps the whole pipeline — pose, optional 3D, and the judge. **ZeroGPU aborts startup with "No @spaces.GPU function detected" unless a decorated function exists at import time**, so the decorator must stay at module level on a top-level function, not buried behind a lazy import. Window length is `config.ZEROGPU_DURATION` (default 120s, `FORMSCOUT_ZEROGPU_DURATION`).
- Verify Gradio APIs against current docs before use — pin exact versions in `requirements.txt`.

## Build phases

1. **Phase 0 — Recon:** ✅ Complete. See `RECON.md`.
2. **Phase 1 — Spine:** ✅ Complete. Deep Squat end-to-end.
3. **Phase 2 — All 7 tests:** ✅ Complete. Classifier, Judge, Report agents; all rubric scorers; Gradio UI.
4. **Phase 3 — Learned scoring + retrieval:** ST-GCN fine-tune on physio clips, publish to Hub. RetrievalAgent with embedding index.
5. **Phase 4 — Polish + ship:** Custom Svelte UI components, agent trace to Hub, blog post. (Overlay video done via `PoseVisualizer`; full 7-test session + PDF export done via `formscout/session.py` + `PdfReportAgent`.)

## Known issues

- `tests/test_biomechanics.py::TestBiomechanicsAgent::test_unimplemented_test_returns_low_confidence` fails: expects `"not yet implemented"` in `result.notes` but biomechanics returns empty string. Minor — low priority.

## Badge checklist (definition of done)

- [ ] Space runs green; upload → scorecard works on real clips
- [ ] Param sum verified ≤ 32B in `MODEL_BUDGET.md`
- [ ] 🔌 **Off the Grid** — no cloud model APIs anywhere in the pipeline
- [ ] 🎯 **Well-Tuned** — fine-tuned ST-GCN head published to Hub with honest model card
- [ ] 🎨 **Off-Brand** — custom, non-default Gradio UI (scout/trail theme)
- [ ] 🦙 **Llama Champion** — VLM + embedder served via llama.cpp (GGUF)
- [ ] 📡 **Sharing is Caring** — one full agent trace (all I/O) published to Hub
- [ ] 📓 **Field Notes** — blog post written, honesty section (FMS limitations) front-and-center
- [ ] Demo video + social post recorded
- [ ] Safety banner present; pain/clearing never auto-scored; low-confidence flagged