README.md

---
title: HARvestGym
emoji: 🕸️
colorFrom: blue
colorTo: purple
sdk: docker
pinned: false
tags:
  - openenv
  - reinforcement-learning
  - api-agent
  - web-tasks
base_path: /web
---

# HARvestGym

### Can a small model learn to reverse-engineer any web application's API — and complete real tasks through those APIs, without ever opening a browser?

Web applications are full of APIs. Every click in a browser triggers an HTTP call with a precise schema, a specific authentication header, an exact sequence of prerequisites. **HARvestGym trains a small model to do all of that directly** — given a task and a URL, it discovers the relevant endpoints, figures out what each one needs, chains the calls in the right order, and completes the task without any browser.

The model starts with nothing: no schema, no documentation, no endpoint list. It uses tools to explore — issuing requests, inspecting responses, building up its own understanding of how the application works. This is what a developer does when they reverse-engineer an API. The model learns to do the same.

---

## How It Works

```
Task + App URL
      │
      ▼
Policy Model (RL Agent)
  small model — no prior knowledge of the app

  Step 1     ──► browser_agent(task, url)     → filtered API endpoint list
  Step 2+    ──► search_endpoints(query)      → full schema for a specific endpoint
             ──► curl_exec(command)           → execute HTTP call, get response
             ──► search_episode_data(query)   → search prior response bodies
             ──► done(result)                 → declare task complete
      │
      ▼
Live WebArena Apps (EC2)  ←── real HTTP responses (always live, never mocked)
      │
      ▼
Deterministic Judge (compares against ground truth API catalog)
      │
      ▼
Reward Signal  ──►  GRPO  ──►  updated policy
```

The agent calls `browser_agent` once at the start — this runs a real browser to complete the same task while recording all network traffic, then returns the filtered list of API endpoints observed. The agent now has a map of what endpoints exist. What it does *not* know:

- which of those endpoints are actually needed for this specific task
- in what order they must be called (you cannot add to a cart before the cart exists)
- where each required parameter value comes from
- how to re-authenticate if a session expires mid-episode

The model must learn to discover all of this on its own.

---

## Architecture

```
                          TRAINING LOOP
┌─────────────────────────────────────────────────────────────────────────┐
│                                                                         │
│  Task + App URL                                                         │
│       │                                                                 │
│       ▼                                                                 │
│  ┌────────────────────────────────────────────────────────────────┐     │
│  │                  Policy Model (RL Agent)                       │     │
│  │         small model — no prior knowledge of the app            │     │
│  │                                                                │     │
│  │  Observation: task + history + session_state + last_result     │     │
│  │                                                                │     │
│  │  Step 1   ──► browser_agent(task, url)                         │     │
│  │  Step 2+  ──► search_endpoints(query)                          │     │
│  │           ──► curl_exec(command)                               │     │
│  │           ──► search_episode_data(query)                       │     │
│  │           ──► done(result)                                     │     │
│  └────────┬───────────────────────────────────────────────────────┘     │
│           │                                                             │
│    ┌──────┴──────────────────────────────┐                              │
│    │                                     │                              │
│    ▼                                     ▼                              │
│  ┌─────────────────────┐    ┌─────────────────────────────────────┐     │
│  │   Browser Agent     │    │         Environment                 │     │
│  │  (step 1 only)      │    │                                     │     │
│  │                     │    │  • Executes curl_exec via subprocess│     │
│  │ Training:           │    │  • Auto-injects session cookies     │     │
│  │  Load pre-recorded  │    │  • Smart-truncates response bodies  │     │
│  │  cached HAR from    │    │  • Indexes full responses into      │     │
│  │   disk or launch    │    │    per-episode BM25 + GEMMA store   │     │
│  │   on real browser   │    │  • Manages session_state: cookies,  │     │
│  │                     │    │    CSRF tokens, auth headers        │     │
│  │ Inference:          │    └──────────────┬──────────────────────┘     │
│  │  Launch real browser│                   │                            │
│  │  via Playwright +   │                   │ HTTP calls (always live)   │
│  │  bu-30b-a3b-preview │                   ▼                            │
│  │                     │    ┌─────────────────────────────────────┐     │
│  │ Both paths produce: │    │     WebArena EC2 (live apps)        │     │
│  │  • Filtered HAR     │    │                                     │     │
│  │  • OpenAPI-like spec│    │  :7770  Shopping (Magento 2)        │     │
│  │  • GEMMA embeddings │    │  :7780  Shopping Admin              │     │
│  │    for search_      │    │  :9999  Forum (Postmill)            │     │
│  │    endpoints()      │    │  :8888  Wikipedia (Kiwix)           │     │
│  └─────────────────────┘    │  :3000  Map (OpenStreetMap)         │     │
│                             └──────────────┬──────────────────────┘     │
│                                            │                            │
│                                            │ episode trajectory         │
│                                            ▼                            │
│                             ┌─────────────────────────────────────┐     │
│                             │    Deterministic Judge              │     │
│                             │                                     │     │
│                             │  Per-template programmatic grader:  │     │
│                             │  • Inspects episode trajectory      │     │
│                             │  • Optionally probes live app state │     │
│                             │  • Verifies parameter sourcing      │     │
│                             │    (TASK_SPEC / PREV_CALL /         │     │
│                             │     AUTH_FLOW / STATIC / DERIVED)   │     │
│                             │  • Scores [0.0 → 1.0]               │     │
│                             └──────────────┬──────────────────────┘     │
│                                            │                            │
│                                            ▼                            │
│                             ┌─────────────────────────────────────┐     │
│                             │         Reward Signal               │     │
│                             │                                     │     │
│                             │  Per-step:                          │     │
│                             │   +0.2  valid API call (2xx)        │     │
│                             │   +0.1  new path explored           │     │
│                             │   +0.25 correct param sourcing      │     │
│                             │   −0.15 repeated identical call     │     │
│                             │   −0.3  browser_agent called again  │     │
│                             │                                     │     │
│                             │  Episode end:                       │     │
│                             │   +2.0–+5.0 task complete (easy→hard│     │
│                             │   −1.5      task failed             │     │
│                             └──────────────┬──────────────────────┘     │
│                                            │                            │
│                                            ▼                            │
│                             ┌─────────────────────────────────────┐     │
│                             │    GRPO (via HF TRL)                │     │
│                             │                                     │     │
│                             │  8 parallel rollouts per prompt     │     │
│                             │  Computes advantages without        │     │
│                             │  a value function                   │     │
│                             │  Updates policy weights             │     │
│                             └─────────────────────────────────────┘     │
│                                            │                            │
│                                            └──► updated Policy Model    │
└─────────────────────────────────────────────────────────────────────────┘
```

---

## Target Applications

All running on a single AWS EC2 instance — real production software, no simulation.

| App            | Port | Software                                          |
| -------------- | ---- | ------------------------------------------------- |
| Shopping       | 7770 | Magento 2 — open-source e-commerce platform       |
| Shopping Admin | 7780 | Magento 2 Admin — backend panel for the same store|
| Forum          | 9999 | Postmill — open-source Reddit-like forum          |
| Wikipedia      | 8888 | Kiwix — read-only offline mirror of Wikipedia     |
| Map            | 3000 | OpenStreetMap — collaborative mapping platform    |

Source: [WebArena environment_docker](https://github.com/web-arena-x/webarena/tree/main/environment_docker)

---

## Tasks

HARvestGym trains on **7 task templates** across three complexity tiers. Each template is a parameterized scenario: one reward function, one ground truth catalog entry, one grader — but potentially hundreds of distinct episode variations produced by substituting different values for the template slots (`{product_name}`, `{category_name}`, etc.).

### Complexity Tiers

| Tier   | Characteristic                                | API calls required |
| ------ | --------------------------------------------- | ------------------ |
| Easy   | Single call, no auth                          | 1                  |
| Medium | Auth + 1–2 dependent calls                    | 2–3                |
| Hard   | Multi-step chain with ID threading, full auth | 4–8+               |

The model only graduates to harder templates once it reliably solves easier ones.

### Task Templates

| #   | Tier   | App            | Template                                               | Key Challenge                                           |
| --- | ------ | -------------- | ------------------------------------------------------ | ------------------------------------------------------- |
| 1   | Easy   | Shopping       | List products in category `{category_name}`            | Single GET with query params                            |
| 2   | Easy   | Wikipedia      | Retrieve article summary for `{title}`                 | Single GET, path parameter resolution                   |
| 3   | Medium | Shopping       | Add `{product_name}` to a guest cart                   | 2 calls: create cart → add item; ID threading           |
| 4   | Medium | Forum          | Retrieve all posts in `{forum_category}` (authed)      | Login → extract session → GET                           |
| 5   | Hard   | Forum          | Create a post titled `{title}` in `{category}`         | Login → extract CSRF `form_key` → POST with full schema |
| 6   | Hard   | Shopping       | Guest checkout for `{product_name}`                    | 5+ chained calls; cart → item → shipping → payment      |
| 7   | Hard   | Shopping Admin | Create a new product with SKU `{sku}`, price `{price}` | Admin bearer token → full Magento product schema        |

**Template parameters** are populated from a static parameter pool built by querying the live applications before training (see `parameter_pools.json`, refreshed via `scripts/build_parameter_pools.py`). Each episode samples randomly from its pool — the model never sees the pool directly, it must discover the correct values through its own API calls.

Each task has a deterministic programmatic grader (score in `[0.0, 1.0]`):
- **Easy graders**: check HTTP response body for expected values
- **Medium graders**: probe application state after episode (e.g., fetch the cart, verify item is present)
- **Hard graders**: verify multi-step state change in the application (e.g., post exists, checkout created)

---

## Spaces

### Observation Space

What the model sees at each step:

```python
class Observation(BaseModel):
    task: str                  # Natural language task
    app_base_url: str          # Root URL of the target application
    last_tool_result: Any      # Result of last tool call
    history: list[dict]        # Full episode trajectory: [{action, tool_result}, ...]
    session_state: dict        # Auto-managed: cookies, tokens, CSRF values
    step_count: int
    max_steps: int             # 20
```

`session_state` is maintained by the environment — the model decides *when* to authenticate and *which* session values to use; the environment handles *extraction* from `Set-Cookie` headers and response bodies.

**Response truncation** rules applied in order:
1. Non-JSON body (HTML, CSS): truncated to 3,000 characters
2. JSON primitive (string, number): never truncated — these are tokens, IDs
3. Error response (4xx/5xx): never truncated — the model needs every word to self-correct
4. Small JSON (no large arrays): returned as-is
5. Large JSON array (≥ 3 items): first 2 items shown + `_list_truncated` annotation + hint to call `search_episode_data()`

Every `curl_exec` call indexes the *full* response into a per-episode hybrid index (BM25 + GEMMA embeddings) *before* truncation — so all items are always retrievable even when only 2 were shown.

### Action Space

The model outputs a single tool call per step.

| Tool                         | Input                             | Output                                                                          |
| ---------------------------- | --------------------------------- | ------------------------------------------------------------------------------- |
| `browser_agent(task, url)`   | Task string + app base URL        | Summary list of API endpoint names + methods (e.g. `GET /products`)             |
| `search_endpoints(query)`    | Natural language query            | Top-3 endpoint schemas (method, path, auth, params with sources, response fields)|
| `curl_exec(command)`         | Full curl command string          | `{status_code, headers, body}` — body smart-truncated; full body indexed        |
| `search_episode_data(query)` | Keyword or natural language query | Top-5 JSON objects from this episode's request/response history                 |
| `done(result?)`              | Optional result string            | Ends episode, triggers judge evaluation                                         |

`browser_agent` is called **exactly once per episode at step 1**. Calling it again applies a −0.3 penalty. During training, it loads a cached HAR file; at inference, it launches a live browser session.

Full technical specifications for all tools: [`TOOLS.md`](./TOOLS.md)

### Reward Space

**Per-step:**

| Signal                       | Value  | Trigger                                                              |
| ---------------------------- | ------ | -------------------------------------------------------------------- |
| Valid API call (2xx)         | +0.2   | `curl_exec` returns 2xx status                                       |
| New path called this episode | +0.1   | Normalized path not called before — discourages looping              |
| Correct parameter sourcing   | +0.25  | Judge: value came from the correct source type                       |
| Session value correctly used | +0.1   | Auth token/cookie present and correct in curl call                   |
| Repeated identical call      | −0.15  | Exact duplicate curl command issued twice                            |
| browser_agent called again   | −0.3   | `browser_agent` called after step 1                                  |
| Malformed curl command       | −0.1   | curl cannot be parsed or executed                                    |
| 4xx response (recoverable)   | −0.05  | Call failed but episode continues                                    |

**Episode end:**

| Outcome                                                     | Reward                                     |
| ----------------------------------------------------------- | ------------------------------------------ |
| Task completed correctly                                    | +2.0 to +5.0 (scales with difficulty tier) |
| Partial completion (right endpoints, wrong param threading) | +0.5 to +1.5                               |
| Authentication correctly obtained (even if task fails)      | +0.3                                       |
| Timeout / task failed entirely                              | −1.5                                       |

Target signal separation: successful episodes `+3` to `+7`, failed episodes `−2` to `−1`. Required for GRPO.

> **Reward design note:** Pure step-level rewards can teach a model to "look busy" — accumulating exploration rewards while never completing the task. The terminal outcome reward is designed to dominate the sum of all per-step rewards. The curriculum is the primary defense: Easy tasks have a trivially short optimal path (2 steps), so there's no room to accumulate fake exploration reward before the model learns that the terminal reward is what matters.

---

## Key Design Decisions

### Browser Agent as a Discovery Tool

The RL agent has access to a **browser agent tool** powered by [`bu-30b-a3b-preview`](https://huggingface.co/browser-use/bu-30b-a3b-preview) — a 30B MoE vision-language model (3B active parameters) served via the [browser-use](https://github.com/browser-use/browser-use) library on Playwright. When called, it completes the task in a real browser while intercepting all network traffic, then returns the filtered API call list.

**Training vs. inference:** The browser agent output is pre-computed and cached per task during training — the RL model receives it instantly, no live browser session runs. At inference, the browser agent runs live to handle novel tasks.

Full details: [`BROWSER_AGENT.md`](BROWSER_AGENT.md)

### Ground Truth from the Codebase, Not the Browser

The browser agent shows *what* API calls happen. It does not explain *why* — where each parameter comes from or what field constraints exist. That comes from a one-time static analysis of each WebArena application's Docker image source, producing a **ground truth API catalog**:

```
endpoint:    POST /rest/V1/guest-carts/{cartId}/items
path_params:
  cartId:    obtained from: POST /rest/V1/guest-carts → response body
body:
  cartItem.sku:       the product's SKU, from: GET /rest/V1/products → items[].sku
  cartItem.qty:       quantity, from: task specification
  cartItem.quote_id:  same as cartId
```

The judge uses this to verify not just *what* the model called, but *where each parameter value came from*. Source types: `TASK_SPEC`, `PREV_CALL`, `AUTH_FLOW`, `STATIC`, `DERIVED`. This is how partial credit works — the model gets reward for correctly threading a `cart_id` even if the final call had a wrong field elsewhere.

Full extraction process: [`GROUND_TRUTH_EXTRACTION.md`](GROUND_TRUTH_EXTRACTION.md)

### HTML and Form-Based Applications

Not every endpoint returns JSON. The Forum (Postmill) relies on HTML form submissions with CSRF tokens; Wikipedia (Kiwix) serves static HTML pages. The agent handles both:

- **CSRF tokens**: The model GETs the form page, reads the `value` attribute of `input[name="_csrf_token"]` from the returned HTML, and places it in the subsequent POST. If the token is beyond the 3,000-character truncation point, it calls `search_episode_data("_csrf_token")` — the full HTML is indexed before truncation.
- **HTML-only responses**: Wikipedia responses are returned in the standard `{status_code, headers, body}` envelope. Search results appear in `<a href>` tags; article content in `<p>` tags.

---

## Example Episode

```
TASK: Add "Radiant Tee" to a guest cart at http://ec2-16-59-2-56.../

STEP 1: browser_agent("Add Radiant Tee to a guest cart", "http://ec2-...:7770/")
→ {"endpoints": ["POST /rest/V1/guest-carts", "GET /rest/V1/products",
                  "POST /rest/V1/guest-carts/{id}/items", ...]}

STEP 2: search_endpoints("create guest cart")
→ ["endpoint: POST /rest/V1/guest-carts | auth: none | returns: string (cartId)"]

STEP 3: curl_exec("curl -X POST 'http://ec2-.../rest/V1/guest-carts' -H 'Content-Type: application/json'")
→ {status_code: 200, body: "cart-abc123"}

STEP 4: search_endpoints("find product by name get sku")
→ ["endpoint: GET /rest/V1/products | query: searchCriteria filters | returns: .items[].sku"]

STEP 5: curl_exec("curl 'http://ec2-.../rest/V1/products?searchCriteria[filter_groups][0][filters][0][field]=name&...[value]=Radiant+Tee'")
→ {status_code: 200, body: {"items":[{"sku":"MH01","name":"Radiant Tee","price":22.0}]}}

STEP 6: search_endpoints("add item to guest cart cartId")
→ ["endpoint: POST /rest/V1/guest-carts/{cartId}/items | path: cartId from POST /rest/V1/guest-carts | body: cartItem.sku, cartItem.qty, cartItem.quote_id"]

STEP 7: curl_exec("curl -X POST 'http://ec2-.../rest/V1/guest-carts/cart-abc123/items' -H 'Content-Type: application/json' -d '{\"cartItem\":{\"sku\":\"MH01\",\"qty\":1,\"quote_id\":\"cart-abc123\"}}'")
→ {status_code: 200, body: {"item_id": 5, "sku": "MH01", "qty": 1}}

STEP 8: done("Radiant Tee added to cart")
```

Values from prior responses (`cart-abc123`, `MH01`) are threaded directly from the growing episode history. If step 5 had returned 200 products truncated to 2, the model would call `search_episode_data("Radiant Tee sku")` to retrieve `MH01` from the episode index.

---

## Setup

### Prerequisites

- Docker installed and running
- Python 3.11+ with [`uv`](https://github.com/astral-sh/uv)
- A Hugging Face token with read access

### Local Development

```bash
# Clone and enter the project
git clone <your-hf-space-url>
cd HARvestGym

# Install dependencies
uv sync

# Validate the OpenEnv spec
openenv validate

# Build and run the Docker image
docker build -t harvgym .
docker run -p 8000:8000 harvgym

# Run the inference script
HF_TOKEN=hf_xxx uv run inference.py
```

### Environment Variables

| Variable       | Default                              | Required | Purpose                                   |
| -------------- | ------------------------------------ | -------- | ----------------------------------------- |
| `HF_TOKEN`     | —                                    | **Yes**  | HuggingFace auth token                    |
| `API_BASE_URL` | `https://router.huggingface.co/v1`   | No       | LLM API endpoint                          |
| `MODEL_NAME`   | `google/gemma-4-31B-it`              | No       | Model for inference                       |
| `HARVGYM_TASK` | `har_classify_easy`                  | No       | Override which task to run                |

### API Endpoints

```bash
# Reset episode
curl -X POST http://localhost:8000/reset

# Execute a step
curl -X POST http://localhost:8000/step \
  -H "Content-Type: application/json" \
  -d '{"tool": "browser_agent", "args": {"task": "...", "url": "..."}}'

# Get current state
curl http://localhost:8000/state
```

---

## Baseline Performance

Scores generated by running `uv run inference.py` with `google/gemma-4-31B-it` via the HuggingFace Router.

| Task | Difficulty | Score | Steps | Result | Notes |
| ---- | ---------- | ----- | ----- | ------ | ----- |
| `easy_list_pants` | Easy | **0.74** | 6 | PASS | List products in 'Pants' category |
| `medium_cart_camera_backpack` | Medium | **0.56** | 20 | PASS | Add Camera Backpack to guest cart |
| `medium_cart_flannel_jacket` | Medium | **0.60** | 20 | PASS | Add Flannel Jacket to guest cart |
| `hard_checkout_ripstop_pants` | Hard | **0.22** | 20 | FAIL | Full guest checkout (hit step limit) |
| **Overall** | — | **0.53** | — | **3/4 passed** | |

> **To regenerate:** `HF_TOKEN=hf_xxx uv run inference.py`