BROWSER_AGENT.md

Browser Agent Component

This document describes the browser agent tool used by the HARvestGym RL agent — how it works, how to build it, and how it integrates with the environment.

---

What It Is

The browser agent is a multi-stage tool the RL agent calls at the start of every episode. Given a natural language task and a URL, it:

1. Checks if a pre-recorded HAR file exists for this application
2. If HAR exists → loads it directly (no browser launched)
3. If no HAR → launches a real browser (Chromium via Playwright), connects an LLM, performs the task, and records all network traffic as a HAR file
4. Processes the HAR (from either source) to extract an OpenAPI-like spec
5. Builds GEMMA embeddings over the extracted spec so search_endpoints() can do semantic search
6. Returns a summary — the list of API endpoint names and HTTP methods only

The browser agent is a script that orchestrates multiple processing stages. The RL agent sees only the final summary output — a list of endpoints like GET /products, POST /guest-carts. No headers, no body schemas, no parameter details. To get full details about any endpoint, the agent calls search_endpoints() with a natural language query — this searches the GEMMA embeddings built during the browser agent's processing stage.

---

Library: browser-use

Repository: browser-use/browser-use
Stars: 86k+ (April 2026)
License: MIT
Language: Python 3.11+

browser-use connects any LLM to a Playwright-controlled browser. The LLM receives the page state (DOM, screenshot, or both), decides on an action (click, type, navigate, extract), and browser-use executes it. It uses a sense-plan-act loop with built-in error handling.

Install:

pip install browser-use
playwright install chromium

---

How It Works: Full Pipeline

Stage 1 — Obtain HAR Data

The browser agent first checks whether a pre-recorded HAR file exists for the target application. If it does, the browser is never launched — this saves 30–120 seconds per episode.

import json, os
from urllib.parse import urlparse

HAR_MAP = {
    ":7770": "hars/shopping.har",
    ":7780": "hars/shopping_admin.har",
    ":9999": "hars/forum.har",
    ":3000": "hars/osm.har",
    ":8888": "hars/wikipedia.har",
}

def resolve_har_path(url: str) -> str | None:
    """Check if a pre-recorded HAR exists for this app."""
    for port_key, path in HAR_MAP.items():
        if port_key in url:
            if os.path.exists(path):
                return path
    return None


async def get_har_data(task: str, url: str, llm_model: str) -> dict:
    """
    Stage 1: Get HAR data — from file if available, from live browser otherwise.
    Returns the parsed HAR JSON.
    """
    har_path = resolve_har_path(url)

    if har_path:
        # HAR exists — load directly, no browser needed
        with open(har_path) as f:
            return json.load(f)
    else:
        # No HAR — run live browser session and capture traffic
        raw_log = await run_browser_agent_live(task, url, llm_model)
        return convert_raw_log_to_har(raw_log)

Stage 2 — Live Browser Session (only if no HAR exists)

When no pre-recorded HAR is available, the browser agent launches a real Chromium browser, connects the LLM, and performs the task while intercepting all network traffic:

from playwright.async_api import async_playwright
from browser_use import Agent
from langchain_openai import ChatOpenAI

async def run_browser_agent_live(task: str, url: str, llm_model: str) -> list[dict]:
    """
    Runs browser-use agent on the given task, intercepts all network traffic,
    returns raw request/response log.
    """
    requests_log = []

    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)
        context = await browser.new_context()
        page = await context.new_page()

        # Attach network interceptor
        async def on_request(request):
            requests_log.append({
                "type": "request",
                "url": request.url,
                "method": request.method,
                "headers": dict(request.headers),
                "post_data": request.post_data,
            })

        async def on_response(response):
            try:
                body = await response.text()
            except Exception:
                body = None
            requests_log.append({
                "type": "response",
                "url": response.url,
                "status": response.status,
                "headers": dict(response.headers),
                "body": body,
            })

        page.on("request", on_request)
        page.on("response", on_response)

        # Navigate to app first
        await page.goto(url)

        # Run browser agent
        llm = ChatOpenAI(model=llm_model, base_url="https://router.huggingface.co/v1")
        agent = Agent(task=task, llm=llm, page=page)
        await agent.run()

        await browser.close()

    return requests_log

Stage 3 — Filter and Extract OpenAPI-like Spec

The HAR data (from either source) contains everything: fonts, analytics, CDN requests, JS bundles, CSS. The browser agent filters this down and extracts a structured OpenAPI-like specification:

import re
from urllib.parse import urlparse

SKIP_EXTENSIONS = {".css", ".png", ".jpg", ".svg", ".ico", ".woff", ".woff2", ".ttf", ".gif"}
SKIP_DOMAINS = {"google-analytics.com", "doubleclick.net", "cloudflare.com", "cdn.", "fonts.googleapis.com"}
SKIP_PATH_PREFIXES = ["/static/", "/media/", "/_next/", "/assets/", "/__webpack"]

def is_application_api_call(url: str, app_base_url: str) -> bool:
    parsed = urlparse(url)
    app_host = urlparse(app_base_url).netloc

    if parsed.netloc != app_host:
        return False

    path = parsed.path.lower()
    for ext in SKIP_EXTENSIONS:
        if path.endswith(ext):
            return False
    for prefix in SKIP_PATH_PREFIXES:
        if path.startswith(prefix):
            return False

    return True


def extract_openapi_spec(har_data: dict, app_base_url: str) -> list[dict]:
    """
    Stage 3: Process HAR entries into an OpenAPI-like spec.
    Each entry becomes a structured endpoint document with method, path,
    query params, request body schema, response body schema, status codes, auth info.
    """
    entries = har_data["log"]["entries"]
    seen = set()
    spec_entries = []

    for entry in entries:
        req = entry["request"]
        resp = entry["response"]
        raw_url = req["url"]
        method = req["method"]

        # Filter non-API traffic
        if not is_application_api_call(raw_url, app_base_url):
            continue

        # Skip HTML page navigations
        content_type = _get_response_content_type(resp)
        if "text/html" in content_type and method == "GET":
            continue

        # Normalise path: replace IDs with {id}
        parsed = urlparse(raw_url)
        path = _normalise_path(parsed.path)

        # Deduplicate by (method, normalised_path)
        key = f"{method} {path}"
        if key in seen:
            continue
        seen.add(key)

        # Extract auth info
        has_auth = any(
            h["name"].lower() in ("authorization", "x-api-key", "cookie")
            for h in req["headers"]
        )

        # Build endpoint spec document
        spec_entry = {
            "method": method,
            "path": path,
            "query_params": parsed.query or None,
            "request_headers": {h["name"]: h["value"] for h in req["headers"]
                                if h["name"].lower() in ("content-type", "authorization", "x-requested-with")},
            "request_body": _extract_body(req),
            "status_code": resp["status"],
            "response_content_type": content_type,
            "response_body_sample": _truncate_body(resp),
            "auth_observed": has_auth,
        }
        spec_entries.append(spec_entry)

    return spec_entries

Stage 4 — Build GEMMA Embeddings for Search

The extracted spec entries are converted to text documents and embedded using GEMMA embeddings. These embeddings power the search_endpoints() tool — when the RL agent queries "how to add item to cart", the semantic search finds the matching endpoint spec.

from sentence_transformers import SentenceTransformer
import numpy as np

def build_endpoint_embeddings(spec_entries: list[dict], app_name: str) -> tuple[np.ndarray, list[str]]:
    """
    Stage 4: Convert spec entries to text chunks and build GEMMA embeddings.
    These embeddings are stored in memory for the duration of the episode,
    enabling search_endpoints() to do semantic search.
    """
    model = SentenceTransformer("google/embeddinggemma-300m")

    chunks = []
    for entry in spec_entries:
        chunk = spec_entry_to_text(entry, app_name)
        chunks.append(chunk)

    # GEMMA encode_document: "title: {endpoint} | text: {rest of chunk}"
    embeddings = model.encode_document(chunks, batch_size=32)
    # Use similarity metric from google/embeddinggemma-300m model card

    return embeddings, chunks


def spec_entry_to_text(entry: dict, app_name: str) -> str:
    """Convert a single spec entry to a searchable text document."""
    parts = [
        f"app: {app_name}",
        f"endpoint: {entry['method']} {entry['path']}",
        f"status: {entry['status_code']}",
        f"auth: {'required' if entry['auth_observed'] else 'none'}",
    ]
    if entry.get("query_params"):
        parts.append(f"query: {entry['query_params']}")
    if entry.get("request_body"):
        parts.append(f"body: {entry['request_body']}")
    if entry.get("response_body_sample"):
        parts.append(f"response_sample: {entry['response_body_sample']}")
    return " | ".join(parts)

Stage 5 — Return Summary to RL Agent

The browser agent returns only a summary — endpoint names and HTTP methods. No headers, no body schemas, no parameter details. The agent must call search_endpoints() to get the full details.

def build_browser_agent_output(spec_entries: list[dict], app_name: str) -> dict:
    """
    Stage 5: Build the summary output returned to the RL agent.
    This is intentionally sparse — just endpoint names and methods.
    """
    summary_endpoints = []
    for entry in spec_entries:
        summary_endpoints.append({
            "method": entry["method"],
            "path": entry["path"],
        })

    return {
        "app": app_name,
        "endpoints": summary_endpoints,
        "total_endpoints": len(summary_endpoints),
        "note": (
            "These endpoints were observed for this application. "
            "Use search_endpoints() with a natural language query to get "
            "the full schema, parameters, and auth details for any endpoint."
        )
    }

Full Orchestration

async def browser_agent(task: str, url: str) -> dict:
    """
    Complete browser agent pipeline:
    1. Get HAR data (from file or live browser)
    2. Filter and extract OpenAPI-like spec
    3. Build GEMMA embeddings for search_endpoints()
    4. Return summary endpoint list to RL agent
    """
    app_name = resolve_app_name(url)
    llm_model = "browser-use/bu-30b-a3b-preview"

    # Stage 1-2: Get HAR data
    har_data = await get_har_data(task, url, llm_model)

    # Stage 3: Extract OpenAPI-like spec
    spec_entries = extract_openapi_spec(har_data, url)

    # Stage 4: Build GEMMA embeddings (stored in environment for search_endpoints)
    embeddings, chunks = build_endpoint_embeddings(spec_entries, app_name)
    store_episode_embeddings(app_name, embeddings, chunks)  # makes search_endpoints() work

    # Stage 5: Return summary to RL agent
    return build_browser_agent_output(spec_entries, app_name)

---

Output Example

What the RL agent sees (summary only — no schemas, no headers, no body details):

{
  "app": "shopping",
  "endpoints": [
    {"method": "POST", "path": "/rest/V1/integration/customer/token"},
    {"method": "GET",  "path": "/rest/V1/products"},
    {"method": "GET",  "path": "/rest/V1/products/{id}"},
    {"method": "POST", "path": "/rest/V1/guest-carts"},
    {"method": "POST", "path": "/rest/V1/guest-carts/{id}/items"},
    {"method": "GET",  "path": "/rest/V1/guest-carts/{id}/totals"},
    {"method": "POST", "path": "/rest/V1/guest-carts/{id}/order"},
    {"method": "GET",  "path": "/rest/V1/categories"}
  ],
  "total_endpoints": 8,
  "note": "These endpoints were observed for this application. Use search_endpoints() with a natural language query to get the full schema, parameters, and auth details for any endpoint."
}

To get full details, the agent calls:

search_endpoints("add item to guest cart")
→ returns full schema: POST /rest/V1/guest-carts/{cartId}/items, body params, auth, response fields

---

How search_endpoints() Uses the Embeddings

The GEMMA embeddings built in Stage 4 are what power search_endpoints(). When the RL agent calls search_endpoints("create guest cart"):

1. The query is encoded using GEMMA encode_query
2. Dot product similarity against all endpoint embeddings
3. Top-3 matching endpoint spec documents are returned with full details

def search_endpoints(query: str, embeddings, texts, model, top_k=3) -> list[str]:
    q_emb = model.encode_query(query)          # shape: (D,)
    # Use similarity metric specified by google/embeddinggemma-300m model card
    scores = model.similarity(q_emb, embeddings).squeeze(0)  # shape: (N,)
    top_idx = np.argsort(scores)[::-1][:top_k]
    return [texts[i] for i in top_idx]

The endpoint documents returned by search contain the full extracted spec — method, path, query params, request body structure, response samples, auth requirements. This is the detailed view that complements the summary list from browser_agent.

---

LLM Choice for Browser Agent

We use browser-use/bu-30b-a3b-preview — a model purpose-built and fine-tuned specifically for browser-use tasks.

Property	Value
Base model	Qwen3-VL-30B-A3B-Instruct
Architecture	Vision-Language MoE (Mixture of Experts)
Total parameters	30B
Active parameters	3B (MoE — only 3B fire per forward pass)
Context length	65,536 tokens
Specialization	Superior DOM understanding + visual reasoning for web tasks

This model is designed to be served with vLLM and integrates directly with the browser-use library via its ChatOpenAI-compatible interface:

from browser_use import Agent, ChatOpenAI

llm = ChatOpenAI(
    base_url="http://localhost:8000/v1",    # vLLM server
    model="browser-use/bu-30b-a3b-preview",
    temperature=0.6,
    top_p=0.95,
    dont_force_structured_output=True,      # speeds up inference
)

agent = Agent(task=task, llm=llm)
agent.run_sync()

Serve with vLLM:

vllm serve browser-use/bu-30b-a3b-preview \
  --max-model-len 65536 \
  --host 0.0.0.0 \
  --port 8000

Because 3B parameters are active per forward pass (MoE), this model is fast enough for deployment without requiring a full large-model GPU allocation.

---

Training vs. Inference: What Changes

                  Training                          Inference
                     │                                  │
    browser_agent    │  HAR file exists → loads from    │  HAR file may not exist
    Stage 1          │  disk, no browser launched       │  → launches live browser session
                     │                                  │  → records traffic as HAR
                     │                                  │
    browser_agent    │  Processes HAR → extracts spec   │  Same processing pipeline
    Stages 2-4       │  → builds GEMMA embeddings       │  on the live-captured traffic
                     │  → returns summary               │
                     │                                  │
    curl_exec        │  hits REAL live server           │  hits REAL live server
    calls            │  (WebArena EC2)                  │  (WebArena EC2)
                     │                                  │
    judge            │  probes REAL live server         │  probes REAL live server
    verification     │  to verify task completion       │  to verify task completion

What changes between training and inference: only Stage 1 — where the HAR data comes from. During training, pre-recorded HAR files exist for all tasks, so the browser is never launched. At inference, the HAR may not exist for novel tasks, so the browser runs live.

What never changes: Stages 3-5 (spec extraction, embedding, summary output) run identically regardless of the HAR source. And curl_exec always hits the real live server — no responses are ever mocked.

---

Integration with the Environment

RL Environment (FastAPI server)
    │
    ├── receives Action: {tool: "browser_agent", input: {task, url}}
    │
    ├── Stage 1: HAR file exists?
    │     ├── YES → load HAR from disk (~0ms)
    │     └── NO  → spawn live browser session (30-120s)
    │               ├── Playwright + bu-30b-a3b-preview
    │               ├── intercept all HTTP traffic
    │               └── produce HAR data
    │
    ├── Stage 3: Extract OpenAPI-like spec from HAR
    │
    ├── Stage 4: Build GEMMA embeddings → stored in env for search_endpoints()
    │
    └── Stage 5: Return summary endpoint list as Observation.last_tool_result
          │
          │  (agent now knows WHAT endpoints exist, but not HOW to call them)
          │
          ▼
    search_endpoints("natural language query")
          │  → semantic search over GEMMA embeddings
          │  → returns full endpoint schema with params, auth, response fields
          ▼
    curl_exec("curl -X POST ...")
          │  → executes against real live WebArena server (EC2)
          │  → indexes full response into episode BM25 store
          ▼
    search_episode_data("keyword query")
          │  → BM25 search over indexed responses from this episode
          ▼
    done() → judge evaluates against ground truth

---

Reference Tools

• browser-use GitHub — the core library
• browser-use docs — configuration, custom actions, LLM setup
• Playwright network events — request/response interception API
• har-to-openapi — alternative: convert HAR files to OpenAPI spec format
• jsluice — extract API routes from JavaScript bundles (useful supplement to network interception) - Future Scope