| --- |
| title: BrowserGym Leaderboard |
| emoji: 🏆 |
| colorFrom: purple |
| colorTo: green |
| sdk: docker |
| pinned: false |
| license: mit |
| short_description: Tracks perf of LLMs, VLMs and agents on web navigation tasks |
| --- |
| |
| # BrowserGym Leaderboard |
|
|
| This leaderboard tracks performance of various agents on web navigation tasks. |
|
|
| ## How to Submit Results for New Agents |
|
|
| ### 1. Create Results Directory |
| Create a new folder in the `results` directory with your agent's name: |
| ```bash |
| results/ |
| └── your-agent-name/ |
| ├── README.md |
| ├── webarena.json |
| ├── workarena-l1.json |
| ├── workarena++-l2.json |
| ├── workarena++-l3.json |
| └── miniwob.json |
| ``` |
|
|
|
|
| ### 2. Add Agent Details |
|
|
| Create a `README.md` in your agent's folder with the following details: |
|
|
| #### Required Information |
| - **Model Name**: Base model used (e.g., GPT-4, Claude-2) |
| - **Model Architecture**: Architecture details and any modifications |
| - **Input/Output Format**: How inputs are processed and outputs generated |
| - **Training Details**: Training configuration if applicable |
| - Dataset used |
| - Number of training steps |
| - Hardware used |
| - Training time |
|
|
| #### Optional Information |
| - **Paper Link**: Link to published paper/preprint if available |
| - **Code Repository**: Link to public code implementation |
| - **Additional Notes**: Any special configurations or requirements |
| - **License**: License information for your agent |
|
|
| Make sure to organize the information in clear sections using Markdown. |
|
|
| ### 3. Add Benchmark Results |
|
|
| Create separate JSON files for each benchmark following this format: |
|
|
| ```json |
| [ |
| { |
| "agent_name": "your-agent-name", |
| "study_id": "unique-study-identifier-from-agentlab", |
| "date_time": "YYYY-MM-DD HH:MM:SS", |
| "benchmark": "WebArena", |
| "score": 0.0, |
| "std_err": 0.0, |
| "benchmark_specific": "Yes/No", |
| "benchmark_tuned": "Yes/No", |
| "followed_evaluation_protocol": "Yes/No", |
| "reproducible": "Yes/No", |
| "comments": "Additional details", |
| "original_or_reproduced": "Original" |
| } |
| ] |
| ``` |
|
|
| Please add all the benchmark files in separate json files named as follows: |
|
|
| - `webarena.json` |
| - `workarena-l1.json` |
| - `workarena-l2.json` |
| - `workarena-l3.json` |
| - `miniwob.json` |
|
|
| Each file must contain a JSON array with a single object following the format above. The benchmark field in each file must match the benchmark name exactly ([`WebArena`, `WorkArena-L1`, `WorkArena-L2`, `WorkArena-L3`, `MiniWoB`]) and benchmark_lowercase.json as the filename. |
| |
| ### 4. Submit PR |
| |
| 1. Open the community tab and press "New Pull Request" |
| 2. Give it a new title to the PR and follow the steps mentioned |
| 3. Publish the branch |
| |
| ## How to Submit Reproducibility Results for Existing Agents |
| |
| Open the results file for the agent and benchmark you reproduced the results for. |
| |
| ### 1. Add reproduced results |
| |
| |
| Append the following entry in the json file. Ensure you set `original_or_reproduced` as `Reproduced`. |
| |
| ```json |
| [ |
| { |
| "agent_name": "your-agent-name", |
| "study_id": "unique-study-identifier-from-agentlab", |
| "date_time": "YYYY-MM-DD HH:MM:SS", |
| "benchmark": "WebArena", |
| "score": 0.0, |
| "std_err": 0.0, |
| "benchmark_specific": "Yes/No", |
| "benchmark_tuned": "Yes/No", |
| "followed_evaluation_protocol": "Yes/No", |
| "reproducible": "Yes/No", |
| "comments": "Additional details", |
| "original_or_reproduced": "Reproduced" |
| } |
| ] |
| ``` |
| |
| ### 2. Submit PR |
|
|
| 1. Open the community tab and press "New Pull Request" |
| 2. Give it a new title to the PR and follow the steps mentioned |
| 3. Publish the branch |
|
|
| ## License |
|
|
| MIT |
