WebArena: a realistic web environment to measure whether an agent can actually do the work

“With advances in generative AI, there is now potential for autonomous agents to manage daily tasks via natural language commands.”

Shuyan Zhou, Frank F. Xu, Hao Zhu +9 · 12 authors · Carnegie Mellon University · 2023 · arXiv:2307.13854

Agents look good on fake websites: what happens on real ones?

Opening

Web-controlling agents are usually built and tested on simple synthetic environments, so a result from the demo says little about real work. Put the same agent on a live website with real data, accounts, and workflows, and the old way of measuring barely reflects what actually happens. The operator’s question is a practical one: how do you know the agent did the job instead of hearing it say it did, and which number proves that before you hand over real work?

What the paper does

WebArena builds an environment for language-guided agents with fully functional websites across four domains: e-commerce, social forum discussions, collaborative software development, and content management. The environment is enriched with tools such as a map and external knowledge bases such as user manuals, which pushes task-solving toward how a human would approach it. On top of that the authors release a set of benchmark tasks focused on the functional correctness of task completions, with tasks modeled on what people routinely do on the internet. They experiment with several baseline agents, including recent techniques such as reasoning before acting, and report the success rates.

Findings

  1. WebArena builds a realistic and reproducible environment for language-guided agents, with fully functional websites spanning four common domains rather than a mock page rebuilt for each experiment. (In this paper, we build an environment for language-guided agents that is highly realistic and reproducible. Specifically, we focus on agents that perform tasks on the web, and create an environment with fully functional websites from four common domains: e-commerce, social forum discussions, collaborative software development, and content management.)

  2. The environment carries tools and external knowledge bases, such as a map and user manuals, to encourage human-like task-solving instead of only executing the instruction it was handed. (Our environment is enriched with tools (e.g., a map) and external knowledge bases (e.g., user manuals) to encourage human-like task-solving)

  3. The measured gap is still wide: among the agents tried, the best GPT-4-based agent reaches an end-to-end task success rate of 14.41%, far below the 78.24% achieved by humans. (our best GPT-4-based agent only achieves an end-to-end task success rate of 14.41%, significantly lower than the human performance of 78.24%)

Caveat

The abstract does not present limitations of the paper; the number the authors give is a measured result, and the correct reading is that real web tasks were still out of reach for agents at the time of publication. It should be read as a yardstick rather than a failure, because the gap between agents and humans is exactly what WebArena was built to track. The paper does not claim that the gap has narrowed, and it does not state limits of the experimental environment itself.

What this means for 5ac

The lesson from WebArena is to score by real output rather than by the agent’s own account. In 5ac’s Kanban pipeline, the verification gate requires a worker to paste real command output as evidence before a card counts as done, instead of grading itself, and that is functional correctness in the paper’s sense. 5ac can build a reproducible sandbox per card type, each run starting from the same initialized state, so results are comparable across runs rather than judged by feel. When completion rates are as low as WebArena measured, the scoring step is what decides whether the claim can be trusted.

Source