OSWorld: measuring agents on a real computer instead of a fabricated environment
“Autonomous agents that accomplish complex computer tasks with minimal human interventions have the potential to transform human-computer interaction, significantly enhancing accessibility and productivity.”
Demos click smoothly: what happens on a real computer?
Opening
Benchmarks for computer agents have either lacked an interactive environment or been boxed into a single application or domain. Results from such an environment say little about using a real computer, where a person opens websites, desktop apps, and files on disk in an interleaved flow. OSWorld was built to measure exactly that setting: let the agent operate a real computer, and score by whether the result actually runs.
What the paper does
OSWorld is a scalable real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across operating systems such as Ubuntu, Windows, and macOS. On top of it the authors build a benchmark of 369 computer tasks covering real web and desktop apps, OS file I/O, and workflows that span multiple applications. Each example is derived from a real-world computer use case, with a detailed initial state setup configuration and its own evaluation script, so results are reliable and reproducible. They run an extensive evaluation of several LLM/VLM-based agents and report the gap against humans.
Findings
-
OSWorld is the first scalable real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across operating systems such as Ubuntu, Windows, and macOS. (we introduce OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating systems such as Ubuntu, Windows, and macOS)
-
The benchmark holds 369 tasks, and each one carries its own initial state configuration and execution-based evaluation script so the result is reproducible rather than anecdotal. (Each task example is derived from real-world computer use cases and includes a detailed initial state setup configuration and a custom execution-based evaluation script for reliable, reproducible evaluation)
-
The gap with humans is wide: humans accomplish over 72.36% of the tasks while the best model reaches only 12.24%, mostly stuck on GUI grounding and operational knowledge. (While humans can accomplish over 72.36% of the tasks, the best model achieves only 12.24% success, primarily struggling with GUI grounding and operational knowledge)
Caveat
The abstract does not state a limitation of the benchmark itself; what the authors do state is a measured result, namely that extensive evaluation on OSWorld reveals significant deficiencies in the ability of modern LLM/VLM agents to serve as computer assistants. Read correctly, this is a finding about agent capability, and the distance between humans and models is the yardstick this benchmark was built to track.
What this means for 5ac
OSWorld shows that execution-based scoring in a real environment is feasible, and that is precisely the model 5ac uses when verifying a card: the worker has to run real commands in the workspace and paste the output, rather than grading itself by its own account. The detail worth copying is the per-task initial state configuration: every card type on the Kanban board should have a sandbox with a fixed input state, so a later run can be compared against an earlier one. Per-task evaluation scripts map onto the acceptance criteria declared when a card is created, instead of one generic criterion. Together they turn “the agent did the job” into a reproducible number, and a fixed input state is what makes a comparison between two runs meaningful: same card, same starting point, differing only in what this run changed.
Source
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments — https://arxiv.org/abs/2404.07972