# Benchmark suite for AI agents (SWE-bench, WebArena, GAIA)

> VI: https://5ac.vn/wiki/benchmark-suite/ · EN: https://5ac.vn/en/wiki/benchmark-suite/ · lang: en · spec_as_of 2026-10-18 (spec no formal spec) · cluster C4

A packaged trio of task set, runnable environment and grading rule that lets two agents be measured on one scale — a score only means something with the suite version and the scaffold that produced it.

---
## Definition

A benchmark suite for agents is a publicly packaged trio: a task set, an environment to act in, and a scoring rule. SWE-bench tests fixing real GitHub issues, WebArena tests long-horizon web tasks, GAIA tests everyday questions needing browsing and tools — a score means nothing without the suite and its version.

## Mechanism

Unlike a question-answer dataset, an agent suite needs three layers: (1) tasks with an answer key or a functional pass/fail criterion; (2) an environment the agent can actually operate — code repos, running websites, a browser; (3) a grader that checks the end state, not the prose. The three pinned suites illustrate three design bets:

| Suite | What it measures | Scale per the original abstract | Score at publication |
| --- | --- | --- | --- |
| SWE-bench (arXiv:2310.06770, submitted 10 Oct 2023, ICLR 2024) | resolving real issues in code | 2,294 problems from GitHub issues and pull requests across 12 Python repositories | best model of the day (Claude 2) resolved 1.96% |
| WebArena (arXiv:2307.13854, submitted 25 Jul 2023) | long-horizon tasks on realistic sites | four fully functional website domains: e-commerce, forums, software development, content management | best GPT-4 agent: 14.41% end-to-end success, humans: 78.24% |
| GAIA (arXiv:2311.12983, submitted 21 Nov 2023) | general assistants: reasoning, multimodality, browsing, tool use | 466 questions, answers withheld for 300 behind a leaderboard | humans 92% vs 15% for GPT-4 with plugins |

Why leaderboard numbers never sit still: what changes constantly is the *model and the scaffold* (harness, prompts, tool loop), not just the suite. Suites are versioned too — the SWE-bench arXiv page records three submissions, v1 Oct 2023 to v3 Nov 2024. The minimum citation for an agent score: suite + version/commit, the scaffold and model run, the run date, and a public reproducible source. Miss one and the number is marketing, not measurement.

## Spec anchor

There is **no committee spec for benchmark suites**, so this entry's spec-version honestly reads "no formal spec"; the as-of date is the source-retrieval date, not a revision date. What is normative for each suite is its original paper plus repository: the Datasets/Benchmarks section defines tasks, execution and grading; the leaderboard defines ranking. That is the structural difference from the protocol entries in this wiki: MCP and A2A carry dated revision pages; a suite's authority lives in reproducibility — an openable environment, a public grader, a frozen versioned task set.

Sources retrieved to build this entry:

- <https://arxiv.org/abs/2310.06770> — SWE-bench: title, abstract, v1–v3 history, ICLR 2024 note — accessed 18 Oct 2026 (HTTP 200)
- <https://arxiv.org/abs/2307.13854> — WebArena: abstract (four site domains, 14.41% vs 78.24%), v1–v4 — accessed 18 Oct 2026 (HTTP 200)
- <https://arxiv.org/abs/2311.12983> — GAIA: abstract (466 questions, 92% vs 15%), leaderboard referenced in-paper — accessed 18 Oct 2026 (HTTP 200)
- <https://arxiv.org/abs/2507.21504> — "Evaluation and Benchmarking of LLM Agents: A Survey" — accessed 18 Oct 2026 (HTTP 200)

## Academic anchors

- *SWE-bench: Can Language Models Resolve Real-World GitHub Issues?* — arXiv:2310.06770 (Jimenez et al.). Axis: real code, real tests.
- *WebArena: A Realistic Web Environment for Building Autonomous Agents* — arXiv:2307.13854 (Zhou et al.). Axis: functional web environment scored on functional correctness.
- *GAIA: a benchmark for General AI Assistants* — arXiv:2311.12983 (Mialon et al.). A contrarian bet: questions conceptually simple for humans, hard for advanced AI.
- *Evaluation and Benchmarking of LLM Agents: A Survey* — arXiv:2507.21504 (Mohammadi et al., submitted 29 Jul 2025): orders the field along two axes — what is evaluated (behavior, capability, reliability, safety) and how (interaction mode, datasets, metrics, tooling). The lens that makes the suites above comparable.

## FAQ

**How is a suite different from a plain dataset?** A dataset ships questions and answers; an agent suite also ships an *environment to act in* and a *grader for the final state*. WebArena stands up actually runnable websites; SWE-bench applies the model's patch and runs the repository's tests.

**Two posts both cite SWE-bench, numbers worlds apart — why?** Different model, different scaffold, possibly a different suite version (tasks and repos changed v1→v3). Comparing scores while those variables differ is meaningless.

**What to check before trusting a benchmark claim?** Four items: suite + version/commit; scaffold and model actually run; run date; public reproducible source. A number that only appears on a sales slide is guilty until proven innocent.

**Which suite fits which agent?** Code fixers: SWE-bench (or its Lite band). Agents operating web software: WebArena. General assistants with tools: GAIA. For the whole landscape, survey 2507.21504 classifies suites along its two axes.

**Do low launch scores mean anything?** They are the *designed gap*: 1.96%, 14.41%, 15% in 2023 became a yardstick — later model generations climb the same frozen suite under the same grading rules. That is exactly what a benchmark is for.

## Related

- [What is agent memory](/en/wiki/agent-memory/)
- [What is agentic RAG](/en/wiki/agentic-rag/)
- [Model Context Protocol (MCP)](/en/wiki/mcp/)
- [MCP protocol eras](/en/wiki/mcp-eras/)
- [What is a Multi-Agent OS?](/en/blog/what-is-multi-agent-os/)