SWE-bench: how many real GitHub issues can a language model actually fix?

“Language models have outpaced our ability to evaluate them effectively, but for their future development it is essential to study the frontier of their capabilities.”

Carlos E. Jimenez, John Yang, Alexander Wettig +4 · 7 authors · Princeton University · 2023 · arXiv:2310.06770

Models are good at toy functions: what happens with a real issue in a real repo?

Opening

Evaluating language models with a single-function problem has become habit: a short description, one input, one output, right or wrong. But a developer’s real work is reading an issue in an existing repo, locating what needs to change among many files, and fixing it without breaking the rest. The two problem types differ enough that model rankings from the old format say nothing once real work is involved, and SWE-bench was built to measure exactly that gap.

What the paper does

SWE-bench is a test set drawn from real GitHub issues, with merged pull requests as reference answers. Each task hands the model a codebase and an issue description and asks it to edit the codebase so the issue is resolved. The authors note that resolving issues frequently requires understanding and coordinating changes across multiple functions, classes, and even files, while calling the execution environment, processing extremely long contexts, and performing complex reasoning, far beyond isolated code generation. On top of this test set they evaluate both leading proprietary models and their own fine-tuned model.

Findings

  1. The benchmark consists of 2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories. (we introduce SWE-bench, an evaluation framework consisting of $2,294$ software engineering problems drawn from real GitHub issues and corresponding pull requests across $12$ popular Python repositories)

  2. Each task stops short of writing one function: the model has to coordinate changes across functions, classes, and files, while interacting with execution environments and handling extremely long contexts. (Resolving issues in SWE-bench frequently requires understanding and coordinating changes across multiple functions, classes, and even files simultaneously, calling for models to interact with execution environments, process extremely long contexts and perform complex reasoning that goes far beyond traditional code generation tasks)

  3. At publication, the best-performing model, Claude 2, solves a mere 1.96% of the issues, and neither the proprietary models nor the fine-tuned one get past the simplest problems. (The best-performing model, Claude 2, is able to solve a mere $1.96$% of the issues)

Caveat

The abstract does not state a limitation of the benchmark itself. What the authors do state is a measured result: both state-of-the-art proprietary models and their fine-tuned model SWE-Llama can resolve only the simplest issues. The correct reading is that this describes model capability at the time of publication rather than a flaw in the benchmark; SWE-bench was built so people can track how that level improves over time.

What this means for 5ac

The SWE-bench recipe is exactly what the verification gate in 5ac’s Kanban pipeline uses: a card counts as done only when real command output is pasted, not when someone claims the work is finished. A real repo with runnable tests maps onto the requirement that every artifact produced must run in the workspace before handoff, which is what each card’s acceptance criteria already declare. We also learned something about choosing evaluations: do not score an agent with easy demo prompts, score it with the kind of work it will do every day. When model routing picks a model for a card, the real difficulty of that job is what decides which model runs. And when an old card is reopened, the pasted command output is what tells you whether the result still reproduces, instead of having to trust anyone’s memory.

Source