Rethinking Verification in Agent Harnesses

“Agent harnesses enable large language models to act through tools, observe the results, and iteratively work toward a task goal.”

Zhengxin Zhang, Erkang Zhu, Yin Lin +3 · 6 authors · Cornell University and Alibaba Group · 2026 · arXiv:2610.34577

Where in the agent lifecycle is checking worth the spend?

Opening

Most harnesses check the agent in one place only, something like telling the model to re-read its work before submitting, and treat that as verification. Checking and revising like that spends tokens and time, and placed in the wrong spot it both costs more and catches less. This paper turns verification into an axis of test-time scaling and asks three questions: when to intervene, who checks the work, and how checking leads to revision.

What the paper does

The authors split verification into four layers along the agent lifecycle: initial instructions, mid-turn reminders, completion gates at submission, and independent review by a separate critic. They compare 17 designs across seven frontier coding models on Terminal-Bench and SWE-bench Verified, and build VisualBench so experts can judge visual and interactive artifacts. The comparison shows initial instructions and execution-time reminders bring only modest, model-dependent gains without substantially raising average output token use, while completion gates can give larger gains but push cost up clearly, so the authors treat them as a separate option rather than grouping them with the other kinds of checking. Independent review by a critic comes out with the more favorable accuracy-cost tradeoff. On VisualBench it exceeds every evaluated L0-L2 design, and additional review rounds or independent critics lift average Terminal-Bench accuracy by up to another 3.5 points.

Findings

  1. The framework spans four layers of the agent lifecycle, from initial instructions to independent review. (Our framework spans four layers of the agent lifecycle: initial instructions, mid-turn reminders, completion gates, and independent review)

  2. The study compares 17 designs across seven frontier coding models on two established benchmarks. (We compare 17 designs across seven frontier coding models on Terminal-Bench (TB) and SWE-bench Verified (SWE))

  3. Independent review lifts average accuracy 5.9-9.5 percentage points on Terminal-Bench and 3.9-6.6 on SWE-bench at a small multiple of baseline cost; on VisualBench it reaches net preferences of 13.2-23.8 points over baseline. (improving average accuracy by 5.9-9.5 percentage points on TB and 3.9-6.6 on SWE at 1.13-1.25 and 1.00-1.04 times baseline cost, respectively)

Caveat

The tradeoff the authors state: completion gates can yield larger gains but may substantially increase cost. That is why independent review emerges as the option with the better accuracy-for-cost tradeoff, rather than a conclusion that every kind of checking pays equally. The authors also note the gains remain model-dependent, so a design that works well with one model is not automatically good with another.

What this means for 5ac

The Kanban at 5ac already runs this axis: a card self-checks first, then a different reviewer profile checks at the completion gate, and the paper’s numbers defend that arrangement with data instead of belief. Adding independent review costs a little more but buys higher accuracy, exactly the kind of cost 5ac accepts for an important lane. The job is to stop routing every card through the same gate: an experiment card can stop at a completion gate, while a card touching money or production must get a separate reviewer. When a lane needs more accuracy, adding an independent review round is the option with data behind it, rather than stacking more instructions into the prompt. Visual artifacts are where independent review showed its clearest edge in the paper, and 5ac meets that kind of work whenever a reviewer judges a page, an OG image or a render instead of only text.

Source