Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

“LLM agents are increasingly used to solve real-world problems, and their effectiveness depends on both the underlying models and the harnesses that support their operation.”

Zhiyong Cao, Hengjie Zhu, Yongxin Wang +20 · 23 authors · Chinese Academy of Sciences, Tsinghua University, and Beijing Institute of Technology · 2026 · arXiv:2610.32285

Benchmarks grade the model, but who grades the broken harness?

Opening

When an agent goes wrong, teams usually blame the model, while most of what decides the result sits around the model: prompts, tools, the loop, the gates, and that part almost never gets graded on its own. Existing benchmarks mostly score the model and the overall agent, so a run can still reach the right answer while some stage of the harness is already failing. This paper builds a benchmark that grades the harness directly, with executable ground truth instead of impressions, and with evidence pulled from multiple sources instead of the agent’s own account.

What the paper does

TraceTrial-Bench splits harness reliability into seven components: context, execution, governance, lifecycle, observability, tools, and verification. The authors construct 95 expert-level, mechanism-oriented tasks with executable or state-based ground truth, so semantically equivalent comparisons still preserve the native behavior of each harness. Scoring runs through a two-layer framework that separately assesses agent performance and harness reliability, which lets a run complete the task while still violating a harness criterion. Three modules work side by side: Trajectory Tracing captures the execution traces, Multi-source Evidence Acquisition provides evidence from the agent context, the harness and the execution environment for LLM-based grading, and Failure Attribution traces a failure back through explicit mechanism chains at the run level. The evaluation covers 4 harnesses, 9 models and 95 tasks, and because each run’s verdict is traceable to evidence, a violation tells you which part of the harness broke instead of only showing a total score.

Findings

  1. Existing benchmarks focus on model capability and overall agent performance, with almost no attention to grading the harness. (most existing benchmarks focus on model capabilities and overall agent performance, with limited attention to the evaluation of agent harnesses)

  2. TraceTrial-Bench organizes harness reliability into seven components, from context to verification. (we introduce TraceTrial-Bench, which organizes harness reliability into seven components covering context, execution, governance, lifecycle, observability, tools, and verification)

  3. Across 4 harnesses, 9 models and 95 tasks, 3,420 runs yield macro-averaged harness scores from 59.8% to 72.0%, while 31.1% of runs achieve task success despite failing at least one harness criterion. (Across 4 harnesses, 9 models, and 95 tasks, our evaluation of 3,420 runs yields macro-averaged harness scores ranging from 59.8% to 72.0%)

Caveat

The abstract states no limitation of the paper. The closest sentence is a gap statement, describing what existing benchmarks still lack: they focus on model capabilities and overall agent performance with limited attention to harness evaluation. The authors use that gap as the motivation for their benchmark, so it has to be read as the reason the paper exists, not a weakness of the study. Readers should also keep in mind it marks a hole in the ecosystem, not a verdict that older benchmarks were done wrong.

What this means for 5ac

The paper’s seven components map straight onto the 5ac stack: context is the prompt and memory loaded in, execution is the sandbox and terminal, governance is the gates and permissions, lifecycle is the Kanban card, observability is the run log, tools are skills and MCP servers, verification is the acceptance script. With that frame, each profile can be scored on its own instead of merged into one general rating. The concrete job is a periodic cron that runs a set of per-component questions against every profile and stores the results in Gbrain so the numbers can be watched over time. When a card fails, the record points at the exact component that broke, instead of asking the vague question of whether the model got weak.

Source