How Do Coding-Agent Harnesses Self-Evolve? An Empirical Study of Retrospective Adaptation

“Coding agents can adapt their harnesses, the software, configuration, and instructions used to run them, based on experience from earlier tasks, but whether these adaptations improve their behavior on later tasks remains unclear.”

Christoph Treude, Sebastian Baltes · 2 authors · Singapore Management University and Heidelberg University · 2026 · arXiv:2610.34938

Skills get patched after every session — what proves the patch helps the next one?

Opening

Coding agents increasingly propose harness changes after finishing a session, editing their own software, configuration and instructions, and treat that as a kind of progress. Nobody had answered whether those changes actually help later work. This paper tests exactly that question, on the same repositories, with fresh sessions for each variant, instead of assuming every self-edit is an improvement.

What the paper does

The authors run Claude Code on 30 SWE-bench Verified issues and collect 125 proposals from a retrospective skill that asks an agent to review a completed session and propose harness changes. They implement 24 of the proposals, then evaluate 12 harness changes on later issues from the same repositories in fresh sessions with and without each change. Every proposal is coded qualitatively, and the coding shows most proposals referred to an event in the session, reacting to what had just happened rather than to a long-term principle, while 18 of 43 lint-related proposals recommended a practice regardless of what had happened. The paired evaluation then tells the harder story: only some of the implemented changes reduced the repeated work they targeted, and some changes added work on unrelated issues.

Findings

  1. The paper asks straight whether an agent adapting its own harness actually improves behavior on later tasks. (Coding agents can adapt their harnesses, the software, configuration, and instructions used to run them, based on experience from earlier tasks, but whether these adaptations improve their behavior on later tasks remains unclear)

  2. Claude Code ran on 30 SWE-bench Verified issues; 125 proposals were coded, 24 implemented, and 12 changes evaluated in fresh sessions on later issues. (We ran Claude Code on 30 SWE-bench Verified issues, qualitatively coded the 125 proposals from \retro, implemented 24, and evaluated 12 harness changes on later issues from the same repositories in fresh sessions with and without each change)

  3. Of the seven evaluations where the control sessions actually hit the targeted problem, two changes eliminated the action, one reduced it, and four showed no reduction at all. (Among the seven evaluations in which sessions without the change encountered the targeted problem, two changes eliminated the targeted action, one reduced its occurrence, four showed no reduction, and some changes added work on unrelated issues)

Caveat

The limitation the authors state: harness self-evolution requires more than generating and implementing changes; it needs ways to establish whether a change helps on later tasks and to revise or remove changes that do not. The paper does not supply that method, so a good feeling after a session cannot count as evidence, and the paper’s own numbers show why: four of seven informative changes produced no reduction.

What this means for 5ac

5ac meets this pattern head-on: after each hard card, a skill gets another preventive step, and the real question is whether that step reduces repeated work on the next card. The paper’s answer is to test on real traffic: hold half of the upcoming cards on the old skill, run the other half with the patch, and compare the counts of repeated errors. That rule goes into the skill-review gate: every patch must carry evidence from a later run, and a patch that merely sounds reasonable is not accepted. A change without evidence gets removed rather than accumulating into an instruction nobody remembers the reason for. The same discipline keeps a daily reminder cron only after a run has shown it caught a real failure, and retire it when it stops showing that.

Source