Self-editing agents need an outside gate: keep a change only when the old tests still pass

Sina Tayebati et al. · 1 authors · University of Illinois Chicago and Capital Labs (Capital One) · 2026 · arXiv:2609.24130

Self-editing agents need an outside gate: keep a change only when the old tests still pass

Opening

An agent allowed to rewrite its own operating instructions will fix the failure in front of it and quietly break something that worked yesterday. The paper’s core observation is that both outcomes come from the same edit, so the useful question is not whether agents may self-modify but which changes are allowed to persist. Its answer is admission control: the agent proposes, an external runtime gate decides.

What the paper does

The agent authors candidate behavioral rules in an external workspace. A rule gets provisional execution authority during evaluation, then persistent cross-episode authority only after measured improvement on the triggering failure without regression beyond a fixed margin on protected cases. Replay supplies matched evidence when available, forward trials are the weaker fallback, and a corpus-level guard re-tests the accumulated active rule set. A Detect, Notice, Heal, Validate loop runs around an otherwise unmodified agent.

The agent may propose changes to its operating instructions, while an external runtime gate controls persistence.

Because adaptation changes the policy-inducing context and leaves model weights fixed, admitted edits stay inspectable and reversible and keep working with closed-weight models.

Findings

  1. The outside gate refused 383 proposals — across 16 matched Baseline and Harness runs, the runtime gate rejected 383 replay-decided self-modification proposals.

the gate rejected 383 replay-decided proposals

  1. Most rejected edits traded one bug for another — of those, 211 proposals, 55%, improved the failure that triggered them while degrading a case that previously worked.

Of these, 211 (55%) improved their triggering failure while degrading a case that previously worked.

  1. Refusing that much still raised task-completion score in every pair — admitting changes only after measured improvement did not cost capability, and repeated-trial reliability never got worse.

Task-completion score is higher under the Harness in all 16 pairs, with two paired bootstrap intervals excluding zero, while repeated-trial reliability is higher in 12 pairs, tied in 4, and lower in none.

Caveat

The abstract gives no compute or latency cost for replay and forward trials, and statistical support is thin: only two of the 16 pairs have bootstrap intervals excluding zero. The fixed regression margin that decides when a change counts as harmful is never varied, so how sensitive the gate is to that setting is not established.

What this means for 5ac

Our workers already modify themselves: they edit skills, prompts, and even the checker scripts that judge them. Today the edit usually sticks the moment it is written. Put a replay gate in front of it — run the existing checks against the old corpus, keep the change only if the failing case improves and the passing ones hold. The 55% collateral figure says most “fixes” would break something that already worked. No new model is required; the checkers we run daily are already the replay corpus.

Source