Same failure score, different payoff: judge an oversight signal by the decisions it drives

Chubin Zhang et al. · 1 authors · Nanyang Technological University, National University of Singapore, A*STAR CFAR and UNC-Chapel Hill · 2026 · arXiv:2606.21399

Same failure score, different payoff: judge an oversight signal by the decisions it drives

Opening

Your approval gate fires when the agent’s failure score crosses a threshold, and that feels like the right moment to intervene. The paper shows why the feeling can be wrong. Two states can carry the same failure risk while only one of them gains anything from a human stepping in. A score built to predict failure cannot separate those states, and recalibration that preserves the score’s ordering cannot recover the difference either. What separates them is whether the intervention itself pays off.

What the paper does

The authors formalize when a summary of the state is sufficient for an intervention decision, and the utility lost when it is not. They measure that gap by replaying agent prefixes and executing the alternative action from the same state, holding features, estimator, and router fixed and changing only the supervision target from failure risk to intervention utility. The main run is on ALFWorld, the comparison replicates on a second set of mid-episode prefixes, and an online test covers 300 unseen tasks with a fixed stronger-model handoff.

Findings

  1. They formalize the sufficiency question — the paper states when a state summary is enough to decide an intervention, and puts a figure on the utility lost when it is not.

We formalize when a summary is sufficient for intervention decisions and the utility lost when it is not.

  1. Changing the target changes the result — with features, estimator, and router fixed, moving supervision from failure risk to intervention utility lowered regret from 0.51 to 0.09 on ALFWorld.

changing the supervision target from failure to intervention utility lowers regret from 0.51 to 0.09

  1. The intervention-trained controller hands off less and succeeds more — on 300 unseen tasks it handed off 35% of the time against 48% for failure-triggered routing, and succeeded 45% against 37%.

a frozen prefix-feature controller improves utility over failure-triggered routing, handing off less often (35% vs 48%) and succeeding more often (45% vs 37%)

  1. Judge the signal by the decision it drives — the closing rule is that oversight signals are evaluated by the decisions they support next to their predictive quality.

Oversight signals should be evaluated by the decisions they support alongside their predictive quality.

Caveat

The abstract reports no dollar cost and names no baseline beyond failure-triggered routing. The largest gains sit on ALFWorld and one set of mid-episode prefixes, and gains are small on two reasoning benchmarks, so the mechanism is demonstrated in that setting rather than established across task types.

What this means for 5ac

Every gate we run is an intervention: the kanban approval card, the secret-triage hook, the moment route.sh pushes a job to a stronger model. Stop asking only whether the guard’s risk score predicts failure. Ask what changes when it fires. Keep two numbers for each guard — how often it hands off, and how often the handed-off job completes — and score the guard on their movement. A gate that flags risk all day without changing a decision is noise, not safety.

Source