Send everything to the human and safety falls: the reviewer fatigues
Send everything to the human and safety falls: the reviewer fatigues
Opening
The standard pattern for irreversible actions — shell commands, file edits, deploys — is a human approval gate: the risky action pauses and a person decides. The paper’s argument is that the gate is the easy part. The hard part is the judgment of which actions to stop, and that judgment is evaluated against two false assumptions: that a ground-truth notion of “risky” exists, and that the reviewer is a perfect, infinitely available oracle.
What the paper does
The authors hand-label 125 adversarially-weighted agent actions and measure how far reviewers agree on which are risky. They frame the guard as selective classification under asymmetric cost, which makes its operating limits measurable and shows that on hard inputs the guard cannot safely auto-decide. They then model the reviewer as endogenous — fatiguing as escalation load grows — and turn realized safety into a curve over the escalation rate. Their stated contribution is an open-source agent-oversight system that operationalizes and measures prior art (fatigue-aware learning-to-defer, cost-sensitive deferral under workload constraints, trajectory-level guarding, reviewer-flooding attacks) in the LLM-agent action-gating setting.
Findings
- There is no single correct risk label — on a hand-labeled set of 125 adversarially-weighted agent actions, reviewers only moderately agree on what is risky, with Fleiss’ kappa at 0.52.
On a hand-labeled set of 125 adversarially-weighted agent actions we show that (i) reviewers only moderately agree on what is risky (Fleiss’ kappa = 0.52), so there is no single correct label
- More oversight can mean less safety — once the reviewer fatigues as escalation load grows, realized safety becomes an inverted-U in the escalation rate.
when the reviewer is modeled as endogenous (fatiguing as escalation load grows), realized safety becomes an inverted-U in the escalation rate: more human oversight can make a system less safe
- The safe guard escalates below full escalation — a load-aware policy uses that same setting to resist a flooding attack that slips a malicious action past a fatigued reviewer.
the safety-optimal guard escalates below full escalation - a setting a load-aware policy also uses to resist a flooding attack that slips a malicious action past a fatigued reviewer
- The hard part is allocating attention, not building the gate — which actions to stop is a resource-allocation problem over finite human attention.
We argue the gate is the easy part; the hard part is the judgment - which actions to stop
Caveat
The abstract states that the inverted-U and the flooding attack are modeling results that motivate a human study, so no measured reviewer data backs the fatigue curve yet. Agreement rests on one hand-labeled set of 125 actions, no dollar cost or latency figure is given, and the mechanisms themselves are claimed as prior art rather than novel.
What this means for 5ac
Our review cards, cron HITL prompts, and customer-delivery approvals all draw on the same pool of attention. Set an escalation budget before tuning any guard: how many items per day a person can actually review at quality, then make the guard spend against that number. Below the budget, more escalation helps; past it, each extra prompt degrades the reviews that remain, which is exactly the inverted-U this paper models. Measure the load, not just the flag rate.
Source
- Oversight Has a Capacity: Calibrating Agent Guards to a Subjective, Fatiguing Human — https://arxiv.org/abs/2606.08919