Three annotators, one rubric, almost no agreement: do not tune interrupt timing on F1

Manvendra Modgil · 1 authors · Independent Researcher · 2026 · arXiv:2606.04296

Three annotators, one rubric, almost no agreement: do not tune interrupt timing on F1

Opening

A runtime safety layer that decides when to interrupt an agent needs a target: a moment labeled “a human should step in here.” This paper tested four families of triggers against human-annotated intervention points on SWE-bench-Verified debugging traces and found the target itself does not hold still. Thresholds saturate into near-constant firing, LLM judges barely fire, and the annotators disagree with each other about where the moment is.

What the paper does

The study uses a continuous 18-dimensional affective-dynamics engine (HEART) as a diagnostic probe and evaluates four trigger families — absolute state thresholds, composite state-action patterns, regex reasoning-feature extraction, and zero-shot LLM-as-judge — against human-annotated intervention points. Three trained annotators label a 56-action trajectory with the same rubric to test whether the supervised target reproduces at all. Firing rates are measured across five trajectories, and LLM judges are swept across vendors with and without full-trajectory context.

Findings

  1. State thresholds stop detecting moments — the state saturation trap converted threshold-on-state triggers into near-constant indicators that fired on 39-83% of actions across five trajectories.

converting threshold-on-state triggers from moment detectors into near-constant indicators that fire on 39-83% of actions across five trajectories

  1. Humans barely agree on where to intervene — three trained annotators using the same rubric on a 56-action trajectory agreed only slightly above chance, with a location Krippendorff’s alpha of +0.047.

three trained annotators using the same rubric on a 56-action trajectory agree on where to intervene only slightly above chance (location Krippendorff’s alpha = +0.047

  1. LLM judges fire weakly and cost more — frontier and cross-vendor models escaped the zero-firing floor only with full-trajectory context, reaching F1 between 0.17 and 0.40 at up to 90x the cost, while a small model never fires.

frontier and cross-vendor models escape the zero-firing floor only with full-trajectory context, reaching only F1 0.17-0.40 at up to 90x the cost

  1. Single-annotator F1 is the wrong target — intervention timing is a low-reliability construct, which makes single-annotator F1 unsuitable as an optimization target for deciding when to interrupt an agent.

We conclude that intervention timing is a low-reliability construct, making single-annotator F1 an unsuitable optimization target.

Caveat

The abstract carries a v2 erratum — the observer could not register recovery and decay never ran — so part of the saturation effect traces to that failure. Human agreement comes from three annotators on a single 56-action trajectory, and the cost comparison reports a relative 90x figure with no dollar amounts.

What this means for 5ac

Before anyone tunes a guard, write the rubric, have two people label the same sample, and compute agreement. If agreement is near chance, stop optimizing F1 — pick decisions instead: which actions get paused, who reviews them, what happens after. For cron and bot ops, watch firing rates. A trigger that fires on most actions is not protecting anything; it is training the reviewer to click approve.

Source