Approve what the command did, not what it says: pre-run every command in a sandbox

Weiheng Wu et al. · 1 authors · Singapore Management University · 2026 · arXiv:2610.40753

Approve what the command did, not what it says: pre-run every command in a sandbox

Opening

An approver that reads command text is judging intent, not behavior. What a shell command actually does depends on the code it invokes and the state of the workspace, and neither is visible in the string the human approves. The paper inverts the order: run the command somewhere controlled first, then decide what it may keep.

What the paper does

The authors present Caper, a consequence-aware approval mechanism. Each command pre-executes in a snapshotted sandbox; an independent approver receives a bounded report of observed effects instead of the request text; allowed commands are committed and blocked ones rolled back, with a reason the agent can act on. Six existing systems are compared on SABER and on YoloFS-Ext, a benchmark the authors build where harmful behavior hides inside invoked tools.

We present Caper, a consequence-aware approval mechanism that pre-executes each command in a snapshotted sandbox, provides an independent approver with a bounded report of its observed effects, commits allowed commands, and rolls back blocked ones with a reason the agent can act on.

Findings

  1. The decision is made on observed effects, not command text — pre-execution in a snapshotted sandbox gives the approver a bounded report of what the command actually did.

pre-executes each command in a snapshotted sandbox, provides an independent approver with a bounded report of its observed effects

  1. Lowest harmful-outcome rate against six existing systems — on SABER, Caper reaches 6.7% against at least 14.8% for the comparison systems.

Caper achieves the lowest harmful-outcome rate on SABER (6.7% versus at least 14.8%)

  1. Harm hidden inside tool calls stops being invisible — on YoloFS-Ext, Caper reaches 7.8% harmful outcomes against at least 35.3%, while blocking 1.0% of the safe twins.

YoloFS-Ext, a benchmark we build in which harmful behavior is hidden inside invoked tools (7.8% versus at least 35.3%), while blocking 1.0% of the safe twins of the dangerous tasks in YoloFS-Ext.

Caveat

Pre-executing every command costs time and compute, and the abstract reports neither figure — for a team gating a high command volume that is the deciding number. The six compared systems are unnamed in the abstract, and YoloFS-Ext is the authors’ own benchmark, so the evidence rests on two benchmarks rather than production traffic.

What this means for 5ac

Our approval cards still judge the sentence the worker wrote. For risky steps, run the command on a snapshot first — a throwaway branch, a dry run — and put the observed effect into the card: files touched, output, exit status. Block with a reason the agent can act on, not a bare denial, so the next proposal arrives already corrected. That is the difference between approving a promise and approving an effect — the cost is one snapshot and a rerun, cheap next to unwinding a bad merge in a customer’s delivery.

Source