Can your agent delete a secret, or only hide it from search?

Cheng Zhang et al. · 1 authors · The Hong Kong University of Science and Technology (Guangzhou) · 2026 · arXiv:2610.21405

Can your agent delete a secret, or only hide it from search?

Opening

A customer writes “please delete my data,” you run a delete against the store, and the ticket closes. Weeks later the agent answers with something it should no longer know, because the secret never lived in one row: it leaked into summaries, tags, and retrieval candidates built from the original conversation. Retrieval research has spent years getting better at finding information; forgetting it on request is the half nobody measures. MemCut turns that gap into a test you can run.

What the paper does

MemCut is a systematic benchmark for user-requested forgetting. The authors hand-design 456 secrets and place them inside long-horizon conversations, each with a fully annotated evidence trail so recovery can be tested after deletion, including from messages that never state the secret directly. Two recovery metrics separate the layers: native recovery measures whether the system’s own retrieval still surfaces the secret, while residual recovery hands the reader the surviving objects directly. Three reference deletion baselines are then run across five memory designs and four LLMs to see how far existing retrievers and write models actually support forgetting.

Findings

  1. Forgetting is half of a dependable memory — the benchmark starts from a requirement that accurate retrieval alone does not satisfy: when the user asks, sensitive data has to go.

a dependable memory system must also forget sensitive user data upon request

  1. Secrets are planted with trails, not just texts — 456 hand-designed secrets sit inside long conversations, each carrying an annotated evidence trail for testing recovery after deletion, including secrets never stated directly.

Its 456 hand-designed secrets in long-horizon conversations have fully annotated evidence trails to test recovery after deletion

  1. Two metrics split retrieval failure from real erasure — native recovery asks whether the system’s retrieval still finds the secret; residual recovery gives the reader the surviving objects directly, which exposes what deletion left behind in storage.

Native recovery measures reader success through the system’s retrieval, whereas residual recovery measures the same reader’s success when directly given surviving objects

  1. Deleting more conversation still leaves the secret reachable — after LLM-guided deletion of ten additional turns drawn from twenty retrieval candidates, residual recovery remains between 39% and 76%.

LLM-guided deletion of ten additional turns from twenty retrieval candidates leaves 39-76% residual recovery

Caveat

The residual-recovery range comes from three reference deletion baselines rather than shipped memory products, so it describes those five designs and four models, and it certifies no vendor.

What this means for 5ac

If a 5ac agent stores customer conversations, “delete my data” will arrive as a real ticket, and the honest scope of that deletion is the whole derived chain: summaries, indexes, tags, and retrieval candidates, not the message row. The practical move is to keep provenance on every fact written to memory, so a deletion request can be traced back and purged everywhere it propagated instead of deleting one record and hoping. Before accepting a “data deletion” feature from any vendor, ask the two questions this benchmark makes concrete: does retrieval still surface the deleted item afterwards, and what survives in storage when retrieval is bypassed entirely?

Source