Can pooled accuracy tell you your agent tracks a changing fact?

Xianxuan Long et al. · 1 authors · Michigan State University · 2026 · arXiv:2606.17328

Can pooled accuracy tell you your agent tracks a changing fact?

Opening

You re-run the same short list of questions after every memory change, watch the average climb, and take it as proof that the memory works. What that average cannot tell you is how one stored fact behaves when conditions change: the customer moves house, a preference gets reversed, a question arrives built on a false premise. Question-by-question scoring answers whether the model said the right thing this time. It stays quiet about the thing an operator actually needs to know, which is whether one fact holds up across those shifts.

What the paper does

MemTrace is a benchmark that swaps the unit of measurement. Instead of a question row, the unit is the knowledge point: one typed fact about the user, probed along three controlled dimensions (memory age, question type, and evidence condition), with 13 memory-system configurations evaluated across four paradigms. The design exists to separate failures that pooled accuracy blends together.

Findings

  1. Scoring rows independently hides how the fact behaves — Each question is graded on its own, so even when several questions probe the same fact, the score never shows how that fact reacts as conditions shift. You can post a clean dashboard and still not know what the agent does with change.

Because this approach scores question rows independently, even when several questions probe the same fact, it cannot show how that fact behaves as conditions change

  1. The unit becomes the knowledge point — A single typed fact about the user replaces the individual question, so the same fact gets probed under several conditions instead of being scattered across disconnected rows.

a benchmark whose unit of measurement is the knowledge point: a single typed fact about the user, rather than an individual question

  1. Recalling states is not the same as tracking change — Getting the current and earlier states of a fact right does not mean the system can describe how that fact changed, and declining to answer safely is not the same as correcting a false premise.

recovering a fact’s current and earlier states does not imply tracking how it changed

  1. The bottleneck is evidence use, not retrieval — On the failures the authors examined, the evidence was usually findable and simply went unused. Extra storage and better retrieval do not touch that failure mode.

when systems fail, the evidence was retrievable 10 times more often than it was missing

Caveat

The study covers thirteen configurations on one benchmark built from synthetic facts, so read it as a diagnostic about how systems use evidence rather than as a ranking of commercial memory products.

What this means for 5ac

When a 5ac agent answers wrongly, the first move is not to buy more retrieval or grow the store. Trace whether the evidence was already in the context at answer time; if it was, the failure sits in how the model used it, and the fix belongs in prompting, citation discipline, or question handling. The benchmark also suggests a cheap internal test: stop grading scattered questions one by one, group the ones that probe the same fact, and see whether the agent follows that fact through change. Spending follows the same logic, since using stored evidence well beats paying to store more of it.

Source