Is your agent still quoting last quarter's price?

Md Nayem Uddin et al. · 1 authors · Arizona State University · 2026 · arXiv:2604.20006

Is your agent still quoting last quarter’s price?

Opening

Personalised agents are supposed to remember you, but circumstances change: the address moves, the discount ends, the contact leaves. A benchmark that only asks what the user said last month will happily reward an agent that retrieves the obsolete answer with confidence. The gap this paper points at is that long-term memory evaluation is mostly framed as fact retrieval from past conversations, with little insight into whether the agent consolidates memory over time or handles frequent knowledge updates.

What the paper does

The authors introduce Memora, a long-term memory benchmark spanning weeks-to-months-long user conversations. It evaluates three memory-grounded tasks, remembering, reasoning and recommending, and protects data quality with automated memory-grounding checks plus human evaluation. On top of the benchmark sits Forgetting-Aware Memory Accuracy (FAMA), a metric that penalizes reliance on obsolete or invalidated memory when scoring long-term memory. The evaluation covers four LLMs and six memory agents.

Findings

  1. The old framing is retrieval-shaped — existing benchmarks predominantly frame long-term memory evaluation as fact retrieval from past conversations, which never tests what happens after the facts change.

existing benchmarks predominantly frame long-term memory evaluation as fact retrieval from past conversations

  1. Three tasks instead of one — the benchmark evaluates remembering, reasoning and recommending, so an agent must not only recall a fact but draw conclusions and act on them.

The benchmark evaluates three memory-grounded tasks: remembering, reasoning, and recommending

  1. Obsolete memory now costs points — FAMA penalizes reliance on obsolete or invalidated memory, which is exactly the property that breaks personalisation once a customer’s circumstances move on.

Forgetting-Aware Memory Accuracy (FAMA), a metric that penalizes reliance on obsolete or invalidated memory

  1. Memory agents barely move the needle — across four LLMs and six memory agents, the study finds frequent reuse of invalid memories and failures to reconcile evolving memories, and concludes that memory agents offer marginal improvements.

Memory agents offer marginal improvements

Caveat

The benchmark is new and its human-evaluation step is described only in outline, so the marginal-gain conclusion rests on this single dataset rather than on a replicated result.

What this means for 5ac

Personalisation breaks quietly: the agent keeps using last quarter’s address or price, and the customer notices before your dashboard does. Two habits follow. First, treat staleness as a scored failure in your own evaluations: include questions whose correct answer changed, and count confident use of the old value as wrong, the way FAMA does. Second, ask any memory vendor how it reconciles evolving facts, because the six memory agents in this study failed at exactly that. Memory that only grows is a liability; memory that is revised on schedule is the product, and this paper gives you the metric to demand it.

Source