When old and new facts collide, which one does your memory answer with?

Zhen Tao et al. · 1 authors · Renmin University of China · 2026 · arXiv:2605.20926

When old and new facts collide, which one does your memory answer with?

Opening

Your CRM holds last year’s price and this year’s price, two contacts share a name, and a client’s address changed in March. A memory system will retrieve something for every question; the hard part is whether it retrieves the fact that is valid for this question, in this context, right now. Existing evaluations mostly score the final answer or check that newer facts replace older ones, which leaves the ranking behavior in between almost unmeasured.

What the paper does

MemConflict is a diagnostic framework that treats memory validity as a query-conditioned fitness-for-use problem. It formalizes three conflict types, dynamic, static and conditional, across three validity dimensions: temporal validity, factual correctness and contextual applicability. The benchmark simulates controlled long-horizon histories from structured user profiles, introduces cross-session conflicts, and injects semantically similar distractors so that memory candidates compete with each other. Evaluation runs at two levels: black-box scoring of final answers, and white-box analysis of how supporting memories were retrieved and ranked, exercised on six representative long-term memory systems.

Findings

  1. Current evaluations skip the ranking question — they mainly assess outcome-level performance or temporal updating, giving limited insight into how systems retrieve and rank valid, correct, applicable evidence when alternatives conflict.

existing evaluations mainly assess outcome-level performance or temporal updating, providing limited insight into how systems retrieve and rank temporally valid, factually correct, and contextually applicable memory evidence under conflicting alternatives

  1. Conflicts have a taxonomy — dynamic, static and conditional conflicts are defined over temporal validity, factual correctness and contextual applicability, so “the memory was wrong” turns into a diagnosable failure type instead of a shrug.

MemConflict formalizes dynamic, static, and conditional conflicts over temporal validity, factual correctness, and contextual applicability

  1. The histories are controlled, not captured — structured profiles generate long-horizon histories with injected cross-session conflicts and semantically similar distractors, which makes each failure reproducible on demand.

It simulates controlled long-horizon histories from structured user profiles, introduces cross-session conflicts, and injects semantically similar distractors

  1. Six systems disagree with themselves — strengths are uneven across conflict types, and answer correctness often diverges from memory retrieval and ranking quality, so a right answer does not prove right memory behavior.

Experiments on six representative long-term memory systems show uneven strengths across conflict types, with answer correctness often diverging from memory retrieval and ranking

Caveat

Histories are simulated from structured user profiles rather than captured from real users, and the study measures diagnostics rather than deployed outcomes, so treat it as a failure map, not a leaderboard of production systems.

What this means for 5ac

SMB data contradicts itself constantly: old and new prices, two contacts at one client, a policy that changed mid-conversation. A memory that ranks well can still answer wrong, and a correct final answer can hide bad retrieval. The sensitivity results are the practical part: longer histories, injected distractors, implicit queries and larger conflict distances all degrade performance, which means the failure worsens as an account ages. For a 5ac build that argues for explicit validity handling on the write path, timestamps and supersession on every fact, instead of trusting the retriever to sort conflicts out at read time, plus a test pass with distractor-heavy histories before the memory meets a customer.

Source