Can you rank stored memories like search results and stop paying an LLM per query?

Xinyu Zhang et al. · 1 authors · Fudan University · 2026 · arXiv:2610.34673

Can you rank stored memories like search results and stop paying an LLM per query?

Opening

Every single request, a long-term memory agent has to decide which handful of stored notes will be allowed into the context window, and that decision largely decides whether the answer is right. It happens on every request, under a cost budget, while the store keeps growing, and multi-session questions make it harder because all of their evidence has to make it in. Meanwhile many memory systems build their notes and graphs with hundreds to thousands of LLM calls per history, each component designed on its own, so nobody can say which part is actually responsible for the accuracy.

What the paper does

The paper formalizes agent memory as budgeted slate recommendation, with the request as the user, stored memories as items, and the context window as a slate, which places memory systems inside one design space of recommender cascades and factorizes accuracy into the completeness each stage controls. Guided by that formalization, the authors propose AgentMemRec, which adapts candidate generation, learning to rank, and slate construction from recommender systems. The result selects memories without any LLM call and is evaluated on multi-session questions of LongMemEval and on LoCoMo.

Findings

  1. The per-request pick is the whole game — every request must choose the few stored memories that enter a limited context window, and that choice largely decides whether the agent answers correctly.

must choose, for every request, the few stored memories that enter a limited context window

  1. Memory selection is a recommendation problem — the request plays the user, the stored memories play the items, and the context window is the slate to be filled under budget.

we formalize agent memory as budgeted slate recommendation, with the request as the user, stored memories as items and the context window as a slate

  1. Selection can be LLM-free — the proposed system picks the memories that enter the window without any LLM call, which is where most read-time cost normally goes.

selects memories without any LLM call

  1. It matches the expensive reranker for less — AgentMemRec matches an LLM reranker that makes 64 calls per query while costing 3.2 times less to serve, and it beats A-Mem, HippoRAG 2, DynamicRAG, and T-Mem by 10 to 31 points of accuracy on LongMemEval.

It matches an LLM reranker that makes 64 calls per query at 3.2 times lower serving cost

Caveat

Gains are measured on two multi-session question-answering benchmarks, and the method assumes memories already exist, so first-pass construction cost and write-time quality stay out of scope.

What this means for 5ac

If a 5ac agent reads memories on every request, the recurring bill is the read-time LLM call that ranks candidates, and this is direct evidence that a conventional ranker can do that job with zero LLM calls per query while matching a 64-call reranker. The practical move is to move memory selection out of the model and into candidate generation plus learning to rank, then spend the saved budget on better notes rather than on more ranking calls. The scope limit matters just as much: this paper starts from a store that already exists, so construction and write cost still have to be metered separately.

Source