Is your memory bill about what you keep, or about how often you write it?

Dongfang Li et al. · 1 authors · Harbin Institute of Technology, Shenzhen · 2026 · arXiv:2608.12990

Is your memory bill about what you keep, or about how often you write it?

Opening

Ask why agent memory is expensive and the answer you get is usually about retrieval: bigger contexts, more hops, more candidates. Look at the write side instead and a different number shows up. Eager memory systems call the model after every interaction to extract, summarize, or update what should be remembered, so the construction bill grows with every exchange in the conversation. The two known shortcuts are unattractive: summarize coarsely and you throw away fine-grained evidence, or push the overhead to query time with larger retrieval contexts and multi-hop reasoning. This paper attacks the variable nobody quotes first, the granularity at which memory is consolidated.

What the paper does

LycheeMemory V2 replaces turn-level consolidation with semantic segment-level consolidation. Instead of consolidating every interaction, it batches multiple exchanges into segments and encodes each finalized segment into context-independent typed memory records, with semantic boundary detection meant to preserve coherent event-level and temporal evidence compared with fixed-window batching. The resulting records are organized with lightweight structured indexes for query-planned evidence retrieval, and the experiments use GPT-4.1-Mini on LoCoMo and LongMemEval-S.

Findings

  1. Eager consolidation makes the bill grow with the chat — calling the model after every interaction to extract, summarize, or update memories means construction cost rises as conversations get longer.

invoking LLMs after each interaction to extract, summarize, or update memories

  1. The cheap shortcut loses evidence — coarse summarization cuts construction cost but risks discarding fine-grained contextual evidence, which is exactly what later questions turn out to need.

Coarse summarization can reduce construction cost but risks discarding fine-grained contextual evidence

  1. Batch at semantic boundaries instead — exchanges are grouped into segments, and only finalized segments are encoded into context-independent typed memory records, so the model is called less often per conversation.

LycheeMemory batches multiple exchanges into segments and encodes each finalized segment into context-independent typed memory records

  1. The savings are large and stay off the query path — against A-Mem, construction tokens drop by 86.0% on LoCoMo and 75.9% on LongMemEval-S without increasing query-time token usage.

it reduces construction tokens by 86.0% on LoCoMo and 75.9% on LongMemEval-S without increasing query-time token usage

Caveat

All numbers come from GPT-4.1-Mini on two conversational benchmarks, so the segment boundaries and the token savings are demonstrated for one model and one task shape only.

What this means for 5ac

If a 5ac agent writes memory continuously, the write frequency is a lever you can pull without touching the product: stop calling the model after every exchange, batch exchanges into coherent segments, and encode only segments that have actually finished. The saving does not come back as extra query cost, because query-time token usage stays flat, which means the freed budget can go to the answer itself. Two guardrails from the paper keep this honest. Boundaries have to be semantic rather than fixed windows, or fine evidence disappears the way it does under coarse summarization, and every number here belongs to one model on two benchmarks, so the batching rule has to be re-measured on the model 5ac actually runs.

Source