Can an agent read a document bigger than its window by paging memory like an OS?

Charles Packer et al. · 1 authors · University of California, Berkeley · 2023 · arXiv:2310.08560

Can an agent read a document bigger than its window by paging memory like an OS?

Opening

You hand your assistant a contract that is longer than the model’s context window, or a customer comes back several sessions later asking about something you both discussed, and the answer arrives as if the two of you had never met. The usual cure is a model with a bigger window, but a bigger window still does not decide which part of the pile deserves to sit inside it. The real constraint is not length. It is that nobody is choosing what enters the window and what gets pushed back out, and operating systems have managed exactly this problem with memory tiers for a long time.

What the paper does

The paper proposes virtual context management, a technique that borrows from hierarchical memory in traditional operating systems, where data moves between fast and slow memory so a small fast pool feels like a large one. On top of that technique sits MemGPT, a system that manages memory tiers inside the model’s limited window and uses interrupts to manage control flow between itself and the user. The evaluation runs in two domains where limited windows hurt the most: document analysis on inputs that far exceed the underlying model’s window, and multi-session chat.

Findings

  1. Context length is the binding constraint — long conversations and large documents fail because the window is small, not because users ask bad questions.

constrained by limited context windows, hindering their utility in tasks like extended conversations and document analysis

  1. The fix comes from operating-system design — instead of widening the window, the paper moves data between a fast tier and a slow tier behind one small window that never grows.

we propose virtual context management, a technique drawing inspiration from hierarchical memory systems in traditional operating systems

  1. The agent itself chooses what to page in — deciding which tier to read or write sits with the agent rather than with a fixed retrieval function, and interrupts carry control flow back and forth with the user.

intelligently manages different memory tiers in order to effectively provide extended context within the LLM’s limited context window

  1. Two evaluation domains set the scope — one is document analysis on inputs far larger than the window, the other is chat where the agent remembers, reflects, and changes over long interactions.

multi-session chat, where MemGPT can create conversational agents that remember, reflect, and evolve dynamically through long-term interactions with their users

Caveat

The evaluation covers only document analysis and multi-session chat, with no token or dollar accounting anywhere, so it demonstrates that the mechanism runs in two domains rather than that the mechanism pays for itself once real usage is billed.

What this means for 5ac

If a 5ac assistant has to keep working across many sessions or read documents larger than the window, this is the template to copy: split memory into tiers, keep the live conversation in the fast tier, park documents and old notes in the slow tier, and let the agent decide what gets paged into the window. What the paper deliberately skips is the bill, so any build that follows this architecture has to ship with its own token and cost metering attached. Without that meter you will know the mechanism runs, and you will not know whether it was worth it.

Source