Reflexion: an agent that learns from verbal feedback instead of weight updates

“Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents.”

Noah Shinn, Federico Cassano, Edward Berman +3 · 6 authors · Northeastern University · 2023 · arXiv:2303.11366

The agent got it wrong: keep the lesson in words, not in retraining

Opening

Every failed attempt in a real environment costs money in model calls, and for most agents today the second attempt is no smarter than the first: whatever the first attempt learned disappears when the session ends. The traditional reinforcement-learning answer is to collect more training samples and fine-tune the weights, a process that is slow and expensive, and that not every team has the data or the compute for. The practical question for anyone running an agent on their own server is simpler: how do you make each attempt leave a lesson behind without touching the model?

What the paper does

Reflexion lets the agent reflect in words after each attempt. It receives a feedback signal, either a scalar value or a written comment, and writes its own reflection into an episodic memory buffer. On the next attempt that note is read back and used to make better decisions, while the model’s weights do not change at all. The framework is flexible about where feedback comes from, external or internally simulated, and about its form, scalar values or free-form language. The authors try it on sequential decision-making, coding, and language reasoning, and run ablations to see how each kind of feedback moves the result.

Findings

  1. Reflexion reinforces the agent with linguistic feedback instead of weight updates: each attempt, the agent comments on itself in text rather than waiting for another training round, so improvement happens while the run is still going. (We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback.)

  2. The reflection is kept in an episodic memory buffer and read back on later attempts, so a failed attempt becomes context for the next one instead of vanishing with the session. (maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials)

  3. On HumanEval, Reflexion reaches 91% pass@1 against the 80% of the previous state of the art, GPT-4; that number comes from verbal reflection, not from fine-tuning the model. (Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%)

Caveat

The authors state no limitation in the abstract. The summary reports results and says it ran ablation and analysis studies, but it does not say where Reflexion is weaker, whether internally simulated feedback is a real substitute for feedback from the environment, or what happens when the reflective memory grows without bound. Those boundaries live in the full paper, not in the abstract, and an abstract that states no limitation is not evidence that none exist.

What this means for 5ac

Reflexion’s reflective memory is Gbrain and Hermes memory: the agent writes lessons down between attempts without fine-tuning, and the next run reads those notes back. On the Kanban board, a failed card gets a comment naming the cause and is retried, and that comment is exactly the linguistic feedback Reflexion talks about, with the retry acting as a small reflection loop. Hermes skills follow the same logic: when a skill is patched in after evidence that it works, every later run carries the lesson without any change to the model. The whole mechanism lives in the harness layer, so 5ac can change models at the model-routing layer without losing what has been accumulated.

Source