Multi-turn orchestration matches the best single model, and the voting UI changes the answer
Does asking four models to agree beat asking the best one?
Opening
The seductive promise of orchestration is that a committee of frontier models will out-think any single member. This paper puts that promise to the test directly: four strong LLMs propose answers and vote over multiple turns until consensus, on three established benchmarks, against single-model baselines. The headline result is sobering, the committee matches but does not clearly beat its strongest member. The more useful finding is about the interface: showing who authored an answer, and showing the live tally, quietly changes what the committee decides. Protocol details carry as much weight as model choice, and most teams ship them without a second thought.
What the paper does
The study defines multi-turn multi-agent orchestration, where multiple LLM agents interact over multiple turns by iteratively proposing answers or casting votes until reaching consensus. Two experiments run with four LLMs, Gemini 2.5 Pro, GPT-5, Grok 4, and Claude Sonnet 4, on GPQA-Diamond, IFEval, and MuSR. The first benchmarks orchestration against single-LLM baselines; the second, an ablation on GPQA-Diamond, varies whether agents see who authored answers and whether they can observe ongoing votes. That ablation turns the voting protocol itself into an independent variable instead of an implementation detail, which is what makes the study usable by anyone designing a review round rather than just interesting.
Findings
Ceiling at the strongest member — The committee reliably beats the weaker models but only matches or exceeds its best member, so orchestration lifts the floor more than it raises the ceiling. Its value is insurance against a weak member rather than a route past the strongest one.
Orchestration matches or exceeds the strongest single model and consistently outperforms the others.
Authorship bias — Show agents who wrote each answer and they start voting for their own, so self-voting and ties both climb once authorship is on the table. Identity leaks into the vote even when nothing else changes, which makes authorship visibility a protocol decision rather than a model property.
revealing authorship increases self-voting and ties
Visible votes herd — Streaming the tally speeds convergence but can lock the group in early, and faster agreement is not the same as better agreement. A panel that converges quickly can converge on the wrong answer, and the tally itself is what pushes it there.
showing ongoing votes amplifies herding, which speeds convergence but can sometimes yield premature consensus
Caveat
The authors state no limitation in the abstract. Everything reported comes from four named LLMs on three benchmarks, with the interface ablations run on GPQA-Diamond only, so the boundary of the evidence is exactly what the abstract describes and nothing beyond it is claimed.
What this means for 5ac
A 5ac review card that fans out to several profiles gets the floor lifted, weaker drafts get caught, but it should not be sold as beating its best single reviewer. The concrete change is in the protocol, not the roster: hide authorship in multi-agent review rounds so profiles stop self-voting, and decide before dispatch whether live vote tallies are visible, because showing them herds the panel toward premature consensus. When a quick correct answer matters more than agreement, cap the rounds and let the strongest model answer alone, since the paper shows the committee edge over it is thin.
Source
- Beyond the Strongest LLM: Multi-Turn Multi-Agent Orchestration vs. Single LLMs on Benchmarks — https://arxiv.org/abs/2509.23537