Scale the number of verifiers, not the size of the model
What if reliability came from more judges instead of a better model?
Opening
There are two ways to make an agent’s output more trustworthy: pay for a stronger model, or spend the same test-time budget checking the answer more times. Verifier-based checking is the second route, and it usually collapses into one of two shapes — sample repeatedly and hope, or trust a single reward model to grade. Both spend compute on one axis only. Neither spends it on breadth of judgment. This paper opens a third: scale the number of verifiers, have each check a different aspect of the output, and let the checks combine.
What the paper does
The authors propose Multi-Agent Verification (MAV), a test-time compute paradigm that combines multiple verifiers to improve performance, and instantiate it with Aspect Verifiers (AVs): off-the-shelf LLMs prompted to verify different aspects of outputs, chosen because they can be combined without additional training. Their algorithm, BoN-MAV, combines best-of-n sampling with multiple verifiers. The evaluation reports stronger scaling patterns than self-consistency and reward model verification, and demonstrates two regimes — weak-to-strong generalization, where combining weak verifiers improves even stronger LLMs, and self-improvement, where the same base model is used to both generate and verify outputs.
Findings
A new axis of test-time compute — Verifier count is presented as a scaling dimension in its own right, separate from sample count and separate from model size. Compute can be spent on judges, not only on generation. Sample count and model size were the familiar levers; this adds a third one.
a novel scaling dimension for test-time compute: scaling the number of verifiers
Best-of-n plus many verifiers beats the usual checks — Against self-consistency and reward model verification, the combined approach scales more strongly as compute grows. The gain comes from combining independent checks rather than trusting one check repeated.
BoN-MAV demonstrates stronger scaling patterns than self-consistency and reward model verification
Weak judges lift strong models — The most budget-relevant result: the verifier does not need to be stronger than the generator. Combining weak verifiers still improves even the stronger LLM being checked, and the same base model can be used to both generate and verify outputs.
combining weak verifiers improves even stronger LLMs
Caveat
The authors state no limitation in the abstract. That abstract also gives no domain list, no verifier counts, and no per-round cost, so the scaling claim should be read as a demonstrated direction rather than a rate a team can price a deployment against.
What this means for 5ac
For a customer on a fixed monthly token budget, MAV is the cheapest reliability upgrade available: keep the generator, add independent cheap checkers. On the Kanban board that translates into verification gates — a card moves to done only after two or three weak, differently prompted reviewer passes agree, instead of one expensive model’s single opinion. The weak-to-strong result also settles a routing question: reviewer profiles do not need the strongest model on the roster. Spend the strong model on generation, let several lighter profiles check, and put the gate only on high-stakes cards — money, production, external-facing — since every extra verifier costs tokens on every run.
Source
- Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifiers — https://arxiv.org/abs/2502.20379