Same task, same models, different topology: over 30 points and double the time

Yanyu Ren, Yunfeng Bai, Xizheng Wang +2 · 5 authors · Tsinghua University and Zhongguancun Laboratory · 2026 · arXiv:2607.27877

Does the org chart change the build when the models stay the same?

Opening

Multi-agent vibe coding promises faster software development, and the benchmarks backing the promise habitually look away from the two things that decide whether it holds in practice: time and money. Existing evaluations rely on synthetic environments that ignore practical time and monetary costs, conflate reasoning with communication, and reward only superficial completion. A team can therefore win a benchmark while burning twice the clock and twice the tokens. MSEval was built to close that gap on real projects with real deployment, and it puts a number on how much the org chart itself is worth.

What the paper does

MSEval grounds evaluation in 10 authentic, full-stack projects across 10 domains, scoring performance using hierarchical requirements and deterministic rubrics. Its execution engine, LegoGent, tests 10 collaboration topologies where agents coordinate via periodic sync intervals and deploy through native CI/CD pipelines. A second component, the automated grader TAgent, dynamically probes implementations to jointly measure functional success, latency, and prefix-cached token cost. Across 100 runs the benchmark holds the task and the models fixed and varies only the organization, which is what lets the authors separate topology effects from capability effects.

Findings

Topology rivals capability — Who reports to whom moves the speed-cost-quality trade-off about as much as which model sits in the seat. That reframes topology from plumbing into a first-class design variable that deserves the same attention as model choice.

organizational topology rivals model capability in shaping the speed–cost–quality trade-off

Same inputs, over 30 points apart — With identical tasks and models, only the wiring changed, and scores shifted by over 30 points while wall-clock time doubled. Variation teams habitually blame on the model is often the org chart.

varying the topology shifts scores by over 30 points and doubles wall-clock time

Structure beats supervision — Structured pipelines converge fastest with the highest quality, while heavy managerial oversight degrades performance: adding a manager layer does not buy quality on these tasks, it costs it.

Structured pipelines converge fastest with the highest quality, whereas heavy managerial oversight degrades performance.

Caveat

The limitation the authors state belongs to the benchmarks they set out to replace: those rely on synthetic environments that ignore practical time and monetary costs, conflate reasoning with communication, and reward only superficial completion. MSEval answers with real projects and native CI/CD, but the stated gap is precisely what the new benchmark was designed to fix.

What this means for 5ac

The concrete warning is for pipeline design: every extra review layer 5ac adds between a draft and a merge is managerial oversight, and on MSEval’s evidence it degrades performance rather than improving it. The change to make is to prefer a structured pipeline — a fixed sequence of stages with clear handoffs — over a stack of concurrent reviewers debating one card. Cost per task should be measured the way TAgent measures it: functional success, latency, and token cost together, not accuracy alone. When two profiles can do the same job, the topology that reaches done in fewer sync intervals wins on all three axes at once.

Source