Build or Adopt? Custom-Built vs. General-Purpose LLM Agent Harnesses for Automotive GUI Testing

“As general-purpose LLM agent harnesses mature, organizations must decide whether to continue developing in-house systems or adopt existing alternatives.”

Arsham Gholamzadeh Khoee, Ruyu Fu, Jiaqi Wei +4 · 7 authors · Chalmers University of Technology and Volvo Group · 2026 · arXiv:2610.33636

Keep the in-house harness or adopt a general one: Volvo ran both

Opening

Every team that uses agents long enough reaches the fork: keep developing an in-house harness, or adopt a general-purpose one, and both sides have real reasons. The question usually gets answered by gut feelings about license cost and control, while the real cost sits where nobody counts. This paper answers it the honest way: by running both sides on the same test cases, the same models and the same set of tools, inside a company that ships real software.

What the paper does

The authors compare Volvo Group’s production, custom-built harness with two general-purpose harnesses, Claude Code and OpenCode, keeping the tools, accessed through the Model Context Protocol, and the models identical. They run up to four models on Volvo’s stepwise IVI test cases for Android-based in-vehicle infotainment, and on open-ended tasks in public apps, and interview ten practitioners: developers, the product owner and testers. The comparison separates success rate, inference cost per task, and total monthly cost including engineering effort, instead of looking at one number alone. On the stepwise IVI tests the general-purpose harnesses match the custom-built harness in success but vary more across runs; on the open-ended tasks they succeed more often, mainly because they recover better after an incorrect action.

Findings

  1. A production custom-built harness is compared against Claude Code and OpenCode with the same MCP tools and the same models. (We compare a production, custom-built harness with two general-purpose harnesses, Claude Code and OpenCode, keeping the tools, accessed through the Model Context Protocol (MCP), and the models the same)

  2. Inference per task costs several times more, but the engineering effort saved outweighs it: for a nightly IVI regression suite the total monthly cost runs 3-66% lower. (Their inference cost per task is 2.6-10.7 times higher, yet the engineering effort they save outweighs this: for a nightly IVI regression suite, total monthly cost, including engineering effort and inference, is 3-66% lower)

  3. Progress made outside the team can turn an in-house harness into technical debt even while the harness itself never deteriorates. (Interviews suggest that external advances can turn an in-house harness into technical debt even without deterioration)

Caveat

The tradeoff the authors state: adopting a general-purpose harness shifts maintenance from the agent loop to tools and instructions, but it gives the team less control over what the model receives. That is why the testers in the interviews still want to inspect the agent’s actions and make sure it follows the prescribed test scenario, regardless of which harness runs the loop.

What this means for 5ac

5ac stands at the same fork. Keeping the agent loop and the skills in-house controls everything the model sees, but the team must maintain all of it alone; adopting a ready harness like OpenCode moves the maintenance burden onto tools and instructions. The decision that fits this stack is to keep Hermes as the primary loop, because it controls the prompts and the gates, and to use OpenCode for the coding lane, where looser control is acceptable. Volvo’s lesson is not to compare single inference costs: the number worth comparing is total monthly cost including human effort, and every lane running an adopted harness must keep its own verification path, meaning a Kanban card that records each agent action so a reviewer can inspect it instead of only receiving the final output. When the in-house harness stops keeping pace with external progress is the moment to revisit the choice, not when it breaks.

Source