A portable specification for the agent harness: which model ran inside which harness

“Large language models act only through the infrastructure that surrounds them.”

Nicolas Gertler, Claudio Novelli, Luciano Floridi · 3 authors · Yale University (Digital Ethics Center) and University of Bologna · 2026 · arXiv:2610.40917

Benchmark results change with the harness, yet nobody records which harness

Opening

Same model, a different way of building the tool loop, a different system prompt, a different runtime configuration, and the reported number moves in the direction that favors whoever measured it. There is still no standard place to record that configuration, so a reader of a report cannot tell what the model actually ran inside. This paper starts from exactly that gap.

What the paper does

The authors argue that a language model acts only through the infrastructure surrounding it, and that infrastructure, the harness, decides what the model can accomplish and how it fails, while the term itself has no settled definition. To answer that, they break the harness into eight functional components that can be specified and compared across systems, then propose a portable specification framework that records which model ran in which harness. Three reasons are given for keeping that record: different harness configurations can produce several-fold differences in reported scores for the same model, much of an agent’s attack surface lies in its harness, and existing model cards and agent cards do not establish the sequence of effective configurations within a run. A harness can change between releases and between runs, and, where agents edit their own tools and configuration, within a run as well.

Findings

  1. A model acts only through the infrastructure around it, and that infrastructure, the harness, decides what it can accomplish and how it fails, yet the term has no settled definition. (Large language models act only through the infrastructure that surrounds them. That infrastructure, the harness, determines what a model can accomplish and how it fails, yet it has no settled definition)

  2. The harness is decomposed into eight functional components that can be specified and compared across systems, instead of each team reading the word its own way. (we decompose the harness into eight functional components that can be specified and compared across systems)

  3. Different harness configurations can produce several-fold differences in the reported scores of the same model, which means a model card cannot stand on its own. (different harness configurations can yield several-fold differences in reported scores for the same model)

Caveat

The abstract does not state a limitation of the paper; the nearest sentence belongs to the gap-statement kind, describing what current tooling still lacks: model cards and agent cards provide release and operational information but do not establish the sequence of effective configurations within a run. The authors use that sentence as motivation for the versioned harness profile they propose, so it has to be read as a gap in the ecosystem rather than a weakness of the study itself.

What this means for 5ac

This is what 5ac has to settle for G-Company OS: every agent version must record its harness version so that results can be reproduced. In practice the pieces of that record already exist in several places: the Kanban card stores the model and run configuration for each run, skills and cron jobs define the loop around the model, and Gbrain keeps the context of earlier runs. The work is to gather them into one record per version instead of leaving them scattered across files. When a card’s result changes after a skill upgrade or a switch in model routing, that record is what shows what actually changed. It also answers the question that comes up whenever a new model goes into production: is this change in the model or in the harness, and if it is in the harness, who is holding that configuration.

Source