Generating changes to an agent harness is the easy part. Knowing which of those changes genuinely helped is the hard part — and it is the part most self-evolution work quietly skips. Our new paper, HarnessBank, is about closing that gap.

HarnessBank Research · 4 min read · Read the paper on arXiv

The measurement problem

An agent's performance is not governed by its foundation model alone. It is governed by everything wrapped around that model — the prompts, the knowledge it can reach, the runtime, the tools, the configuration. That wrapper is the agent harness, and evolving it automatically is one of the most promising paths to a better agent.

The catch is verification. Ask an LLM to propose harness improvements and you will get plenty of them. Ask whether they worked and the honest answer is usually unclear, because a candidate that scores higher on a training split may have done so for at least three reasons that have nothing to do with genuine improvement:

It got lucky — the gain sits inside run-to-run noise. It overfit — the gain is real on the tasks you searched against and evaporates everywhere else. Or it never activated at all — the change was written into the harness but no code path ever executed it, and the score moved for unrelated reasons.

Existing approaches tend to compound the problem by leaning on greedy candidate selection and noisy self-generated feedback: the same model that proposes a change also grades it. That makes gains fragile — vulnerable to search collapse, task-specific overfitting, and poor verifiability.

Split the roles: LLMs propose, deterministic code credits

HarnessBank's central design decision is a division of labour. LLMs do the open-ended work; deterministic code owns the credit.

A dedicated evolver agent, kept separate from the task agent it is improving, handles what language models are genuinely good at: reading failure traces, diagnosing what went wrong, and proposing concrete changes across all five surfaces of the harness — prompts, knowledge, runtime, tools, and configuration.

What the evolver does not get to do is decide whether its own proposals worked. That judgment is handed to a fixed, deterministic gate with four stages:

Validity checks confirm the candidate harness is well-formed and runnable at all. Activation checks confirm the change actually executed — closing off the largest and least-discussed source of phantom gains. Paired significance testing compares candidate against parent on the same tasks, so a difference has to clear statistical noise rather than merely exceed it. And a sealed test, never touched during search, provides the final number.

Because no LLM sits anywhere in that chain, credit cannot be talked into existence.

A gene bank instead of a greedy climb

Gating alone would make search expensive and narrow. HarnessBank pairs it with a Harness Gene Bank: a population of high-performing harnesses held at different semantic coordinates rather than a single best-so-far champion. Candidates are reinvented, recombined, screened, and selected from that bank across evolution rounds, which keeps diversity alive where a greedy hill-climb would collapse onto one lineage.

Screening the resulting offspring is the cost bottleneck, so a Gated Harness Screening mechanism filters aggressively and early — spending full evaluation budget only on candidates that have already earned it.

What the numbers show

We evolved harnesses across seven agent domains. On the six sealed tests that earned statistical credit, the evolved harnesses gained +9.0 to +15.5 percentage points — and retained 86–147% of the gain measured during training.

That retention figure is the one we would point to first. A number near or above 100% means the improvement was a property of the harness rather than an artifact of the search: it survived contact with data the search never saw. Overfitting shows up as retention collapsing toward zero, and that is not what we observed.

It is worth being precise about the arithmetic: six of seven domains earned credit, not seven of seven. One did not clear the gate, and under this framework it therefore gets reported as no credited gain rather than folded into an average. A measurement discipline that can never return "no" is not measuring anything — so we would rather publish the six than round up to seven.

What actually transfers

The most useful negative result in the paper is about transfer. Cross-model experiments confirm the improvements come from the model-specific self-evolving process — not from discovering some universally optimal harness that could be lifted from one model and dropped onto another.

In other words: there is no single best harness waiting to be found, and searching for one is the wrong goal. What generalizes is the diagnose-and-credit loop — the procedure of finding failures, proposing targeted changes, and refusing to bank a gain until deterministic checks have signed off on it. Point that loop at a new model or a new domain and it will build the harness that model needs. That is the transferable artifact.

For teams building on agent harnesses, the practical takeaway is narrower than "use our framework" and more demanding than it first sounds: make something other than your generator responsible for deciding whether a change helped, and make activation something you verify rather than assume.

Read the paper

HarnessBank: Semantic Gene-Bank Search with Gated Verification for Agent-Harness Self-Evolution — Xiaotian Luo, Dizhan Xue, Fengxingyu Wang, Chuanrui Hu, Yafeng Deng. Available on arXiv. Code will be publicly available upon acceptance.

Reported gains are measured on held-out sealed tests that were never used during harness search, with candidate-versus-parent comparisons evaluated by paired significance testing; domains that did not clear the significance gate are reported as uncredited rather than averaged in.