Introduction
A growing body of research seeks to improve frozen large language models (LLMs) as agents by evolving their harness—the textual scaffolding that surrounds the model, encompassing elements such as persona, task strategy, format rules, and control heuristics. Existing reflective prompt-evolution methods typically treat this harness as a single flat string, optimizing it holistically. However, this approach overlooks the possibility that different components of the harness contribute unequally to overall agent performance. In this work, we ask a fundamental question: where does the optimization value truly reside?
To address this, we introduce HARNESSEVO, a framework that decomposes the harness into four independently evolvable slots: role, task-strategy, tool/format-rules, and reflection/control. Using a shared reflective optimizer under an iso-budget constraint, we pair this decomposition with leave-one-in and leave-one-out attribution analyses to systematically quantify the contribution of each slot to agent success.
Experimental Setup
We evaluated HARNESSEVO on ALFWorld using a frozen 7B-backbone LLM. Performance was measured under three conditions: (1) a stock (non-evolved) harness, (2) flat-string evolution as a baseline, and (3) slot-decomposed evolution via HARNESSEVO. All experiments adhered to an equal computational budget to ensure fair comparison.
Results
Overall Success Rates
HARNESSEVO did not significantly outperform the baselines on the overall binary success rate: it achieved 0.657 versus 0.642 for both the stock harness and flat-string evolution. At first glance, this suggests that harness decomposition offers little benefit. However, a deeper slot-level analysis reveals a more nuanced picture.
Localized Gains
Nearly all useful optimization value was concentrated in the reflection/control slot, which exhibited a leave-one-in gain of +0.119. The other three slots—role, task-strategy, and tool/format-rules—were individually null, contributing no measurable improvement when isolated.
The Budget-Splitting Trap
We identified a critical pitfall: uniform budget splitting is actively harmful. Allocating 64 rollouts across four slots leaves only 16 rollouts per slot, which falls below the optimizer's effective search floor. As a result, every slot freezes at its empty seed state, rendering the evolution ineffective. In contrast, concentrating the entire budget on the high-credit reflection/control slot fully recovers the lost gains, achieving a success rate of 0.761—a substantial improvement over the split-budget result—and does so using only half the total rollouts.
Task Contingency
The benefit of harness optimization is highly task-dependent. On the WebShop benchmark, all slots froze at empty seed states, and every method tied in performance. This outcome suggests that WebShop presents a genuine absence of recurrent, verbalizable control failures—not a budget-starved scenario—meaning that harness evolution has little to exploit regardless of allocation strategy.
Conclusion
Our findings challenge the prevailing assumption that harness optimization should treat the scaffolding as a monolithic string. Instead, we show that:
- Harness value is localized: meaningful gains are concentrated in specific slots, such as reflection/control, rather than distributed evenly.
- Uniform budget splitting can be counterproductive: it starves high-credit slots of sufficient search budget, leading to frozen, suboptimal outcomes.
- Credit assignment must precede structured evolution: identifying which components are worth optimizing—and allocating budget accordingly—is essential for effective self-evolving LLM agents.
As the field moves toward 2026 and beyond, with increasingly capable LLMs and more sophisticated agentic frameworks, these insights underscore the need for principled resource allocation in prompt evolution. Future work should explore adaptive budget allocation and richer slot taxonomies to unlock further gains.
Keywords: large language models; agent harness optimization; prompt evolution; budget allocation; slot-level attribution; task contingency
via ArXiv CL+LG