Overview
An agent harness is the code surrounding a model: the execution loop, tools, context management, state handling, recovery logic, and verification. According to the Terminal-Bench 2.1 leaderboard, GPT-5 solves 35.2% of tasks inside Terminus 2 but 49.6% inside Codex CLI with identical model weights. Most benchmarks hold that harness fixed. HarnessDev, proposed by researchers from ByteDance Seed, Singapore University of Technology and Design, Georgia Institute of Technology, M-A-P, and TokenWave.AI, flips the target: the artifact under evaluation is the runnable harness the model writes, not the answers it produces.
Two Stages: Creation and Evolution
In Creation, every creator receives the same weak seed: passive file, search, and process primitives plus result and trajectory writers, with no loop, planner, verifier, retry logic, or stopping rule. Unmodified, it scores zero everywhere. The creator is given a task-family spec, a short design tutorial, and 1 to 3 development cases, then builds a full harness. That harness is frozen before any hidden tasks are revealed.
In Evolution, the creator starts from its own frozen Creation harness and revises it using execution feedback from a fixed set of 100 SWE-bench Pro tasks and all 89 Terminal-Bench 2.1 tasks. Each official candidate must complete both evaluations as a pair, with a budget of 10 pairs and at most 2 five-task probes between pairs. Every official version is later scored on 630 held-out SWE-Pro instances the creator never sees.
Harnesses are graded on capability (task success) and efficiency (executor tokens, with creator tokens excluded).
Experimental Setup
Six creator LLMs were tested: Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro, working inside Claude Code 2.1.177 (GPT-5.5 used Codex 0.144.3). Creation spans 4 domains and 5 benchmarks totaling 2,207 instances: SWE-bench Pro public split (731), Terminal-Bench 2.1 (89), MLE-bench (75), EQ-Bench3 (46), and BrowseComp (1,266). Each creator builds 3 harnesses per benchmark, reported as avg@3. Self-Eval runs each harness with its creator; Unified-Eval runs all with Gemini 3.1 Pro.
Creation Results
Harnesses generated by the creators substantially outperform the weak seed across every benchmark. Significantly, they also exceed the strong reference harnesses (Terminus 2 for Terminal-Bench, OpenHands for SWE-bench Pro) in several settings.
The generated harnesses vary widely in architecture. Common patterns include an outer ReAct-style loop, subagents for context isolation, file-oriented memory, and error-driven refinement. However, their efficiency profiles differ markedly. Gemini 3.1 Pro's harnesses are the most token-efficient, while GPT-5.5's harnesses consume far more tokens for comparable success rates.
Evolution Results: The Generalization Gap
Evolution provides a clear capability boost on in-distribution tasks, but the gains do not transfer cleanly. Of 64 total changes the creators made during evolution, only 34 improved performance on the held-out SWE-bench Pro instances. The remaining 30 changes either had no effect or actively hurt generalization, indicating that a substantial portion of self-directed harness edits are overfitting to the feedback signals rather than learning transferable engineering principles.
Successful, generalizable changes fell into recognizable categories: adding explicit verification steps, tightening context windows, introducing structured retries with error classification, and improving stopping criteria. Changes that failed to generalize often involved task-specific heuristics, overly aggressive context pruning, or new tools tuned to the characteristics of the feedback tasks.
Implications for Agent Engineering in 2026
As of 2026, the frontier of agent development has shifted from prompt engineering to harness engineering. HarnessDev demonstrates both the promise and the limits of automating that process. LLMs can write functional harnesses from intentionally impoverished seeds, and those harnesses can surpass human-designed references. But when asked to improve those harnesses through iterative feedback, current models generalize at only a ~53% rate (34 of 64 changes).
This has direct implications for production agent systems. Teams relying on self-improving agents should validate changes on truly held-out task distributions before deployment. The result also suggests a role for human-in-the-loop oversight: reviewing proposed harness modifications against generalizable design principles rather than accepting feedback-driven edits at face value.
Key Takeaways
- Harness matters more than model weights alone: identical weights can swing task success by 14+ points depending on the surrounding harness.
- LLMs can build functional harnesses from scratch: generated harnesses beat the weak seed everywhere and sometimes outperform strong human-designed references.
- Self-improvement is only partially reliable: only 34 of 64 evolutionary changes generalized to held-out tasks; the rest were noise or overfitting.
- Generalizable gains follow patterns: verification, context discipline, structured retries, and stopping criteria transfer; task-specific hacks do not.
- Validate on truly held-out data: feedback-driven edits require independent generalization checks before production deployment.
via MarkTechPost