EVOHARNESSBENCH: Can Your Agents Keep Pace with an Evolving Harness?

On September 7, 2026, the researchers released EVOHARNESSBENCH, a benchmark used to evaluate the controlled evolution of agents in three dimensions: tools, skills, and expert agents. This benchmark includes 17 multi-stage flows, covering 802 tasks, 520 types of tools, 42 skills, and 62 agents. Non-stationarity is considered within the externally provided harness itself, rather than within the task flows. The study evaluated two complementary settings: deployment evaluation (retention of isolation capabilities) and self-evolution adaptation evaluation (testing the utility of experience). The results revealed three persistent challenges: first, harness expansion can lead to a decline in task performance for previously solved tasks, resulting in harness-induced forgetting; second, the benefits of self-evolution adaptation are inconsistent across stages, abilities, and environments; third, retaining early abilities often conflicts with adapting to new abilities. These findings highlight the unique challenges posed by harness evolution in building agents that can evolve and maintain effective behavior.