Newshot

One Code Change Lets Google AI Agent Boost Task Success Rate 8x After Harness Optimization

AI summary

A new Google research paper, "Harnesses That Improve Themselves," shows that letting an AI agent gradually modify its own runtime harness and validating each change can significantly raise software engineering task completion rates. In a SWE-bench Lite test, an agent using a modular optimization approach raised its task success rate from 5.0% to 40.0%, an 8x increase, at an average of about 46,700 tokens per task.

Translation(model translation, not the source's own text)

One Code Change: Google AI Agent Task Success Rate Up 8x After Harness Optimization

IT Home, November 20: A research paper titled Harnesses That Improve Themselves, newly published by Google researchers on November 17 local time, shows that by having an AI agent incrementally modify its own runtime framework and validate each modification, the completion rate of software engineering tasks can be significantly improved.

In a SWE-bench Lite test, the task success rate of an agent using the modular optimization approach rose from 5.0% to 40.0%, reaching 8 times the original, with an average of about 46,700 tokens consumed per task.

The research focuses on the AI agent's Harness. It is not the underlying AI model itself, but the runtime framework built around the model, responsible for managing context and memory, controlling multi-turn task execution, invoking external tools, and validating execution results before a task ends.

The researchers point out that different models may show notable differences in how they use tools and the weaknesses they exhibit during task execution. Therefore, a Harness optimized for one model, when transferred to another model, may lead to task stagnation, excessive token consumption, or degraded execution performance.

To solve this problem, the researchers propose a self-improving Harness architecture. Unlike the traditional approach of constraining agents by modifying prompts, this approach allows the optimizing model to directly modify the Harness code, and decides through testing whether to keep the modifications.

The experiment uses an initial programming agent Harness of only 142 lines of code, split into four editable modules, respectively responsible for context and state management, agent control, the tool invocation interface, and validation and failure recovery.

In each optimization round, the system allows modification of only one of the modules. The optimizing model first identifies problems based on the agent's execution records and proposes code modification plans, then checks the effect of the modifications through a validation set, while being subject to a token usage cap. Only modifications that pass validation are retained, and plans that fail to meet requirements are discarded.

In the core test, the researchers used Gemini 3.8 Flash as the base model for executing tasks, and evaluated the performance of different approaches on 40 SWE-bench Lite software engineering tasks that were not involved in optimization.

The results show that the initial Harness had a task success rate of 5.0%; with the modular modification and validation mechanism, the success rate rose to 40.0%, with an average of about 46,700 tokens consumed per task.

As a control, if the optimizing model is allowed to rewrite the Harness as a whole without restriction, the task success rate can reach 42.5%, slightly higher than the modular approach, but an average of about 451,600 tokens is needed per task, nearly 10 times that of the modular approach.

IT Home notes that another experiment constrained the agent only by modifying the prompt rule file, and the task success rate instead dropped from the initial 5.0% to 2.5%.

The research also examined the performance of the optimized Harness when transferred to other models. Without conducting a new optimization search, the modular approach, when transferred to Gemini 4 Argon, achieved a task success rate of 37.5%, while the whole-rewrite approach achieved 30.0%.

The researchers believe that the self-improvement of the Harness should not rely solely on prompts, but should implement important constraints in actual code. For example, by automatically testing and checking task execution results, identifying repeated failures, and preventing unvalidated task completion, agents can be made to complete complex tasks more reliably.

The paper also points out that allowing agents to modify their own runtime code brings new risks, including constantly bloating prompt content, over-optimizing for benchmarks, and tampering with evaluation programs. Therefore, code modifications need to be constrained by validation mechanisms and permission boundaries.

It should be noted that the experiments in this research mainly revolve around specific models and SWE-bench Lite software engineering tasks, and the core test set contains only 40 tasks that were not involved in optimization, which is not yet sufficient to prove that the method can achieve the same results across all models and agent application scenarios.

Related Articles

  • Despite strong opposition: Google will still globally roll out Android developer verification in January 2027, covering Google Play, Samsung, Honor, OPPO, vivo, Xiaomi, and Transsion
  • Apple is developing its own AI agent: able to control devices, browse the web, and write code, in a bid to compete with OpenAI and Anthropic
  • Google Gemini can now directly operate Apple Mac computers, enabling cross-app operations, creating PPTs, and filling out forms
  • Google partners with Blue Origin, plans to launch AI server satellites in 2029
  • Who is responsible when AI agents transfer money or make purchases without authorization? Multiple banks warn users that losses may have to be borne by themselves
  • Filling a years-long experience gap, Google will add an "App Lock" feature to native Android

The translation is machine-generated and individual terms may be off. Nothing is rewritten or cut; the original is authoritative.

Published by IT之家
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗