Computer Science > Software Engineering
Title:Does Your Agent Need to Know Its Budget? A Placebo-Controlled Experiment on LLM Coding Agents
Abstract:Organizations deploying autonomous coding agents impose token budgets, yet the causal effects of a budget, and of telling the agent about it, are unmeasured. We conduct a placebo-controlled factorial experiment under a frozen, timestamped protocol, crossing budget enforcement with budget disclosure on two executable program-repair benchmarks (5,835 runs, two open models, three cap levels). Enforcing a calibrated cap reduced task resolution by 20.0 percentage points while reducing tokens per resolved task roughly twelvefold. Disclosing the enforced budget never helped: the placebo-controlled risk difference was -2.8 pp (95% CI [-6.2, +0.7]) at the medium cap and -4.2 pp ([-8.1, -0.2], nominally significant, unadjusted) at the tight cap, with every prompt wording and per-turn feedback variant far below the 10-point effect size of interest. Observed run-to-run outcome variance was roughly 1.5x the uncapped level under caps, though part of this reflects resolution rates moving nearer one half. These results make budget governance for coding agents an enforcement-design problem: no communication variant produced an effect beyond that of an unrelated sentence, which itself raised resolution by 7.2 pp (95% CI [+4.0, +10.5]) over silent enforcement, a sentence effect that only a placebo control removes.
| Comments: | 11 pages, 2 figures, 3 tables. Study archive (frozen protocol, per-run logs, and analysis code) to be deposited in a DOI-issuing repository |
|---|---|
| Subjects: | Software Engineering (cs.SE) |
| ACM classes: | I.2.2; D.2.5 |
| Cite as: | arXiv:2611.21614 [cs.SE] |
| (or arXiv:2611.21614v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2611.21614 arXiv-issued DOI via DataCite |
Submission history
Access Paper:
- HTML (experimental)
Additional Features
- Audio Summary
Current browse context:
References & Citations
- Google Scholar
- Semantic Scholar
BibTeX formatted citation
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.