Computer Science > Software Engineering
Title:Decided or Open? Comparing Program Repair Techniques When Patch Correctness Is Unknown
Abstract:Comparisons of automated program repair (APR) techniques depend on knowing which patches are correct, but many patches are never checked and some archived test results are wrong. We propose reporting each comparison as decided or open. For two techniques that each pick one patch per bug, we compute the full range of the difference in their success rates over every possible value of the unknown labels, and we call the comparison decided when that range excludes zero. Because a test can only show that a patch is wrong, more testing can move this range in one direction only. This lets us tell, before any test runs, which open comparisons testing could never settle. We study eight settings, including 55,466 candidate patches for Defects4J bugs. The labels already in the archive decide 12 of 16 comparisons between a learned patch selector and its baselines, and adding further evidence decides all 16. In a campaign whose predictions we published in advance, all eight predictions held. We also show that copies of the developer's fix, which must pass, are recorded as failing for 33.5% of GPT-4's copies but for only 2.5% of the RepairLLaMA model's, so the reliability of archived test results differs between models. With these checks, researchers can see which comparisons need more evidence and which kind of evidence can help.
| Comments: | 40 pages. Under review at Empirical Software Engineering. Also available at Research Square, doi: https://doi.org/10.21203/rs . this http URL -11480796/v1. Replication package: doi: https://doi.org/10.5281/zenodo.23590962 |
|---|---|
| Subjects: | Software Engineering (cs.SE) |
| Cite as: | arXiv:2611.23287 [cs.SE] |
| (or arXiv:2611.23287v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23287 arXiv-issued DOI via DataCite |
Submission history
Access Paper:
- HTML (experimental)
Additional Features
- Audio Summary
Current browse context:
References & Citations
- Google Scholar
- Semantic Scholar
BibTeX formatted citation
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.