This source only provides a headline — there is no body to show. Use “Read the original” below to read it at the source.
Programmable Distributions of Semantics-Preserving Variants for Evaluating Code Agents
arXiv:2608.18389v2 Announce Type: replace-cross Abstract: Code agents are increasingly deployed to resolve real software issues, yet their reliability under superficial code variations remains poorly understood. We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form. We sample semantically equivalent variants of a repository from an explicitly defined distribution and compare the agent's resolve rate on them with that on the original. Different distributions reveal different aspects of an agent, such as its sensitivity to routine refactoring or to misleading code near the location of a fix. We develop VDL, a probabilistic programming language for expressing such distributions, whose programs randomly apply semantics-preserving transformations (SPTs) to a repository. For our evaluation, we use it to define a generic distribution that is not targeted at any particular failure mod…