Computer Science > Software Engineering
Title:We Fixed the Scorer First, and Cluster Correction Killed Our Own Planted-Defect Result in AI Code Review
Abstract:Judging an AI code-review pipeline only by how much it catches hides an axis: what it dropped. Stronger selection raises precision for free; the cost never reaches the report. We ran four arms on the same eight cases, one scorer, asking whether planted-defect detection can rise without raising the claim error rate. Three repetitions are nested in eight cases, so p is reported per contrast and twice: under an exact permutation test over cases (2^8 arrangements) the two contrasts publishable under independence (Fisher p=0.017, 0.019) move to 0.0625 and 0.109, and after Holm correction no contrast separates. This is "not observable with this design," not "no effect": the sign test needs six discordant cases in one direction, its realized type-I rate under the null is 0.0078, and 80% power needs a per-case win probability near 0.80; closing it takes 12-25 cases, not repetitions. Within a run, a second pass never removes what the first found (0/24, 0/24 paired losses) while its gains (+3, +2) fall in one case; baselines on byte-identical prompts differed 15/24 vs 20/24, so run-to-run variance exceeds the effect. The claim error rate is 0.0% (0/178), with no upper bound on the rate claimed; scorer discrimination was measured with pre-registered positive controls (12/12 fabricated claims rejected, 0 false rejections). We retract our original causal explanation and report the instrument defects we hit. The contributions are not the numbers: (1) fixing the instrument before seeing results, then killing our own main effect with it; (2) the identity eps-hat = eps-hat(pi): a reported error rate is computed only on the subset a retention policy kept, silent about what was dropped, so it does not identify the quantity an irreversible gate needs; no improvement in the number promotes an automated verdict to publish, delete, or history-rewrite surfaces; a mechanized refusal enforces this.
| Subjects: | Software Engineering (cs.SE) |
|---|---|
| Cite as: | arXiv:2611.21635 [cs.SE] |
| (or arXiv:2611.21635v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2611.21635 arXiv-issued DOI via DataCite (pending registration) |
Submission history
Access Paper:
Additional Features
- Audio Summary
Current browse context:
References & Citations
- Google Scholar
- Semantic Scholar
BibTeX formatted citation
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.