Newshot

Meta: A Preliminary Analysis of the Impact of AI Asst'd/Gen'd Code [..] at Scale

Original text(via HN AI)
  • Rui Abreu, Vijayaraghavan Murali, Peter C. Rigby, Chandra Maddila, Weiyan Sun, Jun Ge, Kaavya Chinniah, Audris Mockus, Megh Mehta, and Nachiappan Nagappan. Moving faster and reducing risk: Using llms in release deployment. arXiv preprint, abs/2410.06351, 2024. URL: https://doi.org/10.48550/arXiv.2410.06351.
  • Amazon. At amazon, some coders say their jobs have begun to resemble warehouse work, 2025. URL: https://www.msn.com/en-in/technology/artificial-intelligence/at-amazon-some-coders-say-their-jobs-have-begun-to-resemble-warehouse-work/ar-AA1FrEb4.
  • Peter C. Rigby Andrey Krutauz, Tapajit Dey and Audris Mockus. Do code review measures explain the incidence of post-release defects? International Journal of Empirical Software Engineering, 25:3323-3356, 2020. URL: https://doi.org/10.1007/S10664-020-09837-4.
  • Joel Becker, Nate Rush, Elizabeth Barnes, and David Rein. Measuring the impact of early-2025 ai on experienced open-source developer productivity, 2025. URL: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf.
  • Christian Bird, Nachiappan Nagappan, Harald Gall, Brendan Murphy, and Premkumar Devanbu. Putting it all together: Using socio-technical networks to predict failures. In 2009 20th International Symposium on Software Reliability Engineering, pages 109-119. IEEE, 2009. URL: https://doi.org/10.1109/ISSRE.2009.17.
  • Christian Bird, Nachiappan Nagappan, Brendan Murphy, Harald Gall, and Premkumar Devanbu. Don't touch my code! examining the effects of ownership on software quality. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, pages 4-14, 2011. URL: https://doi.org/10.1145/2025113.2025119.
  • B.W. Boehm. Software Engineering Economics. Prentice-Hall, 1981.
Google Scholar
Google Scholar
  • Marcelo Cataldo, Audris Mockus, Jeffrey A. Roberts, and James D. Herbsleb. Software dependencies, the structure of work dependencies and their impact on failures. IEEE Transactions on Software Engineering, 2009.
  • Zheyuan Kevin Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz. The effects of generative ai on high skilled work: Evidence from three field experiments with software developers. Available at SSRN 4945566, 2024.
  • Bates D and Maechler M. Package lme4, 2009. URL: http://lme4.r-forge.r-project.org.
  • Marco D'Ambros, Michele Lanza, and Romain Robbes. On the relationship between change coupling and software defects. In 2009 16th Working Conference on Reverse Engineering, pages 135-144. IEEE, 2009. URL: https://doi.org/10.1109/WCRE.2009.19.
  • Thomas Dohmke, Marco Iansiti, and Greg Richards. Sea change in software development: Economic and productivity analysis of the ai-powered developer lifecycle, 2023. URL: https://arxiv.org/abs/2306.15033.
  • Carsten F Dormann, Jane Elith, Sven Bacher, Carsten Buchmann, Gudrun Carl, Gabriel Carré, Jaime R García Marquéz, Bernd Gruber, Bruno Lafourcade, Pedro J Leitão, et al. Collinearity: a review of methods to deal with it and a simulation study evaluating their performance. Ecography, 36(1):27-46, 2013.
  • Omer Dunay, Daniel Cheng, Adam Tait, Parth Thakkar, Peter C Rigby, Andy Chiu, Imad Ahmad, Arun Ganesan, Chandra Maddila, Vijayaraghavan Murali, et al. Multi-line ai-assisted code authoring. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, pages 150-160, 2024.
  • M. E. Fagan. Design and Code Inspections to Reduce Errors in Program Development. IBM Systems Journal, 15(3):182-211, 1976. URL: https://doi.org/10.1147/SJ.153.0182.
  • David Firth. Bias reduction of maximum likelihood estimates. Biometrika, 80(1):27-38, 1993.
  • John Fox and Georges Monette. Generalized collinearity diagnostics. Journal of the American Statistical Association, 87(417):178-183, 1992.
  • John Fox and Sanford Weisberg. An R companion to applied regression. Sage publications, 2018.
  • Google Cloud. Devops research and assessment (DORA) report, 2024. URL: https://services.google.com/fh/files/misc/2024_final_dora_report.pdf.
  • T. L. Graves, A. F. Karr, J. S. Marron, and H. Siy. Predicting fault incidence using software change history. IEEE Transactions on Software Engineering, 26(2), 2000.
  • William Harding and Matthew Kloster. Coding on copilot: 2023 data suggests downward pressure on code quality. GitClear. Accessed: Mar, 1:58-99, 2024.
  • James Herbsleb and Audris Mockus. Formulation and preliminary test of an empirical theory of coordination in software engineering. In 2003 International Conference on Foundations of Software Engineering, Helsinki, Finland, October 2003. ACM Press. URL: http://dl.acm.org/authorize?787510.
  • Saki Imai. Is github copilot a substitute for human pair-programming? an empirical study. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, pages 319-321, 2022. URL: https://doi.org/10.1145/3510454.3522684.
  • Laura Inozemtseva and Reid Holmes. Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the 36th international conference on software engineering, pages 435-445, 2014. URL: https://doi.org/10.1145/2568225.2568271.
  • Nick Jiang, Anshul Ramachandran, Mehul Raheja, and Michael Li. The golden metrics: Characters per opportunity and percentage code written, 2024. URL: https://windsurf.com/blog/golden-metrics-characters-per-opportunity-percentage-code-written.
  • Yasutaka Kamei, Emad Shihab, Bram Adams, Ahmed E. Hassan, Audris Mockus, Anand Sinha, and Naoyasu Ubayashi. A large-scale empirical study of just-in-time quality assurance. IEEE Transactions on Software Engineering, 2013. URL: https://doi.org/10.1109/TSE.2012.70.
  • Leo Katz. A new status index derived from sociometric analysis. Psychometrika, 18(1):39-43, 1953.
  • Gary King and Langche Zeng. Logistic regression in rare events data. Political Analysis, 9(2):137-163, 2001.
  • Pavneet Singh Kochhar, D. Lo, Julia L. Lawall, and Nachiappan Nagappan. Code coverage and postrelease defects: A large-scale study on open source projects. IEEE Transactions on Reliability, 66:1213-1228, 2017. URL: https://doi.org/10.1109/TR.2017.2727062.
  • David Kreitmeir and Paul A. Raschky. The heterogeneous productivity effects of generative ai, 2024. URL: https://doi.org/10.48550/arXiv.2403.01964.
  • Andrey Krutauz, Tapajit Dey, Peter Rigby, and Audris Mockus. Do code review measures explain the incidence of post-release defects? case study replications and bayesian networks. In ESEC/FSE'20 Journal First Track, November 2020.
  • Matthew Li, Mehul Raheja, and Douglas Chen. Why you should not trust all the numbers you see, 2023. URL: https://windsurf.com/blog/code-llm-eval-why-you-should-not-trust-all-the-numbers-you-see.
  • Jenny T Liang, Chenyang Yang, and Brad A Myers. A large-scale survey on the usability of ai programming assistants: Successes and challenges. In Proceedings of the 46th IEEE/ACM international conference on software engineering, pages 1-13, 2024. URL: https://doi.org/10.1145/3597503.3608128.
  • Jinping Liu, Yuming Zhou, Yibiao Yang, Hongmin Lu, and Baowen Xu. Code churn: A neglected metric in effort-aware just-in-time defect prediction. In 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pages 11-19. IEEE, 2017. URL: https://doi.org/10.1109/ESEM.2017.8.
  • T. J. McCabe. A complexity measure. IEEE Transactions on Software Engineering, 2(4):308-320, December 1976. URL: https://doi.org/10.1109/TSE.1976.233837.
  • Daniel McFadden. Quantitative methods for analysing travel behaviour of individuals: some recent developments. In Behavioural travel modelling, pages 279-318. Routledge, 2021.
  • Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E Hassan. An empirical study of the impact of modern code review practices on software quality. Empirical Software Engineering, 21:2146-2189, 2016. URL: https://doi.org/10.1007/S10664-015-9381-9.
  • Andrew Meneely, Laurie Williams, Will Snipes, and Jason Osborne. Predicting failures with developer networks and social network analysis. In Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering, pages 13-23, 2008. URL: https://doi.org/10.1145/1453101.1453106.
  • Meta. Glean: Code indexing and search. https://glean.software/, 2025.
  • Audris Mockus. Software support tools and experimental work. In V Basili and et al, editors, Empirical Software Engineering Issues: Critical Assessments and Future Directions, volume LNCS 4336, pages 91-99. Springer, 2007. URL: https://doi.org/10.1007/978-3-540-71301-2_25.
  • Audris Mockus. Missing data in software engineering. In J. Singer et al., editor, Guide to Advanced Empirical Software Engineering, pages 185-200. Springer-Verlag, 2008. URL: https://doi.org/10.1007/978-1-84800-044-5_7.
  • Audris Mockus. Organizational volatility and its effects on software defects. In ACM SIGSOFT / FSE, pages 117-126, Santa Fe, New Mexico, november 7-11 2010. URL: https://doi.org/10.1145/1882291.1882311.
  • Audris Mockus. Engineering big data solutions. In ICSE'14 FOSE, 2014. URL: https://dl.acm.org/authorize?N14216.
  • Audris Mockus, Rui Abreu, Peter C. Rigby, David Amsallem, Parveen Bansal, Kaavya Chinniah, Brian Ellis, Peng Fan, Jun Ge, Bingjie He, Kelly Hirano, Sahil Kumar, Ajay Lingapuram, Andrew Loe, Megh Mehta, Venus Montes, Maher Saba, Gursharan Singh, Matt Steiner, Weiyan Sun, Siri Uppalapati, and Nachiappan Nagappan. Leveraging risk models to improve productivity for effective code un-freeze at scale. ACM Transactions on Software Engineering and Methodology, 2025. URL: https://api.semanticscholar.org/CorpusID:276950332.
  • Audris Mockus, Nachiappan Nagappan, and T Dinh-Trong, Trung. Test coverage and post-verification defects: A multiple case study. In International Conference on Empirical Software Engineering and Measurement, Lake Buena Vista, Florida USA, October 2009. ACM.
  • Audris Mockus and David M. Weiss. Predicting risk of software changes. Bell Labs Technical Journal, 5(2):169-180, april-june 2000. URL: https://doi.org/10.1002/BLTJ.2229.
  • Vijayaraghavan Murali, Chandra Maddila, Imad Ahmad, Michael Bolin, Daniel Cheng, Negar Ghorbani, Renuka Fernandez, Nachiappan Nagappan, and Peter C Rigby. Ai-assisted code authoring at scale: Fine-tuning, deploying, and mixed methods evaluation. Proceedings of the ACM on Software Engineering, 1(FSE):1066-1085, 2024. URL: https://doi.org/10.1145/3643774.
  • Nachiappan Nagappan and Thomas Ball. Use of relative code churn measures to predict system defect density. In Proceedings of the 27th international conference on Software engineering, pages 284-292, 2005. URL: https://doi.org/10.1145/1062455.1062514.
  • Ramzi W Nahhas. Introduction to Regression Methods for Public Health Using R. CRC Press, 2024.
  • Elise Paradis, Kate Grey, Quinn Madison, Daye Nam, Andrew Macvean, Vahid Meimand, Nan Zhang, Ben Ferrari-Church, and Satish Chandra. How much does ai impact development speed? an enterprise-based randomized controlled trial. arXiv preprint, 2024. URL: https://arxiv.org/abs/2410.12944.
  • Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. The impact of ai on developer productivity: Evidence from github copilot. arXiv preprint, 2023. URL: https://doi.org/10.48550/arXiv.2302.06590.
  • Adam Porter, Harvey Siy, Audris Mockus, and Lawrence Votta. Understanding the sources of variation in software inspections. ACM Transactions on Software Engineering and Methodology (TOSEM), 7(1):41-79, 1998. URL: https://doi.org/10.1145/268411.268421.
  • Liam Proven. Ai-generated bug reports are seriously annoying for developers, 2024. URL: https://www.theregister.com/2024/01/04/aiassisted_bug_reports_make_developers/.
  • Peter C Rigby, Daniel M German, Laura Cowen, and Margaret-Anne Storey. Peer review on open-source software projects: Parameters, statistical models, and theory. ACM Transactions on Software Engineering and Methodology (TOSEM), 23(4):35, 2014.
  • Dag I.K. Sjøberg, Bente Anda, and Audris Mockus. Questioning software maintenance metrics: a comparative case study. In Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement, ESEM'12, pages 107-110, New York, NY, USA, 2012. ACM. URL: https://doi.org/10.1145/2372251.2372269.
  • Microsoft: VS Code team. Vs code: Open source ai editor, 2025. URL: https://code.visualstudio.com/blogs/2025/05/19/openSourceAIEditor.
  • Tue Tjur. Coefficients of determination in logistic regression models—a new proposal: The coefficient of discrimination. The American Statistician, 63(4):366-372, 2009. URL: https://doi.org/10.1198/tast.2009.08210.
  • Uplevel. Can generative ai improve developer productivity?, 2025. URL: https://resources.uplevelteam.com/gen-ai-for-coding.
  • Thomas Weber, Maximilian Brandmaier, Albrecht Schmidt, and Sven Mayer. Significant productivity gains through programming with large language models. Proceedings of the ACM on Human-Computer Interaction, 8:1-29, 2024. URL: https://doi.org/10.1145/3661145.
  • Justin D. Weisz, Shraddha Kumar, Michael J. Muller, Karen-Ellen Browne, Arielle Goldberg, Katrin Ellice Heintze, and Shagun Bajpai. Examining the use and impact of an ai code assistant on developer productivity and experience in the enterprise. Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, 2025.
  • Elaine J Weyuker, Thomas J Ostrand, and Robert M Bell. Do too many cooks spoil the broth? using the number of developers to enhance defect prediction models. Empirical Software Engineering, 13:539-559, 2008. URL: https://doi.org/10.1007/S10664-008-9082-8.
  • Marcelo Serrano Zanetti, Ingo Scholtes, Claudio Juan Tessone, and Frank Schweitzer. Categorizing bugs with social networks: a case study on four open source software communities. In 2013 35th International Conference on Software Engineering (ICSE), pages 1032-1041. IEEE, 2013. URL: https://doi.org/10.1109/ICSE.2013.6606653.
  • Yunhua Zhao, Kostadin Damevski, and Hui Chen. A systematic survey of just-in-time software defect prediction. ACM Computing Surveys, 55(10):1-35, 2023. URL: https://doi.org/10.1145/3567550.
  • Minghui Zhou and Audris Mockus. Developer fluency: Achieving true mastery in software projects. In ACM SIGSOFT / FSE, pages 137-146, Santa Fe, New Mexico, november 7-11 2010. URL: https://doi.org/10.1145/1882291.1882313.
  • Albert Ziegler, Eirini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. Productivity assessment of neural code completion. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 21-29, 2022. URL: https://doi.org/10.1145/3520312.3534864.
  • Albert Ziegler, Eirini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. Measuring github copilot’s impact on productivity. Communications of the ACM, 67(3):54-63, 2024. URL: https://doi.org/10.1145/3633453.
  • Thomas Zimmermann and Nachiappan Nagappan. Predicting defects using network analysis on dependency graphs. In Proceedings of the 30th international conference on Software engineering, pages 531-540, 2008. URL: https://doi.org/10.1145/1368088.1368161.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by HN AI
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗