An LLM Workflow That Reproduces, Improves, and Extends Published Economics Research
We introduce an open-source workflow that enables an LLM to reproduce, improve, and extend an economics article using the article’s published replication package. First, the workflow attempts to reproduce the original calculations, checks for discrepancies with published findings, and performs automated sensitivity analysis. Across 4,452 published replication packages for five economics journals, the workflow flags discrepancies in 3,460 articles or their appendices. Second, the workflow improves the original calculations by using a different implementation or algorithm. In 496 articles, the workflow is able to reduce a calculation’s computation time, at similar or greater accuracy, by more than a factor of 10. Third, the workflow extends the original analysis. In 923 articles, the workflow develops an extension that does not appear in the original article and that is aligned with the original article’s goals and assumptions.
- LLMs were used in the analysis and writing of this article. For this project, Schwartz worked as a contractor for Anthropic; the results and views are not endorsed by Anthropic. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research. Jesse M. Shapiro Shapiro has, in the past, been a paid visitor at Microsoft Research New England and a paid consultant for FutureOfCapitalism, LLC. Shapiro has been paid for writing by the New York Times. Shapiro's spouse has a disclosure statement posted at https://vivo.brown.edu/display/eoster1#Affiliations.
LLMs were used in the analysis and writing of this article. For this project, Schwartz worked as a contractor for Anthropic; the results and views are not endorsed by Anthropic. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research.
Shapiro has, in the past, been a paid visitor at Microsoft Research New England and a paid consultant for FutureOfCapitalism, LLC. Shapiro has been paid for writing by the New York Times.
Shapiro's spouse has a disclosure statement posted at https://vivo.brown.edu/display/eoster1#Affiliations.
- Copy Citation Matthew Schwartz, Isaiah Andrews, and Jesse M. Shapiro, "An LLM Workflow That Reproduces, Improves, and Extends Published Economics Research," NBER Working Paper 35782 (2026), https://doi.org/10.3386/w35782. Download Citation
- Location for public releases of project materials
Related
Topics
Programs
More from the NBER
In addition to working papers, the NBER disseminates affiliates’ latest findings through a range of free periodicals—the NBER Reporter, the NBER Digest, the Bulletin on Health, the Bulletin on the Economics of Alzheimer's, and the Bulletin on Entrepreneurship—as well as research spotlights, lectures, and panel videos.

- Feldstein Lecture
- Presenter: Mark Duggan

- Methods Lectures
- Presenters: Melissa Dell & Ashesh Rambachan

- Presenter: Paul Goldsmith-Pinkham