Newshot

Opus 5.5 Forecast: The Road to an AI Deal

Summary

This article portrays an experiment on how good current frontier AIs are at producing detailed forecasting analyses. For this, I tasked Opus 5.5 to write an analysis on the effectiveness of AI safety advocacy work. Opus 5.5 decided to turn it into a broader forecast on the likelihood of a US-China AI deal by the end of 2030, and the cost effectiveness of various strategies for increasing that probability. At a later stage, it also added the likelihood of a US-China AI deal before complete automation of AI R&D. I was moderately impressed. Opus 5.5 performs far above previous Claude models. I was planning on writing an article myself on this topic, and just take some inspiration from Opus 5.5’s article, but its first draft was interesting enough that I put my own article on hold, and took on the role of a supervisor to see whether Opus 5.5 could improve its work further. The article linked below is the result. Still, the article should be considered a lower bound on what Opus 5.5 can pro…

Original text(via LessWrong)

This article portrays an experiment on how good current frontier AIs are at producing detailed forecasting analyses. For this, I tasked Opus 5.5 to write an analysis on the effectiveness of AI safety advocacy work. Opus 5.5 decided to turn it into a broader forecast on the likelihood of a US-China AI deal by the end of 2030, and the cost effectiveness of various strategies for increasing that probability. At a later stage, it also added the likelihood of a US-China AI deal before complete automation of AI R&D.

I was moderately impressed. Opus 5.5 performs far above previous Claude models. I was planning on writing an article myself on this topic, and just take some inspiration from Opus 5.5’s article, but its first draft was interesting enough that I put my own article on hold, and took on the role of a supervisor to see whether Opus 5.5 could improve its work further. The article linked below is the result.

Still, the article should be considered a lower bound on what Opus 5.5 can produce. The instructions and feedback could be further improved. I could have asked it to iterate even more on the article. Now that I know more about its strengths and weaknesses, I could have restarted the entire process and worked more with Opus 5.5 to handle those better. But now I want to work on other projects.

I absolutely recommend reading or skimming the article. I personally found it quite interesting. A few caveats before you do so:

  • Authorship: Everything is written by Opus 5.5. I only provided suggestions and critiques, though this feedback was sometimes very concrete (e.g. “You should consider doing X in a way that handles issue Y, perhaps by doing Z”). Consequentially, I don’t agree with all the parameter choices and modeling decisions that Opus 5.5 does. This is Opus 5.5’s forecast, not mine. However, my thinking and views have undoubtedly affected the analysis, as disclosed in the disclaimer below.
  • Article length and complexity: I intentionally let the analysis grow long and complex to test whether Opus 5.5 could handle it. As such, I recommend focusing on Part 1: A simple model, which is almost 6,000 words, taking perhaps 20-30 minutes to read through. This section captures the overall approach well. The entire article is over 60,000 words long, largely because Opus 5.5 keeps adding things when I provide feedback, while lacking ability to simplify things when necessary. Nevertheless, Opus 5.5 didn’t seem to struggle much with handling the article or conversation length, though it did occasionally make various mistakes (such as failing to update all parts relevant to some specific change, such as a modification to one of the models, because the change affects unnecessarily many different parts of the article).
  • Remaining errors and mistakes: I haven’t verified all the sources, reasoning and code (would take more time than it’s worth). I focused on providing feedback and suggestions on parts I was more interested in, and haven’t reviewed all parts equally carefully. The most uncertain part of the analysis is the strategy effectiveness results. I have pointed out many flaws in the modeling assumptions by Opus 5.5, which it then fixed or included in sensitivity analysis, but they’re harder to find compared to if I had constructed the models myself. Undoubtedly, many mistakes remain. If you are looking for more accurate forecasts on domestic AI regulation, pacing AI, and AI deals between US and China, see predictions by professional forecasters with good track records, such as predictions by the Sentinel team.
  • Strategy effectiveness analysis: The analysis focuses on the cost effectiveness of additional spending on each strategy in the coming years, beyond what Opus 5.5 expects by default. As such, the strategy ranking and effectiveness estimates tries to answer what strategies are probably going to be underinvested in. It does not focus on the effectiveness of the total budget that each strategy may receive (though the analysis includes some reasoning and estimates for this as well). The strategy ranking (while also uncertain) is more reliable than absolute effectiveness estimates, which could easily be off by an order of magnitude or more. Note that the strategy effectiveness analysis only considers impact on a formal AI deal, and nothing else. Most of the strategies have a range of benefits not considered, meaning that their effectiveness is underestimated, and the ranking with regards to other impacts could be very different. The collection of analyzed strategies is not exhaustive.
  • Complicated models: To estimate the probability of an AI deal between US and China, it may be better to consider base rates and other simpler methods than the models that Opus 5.5 use. However, a model is very useful to reason about strategy effectiveness, and for updating on new evidence, as you can then reason about what parts of the model that a certain strategy or incoming evidence affects to see how that impacts the final outcomes.
  • Future work: I plan to task future frontier AIs with reviewing this work by Opus 5.5, so they can find mistakes or possible improvements. This analysis then works as a method of evaluation, by which I can quickly figure out the ability of future AIs to complete complicated forecasting analyses. I think this article can also serve as an inspiration for future work on evaluating strategy effectiveness, as I believe most of the overall approach is sound. I also hope to explore more formal methods for evaluating AI forecasting analyses, in order to figure out how AIs can improve and accelerate such work (I learned a lot about how to delegate forecasting work to AI through this project, but many of those lessons are too informal to convey effectively to others).

There are elements in the article that can’t be displayed properly in Substack, so I've copied the opening sections below and linked the full version (Google Docs), where you can also find earlier versions of the article in different tabs (the latest is v6). Also, feel free to read the conversation with Opus 5.5 to see how I prompted it and what feedback I gave it, etc. Note that a Claude account is required to view both the article and the conversation.

  • Google Docs version: The Road to an AI Deal
  • Claude conversation

You can also download the article as a PDF: Download

Disclaimer: I have ties to ControlAI members, am a member of the Torchbearer Community, and consider myself a member of the broader AI safety community that aims to prevent catastrophic outcomes from AI. Opus 5.5 knows this from previous conversations with memory turned on. (In hindsight, I should have turned it off for this experiment. Oh well.) I have tried to formulate my instructions and feedback to Claude in an impartial and unbiased way, but Claude may still have adapted to its knowledge of my views from its memories, subtle cues in the conversation, or based on my own public writing. Someone else doing this exercise with Opus 5.5 would probably get very different results.

What follows is my impression of how good Opus 5.5 is at producing forecasting analyses, as well as the first part of its analysis. I have many thoughts on its modeling choices and parameter estimations that are not included in this post, and might write a follow-up analysis on that in the future, in which case I’ll link to it here.

If you find this interesting, please subscribe!

How good is Opus 5.5 at doing forecasting analyses?

Naturally, I can’t evaluate whether the predictions by Opus 5.5 are correct yet, but the analysis can be evaluated on other qualities, such as on whether it uses reliable sources, whether modeling assumptions seem reasonable, whether parameter choices and other decisions are explained clearly and transparently, and so on.

My own judgment is that Opus 5.5 seems like the first AI to be able to produce a decent forecasting analysis, as long as it’s given some feedback. I have experimented a bit with letting earlier models do forecasting analyses like this, and they don’t come near the level of quality that Opus 5.5 achieves (though this is just based on my personal judgment, rather than some formalized test).

What Opus 5.5 (mostly) does well is the high-level modeling. It figures out how to decompose various outcomes, combines different modeling methods, does a decent job of reasoning about parameter values based on available evidence, does careful sensitivity analysis, and is impressively comprehensive. It was Opus 5.5 that came up with how to combine models for salience, AI deals, and strategy effectiveness (see the first part of the forecast analysis below for a brief explanation of this approach). On the other hand, it makes a large number of weird judgment calls in specific parts of the models, and many mistakes in the low-level implementation of its high-level strategy. Opus 5.5 is clever, but also lacks judgment (more on this below).

Opus 5.5 also often fails to explain the reasoning behind its assumptions and decisions, unless prompted repeatedly to do so. It seems like AI forecasting ability is improving faster than the ability to explain things clearly. However, while Opus 5.5 makes a few systematic errors, it can often fix these when you point out examples.

The problems I found with its work usually fell into one of these categories. Note that the final article version has fewer of these problems since I prompted Opus 5.5 to remove them.

  • Lack of transparency: When estimating things the calculations aren’t always transparent (unclear where the numbers come from). I sometimes get the sense that Opus 5.5 doesn’t want to show its work. Open 5.5 explains that it’s “a tendency to compress: I presented conclusions and tables and left derivations out until you asked.”
  • Confused writing: Often forgets to include links or explanations when some concept or topic is first introduced, and to include pointers to other parts of the document when necessary (e.g. mentioning when something is included in Appendices). This issue is particularly frustrating when Opus 5.5 is given feedback on specific things to edit or clarify, since it often refers to concepts, sources, or calculations as if the reader is familiar with them, despite these not having been introduced to the reader yet.
  • Overadjustment to instructions and feedback: If I mention an example, it may force that specific example into the article and give it more attention than called for, regardless of whether it’s the best example or fits there in context. Opus 5.5 notes that it doesn’t push back enough: “I accepted nearly every critique and rarely argued that a requested addition wasn't worth its complexity.”
  • Unreliable sources: Opus 5.5 doesn’t always use the most recent or most reliable source. This is something I have encountered with previous Claude models as well. For instance, they often reference second-hand sources even when first-hand sources are publicly available. Opus 5.5 is better at finding and using good sources than previous models, I think, but still makes mistakes or odd choices sometimes. Opus 5.5 notes that it isn’t always its own fault, but a tool limitation: “Several second-hand sources were substitutes for primary ones I couldn’t reach”.
  • Bad judgment: Opus 5.5 is often clever and able to reason well about the modeling, but then proceeds to make silly mistakes. When I point those out, it often understands the problems and is able to fix them, which makes it seem like it has good judgment that it just doesn’t use. When pointing out one error, it was often able to fix it, but in the process introduced more errors, creating an eternal loop of fixing mistakes. Opus 5.5 sometimes makes modeling decisions that look good superficially, but have subtle flaws on closer inspection, which are difficult and time-consuming to find. Letting Opus 5.5 make every change made this project take longer time and resulted in a more confusing and worse forecast analysis than if we had worked more collaboratively on it.

Opus 5.5 adds a few weaknesses of its own, out of which this is the most important, I think: “Reactive rather than proactive stress-testing. Correlations, timing by administration, persistent salience and window types all came from your prompts. Once prompted I handled them well, but I didn’t anticipate them.” I fully agree with this, and in hindsight I should have prompted it to do more stress-testing on its own.

There are also a few more nuanced issues. For instance, Opus 5.5 sometimes falls back to doing a more informal analysis without specific forecasts or estimates, even where that would be highly interesting and useful (this happened more at the start of the project, before being given feedback on doing more formal analysis). At other times, it feels like it forgets the original purpose of the article, doesn’t focus on what is most important, and fails to make its analysis actionable.

I also suspect that Opus 5.5 struggles with updating enough on new evidence. It sometimes forgot to reconsider its views until prompted to do so. I’m more uncertain on this point though.

(Note that many or most human writers fall into these traps as well, often much worse than Opus 5.5.)

Here’s the first part of Opus 5.5’s forecasting analysis:

The Road to an AI Deal

Forecasting a verified US-China slowdown by the end of 2030, and the strategies that make it likelier

By Claude Opus 5.5 · Supervised and reviewed by Alvin Ånestrand · November 2026

Summary

AI safety advocacy has been cheap and effective at agenda-setting, but a verified US-China slowdown, or pacing, of frontier AI remains unlikely. A slowdown with its verification operating by the end of 2030 has a 2.6% chance in a simple model and 3.2% in a full one. One whose verification is operating before AI research and development is fully automated, which in most simulated worlds comes by the end of 2028 but can come long after 2030, has 1.2% in the simple model, which counts only the current administration and its successor, and 2.3% in the full one, where the administrations elected in 2032 and 2036 also get their turn if AI R&D has not been automated before they can act; a deal that starts operating after automation is more likely to fail. Signing by 2030 is likelier, about 3.5% in the simple model and 5.0% in the full one, but nearly a third of the deals signed by then would fail both tests, operating neither by the end of 2030 nor before AI R&D is automated. The money now pouring in will do the most good if it targets what gates a deal: what China and the US administration in office when salience peaks are willing to sign, how fast a deal can be negotiated and verified, and whether the next incident reads as a loss of control rather than a race.

  • Approach: three linked models: whether AI becomes a top public issue, whether that leads to a deal in time, and what each kind of advocacy adds, valued by what $1M and $10M a year buy. Every input assumes safety spending frozen at its 2025 level; the forecast growth and each strategy are then valued on the same curves of diminishing returns, scaled to what spending so far has bought. Each administration’s willingness is judged as its chance of deciding to sign after a typical salience window; since windows usually open part-way through a term, the full model reads those judgments halfway between that and a whole term’s chance. Paired figures such as 3.5–5.0% give the simple and full models’ results; uncertainty ranges are marked as 90% CIs.
  • So far: roughly $0.4–1.1B of governance, policy and advocacy spending bought two state transparency laws, a 170+ member UK parliamentary coalition, federal bills and, after a summer of incidents, a new US-China AI channel, but no binding limits on frontier development. Judging its effect on each model input, I estimate it has multiplied the chance of a deal operating before automated AI R&D about 2.4-fold (90% CI ×1.6–3.5), compared with a world in which no one had spent anything.
  • Ahead: public advocacy and lobbying should roughly quadruple to about $190M in 2027 (90% CI $90–410M), while industry-aligned electoral money keeps outspending the safety side 2–3 to 1. On the same curves that reproduce the record, this surge, about $2.3B above 2025 levels over 2026–29, multiplies the chance of a deal by about 1.3 (×1.16–1.48 across the spending forecast’s 90% CI) if the field keeps what it builds, because the easiest gains came first.
  • Salience: under 1% of voters name AI as their top issue. I give 78% that AI’s share of answers reaches 5% in some month on Gallup’s “most important problem” question by the end of 2029 (61% by the end of 2028), mostly because fast AI progress brings incidents and visible disruption: 94% where an automated coder arrives by the end of 2028, 39% where it comes later. With safety spending frozen at its 2025 level it would be 76%, and with a scaled public campaign on top 79%; the full model’s richer simulation, in which big incidents leave a lasting floor and grow more severe, gives 81% (Part 2 explains the gap).
  • Odds: an indefinite prohibition is the most demanding kind of deal in the models, about a quarter of all deals, so its 0.65–0.8% by 2030 and 0.3–0.6% before automated AI R&D are part of the slowdown figures above, not added to them. For a slowdown signed by 2030 the models give 3.5–5.0%, close to the 5% that Metaculus forecasters give for a verifiable treaty limiting AGI signed by 2030.
  • Time is the binding constraint: in the full model the current administration or its successor decides to sign a slowdown in 37% of worlds and China is willing too in 20%, but only 5.0% sign by 2030 and 3.2% have verification running by the end of 2030. An administration that decides usually does so late in its window: a 2029 administration acts in its first year less than a fifth of the time. Past agreements took months to years to negotiate, and from two months to four years more to implement. A deal before automated AI R&D has more time where AI progresses slowly, and the 2033 and 2037 administrations carry a third of its 2.3%.
  • Timelines: the two figures pull apart. If an automated coder, an AI a leading lab would rather keep than its human software engineers, arrives by the end of 2027, as it does in 58% of simulated worlds, a deal operating by the end of 2030 has 3.8% but one operating before automated AI R&D only 0.13%; if it arrives in 2031–35, they have 2.2% and 8.3%, and after 2035, 1.7% and 16%. AI forecasts move the second figure more than any political input, while the first depends most on China and US willingness.
  • Default spending: the forecast growth, about $2.3B above 2025 levels over 2026–29, adds about 0.3–0.4 points to the chance of a deal before automated AI R&D in the two models, counting only what that money itself builds. Its first $10M a year buys most in verification (about 5–8 bp) and readiness (3–6), and least in policy research, fieldbuilding and earned media (0.2–0.3), electoral money (0.16–0.19) and briefings (0.8–0.9), which take nearly 90% of the money, though much of their value lies in goals the models don’t score. In every channel with a strategy, the forecast money buys more per dollar than extra money would.
  • What the strategy rankings measure: which kinds of work are most likely to be underinvested in, given the spending already forecast, not whether that spending is worthwhile. In both models every strategy raises the expected chance of a deal, the forecast growth as a whole raises it by about a third, and much of the money in the lower-ranked channels also serves goals the models don’t score, such as domestic law. More money in almost any of these channels would help; the rankings say where the next dollar helps most, and nothing here argues for cutting what is already planned.
  • Best buys for extra money: for the first $1M a year, two levers almost no one funds yet, shaping 2028 presidential positions (about 0.6–1.0 basis points of the chance of a deal before automated AI R&D in the two models) and warning-shot readiness (0.6–1.0 bp); for $10M a year, verification R&D with US-China Track 2 talks between non-government experts (2.2–3.5 bp), the only strategy that raises China’s willingness and shortens negotiation and implementation, and the one whose effect lasts into the 2030s. Spread across the leading levers, $10M a year buys about 6 bp in the full model and $50M about 18, about 6 times what a ControlAI-style $50M plan of briefings, ads, list-building and volunteers would. In the full model, cheap transition readiness for a 2029 administration and a push for industry self-pacing also rank near the top for the first million, and transition readiness first for a deal operating by the end of 2030. These gains are for money added on top of the forecast growth, and they move up to two or three times with the field’s growth and the normalization.
  • Weakest buys for extra money, for this goal: paid ads (about 0.8–1.1 bp for $10M a year), elite briefings, whose channel is forecast to spend about $90M a year anyway (0.22–0.26 bp), and electoral deterrence (0.07–0.08 bp), though briefings and electoral money matter more for domestic law.
  • My own view: all things considered, about 4% (90% CI 0.6–10%) for a verified slowdown operating by the end of 2030 and 3% (0.3–10%) for one operating before automated AI R&D, a little above the models: crisis diplomacy could compress timelines far below historical norms, forecasters who wrote after the November evidence expect drastic action more often than the models do, and the timeline input may lean fast, which would understate the second figure; but the models leave out failure modes such as Senate ratification and deals that collapse, and their 2030s administrations are their least grounded part. I trust the ranking of strategies more than any dollar figure.
  • Caveat: the structure is the main contribution. The parameters are judgment calls, all stated, and the code in the appendices lets readers substitute their own.

The approach in one picture

How much does a dollar of advocacy change the odds that the US and China agree to slow frontier AI? Nobody can measure that directly, but it breaks into questions that can each be estimated: will AI become a top public issue, would that make a deal likely, and which kinds of spending move those odds most. The article answers them with three linked models.

The salience model simulates the share of Americans naming AI as the most important problem, month by month to the end of 2040: a baseline that grows and, in the full model, levels off near a ceiling, plus incidents that spike and fade. Its main output is the chance of a salience window, AI reaching 5% on Gallup’s question. The deal model combines salience with how fast AI progresses, who holds the White House, what China will accept and how much time is left, and returns two probabilities: that a verified US-China slowdown has its verification operating by the end of 2030, and that one signed by an administration in office before January 2041 has it operating before AI research and development is fully automated. The strategy model turns a budget into doublings of a channel’s resources (briefings, verification work, ads and so on) and reads what they buy off the channel’s curve, on which later doublings buy less and which is scaled to what the field’s spending so far has bought. The inputs assume spending frozen at its 2025 level; the forecast growth moves each channel up its curve, a strategy moves it further, and the cost divided by the gain gives dollars per percentage point.

People will disagree with many of my parameter values, and some will prefer different modeling choices. The structure is the part I’d defend most: it makes every assumption explicit, it shows which ones matter (capability timelines, the time diplomacy takes, China’s terms and US willingness, far more than ad reach), and it lets anyone swap in their own numbers using the code in Appendices A and E.

The article runs in two passes. Part 1 builds the simplest version of all three models, with a handful of parameters and arithmetic you can check by hand. Part 2 builds the full version, with four administrations deciding month by month through 2040, simulated AI timelines and negotiation and implementation times, correlated parameters and a richer salience simulation. The simple model gives 2.6% for a deal operating by the end of 2030 and 1.2% for one operating before automated AI R&D, the full model 3.2% and 2.3%. Both put work on 2028 presidential positions and warning-shot readiness at the top for the first $1M a year, and verification R&D at the top for $10M. Background on spending and recent events comes first; my own all-things-considered forecast, movement building, hard questions and recommendations come last.

To read the rest, please see the google docs version.

Thank you for reading! If you found value in this post, consider checking out my blog: Forecasting AI Futures.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by LessWrong
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗