Newshot

We can still win against ASI (maybe)

Summary

This essay is a combination of personal notes, discussions I had at the MAIA/AIST workshop, and slides from a talk I gave with JHU ASBS and BlueDot Impact. Premise A common theme I noticed, particularly among viewpoints with short timelines, is the idea that the introduction of ASI is inherently and instantaneously the loss condition. This stems, I think, from Yudkowsky-ian philosophy that ASI = unavoidable doom. [1] I think this defeatist attitude is counterproductive and detracts from real, important, effective avenues of action. A superintelligence is bounded through interaction with the physical world; there are fundamental limitations on feedback loops which are crucial to enact any kind of influence. [2] ASI on a laptop in a concrete room in 1850 cannot escape with any high probability, and even if it did, it would take a very long time for it to do anything – conceivably much longer than the physical lifetime of the laptop. In well-defined worlds like video games, it is not hard…

Original text(via LessWrong)

This essay is a combination of personal notes, discussions I had at the MAIA/AIST workshop, and slides from a talk I gave with JHU ASBS and BlueDot Impact.

Premise

A common theme I noticed, particularly among viewpoints with short timelines, is the idea that the introduction of ASI is inherently and instantaneously the loss condition. This stems, I think, from Yudkowsky-ian philosophy that ASI = unavoidable doom.[1] I think this defeatist attitude is counterproductive and detracts from real, important, effective avenues of action.

A superintelligence is bounded through interaction with the physical world; there are fundamental limitations on feedback loops which are crucial to enact any kind of influence.[2] ASI on a laptop in a concrete room in 1850 cannot escape with any high probability, and even if it did, it would take a very long time for it to do anything – conceivably much longer than the physical lifetime of the laptop. In well-defined worlds like video games, it is not hard to imagine scenarios where you are mathematically guaranteed to lose even with perfect skill and play – for example, if the ball in pong is much much faster than your paddle. There is an analogy here to galactic algorithms in computer science[3]: just because an ASI has an algorithm that scales better asymptotically doesn't mean the constant factors (e.g. time needed to conduct experiments; time needed to manufacture a quantity of robots) allow it to win within a finite time horizon.

In the present world, incidents like HuggingFace[4] can be largely attributed to 1. Ease: cybersecurity and the internet is an extremely accessible attack surface for AI, 2. Lack of foresight: most of the internet was not designed in anticipation of highly intelligent adversaries and widespread integration into society (cybersecurity is almost always reactionary), and 3. Ubiquity: the internet is a widespread and well-connected infrastructure tied deeply to society. As I, and many others have previously wrote, it is extremely difficult for ASI, if incarnated right now, to rapidly destroy all of humanity, or even do more mundane things like synthesize a virus or take over a factory. It would almost necessarily have to use humans, through superpersuasion or blackmail, limiting the effective set of “hands” to a few experts which it must coordinate… not impossible, but inherently gated by the limitations of humans as tools: we suck at following instructions, general competency, etc. Physical limits also significantly bound the influence of ASI in propagation time and physical limits (e.g. Control Theory); no matter what, noise and experiment mean you cannot have iteration speeds past a certain limit nor achieve results under a certain statistical confidence. All of this is further compounded by the need to avoid detection prior to achieving independence or dominance over humanity. This is why “gradual disempowerment” scenarios are generally more favored.[5]

However, as I have previously argued, this will change.[6] It’s only through luck that the robotics and biotech industries have not achieved the same widespread diffusion that the internet has (at this point in time), and all signs point towards this changing rapidly. This presents us a simple theory of change: designing these new systems with foresight, similarly to how we treat other high-risk technologies like nuclear weapons. Through proactively “hardening” these systems – these “hands” for AI – can we provide an effectively externally-imposed slowdown (allowing other actions) or even shut down ASI?

Adjusting time-to-doom through slowing the trajectory of ASI independence

We have three basic actions to combat against ASI: slow it, see it, stop it. Suppose a misaligned ASI has escaped: we define the win condition as a physical base (power, chips, robots, etc.) entirely independent of humanity. Here, we have four steps:

  • Building/accumulating physical assets takes some amount of time.
  • Each month, there is some chance someone notices.
  • Each time someone notices, some action may be taken for a response.
  • A response either terminates the ASI, or sets it back.[7]

where the first three steps can be convolved into one number: the number of times we expect to catch an attempt and act on it. Then,

where T is months to build, λ is the monthly chance of being noticed, a is the chance a notice is acted on, and d is the chance a response terminates the ASI. That gives three dials: slow it (T), see it (λ), and stop it (a and d). Then, we can make some starting estimates:

InputMedianSampled range (10th to 90th)Basis
Months to build (T)457 to 172Mostly follows from the next three rows
↳ Robot doubling time, months124 to 36Forethought’s estimate of about a year [8]
↳ Robots needed320,00010,000 to 10 millionGuess
↳ Robots at the start1,700100 to 30,000Guess
Monthly chance of notice (λ)2%0.5% to 8%Two covert programmes hidden 5 and 15 years [9] imply 0.5% to 2%
Notice acted on (a)42%15% to 75%Guess; no reference class
Response ends it (d)22%5% to 60%Guess; large botnets came back after takedowns [10]
Unmodelled route exists15%Fixed; tested from 0% to 100%Guess
Catches per attempt (X)0.40.03 to 3.9Result of the rows above

Sampled range (10th to 90th)

Months to build (T)

Mostly follows from the next three rows

↳ Robot doubling time, months

Forethought’s estimate of about a year[8]

↳ Robots needed

10,000 to 10 million

↳ Robots at the start

100 to 30,000

Monthly chance of notice (λ)

Two covert programmes hidden 5 and 15 years[9] imply 0.5% to 2%

Notice acted on (a)

Guess; no reference class

Response ends it (d)

Guess; large botnets came back after takedowns[10]

Unmodelled route exists

Fixed; tested from 0% to 100%

Catches per attempt (X)

Result of the rows above

but also sample from a wide range for all parameters. Simulating many different worlds with different combinations of parameters:

Where the worlds sit
Where the worlds sit

Clearly, it doesn’t seem terribly unlikely (black line = even odds) that we can indeed detect and catch nascent ASI even in the absence of interventions (orange dots). Nevertheless, the odds are not great. To model the effect of hardening (green dots), we can adjust the parameters:

DialFactor (median; range)Median worldBasisEvidence
T×1.8 per stage; 1.1 to 345 → 95 monthsNuclear projects after 1970 took 2.4× longer (17 years against 7) [11]Loose
T÷6.3; 2 to 20Included in the row aboveNone; today’s nearest control stops about 5% of ordersGuess
λ×3.5; 1.5 to 82% → 7%Monitoring speeds detection about 2× (fraud: 6 months against 12 [12] ; intrusions: 9 days against 20 [13] )Anchored
a×4; 2 to 842% → 74%None; history shows notices are often ignored [14]Guess
d×2.3 twice; 1.3 to 4 each22% → 59%None; kept modest because strong takedowns have failedGuess
X0.4 → 5.5Follows from the rows above

Factor (median; range)

Median world

×1.8 per stage; 1.1 to 3

45 → 95 months

Nuclear projects after 1970 took 2.4× longer (17 years against 7)[11]

÷6.3; 2 to 20

Included in the row above

None; today’s nearest control stops about 5% of orders

×3.5; 1.5 to 8

Monitoring speeds detection about 2× (fraud: 6 months against 12[12]; intrusions: 9 days against 20[13])

None; history shows notices are often ignored[14]

×2.3 twice; 1.3 to 4 each

None; kept modest because strong takedowns have failed

Follows from the rows above

Some interesting results follow. Win/loss probability in worlds is largely a two-state readout, with the majority of probability mass either on 0% or 100%; hardening therefore flips many worlds from certain doom to certain success.

Most worlds get settled
Most worlds get settled

The median time-to-notice required for even odds also increases to historically plausible levels:

Years hidden
Years hidden

Finally, we can decompose the expected effect of interventions at each level:

Points lost without each measure and slow it see it stop it
Points lost without each measure and slow it see it stop it

and, as expected, speedups in thinking speed quickly saturate and do not increase ASI-win probability:

Thinking has saturated building has not
Thinking has saturated building has not

(1x ASI = 100x human)

Lastly, I look at some pessimistic and optimistic assumptions:

InputFastCentralBottleneck
A route the model misses exists60%15%3%
Robot doubling time, months0.5 to 64 to 369 to 60
How far physical steps compress5 to 300×1.2 to 10×1.05 to 3×
Monthly chance of noticeA quarter of centralAbout 2%Double central
A response ends it0.5% to 15%5% to 60%15% to 80%
Hardening factorsWeakAs aboveStronger
Median months to build745109
Result, without → with hardening99% → 94%72% → 26%39% → 5%

A route the model misses exists

Robot doubling time, months

How far physical steps compress

Monthly chance of notice

A quarter of central

Double central

A response ends it

Hardening factors

Median months to build

Result, without → with hardening

ASI wins by worldview
ASI wins by worldview

Obviously, if most probability is dominated by definitionally unforeseeable routes, hardening does not provide much gain. Outside of this, however, hardening seems to robustly increase our chances of winning. Therefore, it seems like hardening may represent a third option to policy and technical alignment work amenable to great interdisciplinary efforts.

Thanks to Elaine Ly and Peyton Li for comments.

  • Eliezer Yudkowsky, “AGI Ruin: A List of Lethalities,” LessWrong, June 5, 2022, https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities (point 2: losing to a high-powered cognitive system “looks at least as deadly as ‘everybody on the face of the Earth suddenly falls over dead within the same second’”).
  • On physical limits bounding a superintelligence: Jeffrey Heninger and Aysja Johnson, “Superintelligence Is Not Omniscience,” LessWrong, April 7, 2023, https://www.lesswrong.com/posts/qpgkttrxkvGrH9BRr/superintelligence-is-not-omniscience ; Shahin Kaveh, “Why AI Won’t Kill Us by 2030, and Probably Ever,” LessWrong, November 14, 2026, https://www.lesswrong.com/posts/QgSvF5yDJjkn3NZTY/why-ai-won-t-kill-us-by-2030-and-probably-ever ; MarkelKori, “Where ASI Could Stumble,” LessWrong, November 15, 2026, https://www.lesswrong.com/posts/Jj2aLyZPYDv8ZsnCj/where-asi-could-stumble-1 . For the counterargument, see Algon, “Chaos Alone is No Bar to Superintelligence,” LessWrong, October 6, 2025, https://www.lesswrong.com/posts/4AxYNx5HxjKKyfJyG/chaos-alone-is-no-bar-to-superintelligence .
  • Richard J. Lipton, “Galactic Algorithms,” Gödel’s Lost Letter and P=NP, October 23, 2010, https://rjlipton.com/2010/10/23/galactic-algorithms/ (an algorithm “wonderful in its asymptotic behavior” but never used in practice).
  • OpenAI, “OpenAI – Hugging Face Incident Technical Report,” August 26, 2026, https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf .
  • Jan Kulveit, Raymond Douglas, Nora Ammann, Deger Turan, David Krueger, and David Duvenaud, “Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development,” arXiv:2501.16946, January 28, 2025, https://arxiv.org/abs/2501.16946 .
  • Jonathan Ouyang, “Let’s be realistic about AI-enabled bioweapons,” The Johns Hopkins News-Letter, October 5, 2026, https://www.jhunewsletter.com/article/2026/10/lets-be-realistic-about-ai-enabled-bioweapons .
  • On catching an AI attempt and acting on it: Ryan Greenblatt and Buck Shlegeris, “Catching AIs red-handed,” LessWrong, January 5, 2024, https://www.lesswrong.com/posts/i2nmBfCXnadeGmhzW/catching-ais-red-handed ; Buck Shlegeris, “Handling schemers if shutdown is not an option,” LessWrong, April 18, 2025, https://www.lesswrong.com/posts/XxjScx4niRLWTfuD5/handling-schemers-if-shutdown-is-not-an-option .
  • Tom Davidson and Rose Hadshar, “The Industrial Explosion,” Forethought, May 21, 2025, https://www.forethought.org/research/the-industrial-explosion (estimates “an initial robot doubling time of roughly 1 year or less”; its summary is more conservative, “on the order of a few years”). Also posted to LessWrong: https://www.lesswrong.com/posts/Na2CBmNY7otypEmto/the-industrial-explosion .
  • Biopreparat was created on April 24, 1974 (Glavmikrobioprom Order 131 DSP), per “Biopreparat,” Wikipedia, citing Anthony Rimmington, The Soviet Union’s Invisible Weapons of Mass Destruction (Springer, 2021), https://en.wikipedia.org/wiki/Biopreparat , and was exposed when Vladimir Pasechnik defected to Britain in late October 1989, per David E. Hoffman, The Dead Hand (Doubleday, 2009), excerpted by the National Security Archive, May 6, 2010, https://nsarchive2.gwu.edu/NSAEBB/NSAEBB315/ . Aum Shinrikyo began producing botulinum toxin and phosgene in 1990 and carried out the Tokyo subway sarin attack on March 20, 1995, per “Tokyo subway sarin attack,” Wikipedia, https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack ; see also Richard Danzig et al., Aum Shinrikyo: Insights Into How Terrorists Develop Biological and Chemical Weapons, 2nd ed. (CNAS, 2012), https://www.cnas.org/publications/reports/aum-shinrikyo-second-edition-english .
  • For example, Emotet, taken down in an international operation coordinated by Europol and Eurojust at the start of 2021, was rebuilding its botnet by November 2021: Lawrence Abrams, “Emotet malware is back and rebuilding its botnet via TrickBot,” BleepingComputer, November 15, 2021, https://www.bleepingcomputer.com/news/security/the-emotet-malware-is-back-and-rebuilding-the-botnet-via-trickbot/ .
  • Jacques E. C. Hymans, “The Threat of Nuclear Proliferation: Perception and Reality,” Ethics & International Affairs 27, no. 3 (2013): 281–298, https://doi.org/10.1017/S089267941300021X (all dedicated projects launched before 1970 succeeded, averaging about seven years to a first test; only three of the ten launched since 1970 succeeded, needing “an average of about seventeen years”). See also Hymans, Achieving Nuclear Ambitions (Cambridge University Press, 2012).
  • Association of Certified Fraud Examiners, Occupational Fraud 2024: A Report to the Nations, as summarized in Alexander Gershner, “Be Proactive & Limit Asset Misappropriation Schemes,” Forvis Mazars, August 30, 2024, https://www.forvismazars.us/forsights/2024/08/be-proactive-limit-asset-misappropriation-schemes (proactive surveillance and data monitoring detected frauds at a median of six months; the typical scheme lasted 12 months).
  • Mandiant M-Trends 2024, as reported in Connor Jones, “Mandiant: Orgs are detecting cybercriminals faster than ever,” The Register, April 23, 2024, https://www.theregister.com/2024/04/23/mandiant_orgs_are_detecting_cybercrims/ (global median dwell time of ten days; non-ransomware intrusions found by internal teams in nine days against 20 days when an outside party notified the victim).
  • On warning signs failing to trigger action: Eliezer Yudkowsky, “There’s No Fire Alarm for Artificial General Intelligence,” LessWrong, October 13, 2017, https://www.lesswrong.com/posts/BEtzRE2M5m9YEAQpX/there-s-no-fire-alarm-for-artificial-general-intelligence .

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by LessWrong
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗