Newshot

David Duvenaud on Post-AGI Futures

Summary

What sort of futures can we expect for civilization after AGI? Ideally, our future satisfies all of these properties: tazik's blog is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. plausible stable desirable Surprisingly, while current visions meet some of these criteria, none fully satisfy all of them. Let’s explore some of the most commonly presented post-AGI futures. type 1: nice but unstable post-scarcity If labor is no longer scarce (because of robotics explosion), wages will collapse Resources that are currently in surplus will eventually be contested (e.g. land, energy) Even if we (humans) don't fall into a Malthusian trap, AIs (embodied or not) will replicate at a faster rate, leading to scarcity Example: The sun is our primary source of energy. We are 13 doublings away from using up all of the sunlight hitting Earth (1.7x10^17 W). And 44 doublings away from needing all of the Sun's output (3.8x10^26 W) Sou…

Original text(via LessWrong)
1440p version of the image used in many "The world if" memes :  r/MemeRestoration
1440p version of the image used in many "The world if" memes : r/MemeRestoration

What sort of futures can we expect for civilization after AGI?

Ideally, our future satisfies all of these properties:

tazik's blog is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Surprisingly, while current visions meet some of these criteria, none fully satisfy all of them.

Let’s explore some of the most commonly presented post-AGI futures.

type 1: nice but unstable

post-scarcity

  • If labor is no longer scarce (because of robotics explosion), wages will collapse
  • Resources that are currently in surplus will eventually be contested (e.g. land, energy)
  • Even if we (humans) don't fall into a Malthusian trap, AIs (embodied or not) will replicate at a faster rate, leading to scarcity
  • Example: The sun is our primary source of energy. We are 13 doublings away from using up all of the sunlight hitting Earth (1.7x10^17 W). And 44 doublings away from needing all of the Sun's output (3.8x10^26 W)

Post-scarcity sounds appealing, but is incoherent upon inspection.

laissez-faire

  • IdeaWe can still have robust trade between humans and AI even when they are abundant in the physical and digital worldConcretely, rich humans will want services from only humans. Example: horse-drawn carriages
  • We can still have robust trade between humans and AI even when they are abundant in the physical and digital world
  • Concretely, rich humans will want services from only humans. Example: horse-drawn carriages
  • ProblemThis will necessarily be a niche market, and a tiny fraction of the total economy (to see this: consider what % of the economy is currently occupied by this category)In most of the economy, competition is wont to drive wages below human subsistenceThis leads to human loss of cultural, political, and hard power
  • This will necessarily be a niche market, and a tiny fraction of the total economy (to see this: consider what % of the economy is currently occupied by this category)
  • In most of the economy, competition is wont to drive wages below human subsistence
  • This leads to human loss of cultural, political, and hard power

type 2: stable but bad

locusts

File:Foster Bible Pictures 0074-1 Offering to Molech.jpg
File:Foster Bible Pictures 0074-1 Offering to Molech.jpg

Source: Can goodness compete, Joe Carlsmith

  • Problem: Goodness may not be robust to competition (Joe Carlsmith, 2025)The winner of competition is powerPower is not the same as goodness (~orthogonal)So every opportunity to sacrifice goodness for power makes you more likely to win (we see this in human society)
  • The winner of competition is power
  • Power is not the same as goodness (~orthogonal)
  • So every opportunity to sacrifice goodness for power makes you more likely to win (we see this in human society)
  • So if the future involves a lot of competition (does it?)... goodness loses
  • Can goodness compete?Locusts aren’t entirely foreign; human civilization is a locust in some waysPower might be dominant, yet we still managed to select for (or at least, not entirely select away)friendshiployaltyartchildren’s laughterplayIMO, some of the above feel like side effects, or local instantiations of phenomena that are globally advantageous (e.g. friendship, loyalty are instances of cooperation, so it’s not that surprising that they’d develop if we select for coordination)But art and play do feel harder to argue for on instrumental grounds (yay for terminal goods)
  • Locusts aren’t entirely foreign; human civilization is a locust in some ways
  • Power might be dominant, yet we still managed to select for (or at least, not entirely select away)friendshiployaltyartchildren’s laughterplay
  • children’s laughter
  • IMO, some of the above feel like side effects, or local instantiations of phenomena that are globally advantageous (e.g. friendship, loyalty are instances of cooperation, so it’s not that surprising that they’d develop if we select for coordination)
  • But art and play do feel harder to argue for on instrumental grounds (yay for terminal goods)
  • This would be a plausible and stable but not good future

Source: Post-AGI Civilization Talk, David Duvenaud

totalitarian stasis

  • Let’s say the government stops or pauses AI development, so as to keep humans competitive
  • In the limit, you cannot stop AI development without implementing a panopticon and surveilling all your citizens (it seems possible to stop large scale development, but perhaps not smaller scale ones, without devolving to an Orwellian state. But this is not robust to development: one can imagine underground data centers, or even decentralized AI training)
  • Similarly, if the state has legal precedent for imposing restrictions on progress, we could see this lead to encroachment of other freedoms e.g. reproductive rights (tbh, I’m not sure I fully see the argument here)
  • If it is the case that if anyone builds it, everyone dies, then we need global governance for stability!There will then be no freedom, even in Mars colonies (at temporal limits)
  • There will then be no freedom, even in Mars colonies (at temporal limits)
  • North Korea is an example of totalitarian stasis
  • This would be stable, perhaps, but not nice

type 3: stable, nice, but plausible?

benevolent singleton

  • Aligned AI gets total control
  • This AI has humanity’s best values upon reflection, e.g. Coherent Extrapolated Volition (CEV)Note: long reflection feels intuitively silly to me
  • Note: long reflection feels intuitively silly to me
  • You need to load our values correctly and really nail it before “The Handoff”Incredibly hard to robustly validate all possible failure modes“Need to grab power before reflection” (not sure what he meant by this)
  • Incredibly hard to robustly validate all possible failure modes
  • “Need to grab power before reflection” (not sure what he meant by this)
  • Powers-that-be will fight challengersIt’s not hard to imagine the government, once AGI-pilled, asks/demands to write Claude’s Constitution
  • It’s not hard to imagine the government, once AGI-pilled, asks/demands to write Claude’s Constitution

Overall, this is stable and good, but may not be plausible (or is at least an unknown unknown).

summary

Existing visions of futurism by philosophers and tech company CEOs are unlikely, non-robust, or not good for humans.

green = yes, red = no, grey = maybe

But we can and should conceive of a plausible, stable, and desirable future for human civilization.

credits

This blogpost is based on David Duvenaud's excellent talk at AI Safety Week 2026 in Toronto. If you enjoyed this, I highly recommend checking out: https://post-agi.org/

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by LessWrong
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗