Newshot

Quick Thoughts on the SURA Debate

Summary

The SURA debate seems remarkably confused to me, for the reasons described below. I think they point to a broader... missing mood? Lack of on-the-ball-ness? Lack of creativity? Tangled web of biases for tomorrow's historians to unravel? The topic seems too urgent to not tell others about, even before Searching For The Central Problem in depth. The frontier models are basically all trained in the US, by two bleeding-edge companies and a handful more deep-pocketed companies and also Meta. Then, Chinese labs usually distill the models, or otherwise copy advances by the US labs. China has, and makes, fewer high-end chips than the US, but they are working on building factories to make those. These factories seem like they'll be up mostly after 2027 (see "pre-2028 timescale" below). China, unlike the US, has also exercised some kinds of direct power against their own (admittedly less-powerful homegrown versions of) top-level tech billionaires. I tried thinking of central cruxes to This Sort…

Original text(via LessWrong)

The SURA debate seems remarkably confused to me, for the reasons described below.

I think they point to a broader... missing mood? Lack of on-the-ball-ness? Lack of creativity? Tangled web of biases for tomorrow's historians to unravel? The topic seems too urgent to not tell others about, even before Searching For The Central Problem in depth.

  • The frontier models are basically all trained in the US, by two bleeding-edge companies and a handful more deep-pocketed companies and also Meta. Then, Chinese labs usually distill the models, or otherwise copy advances by the US labs. China has, and makes, fewer high-end chips than the US, but they are working on building factories to make those. These factories seem like they'll be up mostly after 2027 (see "pre-2028 timescale" below). China, unlike the US, has also exercised some kinds of direct power against their own (admittedly less-powerful homegrown versions of) top-level tech billionaires.
  • I tried thinking of central cruxes to This Sort Of Thing, which crop up often in our sphere, that persist despite our otherwise often-shared ethical priors (like consequentialism) and causal priors (like materialism). Some I thought of are "demarcation problem, is this 'AGI builders vs the public' or 'LessWrongers vs non/anti-LessWrongers'?" or "different autism levels" or "conflict vs mistake theory" or "ye olde radical vs incremental". None of these are satisfying, but that doesn't mean they're necessarily wrong. History is full of one brute Aesop after another, destroying each other. A depressing amount of this can still come down to luck.
  • Frontier lab employees (at all frontier labs) have failed to unionize, even merely internally rather than across the industry. To me, this seems like a historical accident involving neoliberal ideology, US academia, and the superpowers' decisionmaking throughout the Cold War.
  • No, handful of other lefties on LessWrong, this doesn't give a shred of excuse for the failings of the political left in the USA in general and on AI safety/governance in particular. Like wealth, blame is mostly not zero-sum.
  • You work a frontier AI lab in the USA. (If you aren't, imagine that you are). Think of a scenario where China just took Taiwan. Quick, Trump has ordered your company to Do Superintelligence For The US Military! Your CEO goes along with it, and allows military observers to prevent internal planning. What do you do? Does your plan work against straightforward brute coercion? Why, precisely?
  • Large classes of strategies along the SURA-or-not continuum, good and bad, sane and non-sane, unilateralist-curse-y and not, are outside the relevant decisionmaking people and groups' Overton Windows.
  • I assume typical cognitive biases grease the wheels of all of the above.
  • I am somewhat left-leaning (or what some might call "liberatarian-center") in economic opinions, so I also assume capitalism spins the wheels at 1000 MPH and also lights them on fire (see "grease" above).
  • All of this is less relevant than the fact that Trump Is In Charge During The Maybe-Singularity, at the same time that various problems are coming home to roost on a pre-2028 timescale, such as extreme climate-change-related weather events and possible oil/food shortages. If I were a consumer-grade chatbot from like a year ago, I'd end this bullet point with "It's not AI 2027, it's Everything 2027". Since I'm a human, I think it's closer to "Multiple Risks On The 80K List 2027".

So I guess I'm overall pro-SURA-as-main-community-default-strategy, with plenty of room for technical alignment work and/or political intrigue, but with the caveat of "if your elite group can raise large amounts of money, it's default-evil-encouraging". If want to run some vanguard-like small alignment research group, I advise you to continue to do your thing, especially theoretical/serial-time/agent-foundations work (or donate to such a group, e.g. Orthogonal [1] ). If you need enough computing power to be Beholden To Capitalism about it, then you are doing the wrong thing, one way or another. I think the community can usefully "hedge" by funding an array of people/orgs with their own strategies for an array of timelines.

Overall I'm confused. Luckily, if the rest of the debate is any indication, the comments will frequently have superior frames to the posts. Please discuss and keep it cruxy!

  • If you want to fund me in particular, or more-talented people I know, in this kind of research work, DM me. Yes, this includes if you're broadly anti-SURA or otherwise uncomfortable with this post, see "hedge" above. This goes doubly-so for our political differences, see discussions of incentives elsewhere in this debate. ↩︎

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by LessWrong
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗