I'm going to protest the Anthropic IPO tomorrow, at 11am outside Anthropic's offices at 500 Howard St, San Francisco. The primary ask of the protest is for the government to ban recursive self improvement, and/or for labs to stop doing it voluntarily.
See: https://banrsi.ai/ I think Anthropic leaders and employees are extremely overconfident about their ability to manage RSI in a way that is safe. It currently looks to me like they are not only doing extremely reckless AI development, but causing others to as well (i.e. my current impression is that OpenAI would be less RSI pilled if Anthropic was less so).Meanwhile, everything I know about Anthropic political advocacy suggests the org is actively steering towards milquetoast regulations that won't come anywhere close to making the world safe enough.In the past few months, Anthropic leadership could have made different choices about what political choices to advocate for, which could have made the Anthropic strategy worthwhile. (i.e. be a leading lab, get a set at the table to meaningfully negotiate for and encourage safety). I think some Anthropic employees think "it's still not quite the time to pull out all the stops on arguing 'guys we all need to stop', and then put their money where their mouth was and stop." I will try to followup this post with clear arguments for why I think this is false. (I think, to be fair, it's not entirely obvious)I think it's salient to Anthropic employees how hard they're working on safety, and how many tradeoffs they (and the org) are making for safety. And they feel like AI safety advocates are painting them with too broad a brush. Sometimes AI safety advocates say "Anthropic people think alignment is easy." My model of Anthropic people think "what? We are working so hard! We know it's not easy."But, alas, there's a difference between "it's hard", and "it's so hard that Anthropic's strategy is fundamentally unworkable and misguided."Over the years, I have tried to engage with Anthropic on its merits and not just round them off to a stereotype. I think Dario has real convictions, is clearly willing to stand up for some important principles. I think he earnestly thinks Loss of Control risk is real but manageable, but is genuinely fearful of misuse and human takeover.My understand is that the Anthropic plan to not "try to align vast-superintelligence" but "try to race to the edge of the safely-alignable superintelligence, and then, use the strongest achievably-alignable-AI to improve the Earth's position." (I have at least talked to some employees who seemed to imply this is the plan)I think this strategy isn't completely ruled out as plausibly good. I could hypothetically imagine them still pulling it off.But, in addition to vastly underestimating how hard alignment is, I think they are vastly overestimating their organizational capacity to correctly race to the edge safely and navigate that situation. People are known to be overly optimistic about their own plans. Startup founders are known to have a reality warping bubble that makes it feel like ambitious, crazy plans are more achievable this is. Normally, this is okay, because when they fail, the company often still achieves great things, and startups on average achieve valuable things.But, this is not a situation where that level of mistake is acceptable.At this point I think Anthropic employees need a very strong reason to think "do their day job" is remotely competitive with "spend all their time and social capital either arguing with leadership to change course, or, argue to politicians that we need laws that would meaningfully stop Anthropic (and OpenAI and everyone else) from pursuing RSI.