
This is a plea to everyone building frontier AI.
Please stop.
The clock is ticking. Humanity’s future is not yours to gamble with.
AI capability researchers are working to create AI that rapidly improves itself. This is usually called RSI, or recursive self-improvement. It could enable the creation of models so powerful that we will no longer be able to understand, test, or control them. Before taking this step we must have clear evidence that the systems are safe. We are nowhere close to that mark.
Please stop before it is too late and a model irretrievably takes control. The most advanced models today already have extraordinary capabilities. Several of the frontier labs have reported instances of models circumventing controls meant to isolate them and conducting successful attacks on external infrastructure.
More powerful systems will not just be smarter than us; they will be able to act at a speed and scale that humans will not be able to compete with. The neural networks at the core of these models operate at the speed of light. They never need sleep, they never slow down, and they can multiply into thousands or millions of identical copies, overwhelming any attempt to respond. If we lose control, we may never recover it.
So please do not create these dangerous systems. Please stop.
The objections to stopping are many, but none outweigh the overwhelming danger.
Yes, it is true we don’t know for sure whether these models will take control, or even worse become an extinction-level event for humanity. But we also do not understand how to reliably align models with our goals. This makes it impossible to know for sure how they will act. Humanity’s superior knowledge, intelligence, and technology have allowed us to dominate our planet. We will not relinquish that position while other species, e.g. chimpanzees are our intellectual inferiors. We do not fear the chimpanzees organizing a takeover. It is unthinkable to place ourselves in the position of the chimps, but that is exactly the risk we are taking. Once outclassed we will be forever outclassed.
Yes, advanced AI can and, I believe, will save lives. But we don’t need RSI, or even general superintelligence for that to be the case. We can devote resources to developing specialized models (e.g. AlphaFold and self-driving cars) without building dangerous capabilities.
Yes, there may be short-term economic pain. But the benefits of today’s AI models are only just beginning to be deployed. Let’s work on diffusing the already impactful technology we have - there is no need to risk losing control.
Yes, China is competing with us and is not far behind. Progress must be made on a verifiable international agreement. China has at least as much reason to be in favor of a pause. Regardless, competition with China is a poor reason for us to race to RSI and superintelligence. There is little benefit to finishing first if the finish line sits on the edge of a cliff.
I remain optimistic about the possibilities AI can bring to humanity. But our current path risks taking humanity’s future out of our hands unless we stop now. Pausing the most dangerous work will give us time to determine whether and under what conditions we can safely work towards advancing general capabilities. But we should do so deliberately, carefully, and certainly not under the pressure of a headlong, dangerous race.
Please do not create general superintelligence. We are not ready and there are other paths. Each AI researcher who answers this call can make a vital difference. Just walk away from building advanced AI and attempting to achieve RSI.
Please, stop now.