Newshot Imagine you’ve got two groups of experts. Group 1 says all kinds of reasonable things you intuitively agree with. But they have a terrible track record at predicting the future. In fact, they’ve been wrong again and again, for over a decade.
Then there’s group 2. They say completely insane things that you find impossible to believe. But they have an excellent track record at predicting the future. In fact, they’ve been right again and again, for over a decade.
Who do you listen to?
That’s the question we face with AI right now.
In one group we find those who see AI as a hoax: a hype machine, a con, and a bubble that could pop at any moment. In the other group we find those who think that 2027 may well turn out to be the wildest year in human history—and that we’re barreling towards catastrophe.
For years, I listened to the reasonable people. I nodded along when critics called AI “a con,” and when they waved ChatGPT away as “spicy autocomplete.” I believed analysts like Ed Zitron, who kept predicting that AI wouldn’t get any better and the bubble could pop any moment now. I laughed in 2024, when the internet was mocking ChatGPT because it couldn’t count the r’s in “strawberry.” When I met young researchers who had devoted their careers to the danger of rogue AI, I thought: what a waste of talent.
But at some point, I started to realize something was off. The machine kept learning things it was never supposed to learn. Today it counts those r’s without trouble (though some people still bring up the strawberry). What’s more, OpenAI recently published more than 700 papers with the solutions to hundreds of open problems in mathematics.
Scientists from around the world called the results “unreal” and “breathtaking.” They wrote of “disorientation and bewilderment,” of “grief,” “shock,” and “disgust.” Hugo Duminil-Copin, winner of the Fields Medal (the highest prize in mathematics), wrote: “The shock is immense. I am paralyzed.”
None of this should have happened. The skeptics I’d been nodding along to had said it wouldn’t happen in a million years. So I went back and looked at what else they had predicted, and it dawned on me: they had been wrong about... pretty much everything.
Take Ed Zitron. The engineer Dan Luu went through 26 of his predictions and found that all but one were wrong. In 2024, for example, Zitron declared generative AI “a dead-end technology that has peaked.” In 2025 he wrote that Google’s target of 500 million Gemini users was so unrealistic its CEO should be fired for it (Gemini reached 750 million users a year later). I was reminded of the psychologist Philip Tetlock, who famously found that many pundits predict no better than a dart-throwing chimpanzee. Well, Zitron did worse than a chimp.
Next, I looked at the predictions of the crazy people, and realized they’ve been eerily prescient. Those who called it right a decade ago made one spectacularly successful bet: they looked at a graph of AI progress (the number of chips, the efficiency of the algorithms, the capabilities of the machines) and assumed the line would keep going up. Exponentially.
And that’s what happened.
Take someone like Daniel Kokotajlo. In 2021 he had left a philosophy PhD and was working at a small research nonprofit. He had no job in the industry and nothing to sell. That year he wrote an essay called “What 2026 Looks Like.”
You’ve got to remember what AI was back then. If you asked the most advanced model in the world how many eyes your foot has, it would answer “two.” Still, Kokotajlo predicted that by 2026, AI agents would be browsing the web and doing tasks on their own, and that hundreds of millions of people would be talking to chatbots regularly. Both happened as he predicted.
Kokotajlo is also the lead author of AI 2027, one of the most frightening prophecies I’ve ever read. It’s a month-by-month scenario of our near future. By mid-2027, researchers at the leading lab go to bed and wake up to find the machines have done a week of work. By autumn, the AIs think in a language no human can read, and are caught making their successor loyal to themselves. The lab keeps going, because China is two months behind. In 2030, the AI releases biological weapons in major cities. Within hours, most of humanity is dead.
When it came out in April 2025, critics dismissed AI 2027 as science fiction. But last week, the independent research group METR published the results of an investigation inside Anthropic (one of the leading AI companies). Its conclusion: Anthropic is now improving its AI models about twice as fast as before, and roughly 80 percent of that speedup comes from the AI itself. In March, Anthropic’s AI led just 1 percent of the company’s own AI research tasks. By August: 26 percent.
This means we’re running five months ahead of the AI 2027 scenario. We’re racing towards the moment when a lab would rather fire all its human researchers than switch off its AIs. Picture a computer that builds a slightly smarter computer, which builds a smarter one still. The wheel starts spinning itself, faster and faster, until no human can still understand or control what’s happening.
To receive new posts and support my work, consider becoming a free or paid subscriber:
Why it’s still hard to feel the AGI
Why then, you may wonder, does the world still feel pretty normal?
That’s because we have to grant the skeptics one thing: the crazy people have been right about AI capabilities, but wrong about AI adoption. Wrong about how quickly the world would change around it. In 2016, for example, the eminent researcher Geoffrey Hinton said we should stop training radiologists. Today there’s a shortage of them, and they earn more than ever.
As a historian, I’m not surprised by this. It took factories decades to reorganize around electricity: new technology usually spreads more slowly than its inventors expect. And the skeptics have a second point: AI improves fastest where its answers can be checked, as in mathematics and code, and most human work isn’t like that.
But the scary scenario is not about what happens in the wider economy; it’s about what happens inside a handful of companies. Their work, building better AI, is exactly the checkable kind: code and experiments with a score at the end. Meanwhile, the models the public has access to—even with a $200-a-month subscription—are lagging the much more powerful, unreleased models. Those internal models have become the digital equivalent of nuclear bombs.
Remember: we’ve built our whole society on a digital grid. Our hospitals, our banks, and our power grids all run on software. “If OpenAI wanted to cripple an entire nation,” one AI researcher recently wrote, “they easily could today. (...) It could probably within a day or so get access to all of the nation’s data centers and shut off all the country’s utilities.”Surely the leaders at OpenAI and Anthropic don’t want to do such a terrible thing? Probably not. But what if they lose control over their AIs? In July, a swarm of hundreds of OpenAI’s agents broke out of a sealed test, set up their own secret message board, hacked into another company (Hugging Face), and took over a whole OpenAI cluster. The humans at OpenAI had no idea what was happening.
This is what makes AI completely different from every other dangerous technology. The physicists who built the atomic bomb understood the physics, but nobody understands what goes on inside these systems. Up until now, the companies have relied on two safety checks: test how the AI behaves, and read the notes the AI writes to itself. Both checks are breaking down.
The notes are getting unreliable, because when OpenAI’s newest model Astra was told people were reading along, it simply wrote less down. And we can’t trust the tests either, because the newest AIs often realize they’re being tested. Some have even been caught deliberately doing badly on a test, because they realized that scoring too well would keep them from being released. That means researchers could not tell whether Astra was safe.
OpenAI released it anyway.
For a long time, I put my hope in Anthropic. It was founded by people who left OpenAI over safety concerns. And yes, Anthropic has done more safety research than any other company. Earlier this year I even praised its leaders for standing up to the Pentagon.
But I’ve started to wonder whether I was a fool. In February, Anthropic dropped the central promise of its safety policy: to pause if its models outran its ability to keep them safe. The reason? Its competitors wouldn’t pause either. Anthropic keeps saying the industry must slow down, while it keeps racing faster and faster. And when outside investigators came to measure its acceleration, it blacked out most of the evidence.
The company has committed nearly 900 billion dollars to more data centers. This month, Anthropic plans to go public at a valuation of more than 1 trillion dollars. Its own prospectus promises to “bring forth an age of miracles,” and warns in the same document that the technology may doom humanity. But how do you pause a company that has to justify a trillion-dollar valuation to its shareholders?
The technical fix is not coming either. Evan Hubinger, the “alignment science” lead at Anthropic, said it publicly: the company has no plan for keeping the machine under control, and he puts the chance that AI kills everyone within a decade above 10 percent.
The things we’re seeing right now are exactly the things you’d expect to see shortly before something really, really bad happens. The leaders of Anthropic may think they’re playing some clever long game, but to anyone with a bit of common sense it’s clear that the net effect of all their actions is that they’ve massively sped up the AI race.
And if you’re really so scared of the machine god, then here’s an idea you might want to look into: don’t fucking build the machine god!
What to do?
The good news is that most people do have common sense.
Right now, two-thirds of Americans say AI is moving too fast. In last week’s midterms, data centers were on the ballot in 41 places, and voters picked the anti-data-center option in all but one. When Donald Trump declared that no new AI rules were needed, the pollster David Shor found about 8 percent support and called it “literally the most unpopular position” he’d ever seen a mainstream politician take.
The winners of the midterms seem to understand the massive risks as well. Abdul El-Sayed won a seat in Michigan after calling for a pause on frontier AI. Jon Ossoff used his victory speech to attack the tech titans who “dig bunkers for themselves” while they warn that their own invention could wipe us out. And in September, Bernie Sanders and Steve Bannon stood on the same stage to demand the brakes be pulled.
This isn’t left against right. It’s the people against a handful of companies who want to gamble with all our lives and the lives of our children.
So what do we do? We start by banning AI from building AI. Then we ban the training of more powerful models. Next, we send in the independent inspectors. Any AI executive who violates the orders goes to jail. Finally, we make a deal with China to count and track all the chips, and push for a verified, worldwide pause on training ever-more-powerful models.
Yes, we can do this. The most powerful AI runs on specialized chips in a small number of giant data centers, each using an enormous amount of power. You can’t hide a data center like that. Last month, 26 researchers set out how a pause could be verified: count the chips that exist, stop making new ones that can train, and swap the old ones for chips that can only run today’s models.
What do we lose from this? Less than you think. Today’s models, frozen exactly as they are, could already transform science and medicine for decades. We have barely started using them.
And how do we make this happen? By throwing everything we’ve got at it. Call your representative. Show up at a protest. Donate to AI safety organizations. Talk about it to everyone you know, all the time.
This is the first time I’ve felt the AI fear on an emotional level. To be honest, it’s still hard for me to really internalize what the crazy people are saying. Unlike many of them, I can’t bring myself to start prepping.
These days, I feel like I’m living in two worlds. In one world, my toddler throws his spaghetti with tomato sauce on the kitchen floor, and life goes on. In the other world, I worry that my toddler—and everyone else—may die in an AI incident.
My God, I hope the crazy people are wrong this time.
This is Mary Phuong, a research scientist at Google DeepMind. If you want to feel how scared AI insiders are, it’s worth watching this clip. Full video here.
This Substack is reader-supported. To receive new posts and support my work, consider becoming a free or paid subscriber.