
To spell out this vibe-comic[1]: Safety made the locally correct decision here at every step of a bad situation. But like in Caleb Biddulph's parable, this globally resulted in an awkward state of affairs.
From the Public's POV (penultimate panel), they would naively see AI safety as complicit in the building of the machine. If it really is a doomsday device, why would Safety involve itself at all?
(Safety in turn has a good answer: Public wasn't willing to listen back in the 2010s. Ordinary people would nod along but not lobby their politicians, and politicians would not take it seriously, because most AI experts wouldn't either.)
The realest case of a doomsday device was probably Project Sundial, a bomb so big as to destroy the Earth in one go. When Edward Teller proposed it in backroom briefings, people thought it was an "advertising stunt."
The prior on gadgets truly being doomsday devices is very low. Hence, when even Safety shows some signs of not taking it seriously, the Public doesn't either.
In the final panel, we have Safety looking at the other two characters. In the past, they would be timid about shouting too much to Public, due to Company's implicit threat of bad relations. Company has become more powerful than they were in the 2010s. But in comparison to the now half-sleeping giant, not so much anymore.
I also spent some time researching historical precedents for groups becoming socially cozy with entities they're advocating against:
- Many of the Manhattan project scientists advocated for treaties, but after they'd already built the A-bomb. Some like Oppenheimer advocated against the H-bomb, and this did not go well. But before the A-bomb most outsiders considered the idea ludicrous (e.g. "That is the biggest fool thing we have ever done. The bomb will never go off, and I speak as an expert in explosives.").
- Recombinant DNA technology was stopped at Asilomar in 1975, but this was internal to the researchers themselves, without having to involve Public. Safety tried that strategy in 2017, but back then mainstream AI experts were not bought into it. Now that pioneers like Yoshua Bengio and Geoffrey Hinton are on board, the political dynamics have become so warped that they have less voice.
- There are various examples of environmental groups working closely with the relevant industry for harm reduction. The Environmental Defense Fund is well-known for doing this openly. But environmentalism also has lots of other dysfunctional dynamics which seem to make the circumstances quite different here.
This was a very shallow review, but unfortunately the record is fairly thin, at least in terms of direct analogies.
But we could imagine that General Electric and Ford Motors were doing some large-scale nuclear fission experiments in the 1930s, before almost anyone understood the implications of this. They notice the atmospheric ignition problem, and run some calculations. At some point, they realize they just don't know what will happen, since it comes down to theories they don't yet understand. All the physicists aware of the implications are either working at the labs or closely tied. I don't know what they should do then.