Newshot

Thoughts on "safety bad bc warning shots good"

Summary

Suppose there was a button which made the current generation of AIs more safe, i.e. more aligned and less capable of subverting control measures. The button has no other effects — in particular, the button wouldn’t make future generations more or less safe. Should you press the button? Maybe you shouldn’t, because of the Warning Shot Argument : Future agents contribute almost all the risk of catastrophe, compared with current agents. Pressing the button makes current agents safer. If current agents are safer, then they are less likely to cause non-catastrophic harms, e.g. cybercrime. If current agents don’t cause non-catastrophic harms, then people will be less aware about catastrophic risks, e.g. lab leadership, governments, legislators, international community, lab employees, general public, media, academia, etc. If people are less aware about catastrophic risks, then they won’t take drastic action to reduce catastrophic risk. So future agents contribute more catastrophic risk. There…

Original text(via LessWrong)

Suppose there was a button which made the current generation of AIs more safe, i.e. more aligned and less capable of subverting control measures. The button has no other effects — in particular, the button wouldn’t make future generations more or less safe. Should you press the button?

Maybe you shouldn’t, because of the Warning Shot Argument:

  • Future agents contribute almost all the risk of catastrophe, compared with current agents.
  • Pressing the button makes current agents safer.
  • If current agents are safer, then they are less likely to cause non-catastrophic harms, e.g. cybercrime.
  • If current agents don’t cause non-catastrophic harms, then people will be less aware about catastrophic risks, e.g. lab leadership, governments, legislators, international community, lab employees, general public, media, academia, etc.
  • If people are less aware about catastrophic risks, then they won’t take drastic action to reduce catastrophic risk.
  • So future agents contribute more catastrophic risk.
  • Therefore pressing the button would increase catastrophic risk.

This might seem very abstract, but consider a concrete example. Suppose you’d pressed the button to make OpenAI’s HPIM more aligned and less capable to subvert control measures. Then the HuggingFace incident would never have happened. And that probably would’ve increased catastrophic risk.

Is this self-defeating?

The story is a little suspicious. In particular, it seems to account the second-order effect, i.e. “more safety -> fewer warning shots -> less awareness -> less safety”. But ignores the higher-order effects like “more safety -> fewer warning shots -> less awareness -> less safety -> more warning shots -> more awareness -> more safety”.

Now, maybe the second-order effect is bigger than the first-order effects and all higher-order effects. But it seems like that would involve pretty messy quantitive modelling that no one has grappled with.

For example, it seems very plausible that the HuggingFace incident actually increased catastrophic risk, because it caused labs to improve their control measures, therefore avoiding the bigger warning shots that would’ve occurred three months later.

My stance towards this is something like cluelessness? So I don’t put much stock in it.

I made this diagram before the METR/Anthropic timelines report, so it places takeover in 2029.

Other considerations

  • Open-weight models will be a source of warning shots. Now, the open-weight models are like 6 months behind the frontier. So the questions is what contributes more warning shots — open-weight models being misused, or closed-weight models being misaligned. Seems maybe like there’s a three month gap here — i.e. closed-source models contribute as much warning shot as open-source models three months later.
  • Current AIs seem useful for reducing catastrophic risk.In March 2025, Ryan Greenblatt wrote AIs at the current capability level may be important for future safety work. And that was more than 18 months ago. Current models are way more important for future safety work. In particular, they are useful for monitoring, grading, engineering, research, etc. In the HF investigation, he used huge amounts of AI labour for analysing the transcripts.
  • In March 2025, Ryan Greenblatt wrote AIs at the current capability level may be important for future safety work. And that was more than 18 months ago. Current models are way more important for future safety work. In particular, they are useful for monitoring, grading, engineering, research, etc. In the HF investigation, he used huge amounts of AI labour for analysing the transcripts.
  • Demonstrating capabilities might be sufficient for awareness. Maybe we could raise awareness by demonstrating that the models could cause a catastrophe. For example, via penetration testing, scary demos, and capability evaluations.
  • Warning shots are scarier if they happen despite our best efforts to mitigate them. HF was scary because they escaped our sandboxes. OpenAI could’ve simply not used sandboxes whatsoever. But that would’ve made the incident less scary.

Overall, my best guess is that you should press the button. But I wouldn’t be shocked if an oracle told me I shouldn’t.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by LessWrong
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗