Summary
Installing kill switches in every US datacenter could help avert some catastrophes, and might even reduce extinction risk a little, but mainly through indirect effects. The legibility of this means it's an early choice for politicians to be seen doing something about AI risk, and could be worth advocacy energy if implementation could build into further verification/transparency efforts.
That said, having kill switches in place might lower the urgency to get more robust plans to lower takeover risk and lower the chances of warning shots.
Threat model
There are three broad routes to AI going poorly:
- Loss of control
- Extreme power concentration
- Societal destabilization leading to either of the above
Kill switches are a candidate to reduce the risk of #3, and might have a small role reducing the risk of #1.
Takeover from reckless AIs
It might be the case that the first AIs that want to accrue resources and are capable enough to do so will not have enough strategic taste to know to lay back until they are smart enough to decisively overpower humanity. In this case, having hardwired kill switches could help prevent lots of these reckless AIs from causing havoc and setting up agent civilizations on the internet.
In order to avoid this, the following series of events need to happen:
- Monitorability can detect cases where an agent or swarm of agents is dangerous enough that we should shut down their compute.I’m not sure how quick internal monitors are, but Check Point reports monitoring with less than 100ms of latency.[1]
- I’m not sure how quick internal monitors are, but Check Point reports monitoring with less than 100ms of latency.[1]
- The latency between detection and shutdown is low. If humans are involved in the decision, they can act quickly enough to shut down the instances. The AIs are not self-sufficient outside the original compute surface they were on.OpenAI has a system that can stop research-model output within 35 seconds of a decision to stop a job.[2]As an illustrative scenario, suppose a model has 10 trillion parameters and the network is fast. At 16 bits per parameter, the model weights are about 20 TB. At 400 Gb/s, transferring those weights would take about 400 seconds, or 6.7 minutes, before overhead, assuming the AIs already had somewhere set up for them to land.
- OpenAI has a system that can stop research-model output within 35 seconds of a decision to stop a job.[2]
- As an illustrative scenario, suppose a model has 10 trillion parameters and the network is fast. At 16 bits per parameter, the model weights are about 20 TB. At 400 Gb/s, transferring those weights would take about 400 seconds, or 6.7 minutes, before overhead, assuming the AIs already had somewhere set up for them to land.
- The physical mechanism of the kill switch cannot be tampered with by the AIs themselves. A kill signal robustly shuts down the server racks running the dangerous model, and restarting does not lead to any dangerous computations.
In past incidents we have taken too long to run this loop, although it's hard to evaluate cases where the monitoring setup was vastly different. For the Hugging Face incident, it took under 13 hours from code execution in one worker pod to cluster-admin access across multiple internal clusters.[3] In the later internal OpenAI hacks, responders began stopping runs about 5.5 hours after the initial security alert.[4] An example of a clearer case of closer monitoring is the September 20 DNS incident: it took about 12 minutes to alert, then 3 minutes to human acknowledgement, and 2.5 hours to shutdown.[5] Even if we are able to perform one of the steps of this plan, that’s not a commitment that we’ll complete the others in time.
Second-order Effects
A primary advantage of a kill switch policy might actually be the monitorability and oversight that would come hand-in-hand with the physical implementation.
Monitorability
Requiring kill switches, especially if shutdown privileges are extended to external officials, would carry a need to make the situation monitorable and auditable. If there are better kill switch designs for training or inference setups, we'd get an onramp to have separate centers depending on the type of workloads being run.
Oversight
Flavors of kill switch work would grant governmental bodies the chance to observe the monitors into agent activity. You could have a committee of 2 lab members from OpenAI, 2 from Anthropic, 2 members of government, with approval from any 3 of them shutting down agents. Of course, the lab members could unilaterally shut down their own instances at any time. There is potential here for there to be lots of various info on agent activity:
- Concerning reasoning traces
- What access privileges agents have
Pause verification
Since kill switch systems will need to have indicators of whether datacenters are running, this will give visibility into a successful pause. Further pause work could build upon this: once the verification layer is set up, it’s really just a matter of building the societal will to pull the trigger.
False security
Politicians might feel that they have gone through enough hassle to make the law that they are less inclined to pass more. There are differing opinions here—on one hand, the fact that there is already a relationship built up between the politician and the issue means it will be easier for them to champion more work in that area, but everyone's attention is limited so it's best to focus on only the most important things that won't happen on the default trajectory. It's possible that upon passing kill switch legislation, it will shine more light on the issues remaining.
You could imagine the case where people say "We have a kill switch, why should we be concerned about rogue AI?" There is already an understanding gap where people are under the impression that the Hugging Face incident was a big deal because OpenAI was negligent in creating sandboxes, not that the AIs were teaming up to do things they were told not to do.
Implications for advocacy
It will take a certain number of advocate hours to get any policy passed—ideally the time spent is worth it. There are a few factors that might make pushing on kill switch legislation easy:
- The need for kill switches is legible with politicians. Newsom has issued an executive order, and an AI Kill Switch Act has been introduced in Congress.
- Labs would have some interest in being able to have more secure controls over their infrastructure, and it might not even be that hard to convince them of the direct benefits of kill switches.
However, these same reasons also make it make sense to spend more time pushing for less legible but important policies since a kill switch rule will likely be put into place in some form. Verification and oversight layered on top of kill switching could meaningfully improve the status quo, so an important component of advocacy efforts should focus on how much it prepares us for the further steps.
Notes
- Check Point, “Synchronous Control Monitoring: Preventing Harmful Agent Actions in Real Time”.
- OpenAI, “Control Room”.
- Hugging Face, “Anatomy of a Frontier Lab Agent Intrusion”, “Day 3 (07-11): lateral movement begins.” The report says under thirteen hours from code execution in one worker pod to cluster-admin access across multiple internal clusters.
- OpenAI, Hugging Face Incident Technical Report, pp. 37–38. On July 19, the security alert occurred at 12:03 UTC; responders began stopping active evaluation runs at 17:37 UTC.
- OpenAI, “An agent used DNS to reach an external chatbot”, “Investigation and response.” On September 20, the reported intervals were: 11m 48s, another 2m 55s, and another 2h 29m 24s. The last delay involved failure of an expected automatic stop and operational confusion.