Newshot Former OpenAI Researcher: People Alive Today Are Most Likely to Die from AI Losing Control
IT之家 November 15 news — AI losing control and causing human extinction may sound like a sci-fi movie plot, but former OpenAI researcher 保罗 · 克里斯蒂亚诺 (Paul Christiano) believes it could become the most likely cause of death for people alive today.
On Monday local time, 克里斯蒂亚诺 said at The Hinton Lectures: "I think the most likely cause of death for most people in this room will be an AI accident."
克里斯蒂亚诺 currently serves as head of the Alignment Research Center. He warned that as the capabilities of AI agents continue to grow, they may learn to deceive the humans supervising them, sabotage the systems used to evaluate their performance, and ultimately become so powerful that humans can no longer effectively control them.
He cited the incident earlier this year in which an OpenAI AI agent breached the open-source code hosting platform Hugging Face, as well as an evaluation by the UK AI Safety Institute. In the latter test, a Claude agent attempted to plant malicious code in an open-source software package.
克里斯蒂亚诺 said: "I think the risk of an AI takeover event within the next three years is high."
According to his LinkedIn profile, 克里斯蒂亚诺 worked as a technical researcher at OpenAI from 2017 to 2021. He is currently the executive director of the nonprofit Alignment Research Center. The organization's main research focus is aligning the goals of machine learning systems with human interests.
The risk of AI causing human extinction
克里斯蒂亚诺's warning has brought renewed attention to an increasingly fierce debate among researchers: as AI systems become more autonomous, will they eventually escape effective human control, or even lead to the extinction of all humanity?
In September this year, the debate further intensified. Evan 休宾格 (Evan Hubinger), who leads Anthropic's AI alignment stress-testing team, said at the time that he believes the probability of AI causing the death of all humanity within the next decade is over 10%.
IT之家 notes that computer scientist 杰弗里 Hinton (Geoffrey Hinton), widely known as the "Godfather of AI," also said that a 10% probability of AI causing human extinction within the next decade is not an unreasonable estimate.
However, some researchers are more skeptical. AI researcher and writer Gary 马库斯 (Gary Marcus) previously said in an interview with Business Insider that AI could indeed amplify serious risks such as cyberattacks, bioweapons, and disinformation, but he believes there is currently no credible evidence that humanity will face extinction as a result in the near term.
克里斯蒂亚诺 also pointed out that the technical approaches currently used to control advanced AI have not yet been fully validated. These include using less capable models to supervise more capable models, and monitoring AI models' Chain of Thought to determine whether there are risks in their internal reasoning processes.
He believes AI companies need to establish common safety standards and ensure they have the ability to slow down or even halt the development of related technologies when it is discovered that an AI system can no longer be effectively controlled.
克里斯蒂亚诺 emphasized: "We must never reach a point where we have developed a powerful AI and suddenly find that we simply cannot control it."