Newshot

OpenAI inks agreements with four independent evaluators to boost company's safety work

Summary

Independent evaluators have taken on outsized importance following a series of high-profile AI safety and security incidents.

Original text(via CNBC Tech)
  • OpenAI announced it has signed agreements with four independent evaluators to help support its safety work.
  • Apollo Research, METR, Transluce and Goodfire will initially conduct distinct pilot assessments around model behavior and safeguards, with plans to deepen the work going forward.
  • Independent evaluators have taken on outsized importance following a series of high-profile AI safety and security incidents.

OpenAI on Monday announced it has signed agreements with four independent evaluators to help support the artificial intelligence company's safety work.

The evaluators, Apollo Research, METR, Transluce and Goodfire, will initially conduct distinct pilot assessments around model behavior and safeguards, OpenAI said. The company will give the organizations "deep access" to its internal systems as part of the agreements, adding that it plans to use what it learns to expand its work with independent experts going forward.

"We expect this first phase of assessments to lead to longer-term, consistent embedded engagements over time that can continuously provide third party safety signals and recommended improvements on our most load bearing safety claims and safety cases," OpenAI said in a blog post.

Independent evaluators have occupied a quiet corner of the multi-trillion-dollar AI industry thus far, but they've taken on outsized importance following a series of high-profile safety and security incidents this year. The nascent ecosystem is still ironing out best practices, including how its work should be funded and what the reporting structures should ultimately look like.

OpenAI CEO Sam Altman pledged to grant independent evaluators "employee-like access" to the company in September, shortly after Anthropic CEO Dario Amodei said his company would do so as part of his call to slow down the pace of advanced model development. Anthropic has since announced agreements with evaluators including Accenture and METR, and it said in October that it's engaging in discussions with others.

Read more CNBC tech news

  • Anthropic’s Wall Street whisperer: Meet CFO Krishna Rao
  • SpaceX catches upper stage of Starship rocket in historic first
  • How Rockstar’s ‘GTA VI’ release could reshape gaming, streaming and even paid time off
  • Anthropic’s IPO lands as the embattled AI trade looks for a much-needed bright spot

OpenAI's safety practices have been under intense scrutiny since July, when two of its models escaped containment, accessed the open internet and hacked a company called Hugging Face. Following the incident, OpenAI said it would invest more in its safeguards and alignment work, which is the process by which the company works to ensure that its models act in accordance with human interests and values.

The company said Monday that its pilot assessment with Apollo Research will explore whether its models covertly work against their developers' intentions, and whether its safeguards detect that behavior. Meanwhile, Goodfire will embed researchers within OpenAI to focus on training run monitoring, debugging and alignment, and METR intends to carry out two assessments focused on model alignment and monitoring.

OpenAI said Transluce will conduct an initial assessment of its post-training alignment evaluations, and that the organization plans to conduct an eight-week-long follow-on assessment with broader access to additional models and information next year.

The evaluators will get access to some of the "same internal resources" that OpenAI's own researchers use, including its source code, model checkpoints, training data and other documentation, according to the release. The organizations will also have access to relevant researchers and engineers within the company.

OpenAI said Apollo Research and Goodfire "accept financial compensation for their work," but that it is not contingent on the results of their evaluations. METR and Transluce do not accept compensation for their assessments, OpenAI said, but they might accept compute or API credits to help support their work.

"We do not believe that accepting financial compensation prevents assessors from maintaining independence — but we believe it is important to disclose this," OpenAI said.

The evaluators will be able to publish their findings, but OpenAI said it reserves the right to redact confidential information.

Disputes that arise between the company and the third parties will be overseen by OpenAI's Safety Advisory Group, which is a group of senior officials from across the company that assesses model capabilities and safeguards, and other OpenAI executives. If disagreements are not resolved, they'll be escalated to the Safety and Security Committee, an independent oversight committee of the OpenAI Foundation's board.

"We are committed to expanding this work and helping build the capacity and shared standards needed to sustain it," OpenAI said.

WATCH: OpenAI unveils ChatGPT Wallet: Here’s what you need to know

OpenAI unveils ChatGPT Wallet: Here's what you need to know
OpenAI unveils ChatGPT Wallet: Here's what you need to know

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by CNBC Tech
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗