Newshot

AI is getting better faster than experts can reliably test whether new systems are safe, METR warns

Summary

METR reports that leading AI models like Opus 5.5 and GPT-6 Astra are improving rapidly at research tasks but getting harder to measure. Current data can't rule out a dangerous acceleration in AI development, the organization warns, though it also says common tests may overestimate what the models can actually do. The article AI is getting better faster than experts can reliably test whether new systems are safe, METR warns appeared first on The Decoder .

Original text(via The Decoder)

AI is getting better faster than experts can reliably test whether new systems are safe, METR warns

AI models are rapidly improving at AI research tasks, and they're getting harder to measure. According to the AI research organization METR, current public data isn't enough to rule out a drastic acceleration in AI research over the next few years. That could leave humans unable "to meaningfully understand, assess, and react to catastrophic risks posed by new AI systems," the organization warns.

On clearly defined coding and computational tasks with direct feedback, the models already beat experienced human professionals most of the time, METR found, though they haven't yet outperformed large groups of top experts. For tasks that take humans longer, the task length the models can handle doubles every 3.6 months. A naive extrapolation of METR's data puts them at a 50 percent success rate on the longest, 200-hour tasks by July 2027.

The systems still struggle with open-ended questions that lack immediate feedback and require real judgment, as well as with situations that are hard to recreate for training, according to METR. The organization also warns that common, easy-to-run tests may overestimate how well the models perform in real-world use, since developers train them in very similar test environments.

THE DECODER+

AI News Without the Hype – Curated by Humans

  • Every Plus article and the full archive
  • No ad banners
  • Frontier Radar: 6 deep dives a year

Already a subscriber? Sign in

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by The Decoder
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗