Newshot

Hillclimbing AISafetyResearchBench

Summary

AI labs' main alignment strategy is using AIs to uplift AI safety research. This way, as AIs get smarter (and likely harder to align?), we also enhance our safety-research capabilities and our understanding of " Misalignment Science ". One way to use AI to speed up AI safety research is to delegate " number-go-up tasks ” to AIs. Number-go-up tasks are problems with a numeric, easily verifiable output, so anyone can measure how well a task was done. This makes them suitable to delegate to AIs (even potentially untrusted AIs). Some examples: auditing games, “ARC-like” white-box challenges, jailbreaks robustness, various small model post training challenges or RL from sparse reward [1] , etc. Fully automating number-go-up tasks isn't sufficient to fully automate AI-safety research. Human researchers still have to decide which metrics are worth maximizing and why. Some problems are also harder to turn into number-go-up tasks at all, such as some flavors of adversarial robustness, or whethe…

Original text(via LessWrong)

AI labs' main alignment strategy is using AIs to uplift AI safety research. This way, as AIs get smarter (and likely harder to align?), we also enhance our safety-research capabilities and our understanding of "Misalignment Science".

One way to use AI to speed up AI safety research is to delegate "number-go-up tasks” to AIs. Number-go-up tasks are problems with a numeric, easily verifiable output, so anyone can measure how well a task was done. This makes them suitable to delegate to AIs (even potentially untrusted AIs). Some examples: auditing games, “ARC-like” white-box challenges, jailbreaks robustness, various small model post training challenges or RL from sparse reward[1], etc.

Fully automating number-go-up tasks isn't sufficient to fully automate AI-safety research. Human researchers still have to decide which metrics are worth maximizing and why. Some problems are also harder to turn into number-go-up tasks at all, such as some flavors of adversarial robustness, or whether a safety technique will keep working on smarter models. But automating number-go-up tasks could free up human researchers to focus on that kind of work: building threat models, designing new metrics, interpreting surprising results, etc.

AI labs are laser-focused on using AI to accelerate AI capabilities R&D (RSI). They do that by automating more and more number-go-up capabilities tasks. RSI benchmarks exist, and some data companies specifically sell RSI tasks. If we want AIs to improve at AI safety research just as quickly, the community should build more benchmarks for it and take them more seriously. "If you can't measure it, you can't improve it."

Labs are already building / buying AI-safety-research tasks and internally measuring such abilities. There are at least two small data companies that are specifically focused on such tasks. There are some benchmarks for specific aspects of AI safety research (e.g AuditBench and SAEScientist-Bench) but nothing large, popular and all encompassing. We should make a large and all encompassing call it: AISafetyResearchBench or something[2]

A good AI safety research benchmark could tell us which models are good at which kinds of AI-safety research tasks. It could tell AI safety researchers which tasks they can safely delegate, and to which models. It could also help us detect whether a new model is sandbagging (or is just surprisingly bad) at AI safety research. This is not a theoretical issue: OpenAI recently revealed that GPT-6 Astra was caught sandbagging on safety research tasks. Such a benchmark would also give the labs a legible leaderboard to compete on. I can easily imagine a world where OpenAI, Anthropic, and GDM race to build the best AI safety research assistant. That race would increase demand for safety-research RL tasks, growing the task market and making it easier for new data companies to build them.

Unlike other races between AI labs, this is one we should want them to run!

  • ^many cool ideas here
  • ^A fair amount of AI safety research is dual-use: a safety breakthrough can also accelerate capabilities (RLHF is the canonical example). I'm not too worried about this here. So much talent and money already goes into capabilities research that the marginal effect of models getting better on AISafetyResearchBench is probably small relative to the benefits.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by LessWrong
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗