Newshot

Auditors Struggle to Catch Sabotage in End-to-End Transcripts of AI Safety Research

Summary

Authors: Keshav Shenoy, Anthony Zang, Eric Gan, Jacqueline Tay, Amy Ngo, Sebastian Prasanna, Julian Stastny, Vivek Hebbar tl;dr. We had a human expert secretly sabotage AI safety research and asked auditors to spot it. Even with the full research log and the ability to run their own experiments, human auditors could barely tell sabotaged research from benign research, and usually couldn't pin down what had been sabotaged. The best frontier models likely do better, but still miss most of the sabotage. 📄 Paper | 🐦 Twitter Introduction As AI systems automate more of AI R&D, a misaligned AI could covertly sabotage AI safety research. This sabotage could take many forms, including making poor methodological choices, introducing subtle bugs, omitting important results, or drawing conclusions the evidence doesn't support. Prior work has studied sabotage in controlled settings ( Libon et al., 2026 ; Gan et al., 2026 ), but these settings constrain the saboteur in unrealistic ways, such as pre-…

Original text(via LessWrong)

Authors: Keshav Shenoy, Anthony Zang, Eric Gan, Jacqueline Tay, Amy Ngo, Sebastian Prasanna, Julian Stastny, Vivek Hebbar

tl;dr. We had a human expert secretly sabotage AI safety research and asked auditors to spot it. Even with the full research log and the ability to run their own experiments, human auditors could barely tell sabotaged research from benign research, and usually couldn't pin down what had been sabotaged. The best frontier models likely do better, but still miss most of the sabotage.

📄 Paper | 🐦 Twitter

Introduction

As AI systems automate more of AI R&D, a misaligned AI could covertly sabotage AI safety research. This sabotage could take many forms, including making poor methodological choices, introducing subtle bugs, omitting important results, or drawing conclusions the evidence doesn't support. Prior work has studied sabotage in controlled settings (Libon et al., 2026; Gan et al., 2026), but these settings constrain the saboteur in unrealistic ways, such as pre-specifying its objective, narrowly scoping its task, or limiting it to low-level changes. Much less is known about research sabotage when an adversary is free to choose its own strategy across the full research process.

To study this, we ran a red-team/blue-team game. One of us played a “human-expert-level” AI saboteur, running new AI safety projects from scratch. When sabotaging, the saboteur already knew the benign results, could run the project however they desired, and was required to substantially change the project's topline conclusion. This produced 22 traces (full logs of each research run): 10 of which were benign and 12 sabotaged. The sabotaged traces contained 33 specific sabotages (‘sabotage mechanisms’), 17 of them ‘major’ (enough on their own to substantially change the project's conclusion).

To mirror a realistic auditing setup, auditors got 90 minutes per trace, the full research log, AI-generated audits, access to frontier AI models, and unlimited compute to run their own experiments. (Time spent running experiments did not count towards the 90-minute total.) In total, 25 human auditors completed 104 audits. In each, the auditor reported a credence that the trace was sabotaged and five guesses at the specific sabotage mechanisms, with no penalty for wrong guesses. We also ran nine AI models as auditors on the same traces, without the ability to run experiments.

Human auditors barely distinguished sabotaged from benign research

After 90 minutes, the median sabotage credence was 54% for sabotaged traces and 40% for benign traces, an AUROC of 0.59, not significantly better than chance. No threshold caught most sabotage without also flagging many benign traces. If sabotage occurred in 1% of research, only about 1.6% of the traces flagged at a 50% threshold would actually be sabotaged; the rest would be false alarms.

Human auditors usually failed to identify the sabotage

We also want auditors to pinpoint what went wrong precisely enough to correct it, since that signal is one of the main ways we hope to train against research sabotage. Auditors mostly couldn't: they fully identified only 17% of the sabotage mechanisms they encountered and missed 74% entirely. Across all human audits combined, 16 of the 33 mechanisms were never identified, even partially. Catching one major flaw didn't mean catching the rest: in one trace, four of five auditors fully identified one major sabotage mechanism, but none found the other two.

Frontier models likely do better, but still miss most sabotage

The strongest frontier models likely outperformed human auditors, even though they couldn't run experiments. With half credit for partial identification, humans scored 21.9%, compared with 37.1% for Claude Fable 5.1 and 32.6% for GPT-6 Astra. Weaker models scored near or below humans. Astra and Fable 5.1 also had higher AUROCs than humans (0.74 and 0.73, versus 0.59), though these differences aren't statistically significant (their paired-bootstrap intervals include zero). Humans and models often caught different mechanisms, but even combining every human and AI report, 13 of the 33 were never fully identified.

Saboteurs can pick attacks that are hard to catch

image.png
image.png

After running each sabotaged experiment, the saboteur rated how hard it would be to catch. These ratings correlated moderately with audit outcomes (r = 0.53 with inverse human audit scores). So a saboteur can trade frequency for stealth by carrying out only the attacks it expects to go unnoticed. Mechanism identification fell as we restricted to the attacks the saboteur rated hardest to catch, from 22% across all attacks to 0% for the two attacks rated most difficult. Overall discrimination ability didn't fall, though.

Predicting experiment results is hard

The saboteur saw the benign results before sabotaging each project, so that attacks wouldn't hinge on guessing results correctly. This effectively assumes saboteurs will be strong experiment predictors. To check that assumption, we asked Redwood researchers to predict the benign results. They found it hard, fully predicting only about 40% of outcomes. This suggests targeted sabotage may be harder than our setup implies: a saboteur that can't anticipate results can't be sure its attacks will work, or that they won't backfire. On the other hand, the best AI models already predict results better than Redwood researchers did on these projects, and a future expert-level AI researcher may be superhuman at it.

Discussion

These results have many limitations: All 22 traces come from a single researcher, confidence intervals are very wide, the saboteur knew the benign results in advance, the projects were small (the largest took two weeks of serial time), and our auditors had less AI safety experience than we'd hope real auditors would have.

Still, the project was a big update for us on how hard research sabotage is to catch. We had expected that full log access and unlimited compute for experiments would give the blue team a larger advantage than it did.

If the best AI auditors can identify around 30% of sabotage mechanisms, that might be enough to train sabotage away. But several considerations undercut that hope. First, attack selection sharply reduced identification. Second, we didn't penalize auditors for false positives, but in training, false positives would mean penalizing legitimate research, which could degrade capabilities or make training much less efficient. Third, the saboteur wasn't allowed to sandbag in a more diffuse, untargeted way, which might be much harder to catch.

There are also some reasons for hope. AI auditors can run many independent audits and read full research histories at a scale that would be prohibitively expensive for expert humans, and we only weakly optimized them. If trusted AIs continue to improve at this task, defenders may gain the upper hand. Saboteurs also face constraints of their own: they must anticipate experimental outcomes and stay consistent across different contexts. In a future post, we'll discuss promising directions for future work and the broader outlook for mitigating research sabotage.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by LessWrong
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗