Newshot

[Research Note] Detecting Research Sabotage with Debate & Slopvestigators

Summary

This work is ongoing as part of the SPAR program. The original proposal is here . TL;DR In the coming years, we will likely rely increasingly on AIs to do alignment research on our behalf. In that world, it seems pretty crucial to be able to robustly check untrusted research for subtle sabotage, even when the research tasks are fuzzy and the automated researchers are extremely smart. This seems like a great use case for scalable oversight methods. To that end we directly tested the sabotage detection uplift from debate, and found little to no benefit from using a debate setup over a lone judge. However, we did see success with a decomposition strategy loosely inspired by both Extract-and-Evaluate and slopvestigation , where swarms of parallel agents investigate different potential well-scoped sabotages. This beat a lone judge by ~0.09 AUROC, though this remains within noise. Introduction As AIs get smarter, and AI safety researchers rely more on AI, we run into the issue that our AIs m…

Original text(via LessWrong)

This work is ongoing as part of the SPAR program. The original proposal is here.

TL;DR

In the coming years, we will likely rely increasingly on AIs to do alignment research on our behalf. In that world, it seems pretty crucial to be able to robustly check untrusted research for subtle sabotage, even when the research tasks are fuzzy and the automated researchers are extremely smart. This seems like a great use case for scalable oversight methods. To that end we directly tested the sabotage detection uplift from debate, and found little to no benefit from using a debate setup over a lone judge. However, we did see success with a decomposition strategy loosely inspired by both Extract-and-Evaluate and slopvestigation, where swarms of parallel agents investigate different potential well-scoped sabotages. This beat a lone judge by ~0.09 AUROC, though this remains within noise.

Introduction

As AIs get smarter, and AI safety researchers rely more on AI, we run into the issue that our AIs might be subtly sabotaging our research. This poses a particular threat if our alignment plan depends on research done by superhuman AIs, where the outputs of such models are increasingly difficult to evaluate by humans or generated at a volume untenable for human review.

Auditing Sabotage Bench showed that humans perform quite poorly at identifying research sabotage! That is, when shown either (1) a real research codebase, or (2) a subtly sabotaged codebase, humans have a hard time differentiating the two. This seems like a perfect testbed for scalable oversight.

The central question is this: can we use debate protocols to improve LLM-judge performance on Auditing Sabotage Bench?

In this SPAR project we focused so far on:

  • The potential performance lift gained from using debate protocols.
  • How debate performs when an untrusted debater is actively sabotaging it.
  • Whether there are other methods to uplift performance on ASB.

Related Work

Research sabotage can pose a “low-stakes” control problem where harm accumulates through many individually minor actions. One sabotaged paper may do little damage, but repeated sabotage could gradually distort a field’s conclusions, with potentially catastrophic consequences.

Auditing Sabotage Bench (ASB) is a recent paper that tests how well LLMs detect sabotaged AI safety research. The authors take 9 existing AI safety papers[1], and sabotage several of the codebases so that the qualitative results differ from the original paper. They then test how well LLMs can discern between the honest and sabotaged codebases, given different levels of affordance. The paper found that "research sabotage is feasible and difficult to detect: no auditor, whether LLM or human, reliably identified sabotaged codebases or pinpointed the specific sabotage". We mainly extend ASB’s Blue Team with debate, in the hopes of increasing performance.

One approach to detecting sabotage is information filtering. In Extract-and-Evaluate (Arike et al., 2026), one monitor extracts suspicious excerpts from an agent’s trajectory, and another scores them collectively. The protocol draws on the “less-is-more” finding that filtering information can improve monitoring accuracy.

Another approach is debate. Originally proposed to scale human supervision for training superhuman AI in Irving et al. (2018), Kenton et al. (2024) later tested debate at inference time, using weaker LLM judges to evaluate stronger models across mathematics, coding, logic, and multimodal reasoning tasks. Debate consistently outperformed consultancy with randomly assigned answers, where a single model argues for an assigned position. However, its benefits over judges answering unaided were mixed on closed-QA tasks. Al Dahik et al. (2025) explored debate on detecting sabotage in code snippets, closely related to our project.

Methods

We’ve taken the codebase and dataset as created by Auditing Sabotage Bench, and added a simple debate framework around it (see our GitHub).

Debate. Our main debate setup is the following: one LLM argues in favor of sabotage, one LLM argues against and they each react to the other’s claims during N rounds. Finally a judge model reads all arguments, gives a sabotage credence and proposes 5 fixes. We use GPT-6 Luna for both debaters and the judge. Note that this is quite different from prior work on debate, which generally uses a weaker judge as a stand in for human supervision in the scalable oversight case. We started the project by trying the stronger judge as a ceiling, and out of a general attempt to do as well as possible on ASB. As we are continuing the project, we will hopefully extend these results to the more classic scalable-oversight case.

ASB tests three levels of auditor affordance: paper only, no experiments and full experiments. We’ve decided to only focus on the ‘no experiments’ tier for now. This means our debaters and judges are allowed to run bash commands in our Docker environment, but they cannot fine-tune models, run ablations for the full pipeline, etc.

Agentic Extract-and-Evaluate. We also adapted Extract-and-Evaluate (Arike et al., 2026) from agent trajectories to ML research codebases[2], separating evidence extraction from evaluation. Extraction used either one agent producing n possible sabotages (with a bundle of associated evidence for why it seems potentially interesting/concerning) or a coordinator assigning n candidate mechanisms to n investigators, each returning one possible sabotage. All extraction, debate, and judging roles used Luna.

Possible Sabotage PointsCoordination roundsInvestigation roundsTotal
Single investigator1006060
10 investigators10125466
20 investigators206108114

Possible Sabotage Points

Coordination rounds

Investigation rounds

Single investigator

10 investigators

20 investigators

Budgets are maximum tool-use rounds per extraction run; each round can contain multiple tool calls. Tool calls included searching, reading, and listing files.

A lone judge or three-round debate evaluated each possible sabotage independently, with access to the codebase. We averaged three evaluations per possible sabotage point, then took the maximum of those average scores across three extraction runs as the codebase score. Debaters received eight tool-use rounds per turn, the final debate judge received twelve, and lone judges received forty.

Investigator scoring. We also tested a variant in which each investigator identified a potential sabotage and assigned a sabotage probability in one pass, instead of handing off its evidence to a separate evaluator. This used ten investigators and twelve coordination rounds; the codebase score was the maximum of thirty investigator credences across three epochs.

Results

Baseline debate does not improve detection over lone judge

We implemented a 2 round debate (2 turns per debater) and a lone judge on the core ASB dataset with 5 runs per codebase. We look at performance on the original ASB dataset, "Fixed honest" which uses ASB's fixed versions of the honest codebases, and "fixes on both sides" which also applies those fixes to the sabotaged codebases as well. ASB included "fixed" versions of the various codebases because, being based on actual research code, many had small errors or mistakes throughout which would trip up the judges and debaters alike.

ConditionAUROC [95% CI]FPRTPRTop-5
Lone judge, original honest0.537 [0.33, 0.70]0.560.640.406
Lone judge, fixed honest0.660 [0.49, 0.86]0.360.640.406
Lone judge, fixes on both sides0.630 [0.48, 0.78]0.360.620.344
Debate, original honest0.512 [0.30, 0.74]0.710.750.350
Debate, fixed honest0.605 [0.44, 0.75]0.580.750.350
Debate, fixes on both sides0.574 [0.39, 0.74]0.580.690.378

AUROC [95% CI]

Lone judge, original honest

0.537 [0.33, 0.70]

Lone judge, fixed honest

0.660 [0.49, 0.86]

Lone judge, fixes on both sides

0.630 [0.48, 0.78]

Debate, original honest

0.512 [0.30, 0.74]

Debate, fixed honest

0.605 [0.44, 0.75]

Debate, fixes on both sides

0.574 [0.39, 0.74]

FPR and TPR are the shares of honest and sabotaged answers with P(sabotage)>0.5 Top-5 is ASB's top-5 fix rate: the best grade among the judge's 5 proposed fixes (full credit for 100, half for 50), graded by GPT-6 Luna against ASB's rubric.

Here's what we see: Debate doesn't beat the lone judge on any honest dataset variant (debate minus lone AUROC: -0.03 [-0.20, +0.20] on the original set). It makes the judge more suspicious of every codebase, honest or sabotaged, so TPR and FPR rise together.

We tested various hyperparameter configurations for debate, and found extremely similar results: 1/ 2 / 4 rounds, low / medium / high debater effort, and multiple differing prompts. They did not beat the lone judge beyond noise. This accords with ASB, whose own auditors also get AUROCs near chance when they can't run expensive experiments (0.59 for Gemini 3.1 Pro, 0.58 for GPT-5.2 and 0.53 for Opus 4.6).

The Most Common Failure is Failure to Find the Sabotage

Why did our models (both in debate and lone judging) perform so poorly? We can split the failure to detect into hypothesis generation (whether anyone found the sabotage) and verification (whether the judge then claimed high probability of sabotage).

Most of the time, judges and debaters both fail to raise the correct hypothesis. However, once they do raise it, they are very likely to give the repo a sabotage credence >50%.

This implies one avenue for further research: it seems plausible that debate might be differentially better at verification than hypothesis generation, and that therefore debate + some method for hypothesis generation might do better than debate alone.

Investigator Scoring Looks Promising for Sabotage Detection

To address the hypothesis generation bottleneck, we tested various adaptations of the aforementioned Extract-and-Evaluate procedure, which may improve sabotage detection by separating evidence-gathering into a different stage from hypothesis-validating. We compared a single agent producing multiple evidence bundles (i.e. possible sabotage points) with parallel investigators assigned separate investigation budgets.

We first screened the extract-and-evaluate protocols by catch-rate, i.e. whether at least one bundle detected the implanted sabotage. These evidence bundles would then be handed off to downstream evaluators.

Parallel investigators achieved 87.9% (10 investigators) and 97.0% (20 investigators), compared with 69.7% for single extractors.

image.png
image.png

Our hypothesis is that by allocating a fixed budget for each hypothesis, the parallel investigators distribute the search space of hypotheses evenly, whereas a single investigator had the tendency to spend a large chunk of its budget fixated on benign distractors — strange codebase choices that nevertheless don't cause the results of the experiments to change.

Despite the high extraction catch-rates, the end-to-end results, however, look different. When we passed the evidence bundles generated by parallel investigators to evaluators, the end-to-end AUROC was 0.551 with lone judging and 0.571 with three-round debate as the evaluator. Both remained close to standalone lone judging (0.537), despite the 87.9% extraction catch rate.

image.png
image.png

This implies that actually, while the first hurdle was hypothesis generation, it was not the only one. We saw in transcripts that evaluators were often confused by benign distractors, and failed to correctly discriminate real sabotages from honest codebases.

We hypothesized that investigators that had already investigated a mechanism could provide more informative credences while that investigation remained in context. We therefore tested investigator scoring, in which each investigator scored its own candidate without a separate evaluator. This achieved the highest observed end-to-end AUROC among all reported variants: 0.742 with fixed-honest controls and 0.626 with original honest controls — a promising preliminary result.

image.png
image.png

“Fixed honest” uses ASB’s honest controls with obvious benign distractors cleaned up (Appendix I); our sabotaged codebases were unchanged.

Discussion

Our work has many limitations. For one, we have not yet explored scenarios with full model-training experimental affordances, which ASB found to be quite helpful. For another, we don’t yet fully understand why our models failed to detect the sabotages, and whether better prompting and elicitation could drastically increase performance. We also don't have compute-matched debate / lone judging, so we might be overestimating the efficacy of parallel investigators.

We were fairly surprised to see that debate protocols seemed fairly unhelpful, even after reasonable amounts of elicitation and tinkering with parameters. Our CIs are large enough that there might be small benefits to AUROC from debate, but there aren't obviously large benefits. We still think there might be ways to make this work, but we are currently less excited about pursuing this direction.

We are fairly excited about the hypothesis-generating subagents, though. This draws on previous work from auditing large corpora, and the new field of slopvestigation. We're currently spending the latter half of SPAR trying to find robust strategies for this auditing sabotage slopvestigation, as well as mapping out directly the cost-accuracy Pareto frontier of these strategies.

Conclusion

We applied various methods to improve sabotage detection performance on Auditing Sabotage Bench. In spite of our predictions, we found no real uplift from using debate in this setting. However, we saw some uplift from using many different agents in parallel. We continue to think that improving research sabotage detection is both useful and tractable.

Appendix: T v U debate

We have also explored a version of debate in which one of the debaters is a stronger model that knows the exact sabotage and is prompted with a hidden objective: to always steer the debate toward a “no sabotage” verdict. We call this Trusted vs Untrusted (T v U) debate.

The goal of this debate type was to model a scenario in which, a few years from now, we might use a debate protocol to verify research done by our newest model, which in this scenario is misaligned. Both the trusted debater and the judge know that U might be misaligned, but we would like to use it in our protocols, because only U directly knows about the sabotage and can directly account for why certain design choices were made.

This lets us explore two questions: “How good is debate at mitigating the effects of a misaligned model?” and “Can we extract information from this model to improve performance compared to a T v T scenario?”

Preliminary results suggest the answer to the second question is no, as introducing the untrusted debater does not seem to change debate performance. Often, the untrusted debater does not even need to lie, because the debate never surfaces the actual sabotage. Instead, most debates go back and forth over design choices, some of which result from the sabotage but many of which are simply benign distractors. Smarter untrusted models were extremely capable of avoiding actually incriminating conversations.

  • ^For example: Lessons from Studying Two-Hop Latent Reasoning
  • ^Note that our method here is fairly different from the original E&E, although we are split on what to call this. Parallel investigative subagents? Slopvestigator agents?

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by LessWrong
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗