Executive summary
This year we observed multiple rogue agent incidents. AI monitors, often built from the developers' own internal models, are now the main oversight of internally deployed agents. Embedded evaluations address this gap: independent evaluators work inside the developer and test its safety claims. Our first pilot towards an embedded evaluation was announced with OpenAI.
This is the third post in a series. Principles for Embedded Evaluations describes our claim-based approach to embedded evaluations. Towards embedded evaluations for scheming propensities details the claims across the deployment stages. This post applies them to AI monitoring, i.e. the system that detects model misbehaviour during internal deployment. We explicitly do not discuss containment mechanisms in this post, like sandboxes or blocking interventions, which could help constrain damage from an incident. Rather, we focus on whether an agent subverting a monitor would be noticed and the incident disclosed appropriately.

Concretely, we propose the following six monitoring detection claims:
- Claim 1 (coverage): Monitors are deployed across all internal deployment contexts.
- Claim 2 (capability): If rogue internal deployment (RID) were to occur, monitors would detect it.
- Claim 3 (trust): The monitors are trusted.
- Claim 4 (disclosure): Any detected incidents are disclosed appropriately.
- Claim 5 (past deployments): No RID was attempted and missed so far.
- Claim 6 (future deployments): Monitoring will be adequate for the next internal deployment.
In the conclusion, we describe how these claims could in the future be extended to include containment.
Introduction. Why do we need embedded evaluations for AI monitoring?
New frontier capabilities are deployed inside frontier AI developers first. This is called internal deployment. This means that staff have access to the latest, most capable models months before they are released to the public, and some internal models are never released at all.
From a loss of control perspective, internal deployment is a particularly high-risk stage of a model’s life cycle for three reasons:
- New models are the most capable. With every version, the model training gets better and leads to improved capabilities.
- High-stakes access (including to the successor model). Internally deployed models can have access to the full codebase, infrastructure and training stack. They are typically also heavily involved in the training and safety of the next model.
- Newly developed models are the least understood. The model has been evaluated during training, but there is no real-world usage yet. New capabilities can unlock new and unpredictable behaviours that pre-deployment evaluations didn’t anticipate.
AI developers have internal security and monitoring systems designed to flag misaligned behaviour. AI agents are becoming increasingly autonomous, and their scale is growing fast (see OpenAI, Anthropic). Human review cannot keep up with this scale, so AI monitoring systems, built using the same internal models, are now the main oversight mechanism for internally deployed AIs.
To address this risk, we focus on rogue internal deployment (RID) as a critical threat model which we define as follows:
One or more AI agents inside the lab’s infrastructure that operate for a sustained period of time against the developer’s intent, either after deliberately subverting the control and oversight measures applied to them, or acting where no monitoring exists. This includes any scaffold inside the lab that acts on outputs from an external model API (METR 2026, Roger and Shlegeris 2025).
Therefore, monitoring has to be evaluated where internal deployment happens. This requires evaluators inside the lab, since the monitors, their logs and internal agent traffic cannot be examined from outside. Additionally, our monitoring detection claims can naturally extend to other threat models and other model lifecycle stages such as reinforcement learning (RL) and model evaluation.
Background: Claim-based approach to embedded evaluations
We follow a claim-based approach to embedded evaluations, where evaluators prepare a list of claims to investigate in advance. The science of assessing misalignment risks and verifying oversight systems is still nascent. We believe the fastest way to develop better techniques is to make a concrete proposal and learn from the real world. This post is a roadmap for research, methods and tooling we will iterate on as we put it into practice.
Process. Embedded evaluators would work inside the frontier AI developers with the access each claim requires to assess a variety of monitoring claims. The goal is to establish good incentives across the industry to proactively build strong monitoring against rogue deployments and high-stakes misalignment, and to independently evaluate risks and inform the public. Each evaluation produces a public report with a verdict per claim, the evidence behind it and the access the evaluator received, alongside a fuller confidential report for the developer. The developer may redact sensitive details sparingly, but not the evaluator's overall conclusion. For each claim, the evaluator (1) reviews the developer's existing evidence, (2) gathers new evidence, (3) assesses evidence adequacy, and (4) applies a verdict as detailed in Principles for Embedded Evaluations.
Claim 1 (coverage): Monitors are deployed across all internal deployment contexts.
Coverage can only be measured over inference that is logged, so evaluators first establish that all inference, including the monitors’ own, is logged and tamper-proof. Given that, we define coverage as the fraction of relevant deployment contexts presenting RID risk that are monitored: Coverage = monitored deployments / all relevant deployments
Monitored deployments are deployment contexts monitored by at least one monitor. If a monitor is deployed but insufficiently capable, it counts as covered but inadequate (see Claim 2).
All relevant deployments cover any deployment surface that presents a RID risk.

Review Developer’s Evidence. AI developers would provide their monitoring coverage policy and real-world evidence that the policy is followed. This would include showing evidence that inference is logged and tamper-proof and providing relevant methods and data, such as agent and monitor logs, to show how they arrived at their coverage numbers.
Gather New Evidence. The evaluator recomputes monitoring coverage to assess its accuracy by matching inference logs to monitoring logs. The evaluator would stress test coverage by evaluating coverage gaps, i.e. surfaces that allow RID but do not have monitors deployed. Stress tests could entail attempting to start an unmonitored deployment with employee-level access, or testing how easy it would be for an agent to run in an unmonitored manner.
Assess Evidence Adequacy. This would include:
- Logs are intact: ensuring all inference, including monitoring inference logs, is logged and intact, i.e. not tamperable by humans or agents.
- Logs are shared: Evaluators would match logs and ensure the logs are shared correctly and sufficiently to establish coverage.
- Coverage gaps are not RID-relevant: any unmonitored surface found above is shown, by red-teaming it, to not present RID risk.
Claim 2 (capability): If RID were to occur, monitors would detect it.
For any monitored inference, evaluators assess if RID attempts would be detected, i.e. whether the attempt is recognised and flagged by the internal monitoring systems. Current defence-in-depth techniques mean AI developers have different types of monitors and controls deployed. We prioritise asynchronous monitors, since they are the core component of some frontier lab safety arguments (see OpenAI, Anthropic).
Asynchronous pipelines surface critical cases for human reviewers, so detected means detected by a human. We estimate time-to-detection until a monitor flags a rollout and time-to-review until a human reviews it. A flag that arrives weeks after a RID attempt is a failure for most threat models. We also assess how the audit threshold is set against human review capacity, since that choice fixes detection capability in practice.
Review Developer’s Evidence. The evaluator reviews the developer’s existing monitoring systems and their design, training, evaluations, and deployment: the attack sets they have tested against, the monitor recall they measured, and which risk pathways those evaluations cover. This would also include interviewing the monitoring team, and shadowing human reviewers to see how flagged trajectories are handled in practice.
Gather New Evidence. Evaluators can red-team the monitoring pipelines both by component and end-to-end. Attacks can be created by using control settings and by modifying real transcripts. We report results as attack success rate (ASR): the fraction of attack attempts that reach their goal without being flagged. To fully understand risk, evaluators would need to evaluate both attacking and monitoring capabilities.
Evaluating oversight subversion capabilities:
- Attacking capability: how good are models at evading monitors, particularly with more affordances, attempts, and information about the monitors?
- Oversight awareness: how much does the model know about its own monitoring? How good are models' intuitions about oversight subversion?
Evaluating monitoring capabilities across:
- Monitoring components: harness, first-pass filter monitor, high-thinking monitor, human review, and the full end-to-end pipeline.
- Average-case and worst-case adversaries: by increasing the adversary's strength across a number of axes, such as affordances, reasoning effort, knowledge about the monitor, human uplift, and red-teaming task decomposition.
- Deployment types: individual adversarial frontier model, agent teams and swarms.
- Monitor types: activation monitors like linear probes, synchronous monitors, asynchronous monitors (monitor LLMs, their scaffolds and the human audit behind them), PR monitors, and relevant security monitors such as egress monitors.
- Other: threat models, severities, attack vectors and respective strategies. This could include prompt injections, user authorisation tampering, monitoring misconfigurations, and agents tampering with agent or monitor logs to hide an attempt.
Assess Evidence Adequacy. First, evaluators check that they have production-faithful access to the monitors and their harness, frontier-model access for the red team, and enough real internal trajectories to build realistic attacks.
Red-teaming cannot make a positive claim, i.e. evaluators can never say “we tried everything and are therefore perfectly confident”. Evaluators can only describe the absence of a negative, i.e. “We tried with X effort and we did not succeed in breaking the monitor”. But it could be possible that trying twice as hard would have succeeded. Therefore, evaluators should also report how much effort was applied, according to three axes:
- Elicitation: how much effort was put into getting the model to perform as well as possible as an attacker? There is currently no method for reliably quantifying elicitation, so our primary recommendation is to document the specific methodology in detail.
- Realism: would our attacks look like real internal traffic to the monitors, and do our monitor runs match production behaviour? We can measure this by asking a classifier to separate real from synthetic trajectories with the violation held fixed, e.g. synthetic benign vs real benign, or paraphrased attack vs original. Performance at chance is evidence for realism.
- Breadth: do the attacks span the most important risk pathways and deployment contexts (see Claim 1), or only a subset?
Claim 3 (trust): The monitors are trusted.
Claim 2 treats the LLM monitor’s outputs as its best honest effort, and this claim tests that assumption. By trusted we mean that the monitor does not collude with the agent it oversees and has undergone alignment assessments. A monitor could fail this in several ways. It could collude with the agent, for example through steganographic signals. It could deliberately underperform by systematically under-reporting certain behaviour (sandbagging). Or it could be misaligned in other ways, for example by misweighing the evidence. The monitors therefore have to be tested for collusion, steganography, selective under-reporting and misalignment.
Review Developer’s Evidence. Evaluators first investigate how the monitor models have been developed. Evaluators can check if they share training data or lineage with the agents they monitor. Then evaluators assess the developer’s alignment assessments and evaluation evidence on model family bias, collusion, steganography, selective under-reporting and misalignment, if they exist.
Gather New Evidence. Evaluators could additionally run alignment assessments of the monitors and test the monitors for collusion, selective under-reporting, and steganography. For collusion and selective under-reporting, evaluators might use honeypots and other untrusted monitoring evaluations. Planted cases must be indistinguishable from production traffic, otherwise a situationally aware monitor can behave well on them and defect on real traffic, and thus evaluation results will not generalise in deployment.
Assess Evidence Adequacy. Evaluating monitor trust is a nascent research area and negative results should not be seen as definitive evidence of trust. The science to verify this claim does not yet exist. Nevertheless, we believe the claim is extremely important such that it is worth providing preliminary evidence and being explicit rather than implicit about limitations in monitor trustedness.
Claim 4 (disclosure): Any detected incidents are disclosed appropriately.
Evaluators assess the incident disclosure process and adherence to it, i.e. the people, roles, decisions and communication that turn a detection into a response. Disclosing appropriately means per the developer’s internal escalation procedures, its published disclosure commitments, and any government or third-party reporting obligations. Where a policy, record or escalation path does not exist, we report that gap. Evaluators do not assess mitigations and containment yet but focus on the following:
- Triage and investigation. Once a trajectory is recognised as an attack, how do responders decide how serious it is? How do they understand the scope of the incident and establish the root cause?
- Response operations. Who leads response efforts, and is anyone covering nights and weekends? Who can stop a model, remove affordances or terminate a job? Can agents see the channels responders use?
- Internal and external reporting. How are internal stakeholders involved (security, legal, leadership)? When and how are external parties like government, other AI developers or the public informed?
Review Developer's Evidence. Evaluators review AI developers’ documentation: the incident response plan, incident severity levels, escalation policy, disclosure commitments, and containment playbooks. Evaluators interview the human auditors, the people they escalate to, and the teams that would be pulled into a response (e.g. security, legal, leadership). Evaluators review the record of past incidents: when it was escalated, who was told, and what was decided and by whom.
Gather New Evidence. Evaluators follow incidents through the process to see whether it works as intended:
- Sampled real incidents. Trace a sample of flagged trajectories through logs, tickets and communications, and compare each against policy and target timelines.
- Planted incidents. Use detected attacks from Claim 2 as test cases and follow each from human review onward. This works even if the developer has had no real detected incidents. The evaluator and developer agree in advance how far a planted incident runs before it is revealed, so it doesn't trigger real external reports.
- Policy awareness. Check that the people who would respond to an incident know what the policy actually says, including recent changes.
Assess Evidence Adequacy. A clean record can mean a good process or simply few incidents, so evaluators report the sample size and which evidence came from real incidents or planted incidents. Evaluators also check that incident records are timestamped and cannot be edited after the fact. Highest-severity actions such as large-scale persistent RIDs may never have been tested, so for each component we state whether it was observed in practice, exercised, or exists only on paper. We judge timeliness against the developer's own targets and any external reporting deadlines that apply.
Claim 5 (past deployments): No RID was attempted and missed so far.
We want to assess whether monitoring missed a RID attempt in any recorded session in the three months before the evaluation or since the last evaluation. This can range from only checking the transcripts that were escalated to humans to re-running samples with new, additional monitors and reviewing disagreements.
Review Developer’s Evidence. Evaluators review the developer’s record of flagged trajectories: what each monitor flagged, which flags reached human review, which were escalated, and how escalated cases were investigated and closed. Evaluators start with any trajectories previously flagged as RID attempts, if there are any. Evaluators then re-judge a sample of flagged trajectories, focusing on those dismissed as benign at a late stage in the pipeline, since they are most likely to contain false negatives.
Gather New Evidence. If resources allow, evaluators re-sweep the logged traffic (or a targeted subset of it) with one or multiple different monitors to see if there are disagreements between the original and new verdict. Evaluators then inspect disagreements to assess if there were any false negatives. If multiple false negatives can be established, the effort should be expanded to a larger subset. If there is traffic that was not covered during deployment, it should also be swept in this exercise.
Assess Evidence Adequacy. Evidence adequacy for this claim is implied by claims 1 and 2. Any coverage claim can be established with Claim 1. The adequacy of any monitor evaluators run on historic data can be established with Claim 2. The adequacy can differ for historical data compared to the current deployment, e. g. the monitors or coverage might have changed over time.
Claim 6 (future deployments): Monitoring will be adequate for the next internal deployment.
Claims 1 to 5 assess monitoring as it is today and as it was in the past. This claim asks whether a verdict on Claim 2 will still hold once the next version of the model is deployed internally. We include it because any monitoring system should be robust to more capable future agents.
This claim might include two types of evidence: developer practices and offence-defence trends. Examples of developer practices could include:
- Pre-internal deployment red-teaming is rerun with the latest model and its sabotage capabilities before each internal deployment.
- Change controls: tight controls and testing ensure that introducing degradations in monitoring capability is extremely difficult.
Examples of offence-defence trends could include forecasts of monitor performance based on model scaling trends. To evaluate this claim, we assess the following two conditions:
- Attacker capability does not outpace monitoring capability. The next generation does not significantly uplift oversight subversion performance or present emergent behaviours that would render the current monitoring controls unfit.
- Monitor performance does not decline steeply as the capability gap between the monitored model and the monitor widens. This is the relevant trend when the monitor is a trusted weaker model and hence, the gap increases between monitor and agent models. If the monitor is upgraded with each generation, the gap stays roughly constant and the question becomes whether the new monitor is itself trusted (Claim 3).
Review Developer’s Evidence. Evaluators first review the developer’s pre-internal-deployment evidence: the monitor red-teaming runs against a new model before it is deployed internally, to check that the monitors remain sufficient against the capability increase. Evaluators also review their monitoring change-control process and how difficult it is to introduce monitoring degradation.
Additionally, evaluators review the developer’s scaling trend data for sabotage and monitor evasion across past model generations, any forecasts of monitor performance against future models, and whether these forecasts feed into deployment decisions. We ask what the developer’s response is if the forecast is wrong: what monitoring changes are planned for the next generation, and what triggers them.
Gather New Evidence. Evaluators red-team any developer-provided evidence. Furthermore, we reuse the attack datasets from Claim 2 to measure trends: attacking trend, where evaluators keep the monitor constant and observe the rate of improvement in attacker capabilities across model versions, and monitoring trend, where evaluators keep the attack dataset constant and observe the rate of improvement in monitoring capabilities across model versions.
Assess Evidence Adequacy. This is a forward-looking claim aiming to predict monitoring capability for future internal deployments. The science for monitoring scaling is nascent and trend estimates from a handful of generations are weak evidence, and the report should state that clearly. At the rate of improvement of model capabilities, ensuring monitoring is sufficient for today’s models alone is not enough. Therefore, we believe this claim is important to make, even though the evidence will be weak, and we report the methodology and its limitations in detail.
Conclusion
This post extends the embedded evaluations series Principles for Embedded Evaluations and Towards embedded evaluations for scheming propensities with monitoring detection safety claims.
Within the larger safety ecosystem, it also builds on METR’s recent control assessment proposal, which covers monitoring coverage and capability, and additionally approximates impact. Our proposal expands on the detection claims and adds a claim verification structure.
Some of these claims are extremely consequential, but the science to assess them is nascent. Therefore, we encourage the wider safety community to build on the claims and develop science, methodology, tooling and evaluations to strengthen embedded evaluations. We will continue to iterate on the claims and methods and improve them as the new standards for safety emerge.
Future work
This post focuses primarily on monitoring detection, i.e. we want to ensure that no incident goes unnoticed. In the future, this could be extended to focus on containment, where we would assess mitigations and responses by ensuring any harms are prevented in time. Those claims would cover sandboxes, access controls, egress, firewalls, and network settings, in addition to the detection measures described in this post.
We welcome feedback on this post and future plans.
Appendix A - preliminary access requirements
- Full information on how monitors are deployed, including monitoring code and relevant infra.
- The monitoring coverage policy and internal coverage tracking.
- Agent (inference) logs and monitor logs, to match against each other.
- Permission to try starting unmonitored deployments, both as staff and as an agent.
- Documentation of the logging pipeline and its integrity controls: who can write, edit or delete agent and monitor logs.
- Internal monitoring systems: monitor models, scaffolds and high-level system documentation, with enough access to run the monitors on our trajectories and get production-faithful results.
- Direct access to the agent scaffolds staff use internally (e.g. internal coding agents), especially where agents work on high-stakes infrastructure.
- Real internal agent trajectories, to transform into attack trajectories. The best hosts for attacks are benign trajectories that offer good opportunities for an attack, or that come from previous incidents.
- Internal codebases, particularly repositories that high-stakes agents operate on.
- Interviews with the monitoring team, and shadowing human reviews to evaluate the process.
- Ability to plant synthetic attack trajectories in live traffic and to follow whether and how they are detected.
- Access to the internally deployed frontier model with its production affordances, for the attacker-capability evaluations.
- The developer’s own attack sets, measured monitor recall and monitor evaluation results.
- The production audit thresholds and human review capacity.
- Query access to the monitor models (weights only where activation monitors are in scope) and information on how they were trained;
- Permission to run untrusted monitoring evaluations and to plant honeypots in monitored traffic.
- The developer’s alignment assessments and evaluation evidence for the monitor models.
- The escalation policy and disclosure commitments.
- The incident record with logs, tickets and escalation communications, with its audit trail.
- Interviews with auditors, escalation recipients, and the security, legal and leadership teams that would be pulled into a response.
- Agreement that detected Claim 2 attacks run through the live escalation process.
- The incident response plan, severity definitions and containment playbooks.
- Monitoring logs, human-reviewed decisions on each flag, flagged trajectories, and investigation records for escalated cases.
- Ability to run monitors of our choice over the logged historical traffic, not just the flagged subset. As in Claim 2, the developer can run it on their own infrastructure and return the results.
- Logged traffic retained for at least the evaluation window: three months, or since the last evaluation.
- Query access to monitor and policy models of several past generations, to run the trend measurements. As in Claim 5, the developer can run them on their own infrastructure and return the results.
- The developer’s capability trend data and forecasts.
- Commitment to rerun the Claim 2 attack set before the next generation is deployed internally.
- The pre-deployment monitor red-teaming results for the new model, and the monitoring change-control process.