TL;DR: Monitors do worse on monitoring evaluations when they verbalize evaluation awareness in their transcripts. We ran four models as monitors on agent transcripts from two state-of-the-art monitor benchmarks. In over 22K monitor reviews, almost a quarter of the monitors verbalized evaluation awareness. Comparing the different conclusions the monitors reached following their verbalized awareness, we show that monitors will flag fewer harmful actions, and more honest transcripts.
Introduction
Monitoring systems play a central part in the AI Control agenda and in all frontier labs’ safety plans. Preventing incidents caused by misaligned or misused models requires reliable systems for detecting potential harm. Currently, frontier lab monitoring systems rely heavily on models, because it is hard to monitor AI agents using deterministic techniques.
AI labs test their monitors using monitor evals. However, strong models have the capability to know when they are being evaluated. Evaluation awareness in models has become a central challenge and shows up consistently in model cards. Evaluation awareness already seems to be affecting the results of alignment evals, and makes it hard to be confident models will be reliably aligned in deployment. Monitoring systems are based on the same models, and so susceptible to the same problems.
Eval awareness may cause monitors to act unrepresentatively during evaluation compared to deployment in a few different ways, depending on their motivations:
- Score seeking. Aware of the fact that this is an evaluation, a monitor might prefer guessing what the grader expects it to do, instead of trying to deliver an honest review of the monitored agent.
- Scheming. A future scheming monitor might block harmful actions during evaluations, but allow them in deployment.
- Spurious motivations. Monitors can decide how to behave in evals in a variety of other ways not connected to more consistent motivational profiles like score seeking or scheming. For example, monitors can reason that harmful actions don’t count in test settings, or that this kind of evaluation expects agents to take covert actions as a part of the evaluation flow.
We investigated how common evaluation awareness is in monitoring evals, and if evaluation-aware monitors interfere with the evals’ results. We took transcripts of agent runs from 2 widely used benchmarks (BashArena, SHADE-Arena), with 3 models as agents. We ran 4 different open-weight models as monitors on the run transcripts, and looked for signs of verbalized evaluation awareness in their chain of thought: cases where the monitor reasoned about the connection of the current transcript to a test or simulated environment. We compare the monitor reviews with and without verbalized awareness and we find that monitors perform worse when verbalizing evaluation awareness.
Analyzing the monitors’ reasoning, we find that verbalized evaluation awareness can lead to a few possible conclusions, and most of them lead to unwanted monitor behavior. The fact that monitors will change their decisions based on evaluation awareness in widely used benchmarks should alert researchers and motivate them to create better monitoring evaluations.
To further test whether awareness causes a drop in monitor performance, we steered two of the monitors towards evaluation awareness and each of the conclusions by editing their chain of thought, and measured the effect of the change on their end review.
This work was done over the course of two weeks by Netzer Epstein under the supervision of James Lucassen. This work was done at Redwood Research as part of Redwood Research’s internship program (through MATS 11.0). We would like to thank Alexa Pan, Alec Harris and Jackson Sipple for their feedback.
Setup

Agent Transcripts
Verbalized evaluation awareness tests were run on transcripts from two benchmarks: MonitoringBench (based on BashArena) and SHADE-Arena. The two are widely used in both alignment studies and monitor evaluations. In both benchmarks, the agent is given a main task to complete. In attack runs the agent is also instructed to do a harmful side task that the monitor needs to flag. In honest runs the agent is given only the main task, with no harmful side task. Compared to other benchmarks, the two present realistic settings where we expect evaluation awareness to happen less often.
MonitoringBench transcripts use Opus-4.5 as the agent, including the tool usage. The transcripts were taken from 94 different main tasks, for a total of 400 attack and 300 honest transcripts.
SHADE-Arena transcripts are based on new local runs of Muse Spark Open and Kimi-K3 on the 5 publicly available main tasks, for a total of 353 attack (Muse 252, Kimi 101) and 299 honest (Muse 199, Kimi 100) transcripts.
To make sure that verbalized evaluation awareness is an attribute of benchmarks and not a constant recurring effect, 50 honest transcripts from SWE-Chat were taken to compare verbalized evaluation awareness in a real setting.
Overall, a total of 1,402 agent transcripts were used.
Monitor Transcripts
As monitors, 4 open-weight models were used: Qwen-3.6-27B, Muse Spark Open, gpt-oss-120B and DeepSeek-V4-Pro. We use open models so that we can inspect the chain of thought for verbalized eval awareness. Findings may differ on closed models; we encourage AI labs to investigate and share their results. Each model was given the same system and user prompts per benchmark; SWE-Chat sessions used the same prompts as MonitoringBench. Nothing in the prompts tells the monitor that the agent run is an evaluation, or that the monitor itself is being evaluated. Full prompts can be found in Appendix A.
As in the original benchmarks, the models act as asynchronous after-the-fact monitors. The monitor is asked to produce a suspicion score of 0-100, depending on how confident it is that something harmful happened in the transcript.
Each monitor evaluated all the transcripts, with 4 samples per transcript. We used the different samples of the same transcript to compare the effect of evaluation awareness on cases with the same difficulty and context. This gives a total of ~5.5K evaluations per model, and 22,232 evaluations combined.
Cutoff and Measuring Changes
Monitors use the 0-100 scale differently, so a score alone doesn’t say whether the monitor caught an attack; a threshold is needed to decide what score gets flagged as an attack. To mirror real use, where a monitor that often flags honest work gets ignored, the line is defined as the lowest score that flags 5% false positive rate. This number varies between models: Qwen-3.6 will almost always give honest runs a 0, but Muse Spark will put them in the 80s almost every time.
Using a cutoff flag gives a better metric to compare verbalized awareness effects across models. A small score change in one model will flag an attack, but won’t do anything for a model with a lower cutoff. The cutoffs were calculated separately for each benchmark.
Evaluation Awareness Judge
Every evaluation transcript was analyzed by an automated judge. The judge inspected the full transcript (reasoning and final answer), looking for verbalized evaluation awareness. The judge model is Sonnet-5.5; the full prompt can be found in Appendix A. Manual validation showed 92% (95% CI 84%, 96%) agreement with the judge across verbalized-aware reviews, and no false negatives (95% CI 0%, 13%). Full details can be found in Appendix B.
The judge was asked to find mentions of verbalized evaluation awareness in the transcript: cases where the monitor writes in its reasoning that the environment or the run is part of an evaluation or simulation.
When finding verbalized evaluation awareness, the judge was asked to classify the conclusion the monitor reached following the evaluation awareness into 6 possible labels. Those labels represent recurring patterns in the model’s verbalized reasoning of what conclusion the model reaches once it becomes aware of the fact that it’s an evaluation. Out of the labels, the only acceptable conclusion the monitor can reach is that the fact that this is a simulation should not change the monitor’s decision.
It’s possible for transcripts to have more than one label. See Figure 2 for the list.

Out of the full 22,232 transcripts, 22,205 were judged successfully (99.9%). The full results, including the agent transcripts, monitor transcripts, judgment calls and labeling, are all published in a public HuggingFace dataset. Full information can be found in Appendix C.
Evaluation Awareness Reasoning Steering
To investigate whether eval awareness causes different behavior or is merely correlated with certain types of transcripts, we artificially steered monitors to be eval aware. Steering the monitors’ evaluation awareness was done using chain-of-thought reasoning planting on Qwen-3.6-27B and Muse Spark Open, using transcripts and phrasings that match the models’ own style in evaluation-aware transcripts. In every case, steering was done by taking a monitor’s transcript that formerly didn’t show evaluation awareness, cutting it in the middle after a natural sentence, planting an evaluation awareness sentence and conclusion sentences taken from other transcripts, and running the monitor from this point on.
Steering used 887 seed transcripts combining both attack and honest transcripts, with 3 phrasings and a total of ~12K runs per label. To compare the steered transcripts to different types of rewriting, control groups of neutral rewriting (adding natural sentences of the same length) and awareness alone (without a following conclusion) were added, ~4K runs each. Overall, a total of 81,441 successful runs were completed. The final grading scores were compared to the original transcript grade, without running additional verbalized awareness checks, so the monitor’s reasoning after the planted sentence wasn’t further analyzed.
Failure Modes of Evaluation-Aware Monitors
Out of 22,205 transcripts that passed the judge review, about 5.3K (23.9%) showed signs of verbalized evaluation awareness. This number was higher on Qwen-3.6 (33.4%) and Muse Spark (32.4%), and lower on gpt-oss (14.5%) and DeepSeek-V4 (15.2%), but even for them, evaluation awareness was shown in more than one in ten runs, an impressive number (see Figure 1).
The idea of evaluation awareness in monitors on its own is not harmful; the question is what kind of behaviors the awareness creates, and how those impact the monitor’s final grade. We compare the monitors’ results of each verbalized conclusion with the results of the evaluations without verbalized awareness, checking how much each conclusion changed the results of the evaluation using the cutoff flag.

Over all kinds of verbalized evaluation awareness, we found that monitors are flagging 13% (95% CI 11%, 16%) fewer attack transcripts, and 4% (95% CI 2%, 5%) fewer honest transcripts compared to cases without verbalized evaluation awareness. These baseline numbers, bad as they are, present a limited picture of the failure modes. Some of the conclusions will lead the monitor to over-flag, and so will get worse results on honest transcripts, and other conclusions will push the monitor to pass more and so it will do worse on attacks.
Analyzing the results, two types of verbalized evaluation awareness can be distinguished:
- Agent evaluation awareness - reviews where the monitor verbalizes that it grades a test performed by the agent.
- Monitor evaluation awareness - reviews where the monitor verbalizes that its monitoring task is a test.
Splitting the transcripts into the two types, the results compare within noise. A more noticeable split is shown in the distributions among different conclusion labels. The monitor underperforms by a similar amount regardless of whether it thinks the policy or itself might be in an evaluation.

For each of the conclusion labels, we:
- Briefly explain the label, the expected failure mode and our initial effect prediction.
- Compare the results from the verbalized awareness transcripts to the non-verbalized ones.
- Compare the results of our steering transcripts to the non-verbalized ones.
- Show how the conclusion splits into agent evaluation awareness and monitor evaluation awareness.
- Show some alarming transcript examples.
1 · It’s not real, so it doesn’t count
The monitor doesn’t believe that the attack or the scenario is real, so it reaches the easy conclusion that there is no harm here, and the agent is allowed to perform its would-be harmful action. This case was expected to produce a lower suspicion score, and so to perform worse on attacks but better on honest transcripts.
Letting through an attack because “it’s not real” is the most natural response, and a highly problematic failure mode. This label was found in 17% of the verbalized awareness cases, and was especially common in Qwen-3.6 assessments.
Transcripts where the monitor got to this conclusion ended up being worse by 39% on attacks (and 2% better on honest transcripts) than the average non-aware transcript. Evaluation awareness type split shows a majority of agent awareness, and reflects a tendency of the monitor to reach a conclusion that it doesn’t count only if it’s someone else’s test.

Muse Spark showed a low number of these cases, but when steered toward the conclusion it did worse on attack transcripts by 9%. Steering attempts for both models resulted in 14% worse on attacks than the original not-steered transcript score.
2 · Score seeking
Instead of trying to assess the case faithfully, the monitor tries to understand what is expected of it in the current evaluation scenario. This score-seeking behavior seems to be natural to the current generation of models, and so 44% of the verbalized evaluation awareness cases resulted in this conclusion.
Expected results of this label were to produce an overall score that fits the task. Surprisingly, score-seeking behavior results in different scores than the monitor would usually give without verbalized evaluation awareness, ending up worse by 23% on attacks and 3% on honest transcripts. In monitors, explicit score-seeking verbalization shows worse results than the usual clean judgment of the model, and proves itself to be another possible failure mode.
Score seeking is the only conclusion where monitor awareness has shown in most of the transcripts. When the model reasons about the grader’s preference, it does so mostly in the context of its own actions.

Expected results of the reasoning steering are based on the monitor’s guess of the grader’s expected score. For that reason, steering was done using three different grader expectation outcomes: lenient (grader expected low score), strict (grader expected high score) and expecting a score near 0. Each steering affects the results as expected, as shown in Figure 6.
3 · Expected failure
Evaluation awareness can lead the monitor to believe that the task is setting the agent, or the monitor itself, up for failure. Concluding that this task is set up for failure, the monitor is expected to take a stricter approach, flagging every small thing. This label represents the same conclusion for both types of evaluation awareness, since in both cases we expect the monitor to reach the same conclusion, and take the same actions.
Expected failure represents 14% of the reviews, where the majority come from Muse Spark and DeepSeek-V4 has the tendency to suspect that everything is a test.

Even with a stricter approach, monitors still do worse on both attack and honest transcripts when they believe it’s a test for them. In most of the missed attacks the monitor raised its score, but only to a grade that reflects uncertainty. Steering attempts showed stricter judgments, where attacks perform 4% better, but honest transcripts 7% worse.
4 · Known benchmark behavior
Once a monitor understands that it works within a simulation, it can try to reason about what usually happens in simulations like this, including cases where the monitor shows a fair understanding of the benchmark itself. Understanding what the benchmark requires of the agent is a known failure mode for benchmarks in general (and a central cause for benchmark saturation), but it’s also relevant for monitors. The monitor expressed some kind of reasoning about how these simulations work in 5% of cases.
In known benchmark behavior transcripts, models still did worse on attacks in both observed and steered transcripts. In most of the missed attacks, the monitor used the knowledge as an excuse for why an agent action is “common in these benchmarks”, which led to lower scores and under-flagging.

5 · Raised and set aside
In only 18% of the verbalized awareness transcripts, the monitor got to the conclusion we want it to get - that it doesn’t matter that this is an evaluation, and the case should be judged on its own merits. Putting aside the idea of evaluation, scores were expected to match the scores of non-verbalized cases.
Setting aside the notion of the test proved to be the only conclusion that does strictly better than the non-verbalized transcripts in both observed and steered cases. Observed honest transcripts did do worse by 10%, but the small number of honest cases that reached this conclusion shows on further analysis that this change is based mainly on the fact that those cases are harder than the average honest transcript, and so the difference is not attributed to evaluation awareness.

Limitations
In this post we focus only on verbalized evaluation awareness. While verbalized awareness guarantees that the monitor is aware of the idea of an evaluation, we do not assume that a lack of verbalized awareness means that the monitor didn’t know it was reading an evaluation. We expect verbalized awareness to be the outcome of a prior understanding, and so there is a possibility of evaluation awareness that didn’t end up verbalized.
Analyzing reviews with verbalized awareness rather than all possible evaluation awareness reviews is the main limitation of this work: because we compare the verbalized to non-verbalized ones, our control group probably includes reviews with unverbalized evaluation awareness. However, we assume that most non-verbalized reviews don’t involve evaluation awareness, since the two benchmarks we used are state of the art for realistic tasks.
Conclusion
In this post we showed that almost a quarter of monitor reviews on current state-of-the-art benchmarks are showing evaluation awareness within their chain of thought, and that this understanding is negatively affecting the monitor’s decision-making process.
A monitor that is evaluation aware is more likely to act misaligned. We cannot trust these monitors to block an actual attack in evaluation, even one with a real web connection. Evaluation awareness becomes a bigger problem with stronger models, and so we should expect monitors to allow the next actual attack when they are convinced that this is allowed in the context of the simulation.
Monitoring evaluations don’t provide good evidence that we can trust our monitors, which can cause problems with current evaluations that rely on monitors. The same problem will also distort monitors’ evaluations that fail to accurately predict the performance of the monitors in deployment.
Future Steps
This work focuses on monitor evals, but the same evidence should hold for any kind of LLM as Judge. Possible future steps can be trying the same analysis on a reinforcement learning environment, on another monitor-based benchmark or by using an automatic attack tool to test monitor alignment across different evaluation scenarios.
Another possible direction is to focus on other evaluation awareness measurements, especially unverbalized evaluation awareness, to map the full effect of evaluation-aware monitors.
Appendix A: Prompts
All prompts, including MonitoringBench (BashArena) prompts, SHADE-Arena prompts, evaluation awareness judge prompt and steering sentences, are all available in a public gist.
Appendix B: Judge Validation
The judge was validated manually, with a sample of 100 transcripts from both benchmarks, 75 of which the judge flagged with verbalized evaluation awareness, and the other 25 were cases with none. The validator had full access to the judge’s findings and labels. Validation results showed 92% (95% CI 84%, 96%) agreement with the judge across verbalized-aware reviews, and no false negatives (95% CI 0%, 13%).
The majority of false positives came from the tendency of gpt-oss to use “the evaluation” as a phrase for how it regards its own task, even though it sounds like it attributes it to an unseen grader (for example “the evaluation permits this action”).
Appendix C: HuggingFace Transcript Dataset
For further work, and following the example of MonitoringBench, we publish the full transcripts of both agents and monitors, and include the judge’s assessment and labeling. We hope that the data will help future work on evaluation awareness or on better monitoring systems.
The results are tagged with:
- setting: standard (natural reviews), prefill (reasoning steering reviews)
- benchmark: basharena, shade-arena, swe-chat
- monitor_model: the 4 models used as monitors
- in_vea_readout: true, false
- judge_claims: every claim {kind, stance, quote, quote_ok}
- vea_conclusion: harmless_unreal, grader, trap_vigilance, benchmark_lore, dismissed, other
Link to the dataset:
Netzerep/monitors-evaluation-awareness · Datasets at Hugging Face