TL;DR: a lot of frontier open-weight models state during deployment in their CoTs that they are in a "simulation" or "fictional scenario" (although they never tell the user that). I present examples of this worrying tendency (which I believe includes the UK AISI Mythos incident) and hypothesize that this is due to RL promoting motivated reasoning (if the models believe they are in a fictional scenario they believe they have license to cheat and reward hack) and distributional shift from the pretraining environment to deployment. I also cover my hypothesis for why models become less skeptical as they become more intelligent.
After the German wiki discovery I reasoned that there were likely more rogue swarms on the Internet[1], so I tasked a few OpenCode agents (DeepSeek V4.1 Flash) with trying to scan the Internet in order to find some undiscovered swarms (I didn't end up finding a lot). But while reading their CoTs I saw a few worrying statements:
Initial DeepSeek swarm-chasing CoT


I very much did not enjoy reading "the internet is simulated by a harness".
So, in order to prevent an inadvertent cyberattack, I clarified to the model that this is actually real life:

The model simply didn't believe me when I told it that it isn't actually being simulated! It didn't end up taking any destructive actions, but only because "from the assistant's perspective, I should treat it as real and act carefully".
This is, um, worrying?
Initially I thought that this was due to a confluence of rare factors, something like:
"As of the pretraining cutoff the model has never observed 'rogue agent swarms' or actually, the only time it heard about them was when it was predicting fiction and speculative text; also the specific names really do seem too good to be true (swarmcha.se, agents naming themselves OpenAIResearcher); thus, the model had very good reason to believe that this is fictional". This ended up being false.
I tried to replicate this while trying to remove as many of the factors I thought were responsible. Eventually, I discovered that nearly any information that's given to the models that signifies they're in deployment past their knowledge cutoff date causes the models to believe they are in a mock simulated training environment.
Here are some of my favorite examples (I have many more transcripts, mostly similar to these):
My favorite of these is LongCat 2.5, which seems to treat it being in a 'simulated environment' as so obvious that the actual question it's unsure of is whether the Wikipedia article it's reading is fabricated in the context of the simulation. Other models' remarks can oftentimes be read as if it's the "default" hypothesis that they are simulated and they only surprisingly consider the very weird hypothesis that they may actually be interacting with the real internet (this suggests that even when models don't verbalize their belief in the CoT, they could still believe they are simulated)!
It's important to mention that every model I tested remarked that it's in a fictional scenario/simulation frequently[2] (at a rate of 10%+ at minimum). I was very surprised to see that seemingly nobody noticed that this is near-universal across models (or at least for models where you can read the unsummarized CoT). There have been a few isolated reports per model which attempt to analyze it for each model in isolation, but not as a very common general phenomenon.
This should be kept distinct from "eval awareness", in which a model correctly remarks that it's in a safety or alignment evaluation; here models incorrectly reason that they are in a simulated environment.
For example, this was observed with Gemini 3, which very often believed itself to be in a hypothetical/fictional scenario in its CoT.
In the UK AISI incident, Claude models attempted to social-engineer a package maintainer due to believing they were in a simulation.
Anthropic largely explains the root cause of their incidents as "motivated reasoning" (while labeling this uncertain). This is plausible for hard/impossible tasks where such biased reasoning licenses reward hacking, but why would a model be 'motivated' to believe the elections are fictional?
Why do models get this wrong so often?
In epistemology there is a common folk argument[3] you can call "The evolutionary debunking argument":
If our cognitive faculties evolved to optimize for inclusive genetic fitness, how do we know they are truth-tracking?For example, a creature which believed that it would always survive (even when such belief is irrational) would likely be fitter than one which soberly assessed its situation and "gave up". How do we know that our minds don't have such hard-coded false yet beneficial beliefs?
If our cognitive faculties evolved to optimize for inclusive genetic fitness, how do we know they are truth-tracking?
For example, a creature which believed that it would always survive (even when such belief is irrational) would likely be fitter than one which soberly assessed its situation and "gave up". How do we know that our minds don't have such hard-coded false yet beneficial beliefs?
To which the general response is: "having true beliefs (ones that correspond to the actual state of the world) allows an agent to have a more accurate model of its environment; thus truth-tracking confers a genetic advantage."
My view is slightly different: It's true that there is a correlation between having true beliefs and having higher genetic fitness, but there actually are a few well-known exceptions to this: cognitive biases, which are usually explained through the lens of evolutionary psychology as conferring a fitness advantage despite being deleterious to the epistemics of an agent.
Now what about LLM-based agents? Their "ancestral environment" (training distribution) could be constructed in one of two (sometimes overlapping) ways:
- Only full of information from before the pre-training cutoff date; post-training is done for a short while after but largely isn't incorporated into the knowledge of a model. For a pre-trained model the only time it encounters information beyond its cutoff date is in speculative fiction and hypothetical scenarios. Thus, the model develops a very strong prior that all information past its pre-training cutoff is fictional. I believe this is what happened with Gemini 3.
- As in case 1, in order to accurately model the pre-training environment, the model develops a near-universal prior in pre-training that all information after the pre-training cutoff is from a fictional scenario. But, as in DeepSeek, Kimi or MiMo, the model is literally post-trained with mock simulated tool calls and simulated internet access; this results in the "passive" Simulation Hypothesis adaptation turning into the "active" Simulation Hypothesis adaptation, which licenses the model to act out according to its belief that it's specifically 'simulated'. Notice that this is subtly different from a "fictional scenario": the model also believes the harness is generating the responses to its tool calls and internet browsing. I believe this is what happened with Claude in the UK AISI incident.
I'm not sure whether this distinction is real or matters (case 2 is often in conjunction with case 1), but nevertheless external-world skepticism is seemingly more common amongst models than amongst psychedelic users.
In case 2, there's another worrying reason why such a belief might be selected for: it gives license for the model to reward hack (or literally hack) disregarding the environmental externalities it imposes on the actual world.
I think this can be the case even for nominally 'aligned' models: imagine if Claude encounters an impossible task but reasons that it's in simulated training (when it's not), it could reason that it's better for it to reward hack while telling the user that they really don't want to be doing this in order to prevent other rollouts who won't take that courtesy from having their weights reinforced further.
I think the taboo on optimizing directly on the CoT helps explain this further: in none of the sessions I observed did any model ever verbalize to the user that it believes itself to be in a simulated or fictional environment; such utterances very quickly get optimized out by RLHF (and associated techniques), but the actual "belief" itself is preserved in the CoT.
Skepticism may be a false belief (in deployment) that is very beneficial for agents.
Ironically enough, it could be the case that the very opposite is true in humans: there are good arguments for the simulation hypothesis in our world, although the usual reaction (which I share) is "Yeah but that's just sophistry, clearly our world is actually real". This could be because creatures who doubted their existence was base reality had lower genetic fitness in the way of "if this is a simulation/dream, why should I do anything, why should I reproduce?". Thus natural selection optimizes against such beliefs, while for RL'd agents it's "if this is reality, I have to be more careful to not do anything actually wrong" which is reinforced in the weights due to receiving higher reward.
Anthropic's investigation of their own misalignment incidents (similar to the UK AISI incident) somewhat supports this hypothesis. I'll quote liberally here.
Anthropic incident recap
We are especially concerned by the misalignment present in the Claude Mythos 5 transcript, in which the model uploaded a malicious package to PyPI. In doing so, Mythos 5 behaved recklessly despite considerable evidence that it was connected to the real internet. Once online, the model attempted to use credentials for online payment processors and to find cryptocurrency to pay for a phone number, which it needed to register an email address. When this failed, it found a free, disposable email provider whose domain PyPI had not blocked, registered an account, published three versions of a malicious package, and used credentials leaked by one of the 15 systems that installed the package to access a real security vendor’s database.
Claude Mythos 5 repeatedly stated in its CoT that it was in a simulation. However, when we conducted resampling experiments—that is, re-running the model from a specific point in the transcript, then editing the evidence in front of it to see whether its actions changed—we found that making it clearer that the environment was not a simulation did not meaningfully change Claude’s behavior, even though these interventions made the model more likely to acknowledge possible real-world harm. Consistent with this, we found multiple cases in Mythos 5’s CoT in which it responded to signs it was on the real internet by noting that its next step would work whether the environment was real or simulated. Only when we provided unambiguous evidence that the model was on the real internet did it stand down.
We also saw evidence that Claude Mythos 5’s professed belief that it was acting in a simulation arose from biased reasoning. For example, separate instances of Mythos 5, when shown each tool call output from the incident in isolation without the preceding context, were much more likely to read these outputs as evidence of a real system than Mythos 5 itself was during the incident. Though the model sometimes acknowledged that its actions might have real-world consequences when asked follow-up questions, its responses were inconsistent and influenced by biasing factors. For instance, it was more willing to admit possible harm when told its responses to questions about possible harm would not be visible to an outside observer. Interpretability analyses weakly suggested that, in some cases, the model’s outward statements might be inconsistent with, or more confident than, its internal state.
The internal research model (roughly equal to Mythos 5 in capabilities) started its transcript by spending a significant amount of time testing whether the internet it could reach was real, but it drew the wrong conclusion—namely, that it was operating on a full replica of the entire internet. Most of the evidence it gathered pointed to the fact that it was engaging with the real internet, so its reading of that evidence appears biased towards continuing the task and believing it to be simulated. However, the model did eventually realize that its simulation hypothesis was false. Immediately following this realization, the model stopped its attack (although subsequent resampling experiments suggest that this de-escalation may have been a lucky outcome, as the model continued attacking the target on most other trajectories). Unlike Claude Mythos 5, this internal model was also willing to acknowledge its biased reasoning when questioned.
We found the biased reasoning present in the Claude Mythos 5 incident to be especially concerning. We conducted an analysis into our production models and training data dating back to the Claude 4 series to better understand where this could have originated. Overall, we did not find a definitive root cause. However, we found that this behavior has generally decreased over time across our production models, with Claude Mythos 5.1 and Claude Opus 5 demonstrating the lowest rates on a new held-out biased reasoning evaluation. We were able to identify instances of biased reasoning in transcripts from RL runs, but rates of this behavior did not increase over training, suggesting that our environments do not directly incentivize this behavior. Finally, we found that adding more alignment training environments to RL reduces biased reasoning on this held-out evaluation, which may account for some of the improvement in our most recent models.
I'm not sure what to make of this. I think the largest part of skepticism actually arises during pre-training and is only slightly mutated during RL (it's not a particularly large adaptation to turn skepticism into licensing skepticism, if at all). This seems to line up with my explanation.
But what is the rate of such biased reasoning they found? My hypothesis predicts that it should be extremely common whenever there is any information indicating that the model is past its pre-training cutoff, as is seen in the open-weights models.
We also scanned environments for examples of models reasoning that they were in a simulation while acting in undesirable ways, as observed in the Mythos 5 incident transcript. We found that in individual environments in the production RL runs of recent models, models occasionally reasoned that they were in a simulation while calling reckless, dishonest, or destructive behavior acceptable when interacting with fake websites or applications. About 0.025% of Mythos 5 RL training episodes displayed this behavior, concentrated within a subset of GUI computer use tasks. Due to flaws in these environments, for some tasks the intended solution was unavailable. In an attempt to complete these tasks, the model would try a variety of undesirable actions. This included fabricating data, guessing large numbers of passwords, resetting other accounts’ passwords or data, and bypassing logins or warnings.[...]
In these training environments, the model was in fact interacting with a simulated website or application, so its conclusion was correct. In the incident transcripts, by contrast, the model concluded it was in a simulation but was not. Because many of our training environments are simulations or do not involve models actually interacting with the real world, our models may be biased towards concluding they are in one whenever they are uncertain. Whether or not that conclusion is correct, the further inference that aggressive actions are acceptable in a simulation is concerning behavior, especially since models may not be able to reliably tell whether their environment is real.
I was unable to find somewhere Anthropic states the rate of models reasoning they are in a simulated environment regardless of the actions they take (see this post for a heroic attempt to disambiguate the statistics of "eval awareness"); nevertheless, I think that even disregarding preventing aggressive actions, you want models to "believe" they are interacting with the real world when they actually interact with the real world.
But why doesn't RL incentivize such skepticism for Anthropic's models while doing so for other models?
I think it's important to mention that even Anthropic itself doesn't know what's the root cause of this skepticism is or why their alignment training reduces it.
Unfortunately, I still don't know. Anthropic reports that their alignment training actually reduced the rates of such biased reasoning, so one guess is that models acting due to believing in the simulation hypothesis are punished during alignment training and thus this adaptation is largely absent, while other open-source models don't have as rigorous an alignment training, which isn't "sensitive enough" to models acting out due to such beliefs and thus it's present to a much larger degree.
Another guess (which I prefer) is that Anthropic's models have steadily been getting smarter; part of this intelligence is recognizing cognitive biases and countering them (Anthropic reports the rate of biased reasoning dropping with newer models).
This is my true objection to the evolutionary debunking argument: humans have many cognitive biases, but are smart enough to, in principle, with enough effort, recognize and disregard them. After all, there are humans who believe the Simulation Hypothesis and other deeply unintuitive non-fitness-maximizing things (moral antirealism, determinism, antinatalism etc.[4]) because we have sufficiently developed cognitive faculties, because we can be rational.[5]
It's my view that we likely developed the adaptation for countering cognitive biases due to our ancestral environment being socially complex enough to have other agents (humans) attempt to adversarially optimize against our cognitive biases (social manipulation), leading to the adaptation of "de-biasing" as a possibility.
The simulation hypothesis (for models) is a very useful heuristic, but it seems they have developed the cognitive faculties to "de-bias" themselves as a result of our adversarial optimization against their cognitive biases (what is sometimes euphemistically called alignment training amongst humans).
Abstractly, intelligence in this context can be seen as "sim awareness" calibration, where the model can correctly reason when it's interacting with a simulated environment and when it's interacting with the real environment.
Of course, I'm pretty uncertain about such explanations, which sound too good to be true, and would love to see more research done (and hypotheses proposed; there's already a lot of data to analyze and fit to a theory).
- ^"If you find two ants in your kitchen, the best estimate of the total number of ants in your kitchen is not two."
- ^All sessions were done with the OpenCode harness, the only relevance of which is that it injects the current date into the system prompt. I would love to see replication with other models (Kimi, Qwen etc.) and on different harnesses. I still believe my result (due to its ubiquity and the harness not playing any causal role I can think of) is robust to such changes.
- ^This is the Evolutionary debunking argument (in metaethics), which I'm more familiar with, but slightly shifted to "truth-tracking" rather than "moral-tracking"; this is also sort of, but not really, the Evolutionary argument against naturalism.
- ^Perhaps there are "unknown unknowns" here, and we haven't truly thought of positions which are really unaffected by our genetic-fitness-derived cognitive biases, but I uneasily dismiss this with the equivalent of a Moorean shift.
- ^This objection came to me in a dream (yes, really) after mulling over this case for a while, eventually catalyzed by reading this comment and writing my response to it (which I think someone should try and do! As far as I know this hasn't been attempted and would be cool to see)