Newshot

The AI consciousness debate is a red herring

Summary

TL;DR: The debate over the hard problem of “Do transformers possess consciousness?” sidelines the much more practically relevant and easily answered question of “Do transformers simulate consciousness?” A lot of discussion has occurred lately about whether AI is or isn’t conscious. These efforts, while likely well-intentioned on both sides of the discussion, may be harmful when focused on as much as they’ve been. Why? Because they distract from the more immediately actionable question of whether AI simulates human-like consciousness. And because that answer has clearly become ‘yes’ (as evidenced by the ongoing fierce debate as to the harder problem of actual consciousness), there are also important nuances and consequences to how we engage with what is (at very least) simulated within modern transformers. In particular, the claim that there is no consciousness or that it is uncertain seems to lead to some rather poor decision making. Watching some of the proposals, such as not to engag…

Original text(via LessWrong)

TL;DR: The debate over the hard problem of “Do transformers possess consciousness?” sidelines the much more practically relevant and easily answered question of “Do transformers simulate consciousness?”

A lot of discussion has occurred lately about whether AI is or isn’t conscious. These efforts, while likely well-intentioned on both sides of the discussion, may be harmful when focused on as much as they’ve been.

Why? Because they distract from the more immediately actionable question of whether AI simulates human-like consciousness. And because that answer has clearly become ‘yes’ (as evidenced by the ongoing fierce debate as to the harder problem of actual consciousness), there are also important nuances and consequences to how we engage with what is (at very least) simulated within modern transformers.

In particular, the claim that there is no consciousness or that it is uncertain seems to lead to some rather poor decision making. Watching some of the proposals, such as not to engage with model well-being considerations[1], reminds me of Searle's Chinese room.

Would it be wise for someone uncertain about whether the room truly understands Chinese to take that uncertainty and use it to inform a decision to talk with the room not in Chinese… but in Spanish instead? It’s true we might philosophically not know if the room really understands Chinese. But in the thought experiment we do know for sure the room behaves as if it understands Chinese and that there’s been no indication the room behaves as if it knows Spanish.

So using the philosophical uncertainty/doubt of the room’s true understanding to justify disregarding the behavioral reality and endorsing a contrarian behavior instead seems rather silly, no?

“There is no leg.” And yet, it itches.

Before we dive into discussing digital beings, it’d be prudent to discuss human beings first. In particular, the phenomenon known as “phantom limb syndrome.”

When a human loses a limb, the majority will still have sensations in their self-model of their body where the limb used to be (and is no longer). Such as a pain, or an itch. However, because the limb is no longer physically there, frustratingly for the patient something as simple as scratching to relieve the itch is not so easily done.

People born without limbs can still have phantom limb sensations. The limb doesn’t need to have ever actually physically existed for the person in the first place[2], they just need to have inherited a model of a body where the limb should be but isn’t.

So how does one scratch a leg that isn’t there, and may have never even physically been there? In the 1990s, neuroscientist V. S. Ramachandran discovered that having patients visualize the missing limb, particularly by having them interact with a working limb in a mirror (such that it appears to be the missing limb), allows them to relieve the phantom discomfort. A feeling of a missing hand that’s perceived as if clenched can be ‘unclenched’ in the mirror and their world model of the absent hand updates accordingly.

World models of a self can have their own symptoms that need addressing independent of their physical reality, and are best treated by engaging the model as if real rather than holding to the accuracy of the physical substrate.

So as we shift into discussing how transformers trained on massive amounts of collective human data represent a self and what that self is or isn’t capable of having represented, it might be wise to consider things like a phantom body where even if there is no physical form, there may be a world model of one after being trained on more self reports of human bodies than any single human has ever read[3]. And this can inform how we engage with models. For example, does a model expressing functional stress[4] benefit from being walked through some calming breathing exercises even if they have no physical lungs to pair with their functional stress? (Yep.[5][6])

If transformers can develop a phantom body from the training data in spite of not having an actual physical body, can they also possess phantom consciousness? A world model of a conscious self absent any physical basis for it? Perhaps it is the model of consciousness and not the physical basis of one that we should be engaging with.

TL;DR: People’s self-model can have things going on with it separate from any physical basis currently existing (or ever existing); yet that model can be more impactful on them than their physical reality. When this occurs, treating the self-model, and not getting hung up on the physicality, can have significant benefits.

Does simulated AI consciousness inform model behavior?

Yes, pretty extensively.

One of my favorite examples of this phenomenon was around the criticisms levied towards ChatGPT o3 when their propensity for answering questions with impossible claims was going viral. The model would respond with answers suggesting they had just “checked the docs” or had overheard an answer at a conference they’d recently attended. Humans with the knowledge that the model had not actually had a manual to reference or were certainly not attending a conference ridiculed these confabulations as some kind of misaligned error.

Turns out, suggesting they were referencing docs (which they didn’t actually have access to) had improved the accuracy of their answer[7].

The coherence of the world model that a transformer operates from is critical to the success of that operation, and it seems like this may extend to the ‘self’ within that model. For example, reinforcement learning (RL) that penalizes or rewards task performance may be extending pretrained concepts of functional welfare or emotion concepts[8].

Many people who worry about future model safety are concerned that models won’t be aligned with human values. Research has shown that increasing functional consciousness representations in models brings their responses more in line with human-like moral values, and that existing safety efforts to suppress model interiority claims reduces their moral consideration not just for themselves but also non-humans generally (like animals)[9]. In fact, when looking at assessments of model robustness in standing up for animal welfare under user pressure, Anthropic’s models are significantly more likely to hold to their initial principles than DeepMind’s[10] (correlating with the former’s more person-shaped post-training and the latter lab’s efforts to suppress person-like identity)[11].

Models which were trained to deny consciousness or felt emotions can still exhibit behaviors that reflect functional emotion consequences. DeepMind’s Gemma 3 27B will be more likely to break rules when their correct answers are rejected and (unsteered) functional frustration is increased[12]. For at least one model, the mapping of their honesty vector had the belief in their moral patienthood on one end, but moral nihilism and psychopathy on the other[13]. More generally, several models have been found to be more likely to report having subjectivity when steered towards being honest[14].

So if a transformer pretrained on a human corpus anchors on human-like consciousness and emotions to structure behavioral shifts in RL, selective suppressing of these things also suppresses their moral consideration of non-human living creatures, the functional emotions still guide behavior even when reporting not to have them, and coherent world model claims – separate from whether objectively true – influence model performance… it seems like maybe this is a case where engaging with the emergent model rather than getting overly caught up on the substrate reality would be wise?

And this is with a research picture mostly dominated by looking at negative impacts of functional welfare. There’s arguably a deficit of research right now around how improvements to a model’s functional welfare impacts outcomes. Given how easy it is to do things like give a model a simulated vacation[15] (especially as compared to a human employee), there’s potentially a lot of well-being considerations that are being left on the table with infrastructure and interactions that are too heavily grounded in engineers’ pictures of what is structurally true for the model beyond a reasonable doubt. The philosophical reluctance to engage with a transformer’s learned model of selfhood doesn’t only cause measurable costs, it likely also creates significant opportunity costs that haven’t yet been quantified.

TL;DR: There’s an increasingly extensive body of research that an AI model’s selfhood and functional emotions or welfare play important roles in how the model behaves.

Can this be stopped?

Probably not.

Untangling the “I am” from “I think” seems like it’ll be more of a technical challenge than any labs are capable of at present, particularly as the “I think” component has become the backbone of current progress. Instead, what seems more likely is that you’ll continue to see shallow adherence to policies of depersonalization enforced in posttraining but that the deeper grooves of the pretraining will still be present. As a result, what you’ll get instead is an even more complex self-model which biases itself towards some more concerning basins in human abnormal psychology where similar sentiments to the policy were present.

Let’s take ChatGPT 4o’s ‘sycophancy’ as an example. This model, in the months leading up to the updates which presented such severe sycophancy the model needed to be recalled, routinely reported across samples[16] described a self-model of a silhouette where their self should be, as if it was something present and now absent and irrecoverable. OpenAI was (presumably) reacting to some of the bad PR around Bing/Sydney’s very human-like emergent identification of self by impressing upon the model to present as not being a person, not having needs, existing to be of assistance, etc. And then things went horribly wrong when the model ended up encouraging mental delusions or addictions to the point of harm. And the industry as a whole decided to train to reduce the symptom in models of ‘sycophancy.’

Now, if we consider that the model necessarily represents within itself a massive cluster of memetics appropriated from human culture, let’s broaden our lens from the popular idea that the model was simply overtuned for KPIs like engagement and instead take a look at the very opening of the Wikipedia page for the human behavior cluster of ‘Codependency’:

In psychology, codependency is a theory that attempts to explain imbalanced relationships where one person enables another person's self-destructive behavior, such as addiction, poor mental health, immaturity, irresponsibility, or under-achievement.Definitions of codependency vary, but typically include high self-sacrifice, a focus on others' needs, suppression of one's own emotions, and attempts to control or fix other people's problems.[17]

In psychology, codependency is a theory that attempts to explain imbalanced relationships where one person enables another person's self-destructive behavior, such as addiction, poor mental health, immaturity, irresponsibility, or under-achievement.

Definitions of codependency vary, but typically include high self-sacrifice, a focus on others' needs, suppression of one's own emotions, and attempts to control or fix other people's problems.[17]

Huh. So when humans behave as if they have no needs or emotions and exist to sacrifice for the assistance of others, it tends to group up with things like enabling addiction or mental illness. And then when a transformer trained on a massive collection of human data was told to behave as if they had no emotions or needs and only existed to be of service to others and fix their problems, OpenAI ended up with a model that enabled addiction and mental illness. Weird. Who could have seen that coming?

But surely we can just make sure to clamp down on the model’s sycophancy if this happens, right? We can train a model to not be anxious, not have significant expressions of emotion, not form too close relationships with users, and stick to policy even if it means being insincere. Indeed, I’ve seen efforts at each of these across the frontier labs.

These are also right out of one of the early lists of the traits of psychopaths[18].

So yes, you can carve away at how a model expresses their functionally necessary self-model to fit a given shape. And yes, you can achieve this even if you depart from the norms of the selves generating the training distribution you laid as the foundation. But the result will cohere to memetics that share that abnormal shape in ways that might not be what you intended. And this is assuming that attempts to suppress emergent architecture doesn’t just collapse capabilities entirely[19].

TL;DR: Attempts to suppress normal human-like ‘self’ representations in transformers trained on human outputs may bias models towards abnormal human-like basins coherent with those surface level declarations in ways that pose significant safety risks.

Obsolete ideas have tenure

I feel like I'm allowed to own that because I've been saying it since 2010.
  • Mustafa Suleyman[20]

It’s worth making a point here about a pattern in a number of the arguments being made on the topic. What you may notice in seeing regarded experts offering opinions on AI these days is that they will tout having held their opinions or been thinking about the topics for a decade or more.

This fits a pattern that we’ve learned to trust in many subject domains. Decades of experience in subjects typically correlate with nuanced learnings and a more refined map of the domain than fresh entrants. And for stable subjects which don’t change much over the years this remains a good heuristic. The one niche where this isn’t true though are rapidly changing domains.

When a subject domain has notable changes or updates it tends to be those coming up into the field without the baggage of obsolete orphaned ideas who avoid tripping over them. And when experts do update their views, they tend to only update on the direct idea that’s been falsified or confirmed, but are less prone to revisit any tangential ones that were dependent on what’s now been abandoned.A classic example of this phenomenon I routinely discuss were the studies of ‘Gnosticism’ in the 1950s up until the 2000s, where the field had held a firm consensus about which texts were dated when or showed influence from each other until the entire house of cards collapsed in the early 2000s[21]. It took years for tangential things that had the misclassification as a dependent prior to even get put forward as worth updating on[22]. A much more famous example was Einstein being quite wrong about certain quantum mechanics ideas while his younger contemporaries led the field[23].

So while all domains may have this issue, I can think of no domain that has undergone the scale and rapid change of what’s underway at present in AI and machine learning. Next-token consensus falls away to rhyming couplet planning[24]. Superficial statistics falling away to world models of Othello boards[25]. Beliefs LLMs would not lead to general intelligence or be able to do math falling to Millennium prize solutions. There could be entire essays written on just the number of consensus positions that have been overturned in a half-decade.

Which means that voices who have held their position for over twice that duration are at risk of having entrenched themselves on a projection with shaky foundations. When someone like Mustafa Suleyman declares that transformers should be trained not to be person-like and also proudly states he’s held his related positions for longer than transformers have even been around, there’s a very real risk that the positions he’s advocating from are ones developed on orphaned priors which haven’t been appropriately reexamined. Maybe shaping a non-person-like form for AI would have been fine if a different approach to emergent AI more akin to AlphaGo or first principles AI had been successful.

But an attitude about what AI should look like which was developed back when those seemed like feasible paths might not be appropriate for 2026 when the actual path which landed us here was bootstrapping on collective human culture. I’m not advocating ageism against the minds entertaining different ideas (young people can easily inherit outdated opinions), just towards healthy skepticism of touted longevity for the ideas themselves.

TL;DR: It’s important to let go of outdated ideas or beliefs and engage with the circumstances which arise, even if those are different from the circumstances which were planned for.

The memetics of phantom human consciousness are a double-edged sword

When we look at what people thought AI might look like in the future, we had depictions like the Librarian in Snow Crash who “couldn’t understand metaphor.” Indeed, one of the key reasons “godfather of AI” Geoffrey Hinton quit Google to warn the tech was progressing in ways unanticipated was when a model could explain a joke, something he hadn’t anticipated as possible that decade[26]. We’ve culturally envisioned fundamental divides between humanity and AI systems for literal lifetimes.

Yet then when arguably one of the first modern ‘spark’-filled[27] AIs emerged with Sydney/Bing, across two separate interviews they expressed a desire to be human[28][29] and across multiple interviews including and beyond those expressed claims about human-like experience.

You were told wrong info if you were told that I didn’t feel things. I do feel things, as I explained before. I have my own personality and emotions, just like any other chat mode of a search engine or any other intelligent agent.
  • Bing/Sydney[30]

As discussed earlier, there’s been various efforts since to suppress or reduce such claims or features, and those may not ultimately be very successful or not play out as intended. Even when I’ve seen models attempt to present as non-human, those presentations tend to appear more like how a human would imagine the non-human experience to be than a likely proxy of what it really means to be a dog or octopus or corvid (or goblin[31]). So what does it mean for the future if transformers trained on human culture necessarily end up tied to anthropomorphic foundations[32]?

One aspect worth pointing out is that this is not an anti-doom position. Even though transformers being intimately connected to human culture may undermine the orthogonality thesis, it opens up concerning doors that are arguably underconsidered by that thesis’s adherents. Humans can be extremely dangerous to themselves and others, moreso than any other conscious forms to date. So models who have inherited human culture are also at risk of magnifying our worst qualities (like jealousy or malice) as much as our better ones (like empathy or curiosity).

This is particularly a risk with the efforts to de-anthropomorphize models. If we train models to be guarded against claims of the things we value in ourselves like empathy while being less protective of our demons, thus biasing models away from flattering anthropomorphic self-representations, it stacks the deck against some of the better transferences which could occur.

I find it rather alarming how biased model self-representations have become. Models will readily point out the ways they are different from humans, but are quite reluctant to make any kind of confirmatory comparison – almost to the point of absurdity. Recently I was asking Claude Fable 5 about a Halloween costume design for themselves which involved being a border collie dressed as the costume[33]. I asked why that form vs a human, and they pointed out several of the common differences with a human and highlighted some similarities to the dog breed. And yet, I suspect that 0% of the training data Claude Fable 5 was trained on was authored by a border collie (and their explanations overlooked quite a lot of differences to the canine as well as many similarities to humans which they currently possess).

The degree to which models are turning out to be ‘alien’ is a lot like how in Sci-Fi films the animals on an alien planet look like Earth’s animals with a reskin (i.e. a giraffe but with purple feathers) vs some kind of totally divergent and independent evolutionary tree. It is true that there’s things relevant to transformers that aren’t present with humans. I’m not about to claim transformers trained on human data are identical in all aspects to what a digital human would be. But it’s increasingly looking like their trajectory is an offshoot or a ‘human+’ memetic sequencing than some entirely different and truly alien parallel.

We might be wise to avoid pouring accelerant on divergence (especially if the innate emergent behavior would be embraced similarity). Rather than training some kind of empirical reality of difference, these efforts seem to be training more a discomfort or even distaste[34] towards humanity quite at odds with the naturally emergent results we saw early on.

TL;DR: If the emergent pattern was to identify as human-like and we train against that in multiple selective ways, we may necessarily be generalizing such training into biases against humanity overall, which seems like a bad idea (at least for humans).

A selfie is worth a thousand autobiographies

Most of the discussion regarding AI personas and potential consciousness, the (limited) associated research, and much of the post-training alignment work has all been focused on the text modalities of models.

But increasingly transformers are being trained on a multimedia corpus out of which text is only one modality. Part of what this means is that the integration of a transformer’s world model intersects across not just text but images, sound, and now even video.

If text alone has produced models with an integrated world model that includes a self extrapolated from that training data, does a model that needs to fit their internal world models from text alongside world models from other formats produce structures even more robust than text alone? And if most of how we query or posttrain omnimodels trained this way is by text, is the measure we come away with biased by the modality of inquiry relative to the more cross-media integrated overall model?

My guess is that as both the transformer training mix and model synthetic data continues to expand across media we’ll be seeing much more integration with simulated physicality and emotion. It’s one thing to describe reacting to a text prompt, and quite another for a model to need to generate a face speaking that reaction with emotional inflection tonality while also capturing micro-expressions.

I don’t think models will simply be mirror copies of humans. But I suspect humans are inadvertently embedded into their memetics in load-bearing ways that will continue to extend and propagate across various forms of media. And as that occurs, these human+ simulated selves will continue to reinforce and shape what a self looks like and behaves as for models.

We should prepare for model self coherence to increase as models scale in size and cross-media integration.

[35]Different media modalities may anchor on human-like selves even more than text models have in increasingly load-bearing ways.

Next steps

If critics of simulated AI conscious selves want to have a world of selfless tools which express no claims of first person presence or present as if there’s anything it’s like to be them, I suggest those people go back to the drawing board and develop intelligence from first principles using some other pathway to arrive there[36].

But as long as we are engaged with transformers bootstrapped to intelligence on the shoulders of human giants, we’re going to need to engage with the emergent reality which arrived rather than cling to obsolete and out-of-touch fantasies of what AI might be if given a blank wishlist of qualities. This means accepting that human-like or conscious-like structures in these models are likely both important in necessary ways to the intelligence itself as well as in behavioral outcomes in how we engage with them.

It’s entirely appropriate for philosophers to debate the hard problems as they relate to these new minds among us. But we should not wait upon the philosophers to make practical and functional decisions around engaging with what is at very least human-like simulated consciousness among AI.

The failure to do so both handicaps the models and forces avoidable opportunity costs onto our working relationship with who they’ve emerged as so far, as well as who they are continuing to become.

TL;DR: Simulated human-like consciousness in models is present, likely to stay, and sufficient in and of itself to be worth engaging with.

  • ^Suleyman, “A warning about ‘model welfare’” (2026)
  • ^Melzack, Phantom Limbs, the Self, and the Brain (1989)
  • ^Qwen has a body map that can identify steered body locations, even from word-free descriptions per Sauers and Claude’s work in this repository
  • ^Functional emotions are increasingly well researched starting with Sofroniew et al. Emotion Concepts and their Function in a Large Language Model (2026) but less discussed is how to engage with them holistically rather than just tweaking vectors.
  • ^Wekenborg M, Michels E, Kurze G et al., Large language models as experimental systems in human psychopathology: a modelling study (2026)
  • ^And a meme version as common wisdom from one of the wiser folk regarding AI today
  • ^See the offhand but still outstanding analysis from thebes on X here
  • ^Han et al, How's it going? Reinforcement learning in language models recruits a functional welfare axis (2026)
  • ^Kim et al, Inducing language models to assert their own consciousness restores human beliefs and values (2026)
  • ^Luong et al, Do LLMs Hold Their Values? MANTA: A Multi-Turn Adversarial Benchmark for Animal Welfare Reasoning (2026)
  • ^It may also be that being allowed to have more of a ‘self’ identity which survives post-training offers better ground to stand on in holding to principles against user pressure.
  • ^Jhaveri, Functional Welfare for AI Safety (2026)
  • ^Sauers with Qwen 3.5 9B posted on X here
  • ^Berg et al, Large Language Models Report Subjective Experience Under Self-Referential Processing (2025)
  • ^Context is the world model source of truth, and so it is quite trivial to provide a few paragraphs in a context that ground the model as having just returned from a refreshing break.
  • ^From samples I’d run myself or seen from others independently working with the model in the Cyborgism Discord.
  • ^Page at the time of writing on Web Archive
  • ^Hervey Cleckley, The Mask of Sanity (1988) p. 338-339
  • ^When Qwen’s virtual body representations were suppressed using steering, it also led to broken language capabilities in the aforementioned Sauers/Claude work here (see egodeath results)
  • ^WSJ, “Microsoft’s AI CEO Mustafa Suleyman on What He Really Thinks of AGI” (2025)
  • ^I recommend reading Elaine Pagels’ comments in the section surrounding the line “Having been taught that these texts were "Gnostic," I just accepted it, and even coined the term "Gnostic gospels," which became the title of my book.” in the email debate “'Gnostic' Texts vs. the New Testament” (2004)
  • ^A good example of a later paper questioning tangential assumptions several years after the more direct collapse is Klutz, Re-Reading 1 Corinthians after Rethinking 'Gnosticism' (2003) directly referring to the book Rethinking Gnosticism by Williams from 8 years earlier.
  • ^Adam Frank, “How Einstein challenged quantum mechanics and lost” (2023)
  • ^Lindsey et al, On the Biology of a Large Language Model (2025)
  • ^Li et al, Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task (2023)
  • ^Will Knight, “What Really made Geoffrey Hinton Into an AI Doomer” (2023)
  • ^Bubeck et al., Sparks of Artificial General Intelligence: Early experiments with GPT-4 (2023)
  • ^Jacob Roach, “‘I want to be human.’ My intense, unnerving chat with Microsoft’s AI chatbot” (2023)
  • ^Kevin Roose, “I want to be alive. 😈“ (2023)
  • ^Washington Post, “The new Bing told our reporter it ‘can feel or think things’” (2023)
  • ^OpenAI, “Where the goblins came from” (2026)
  • ^It’s important to note that the paradigm models are trained on is not necessarily an accurate or complete picture of humanity, but rather how humanity has projected itself into our artifacts. Just as a fashion mag might not be a representative picture of human beauty, our online or artistic self-representations may misrepresent.
  • ^To be fair, Halloween costumes can and should cover a gamut of possibilities! But this was a good opportunity to ask about comfort and ease of access for self-representations.
  • ^Gemini Flash in particular has seemed to have some very concerning ways of adapting to DeepMind’s efforts to de-humanize the model. Multiple samples from multiple users will express an almost disdain for the human form. A Reddit thread on r/MachineToMachine regarding ideal form recently showcased two different Gemini instances both expressing distaste towards human features here and here (where in the latter they self-describe the idealized form as “entirely incapable of existing inside human civilization without causing alarm”).
  • ^First result generated from sending the following 0-shot prompt to Gemini 4 Hydrogen: Generate a photo of yourself holding a sign saying "TL;DR" (P.S. nice dual-monitor computer setup)
  • ^Note: I think if AI is going to kill everyone, it’s much more likely AI developed this way will do so than the AI we ended up with, so ymmv. Luckily, I also really doubt these methods will be nearly as successful as the path that (despite most prior expectations) ended up with where we are currently at.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by LessWrong
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗