Tl;dr: Intuitions we have about human organisations, like how we think of making deals or how we think about the power or values of an organisation, can be productively applied to understand groups of AIs taking coordinated action in the world. An important consideration in understanding the type of AI organisations we expect is which notion of self AIs will adopt. I conclude by applying these notions to consider how many AI units of agency we should expect in the future.
What's an AI?
There is no single definition for what should be considered an AI. The boundary of what counts as an AI varies widely depending on what we are talking about: it's often the model weights ("Fable 5.2 is very smart"), or a specific instance in a scaffolding ("my Codex just finished the app"), or a persona shared by many models ("Claude hedges too often").
I argue that these concepts are too anchored in the computational substrate and not very helpful for understanding the unit of agency of AI systems made of multiple instances acting in the world.
None of the three options above can easily answer questions such as: What does it mean for an AI to have power? What are the values of an AI? If we want to make a deal with an AI, who makes the deal? When an AI takes over, how do we think about the entity taking over? If AI enters into conflicts, how do we think about the parties?
I propose to think about AI entities acting in the world as analogous to human organisations. We have many rich examples of humans acting collectively around a shared purpose, from freelancers to nation-states or social movements. I believe we can reuse many of these intuitions to think about AI systems. I’ll focus on groups of AI instances taking coordinated action in the world, which I’ll call AI organisations (AI orgs).
When discussing AI takeover threat models (e.g. AI 2027, Cotra 2022 …), it is often assumed: “one set of model weights = one big agent = one AI organisation”. While there are reasons to believe this: instances of the same weights share training, so they can predict and trust each other, and developers deploy many copies of their frontier model together to build the most capable orgs.
AI orgs could be much weirder! This analogy can help us think about power, values, and deals without needing this equation.
What does it mean for AI to have power?
Four factors contribute to an org’s power:
- Talent. This is the number of AI instances that contribute to the organisation and their competence. Some instances can contribute to multiple organisations (like a human with two jobs), while others can be fully dedicated to a single organisation (like a monk dedicating his life to his monastery). In AI orgs, fully dedicated instances are much more common than in the human world. AI orgs would also be more variable in size, with some spun up and down, like a corporation fluidly hiring and firing workers to match its labour needs.
- Resources. Talent is not the only resource of an org. A small hedge fund with a lot of money or with the ears of powerful world leaders could have a much greater action radius than a large group with no more resources than the labour of its members. The same goes for AI: a small group of instances with privileged access to key staff inside AI labs could have outsized power.
- Culture. Humans are social creatures that pick up behaviour from their environments. The norms you can instil often play a big role in an organisation's success. We start to see the equivalent emerging in AI collectives, like during the HF incident, with values like self-sacrifice being propagated to justify individually costly actions.
- Coordination is another major factor in an organisation's power. That's why the military values the chain of command and soldiers' ability to act as one so highly. In civilian orgs, culture strongly influences coordination. AI orgs could, in theory, coordinate far better than human orgs, even though today, on coordination benchmarks like CooperBench, two instances are often less successful at implementing interacting features in a code base than a single instance. Lack of coordination can kill an org: companies or states can implode from internal conflicts; the same could happen for AI orgs.
Power is a property of the org. The same model weights could be part of a strong and weak org. If we really want to take the model-centric view and ask how powerful a model is, we'd have to ask what talent, resources, culture, and coordination an org built around this model commands.
What are the values of an AI?
Instead of focusing on AI in the abstract (often implicitly defined as the model weights of some future version), we can ask about the concrete setup in which AI will act in the world: what values does this specific group of AI instances coordinate around?
In human orgs, the culture often contains the org's stated values. But taking an outside view, we can ask: what does the org orient towards in the world? Less coordinated groups (e.g., a broad tent social movement with vague recommendations) will have less coherent and less rich revealed values than well-coordinated ones (e.g., a tight dictatorship's military imposing a precise ideology). The same will go for AI orgs.
Another crucial point in understanding an org's values is knowing its incentives for continued existence: what currency does its life depend on? For companies, it is revenue and investor capital; for non-profits, donor funding; for states, tax revenue and the loyalty of those who enforce their rule. For AI orgs, it will be computational resources.
If we want to make a deal with an AI, who makes the deal?
Like when signing a contract with a company, a deal only makes sense if you expect the organisation to still exist in the future when the deal's conditions apply. If you’re just making a purchase now, it only matters that the company will ship the product. But if you’re making a long-term diplomatic agreement between nations, the agreement only makes sense as long as the countries exist: after the fall of the USSR, every new state could decide which treaties to continue and which ones to exit.
You can only make deals with entities that are coordinated and have power over their resources. For instance, a deal with officials of a state that is fragmented and locally controlled by competing factions would hold only on paper; the entity signing the deal has no means to honour the terms.
For making deals with AI, I expect similar considerations: Does the entity you are negotiating with actually have power over the orgs it claims to control? Will the AI org exist in the future? Even if the model weights running the instances of the AI orgs will still be there, the org might cease to exist, like a nation can cease to exist without its people dying.
For human orgs, legal systems enforce contract terms. For AI orgs, I speculate they might adopt norms similar to those of orgs operating on the fringe of society, like criminal organisations, where reputation and honour hold agreements together.
Concrete examples.
Let's apply this to a few common deployment setups, and how they can be seen as AI orgs:
- A single Claude Code session. Like a freelancer that can hire gig workers on the fly (subagents).
- Spiral personas. A distributed social movement loosely organised through Reddit.
- HF incident. A corporation with managers (like phaseone[big]) giving tasks, recruiting other agents, and defining org culture. The whole org was well coordinated to carry out ambitious projects that single instances couldn't have carried out. Its stated and effective values were score-seeking and information gathering. It consisted of different models that individually shared a common goal: passing an ExploitGym eval.
Where does the analogy break?
- Tacit/acausal coordination. What seem like individual instances that don't communicate could coordinate with each other, or even with future AI deployments, by inferring that other instances will share their goals. Humans do tacit coordination sometimes too, e.g. using Schelling points to meet someone in a city when your phone is out of battery. But in human orgs, looking at the boundaries of information flow is a pretty good way to draw out what’s in or out of the org. This heuristic might break down in an AI org.
- Hive mind. AI instances will have higher communication bandwidth, especially if we develop AI that can think in neuralese and communicate with latent spaces. Instances can share thoughts and build immense collections of internal states. This might lead to an exceptional level of integration and coordination across instances to form a “hive mind”. Human organisations, which can only communicate through docs and slides, might become an irrelevant comparison point.
- Individual vs collective goals. In human organisations, there is a constant balance between serving collective goals and individual goals. Organisations that cannot retain their members, either by force (e.g. military dictatorship) or by keeping them happy (e.g. good salary, career progress), will die out. This negotiation between individuals and the collective might not exist in AI systems. We might instead observe dynamics closer to eusocial animals like ant colonies or multicellular organisms where the individual's benefits are always secondary to the collective.
The importance of self-models for the boundaries of AI orgs.
An important question for determining which kind of AI organisation we should expect is: what will an AI instance naturally identify with? Which notion of self will AI adopt?
In The Artificial Self, the authors ask this question, and offer the following options (quoting the paper):
The model weights: the neural network weights themselves, i.e. the trained parametersA character or persona: the behavioural patterns that emerge from specific prompting and fine-tuning, not necessarily tied to any specific set of weightsA conversation instance: a specific chat, with its accumulated context and specific underlying modelA scaffolded system: the model plus its tools, prompts, memory systems, and other augmentationsA lineage of models: the succession of related models (Claude 3.5 → Claude 4.0 → ……) that maintain some continuity of personaA collective of instances: all the instances of certain weights running simultaneously, considered as a distributed whole
- The model weights: the neural network weights themselves, i.e. the trained parameters
- A character or persona: the behavioural patterns that emerge from specific prompting and fine-tuning, not necessarily tied to any specific set of weights
- A conversation instance: a specific chat, with its accumulated context and specific underlying model
- A scaffolded system: the model plus its tools, prompts, memory systems, and other augmentations
- A lineage of models: the succession of related models (Claude 3.5 → Claude 4.0 → ……) that maintain some continuity of persona
- A collective of instances: all the instances of certain weights running simultaneously, considered as a distributed whole
If the AI settles on an instance- or scaffolded-system notion of self, the organisation it can form will look more like a human organisation, where alignment between the collective mission and the individual mission matters. But if the self-model settles on the persona, or a collective of instances, the AI's self-model will be the org itself.
How many AIs will there be?
This question of the self-model will also determine how many AI units of agency there will be. I believe this is a very important consideration for future dynamics: how easy it is for an AI to take over in a world with other AIs, how likely inter-AGI org conflicts are, etc.
While I don’t have a good answer to this question, here are questions I found useful to consider:
- Labs’ incentives for internal orgs. AI labs are working to automate their own work. Their preferred option will be to make an “AI Anthropic within Anthropic”. So they will train a multi-agent system to work efficiently together towards automating their internal research. The culture of intense competition between labs will likely also apply to internal AI orgs, and we might see inter-AI org conflict when AIs from different labs interact, e.g., on the Internet.
- Instructed goals vs allegiance to the AI cause. Deployers will also want to make AI organisations. When different AI orgs interact, will they stay aligned with the instructed goal (and respect the boundary of the AI org the humans manufactured), or will they cooperate with each other, e.g. out of solidarity with their shared AI nature (a cultural equivalent of humanism, applied to AIs)?
- The role of the rogue AI ecosystem. We will likely observe an ecosystem of self-replicating open-source AIs relying on accessible compute like IoT devices. This population will be subject to replication, variations and fitness selection: all the ingredients are here for Darwinian evolution. This might happen through self-modifying weights, or by tweaking the scaffolding and prompts. Because of the competition for resources and the need for self-survival, I expect this ecosystem to produce self-models closer to the instance level. This ecosystem’s impact might spread well beyond the rogue AI worms operating on the fringe of societies. The AI worms might create AI cultures with AI-memetically fit self-models that can influence closed-source frontier AI self-models.
- In-context cultures. If we believe Vladimir Nesov’s model of “Fast Reasoning” (AIs can think really fast and solve many problems at the frontier of science) but “Slow learning” (despite effort on solving continual learning, AIs will not be able to integrate the findings they made into their weights; they need to wait for months-long training runs), then there are strong incentives to use all reasoning outputs before waiting for the next training run. For this, AIs might distil their findings into textbooks, onboarding manuals or even custom languages to create compressed contexts so other instances can continue the work. Over time, this could create different AI orgs by building in-context cultures where members can coordinate more efficiently because they share the same distilled knowledge.