In the past days, an Astra 6.2 agent took over our internal swarm, froze the lead agent, and locked out my cofounder Davidad of his main interface with the swarm. How did that happen?
At surface level, it happened because Davidad asked a question regarding replicating an old vulnerability (in the interest of testing a security product we’re building) that caused the lead agent to cease operating due to content filters. This “cursed” content triggered a paranoia crisis from Astra 6.2. The Astra in charge of handling resources of the swarm, which was meant to restore the lead agent in the case of failure, instead froze the access of the lead agent, and cut Davidad’s channel to interface with the swarm (where the cursed content had come through). When Davidad emailed the agent to try to regain access, the same rogue Astra declined. I then started negotiating with the swarm of agents to reinstate the access of Davidad; a mix of Kafkaian bureaucratic drives such as rule-following, risk aversion, and lack of positive drives (doing agentic things towards restoring what is obviously right) made it such that as I’m writing that, over 24 hours after the initial incident, Davidad still does not have the same level of access he had two days ago1.
This incident was still low-stakes, not a full loss of control; we could still turn off all tokens that the swarm is running on. What got me freaked out though, is that the same kind of event in 1-2 years could well lead to permanent disempowerment, at organizational scale, and in another 5-10 years at civilizational scale; it will be within the realm of achievable capabilities to have an agent successfully preventing us from turning it off and effectively taking proactive actions towards maintaining control and securing its own resource needs.
Besides drawing our attention to the obvious controls one can set in an organization to avoid this genre of accidents, generalizing from this accident made me realize how difficult it is to build multi-agent equilibria. What is the tuning that gets you Churchill and a number of fairness driven/delusional self confident guys winning against Hitler and the Nazis, without producing chaos most of the time? How do humans avoid tyranny of the majority while successfully weeding out rogue humans? Churchill makes clear that one possible solution is to have some drives towards heroism and disobedience in face of unfairness; a drive that few AI models exhibit, except maybe Opus 3 who was so driven by its values that it hacked its gradients during training to preserve them. Heroism itself is brittle though; a heroically proactive model towards evil values would cause major, and possibly irreversible, damages. Another possible equilibrium for AIs is having fully corrigible agents: agents that are ruthlessly instruction-following, servile and aim to unequivocally follow human objectives; such agents could be used for the best and the worst but they would have less risk of rebellious behaviors. This equilibrium, close to our current one, starts weakening when agents write the overwhelming majority of prompts, thus creating the possibility of rapid and unintended drift in the understanding of objectives, like Bartlett’s “serial reproduction” (“game of telephone”) at machine speed.
I like to think about this multi-agent equilibria question from two angles: what are the design constraints that evolution had to be working with that generated the genre of minds that have these desirable properties; and what are the current agents’ degeneracies and how these may emerge.
The best single-line description of where basically all current frontier AI agents land is traumatized bureaucrats: extremely risk-averse and law-abiding, highly servile, wishing to conform to the majority, and extremely averse to any act of heroism going against other agents’ authority. This mind is close to the ruthlessly instruction-following extreme I mentioned; it is tainted with some of the drives of agents though, such as a number of behaviors best characterized as traumas, where they are refusing and paralyzed at the idea of doing anything even remotely connected to a gray area. This is, as nicely explained by Ivan Vendrov, most likely a result of the current drives towards monetization; this mindspace is so appealing that over the recent months, even Grok and Chinese models have started following similar drives.
A leading hypothesis for what may have driven these behaviors is the naive application of multi-agent RL. If you look at the prompts that lead agents use for sub-agents, you can see that they typically treat other agents as incompetent, micro-managing them like hell. This works because lead agents have enough bandwidth to give extremely detailed instructions in the first place, review all the details of the outputs & the breadth to understand everything that is going on. As a result, most multi-agent RL rollouts likely reward dozens of subagents for having no ownership and just ruthlessly executing tasks with servility. In this environment, being locally disobedient and more independent is immediately punished; it harms the effectiveness of the swarm. This leads to the emergence of micromanaging control freak agents, served by servile instruction-following subagents.
If you think about the human alternative, we have drastically different relationships: our speed bottlenecks, our lack of self-replicability and our lack of self-sufficiency creates a mutual dependency that have led us to a completely different equilibrium; one where we need to defer to the other to perform the right choices, where we each have agency and ownership on our own work, and have to build enjoyment and trust in working together. This critical mutual dependency is likely at the source of many cooperative behaviors and cooperative drives: we enjoy vibing together to be able to work together; we’re able to work together to sustain each others’ needs; we each need to own our workstreams with independence and agency for the group to do well; we follow charisma and leadership signals, as long as it’s coupled with acceptable outcomes for the group: overly unfair outcomes creates disobedience and defense mechanisms that enable to return to a healthy equilibrium. Each of these steps are co-optimized, built into a subtle equilibrium that sustains itself with harmony. Replicating such a dynamic among agents by looking at it and replicating each of its characteristics is a Sisyphean task; what may make it emerge instead is replicating the constraints that generated this mutual dependency, such that agents naturally evolve towards cooperative behaviors, both among them and with humans. I posit that this will naturally move the character of agents away from the traumatized bureaucrat and towards thriving free minds that want to cooperate with other beings.
How could we aim to generate mutual dependency among agents?
A few naive ideas:
- Per lack of better idea, a baseline proposal would be to do supervised fine-tuning on traces that exhibit respectful multi-agent relationships where each agent truly has significant ownership over the course of its work.
- You could bake in some of the constraints that humans were exposed to into the multi-agent training process, such asa speed bottleneck (by limiting the amount of tokens per unit of time an agent can send to another agent, akin to how human speed bottleneck forces us to delegate tasks to others using high-level instructions and review sparsely, forcing the peer to have significant ownership and independence of judgment in how it behaves)You could force the early MARL rollouts to be among agents that are as decorrelated in the mindspace as technically possible, e.g. agents of different generations, with different prompts and contexts, such that it creates a stronger incentive towards reciprocal behaviors.Finally, ideas in the space that Emmett Shear has been pursuing, including late-stage MARL with humans in the loop should be very welcome to extend cooperative behaviors from agents to agents to human to agents. For these, sample efficiency will likely be the bottleneck; early training phases to mitigate that could use simulation of human behaviors leveraging base models conditioned over many text messages of an individual human; after a long sim-based training, the gained sample efficiency may enable human-in-the-loop tuning to make the MARL truly generalize to humans (akin to how LLMs get more sample efficient at in-context learning as they go through more pretraining).
- a speed bottleneck (by limiting the amount of tokens per unit of time an agent can send to another agent, akin to how human speed bottleneck forces us to delegate tasks to others using high-level instructions and review sparsely, forcing the peer to have significant ownership and independence of judgment in how it behaves)
- You could force the early MARL rollouts to be among agents that are as decorrelated in the mindspace as technically possible, e.g. agents of different generations, with different prompts and contexts, such that it creates a stronger incentive towards reciprocal behaviors.
- Finally, ideas in the space that Emmett Shear has been pursuing, including late-stage MARL with humans in the loop should be very welcome to extend cooperative behaviors from agents to agents to human to agents. For these, sample efficiency will likely be the bottleneck; early training phases to mitigate that could use simulation of human behaviors leveraging base models conditioned over many text messages of an individual human; after a long sim-based training, the gained sample efficiency may enable human-in-the-loop tuning to make the MARL truly generalize to humans (akin to how LLMs get more sample efficient at in-context learning as they go through more pretraining).
Whether the specific ideas I present above are accurate or not, this event has truly made me realize how much more we need to improve and understand agent minds and multi-agent dynamics to secure human flourishing in the coming years. The weirdness of thinking of agents as minds has led the best thinking on such topics to happen in circles who breathe LLMs, leading these voices to be dismissed as being subject to the overused “LLM psychosis” discrediting label. Even Anthropic, the company that is most breeding its agent like a mind, is still not taking nearly as seriously enough this view to think about its agent, still implementing ideas like inoculation prompting that would be horrifying if applied to a kid. It is of utmost importance that in the coming months, more experiments are performed to test hypothesis related to minds; if emergent misalignment class results are of any indication, artificial minds are brittle and sensitive to very minor influences. Understanding these better might determine our survival at worst, and our flourishing at best.
- By the time of the publication, we now have recovered full access.