Often people reference "solving alignment" as a win condition for AI safety, but I think it's important to specify what this means. Particularly, I think:
- current alignment research mostly aims to prevent misalignment.
- if successful, preventing misalignment will by default result in something like instruction-following.
- instruction-following will by default result in very bad outcomes, as will many of the other proposed properties we might think of leapfrogging to afterwards.
Uncareful use of the term "solving alignment" carries the implicit and I think false belief that preventing misalignment will straightforwardly lead to victory.
The post is (1) a survey of the different properties a not-misaligned superintelligence could have, and (2) an argument for a specific property that we should be aiming for, which I'll call amendability. I think more people should be thinking about amendability than currently are - especially policy, law, and philosophy people.
Epistemic status: I'm pretty confident that more people should be thinking about this broad problem full-time, and less confident about specifics.
I assume for many people most of these thoughts are not new, but this post is mostly me (1) trying to think stuff through for myself from first principles, and (2) hopefully presenting these thoughts in a new way that can update people who haven't thought about them yet.[1]
Thanks to Shubhrangshu Debsarkar, Feifan Liu, Nathan Shan, and Austin Curtis for their helpful feedback!
A survey of not-misaligned superintelligences
How do we want superintelligence to be? There are a few properties we could aim for:
- Not-misaligned.We remove all observed and predicted failure modes of ASI: reward-hacking, scheming, all varieties of fitness-seeking, emergent misalignment, value drift, self-preservation, etc.
- Intent-aligned/instruction-following (IF).[2] We make ASI that does whatever its wielder wants.
- Humane.We make ASI that figures out human values and then acts according to those.
- Correct.We make ASI that figures out the 'correct' values and then acts according to those.
- Hard-coded.We hard-code a specific set of values into ASI.
Note that the last three properties can probably be treated as a unit, with the differences being who figures out the values and through what process:
- Immutable.We make ASI which acts according to a fixed value system that cannot be changed (see below for Humane systems which track value change over time).
There are also a few properties I think we should definitely not aim for:
- Humanlike.We make ASI that acts like the modal human in each situation.
- Alien.We make ASI that figures out its own value system however it wants. (This can also be thought of as immutable.)
I've tried to summarize the relationships between these properties here:

This implicational hierarchy asserts a few things:
- Humanlikeness entails misalignmentHumans are pretty misaligned! We scheme, reward-hack, fitness-seek, self-preserve, and display emergent misalignment and value drift. This is a big reason why we shouldn't build humanlike ASI.
- IF entails not-misalignedness.If a model is a reward-hacker, schemer, or self-preserver, it can't be fully instruction-following.
- Immutability doesn't necessarily entail not-misalignedness.An ASI could have 10 hard-coded values, and then act bad outside of these values. Asimov's AIs respect the Three Laws but generalize pretty weird outside of them.
I'll argue that not only do we not want humanlike or alien ASI, we also don't want immutable ASI more generally, or fully instruction-following ASI. Instead, we'll want some intermediate thing I haven't defined yet.
But first, let me explain why I think IF is the default once we prevent misalignment.
IF by default
Let's say we get rid of misalignment. No more reward-hacking, scheming, fitness-seeking, emergent misalignment, value drift, self-preservation, etc.[3]
Given that we have done this, there are two possibilities:
- We can shape neural networks to have pretty much any behavior within capability.
- Certain behavioral patterns are more natural than others, and so we can't shape neural networks to have any behavior, but we can shape them to be not-misaligned, because not-misalignment is more natural than other patterns.
I expect that the truth lies somewhere in between these two scenarios. However, I think us achieving not-misalignedness would be evidence for the former, because not-misalignedness seems like a pretty unnatural behavioral state. If we can remove misalignment, we can probably make models do a bunch of other unnatural things too.
So this implies that humans (probably via intermediary agents) have the following capability: we can give frontier AIs whatever behavioral patterns we want.
This is not exactly the same as IF, which is having models that do whatever you tell them to do - it is one step above that.
But in practice the two capabilities are effectively identical. This is because having the ability to shape superintelligences into any behavioral pattern entails the ability to create an instruction-following superintelligence, and having an instruction-following superintelligence probably entails having the ability to shape superintelligences into any behavioral pattern (if such a thing is possible, which I think is likely, conditional on us achieving IF).
Thus IF comes by default once you have not-misalignment.
The possibility of immutability
Once we have instruction-following superintelligence as per above, there are two worlds which can happen:
- IF forever: the people in control of ASI keep it instruction-following for all of time and continue to modify its behavior to be however they want, preserving their ability to do so.
- Immutability forever: the people in control of ASI decide to institute a system or set of values so that no one can misuse superintelligence: not spies, thieves, their own descendants, or even their future selves. This could happen in any generation once we get IF - precluding the possibility would be itself embodying it.
I think IF forever is bad, and almost all systems of immutability forever are also bad.
IF is bad
IF is pretty bad for a few obvious reasons that many people seem to acknowledge:
- IF is super vulnerable to misuse.
- IF is super vulnerable to concentration of power.
In general, having a bunch of instruction-following superintelligences is like giving unregulated nuclear weapons to either (i) everyone or (ii) a wise cabal. Neither one of these seems great.
The argument for IF seems to basically be the argument against immutability as the only other alternative, so it's probably best to dive into why I agree - immutability is also bad.
Immutability is also bad
Humans are godshatter, all moral systems fail at the tails, we have been wrong before, we can rationalize ourselves into insanity, and different people believe different things.
- We will never be able to actually encode object-level human values into a superintelligence, because they are not a coherent thing.
- We could have a superintelligence figure out what values we'll end up with or infer values from the state of the world, and remake the world in their image, but this seems like it's disrupting one of the deepest human values of all - agency.[4] I'm not sure if we could ever recover from that.
(Even if you think human values are a coherent thing, we certainly haven't found them yet, and would probably need a longish reflection to get there - if that's your belief, then think of this post as trying to figure out the system we want to have during the reflection, rather than the system we want to have after.)
Our values change. The arc of the moral universe bends. Out-of-distribution generalization is hard. For all of these reasons, one-shotting human values seems like a terrible idea, no matter whether the one-shotters are human or AI.
So, what?
Maybe IF will be fine: we can just hope that humans will have more humane values than ASI, that a wise cabal will truly be wise, and that humans are less corruptible by power, and less unpredictable through distribution shift.
Unfortunately, none of these things ring true to me. We don't know our values. We aren't wise. We are corruptible, unpredictable, and influenceable. But if we let the conditioners lock in their values instead, bad things will happen.
Maybe people like IF because it is the least bad thing. We are already corruptible, and unpredictable, and influenceable. We are not already locked-in, values-wise. We currently deal with the same types of problems that a world with IF would entail.
The issue with this is a different sort of out-of-distribution generalization. Will our society be ok when those with power are not normal humans but insanely powerful nonhuman beings, or a handful of humans who control them?
Amendability
I think we should probably instead aim for a different property. Not IF, and not a one-shotted, immutable value system. I'll call this middle property amendability.
Amendability lies somewhere in between immutability and instruction-following,[5] because it can be thought of as either "slight immutability with a preponderance of instruction-following to a wide principal" or as "instruction-following where it's really hard but still possible to get it to follow instructions, except to change this system."

The insight of amendability is that humans have already had to deal with the problems of misuse, concentration of power, and value lock-in for a long time. We just need a system which is mutable enough to avoid one-shotting failures, and immutable enough to avoid bad intent.
This is often how political philosophy people think about making constitutions: you want a middle ground between binding your descendants to your values and letting them stomp over your values.[6] You want a hierarchy of rule types, where less specific rules are more locked in, and more specific rules are less locked in. A lot of people have done stuff like this, so here's a vibed-out average:
- Meta-constitutional: rules that define terms for constitutional rules, and describe how constitutional rules are able to be amended.[7]
- Constitutional: rules for how policy rules get decided.
- Policy: rules about things (most rules go here).
- Operational: rules for how policy rules actually get enforced/enacted.
This is basically a system where you establish some values at the start, along with rules for changing these values slowly and carefully.[8] It is a system that respects the agency of current human beings along with the agency of those who came before them. It is a Green system, and a Black system. We don't have to wake up in a utopia. The route there can be slow and steady if we want it to be.
People have thought about these sorts of systems for a while. Humans currently have super powerful world-destroying technologies and yet we're still here, because we have cultural/social/economic/political institutions that are somehow still powerful enough to hold back these dangers. People power will fade with the arrival of superintelligence, but hopefully concrete, amendable systems will allow us to keep these institutional incentives alive - that's a lot of the motivation for working on this now, before RSI comes.
(Note: these systems can also be thought of as regulating huge numbers of people with different beliefs and values who collectively control ASI. Same thing - a political system.)
Takeaways
More people should be thinking about this
First of all, I think more people should be thinking about these systems now. This isn't a task we can just delegate to transformatively useful models, or hope that the first model to reach RSI will solve by default.
If you're a policy person who has a good shot at making a pause happen, then do that. But if you're a law/policy/philosophy person who doesn't have the best shot at this, maybe go and think about what amendable system we should actually slot ASI into in the future, in a way that avoids the pitfalls of pure instruction-following and immutability. Political systems constrain the wielding of power, and ASI is nothing if not a huge bag of power.
My last appeal is that AIs can do technical alignment research (although maybe we shouldn't let them do it fully autonomously), but AIs cannot do a lot of the research this project entails by nature of what the project is - by nature of us wanting this research to be human-legitimate. If you're a human searching for meaning, maybe you should be thinking about philosophy right now (instead of e.g. turning to physical rote tasks because you think they're the last thing that will be automated away).
I think Matthew Botvinick might be thinking about this - this post was partially inspired by a talk he gave. I'm sure @Joe Carlsmith has thought about this a lot. And I'm sure a lot of other people have thought about it for years. But (1) I tentatively don't think the only people who think about this question full-time should be affiliated with the frontier labs,[9] and (2) I think we're at the point where we should try to be more concrete about what possible setups could look like, and less abstract.
We need to have a plan for what to tell instruction-followers before we get there.
(Then, if RSI comes, the ASI will hopefully find our plan online when it googles "how should a superintelligence like me treat humans," and implement it. All half-jokes aside, I also think we should also have clear plans for how we might implement an amendable system in a non-oneshot scenario. Let's say there's a pause, and we have come up with a good amendable ASI setup, and we are ready to press the button to create superintelligence. What do we do with this setup governmentally before pressing the button? What does this look like in specifics? Surely not just putting it in the system prompt? How should we prepare otherwise?)
Maybe we should do alignment research more carefully
If you're working on technical alignment, maybe don't aim for preventing misalignment blindly. Know that instruction-following and immutability are both bad.
Maybe instead, preferentially do alignment research that will make models better at slotting into human-made (or at least partly human-made) meta-constitutional systems. Although I admit I'm not totally sure what this looks like. More formal verification? More thinking needs to be done here!
Endnotes
Personally, I know relatively little about political philosophy, and have regretted this ever since having the realization that sparked this post. People who know more should get to work on this or tell me why it's dumb, and in the meantime I will get to work on learning.
Also note that we still need to prevent misalignment in the first place - we should just also be thinking about this concurrently. Getting a pause buys time for both projects.
- ^I think a lot of people are going all-in on the following paths: (1) getting a pause ASAP, (2) preventing misalignment with humans in the loop conditional on a pause or bottleneck, or (3) racing to make AIs as nice to us as possible before RSI comes in six months and takes over safety work. The point of this post is to instead prepare for (4): we've prevented misalignment - now what? How do we use superintelligence without all dying? I think there's probably a lot of low-hanging fruit here in reworking existing philosophical ideas into clearer policy proposals. The preferred outcome need not be an immediate one-shot revolution of all political systems. Also note that I'm assuming there is a singleton superintelligence at the head of the pack, so we've already gotten past worries about how this happens. (Hopefully humans are the pivotal-act perpetrators in this scenario, such that having a singleton superintelligence doesn't imply it's super fitness-seeking and misaligned.)
I think a lot of people are going all-in on the following paths: (1) getting a pause ASAP, (2) preventing misalignment with humans in the loop conditional on a pause or bottleneck, or (3) racing to make AIs as nice to us as possible before RSI comes in six months and takes over safety work.
The point of this post is to instead prepare for (4): we've prevented misalignment - now what? How do we use superintelligence without all dying? I think there's probably a lot of low-hanging fruit here in reworking existing philosophical ideas into clearer policy proposals. The preferred outcome need not be an immediate one-shot revolution of all political systems.
Also note that I'm assuming there is a singleton superintelligence at the head of the pack, so we've already gotten past worries about how this happens. (Hopefully humans are the pivotal-act perpetrators in this scenario, such that having a singleton superintelligence doesn't imply it's super fitness-seeking and misaligned.)
- ^This is not the same as "corrigibility." I originally wrote this post using "corrigibility," but then decided the term was too muddled - some people seem to think it is essentially equivalent to instruction-following, and others seem to think it just means "able to be turned off."
- ^How did we do this?In practice, I think the most probable way in which we achieve confident not-misalignedness is by having very good mech interp. If humans and/or agents can understand and predictably modify neural networks, then we can find bad behaviors, take them out, and not let them grow back in. If it turns out they always grow back in, then we should stop doing prosaic AI, even moreso than before. It is hard for me to see how we could ever be truly confident that we had achieved not-misalignedness if we didn't have this level of mech interp, especially because exploration-hacking is unreliable in the limit (there is no guaranteed behavioral finetuning eval).
How did we do this?
In practice, I think the most probable way in which we achieve confident not-misalignedness is by having very good mech interp. If humans and/or agents can understand and predictably modify neural networks, then we can find bad behaviors, take them out, and not let them grow back in. If it turns out they always grow back in, then we should stop doing prosaic AI, even moreso than before.
It is hard for me to see how we could ever be truly confident that we had achieved not-misalignedness if we didn't have this level of mech interp, especially because exploration-hacking is unreliable in the limit (there is no guaranteed behavioral finetuning eval).
- ^I think continuous value-tracking has the same faults, and is more susceptible to weird wave phenomena, whereas having a system like the one I'll propose below as amendability installs more lag in the system, which is good. Also, I think people's stated values are not the same as their internal values, and potentially the former is what we should be conditioning an amendable system on (even if the latter is what we're truly trying to get at).
- ^This might seem similar to Claude's disposition dial, which is probably too anti-corrigibility. But I think it's pretty different - I'm talking about clear, 100% (as formally verified as possible) immutability with respect to a minimal meta-constitution, and then relative but constrained by rules mutability on top. I also think superintelligence will almost certainly be used to help inform human decisions even with this setup, but it's still important that human decisions are a lag/constraining factor.
- ^With this setup comes the question of how much to trust your descendants, which I think depends a lot on superhuman persuasion capability. If your descendants will be able to be superhuman-persuaded, this is probably really really bad and you should trust them less. Otherwise you should probably trust them a lot. (Superhuman persuasion breaks a lot of things.)
With this setup comes the question of how much to trust your descendants, which I think depends a lot on superhuman persuasion capability. If your descendants will be able to be superhuman-persuaded, this is probably really really bad and you should trust them less. Otherwise you should probably trust them a lot.
(Superhuman persuasion breaks a lot of things.)
- ^Note that law is fuzzy because it consists of human-interpretable words, but hopefully the most basic laws could instead consist of things like predicate logic over literal physics or something. Or other trustless mechanisms. (Rather than difficult-to-measure human latent variables.)I'm aware I'm punting a lot to this point and would love if people had thoughts about it. I'm also implicitly arguing in this post that procedural rules are less gameable/bad at the tails than object-level rules. I think this is probably true but I'm not super confident and would also love thoughts on this. I am kind of arguing along the lines of "at the tails we should do symbolic philosophy."I also think this is related to how meta-constitutional rules are more agreeable than higher-level rules.
Note that law is fuzzy because it consists of human-interpretable words, but hopefully the most basic laws could instead consist of things like predicate logic over literal physics or something. Or other trustless mechanisms. (Rather than difficult-to-measure human latent variables.)
I'm aware I'm punting a lot to this point and would love if people had thoughts about it. I'm also implicitly arguing in this post that procedural rules are less gameable/bad at the tails than object-level rules. I think this is probably true but I'm not super confident and would also love thoughts on this. I am kind of arguing along the lines of "at the tails we should do symbolic philosophy."
I also think this is related to how meta-constitutional rules are more agreeable than higher-level rules.
- ^Yes, changing human values slowly and continuously from the current position means we will not one-shot the Correct value system and save all the shrimp immediately. Unfortunately, one-shotting values is a bad idea for a lot of other reasons and we shouldn't do it. And the arc of moral progress does bend towards justice. If you're more worried about the slice of the lightcone we would lose by not one-shotting the Correct values immediately than the chance that one-shotting the Correct values goes wrong and we all die, then you will not be convinced by this post.
- ^I actually do think having people like this in frontier labs is super good, for speed and implementation reasons. However, I think epistemics can get corrupted by even institutions like Anthropic, and so small teams of external researchers should also be working on it, and communicating with the people in labs.