Written by Celene Nightingale, with assistance from AprilSR.
The Von Neumann-Morgenstern axioms are completeness, transitivity, continuity, and independence. From these, we derive EU-maximizing behavior and the consequences thereof.
VNM axioms are typically justified by money-pump arguments showing that agents in violation of them either have their money consistently taken away from them, or fail to exploit opportunities to consistently gain more money. Nevertheless, some of these arguments are weaker than others, and many choose to reject one or more of the axioms. Tetraspace rejects continuity,[1] Garrabrant rejects independence, and Thornley rejects completeness.
John Wentworth argues in favor of completeness in Why not subagents?, as well as in various comment sections. Here, I present a simplified framing which is (hopefully) easier to understand.
What is completeness?
Completeness states that, given two options X and Y, you must either prefer X to Y, prefer Y to X, or be indifferent between X and Y. At first, it might seem difficult to imagine an agent which behaves any other way. However, in There are no coherence theorems, Elliott Thornley argues:
Here’s the important point missing from this thought: there are two ways of failing to have a strict preference between X and Y. Being indifferent between X and Y is one way: preferring X at least as much as Y and preferring Y at least as much as X. Having a preferential gap between X and Y is another way: not preferring X at least as much as Y and not preferring Y at least as much as X. If an agent has a preferential gap between any two lotteries, then their preferences violate Completeness.
One might say that the agent is not indifferent between X and Y, but finds them incommensurable. The difference between indifference and incommensurability is that indifference is sensitive to sweetening/souring, whereas incommensurability is not. For example, let's say Alice is having trouble picking between two job offers, one which offers $120,000 a year working on an oil rig, and one which offers $60,000 a year being an artist. On one hand, the oil rig pays more; on the other hand, being an artist is more personally meaningful.
If Alice were indifferent between these two options, it implies that if we sweeten the oil rig, and make it pay $125,000 a year instead, she would take the job, but if we changed it to $115,000 a year, souring the offer, she would become an artist. But if money and meaning are incommensurable, then she would remain undecided even after sweetening/souring.[2][3]
Single Souring Money-Pump
The standard money-pump argument for completeness is the single souring money-pump.
Consider Gail the Gappy Agent. Gail prefers more apples to fewer apples, and prefers more oranges to fewer oranges, but considers apples and oranges incommensurable, and has no set exchange rate between them.
Gail has 5 apples right now, and is offered a trade: she can get 4 oranges in exchange for 5 apples. Since apples and oranges are incommensurable, she has no preference between them, and might choose to accept this trade. Let's suppose she does.
Now that she has 4 oranges, she's offered another trade: 4 apples for 4 oranges. Once again, since apples and oranges are incommensurable, she may choose to accept this trade. If Gail accepts both trades, she ends up with 1 fewer apple than she started with through a series of fully permissible trades!
Now, unlike other money-pumps, this is not a forcing argument, because we've only shown that she can be money-pumped, but not that she has to be. Her incomplete preference ordering is insufficient to rule out the money pump's trades, but it also doesn't mandate them: she can just as easily not accept these trades. In fact, there's an easy history-sensitive policy she can follow to avoid being money-pumped: don't accept trades which leave you strictly worse off than you were at any earlier point.
She's already gone from 5 apples to 4 oranges! So she refuses to go to 4 apples. But 1 apple and 1 orange? Sure, why not, she'll take that trade. And she won't accept 3 oranges, since she already had 4 oranges, but she'll accept 3 oranges and 1 apple...
The result of this, after enough trades, is not an agent which converges towards completeness. The result is an ever expanding Pareto frontier, where Gail might happily jump between points on the frontier, or to points beyond it, but never to points inside it.
Ex ante dominance
Gail the Gappy Agent is feeling pretty happy with herself. She's managed to meaningfully keep her incomplete preferences and avoid comparing apples and oranges, while dodging being money-pumped.
Suddenly, Omega descends upon her, offering the opportunity to play a game. Omega will flip a coin, and if it lands heads, she can choose between +4 apples and +8 oranges. If it lands tails, she can choose between +4 oranges and +8 apples.
So how will she make her choice? Let's say that, since she finds apples and oranges incommensurable, she flips another coin to choose between her prizes. This strategy has an EV of +3 apples, +3 oranges.
But wait! What if she always took the +8 option? In that case, her EV is +4/+4. If Gail has the option to, she should prefer modifying herself into a complete agent which simply maximizes total fruit.[4]
Indeed, any strategy with a chance of taking +4 on both heads and tails can be dominated by another strategy.
(Interactive widget made by Claude Fable 5.2)
Does this necessarily create complete behavior? No. Depending on their exchange rate between apples and oranges, a complete agent will pick one of three strategies: Always Apples (if 1 apple >= 2 oranges), Always Oranges (if 1 apple <= 0.5 oranges), or Always +8 (if 0.5 oranges <= 1 apple <= 2 oranges).
Gail, however, has other options. She can pick any undominated strategy on the top and right edges, not just these three. She could always take +8 oranges on heads and flip a coin on tails, and no strategy would be an improvement on both apples and oranges. While a complete agent with an exchange rate on the boundary between two strategies might also do this, this is not guaranteed.
But then Gail looks to the future. When Gail looks ahead, she sees... Omega running the game again. If she picked Always Apples the first time, she can't pick Always Oranges the next, because that would be dominated by picking Always +8 both times. By making her choice, she binds her hands. And worse yet, she doesn't just see Omega running this game, but diverse and varied games, with different tradeoffs and different mechanics.
So then, looking into the future, she sees the range of undominated strategies shrinking and shrinking, and realizes that in order to avoid being ex ante dominated, she must become complete.[5] Any undominated strategy carries with it an implicit exchange rate between apples and oranges. Gail rewrites her code to use that exchange rate, and that is the end of Gail the Gappy Agent.[6]
In Why Not Subagents?, Wentworth and Lorell make a similar argument. In their framing, Gail can be viewed as two separate agents, Gala and Gardner, who like apples and oranges respectively. Gala and Gardner are in an uneasy truce together, but can benefit in expectation from making a values handshake and becoming a single agent with an exchange rate between apples and oranges.
- ^I agree with her that continuity is the weakest VNM axiom.
- ^Lest you think this makes rejecting completeness obvious: in the real world, Alice would not be certain what her life would be like in each option, and that uncertainty would probably be larger than $5,000 a year, so it's worth gathering more information and considering the details either way. In this toy example, though, we assume that Alice is omniscient and knows everything.
- ^Also, incommensurability is not the only type of preference gap. You could say that Alice doesn't have a preference when the oil rig has a salary between $120,000 and $130,000, but she prefers the oil rig above $130,000 and prefers art below $120,000.
- ^This particular result holds even if Gail is risk-averse, but in general we are assuming that Gail values apples linearly and oranges linearly (or that she already has enough apples and oranges that at this scale, additional fruit is valued approximately linearly).
- ^Or must act like a complete agent would in the limit, but IMO it is more natural to rewrite yourself to use an exchange rate than to carefully keep track of an ever expanding web of commitments.
- ^There are some scenarios in which Gail might benefit from self modifying by selecting an exchange rate out of a distribution of possible exchange rates, which Wentworth argues for in Why Not Subagents?, but it's mostly nonapplicable here.