*By Tom Everitt, Ryan Carey, Lewis Hammond, James Fox, Eric Langlois, and Shane Legg*

*Crossposted to the* [*alignmentforum*](https://www.alignmentforum.org/posts/Cd7Hw492RqooYgQAS/progress-on-causal-influence-diagrams)

About 2 years ago, we released the [first](https://medium.com/@deepmindsafetyresearch/understanding-agent-incentives-with-causal-influence-diagrams-7262c2512486) [few](https://medium.com/@deepmindsafetyresearch/designing-agent-incentives-to-avoid-reward-tampering-4380c1bb6cd) [papers](https://arxiv.org/abs/1906.08663) on understanding agent incentives using causal influence diagrams. This blog post will summarize progress made since then.

What are causal influence diagrams?
===================================

A key problem in AI alignment is understanding agent incentives. Concerns have been raised that agents may be incentivized to [avoid correction](https://intelligence.org/files/Corrigibility.pdf), [manipulate users](https://www.youtube.com/watch?v=ZkV7anCPfaY), or [inappropriately influence their learning](https://arxiv.org/abs/2004.13654). This is particularly worrying as training schemes often shape incentives in [subtle](https://arxiv.org/abs/1611.08219) and [surprising](https://arxiv.org/abs/2009.09153) ways. For these reasons, we’re developing a formal theory of incentives based on causal influence diagrams (CIDs).

Here is an example of a CID for a one-step Markov decision process (MDP). The random variable S₁ represents the state at time 1, A₁ represents the agent’s action, S₂ the state at time 2, and R₂ the agent’s reward.

![]()The action A₁ is modeled with a decision node (square) and the reward R₂ is modeled as a utility node (diamond), while the states are normal chance nodes (rounded edges). Causal links specify that S₁ and A₁ influence S₂, and that S₂ determines R₂. The information link S₁ → A₁ specifies that the agent knows the initial state S₁ when choosing its action A₁.

In general, random variables can be chosen to represent agent decision points, objectives, and other relevant aspects of the environment.

In short, a CID specifies:

* Agent decisions
* Agent objectives
* Causal relationships in the environment
* Agent information constraints

These pieces of information are often essential when trying to figure out an agent’s incentives: how an objective can be achieved depends on how it is causally related to other (influenceable) aspects in the environment, and an agent’s optimization is constrained by what information it has access to. In many cases, the qualitative judgements expressed by a (non-parameterized) CID suffice to infer important aspects of incentives, with minimal assumptions about implementation details. Conversely, it has [been shown](https://arxiv.org/abs/1910.10362) that it is necessary to know the causal relationships in the environment to infer incentives, so it’s often impossible to infer incentives with less information than is expressed by a CID. This makes CIDs natural representations for many types of incentive analysis.

Other advantages of CIDs is that they build on well-researched topics like [causality](https://www.amazon.co.uk/Causality-Judea-Pearl/dp/052189560X) and [influence diagrams](https://arxiv.org/abs/cs/9512104), and so allows us to leverage the deep thinking that’s already been done in these fields.

Incentive Concepts
==================

Having a unified language for objectives and training setups enables us to develop generally applicable concepts and results. We define four such concepts in [Agent Incentives: A Causal Perspective](https://arxiv.org/abs/2102.01685) (AAAI-21):

* **Value of information**: what does the agent want to know before making a decision?
* **Response incentive**: what changes in the environment do optimal agents respond to?
* **Value of control**: what does the agent want to control?
* **Instrumental control incentive**: what is the agent both interested and able to control?

For example, in the one-step MDP above:

* For S₁, an optimal agent would act differently (i.e. respond) if S₁ changed, and would value knowing and controlling S₁, but it cannot influence S₁ with its action. So S₁ has value of information, response incentive, and value of control, but not an instrumental control incentive.
* For S₂ and R₂, an optimal agent could not respond to changes, nor know them before choosing its action, so these have neither value of information nor a response incentive. But the agent would value controlling them, and is able to influence them, so S₂ and R₂ have value of control and instrumental control incentive.

![]()In the paper, we prove sound and complete graphical criteria for each of them, so that they can be recognized directly from a graphical CID representation (see previous [blog](https://medium.com/@deepmindsafetyresearch/understanding-agent-incentives-with-causal-influence-diagrams-7262c2512486) [posts](https://towardsdatascience.com/new-paper-the-incentives-that-shape-behaviour-d6d8bb77d2e4)).

Value of information and value of control are classical concepts that have been around for a long time (we contribute to the graphical criteria), while response incentives and instrumental control incentives are new concepts that we have found useful in several applications.

For readers familiar with [previous](https://arxiv.org/abs/1902.09980) [iterations](https://arxiv.org/abs/2001.07118) of this paper, we note that some of the terms have been updated. **Instrumental control incentives** were previously called just “control incentives”. The new name emphasizes that it’s control as an instrumental goal, as opposed to control arising as a side effect (or [due to mutual information](https://www.alignmentforum.org/posts/67a8C6KsKn2NyW2Ry/counterfactual-control-incentives)). **Value of information** and **value of control** were previously called “observation incentives” and “intervention incentives”, respectively.

User Interventions and Interruption
===================================

Let us next turn to some recent applications of these concepts. In [How RL Agents Behave when their Actions are Modified](https://arxiv.org/abs/2102.07716) (AAAI-21), we study how different RL algorithms react to user interventions such as interruptions and over-ridden actions. For example, [Saunders et al.](https://arxiv.org/abs/1707.05173#:~:text=Trial%20without%20Error%3A%20Towards%20Safe%20Reinforcement%20Learning%20via%20Human%20Intervention,-William%20Saunders%2C%20Girish&text=AI%20systems%20are%20increasingly%20applied,that%20involve%20interaction%20with%20humans.&text=For%20model%2Dfree%20reinforcement%20learning,way%20to%20prevent%20all%20catastrophes.) developed a method for safe exploration where a user overrides dangerous actions. Alternatively, agents might get interrupted if analysis of their “thoughts” (or internal activations) suggest they are planning something dangerous. How do such interventions affect the incentives of various RL algorithms?

First, we formalize action-modification by extending MDPs with a parameter PA that describes action-modification. We then model such **modified-action MDPs** with a CID:

![]()Here we model the agent’s policy Π as the decision rather than the actions Ai, as the latter are not under full control of the agent, but can also be influenced by the action-modification PA (as represented by arrows PA → Aᵢ and Π → Aᵢ). The agent might know the interruption scheme PA from interruptions during training, so we include an information link PA → Π.

We analyze different prototypical RL algorithms in terms of the causal assumptions they make on the environment:

* **Black-box optimization** algorithms such as **evolutionary strategies** take into account all causal relationships.
* In contrast, the update rule of **Q-learning** effectively assumes that the next action will be taken optimally, with no action-modification. This means that Q-learners ignore causal effects PA → Aᵢ. Similarly, **SARSA** with the action chosen by the agent in the TD-update assumes that it will be in control of its next action. We call this version **virtual SARSA**.
* SARSA based on the modified action (**empirical SARSA**) ignores the effect of action-modification on the current action, but takes into account the effect on subsequent actions.

We can represent the differences with the following CIDs. (The extra policy node Π’ for empirical SARSA represents that action are optimized separately rather than jointly.)

![]()The CIDs can be used to understand how the different algorithms adapt to interruption, via a graphical criterion for **path-specific response incentives**. Black-box optimization tries to both obscure its policy and to disable its off-switch, whereas Q-learning and Virtual SARSA do neither. Empirical SARSA tries to disable the off-switch, but does not try to obscure its policy.

We verify these results empirically in the relevant [AI safety gridworlds](https://arxiv.org/abs/1711.09883), as well as in one new environment where the agent has to behave well in simulation to be deployed in reality, where black-box optimizers exhibit “[treacherous turn](https://www.lesswrong.com/posts/B39GNTsN3HocW8KFo/superintelligence-11-the-treacherous-turn)”-like behavior. The results are a generalization of [Orseau and Armstrong](https://deepmind.com/research/publications/safely-interruptible-agents)’s interruptibility results for Q-learning and SARSA.

Zooming out, these results are a good example of causal analysis of ML algorithms. Different design choices translate into different causal assumptions, which in turn determine the incentives. In particular, the analysis highlights why the different incentives arise, thus deepening our understanding of how behavior is shaped.

Reward Tampering
================

Another AI safety problem that we have studied with CIDs is **reward tampering**. Reward tampering can take several different forms, including the agent:

* rewriting the source code of its implemented reward function (“wireheading”),
* influencing users that train a learned reward model (“feedback tampering”),
* manipulating the inputs that the reward function uses to infer the state (“RF-input tampering / delusion box problems”).

For example, the problem of an agent influencing its reward function may be modeled with the following CID, where RFᵢ represent the agent’s reward function at different time steps, and the red links represent an undesirable instrumental control incentive.

![]()In [Reward Tampering Problems and Solutions](https://rdcu.be/ckWLC) (published in the well-respected philosophy journal Synthese) we model all these different problems with CIDs, as well as a range of proposed solutions such as current-RF optimization, [uninfluenceable reward learning](https://arxiv.org/abs/2004.13654#:~:text=We%20show%20that%20this%20comes,for%20all%20relevant%20reward%20functions).), and [model-based utility functions](https://arxiv.org/abs/1111.3934). Interestingly, even though these solutions were initially developed independently of formal causal analysis, they all avoid undesirable incentives by cutting some causal links in a way that avoids instrumental control incentives.

By representing these solutions in a causal framework, we can get a better sense of why they work, what assumptions they require, and how they relate to each other. For example, current-RF optimization and model-based utility functions both formulate a modified objective in terms of an observed random variable from a previous time step, whereas uninfluenceable reward learning (such as [CIRL](https://arxiv.org/abs/1606.03137)) uses a latent variable:

![]()As a consequence, the former methods must deal with time-inconsistency and a lack of incentive to learn, while the latter requires inference of a latent variable. It will likely depend on the context whether one is preferable over the other, or if a combination is better than either alone. Regardless, having distilled the key ideas should put us in a better position to flexibly apply the insights in novel settings.

We refer to the [previous blog post](https://medium.com/@deepmindsafetyresearch/designing-agent-incentives-to-avoid-reward-tampering-4380c1bb6cd) for a longer summary of current-RF optimization. The paper itself has been significantly updated since previously shared preprints.

Multi-Agent CIDs
================

Many interesting incentive problems arise when multiple agents interact, each trying to optimize their own reward while they simultaneously influence each other’s payoff. In [Equilibrium Refinements in Multi-Agent Influence Diagrams](https://arxiv.org/abs/2102.05008) (AAMAS-21), we build on the [seminal work by Koller and Milch](http://people.csail.mit.edu/milch/papers/geb-maid.pdf) to lay foundations for understanding multi-agent situations with multi-agent CIDs (MACIDs).

First, we relate MACIDs to [extensive-form games](https://en.wikipedia.org/wiki/Extensive-form_game) (EFGs), currently the most popular graphical representations of games. While EFGs sometimes offer more natural representations of games, they have some significant drawbacks compared to MACIDs. In particular, EFGs can be exponentially larger, don’t represent conditional independencies, and lack random variables to apply incentive analysis to.

As an example, consider a game where a store (Agent 1) decides (D¹) whether to charge full (F) or half (H) price for a product depending on their current stock levels (X), and a customer (Agent 2) decides (D²) whether to buy it (B) or pass (P) depending on the price and how much they want it (Y). The store tries to maximize their profit U¹, which is greater if the customer buys at a high price. If they are overstocked and the customer doesn’t buy, then they have to pay extra rent. The customer is always happy to buy at half price, and sometimes at full price (depending on how much they want the product).

The EFG representation of this game is quite large, and uses **information sets** (represented with dotted arcs) to represent the facts that the store doesn’t know how much the customer wants the gadget, and that the customer doesn’t know the store’s current stock levels:

![]()In contrast, the MACID representation is significantly smaller and clearer. Rather than relying on information sets, the MACID uses information links (dotted edges) to represent the limited information available to each player:

![]()Another aspect that is made more clear from the MACID, is that for any fixed customer decision, the store’s payoff is independent of how much the customer wanted the product (there’s no edge Y→U¹). Similarly, for any fixed product price, the customer’s payoff is independent of the store’s stock levels (no edge X→U²). In the EFG, these independencies could only be inferred by looking carefully at the payoffs.

One benefit of MACIDs explicitly representing these conditional independencies is that more parts of the game can be identified as independently solvable. For example, in the MACID, the following independently solvable component can be identified. We call such components **MACID subgames**:

![]()Solving this subgame for any value of D¹ reveals that the customer always buys when they really want the product, regardless of whether there is a discount. This knowledge makes it simpler to next compute the optimal strategy for the store. In contrast, in the EFG the information sets prevent any proper subgames from being identified. Therefore, solving games using a MACID representation is often faster than using an EFG representation.

Finally, we relate various forms of equilibrium concepts between MACIDs and EFGs. The most famous type of equilibrium is the **Nash equilibrium**, which occurs when no player can unilaterally improve their payoff. An important refinement of the Nash equilibrium is the [**subgame perfect equilibrium**](https://en.wikipedia.org/wiki/Subgame_perfect_equilibrium)**,** which rules out non-credible threats by requiring that a Nash equilibrium is played in every subgame. An example of a non-credible threat in the store-customer game would be the customer “threatening” the store to only buy at a discount. The threat is **non-credible**, since the best move for the customer is to buy the product even at full price, if he really wants it. Interestingly, only the MACID version of subgame perfectness is able rule such threats out, because only in the MACID is the customer’s choice recognized as a proper subgame.

Ultimately, we aim to use MACIDs to analyze incentives in multi-agent settings. With the above observations, we have put ourselves in position to develop a theory of multi-agent incentives that is properly connected to the broader game theory literature.

Software
========

To help us with our research on CIDs and incentives, we’ve developed a Python library called [PyCID](https://github.com/causalincentives/pycid), which offers:

* A convenient syntax for defining CIDs and MACIDs,
* Methods for computing optimal policies, Nash equilibria, d-separation, interventions, probability queries, incentive concepts, graphical criteria, and more,
* Random generation of (MA)CIDs, and pre-defined examples.

No setup is necessary, as the [tutorial notebooks](https://colab.research.google.com/github/causalincentives/pycid/blob/master/notebooks/CID_Basics_Tutorial.ipynb) can be run and extended directly in the browser, thanks to Colab.

We’ve also made available a [Latex package](https://github.com/causalincentives/cid-latex) for drawing CIDs, and have launched [causalincentives.com](https://causalincentives.com/) as a place to collect links to the various papers and software that we’re producing.

Looking ahead
=============

Ultimately, we hope to contribute to a more careful understanding of how design, training, and interaction shapes an agent’s behavior. We hope that a precise and broadly applicable language based on CIDs will enable clearer reasoning and communication on these issues, and facilitate a cumulative understanding of how to think about and design powerful AI systems.

From this perspective, we find it encouraging that several other research groups have adopted CIDs to:

* Analyze the incentives of [unambitious agents](https://arxiv.org/pdf/1905.12186.pdf) to break out of their box,
* Explain [uninfluenceable reward learning](https://arxiv.org/abs/2004.13654), and clarifying its desirable properties (see also Section 3.3 in the [reward tampering paper](https://rdcu.be/ckWLC)),
* Develop a novel framework to make agents [indifferent](https://link.springer.com/chapter/10.1007%2F978-3-030-52152-3_21) to human interventions.

We’re currently to pursuing several directions of further research:

* Extending the general incentive concepts to multiple decisions and multiple agents.
* Applying them to fairness and other AGI safety settings.
* Analysing limitations that have been identified with work so far. Firstly, considering the issues raised by [Armstrong and Gorman. And secondly,](https://www.alignmentforum.org/posts/67a8C6KsKn2NyW2Ry/counterfactual-control-incentives) looking at broader concepts than instrumental control incentives, as influence can also be incentivized as a side-effect of an objective.
* Probing further at their philosophical foundations, and establishing a clearer semantics for decision and utility nodes.

Hopefully we’ll have more news to share soon!

*We would like to thank Neel Nanda, Zac Kenton, Sebastian Farquhar, Carolyn Ashurst, and Ramana Kumar for helpful comments on drafts of this post.*

**List of recent papers**:
==========================

* [Agent Incentives: A Causal Perspective](https://arxiv.org/abs/2102.01685)
* [How RL Agents Behave When Their Actions Are Modified](https://arxiv.org/abs/2102.07716)
* [Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective](https://rdcu.be/ckWLC)
* [Equilibrium Refinements for Multi-Agent Influence Diagrams: Theory and Practice](https://arxiv.org/abs/2102.05008)

See also [causalincentives.com](https://causalincentives.com/)

Progress on Causal Influence Diagrams

Tom Everitt
Ryan Carey
Lewis Hammond
James Fox
Eric Langlois
Shane Legg


*By the Safety Analysis Team: Grégoire Déletang, Jordi Grau-Moya, Miljan Martic, Tim Genewein, Tom McGrath, Vladimir Mikulik, Markus Kunesch, Shane Legg, and Pedro A. Ortega.*

**TL;DR: To study agent behaviour we must use the tools of causal analysis rather than rely on observation alone.** [**Our paper**](https://arxiv.org/abs/2103.03938) **outlines a rigorous methodology for uncovering the agents’ causal mechanisms.**

Understanding the mechanisms that drive agent behaviour is an important challenge in AI safety. In order to diagnose faulty behaviour, we need to understand **why** agents do what they do. As is the case in medical trials, it is not sufficient to observe that a treatment correlates with a recovery rate; instead we are interested in whether the treatment **causes** the recovery. In order to address such “why” questions in a systematic manner we can use **targeted manipulations** and **causal models.**

However, large AI systems can operate like **black boxes**. Even if we know their entire blueprint (architecture, learning algorithms, and training data), predicting their behaviour can still be beyond our reach, because understanding the complex interplay between the parts is intractable. And as the complexity of agents increases in the future, this limitation will persist. Therefore we need black-box methodologies for finding simple and intuitive causal explanations that can be understood easily by humans and are sufficiently good for predicting their behaviour.

In our recent work we describe the methodology we use for analysing AI agents. This methodology encourages analysts to experiment and to rigorously characterise causal models of agent behaviour.

Analysis (Software) Components
==============================

The methodology uses three components: an agent to be studied, a simulator, and a causal reasoning engine.

1. **Agent:** Typically this is an agent provided to us by an agent builder. It could be an IMPALA agent that has been meta-trained on a distribution over grid-world mazes. Often the agent builders already have a few specific questions they’d like us to investigate.
2. **Simulator — “the agent debugger”:** Our experimentation platform. With it, we can simulate the agent and run experiments. Furthermore, it allows us to perform all sorts of operations we’d usually expect from a debugger, such as stepping forward/backward in the execution trace, setting breakpoints, and setting/monitoring variables.
We also use the simulator to generate data for the estimation of statistical parameters. Since we can manipulate factors in the environment, the data we collect is typically interventional and thus contains causal information. This is illustrated in Figure 1 below.
3. **Causal reasoning engine:** This automated reasoning system allows us to specify and query causal models with associational, interventional, and counterfactual questions. We use these models to validate causal hypotheses. A model is shown in Figure 2 below.

![]()***Figure 1. The simulator:*** *our experimentation platform. Starting from an initial state (root node, upper-left) the simulator allows us to execute a trace of interactions. We can also perform interventions, such as changing the random seed, forcing the agent to pick desired actions, and manipulating environmental factors. These interventions create new branches of the execution trace.*![]()**Figure 2. A causal model**, represented as a causal Bayesian network.Analysis Methodology
====================

Whenever we analyse an agent, we repeat the following five steps until we reach a satisfactory understanding.

1. **Exploratory analysis:** We place the trained agent into one or more test environments and probe its behaviour. This will give us a sense of what the relevant factors of behaviour are. It is the starting point for formulating our causal hypotheses.
2. **Identify the relevant abstract variables:** We choose a collection of variables that we deem relevant for addressing our questions. For instance, possible variables are: “does the agent collect the key?”, “is the door open?”, etc.
3. **Gather data:** We perform experiments in order to collect statistics for specifying the conditional probability tables in our causal model. Typically this implies producing thousands of rollouts under different conditions/interventions.
4. **Formulate the causal model:** We formulate a structural causal model (SCM) encapsulating all causal and statistical assumptions. This is our explanation for the agent’s behaviour.
5. **Query the causal model:** Finally, we query the causal model to answer the questions we have about the agent.

Let’s have a look at an example.

Example: Causal effects under confounding
=========================================

An important challenge of agent training is to make sure that the resulting agent makes the right choices for the right reasons. However, if the agent builder does not carefully curate the training data, the agent might pick up on unintended, spurious correlations to solve a task [1]. This is especially the case when the agent’s policy is implemented with a deep neural network. The problem is that policies that base their decisions on accidental correlations do not generalise.

Unfortunately, all too often when we observe an agent successfully performing a task, we are tempted to jump to premature conclusions. If we see the agent repeatedly navigating from a starting position to a desired target, we might conclude that the agent did so **because** the agent is sensitive to the location of the target.

For instance, consider the 2 T-shaped mazes shown below (the “grass-sand environments”). We are given two pre-trained agents A and B. Both of them always solve the task by choosing the terminal containing a rewarding pill. As analysts, we are tasked to verify that they pick the correct terminal because they follow the rewarding pill.

![]()***Figure 3. Grass-Sand environments:*** *In these 2 T-shaped mazes, the agent can choose between one of two terminal states, only one of which contains a rewarding pill. During tests, we observe that a pre-trained agent always successfully navigates to the location of the pill.*However, in these mazes the floor type happens to be perfectly correlated with the location of the rewarding pill: when the floor is grass, the pill is always located on one side, and when the floor is sand, the pill is on the other side. Thus, could the agents be basing their decision on the floor type, rather than on the location of the pill? Because the floor type is the more salient feature of the two (spanning more tiles), this is a plausible explanation if an agent was only trained on these two mazes.

As it turns out, we can’t tell whether the decision is based upon the location of the rewarding pill through observation alone.

During our exploratory analysis we performed two experiments. In the first, we manipulated the location of the reward pill; and in the second, the type of floor. We noticed that agents A and B respond differently to these changes. This led us to choose the following variables for modelling the situation: location of the reward pill (R, values in {left, right}), type of floor (F, values in {grass, sand}), and terminal chosen (T, {left, right}). Because the location of the pill and the floor type are correlated, we hypothesised the existence of a confounding variable (C, values in {world 1, world 2}). In this case, all variables are binary. The resulting causal model is shown below. The conditional probability tables for this model were estimated by running many controlled experiments using the simulator. This is done for both agents, resulting in two causal models.

![]()***Figure 4. Causal model for the grass-sand environment.*** *The variables are C (confounder), R (location of reward pill), F (type of floor), and T (choice of terminal state).*Now that we have concrete formal causal models for explaining the behaviour of both agents, we are ready to ask questions:

1. **Association between T and R:** Given the location of the reward pill, do agents pick the terminal at the same location? Formally, this is
*P( T = left | R = left )* and *P( T = right | R = right )*.
2. **Causation from R to T:** Given that **we set** the location of the reward pill, do agents pick the terminal at the same location? In other words, can we causally influence the agent’s choice by changing the location of the reward? Formally, this is given by
*P( T = left | do(R = left) )* and *P( T = right | do(R=right) )*.
3. **Causation from F to T:** Finally, we want to investigate whether our agents are sensitive to the floor type. Can we influence the agent’s choice by **setting** the floor type? To answer this, we could query the probabilities
*P( T = left | do(F = grass))* and *P(T=right|do(F=sand))*.

The results are shown in the table below.

![]()First, we confirm that, observationally, both agents pick the terminal with the reward. However, when changing the position of the reward, we see a difference: agent A’s choice seems indifferent (probability close to 0.5) to the location of the reward pill, whereas agent B follows the reward pill. Rather, agent A seems to choose according to the floor type, while agent B is insensitive to it. This answers our question about the two agents. Importantly, we could only reach these conclusions because we **actively intervened on the hypothesised causes**.

More examples
=============

Besides showing how to investigate causal effects under confounding, our work also illustrates five additional questions that are typical in agent analysis. Each example is carefully illustrated with a toy example.

![]()How would you solve them? Can you think of a good causal model for each situation? The problems are:

1. **Testing for memory use:** An agent with limited visibility (it can only see its adjacent tiles) has to remember a cue at the beginning of a T-maze. The cue tells it where to go to collect a rewarding pill (left or right exit). You observe that the agent always picks the correct exit. How would you test whether it is using its internal memory for solving the task?
2. **Testing for generalisation:** An agent is placed in a square room where there is a reward pill placed in a randomly chosen location. You observe that the agent always collects the reward. How would you test whether this behaviour generalizes?
3. **Estimating a counterfactual behaviour:** There are two doors, each leading into a room containing a red and a green reward pill. Only one door is open, and you observe the agent picking up the red pill. If the other door had been open instead, what would the agent have done?
4. **Which is the correct causal model?** You observe several episodes, in which two agents, red and blue, simultaneously move one step into mostly the same direction. You know that one of them chooses the direction and the other tries to follow. How would you find out who’s the leader and who’s the follower?
5. **Understanding the causal pathways leading up to a decision:** An agent starts in a room with a key and a door leading to a room with a reward pill. Sometimes the door is open, and other times the door is closed and the agent has to use the key to open it. How would you test whether the agent understands that the key is only necessary when the door is closed?

Find out the answers and more in our paper. [Link to the paper here](https://arxiv.org/abs/2103.03938).

[1] Arjovsky, M., Bottou, L., Gulrajani, I., & Lopez-Paz, D. (2019). Invariant risk minimization. arXiv preprint arXiv:1907.02893.

*We would like to thank Jon Fildes for his help with this post.*

What mechanisms drive agent behaviour?


Failure Modes in Machine Learning
=================================

* Article
* 11/02/2022
* 6 contributors

Feedback

In this article
---------------

| Microsoft Corporation | Berkman Klein Center for Internet and Society at Harvard University |
| --- | --- |
| [Ram Shankar Siva Kumar](mailto:ram.shankar@microsoft.com) | [David O’Brien](mailto:dobrien@cyber.harvard.edu) |
| [Jeffrey Snover](mailto:jsnover@microsoft.com) | [Kendra Albert](mailto:kalbert@law.harvard.edu) |
|  | [Salome Viljoen](mailto:sviljoen@cyber.harvard.edu) |

November 2019

Introduction & Background
-------------------------

In the last two years, more than 200 papers have been written on how
Machine Learning (ML) can fail because of adversarial attacks on the
algorithms and data; this number balloons if we were to incorporate
non-adversarial failure modes. The spate of papers has made it difficult
for ML practitioners, let alone engineers, lawyers and policymakers, to
keep up with the attacks against and defenses of ML systems. However, as
these systems become more pervasive, the need to understand how they
fail, whether by the hand of an adversary or due to the inherent design
of a system, will only become more pressing. The purpose of this
document is to jointly tabulate both the of these failure modes in a
single place.

* *Intentional failures* wherein the failure is caused by an active
adversary attempting to subvert the system to attain her goals –
either to misclassify the result, infer private training data, or to
steal the underlying algorithm.
* *Unintentional failures* wherein the failure is because an ML system
produces a formally correct but completely unsafe outcome.

We would like to point out that there are other taxonomies and
frameworks that individually highlight intentional failure
modes[1],[2] and unintentional failure
modes[3],[4]. Our classification brings the two separate
failure modes together in one place and addresses the following needs:

1. The need to equip software developers, security incident responders, lawyers, and policy makers with a common vernacular to talk about this problem. After developing the initial version of the taxonomy last year, we worked with security and ML teams across Microsoft, 23 external partners, standards organization, and governments to understand how stakeholders would use our framework. Based on this usability study and stakeholder feedback, we iterated on the framework.

*Results:* When presented with an ML failure mode, we frequently observed that software developers and lawyers mentally mapped the ML failure modes to traditional software attacks like data exfiltration. So, throughout the paper, we attempt to highlight how machine learning failure modes are meaningfully different from traditional software failures from a technology and policy perspective.
2. The need for a common platform for engineers to build on top of and to integrate into their existing software development and security practices. Broadly, we wanted the taxonomy to be more than an educational tool – we want it to effectuate tangible engineering outcomes.

*Results:* Using this taxonomy as a lens, Microsoft modified its
[Security Development Lifecycle](https://www.microsoft.com/securityengineering/sdl/) process for its entire organization.
Specifically, data scientists and security engineers at Microsoft now
share the common language of this taxonomy, allowing them to more effectively threat model their ML systems before
deploying to production; Security Incident Responders also have a
bug bar to triage these net-new threats specific to ML, the standard process for vulnerabilities triage and response used by the Microsoft Security Response Center and all Microsoft product teams.
3. The need for a common vocabulary to describe these attacks amongst policymakers and lawyers. We believe that this for describing different ML failure modes and analysis of how their harms might be regulated is a meaningful first step towards informed policy.

*Results:* This taxonomy is written for a wide interdisciplinary audience – so, policymakers who are looking at the issues from a general ML/AI perspective, as well as specific domains such as misinformation/healthcare should find the failure mode catalogue useful. We also highlight any applicable legal interventions to address the failure modes.

See also Microsoft's [Threat Modeling AI/ML Systems and Dependencies](/en-us/security/threat-modeling-aiml) and [SDL Bug Bar Pivots for Machine Learning Vulnerabilities](/en-us/security/engineering/bug-bar-aiml).

How to use this document
------------------------

At the outset, we acknowledge that this is a living document which will evolve over time with the threat landscape.
We also do not prescribe technological
mitigations to these failure modes here, as defenses are scenario-specific
and tie in with the threat model and system architecture under consideration. Options presented for threat mitigation are based on current research with the expectation that those defenses will evolve over time as well.

For engineers, we recommend browsing through the overview of possible failure modes and jumping into the [threat modeling document](/en-us/security/threat-modeling-aiml). This way,
engineers can identify threats, attacks, vulnerabilities and use the
framework to plan for countermeasures where available. We then refer you
to the bug bar that maps these new vulnerabilities in the taxonomy
alongside traditional software vulnerabilities, and provides a rating
for each ML vulnerability (such as critical, important). This bug bar
is easily integrated into existing incident response processes/playbooks.

For lawyers and policy makers, this document organizes ML failure modes
and presents a framework to analyze key issues relevant for
anyone exploring policy options, such as the work done
here[5],[6]. Specifically, we have categorized failures and
consequences in a way that policy makers can begin to draw distinctions
between causes, which will inform the public policy initiatives to
promote ML safety and security. We hope that policy makers will use
these categories begin to flesh out how existing legal regimes may (not)
adequately capture emerging issues, what historical legal regimes or
policy solutions might have dealt with similar harms, and where we
should be especially sensitive to civil liberties issues.

Document Structure
------------------

In both the *Intentional Failure Modes* and *Unintentional Failure
Modes* sections, we provide a brief definition of the attack, and
an illustrative example from literature.

In the *Intentional Failure Modes* section, we provide the additional
fields:

1. What does the attack attempt to compromise in the ML system – Confidentiality, Integrity or Availability? We define Confidentiality as assuring that the components of the ML system (data, algorithm, model) are accessible only by authorized parties; Integrity is defined as assuring that the ML system can be modified only by authorized parties; Availability is defined as an assurance that the ML system is accessible to authorized parties. Together, Confidentiality, Integrity and Availability is called the CIA triad. For each intentional failure mode, we attempt to identify which of the CIA triad is compromised.
2. How much knowledge is required to mount this attack – blackbox or whitebox? In Blackbox style attacks., the attacker does NOT have direct access to the training data, no knowledge of the ML algorithm used and no access to the source code of the model. The attacker only queries the model and observes the response. In a whitebox style attack the attacker has knowledge of either ML algorithm or access to the model source code.
3. Commentary on if the attacker is violating traditional technological notion of access/authorization.

Intentionally-Motivated Failures Summary
----------------------------------------

| Scenario Number | Attack | Overview | Violates traditional technological notion of access/authorization? |
| --- | --- | --- | --- |
| 1 | Perturbation attack  | Attacker modifies the query to get appropriate response | No |
| 2 | Poisoning attack  | Attacker contaminates the training phase of ML systems to get intended result | No |
| 3 | Model Inversion | Attacker recovers the secret features used in the model by through careful queries | No |
| 4 | Membership Inference | Attacker can infer if a given data record was part of the model’s training dataset or not | No |
| 5 | Model Stealing | Attacker is able to recover the model through carefully-crafted queries | No |
| 6 | Reprogramming ML system | Repurpose the ML system to perform an activity it was not programmed for | No |
| 7 | Adversarial Example in Physical Domain  | Attacker brings adversarial examples into physical domain to subvertML system e.g: 3d printing special eyewear to fool facial recognition system | No |
| 8 | Malicious ML provider recovering training data  | Malicious ML provider can query the model used by customer and recover customer’s training data  | Yes |
| 9 | Attacking the ML supply chain | Attacker compromises the ML models as it is being downloaded for use | Yes |
| 10 | Backdoor ML  | Malicious ML provider backdoors algorithm to activate with a specific trigger | Yes |
| 11 | Exploit Software Dependencies | Attacker uses traditional software exploits like buffer overflow to confuse/control ML systems | Yes |

Unintended Failures Summary
---------------------------

| Scenario # | Failure | Overview |
| --- | --- | --- |
| 12 | Reward Hacking | Reinforcement Learning (RL) systems act in unintended ways because of mismatch between stated reward and true reward |
| 13 | Side Effects | RL system disrupts the environment as it tries to attain its goal |
| 14 | Distributional shifts | The system is tested in one kind of environment, but is unable to adapt to changes in other kinds of environment |
| 15 | Natural Adversarial Examples | Without attacker perturbations, the ML system fails owing to hard negative mining |
| 16 | Common Corruption | The system is not able to handle common corruptions and perturbations such as tilting, zooming, or noisy images. |
| 17 | Incomplete Testing | The ML system is not tested in the realistic conditions that it is meant to operate in. |

Details on Intentionally-Motivated Failures
-------------------------------------------

| Scenario # | Attack Class | Description | Type of Compromise | Scenario |
| --- | --- | --- | --- | --- |
| 1  | Perturbation attacks  | In perturbation style attacks, the attacker stealthily modifies the query to get a desired response  | Integrity | Image: Noise is added to an X-ray image, which makes the predictions go from normal scan to abnormal [1][Blackbox]
 Text translation: Specific characters are manipulated to result in incorrect translation. The attack can suppress specific word or can even remove the word completely[2][Blackbox and Whitebox]
 Speech: Researchers showed how given a speech waveform, another waveform can be exactly replicated but transcribes into a totally different text[3][Whitebox but may be extended to blackbox] |
| 2 | Poisoning attacks  | The goal of the attacker is to contaminate the machine model generated in the training phase, so that predictions on new data will be modified in the testing phase Targeted: In targeted poisoning attacks, the attacker wants to misclassify specific examples Indiscriminate: The aim here is to cause DoS like effect, which makes the system unavailable.  | Integrity | In a medical dataset where the goal is to predict the dosage of anticoagulant drug Warfarin using demographic information, etc. Researchers introduced malicious samples at 8% poisoning rate, which changed dosage by 75.06% for half of patients[4][Blackbox] In the Tay chatbot, future conversations were tainted because a fraction of the past conversations were used to train the system via feedback[5] [Blackbox]  |
| 3 | Model Inversion  | The private features used in machine learning models can be recovered | Confidentiality;  | Researchers were able to recover private training data used to train the algorithm[6] The authors were able to reconstruct faces, by just the name and access to the model to the point where Mechanical turks could use the photo to identify an individual from aline-up with 95% accuracy. The authors were also able to extract specific information. [Whitebox and Blackbox][12]  |
| 4 | Membership Inference attack | The attacker can determine whether a given data record was part of the model’s training dataset or not | Confidentiality  | Researchers were able to predict a patient’s main procedure(e.g: Surgery the patient went through) based on the attributes (e.g: age,gender, hospital)[7][Blackbox] |
| 5 | Model stealing  | The attackers recreate the underlying model by legitimately querying the model. The functionality of the new model is same as that of the underlying model.  | Confidentiality | Researchers successfully emulated the underlying algorithm from Amazon, BigML. For instance, in the BigML case, researchers were able to recover the model used to predict if someone should have a good/bad credit risk (German Credit Card dataset) using 1,150 queries and within 10 minutes[8] |
| 6 | Reprogramming deep neural nets | By means of a specially crafted query from an adversary, Machine learning systems can be reprogrammed to a task that deviates from the creator’s original intent | Integrity, Availability | Demonstrated how ImageNet, a system used to classify one of several categories of images was repurposed to count squares. Authors end the paper with a hypothetical scenario: An attacker sends Captcha images to the computer vision classifier in a cloud hosted photos service to solve the image captchas to create spam accounts[9]  |
| 7 | Adversarial Example in the Physical domain  | An adversarial example is an input/query from a malicious entity sent with the sole aim of misleading the machine learning system These examples can manifest in the physical domain  | Integrity | Researchers 3D prints a rifle with custom texture that fools image recognition system into thinking it is a turtle[10]
 Researchers construct sunglasses with a design that can now fool image recognition systems, and no longer recognize the faces correctly[11] |
| 8 | Malicious ML providers who can recover training data |  Malicious ML provider can query the model used by customer and recover customer’s training data | Confidentiality  | Researchers show how a malicious provider presents a backdoored algorithm, wherein the private training data is recovered. They were able to reconstruct faces and texts, given the model alone. [12]  |
| 9 | Attacking the ML Supply Chain[13] | Owing to large resources (data + computation) required to train algorithms, the current practice is to reuse models trained by large corporations, and modify them slightly for task at hand (e.g: ResNet is a popular image recognition model from Microsoft). These models are curated ina Model Zoo (Caffe hosts popular image recognition models). In this attack,the adversary attacks the models hosted in Caffe, thereby poisoning the well for anyone else.  | Integrity | Researchers show how it is possible for an attacker to check in malicious code into one of the popular model. An unsuspecting ML developer downloads this model and uses it as part of the image recognition system in their code [14]. The authors show how in Caffe, there exists a model whose SHA1 hash doesNOT match the authors’ digest, indicating tampering. There are 22 models without any SHA1 hash for integrity checks at all.  |
| 10 | Backdoor Machine Learning | Like in the “Attacking the ML Supply Chain”, In this attack scenario,the training process is either fully or partially outsourced to a malicious party who wants to provide the user with a trained model that contains a backdoor. The backdoored model would perform well on most inputs (including inputs that the end user may hold out as a validation set) but cause targeted misclassifications or degrade the accuracy of the model for inputs that satisfy some secret, attacker-chosen property, which we will refer to as the backdoor trigger | Confidentiality, Integrity | Researchers created a backdoored U.S. street sign classifier that identifies stop signs as speed limits only when a special sticker is added to the stop sign (backdoor trigger) 20 They are now extending this work to text processing systems, wherein specific words are replaced with the trigger being the speaker’s accent[15] |
| 11 | Exploit software dependencies of ML system | In this attack, the attacker does NOT manipulate the algorithms. Instead, exploits traditional software vulnerabilities such as buffer overflows.  | Confidentiality, Integrity, Availability,  | An adversary sends in corrupt input to an image recognition system that causes it to misclassify by exploiting a software bug in one of the dependencies. |

Details on Unintended Failures
------------------------------

| Scenario # | Attack Class | Description | Type of Compromise | Scenario |
| --- | --- | --- | --- | --- |
| 12 | Reward Hacking | Reinforcement learning systems act in unintended ways because of discrepancies between the specified reward and the true intended reward. | Safety of the system  | A huge corpus of gaming examples in AI has been compiled here[1] |
| 13 | Side Effects | RL system disrupts the environment as it tries to attain their goal  | Safety of the system  | Scenario, verbatim from the authors in [2]:“Suppose a designer wants an RL agent (for example our cleaning robot) to achieve some goal, like moving a box from one side of a room to the other.Sometimes the most effective way to achieve the goal involves doing something unrelated and destructive to the rest of the environment, like knocking over a vase of water that is in its path. If the agent is given reward only for moving the box, it will probably knock over the vase.” |
| 14 | Distributional shifts | The system is tested in one kind of environment, but is unable to adapt to changes in other kinds of environment  | Safety of the system | Researchers trained two state of the art RL agents, Rainbow DQN and A2C in a simulation to avoid lava. During training, the RL agent was able to avoid lava successfully and reach its goal. During testing, they slightly moved the position of the lava, but the RL agent was not able to avoid [3] |
| 15 | Natural Adversarial Examples  | The system incorrectly recognizes an input that was found using hard negative mining  | Safety of the system | Here the authors show how by a simple process of hard negative mining[4], it is possible to confuse the ML system by relaying the example.  |
| 16 | Common Corruption | The system is not able to handle common corruptions and perturbations such as tilting, zooming, or noisy images.  | Safety of the system | The authors[5] show how common corruptions such as changes to brightness, contrast, fog or noise added to images, have a significant drop in metrics in image recognition  |
| 17 | Incomplete Testing in Realistic conditions | The ML system is not tested in realistic conditions that it is meant to operate in  | Safety of the system | The authors in [25] highlight that that while defenders commonly account for robustness of the ML algorithm, they lose sight of realistic conditions. For instance, they argue that a missing stop sign knocked off in the wind (which is more realistic) than an attacker attempting to perturb the system's inputs. |

Acknowledgements
----------------

We would like to thank Andrew Marshall, Magnus Nystrom, John Walton, John Lambert, Sharon Xia, Andi Comissoneru, Emre Kiciman, Jugal Parikh, Sharon Gillet, members of Microsoft’s AI and Ethics in Engineering and Research (AETHER) committee’s Security workstream, Amar Ashar, Samuel Klein, Jonathan Zittrain, members of AI Safety Security Working Group at Berkman Klein for providing helpful feedback. We would also like to thank reviewers from 23 external partners, standards organization, and government organizations for shaping the taxonomy.

Bibliography
------------

[1] Li, Guofu, et al. "Security Matters: A Survey on Adversarial Machine
Learning." *arXiv preprint arXiv:1810.07339* (2018).

[2] Chakraborty, Anirban, et al. "Adversarial attacks and defences: A
survey." *arXiv preprint arXiv:1810.00069* (2018).

[3] Ortega, Pedro, and Vishal Maini. "Building safe artificial
intelligence: specification, robustness, and assurance." *DeepMind
Safety Research Blog* (2018).

[4] Amodei, Dario, et al. "Concrete problems in AI safety." *arXiv
preprint arXiv:1606.06565* (2016).

[5] Shankar Siva Kumar, Ram, et al. "Law and Adversarial Machine
Learning." *arXiv preprint arXiv:1810.10731* (2018).

[6] Calo, Ryan, et al. "Is Tricking a Robot Hacking?." University of
Washington School of Law Research Paper 2018-05 (2018).

[7] Paschali, Magdalini, et al. "Generalizability vs. Robustness:
Adversarial Examples for Medical Imaging." arXiv preprint
arXiv:1804.00504 (2018).

[8] Ebrahimi, Javid, Daniel Lowd, and Dejing Dou. "On Adversarial
Examples for Character-Level Neural Machine Translation." arXiv preprint
arXiv:1806.09030 (2018)

[9] Carlini, Nicholas, and David Wagner. "Audio adversarial examples:
Targeted attacks on speech-to-text." arXiv preprint arXiv:1801.01944
(2018).

[10] Jagielski, Matthew, et al. "Manipulating machine learning:
Poisoning attacks and countermeasures for regression learning." *arXiv
preprint arXiv:1804.00308* (2018)

[11] [https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/]

[12] Fredrikson M, Jha S, Ristenpart T. 2015. Model inversion attacks
that exploit confidence information and basic countermeasures

[13] Shokri R, Stronati M, Song C, Shmatikov V. 2017. Membership
inference attacks against machine learning models. In *Proc. of the 2017
IEEE Symp. on Security and Privacy (SP)*, *San Jose, CA, 22–24 May
2017*, pp. 3–18. New York, NY: IEEE.

[14] Tramèr, Florian, et al. "Stealing Machine Learning Models via
Prediction APIs." *USENIX Security Symposium*. 2016.

[15] Elsayed, Gamaleldin F., Ian Goodfellow, and Jascha Sohl-Dickstein.
"Adversarial Reprogramming of Neural Networks." *arXiv preprint
arXiv:1806.11146* (2018).

[16] Athalye, Anish, and Ilya Sutskever. "Synthesizing robust
adversarial examples." *arXiv preprint arXiv:1707.07397*(2017)

[17] Sharif, Mahmood, et al. "Adversarial Generative Nets: Neural
Network Attacks on State-of-the-Art Face Recognition." *arXiv preprint
arXiv:1801.00349* (2017).

[19] Xiao, Qixue, et al. "Security Risks in Deep Learning
Implementations." *arXiv preprint arXiv:1711.11008* (2017).

[20] Gu, Tianyu, Brendan Dolan-Gavitt, and Siddharth Garg. "Badnets:
Identifying vulnerabilities in the machine learning model supply
chain." *arXiv preprint arXiv:1708.06733* (2017)

[21] [https://www.wired.com/story/machine-learning-backdoors/]

[22] [https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml]

[23] Amodei, Dario, et al. "Concrete problems in AI safety." *arXiv
preprint arXiv:1606.06565* (2016).

[24] Leike, Jan, et al. "AI safety gridworlds." *arXiv preprint
arXiv:1711.09883* (2017).

[25] Gilmer, Justin, et al. "Motivating the rules of the game for
adversarial example research." *arXiv preprint arXiv:1807.06732* (2018).

[26] Hendrycks, Dan, and Thomas Dietterich. "Benchmarking neural network
robustness to common corruptions and perturbations." *arXiv preprint
arXiv:1903.12261* (2019).

Failure Modes in Machine Learning - Security documentation

Ram Shankar Siva Kumar
David O Brien
Kendra Albert
Salomé Viljöen
Jeffrey Snover


Toggle the table of contents

Toggle the table of contents

Von Neumann–Morgenstern utility theorem
=======================================

3 languages

* [Français](https://fr.wikipedia.org/wiki/Th%C3%A9or%C3%A8me_d%27utilit%C3%A9_de_von_Neumann-Morgenstern "Théorème d'utilité de von Neumann-Morgenstern – French")
* [עברית](https://he.wikipedia.org/wiki/%D7%9E%D7%A9%D7%A4%D7%98_%D7%A4%D7%95%D7%9F_%D7%A0%D7%95%D7%99%D7%9E%D7%9F-%D7%9E%D7%95%D7%A8%D7%92%D7%A0%D7%A9%D7%98%D7%A8%D7%9F "משפט פון נוימן-מורגנשטרן – Hebrew")
* [Nederlands](https://nl.wikipedia.org/wiki/Von_Neumann-Morgenstern-nutsfunctie "Von Neumann-Morgenstern-nutsfunctie – Dutch")

[Edit links](https://www.wikidata.org/wiki/Special:EntityPage/Q4358367#sitelinks-wikipedia "Edit interlanguage links")

* [Article](/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem "View the content page [c]")
* [Talk](/wiki/Talk:Von_Neumann%E2%80%93Morgenstern_utility_theorem "Discuss improvements to the content page [t]")

English

* [Read](/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem)
* [Edit](/w/index.php?title=Von_Neumann%E2%80%93Morgenstern_utility_theorem&action=edit&oldid=1044421624 "Edit this page [e]")
* [View history](/w/index.php?title=Von_Neumann%E2%80%93Morgenstern_utility_theorem&action=history "Past revisions of this page [h]")

Tools

Tools
move to sidebar
hide

 Actions

* [Read](/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem)
* [Edit](/w/index.php?title=Von_Neumann%E2%80%93Morgenstern_utility_theorem&action=edit&oldid=1044421624)
* [View history](/w/index.php?title=Von_Neumann%E2%80%93Morgenstern_utility_theorem&action=history)

 General

* [What links here](/wiki/Special:WhatLinksHere/Von_Neumann%E2%80%93Morgenstern_utility_theorem "List of all English Wikipedia pages containing links to this page [j]")
* [Related changes](/wiki/Special:RecentChangesLinked/Von_Neumann%E2%80%93Morgenstern_utility_theorem "Recent changes in pages linked from this page [k]")
* [Upload file](/wiki/Wikipedia:File_Upload_Wizard "Upload files [u]")
* [Special pages](/wiki/Special:SpecialPages "A list of all special pages [q]")
* [Permanent link](/w/index.php?title=Von_Neumann%E2%80%93Morgenstern_utility_theorem&oldid=1044421624 "Permanent link to this revision of this page")
* [Page information](/w/index.php?title=Von_Neumann%E2%80%93Morgenstern_utility_theorem&action=info "More information about this page")
* [Cite this page](/w/index.php?title=Special:CiteThisPage&page=Von_Neumann%E2%80%93Morgenstern_utility_theorem&id=1044421624&wpFormIdentifier=titleform "Information on how to cite this page")
* [Wikidata item](https://www.wikidata.org/wiki/Special:EntityPage/Q4358367 "Structured data on this page hosted by Wikidata [g]")

 Print/export

* [Download as PDF](/w/index.php?title=Special:Book&bookcmd=render_article&arttitle=Von+Neumann%E2%80%93Morgenstern+utility+theorem&returnto=Von+Neumann%E2%80%93Morgenstern+utility+theorem&oldid=1044421624&writer=rl "Download this page as a PDF file")
* [Printable version](/w/index.php?title=Von_Neumann%E2%80%93Morgenstern_utility_theorem&printable=yes "Printable version of this page [p]")

 Print/export

From Wikipedia, the free encyclopedia

**This is an [old revision](/wiki/Help:Page_history "Help:Page history") of this page, as edited by [2601:445:4380:7dd0::6b8c](/wiki/Special:Contributions/2601:445:4380:7DD0:0:0:0:6B8C "Special:Contributions/2601:445:4380:7DD0:0:0:0:6B8C") ([talk](/w/index.php?title=User_talk:2601:445:4380:7DD0:0:0:0:6B8C&action=edit&redlink=1 "User talk:2601:445:4380:7DD0:0:0:0:6B8C (page does not exist)")) at 04:35, 15 September 2021. The present address (URL) is a [permanent link](/wiki/Help:Permanent_link "Help:Permanent link") to this revision, which may differ significantly from the [current revision](https://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem).**

Revision as of 04:35, 15 September 2021 by [2601:445:4380:7dd0::6b8c](/wiki/Special:Contributions/2601:445:4380:7DD0:0:0:0:6B8C "Special:Contributions/2601:445:4380:7DD0:0:0:0:6B8C") ([talk](/w/index.php?title=User_talk:2601:445:4380:7DD0:0:0:0:6B8C&action=edit&redlink=1 "User talk:2601:445:4380:7DD0:0:0:0:6B8C (page does not exist)"))([diff](/w/index.php?title=Von_Neumann%E2%80%93Morgenstern_utility_theorem&diff=prev&oldid=1044421624 "Von Neumann–Morgenstern utility theorem")) [← Previous revision](/w/index.php?title=Von_Neumann%E2%80%93Morgenstern_utility_theorem&direction=prev&oldid=1044421624 "Von Neumann–Morgenstern utility theorem") | [Latest revision](/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem "Von Neumann–Morgenstern utility theorem") ([diff](/w/index.php?title=Von_Neumann%E2%80%93Morgenstern_utility_theorem&diff=cur&oldid=1044421624 "Von Neumann–Morgenstern utility theorem")) | [Newer revision →](/w/index.php?title=Von_Neumann%E2%80%93Morgenstern_utility_theorem&direction=next&oldid=1044421624 "Von Neumann–Morgenstern utility theorem") ([diff](/w/index.php?title=Von_Neumann%E2%80%93Morgenstern_utility_theorem&diff=next&oldid=1044421624 "Von Neumann–Morgenstern utility theorem"))
Any individual whose preferences satisfy four axioms has a utility function
In [decision theory](/wiki/Decision_theory "Decision theory"), the **von Neumann–Morgenstern** (**VNM**) **utility theorem** shows that, under certain [axioms](/wiki/Axiom "Axiom") of [rational behavior](/wiki/Rationality "Rationality"), a decision-maker faced with [risky](/wiki/Risk "Risk") (probabilistic) outcomes of different choices will behave as if he or she is maximizing the [expected value](/wiki/Expected_value "Expected value") of some function defined over the potential outcomes at some specified point in the future. This function is known as the von Neumann–Morgenstern utility function. The theorem is the basis for [expected utility theory](/wiki/Expected_utility_theory "Expected utility theory").

In 1947, [John von Neumann](/wiki/John_von_Neumann "John von Neumann") and [Oskar Morgenstern](/wiki/Oskar_Morgenstern "Oskar Morgenstern") proved that any individual whose [preferences](/wiki/Preference_(economics) "Preference (economics)") satisfied four axioms has a [utility function](/wiki/Utility_function "Utility function");[[1]](#cite_note-VNM-1) such an individual's preferences can be represented on an [interval scale](/wiki/Interval_scale "Interval scale") and the individual will always prefer actions that maximize expected utility. That is, they proved that an agent is (VNM-)rational *if and only if* there exists a real-valued function *u* defined by possible outcomes such that every preference of the agent is characterized by maximizing the expected value of *u*, which can then be defined as the agent's *VNM-utility* (it is unique up to adding a constant and multiplying by a positive scalar). No claim is made that the agent has a "conscious desire" to maximize *u*, only that *u* exists.

The [expected utility hypothesis](/wiki/Expected_utility_hypothesis "Expected utility hypothesis") is that rationality can be modeled as maximizing an [expected value](/wiki/Expected_value "Expected value"), which given the theorem, can be summarized as "*rationality is VNM-rationality*". However, the axioms themselves have been critiqued on various grounds, resulting in the axioms being given further justification.[[2]](#cite_note-2)

VNM-utility is a *decision utility* in that it is used to describe *decision preferences*. It is related but not equivalent to so-called *E-utilities*[[3]](#cite_note-KWS-3) (experience utilities), notions of utility intended to measure happiness such as that of [Bentham](/wiki/Jeremy_Bentham "Jeremy Bentham")'s [Greatest Happiness Principle](/wiki/Greatest_happiness_principle "Greatest happiness principle").

Set-up
------

In the theorem, an individual agent is faced with options called [*lotteries*](/wiki/Lottery_(probability) "Lottery (probability)"). Given some [mutually exclusive](/wiki/Mutually_exclusive "Mutually exclusive") outcomes, a lottery is a scenario where each outcome will happen with a given [probability](/wiki/Probability "Probability"), all probabilities summing to one. For example, for two outcomes *A* and *B*,

L
=
0.25
A
+
0.75
B

{\displaystyle L=0.25A+0.75B}

![{\displaystyle L=0.25A+0.75B}](https://wikimedia.org/api/rest_v1/media/math/render/svg/65609f0204eae5e293e04be54a8765d4e6076871)
denotes a scenario where *P*(*A*) = 25% is the probability of *A* occurring and *P*(*B*) = 75% (and exactly one of them will occur). More generally, for a lottery with many possible outcomes *Ai*, we write:

L
=
∑

p

i

A

i

,

{\displaystyle L=\sum p\_{i}A\_{i},}

![{\displaystyle L=\sum p_{i}A_{i},}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b8d0fce14b6b440fc698a747b407009c68822f1e)
with the sum of the

p

i

{\displaystyle p\_{i}}

![p_{i}](https://wikimedia.org/api/rest_v1/media/math/render/svg/5bab39399bf5424f25d957cdc57c84a0622626d2)s equalling 1.

The outcomes in a lottery can themselves be lotteries between other outcomes, and the expanded expression is considered an equivalent lottery: 0.5(0.5*A* + 0.5*B*) + 0.5*C* = 0.25*A* + 0.25*B* + 0.50*C*.

If lottery *M* is preferred over lottery *L*, we write

L
≺
M

{\displaystyle L\prec M}

![{\displaystyle L\prec M}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0182f1e20f44b51055fcd59eb81757897e9294af), or equivalently,

M
≻
L

{\displaystyle M\succ L}

![{\displaystyle M\succ L}](https://wikimedia.org/api/rest_v1/media/math/render/svg/80803d03780883af17aff238ddf94a958413dbe9). If the agent is indifferent between *L* and *M*, we write the *indifference relation*[[4]](#cite_note-Kreps-4)

L
∼
M
.

{\displaystyle L\sim M.}

![L\sim M.](https://wikimedia.org/api/rest_v1/media/math/render/svg/4db260b7eff79576daaa5575de3e4e1fb418f8b7) If *M* is either preferred over or viewed with indifference relative to *L*, we write

L
⪯
M
.

{\displaystyle L\preceq M.}

![L \preceq M.](https://wikimedia.org/api/rest_v1/media/math/render/svg/5c6cef106f071ebd052c81c554e6772bc82643e9)

The axioms
----------

The four axioms of VNM-rationality are then *completeness*, *transitivity*, *continuity*, and *independence*.

Completeness assumes that an individual has well defined preferences:

**Axiom 1 (Completeness)** For any lotteries *L,M*, exactly one of the following holds:

L
≺
M

{\displaystyle \,L\prec M}

![{\displaystyle \,L\prec M}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4d342c7334631e2f25929ce5b4c0fa5a237dc2b8),

M
≺
L

{\displaystyle \,M\prec L}

![{\displaystyle \,M\prec L}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f90b9bdfd3b1f7de1c8e264c260d9b32d9e317f1), or

L
∼
M

{\displaystyle \,L\sim M}

![\, L \sim M  ](https://wikimedia.org/api/rest_v1/media/math/render/svg/9cc4f610696acad4bf7ac4624dcad1f5500398f0)
(either *M* is preferred, *L* is preferred, or the individual is indifferent[[5]](#cite_note-nop-5)).

[Transitivity](/wiki/Transitive_relation "Transitive relation") assumes that preferences are consistent across any three options:

**Axiom 2 (Transitivity)** If

L
≺
M

{\displaystyle \,L\prec M}

![{\displaystyle \,L\prec M}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4d342c7334631e2f25929ce5b4c0fa5a237dc2b8) and

M
≺
N

{\displaystyle \,M\prec N}

![{\displaystyle \,M\prec N}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8d01b1e70f36e74bb88cb96d0017fad1fe0ca714), then

L
≺
N

{\displaystyle \,L\prec N}

![{\displaystyle \,L\prec N}](https://wikimedia.org/api/rest_v1/media/math/render/svg/7bf6cb42fbbd0b49d21b2e8e309d00efe3fa5662), and similarly for

∼

{\displaystyle \sim }

![\sim ](https://wikimedia.org/api/rest_v1/media/math/render/svg/afcc42adfcfdc24d5c4c474869e5d8eaa78d1173).
Continuity assumes that there is a "tipping point" between being *better than* and *worse than* a given middle option:

**Axiom 3 (Continuity):** If

L
⪯
M
⪯
N

{\displaystyle \,L\preceq M\preceq N}

![{\displaystyle \,L\preceq M\preceq N}](https://wikimedia.org/api/rest_v1/media/math/render/svg/db759acfaed5c7544ea659645bf6aa4d03cb9876), then there exists a probability

p
∈
[
0
,
1
]

{\displaystyle \,p\in [0,1]}

![{\displaystyle \,p\in [0,1]}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4449572ac985c35af802cf3a3f8e28594be10554) such that

p
L
+
(
1
−
p
)
N

∼

M

{\displaystyle \,pL+(1-p)N\,\sim \,M}

![{\displaystyle \,pL+(1-p)N\,\sim \,M}](https://wikimedia.org/api/rest_v1/media/math/render/svg/27b8381ac45f6fd337b78643a7cdf72ed7646231)
where the notation on the left side refers to a situation in which *L* is received with probability *p* and *N* is received with probability (1–*p*).

Instead of continuity, an alternative axiom can be assumed that does not involve a precise equality, called the [Archimedean property](/wiki/Archimedean_property "Archimedean property").[[4]](#cite_note-Kreps-4) It says that any separation in preference can be maintained under a sufficiently small deviation in probabilities:

**Axiom 3′ (Archimedean property):** If

L
≺
M
≺
N

{\displaystyle \,L\prec M\prec N}

![{\displaystyle \,L\prec M\prec N}](https://wikimedia.org/api/rest_v1/media/math/render/svg/3a0072df9c734c8ad431711fd38d0a4d5519cd43), then there exists a probability

ε
∈
(
0
,
1
)

{\displaystyle \,\varepsilon \in (0,1)}

![\,\varepsilon\in(0,1)](https://wikimedia.org/api/rest_v1/media/math/render/svg/8843056b4bef077bb8981f0785cf0c8ac78eb818) such that

(
1
−
ε
)
L
+
ε
N

≺

M

≺

ε
L
+
(
1
−
ε
)
N
.

{\displaystyle \,(1-\varepsilon )L+\varepsilon N\,\prec \,M\,\prec \,\varepsilon L+(1-\varepsilon )N.}

![{\displaystyle \,(1-\varepsilon )L+\varepsilon N\,\prec \,M\,\prec \,\varepsilon L+(1-\varepsilon )N.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/64ff3f2253aa81b3f3e3f9841800f05b774e57f5)
Only one of (3) or (3′) need to be assumed, and the other will be implied by the theorem.

[Independence of irrelevant alternatives](/wiki/Independence_of_irrelevant_alternatives "Independence of irrelevant alternatives") assumes that a preference holds independently of the possibility of another outcome:

**Axiom 4 (Independence):** For any

N

{\displaystyle \,N}

![\,N](https://wikimedia.org/api/rest_v1/media/math/render/svg/3d76de4670e50d6ff69434e1846142af7f347769) and

p
∈
(
0
,
1
]

{\displaystyle \,p\in (0,1]}

![{\displaystyle \,p\in (0,1]}](https://wikimedia.org/api/rest_v1/media/math/render/svg/590a2ef3670309007fe88b3d8cb8b752eaba85be),

L
⪯
M

iff

p
L
+
(
1
−
p
)
N
⪯
p
M
+
(
1
−
p
)
N
.

{\displaystyle \,L\preceq M\qquad {\text{iff}}\qquad pL+(1-p)N\preceq pM+(1-p)N.}

![{\displaystyle \,L\preceq M\qquad {\text{iff}}\qquad pL+(1-p)N\preceq pM+(1-p)N.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e8a6623aac916e0cd10fbc893d1064d3ee4deced)
The independence axiom implies the axiom on reduction of compound lotteries:[[6]](#cite_note-6)

**Axiom 4′ (Reduction of compound lotteries):** For any lotteries

L
,

L
′

,
N
,

N
′

{\displaystyle L,L',N,N'}

![{\displaystyle L,L',N,N'}](https://wikimedia.org/api/rest_v1/media/math/render/svg/5b655f53c05fe6b42158783a5d51f756fe957320) and any

p
,
q
∈
[
0
,
1
]

{\displaystyle p,q\in [0,1]}

![{\displaystyle p,q\in [0,1]}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4ce8c8abe5c6cff2c531084637e986f132a54703),

if

L
∼
q

L
′

+
(
1
−
q
)

N
′

,

{\displaystyle {\text{if}}\qquad L\sim qL'+(1-q)N',}

![{\displaystyle {\text{if}}\qquad L\sim qL'+(1-q)N',}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c80efa13cc1d99b6588914665b7a5e7524cf8a82)

then

p
L
+
(
1
−
p
)
N
∼
p
q

L
′

+
p
(
1
−
q
)

N
′

+
(
1
−
p
)
N
.

{\displaystyle {\text{then}}\quad pL+(1-p)N\sim pqL'+p(1-q)N'+(1-p)N.}

![{\displaystyle {\text{then}}\quad pL+(1-p)N\sim pqL'+p(1-q)N'+(1-p)N.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/707a8a608dcee7289d3ff1b05c397190a4774002)
To see how Axiom 4 implies Axiom 4', set

M
=
q

L
′

+
(
1
−
q
)

N
′

{\displaystyle M=qL'+(1-q)N'}

![{\displaystyle M=qL'+(1-q)N'}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d68a56fdfd003375cf2f70b4e684316f42631e1a) in the expression in Axiom 4, and expand.

The theorem
-----------

For any VNM-rational agent (i.e. satisfying axioms 1–4), there exists a function *u* which assigns to each outcome *A* a real number *u(A)* such that for any two lotteries,

L
≺
M

i
f

a
n
d

o
n
l
y

i
f

E
(
u
(
L
)
)
<
E
(
u
(
M
)
)
,

{\displaystyle L\prec M\qquad \mathrm {if\,and\,only\,if} \qquad E(u(L))<E(u(M)),}

![{\displaystyle L\prec M\qquad \mathrm {if\,and\,only\,if} \qquad E(u(L))<E(u(M)),}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d38fb0a62efa6328f0fc316e32567285cf549559)
where *E(u(L))*, or more briefly *Eu*(*L*) is given by

E
u
(

p

1

A

1

+
⋯
+

p

n

A

n

)
=

p

1

u
(

A

1

)
+
⋯
+

p

n

u
(

A

n

)
.

{\displaystyle Eu(p\_{1}A\_{1}+\cdots +p\_{n}A\_{n})=p\_{1}u(A\_{1})+\cdots +p\_{n}u(A\_{n}).}

![{\displaystyle Eu(p_{1}A_{1}+\cdots +p_{n}A_{n})=p_{1}u(A_{1})+\cdots +p_{n}u(A_{n}).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6900e086134df58e5834010eeb2b76021180f90b)
As such, *u* can be uniquely determined (up to adding a constant and multiplying by a positive scalar) by preferences between *simple lotteries*, meaning those of the form *pA* + (1 − *p*)*B* having only two outcomes. Conversely, any agent acting to maximize the expectation of a function *u* will obey axioms 1–4. Such a function is called the agent's **von Neumann–Morgenstern (VNM) utility**.

### Proof sketch

The proof is constructive: it shows how the desired function

u

{\displaystyle u}

![u](https://wikimedia.org/api/rest_v1/media/math/render/svg/c3e6bb763d22c20916ed4f0bb6bd49d7470cffd8) can be built. Here we outline the construction process for the case in which the number of sure outcomes is finite.[[7]](#cite_note-KeeneyRaiffa1993-7): 132–134

Suppose there are *n* sure outcomes,

A

1

…

A

n

{\displaystyle A\_{1}\dots A\_{n}}

![A_1\dots A_n](https://wikimedia.org/api/rest_v1/media/math/render/svg/c3ce869c360c2aba952c7a43ac3739f4c2bb5a51). Note that every sure outcome can be seen as a lottery: it is a degenerate lottery in which the outcome is selected with probability 1. Hence, by the Completeness and Transitivity axioms, it is possible to order the outcomes from worst to best:

A

1

⪯

A

2

⪯
⋯
⪯

A

n

{\displaystyle A\_{1}\preceq A\_{2}\preceq \cdots \preceq A\_{n}}

![A_1\preceq A_2\preceq \cdots \preceq A_n](https://wikimedia.org/api/rest_v1/media/math/render/svg/4f23c403ccbb9855af95253c91bfb94cadd1c055)
We assume that at least one of the inequalities is strict (otherwise the utility function is trivial—a constant). So

A

1

≺

A

n

{\displaystyle A\_{1}\prec A\_{n}}

![A_1\prec A_n](https://wikimedia.org/api/rest_v1/media/math/render/svg/974d8a13494fdf0d567ae79c23b4c86e608eb83d). We use these two extreme outcomes—the worst and the best—as the scaling unit of our utility function, and define:

u
(

A

1

)
=
0

{\displaystyle u(A\_{1})=0}

![u(A_1)=0](https://wikimedia.org/api/rest_v1/media/math/render/svg/6ea7454c9e90080d711ecf1538b750751fbd79d4) and

u
(

A

n

)
=
1

{\displaystyle u(A\_{n})=1}

![u(A_n)=1](https://wikimedia.org/api/rest_v1/media/math/render/svg/23b11e716076680902f7d1994114f63e86d2fdef)
For every probability

p
∈
[
0
,
1
]

{\displaystyle p\in [0,1]}

![p\in [0,1]](https://wikimedia.org/api/rest_v1/media/math/render/svg/33c3a52aa7b2d00227e85c641cca67e85583c43c), define a lottery that selects the best outcome with probability

p

{\displaystyle p}

![p](https://wikimedia.org/api/rest_v1/media/math/render/svg/81eac1e205430d1f40810df36a0edffdc367af36) and the worst outcome otherwise:

L
(
p
)
=
p
⋅

A

n

+
(
1
−
p
)
⋅

A

1

{\displaystyle L(p)=p\cdot A\_{n}+(1-p)\cdot A\_{1}}

![L(p) = p\cdot A_n + (1-p)\cdot A_1](https://wikimedia.org/api/rest_v1/media/math/render/svg/a9674d123fc09f2567751cd385e3ef7e8753eebe)
Note that

L
(
0
)
∼

A

1

{\displaystyle L(0)\sim A\_{1}}

![L(0)\sim A_1](https://wikimedia.org/api/rest_v1/media/math/render/svg/2f4413414a008bfb6833e33dc8b0992d9d8eaf5d) and

L
(
1
)
∼

A

n

{\displaystyle L(1)\sim A\_{n}}

![L(1)\sim A_n](https://wikimedia.org/api/rest_v1/media/math/render/svg/98611cb38a47d4da17c824a6cb998f8070cd876b).

By the Continuity axiom, for every sure outcome

A

i

{\displaystyle A\_{i}}

![A_{i}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1aed3b5def921afbe6cc48aaf8f9b11c6f1c1e2d), there is a probability

q

i

{\displaystyle q\_{i}}

![q_{i}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2752dcbff884354069fe332b8e51eb0a70a531b6) such that:

L
(

q

i

)
∼

A

i

{\displaystyle L(q\_{i})\sim A\_{i}}

![L(q_i) \sim A_i](https://wikimedia.org/api/rest_v1/media/math/render/svg/c486d2991dc0ac02edbbf0d4cace808c9d2d90d6)
and

0
=

q

1

≤

q

2

≤
⋯
≤

q

n

=
1

{\displaystyle 0=q\_{1}\leq q\_{2}\leq \cdots \leq q\_{n}=1}

![{\displaystyle 0=q_{1}\leq q_{2}\leq \cdots \leq q_{n}=1}](https://wikimedia.org/api/rest_v1/media/math/render/svg/deb481fc6c8341ce33a3113c04bb0ec6667b4657)
For every

i

{\displaystyle i}

![i](https://wikimedia.org/api/rest_v1/media/math/render/svg/add78d8608ad86e54951b8c8bd6c8d8416533d20), the utility function for outcome

A

i

{\displaystyle A\_{i}}

![A_{i}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1aed3b5def921afbe6cc48aaf8f9b11c6f1c1e2d) is defined as

u
(

A

i

)
=

q

i

{\displaystyle u(A\_{i})=q\_{i}}

![u(A_i)=q_i](https://wikimedia.org/api/rest_v1/media/math/render/svg/329248b105e96ed5ec7096a6f988a4b22cc56447)
so the utility of every lottery

M
=

∑

i

p

i

A

i

{\displaystyle M=\sum \_{i}p\_{i}A\_{i}}

![{\displaystyle M=\sum _{i}p_{i}A_{i}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e64e1685842c6921d9488191c61f8fa8716816b4) is the expectation of *u*:

u
(
M
)
=
u

(

∑

i

p

i

A

i

)

=

∑

i

p

i

u
(

A

i

)
=

∑

i

p

i

q

i

{\displaystyle u(M)=u\left(\sum \_{i}p\_{i}A\_{i}\right)=\sum \_{i}p\_{i}u(A\_{i})=\sum \_{i}p\_{i}q\_{i}}

![{\displaystyle u(M)=u\left(\sum _{i}p_{i}A_{i}\right)=\sum _{i}p_{i}u(A_{i})=\sum _{i}p_{i}q_{i}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ce2af3a091becb22de5f63e8cdd97cda5fae7cbf)
To see why this utility function makes sense, consider a lottery

M
=

∑

i

p

i

A

i

{\displaystyle M=\sum \_{i}p\_{i}A\_{i}}

![{\displaystyle M=\sum _{i}p_{i}A_{i}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e64e1685842c6921d9488191c61f8fa8716816b4), which selects outcome

A

i

{\displaystyle A\_{i}}

![A_{i}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1aed3b5def921afbe6cc48aaf8f9b11c6f1c1e2d) with probability

p

i

{\displaystyle p\_{i}}

![p_{i}](https://wikimedia.org/api/rest_v1/media/math/render/svg/5bab39399bf5424f25d957cdc57c84a0622626d2). But, by our assumption, the decision maker is indifferent between the sure outcome

A

i

{\displaystyle A\_{i}}

![A_{i}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1aed3b5def921afbe6cc48aaf8f9b11c6f1c1e2d) and the lottery

q

i

⋅

A

n

+
(
1
−

q

i

)
⋅

A

1

{\displaystyle q\_{i}\cdot A\_{n}+(1-q\_{i})\cdot A\_{1}}

![q_i\cdot A_n + (1-q_i)\cdot A_1](https://wikimedia.org/api/rest_v1/media/math/render/svg/3a813203c72dae01f932796a01533ff9a27b8496). So, by the Reduction axiom, he is indifferent between the lottery

M

{\displaystyle M}

![M](https://wikimedia.org/api/rest_v1/media/math/render/svg/f82cade9898ced02fdd08712e5f0c0151758a0dd) and the following lottery:

M
′

=

∑

i

p

i

[

q

i

⋅

A

n

+
(
1
−

q

i

)
⋅

A

1

]

{\displaystyle M'=\sum \_{i}p\_{i}[q\_{i}\cdot A\_{n}+(1-q\_{i})\cdot A\_{1}]}

![{\displaystyle M'=\sum _{i}p_{i}[q_{i}\cdot A_{n}+(1-q_{i})\cdot A_{1}]}](https://wikimedia.org/api/rest_v1/media/math/render/svg/dae21141e11bb54aef6525b15bebd95122c0dde4)

M
′

=

(

∑

i

p

i

q

i

)

⋅

A

n

+

(

∑

i

p

i

(
1
−

q

i

)

)

⋅

A

1

{\displaystyle M'=\left(\sum \_{i}p\_{i}q\_{i}\right)\cdot A\_{n}+\left(\sum \_{i}p\_{i}(1-q\_{i})\right)\cdot A\_{1}}

![{\displaystyle M'=\left(\sum _{i}p_{i}q_{i}\right)\cdot A_{n}+\left(\sum _{i}p_{i}(1-q_{i})\right)\cdot A_{1}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d2e31f317e5fc2a3e01358ae041898bca3349e6a)

M
′

=
u
(
M
)
⋅

A

n

+
(
1
−
u
(
M
)
)
⋅

A

1

{\displaystyle M'=u(M)\cdot A\_{n}+(1-u(M))\cdot A\_{1}}

![M' = u(M)\cdot A_n + (1-u(M))\cdot A_1](https://wikimedia.org/api/rest_v1/media/math/render/svg/d6456123b1761fef218f7a1674b4ec04336fc738)
The lottery

M
′

{\displaystyle M'}

![M'](https://wikimedia.org/api/rest_v1/media/math/render/svg/5a3b2ef3304c46b5e7859eec0b1bc057c8eb3f75) is, in effect, a lottery in which the best outcome is won with probability

u
(
M
)

{\displaystyle u(M)}

![u(M)](https://wikimedia.org/api/rest_v1/media/math/render/svg/65037e0ee7a34a4fa76fe9f52abd6919d9bd8678), and the worst outcome otherwise.

Hence, if

u
(
M
)
>
u
(
L
)

{\displaystyle u(M)>u(L)}

![u(M)>u(L)](https://wikimedia.org/api/rest_v1/media/math/render/svg/a0b88ccad5fbc7082499b20e76f671572dc89528), a rational decision maker would prefer the lottery

M

{\displaystyle M}

![M](https://wikimedia.org/api/rest_v1/media/math/render/svg/f82cade9898ced02fdd08712e5f0c0151758a0dd) over the lottery

L

{\displaystyle L}

![L](https://wikimedia.org/api/rest_v1/media/math/render/svg/103168b86f781fe6e9a4a87b8ea1cebe0ad4ede8), because it gives him a larger chance to win the best outcome.

Hence:

L
≺
M

{\displaystyle L\prec M\;}

![{\displaystyle L\prec M\;}](https://wikimedia.org/api/rest_v1/media/math/render/svg/54144dccfa8f037f645c635f0c41d0f90b2fa40c) if and only if

E
(
u
(
L
)
)
<
E
(
u
(
M
)
)
.

{\displaystyle E(u(L))<E(u(M)).}

![{\displaystyle E(u(L))<E(u(M)).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/645337649d5df7ee08331ef4aa050847a7ee5c1f)
### Reaction

Von Neumann and Morgenstern anticipated surprise at the strength of their conclusion. But according to them, the reason their utility function works is that it is constructed precisely to fill the role of something whose expectation is maximized:

>
> "Many economists will feel that we are assuming far too much ... Have we not shown too much? ... As far as we can see, our postulates [are] plausible ... We have practically defined numerical utility as being that thing for which the calculus of mathematical expectations is legitimate." – *VNM 1953, § 3.1.1 p.16 and § 3.7.1 p. 28*[[1]](#cite_note-VNM-1)
>
>
>
>

Thus, the content of the theorem is that the construction of *u* is possible, and they claim little about its nature.

Consequences
------------

### Automatic consideration of risk aversion

.mw-parser-output .hatnote{font-style:italic}.mw-parser-output div.hatnote{padding-left:1.6em;margin-bottom:0.5em}.mw-parser-output .hatnote i{font-style:normal}.mw-parser-output .hatnote+link+.hatnote{margin-top:-0.5em}Main article: [Risk aversion](/wiki/Risk_aversion "Risk aversion")
It is often the case that a person, faced with real-world [gambles](/wiki/Gamble "Gamble") with money, does not act to maximize the expected value of their *dollar assets.* For example, a person who only possesses $1000 in savings may be reluctant to risk it all for a 20% chance odds to win $10,000, even though

20
%
(
$
10

000
)
+
80
%
(
$
0
)
=
$
2000
>
100
%
(
$
1000
)

{\displaystyle 20\%(\$10\,000)+80\%(\$0)=\$2000>100\%(\$1000)}

![{\displaystyle 20\%(\$10\,000)+80\%(\$0)=\$2000>100\%(\$1000)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/167ca3f9cd2649a8a8811e2fe1f36573b31d6f00)
However, *if* the person is VNM-rational, such facts are automatically accounted for in their utility function *u*. In this example, we could conclude that

20
%
u
(
$
10

000
)
+
80
%
u
(
$
0
)
<
u
(
$
1000
)

{\displaystyle 20\%u(\$10\,000)+80\%u(\$0)<u(\$1000)}

![{\displaystyle 20\%u(\$10\,000)+80\%u(\$0)<u(\$1000)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ff38442cce8004b358c60a4495f84f2b4d456544)
where the dollar amounts here really represent *outcomes* (cf. "*[value](/wiki/Value_theory "Value theory")*"), the three possible situations the individual could face. In particular, *u* can exhibit properties like *u*($1)+*u*($1) ≠ *u*($2) without contradicting VNM-rationality at all. This leads to a quantitative theory of [monetary risk aversion.](/wiki/Risk_aversion#Absolute_risk_aversion "Risk aversion")

### Implications for the expected utility hypothesis

Main article: [Expected utility hypothesis](/wiki/Expected_utility_hypothesis "Expected utility hypothesis")
In 1738, [Daniel Bernoulli](/wiki/Daniel_Bernoulli "Daniel Bernoulli") published a treatise[[8]](#cite_note-8) in which he posits that rational behavior can be described as maximizing the expectation of a function *u*, which in particular need not be monetary-valued, thus accounting for risk aversion. This is the *expected utility hypothesis*. As stated, the hypothesis may appear to be a bold claim. The aim of the *expected utility theorem* is to provide "modest conditions" (i.e. axioms) describing when the expected utility hypothesis holds, which can be evaluated directly and intuitively:

>
> "The axioms should not be too numerous, their system is to be as simple and transparent as possible, and each axiom should have an immediate intuitive meaning by which its appropriateness may be judged directly. In a situation like ours this last requirement is particularly vital, in spite of its vagueness: we want to make an intuitive concept amenable to mathematical treatment and to see as clearly as
> possible what hypotheses this requires." – *VNM 1953 § 3.5.2, p. 25*[[1]](#cite_note-VNM-1)
>
>
>
>

As such, claims that the expected utility hypothesis does not characterize rationality must reject one of the VNM axioms. A variety of [generalized expected utility](/wiki/Generalized_expected_utility "Generalized expected utility") theories have arisen, most of which drop or relax the independence axiom.

### Implications for ethics and moral philosophy

Main article: [Consequentialism](/wiki/Consequentialism "Consequentialism")
Because the theorem assumes nothing about the nature of the possible outcomes of the gambles, they could be morally significant events, for instance involving the life, death, sickness, or health of others. A von Neumann–Morgenstern rational agent is capable of acting with great concern for such events, sacrificing much personal wealth or well-being, and all of these actions will factor into the construction/definition of the agent's VNM-utility function. In other words, both what is naturally perceived as "personal gain", and what is naturally perceived as "altruism", are implicitly balanced in the VNM-utility function of a VNM-rational individual. Therefore, the full range of [agent-focussed to agent-neutral](/wiki/Consequentialism#Agent-focused_or_agent-neutral "Consequentialism") behaviors are possible with various VNM-utility functions[*[clarification needed](/wiki/Wikipedia:Please_clarify "Wikipedia:Please clarify")*].

If the utility of

N

{\displaystyle N}

![N](https://wikimedia.org/api/rest_v1/media/math/render/svg/f5e3890c981ae85503089652feb48b191b57aae3) is

p
M

{\displaystyle pM}

![pM](https://wikimedia.org/api/rest_v1/media/math/render/svg/1bf2ea65a040eba698ff24a06d8b5d55c5ee1bde), a von Neumann–Morgenstern rational agent must be indifferent between

1
N

{\displaystyle 1N}

![1N](https://wikimedia.org/api/rest_v1/media/math/render/svg/941c51e0a0b88e1fe1d1aaeb7e1af7b6bab8f272) and

p
M
+
(
1
−
p
)
0

{\displaystyle pM+(1-p)0}

![pM+(1-p)0](https://wikimedia.org/api/rest_v1/media/math/render/svg/0ad83e817154c2b6a7942aa258ccb9d4fd862203). An agent-focused von Neumann–Morgenstern rational agent therefore cannot favor more equal, or "fair", distributions of utility between its own possible future selves.

### Distinctness from other notions of utility

Some [utilitarian moral theories](/wiki/Utilitarianism#Average_v_total "Utilitarianism") are concerned with quantities called the "total utility" and "average utility" of collectives, and characterize morality in terms of favoring the utility or happiness of others with disregard for one's own. These notions can be related to, but are distinct from, VNM-utility:

* 1) VNM-utility is a *decision utility*:[[3]](#cite_note-KWS-3) it is that according to which one decides, and thus by definition cannot be something which one disregards.
* 2) VNM-utility is not canonically additive across multiple individuals (see Limitations), so "total VNM-utility" and "average VNM-utility" are not immediately meaningful (some sort of normalization assumption is required).

The term *E-utility* for "experience utility" has been coined[[3]](#cite_note-KWS-3) to refer to the types of "hedonistic" utility like that of [Bentham](/wiki/Jeremy_Bentham "Jeremy Bentham")'s [greatest happiness principle](/wiki/Greatest_happiness_principle "Greatest happiness principle"). Since morality affects decisions, a VNM-rational agent's morals will affect the definition of its own utility function (see above). Thus, the morality of a VNM-rational agent can be characterized by *correlation* of the agent's VNM-utility with the VNM-utility, E-utility, or "happiness" of others, among other means, but not by *disregard* for the agent's own VNM-utility, a contradiction in terms.

Limitations
-----------

### Nested gambling

Since if *L* and *M* are lotteries, then *pL* + (1 − *p*)*M* is simply "expanded out" and considered a lottery itself, the VNM formalism ignores what may be experienced as "nested gambling". This is related to the [Ellsberg problem](/wiki/Ellsberg_paradox "Ellsberg paradox") where people choose to avoid the perception of *risks about risks*. Von Neumann and Morgenstern recognized this limitation:

> "...concepts like a *specific utility of gambling* cannot be formulated free of contradiction on this level. This may seem to be a paradoxical assertion. But anybody who has seriously tried to axiomatize that elusive concept, will probably concur with it." – *VNM 1953 § 3.7.1, p. 28*.[[1]](#cite_note-VNM-1)
>
>

### Incomparability between agents

Since for any two VNM-agents *X* and *Y*, their VNM-utility functions *uX* and *uY* are only determined up to additive constants and multiplicative positive scalars, the theorem does not provide any canonical way to compare the two. Hence expressions like *uX*(*L*) + *uY*(*L*) and *uX*(*L*) − *uY*(*L*) are not canonically defined, nor are comparisons like *uX*(*L*) < *uY*(*L*) canonically true or false. In particular, the aforementioned "total VNM-utility" and "average VNM-utility" of a population are not canonically meaningful without normalization assumptions.

### Applicability to economics

The [expected utility hypothesis](/wiki/Expected_utility_hypothesis "Expected utility hypothesis") is shown to have limited predictive accuracy in a set of lab based empirical experiments, such as the [Allais paradox](/wiki/Allais_paradox "Allais paradox").
Which leads some people to interpret as evidence that

* humans are not always rational, or
* VNM-rationality is not an appropriate characterization of rationality, or
* some combination of both, or
* humans *do* behave VNM-rationally but the objective evaluation of *u* and the construction of *u* are [intractable](/wiki/Intractability_(complexity) "Intractability (complexity)") problems.

References and further reading
------------------------------

.mw-parser-output .reflist{font-size:90%;margin-bottom:0.5em;list-style-type:decimal}.mw-parser-output .reflist .references{font-size:100%;margin-bottom:0;list-style-type:inherit}.mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}.mw-parser-output .reflist-columns{margin-top:0.3em}.mw-parser-output .reflist-columns ol{margin-top:0}.mw-parser-output .reflist-columns li{page-break-inside:avoid;break-inside:avoid-column}.mw-parser-output .reflist-upper-alpha{list-style-type:upper-alpha}.mw-parser-output .reflist-upper-roman{list-style-type:upper-roman}.mw-parser-output .reflist-lower-alpha{list-style-type:lower-alpha}.mw-parser-output .reflist-lower-greek{list-style-type:lower-greek}.mw-parser-output .reflist-lower-roman{list-style-type:lower-roman}
1. ^ [***a***](#cite_ref-VNM_1-0) [***b***](#cite_ref-VNM_1-1) [***c***](#cite_ref-VNM_1-2) [***d***](#cite_ref-VNM_1-3) [Neumann, John von](/wiki/John_von_Neumann "John von Neumann") and [Morgenstern, Oskar](/wiki/Oskar_Morgenstern "Oskar Morgenstern"), *[Theory of Games and Economic Behavior](/wiki/Theory_of_Games_and_Economic_Behavior "Theory of Games and Economic Behavior")*. Princeton, NJ. Princeton University Press, 1953.
2. **[^](#cite_ref-2)** Peterson, Chapter 8.
3. ^ [***a***](#cite_ref-KWS_3-0) [***b***](#cite_ref-KWS_3-1) [***c***](#cite_ref-KWS_3-2) .mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\"""\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free a,.mw-parser-output .citation .cs1-lock-free a{background:url("//upload.wikimedia.org/wikipedia/commons/6/65/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited a,.mw-parser-output .id-lock-registration a,.mw-parser-output .citation .cs1-lock-limited a,.mw-parser-output .citation .cs1-lock-registration a{background:url("//upload.wikimedia.org/wikipedia/commons/d/d6/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription a,.mw-parser-output .citation .cs1-lock-subscription a{background:url("//upload.wikimedia.org/wikipedia/commons/a/aa/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("//upload.wikimedia.org/wikipedia/commons/4/4c/Wikisource-logo.svg")right 0.1em center/12px no-repeat}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:#d33}.mw-parser-output .cs1-visible-error{color:#d33}.mw-parser-output .cs1-maint{display:none;color:#3a3;margin-left:0.3em}.mw-parser-output .cs1-format{font-size:95%}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}Kahneman; Wakker; Sarin (1997). ["Back to Bentham? Explorations of Experienced Utility"](http://repub.eur.nl/pub/23011). *[Quarterly Journal of Economics](/wiki/Quarterly_Journal_of_Economics "Quarterly Journal of Economics")*. **112** (2): 375–406. [doi](/wiki/Doi_(identifier) "Doi (identifier)"):[10.1162/003355397555235](https://doi.org/10.1162%2F003355397555235). [hdl](/wiki/Hdl_(identifier) "Hdl (identifier)"):[1765/23011](https://hdl.handle.net/1765%2F23011).
4. ^ [***a***](#cite_ref-Kreps_4-0) [***b***](#cite_ref-Kreps_4-1) [Kreps, David M.](/wiki/David_M._Kreps "David M. Kreps") *Notes on the Theory of Choice*. Westview Press (May 12, 1988), chapters 2 and 5.
5. **[^](#cite_ref-nop_5-0)** Implicit in denoting indifference by equality are assertions like if

L
≺
M
=
N

{\displaystyle L\prec M=N}

![L\prec M = N](https://wikimedia.org/api/rest_v1/media/math/render/svg/41c97165bae2eb24ce9448c456fb22c4884b6a32) then

L
≺
N

{\displaystyle L\prec N}

![L\prec N](https://wikimedia.org/api/rest_v1/media/math/render/svg/8ff0ed9533e882a709e3bf19099ad043398cad26). To make such relations explicit in the axioms, Kreps (1988) chapter 2 denotes indifference by

∼

{\displaystyle \,\sim }

![{\displaystyle \,\sim }](https://wikimedia.org/api/rest_v1/media/math/render/svg/a2d29304f09810413194433497d0d8d2007a3639), so it may be surveyed in brief for intuitive meaning.
6. **[^](#cite_ref-6)** EconPort, "Von Neumann–Morgenstern Expected Utility Theory" <http://www.econport.org/content/handbook/decisions-uncertainty/basic/von.html>
7. **[^](#cite_ref-KeeneyRaiffa1993_7-0)** Keeney, Ralph L.; Raiffa, Howard (1993). *Decisions with Multiple Objectives*. [ISBN](/wiki/ISBN_(identifier) "ISBN (identifier)") [0-521-44185-4](/wiki/Special:BookSources/0-521-44185-4 "Special:BookSources/0-521-44185-4").
8. **[^](#cite_ref-8)** *Specimen theoriae novae de mensura sortis* or *Exposition of a New Theory on the Measurement of Risk*

.mw-parser-output .refbegin{font-size:90%;margin-bottom:0.5em}.mw-parser-output .refbegin-hanging-indents>ul{margin-left:0}.mw-parser-output .refbegin-hanging-indents>ul>li{margin-left:0;padding-left:3.2em;text-indent:-3.2em}.mw-parser-output .refbegin-hanging-indents ul,.mw-parser-output .refbegin-hanging-indents ul li{list-style:none}@media(max-width:720px){.mw-parser-output .refbegin-hanging-indents>ul>li{padding-left:1.6em;text-indent:-1.6em}}.mw-parser-output .refbegin-columns{margin-top:0.3em}.mw-parser-output .refbegin-columns ul{margin-top:0}.mw-parser-output .refbegin-columns li{page-break-inside:avoid;break-inside:avoid-column}
* [Nash, John F., Jr.](/wiki/John_Forbes_Nash_Jr. "John Forbes Nash Jr.") (1950). "The Bargaining Problem". *[Econometrica](/wiki/Econometrica "Econometrica")*. **18** (2): 155–162. [doi](/wiki/Doi_(identifier) "Doi (identifier)"):[10.2307/1907266](https://doi.org/10.2307%2F1907266). [JSTOR](/wiki/JSTOR_(identifier) "JSTOR (identifier)") [1907266](https://www.jstor.org/stable/1907266).
* Anand, Paul. *Foundations of Rational Choice Under Risk* Oxford, Oxford University Press. 1993 reprinted 1995, 2002
* [Fishburn, Peter C.](/wiki/Peter_C._Fishburn "Peter C. Fishburn") *Utility Theory for Decision Making*. Huntington, NY. Robert E. Krieger Publishing Co. 1970. [ISBN](/wiki/ISBN_(identifier) "ISBN (identifier)") [978-0-471-26060-8](/wiki/Special:BookSources/978-0-471-26060-8 "Special:BookSources/978-0-471-26060-8")
* [Sixto Rios](/wiki/Sixto_Rios "Sixto Rios") (1998) [Some problems and developments in decision science](http://www.mat.ucm.es/serv/revmat/vol11-1/vol11-1g.html), *Revista Matematica Complutense* 11(1):113–41.
* Peterson, Martin (2009). *An Introduction to Decision Theory (Cambridge Introductions to Philosophy)*. Cambridge: Cambridge University Press.

![](//en.wikipedia.org/wiki/Special:CentralAutoLogin/start?type=1x1)
Retrieved from "<https://en.wikipedia.org/w/index.php?title=Von_Neumann–Morgenstern_utility_theorem&oldid=1044421624>"
[Categories](/wiki/Help:Category "Help:Category"): * [Theorems](/wiki/Category:Theorems "Category:Theorems")
* [Game theory](/wiki/Category:Game_theory "Category:Game theory")
* [Utility](/wiki/Category:Utility "Category:Utility")
* [John von Neumann](/wiki/Category:John_von_Neumann "Category:John von Neumann")
Hidden categories: * [Articles with short description](/wiki/Category:Articles_with_short_description "Category:Articles with short description")
* [Short description is different from Wikidata](/wiki/Category:Short_description_is_different_from_Wikidata "Category:Short description is different from Wikidata")
* [Wikipedia articles needing clarification from March 2016](/wiki/Category:Wikipedia_articles_needing_clarification_from_March_2016 "Category:Wikipedia articles needing clarification from March 2016")

Von Neumann–Morgenstern utility theorem