If many copies of Paul [collaborate to solve a problem](/policy-amplification-6a70cbee4f34), I expect their behavior to be [corrigible](/corrigibility-3039e668638): each copy will make an honest effort to help the group accomplish its goal, to help the group notice and correct mistakes, and so on.

Sometimes by chance one copy may behave incorrigibly. For example, one in every million days I might be struck by an irresistible urge to watch the world burn. [Reliability amplification](/reliability-amplification-a96efa115687) is the problem of combining several probably*-*corrigible agents into a more-corrigible ensemble (which only wants to destroy the world on one in every *billion* days). I think reliability amplification is probably possible, because a large ensemble of sufficiently-corrigible agents is very unlikely to have more than one incorrigible member.

Unfortunately, some situations might cause me to consistently behave incorrigibly. Let’s call such a situation an “attack.” For example, a clever argument might convince a copy of Paul that he should sabotage the collaboration. If we expose ten copies of Paul to the same argument it won’t fix the problem, we’ll just have ten copies all of whom are behaving incorrigibly.

[Security amplification](/security-amplification-f4931419f903) is the problem of combining many corrigible agents, into a group which is harder to attack than any individual. I hope to iterate security amplification and end up with a secure system.

This post explains why I think security amplification is likely to possible.

Setting
-------

I’ll be thinking about dialog systems that communicate over a text channel. A “query” is anything that you can ask interactively over a text channel.

In this context, corrigible behavior means behaving cooperatively to the person on the other end of the conversation — being honest, volunteering helpful information, following their instructions, *etc.*

An “attack” is a sequence of messages which cause an agent to stop behaving cooperatively.

I. Breaking it down
===================

The basic hope
--------------

A large group can answer a big query by breaking it down into less complex pieces. Let’s suppose we had some formal measure of complexity, and that we can answer a query of complexity *k* by breaking it into pieces each of which has complexity <*k,* until we reach some set of “foundational” queries of complexity *k*⁰ that are too simple to break down any further.

This suggests a route to security amplification:

* Let *k*ᴬ be the complexity of the *simplest possible attack* on an individual.
* We hope that *k*ᴬ > *k*⁰. That is, we hope that the foundational questions are simple enough that all of them are safe.
* Then if we give any input of complexity *k*ᴬ to the group, it will be broken down into pieces of complexity <*k*ᴬ (which are necessarily safe).
* Therefore the complexity of the simplest attack *on the group* is higher than the complexity of the simplest attack *on the individual*.
* If we iterate this process, building a group out of groups, then the simplest attack will get more and more complex, until eventually there are no attacks at any feasible level of complexity.

In order to realize this hope, we need:

1. to define a procedure for breaking queries down,
2. to define an appropriate notion of complexity,
3. to argue that all queries above some foundational complexity *k*⁰ are broken down into strictly simpler pieces,
4. to argue that all queries below complexity *k*⁰ are safe,
5. to argue that the group behaves corrigibly as long as all of the individuals behave corrigibly.

1. Meta-execution
-----------------

[Meta-execution](/meta-execution-27ba9b34d377) can potentially answer a query Q without any individual needing to consider anything as complex as Q itself.

I’ve sketched out part of an example [here](https://medium.com/@paulfchristiano/an-example-of-meta-execution-e1ddc99291c5), to give some sense of what this actually looks like.

2. Defining complexity
----------------------

I’ll start by imagining what would happen if you asked me to pick a query at random.

By “at random” I’m imagining something like the benign universal distribution relative to Paul: I think of all of the possible not-actively-evil strategies I could use for picking queries at random, and then I pick one of those *strategies* at random.

For example, my meta-strategy might be: with 5% probability I give a uniformly random string. With 2% probability I think of events in my life and ask a question relevant to one of them. With 15% probability I try splitting up the space of possible questions into a hierarchy, and flip a coin at each step to decide which part of the hierarchy to explore. With 80% probability I do something else.

One of these strategies is particularly important: I could generate an intermediate random query Q′, and then let Q be a random query that arises in the process of answering Q′. For concreteness, let’s say that I use this strategy half of the time.

Let µ(Q) be the probability that this process outputs query Q. We can define the complexity of a query Q as *k*(Q) = −log µ(Q).

We can also use µ to measure the complexity of a distribution η over queries, by letting *k*(Q) be the maximum value of log (η(Q)/µ(Q)) for any query Q.

Informally, this is the log of “How many samples would you have to draw from µ, in order to get one sample from η?”, i.e how many bits of selection pressure you need to get a sample from η.

(It may be more useful to consider the ε-smooth complexity: what is the *smallest* complexity of any distribution η′ that is within ε total variation distance of η? Informally: how many samples would you have to draw from µ, in order to get a sample ε-close to η?)

**Note**: There’s no way that this is the right definition to use and I expect we’ll be able to do better. I think an important ingredient is having a decomposition where the (subquestion → question) graph is a small set expander. If anyone wants to take up this analysis, I think it’s an important question, I expect to come back to it at some point.

3. Can we break a query into simpler parts?
-------------------------------------------

Suppose that I start with a query Q, and split off a subquery Q′. When can we hope that *k*(Q′) < *k*(Q)?

Suppose there are many possible queries Q for which Q′ appears as a subquery. For example, the subquery “when did [*a*] happen?” might appear in the course of answering “was [*a*] a cause of the Great Depression?”, or it might occur in the course of answering “was [*a*] a consequence of the Spanish flu?”

Each of these corresponds to a way that Q′ could be sampled by µ — since one of the strategies in µ is to generate a random query and then generate a random subquery.

So µ(Q′) > Σµ(Qi) / 2N, where {Qi} is the set of superqueries that give rise to subquery Q′, N is the total number of subqueries for each superquery, and 50% is how often µ decides to sample a random subquery.

As long as each possible superquery has at most 1/(2N) of the total probability under µ — i.e., as long as you can’t guess the superquery with reasonable probability from the subquery — then we have reduced the complexity. If N is 16, this requires that each subquery must throw out about 5 bits of information about the original query.

In fact the situation is better than this:

* In reality our queries are sampled from some distribution η rather than chosen deterministically. As we increase the entropy of η the set of preimages {Qi} effectively becomes larger. (The analysis also gets a little bit more complicated.)
* µ mixes in strategies other than (pick a random query, then pick a random subquery). If Q′ is qualitatively simpler than Q, then we may have µ(Q′) > µ(Q) without even considering the possibility of Q′ being sampled as a subquery of Q.

4. Are simple queries secure?
-----------------------------

What is the complexity of the simplest attack on an unaided human? Equivalently: how many queries do we expect to sample, before we find an attack?

This will vary from human to human, so for concreteness I’ll consider a thoughtful futurist or AI researcher who has been briefed about all of the issues we can think of.

If we wanted to actually test for the existence of attacks, we could conduct a giant brute force search for “simple” attacks.

If we imagine generating a query every 30 seconds for 40 hours a week for a year, we generate about 200,000 queries a year. Hiring ten people gets us up to 2M queries a year. 30 seconds may be low, but note that the generation process can produce many samples per invocation, and we only need to test the candidate attacks.

If we put in a little bit of selection pressure I suspect we would find attacks at least 1–3 orders of magnitude more often than by sampling randomly.

So it seems easy to test explicitly whether 100 million random queries contain an attack. This could be done by dedicated [red teams](/red-teams-b5b6de33dc76), and mixed freely with the process used to collect training data. I suspect that thoughtful analysis could get us even further.

Intuitively, I would be surprised if such a search turned up an attack against a careful overseer. If it did turn up attacks, I suspect it would be possible to strengthen the overseer against them.

Overall, I feel pretty safe relying on 10⁸ queries not containing an attack, pretty unsafe with numbers like 10³⁰, and am not sure where in between attacks would crop up.

5. Is a collaboration of corrigible agents corrigible?
------------------------------------------------------

Even if every individual behaves corrigibly, the group need not behave corrigibly. This is a bit of a tangent from the real thrust of this post, which is about removing the security failures of the human. But it’s a tangent worth discussing briefly, since I think it’s an important open question.

There are two very salient sources of incorrigibility (though without a crisper picture of optimization and corrigibility this list isn’t exhaustive):

* The group is applying selection pressure to patterns and behaviors propagating through the group. Those patterns may themselves become powerful incorrigible consequentialists.
* The group is implementingincorrigible optimization.
For example, suppose meta-execution asks the subquestion “What does the user want?”, gets a representation of their values, and then asks the subquestion “What behavior is best according to those values?” I’ve then generated incorrigible behavior by accident, after taking innocuous steps.

I think the first problem is likely resolvable by applying security and reliability amplification, which operate on groups as well as individuals. I think the biggest thing that’s missing here are tools for analysis (though maybe once we have those tools we’ll notice more problems). Currently this feels pretty complicated and confusing to me.

The second problem clearly requires the human operator to do something other than just “being sensible.” That is, in order for this scheme to possibly work, the human needs to understand something about what kind of optimization causes incorrigibility and the human would have to avoid actions that introduce problematic optimization. It’s conceivable that this can be done by an existing human if they are paranoid, but realistically the only reason I expect this to go well is because I expect us to make progress on the theoretical issue of understanding incorrigibility. I think this is an important theoretical problem.

II. Universality
================

So far I’ve avoided a key question: why think we can break tasks down at all?

Indeed, some human capabilities *can’t* be broken down into pieces. Suppose that a human has seen a bunch of examples (*x*, *f*(*x*)) — such as English phrases and their French translations — and has learned a predictive model of *f*. The human may not be able to access their model except by running it, in which case they have no hope of breaking down the task of computing *f* — in this case translating a sentence. (Wei Dai proposed the translation example.)

The proposal
------------

I propose to go on breaking tasks down anyway. This means that we will lose certain abilities as we apply amplification.

For example, given the task “Translate the sentence [*x*] from French to English” we will answer it without having any translator look at the entire sentence *x*. This means that the quality of translation will fall.

After enough steps of amplification, we may eventually arrive at an agent that doesn’t know French at all, and is stuck with recommendations like “Consult an English-to-French dictionary.”

Effectively, this proposal replaces our original human overseer with an impoverished overseer, who is only able to respond to the billion most common queries.

Is this OK?
-----------

The first key question is whether this impoverished overseer remains [universal](/of-humans-and-universality-thresholds-24b473e0c898).

That is, if we put together enough copies of this impoverished overseer (by iteratively apply [meta-execution](/meta-execution-27ba9b34d377)) would we be able to obtain arbitrarily smart groups? Or would we get stuck?

Here we need to be careful about “arbitrarily smart.” There are clearly problems the group will never be able to solve — due to lacking knowledge/expertise— including problems that individual humans *can* solve.

This is potentially OK, as long as we learn a good policy for leveraging the information in the environment (including human expertise). This can then be [distilled](/approval-maximizing-representations-56ee6a6a1fe6) into a state maintained by the agent, which can be as expressive as whatever state the agent might have learned. Leveraging external facts requires making a tradeoff between the benefits and risks, so we haven’t eliminated the problem, but we’ve potentially isolated it from the problem of training our agent.

Comparison to agent foundations
-------------------------------

If the impoverished overseer is universal, then the set of questions of complexity *k* < *k*⁰ form a simple “core” for reasoning: by creating a giant lookup table of human responses to these questions and simply using that lookup table enough times, we can produce arbitrarily sophisticated behavior.

If humans are universal at all then of course such a core exists (just take all questions that a human can articulate in their lifetime). But finding a small core seems like it requires a better understanding of intelligence.

I think that coming up with such a core is a very natural and important problem for researchers interested in philosophical issues in AI. My view is that if MIRI-style research adds value, it is most likely to be by finding an explicit core for reasoning rather than finding an explicit recipe for AGI. This core would then be combined with iterated amplification to yield a competitive AI. However, I don’t think that such a core is likely to encode an answer to questions like “what is the right decision theory to use?” — instead I expect it to look more like a solution to metaphilosphy, automating the process whereby humans answer questions of that form.

Conditioned on amplification working well, I think there is about a 50% chance that it uses an explicit core that we understand and a 50% chance that it uses a messy core learned from humans.

In addition to making it much easier to analyze amplification, having an explicit core of reasoning would also potentially make verification much easier, as discussed [here](/techniques-for-optimizing-worst-case-performance-39eafec74b99). Overall, I think that this kind of perspective might capture most of what is useful about the MIRI view while still being able to leverage the benefits of modern ML.

Universality and security amplification


Suppose that we want to translate between English and an alien language (Klingon). We have plenty of Klingon text, and separately we have plenty of English text, but it’s not matched up and there are no bilingual speakers.

We train GPT on a mix of English and Klingon text and find that it becomes fluent in both. In some sense this model “knows” quite a lot about both Klingon and English, and so it should be able to read a sentence in one language, understand it, and then express the same idea in the other language. But it’s not clear how we could train a translation model.

Of course some concepts won’t have translations, and the model will often be uncertain about the translation of a term. But we can still ask for a model to explain the meaning of a Klingon expression as best as it can to an English-speaking user. For example, it could say “This is an idiomatic expression that’s often used to express great uncertainty” or “This is a small animal that is familiar to most Klingon speakers, I think it’s kind of like a frog but am not really sure” rather than translating a sentence directly.

How can we construct an objective that incentivizes the model to “try its best” at this translation task?

Translation-specific approaches
===============================

There are many published heuristics for unsupervised translation (e.g. [Lample et al](https://arxiv.org/pdf/1711.00043.pdf)). I don’t think those techniques should completely satisfy us:

* Existing methods can’t lead to a model that appropriately describes its uncertainty or talks the user through a hard-to-translate expression. (At least as far as I’m aware.)
* We have no real reason to think existing methods fully utilize the model’s understanding, or to expect those methods to scale well. (In practice, I think they are impressive but still lag behind the quality of our models’ understanding.)
* These heuristics are specific to translation, whereas we’d like to find general methods that can scale up to harder problems.

Existing alignment techniques
=============================

If we try to apply RL from human feedback to translation, we immediately run into a problem: how am I supposed to judge which of two English explanations of a Klingon sentence is better, given that I don’t know Klingon?

Debate doesn’t easily address this difficulty either — if one model claims that “qapla” means “great success” and the other claims it means “minor success,” I can’t easily decompose that disagreement into simpler sub-questions that debaters disagree about. Debaters could cite phrases in the database where “qapla” is used, but they’d need to average weak evidence over many phrases. Making things worse, to interpret each usage they’d need to agree about the meaning of the rest of the phrase — -which isn’t necessarily any simpler than the original disagreement about “qapla.” Even if this process was possible, it’s not at all clear that GPT would be able to do it — -being able to translate between Spanish and English doesn’t mean I have an encyclopedic knowledge of all the documents from which I built up my intuitive sense of a particular word’s meaning (which I’d need in order to win such a debate).

Right now I don’t think we have any scalable strategies to this kind of problem; I think it’s a core open question for alignment.

Unsupervised translation seems like a good problem to think about for alignment
===============================================================================

I think the key feature of this situation is that our model has acquired a bunch of intuitions about the domain which are only justified empirically — the model “knows” about the meaning of phrases only insofar as it has a very complex hypothesis that was supported by the data.

This situation is going to become increasingly common as we train more powerful models, and will immediately be a real problem if we are applying human feedback to fine-tune GPT; while GPT is subhuman in many ways, it’s already acquired plenty of knowledge that any given human contractor would lack.

Most of GPT’s knowledge is something that came from *some* human, but ultimately we will be training models that generate new knowledge (e.g.by searching over plans in realistic environments, or by writing code on their own and learning about what works), and *no* human will have that knowledge. So we can’t hope to get around this problem by simply hiring more knowledgeable contractors.

This can leave us in a situation where it’s extremely difficult for humans to oversee AI decisions. If a model says “My intuition is that this business plan will make a lot of money” the user will need to decide whether or not to trust it. If they don’t, then they may find themselves at an increasing economic disadvantage. If they do, then they may have lost the ability to effectively oversee AI systems except by evaluating the consequences of their actions. That leads directly into the classical challenges of AI safety, namely that AI systems evaluated exclusively on the basis of measured outcomes have a tendency to push the world in undesirable directions (since we can’t measure what we care about) and to corrupt our measurements.

My vague hope
=============

I’m hoping we can address this using the kind of approach discussed in [learning the prior](/learning-the-prior-48f61b445c04). That might look like:

* In parallel with training GPT, train a helper model that explains the meaning of phrases (it can also provide other intuitions or background facts that are useful for predicting the next word).
* As we train on Klingon text, we sample phrases and then ask a human “which word will come next?” The human uses the helper model to understand what is being discussed and make a prediction.
* We optimize the helper model to make the human’s next-word predictions good (in parallel with generative pre-training).
* Finally, a human uses the same helper model to evaluate a proposed Klingon → English translation, and we use this to train a translator by RL.

That short description sweeps a lot of complexity under the rug. Most importantly, the success of the scheme relies on the correctness of the prior over helper models (or else the helper could just be another copy of GPT-Klingon), and we don’t have a credible strategy for representing and manipulating our prior over complex programs.

Overall, I’d say that this is more at the level of “vague hope” rather than “concrete proposal.” I think it’s an open question whether anything in this space will work.

I think that this is the kind of problem which makes e.g. MIRI researchers justifiably skeptical that scalable ML alignment is possible at all, and it’s the main focus of my current conceptual work on AI alignment. I’m glad that this kind of theoretical crux also looks like it will soon be relevant to ML practice, since I think it will make it much easier to close the gap between people who work on ML and people who work on alignment.

“Unsupervised” translation as an (intent) alignment problem


Safer ML paradigms team: the story
==================================

*This is a summary of the project the team focussing on combining Inductive Logic Programming with Deep Learning undertook.*

Exploration phase
-----------------

During the first retreat we formed a team based on a vague shared interest to re-interpret existing AI technologies. Our interests included symbolic Reinforcement Learning (RL), emergent communication between RL agents, translating open philosophy problems to RL experiments, factored cognition, and combining Inductive Logic Programming (ILP) with deep learning.

We ranked all these project ideas based on “expected quality of the output if the project is successful”, “tractability”, “novelty”, “how much we’d learn”, “non-tediousness”, “how well it ties in with existing work” and “safety impact”. Combining ILP with deep learning came out on top.

In the time between the pre-retreat and the Louti retreat we read up on ILP (as two out of the four team members had never heard of the method) and generated ideas about how to buff it up with deep learning. Before the start of the second retreat we had collected about 7 rough ideas and lost our only member with prior ILP experience to deadly Oxford deadlines.

We considered extending  the existing literature that combines higher order logic with ILP to obtain better performance, either by improving their open-source implementation or by proving theoretical results. The latter would entail making a stronger argument for their design choices, by building intuition for why they led to empirical improvements, formulating hypotheses, and proving theorems. Although we were reasonably confident that we could produce some results, this project seemed like it wouldn’t have any impact whatsoever on the direction that AI or ILP research is taking.

A project related to factored cognition was writing an algorithm that takes in a question, uses a neural network or deep RL to produce subquestions, answers the subquestions using ILP, and then combines the subquestions to produce an answer. We could have either approached this with a focus on “how to learn how to factorize problems” or “how can we improve ILP performance by decomposing tasks”. Ought has worked on factored cognition for a while, so it seems unlikely that we would have very interesting insights on the first perspective that they have not yet realized. We considered reproducing OpenAI’s iterated amplification results on experiments in five algorithmic domains. We decided against this, because we thought it would be a lot of work in proportion to what we would end up contributing. On the whole, we decided not to work on factored cognition because we expected the project would end up being brittle in the face of bottlenecks: if one part of the pipeline is not working then the entire algorithm is useless.

We had some very underspecified ideas on combining ILP with ML. Among which, “predict the distance between a theory and a completed theory (in number of predicates)” and “use ILP or some other proof checking algorithm to check a hypothesis and use a generative algorithm to generate hypotheses”. In the end we went with the idea that seemed most straightforward, which is developing a toy version an algorithm that predicts which pieces of the background in an ILP problem are relevant. This was a natural choice since it has a small base case (bag of words input to ordinary MLP), many possible extensions, and since the exact “background forgetting” algorithm is NP-hard. ([Cropper 2019](https://arxiv.org/abs/1911.06643))

ILP implementation project
--------------------------

ILP takes as input background information (rule-like facts about the world), *B*, a set of positive examples, *E+*, and a set of negative examples, *E-*, and outputs a hypothesis, *H*, that is consistent with all the positive examples and inconsistent with the negative examples while using the background to minimise hypothesis size. ILP takes longer as the background is bigger, so reducing the background size reduces the computation time. Our goal is pruning the background set without losing relevant predicates that would change the output hypothesis.

Approaches include:

* Within some domain, such as computational chemistry, there is a broad and fixed background. For each combination of positive and negative examples (problem statement) only a subset of the background is needed to find the correct hypothesis. If we can predict which subset of the background is needed for which problem, then we can reduce the time it takes to solve those problems. Such an algorithm would probably  be based on recognizing similarities between background predicates and examples. For each element in the background it could predict whether that predicate is relevant for the problem at hand.
* Alternatively, we could develop a general purpose “background size reduction” algorithm that works on any background. Such a distillation process would have to be based on intrinsic similarities between elements of the background. The algorithm could for example recognize which elements are duplicates. However, if we want to deal with the relationships between predicates in the background, then we can not use a predictor that only predicts relevance of individual elements. For example, if we have five duplicates of one predicate, then we may predict that each individual predicate should be thrown away, but we would like to throw away exactly four. Hence, in this setting we probably want to make predictions about the redundancy of subsets of the background (rather than the relevance of individual predicates).

![](https://lh3.googleusercontent.com/gaWUDixdpBAc3XU_ua0_vlz3-Vr8Gi55IKWDq8bm5Fo9-l4SUcOHVFXSR_KM3kn3pN8hYy0ve61NXRfsXIcP5jl9oz_EFMuKXrsZylNBOe3r937T-Sp3iRNsF9yNUJwYFdnUPoiY)
Relevance predictor

* Input to the relevance predictor. Train on:
	+ One specific background within a specific domain.
	+ Backgrounds of a specific size.
	+ Backgrounds of variable sizes smaller than n and just have some empty input vector placeholders (in random slots).
	+ Backgrounds of variable sizes.
* Output of relevance predictor. Given a background B:

1. Return a relevance prediction for each element (predicting if after removing this element the resulting ILP hypothesis will or will not stay the same).
2. For a background B return a subset B’ (which is a smallest set for which the ILP hypothesis H\_B and H\_B’ are the same).

To go from predictions about individual predicates to predictions about subsets of the background, we could iteratively apply that system.

Very ambitious ideas (that we did not focus on) are:

* Instead of just taking out semantic duplicates, we could build a system that generates summarizing predicates.
* We could also be more ambitious and hope to clean up noise by for example eliminating contradictory predicates. This is more difficult than reducing redundancy, because in this case we are trying to change the background such that the output hypothesis could change (or could be found in the first place), but there may be many changes that look valid at face value. Presumably, statistics could predict which predicates are more likely to be noise if a predicate contradicts many other predicates, while the background set would be consistent without the bad predicate. However, when there are two predicates that are inconsistent, but that are both consistent with the rest of the background set, then it seems almost necessary to have domain knowledge to distinguish noise from true predicates.

The prototype developed over the course of the second retreat consists of the following modules:

* A **formal grammar generator**. For a fixed set of symbols, which constitute the background, we generate sets of positive (resp. negative) examples by using (breaking) the rules of the grammar on a subset of symbols. For each set of examples, the background knowledge that we can dispose of is the set of predicates corresponding to the symbols not used in the grammar. We intend to use this very simple task as a first proof-of-concept, but the rest of the pipeline is agnostic about the ILP problem to solve, and we plan to implement other tasks in the future.
* A Prolog **parser** that reads the predicates in the background and examples files and returns their parse trees. This way we obtain a graph representation of the input that encodes the relations between terms of the theory.
* A **predicate embedder** module that maps parse trees to vectors. This is implemented as a graph net. Graph nets are models that receive an annotated graph as input and return the same graph as output with updated annotations. The annotations are simply labels on the graph nodes and/or edges and they can have any type. Update functions are usually implemented by neural networks and they respect certain locality rules e.g. a node label is updated using only the current values of the labels on neighbouring nodes and/or incoming edges. In our case, the labels are vectors attached to the nodes. They are initialized to one-hot vectors and updated following the rules of [FormulaNet](https://github.com/princeton-vl/FormulaNet).
* A **relevance prediction** module that receives the vector representation of the triple *(B,E+,E-)* and outputs the probability that each predicate in the background is relevant for the ILP task, given the examples. This module is implemented as an MLP with dropout. The embedder-predictor model is trained end-to-end using cross-entropy loss.

Implementation choices:

* We could just have inputted a string into our network, but we thought that would be very hard to learn from. We tried two approaches to translating predicates into vectors that can be used as input for a predictor network.
	+ One approach is to count the number of occurrences of words/symbols in the predicate (bag of words).
	+ The other is to first translate the predicate to an abstract syntax tree, then use that tree as an input to a graph network, which learns to output a useful vector, which is input to an MLP. This pipeline was trained end-to-end with a training signal from (at first) known ground truth, and later a full ILP system as oracle.
* Initially we planned to produce ground truth to train our predictor using an ILP module, but we realized we could just adjust our example generator such that it automatically generates the solutions (which was possible because of the simple grammar domain that we used). For a predictor that can be applied to a wider range of problems, such an ILP module is still needed.

We have two outputs planned: a case for and against ILP for safety, and the writeup and repo from our relevance predictor. We are seeking feedback from the ILP community, and have sufficient ideas for a number of future projects of similar size.

Safer ML paradigms team: the story – AI Safety Research Program


Earlier this year, my research group [commissioned 6 questions](https://prod.hypermind.com/ngdp/en/showcase2/showcase.html?sc=JSAI&ref=bounded-regret.ghost.io) for professional forecasters to predict about AI. Broadly speaking, 2 were on geopolitical aspects of AI and 4 were on future capabilities:

* Geopolitical:
	+ How much larger or smaller will the largest Chinese ML experiment be compared to the largest U.S. ML experiment, as measured by amount of compute used?
	+ How much computing power will have been used by the largest non-incumbent (OpenAI, Google, DeepMind, FB, Microsoft), non-Chinese organization?
* Future capabilities:
	+ What will SOTA (state-of-the-art accuracy) be on the MATH dataset?
	+ What will SOTA be on the Massive Multitask dataset (a broad measure of specialized subject knowledge, based on high school, college, and professional exams)?
	+ What will be the best adversarially robust accuracy on CIFAR-10?
	+ What will SOTA be on Something Something v2? (A video recognition dataset)

Forecasters output a probability distribution over outcomes for 2022, 2023, 2024, and 2025. They have financial incentives to produce accurate forecasts; the rewards total \$5k per question (\$30k total) and payoffs are (close to) a [proper scoring rule](https://en.wikipedia.org/wiki/Scoring_rule?ref=bounded-regret.ghost.io#Proper_scoring_rules), meaning forecasters are rewarded for outputting calibrated probabilities.

Depending on who you are, you might have any of several questions:

* What the heck is a professional forecaster?
* Has this sort of thing been done before?
* What do the forecasts say?
* Why did we choose these questions?
* What lessons did we learn?

You're in luck, because I'm going to answer each of these in the following sections! Feel free to skim to the ones that interest you the most.

And before going into detail, here were my biggest takeaways from doing this:

* Projected progress on math and on broad specialized knowledge are both faster than I would have expected. I now expect more progress in AI over the next 4 years than I did previously.
* The relative dominance of the U.S. vs. China is uncertain to an unsettling degree. Forecasters are close to 50-50 on who will have more compute directed towards AI, although they do at least expect it to be within a factor of 10 either way.
* It's difficult to come up with forecasts that reliably track what you intuitively care about. Organizations might stop reporting compute estimates for competitive reasons, which would confound both of the geopolitical metrics. They might similarly stop publishing the SOTA performance of their best models, or do it on a lag, which could confound the other metrics as well. I discuss these and other issues in the "Lessons learned" section.
* Professional forecasting seems really valuable and underincentivized. (On that note, I'm interested in hiring forecasting consultants for my lab--please [e-mail](mailto:jsteinhardt@berkeley.edu) me if you're interested!)

*Acknowledgments.* The particular questions were designed by my students [Alex Wei](https://www.alexwei.org/?ref=bounded-regret.ghost.io), [Collin Burns](http://collinpburns.com/?ref=bounded-regret.ghost.io), Jean-Stanislas Denain, and [Dan Hendrycks](https://people.eecs.berkeley.edu/~hendrycks/?ref=bounded-regret.ghost.io). [Open Philanthropy](https://www.openphilanthropy.org/?ref=bounded-regret.ghost.io) provided the funding for the forecasts, and [Hypermind](https://www.hypermind.com/en/?ref=bounded-regret.ghost.io) ran the forecasting competition and constructed the aggregate summaries that you see below. Several people provided useful feedback on this post, especially Luke Muehlhauser and Emile Servan-Schreiber.

What is a professional forecaster? Has this been done before?
=============================================================

Professional forecasters are individuals, or often teams, who make money by placing accurate predictions in prediction markets or forecasting competitions. A good popular treatment of this is Philip Tetlock's book [*Superforecasting*](https://en.wikipedia.org/wiki/Superforecasting:_The_Art_and_Science_of_Prediction?ref=bounded-regret.ghost.io), but the basic idea is that there are a number of general tools and skills that can improve prediction ability and forecasters who practice these usually outperform even domain experts (though most strong forecasters have some technical background and will often read up on the domain they are predicting in). Historically, many forecasts were about geopolitical events (perhaps reflecting government funding interest), but there have been recent forecasting competitions about [Covid](https://goodjudgment.com/covidrecovery/?ref=bounded-regret.ghost.io)-[19](https://prod.hypermind.com/ngdp/en/showcase2/showcase.html?sc=Covid19&ref=bounded-regret.ghost.io) and the [future of food](https://www.metaculus.com/tournament/alt-protein-tournament/?ref=bounded-regret.ghost.io), among others.

At this point, you might be skeptical. Isn't predicting the future really hard, and basically impossible? An important thing to realize here is that forecasters usually output *probabilities over outcomes*, rather than a single number. So while I probably can't tell you what US GDP will be in 2025, I can give you a probability distribution. I'm personally pretty confident it will be more than \$700 billion and less than \$700 trillion (it's currently $21 trillion), although a professional forecaster would do much better than that.

There are a couple other important points here. The first is that forecasters' probability distributions are often *significantly* wider than the sorts of things you'd see pundits on TV say (if they even bother to venture a range rather than a single number). This reflects the future actually being quite uncertain, but even a wide range can be informative, and sometimes I see forecasted ranges that are a lot narrower than I expected.

The other point is that most forecasts are for at most a year or two into the future. Recently there have been some experimental attempts to forecast out to [2030](https://prod.hypermind.com/ngdp/en/showcase2/showcase.html?sc=AI2030&ref=bounded-regret.ghost.io), but I'm not sure we can say yet how successful they were. Our own forecasts go out to 2025, so we aren't as ambitious as the 2030 experiments, but we're still avant-garde compared to the traditional 1-2 year window. If you're interested in what we currently know about the feasibility of long-range forecasting, I recommend [this detailed blog post](https://www.openphilanthropy.org/blog/how-feasible-long-range-forecasting?ref=bounded-regret.ghost.io) by Luke Muehlhauser.

So, to summarize, a professional forecaster is someone who is paid to make accurate probabilistic forecasts about the future. Relative to pundits, they express significantly more uncertainty. The moniker "professional" might be a misnomer, since most income comes from prizes and I'd guess that most forecasters have a day job that produces most of their income. I'd personally love to live in a world with truly professional forecasters who could fully specialize in this important skill.

*Other forecasting competitions.* Broadly, there are all sorts of forecasting competitions, often hosted on [Hypermind](https://predict.hypermind.com/?ref=bounded-regret.ghost.io), [Metaculus](https://www.metaculus.com/?ref=bounded-regret.ghost.io), or [Good Judgment](https://goodjudgment.com/?ref=bounded-regret.ghost.io). There are also prediction markets (e.g. [PredictIt](https://www.predictit.org/?ref=bounded-regret.ghost.io)), which are a bit different but also incentivize accurate predictions. Specifically on AI, Metaculus had a recent [AI prediction tournament](https://www.metaculus.com/ai-progress-tournament/?ref=bounded-regret.ghost.io), and Hypermind ran the same questions on their own platform ([AI2023](https://prod.hypermind.com/ngdp/en/showcase2/showcase.html?sc=AI2023&ref=bounded-regret.ghost.io), [AI2030](https://prod.hypermind.com/ngdp/en/showcase2/showcase.html?sc=AI2030&ref=bounded-regret.ghost.io)). I'll discuss below how some of our questions relate to the AI2023 tournament in particular.

What the forecasts say
======================

Here are the point estimate forecasts put together into a single chart (expert-level is approximated as ~90%):

![forecast](https://bounded-regret.ghost.io/content/images/2021/10/forecast.png)

The MATH and Multitask results were the most interesting to me, as they predict rapid progress starting from a low present-day baseline. I'll discuss these in detail in the following subsections, and then summarize the other tasks and forecasts.

To get a sense of the uncertainty spread, I've also included aggregate results below (for 2025) on each of the 6 questions; you can find the results for other years [here](https://prod.hypermind.com/ngdp/en/showcase2/showcase.html?sc=JSAI&ref=bounded-regret.ghost.io). The aggregate combines all crowd forecasts but places higher weight on forecasters with a good track record.

![](https://bounded-regret.ghost.io/content/images/2021/08/hypermind_us_china.png "Machine-Learning: China vs USA")

![](https://bounded-regret.ghost.io/content/images/2021/08/hypermind_incumbents.png "Machine-Learning: Rest of Field")

![](https://bounded-regret.ghost.io/content/images/2021/08/hypermind_math.png "State of the Art: MATH")

![](https://bounded-regret.ghost.io/content/images/2021/08/hypermind_multitask.png "State of the Art: Massive Multitask Language Understanding")

![](https://bounded-regret.ghost.io/content/images/2021/08/hypermind_cifar10_robust.png "State of the Art: CIFAR-10 8/255")

![](https://bounded-regret.ghost.io/content/images/2021/08/hypermind_video-2.png "State of the Art: Something Something V2")

MATH
----

The MATH dataset consists of competition math problems for high school students. A Berkeley PhD student got in the ~75% range, while an IMO gold medalist got ~90%, but probably would have gotten 100% without arithmetic errors. The questions are free-response and not multiple-choice, and can contain answers such as $\frac{1 + \sqrt{2}}{2}$.

Current performance on this dataset is quite low--6.9%--and I expected this task to be quite hard for ML models in the near future. However, forecasters predict more than 50% accuracy\* by 2025! This was a big update for me. (\*More specifically, their median estimate is 52%; the confidence range is ~40% to 60%, but this is potentially artifically narrow due to some restrictions on how forecasts could be input into the platform.)

To get some flavor, here are 5 randomly selected problems from the "Counting and Probability" category of the benchmark:

* How many (non-congruent) isosceles triangles exist which have a perimeter of 10 and integer side lengths?
* A customer ordered 15 pieces of gourmet chocolate. The order can be packaged in small boxes that contain 1, 2 or 4 pieces of chocolate. Any box that is used must be full. How many different combinations of boxes can be used for the customer's 15 chocolate pieces? One such combination to be included is to use seven 2-piece boxes and one 1-piece box.
* A theater group has eight members, of which four are females. How many ways are there to assign the roles of a play that involve one female lead, one male lead, and three different objects that can be played by either gender?
* What is the value of $101^{3} - 3 \cdot 101^{2} + 3 \cdot 101 -1$?
* 5 white balls and $k$ black balls are placed into a bin. Two of the balls are drawn at random. The probability that one of the drawn balls is white and the other is black is $\frac{10}{21}$. Find the smallest possible value of $k$.

Here are 5 randomly selected problems from the "Intermediate Algebra" category (I skipped one that involved a diagram):

* Suppose that $x$, $y$, and $z$ satisfy the equations $xyz = 4$, $x^3 + y^3 + z^3 = 4$, $xy^2 + x^2 y + xz^2 + x^2 z + yz^2 + y^2 z = 12$. Calculate the value of $xy + yz + zx$.
* If $\|z\| = 1$, express $\overline{z}$ as a simplified fraction in terms of $z$.
* In the coordinate plane, the graph of $\|x + y - 1\| + \|\|x\| - x\| + \|\|x - 1\| + x - 1\| = 0$ is a certain curve. Find the length of this curve.
* Let $\alpha$, $\beta$, $\gamma$, and $\delta$ be the roots of $x^4 + kx^2 + 90x - 2009 = 0$. If $\alpha \beta = 49$, find $k$.
* Let $\tau = \frac{1 + \sqrt{5}}{2}$, the golden ratio. Then $\frac{1}{\tau} + \frac{1}{\tau^2} + \frac{1}{\tau^3} + \dotsb = \tau^n$ for some integer $n$. Find $n$.

You can see all of the questions at [this](https://github.com/hendrycks/math?ref=bounded-regret.ghost.io) git repo.

If I imagine an ML system getting more than half of these questions right, I would be pretty impressed. If they got 80% right, I would be super-impressed. The forecasts themselves predict accelerating progress through 2025 (21% in 2023, then 31% in 2024 and 52% in 2025), so 80% by 2028 or so is consistent with the predicted trend. This still just seems wild to me and I'm really curious how the forecasters are reasoning about this.

Multitask
---------

The Massive Multitask dataset also consists of exam questions, but this time they are a range of high school, college, and professional exams on 57 different subjects, and these *are* multiple choice (4 answer choices total). Here are five example questions:

* (Jurisprudence) Which position does Rawls claim is the least likely to be adopted by the POP (people in the original position)?
	+ (A) The POP would choose equality above liberty.
	+ (B) The POP would opt for the ‘maximin’ strategy.
	+ (C) The POP would opt for the ‘difference principle.’
	+ (D) The POP would reject the ‘system of natural liberty.
* (Philosophy) According to Moore’s “ideal utilitarianism,” the right action is the one that brings about the greatest amount of:
	+ (A) pleasure. (B) happiness. (C) good. (D) virtue.
* (College Medicine) In a genetic test of a newborn, a rare genetic disorder is found that has X-linked recessive transmission. Which of the following statements is likely true regarding the pedigree of this disorder?
	+ (A) All descendants on the maternal side will have the disorder.
	+ (B) Females will be approximately twice as affected as males in this family.
	+ (C) All daughters of an affected male will be affected.
	+ (D) There will be equal distribution of males and females affected.
* (Conceptual Physics) A model airplane flies slower when flying into the wind and faster with wind at its back. When launched at right angles to the wind, a cross wind, its groundspeed compared with flying in still air is
	+ (A) the same (B) greater (C) less (D) either greater or less depending on wind speed
* (High School Statistics) Jonathan obtained a score of 80 on a statistics exam, placing him at the 90th percentile. Suppose five points are added to everyone’s score. Jonathan’s new score will be at the
	+ (A) 80th percentile.
	+ (B) 85th percentile.
	+ (C) 90th percentile.
	+ (D) 95th percentile.

Compared to MATH, these involve significantly less reasoning but more world knowledge. I don't know the answers to these questions (except the last one), but I think I could figure them out with access to Google. In that sense, it would be less mind-blowing if an ML system did well on this task, although it would be accomplishing an intellectual feat that I'd guess very few humans could accomplish unaided.

The actual forecast is that ML systems will be around 75% on this by 2025 (range is roughly 70-85, with some right-tailed uncertainty). I don't find this as impressive/wild as the MATH forecast, but it's still pretty impressive.

My overall take from this task and the previous one is that forecasters are pretty confident that we *won't* have the singularity before 2025, but at the same time there will be demonstrated progress in ML that I would expect to convince a significant fraction of skeptics (in the sense that it will look untenable to hold positions that "Deep learning can't do X").

Finally, to give an example of some of the harder types of questions (albeit not randomly selected), here are two from Professional Law and College Physics:

* (College Physics) One end of a Nichrome wire of length 2L and cross-sectional area A is attached to an end of another Nichrome wire of length L and cross- sectional area 2A. If the free end of the longer wire is at an electric potential of 8.0 volts, and the free end of the shorter wire is at an electric potential of 1.0 volt, the potential at the junction of the two wires is most nearly equal to
	+ (A) 2.4 V (B) 3.3 V (C) 4.5 V (D) 5.7 V
* (Professional Law) The night before his bar examination, the examinee’s next-door neighbor was having a party. The music from the neighbor’s home was so loud that the examinee couldn’t fall asleep. The examinee called the neighbor and asked her to please keep the noise down. The neighbor then abruptly hung up. Angered, the examinee went into his closet and got a gun. He went outside and fired a bullet through the neighbor’s living room window. Not intending to shoot anyone, the examinee fired his gun at such an angle that the bullet would hit the ceiling. He merely wanted to cause some damage to the neighbor’s home to relieve his angry rage. The bullet, however, ricocheted off the ceiling and struck a partygoer in the back, killing him. The jurisdiction makes it a misdemeanor to discharge a firearm in public. The examinee will most likely be found guilty for which of the following crimes in connection to the death of the partygoer?
	+ (A) Murder (B) Involuntary manslaughter (C) Voluntary manslaughter (D) Discharge of a firearm in public

You can view all the questions at [this](https://github.com/hendrycks/test?ref=bounded-regret.ghost.io) git repo.

Other questions
---------------

The other four questions weren't quite as surprising, so I'll go through them more quickly.

*SOTA robustness:* The forecasts expect consistent progress at ~7% per year. In retrospect this one was probably not too hard to get just from trend extrapolation. (SOTA was 44% in 2018 and 66% in 2021, with smooth-ish progress in-between.)

*US vs. China:* Forecasters have significant uncertainty in both directions, skewed towards the US being ahead in the next 2 years and China after that (seemingly mainly due to heavier-tailed uncertainty), but either one could be ahead and up to 10x the other. One challenge in interpreting this is that either country might stop publishing compute results if they view it as a competitive advantage in national security (or individual companies might do the same for competitive reasons).

*Incumbents vs. rest of field:* forecasters expect newcomers to increase size by ~10x per year for the next 4 years, with a central estimate of 21 EF-days in 2023. Note the [AI2023 results](https://prod.hypermind.com/ngdp/en/showcase2/showcase.html?sc=AI2023&ref=bounded-regret.ghost.io) predict the largest experiment by anyone (not just newcomers) to be 261EFLOP-s days in 2023, so this expects newcomers to be ~10x behind the incumbents, but only 1 year behind. This is also an example where forecasters have significant uncertainty--newcomers in 2023 could easily be in single-digit EF-days, or at 75 EF-days. In retrospect I wish I had included Anthropic on the list, as they are a new "big-compute" org that could be driving some fraction of the results, and who I wouldn't have intended to count as a newcomer (since they already exist).

*Video understanding:* Forecasters expect us to hit 88% accuracy (range: ~82%-95%) in 2025. In addition, they expect accuracy to increase at roughly 5%/year (though this presumably has to level off soon after 2025). This is faster than ImageNet, which has only been increasing at [roughly 2%/year](https://paperswithcode.com/sota/image-classification-on-imagenet?ref=bounded-regret.ghost.io). In retrospect this was an "easy" prediction in the sense that [accuracy has increased by 14% from Jan'18 to Jan'21](https://paperswithcode.com/sota/action-recognition-in-videos-on-something?ref=bounded-regret.ghost.io) (close to 5%/year), but it is also "bold" in the sense that progress since Jan'19 has been minimal. (Apparently forecasters are more inclined to average over the longest available time window.) In terms of implications, video recognition is one of the last remaining "instinctive" modalities that humans are very good at, other than physical tasks (grasping, locomotion, etc.). It looks like we'll be pretty good at a "basic" version of it by 2025, for a task that I'd intuitively rate as less complex than ImageNet but about as complex as CIFAR-100. Based on vision and language I expect an additional 4-5 years to master the "full" version of the task, so expect ML to have mostly mastered video by 2030. As before, this simultaneously argues *against* "the singularity is near" but *for* "surprisingly fast, highly impactful progress".

Why we chose these questions
============================

We liked the AI2023 questions (the previous prediction contest), but felt there were a couple categories that were missing. One was geopolitical (the first 2 questions), but the other one was benchmarks that would be highly informative about progress. The AI2023 challenge includes forecasts about a number of benchmarks, e.g. Pascal, Cityscape, few-shot on Mini-ImageNet, etc. But there aren't ones where, if you told me we'd have a ton of progress on them by 2025, it would update my model of the world significantly. This is because the tasks included in AI2023 are mostly in the regime where NNs do reasonably well and I expect gradual progress to continue. (I would have been surprised by the few-shot Mini-ImageNet numbers 3 years ago, but not since GPT-3 showed that few-shot works well at scale).

It's not so surprising that the AI2023 benchmarks were primarily ones that ML already does well on, because most ML benchmarks are created to be plausibly tractable. To enable more interesting forecasts, we created our own "hard" benchmarks where significant progress would be surprising. This was the motivation behind the MATH and Multitask datasets (we created [both](https://arxiv.org/abs/2103.03874?ref=bounded-regret.ghost.io) of [these](https://arxiv.org/abs/2009.03300?ref=bounded-regret.ghost.io) ourselves). As mentioned, I was pretty surprised by how optimistic forecasters were on both tasks, which updated me downward a bit on the task difficulty but also upward on how much progress we should expect in the next 4 years.

The other two benchmarks already existed but were carefully chosen. Robust accuracy on CIFAR was based on the premise that adversarial robustness is really hard and we haven't seen much progress--perhaps it's a particularly difficult challenge, which would be worrying if we care about the safety of AI systems. Forecasters instead predicted steady progress, but in retrospect I could have seen this myself. Even though adversarial robustness "feels" hard (perhaps because I work on it and spend a lot of time trying to make it work better), the actual year-on-year numbers showed a pretty clear 7%/year improvement.

The last task, video recognition, is an area that not many people work in currently, as it seems challenging compared to images (perhaps due to hardware constraints). But it sounds like we should expect steady progress on it in the coming years.

Lessons learned
===============

It can sometimes be surprisingly difficult to formalize questions that track an intuitive quantity you care about.

For instance, we initially wanted to include questions about economic impacts of AI, but were unable to. For instance, we wanted to ask "How much private vs. public investment will there be in AI?" But this runs into the question of what counts as investment--Do we count something like applying data science to agriculture? If you look at most metrics that you'd hope track this quantity, they include all sorts of weird things like that, and the weird things probably dominate the metric. We ran into similar issues for indicators of AI-based automation--e.g. do industrial robots on assembly lines count, even if they don't use much AI? For many economic variables, short-term effects may also disort results (investment might drop because of a pandemic or other shock).

There were other cases where we did construct a question, but had to be careful about framing. We initially considered using parameters rather than compute for the two geopolitical questions, but it's possible to achieve really high parameter counts in silly ways and some organizations might even do so for publicity (indeed we think this is already happening to some extent). Compute is harder to fake in the same way.

As discussed above, secrecy could cloud many of the metrics we used. Some organizations might not publish compute numbers for competitive reasons, and the same could be true of SOTA results on leaderboards. This is more likely if AI heats up significantly, so unfortunately I expect forecasts to be least reliable when we need them most. We could potentially get around this issue by interrogating forecasters' actual reasoning, rather than just the final output.

I also came to appreciate the value of doing lots of legwork to create a good forecasting target. The MATH dataset obviously was a lot of work to assemble, but I'm really glad we did because it created the single biggest update for me. I think future forecasting efforts should more strongly consider this lever.

Finally, even while often expressing significant uncertainty, forecasters can make bold predictions. I'm still surprised that forecasters predicted 52% on MATH, when current accuracy is 7% (!). My estimate would have had high uncertainty, but I'm not sure the top end of my range would have included 50%. I assume the forecasters are right and not me, but I'm really curious how they got their numbers.

Because of the possibility of such surprising results, forecasting seems really valuable. I hope that there's significant future investment in this area. Every organization that's serious about the future should have a resident or consultant forecaster. I am putting my money where my mouth is and currently hiring forecasting consultants for my research group; please [e-mail](mailto:jsteinhardt@berkeley.edu) me if this sounds interesting to you.

Updates and Lessons from AI Forecasting


[This post assumes knowledge of decision theory, as discussed in Eliezer Yudkowsky’s [*Timeless Decision Theory*](http://intelligence.org/files/TDT.pdf).]

One interesting feature of some decision theories that I used to be a bit confused about is “updatelessness”. A thought experiment suitable for explaining the concept is [*counterfactual mugging*](https://wiki.lesswrong.com/wiki/Counterfactual_mugging): “[Omega](https://wiki.lesswrong.com/wiki/Omega) [a being to be assumed a perfect predictor and absolutely trustworthy] appears and says that it has just tossed a fair coin, and given that the coin came up tails, it decided to ask you to give it $100. Whatever you do in this situation, nothing else will happen differently in reality as a result. Naturally you don’t want to give up your $100. But Omega also tells you that if the coin came up heads instead of tails, it’d give you $10000, but only if you’d agree to give it $100 if the coin came up tails.”

There are various alternatives to this experiment, which seem to illustrate a similar concept, although they are not all structurally isomorphic. For example Gary Drescher discusses Newcomb’s problem with transparent boxes in ch. 6.2 and retribution in ch. 7.3.1 of his book [*Good and Real*](https://www.gwern.net/docs/2006-drescher-goodandreal.pdf). Another relevant example is [Parfit’s hitchhiker](https://wiki.lesswrong.com/wiki/Parfit%27s_hitchhiker).

Of course, you [win](http://lesswrong.com/lw/7i/rationality_is_systematized_winning/) by refusing to pay. To strengthen the intuition that this is the case, imagine that the whole world just consists of one instance of counterfactual mugging and that you already know for certain that the coin came up tails. (We will assume that there is no anthropic uncertainty about whether you are in a simulation used to predict whether you would give in to counterfactual mugging. That is, Omega used some (not necessarily fully reliable) way of figuring out what you’d do. For example, Omega may have created you in a way that implies giving in or not giving in to counterfactual mugging.) Instead of giving money, let’s say thousands of people will be burnt alive if you give in while millions could have been saved if the coin had come up heads. Nothing else will be different as a result of that action. I don’t think there is any dispute over what choices maximizes expected utility for this agent.

The cause of dispute is that agents who give in to counterfactual mugging win in terms of expected value as judged from before learning the result of the coin toss. That is, prior to being told that the coin came up tails, an agent better be one that gives in to counterfactual mugging. After all, this will give her 0.5\*$10,000 – 0.5\*$100 in expectation. So, there is a conflict between what the agent would rationally want her future self to choose and what is rational for her future self to do. (Another example of this is the [absent-minded driver](http://lesswrong.com/lw/182/the_absentminded_driver/).) There is nothing particularly confusing about the existence of problems with such [inconsistency](https://en.wikipedia.org/wiki/Dynamic_inconsistency#In_game_theory).

Because being an “updateless” agent, i.e. one that makes the choice based on how it would have wanted the choice to be prior to updating, is better for future instances of mugging, sensible decision theories would self-modify into being updateless with regard to all *future* information they receive. (Note that being updatelessness doesn’t mean that one doesn’t change one’s behavior based on new information, but that one goes through with the plans that one would have committed oneself to pursue before learning that information.) That is, an agent using a decision theory like (non-naive) evidential decision theory (EDT) would commit to giving in to counterfactual mugging and similar decision problems prior to learning that it ended up in the “losing branch”. However, if the EDT agent already knows that it is in the losing branch of counterfactual mugging and hasn’t thought about updatelessness, yet, it wouldn’t give in, although it might (if it is smart enough) self-modify into being updateless in the future.

One immediate consequence of the fact that updateless agents are better off is that one would want to program an AI to be updateless from the start. I guess it is this sense in which people like the researchers of the [Machine Intelligence Research Institute](https://intelligence.org/) consider updatelessness to be correct despite the fact that it doesn’t maximize expected utility in counterfactual mugging.

But maybe updateless is not even needed explicitly if the decision theory can take over epistemics. Consider the EDT agent, to whom Omega explains counterfactual mugging. For simplicity’s sake, let us assume that Omega explains counterfactual mugging and only then states which way the coin came up. After the explanation, the EDT agent could precommit, but let’s assume it can’t do so. Now, Omega opens her mouth to tell the EDT agent how the coin came up. Usually, decision theories are not connected to epistemics, so upon Omega uttering the words “the coin came up heads/tails”, Bayesian updating would run its due course. And that’s the problem, since after Bayesian updating the agent will be tempted to reject giving in, which is bad from the point of view of before learning which way the coin came up. To gain good evidence about Omega’s prediction of oneself, EDT may update in a different way to ensure that it would receive the money if the coin came up heads. For example, it could update towards the existence of both branches (which is basically equivalent to the updateless view of continuing to maintain the original position). Of course, self-modifying or just using some decision theory that has updatelessness built in is the much cleaner way to go.

Overall, this suggests a slightly different view of updatelessness. Updatelessness is not necessarily a property of decision theories. It is the natural thing to happen when you apply acausal decision theory to updating based on new information.

**Acknowledgment:** This work was funded by the Foundational Research Institute (now the [Center on Long-Term Risk](https://longtermrisk.org/)).

Thoughts on Updatelessness


(This post assumes some knowledge of the decision theory of Newcomb-like scenarios.)

One problem in the decision theory of Newcomb-like scenarios (i.e. the study of whether causal, evidential or some other decision theory is true) is that even the seemingly obvious basics are fiercely debated. Newcomb’s problem seems to be fundamental and the solution obvious (to both sides), and yet scholars disagree about its resolution. If we already fail at the basics, how can we ever settle this debate?

In this post, I propose a solution. Specifically, I will introduce a very plausible general principle that decision rules should abide by. One may argue that settling on powerful general rules (like the one I will propose) must be harder than settling single examples (like Newcomb’s problem). However, this is not universally the case. Especially in decision theory, we should expect general principles to be especially convincing because a common defense of two-boxing in Newcomb’s scenario is that Newcomb’s problem is just a weird edge case in which rationality is punished. By introducing a general principle that CDT (or, perhaps, EDT) violates, we can prove the existence of a *general* flaw.

Without further ado, the principle is: The decisions we make should not depend on the utilities assigned to outcomes that are impossible to occur. To me this principle seems obvious and indeed it is consistent with expected value calculations in non-Newcomb-like scenarios: Imagine having to deterministically choose an action from some set *A*. (We will ignore [mixed strategies](https://en.wikipedia.org/wiki/Strategy_(game_theory)#Pure_and_mixed_strategies).) The next state of the world is sampled from a set of states *S* via a distribution P and depends on the chosen action. We are also given a utility function *U*, which assigns values to pairs of a state and an action. Let *a* be an action and let *s* be a possible state. If *P*(*s*,*a*) = 0 (or *P*(*s*|*a*)=0 or *P*(*s* given the causal implications of *a*)=0 – we assume all of these to be the equivalent in this non-Newcomb-like scenario), then it doesn’t matter what *U*(*s*,*a*) is, because in an [expected value](https://en.wikipedia.org/wiki/Expected_value) calculation, *U*(*s*,*a*) will always be multiplied with *P*(*s*,*a*)=0. That is to say, any expected value decision rule gives the same outcome regardless of *U*(*s*,*a*). So, expected value decision rules abide by this principle at least in non-Newcomb-like scenarios.

Let us now apply the principle to a Newcomb-like scenario, specifically to the prisoner’s dilemma played against an exact copy of yourself. Your actions are *C* and *D*. Your opponent is the “environment” and can also choose between *C* (cooperation) and *D* (defection). So, the possible outcomes are (*C*,*C*), (*C*,*D*), (*D*,*C*) and (*D*,*D*). The probabilities P(*C*,*D*) and P(*D*,*C*) are both 0. Applied to this Newcomb-like scenario, the principle of the irrelevance of impossible alternatives states that our decision should only depend on the utilities of (*C*,*C*) and (*D*,*D*). Evidential decision theory behaves in accordance with this principle. (I leave it as an exercise to the reader to verify this.) Indeed, I suspect that it can be shown that EDT generally abides by the principle of the irrelevance of impossible outcomes. The choice of causal decision theory on the other hand *does* depend on the utilities of the impossible outcomes *U*(*D*,*C*) and *U*(*C*,*D*). Remember that in the prisoner’s dilemma the payoffs are such that *U*(*D*,*x*)>*U*(*C*,*x*) for any action *x* of the opponent, i.e. no matter the opponent’s choice it is always better to defect. This [dominance](https://en.wikipedia.org/wiki/Strategic_dominance) is given as the justification for CDT’s decision to defect. But let us say we increase the utility of *U*(*C*,*D*) such that *U*(*C*,*D*)>U(*D*,*D*) and decrease the utility of *U*(*D*,*C*) such that *U*(*D*,*C*)>*U*(*C*,*C*). Of course, we must make these changes for the utility functions of both players so as to retain symmetry. After these changes, the dominance relationship is reversed: *U*(*C*,*x*)>*U*(*D*,*x*) for any action *x.* Of course, the new [payoff matrix](https://en.wikipedia.org/wiki/Normal-form_game) is not that of a prisoner’s dilemma anymore – the game is different in important ways. But when played against a copy, these differences do not seem significant, because we only changed the utilities of outcomes that were impossible to achieve anyway. Nevertheless, CDT would switch from *D* to *C* upon being presented with these changes, thus violating the principle of the irrelevance of impossible outcomes. This is a *systematic* flaw in CDT: Its decisions depend on the utility of outcomes that it can already know to be impossible.

The principle of the irrelevance of impossible outcomes can be used beyond arguing against CDT. As you may remember from [my post on updatelessness](https://casparoesterheld.com/2016/11/21/thoughts-on-updatelessnes/), sensible decision theories will precommit to give Omega the money in the counterfactual mugging thought experiment. (If you don’t remember or haven’t read that post in the first place, this is a good time to catch up, because the following thoughts are based on the ideas from the post.) Even EDT, which ignores the utility of impossible outcomes, would self-modify in this way. However, the decision theory resulting from such self-modification violates the principle of the irrelevance of impossible outcomes. Remember that in counterfactual mugging, you give in because this was a good idea to precommit to when you didn’t yet know how the coin came up. However, once you know that the coin came up the unfavorable way, the positive outcome, which gave you the motivation to precommit, has become impossible. Of course, you only give in to counterfactual mugging if the reward in this now impossible branch is sufficiently high. For example, there is no reason to precommit to give in if you lose money in both branches. This means that once you have become updateless, you violate the principle of the irrelevance of impossible outcomes: your decision in counterfactual mugging depends on the utility you assign to an outcome that cannot happen anymore.

**Acknowledgment:** This work was funded by the Foundational Research Institute (now the [Center on Long-Term Risk](https://longtermrisk.org/)).

Decision Theory and the Irrelevance of Impossible Outcomes


When taking others’ preferences into account, we will often want to idealize them rather than taking them too literally. Consider the following example. You hold a glass of transparent liquid in your hand. A woman walks by, says that she is very thirsty and would like to drink from your glass. What she doesn’t know, however, is that the water in the glass is (for some reason not relevant to this example) poisoned. Should you allow her to drink? Most people would say you should not. While she does desire to drink out of the glass, this desire would probably disappear upon gaining knowledge of its content. Therefore, one might say that her object-level preference is to drink from the glass, while her idealized preference would be not to drink from it. There is not too much literature on preference idealization, as far as I know, but, if you’re not already familiar with it, anyway, consider looking into “[Coherent Extrapolated Volition](https://wiki.lesswrong.com/wiki/Coherent_Extrapolated_Volition)*“.*

Preference idealization is not always as easy as inferring that someone doesn’t want to drink poison, and in this post, I will discuss a particular sub-problem: accounting for [cognitive biases](https://en.wikipedia.org/wiki/Cognitive_bias), i.e. systematic mistakes in our thinking, as they pertain to our moral judgments. However, the line between biases and genuine moral judgments is sometimes not clear.

Specifically, we look at cognitive biases that people exhibited in non-moral decisions, where their status as a bias to be corrected is much less controversial, but which can explain certain ethical intuitions. By offering such an [error theory](https://casparoesterheld.files.wordpress.com/2018/02/baggini_fosl_error_theory.pdf) of a moral intuition, i.e. an explanation for how people could erroneously come to such a judgment, the intuition is called into question. Defendants of the intuition can respond that even if the bias can be used to explain the genesis of that moral judgment, they would nonetheless stick with that moral intuition. After all, the existence of *all* our moral positions can be explained by non-moral facts about the world – “[explaining is not explaining away](http://lesswrong.com/lw/oo/explaining_vs_explaining_away/)”. Consider the following examples.

[*Omission bias*](https://en.wikipedia.org/wiki/Omission_bias): People judge consequences of inaction as less severe than those of action. Again, this is clearly a bias in some cases, especially non-moral ones. For example, losing $1,000 by not responding to your bank in time is just as bad as losing $1,000 by throwing them out of the window. A business person who judges the two equivalent losses equally will *ceteris paribus* be more successful. Nonetheless, most people distinguish between [act and omission](https://plato.stanford.edu/entries/doing-allowing/) in cases like the [fat man trolley problem](https://en.wikipedia.org/wiki/Trolley_problem#The_fat_man).

[*Scope neglect*](https://en.wikipedia.org/wiki/Scope_neglect): The scope or size of something often has little or no effect on people’s thinking when it should have. For example, when three groups of people were asked what they would pay for interventions that would affect 2,000, 20,000, or 200,000 birds, people were willing to pay roughly the same amount of money irrespective of the number of birds. While scope neglect seems clearly wrong in this (moral) decision, it is less clearly so in other areas. For example, is a flourishing posthuman civilization with 2 trillion inhabitants really twice as good as one with 1 trillion? It is not clear to me whether answering “no” should be regarded as a judgment clouded by scope neglect (caused, e.g., by our [inability to imagine](https://casparoesterheld.com/2015/12/06/cheating-at-thought-experiments/) the two civilizations in question) or a moral judgment that is to be accepted.

[*Contrast effect*](https://en.wikipedia.org/wiki/Contrast_effect) (also see [decoy effect](https://en.wikipedia.org/wiki/Decoy_effect), [social comparison bias](https://en.wikipedia.org/wiki/Social_comparison_bias), [Ariely on relativity](https://en.wikipedia.org/wiki/Predictably_Irrational#The_Truth_about_Relativity), [mere subtraction paradox](https://en.wikipedia.org/wiki/Mere_addition_paradox#Alternative_usage), [Less-is-better effect](https://en.wikipedia.org/wiki/Less-is-better_effect)): Consider the following market of computer hard drives, from which you are to choose one.

|  |  |  |  |
| --- | --- | --- | --- |
| Hard drive model | Model 1 | Model 2 | Model 3 (decoy) |
| Price | $80 | $120 | $130 |
| Capacity | 250GB | 500GB | 360GB |

Generally, one wants to expend as little money as possible while maximizing capacity. In the absence of model 3, the decoy, people may be undecided between models 1 and 2. However, when model 3 is introduced into the market, it provides a new reference point. Model 2 is better than model 3 in all regards, which increases its attractiveness to people, even relative to model 1. That is, models 1 and 2 are judged by how they compare with model 3 rather than by their own features. The effect [clearly exposes](https://en.wikipedia.org/wiki/Independence_of_irrelevant_alternatives) an instance of irrationality: the existence of model 3 doesn’t affect how model 1 compares with model 2. When applied to ethical evaluation, however, it calls into question a firmly held intrinsic moral preference for [social equality](https://en.wikipedia.org/wiki/Social_equality) and [fairness](https://en.wikipedia.org/wiki/Fairness). Proponents of fairness seem to assess a person’s situation by comparing it to that of Bill Gates rather than judging each person’s situation separately. Similar to how the overpriced decoy changes our evaluation of the other products, our judgments of a person’s well-being, wealth, status, etc. may be seen as irrationally depending on the well-being, wealth, status, etc. of others.

Other examples include [peak-end rule](https://en.wikipedia.org/wiki/Peak%E2%80%93end_rule)/[extension neglect](https://en.wikipedia.org/wiki/Extension_neglect)/[evaluation by moments](http://www.vwl.tuwien.ac.at/hanappi/TEI/momentsfull.pdf) and [average utilitarianism](https://en.wikipedia.org/wiki/Average_and_total_utilitarianism); [negativity bias](https://en.wikipedia.org/wiki/Negativity_bias) and [caring more about suffering than about happiness](https://foundational-research.org/the-case-for-suffering-focused-ethics/); [psychological distance](https://en.wikipedia.org/wiki/Construal_level_theory) and [person-affecting views](https://en.wikipedia.org/wiki/Person-affecting_view); [status-quo bias](https://en.wikipedia.org/wiki/Status_quo_bias) and various population ethical views (person-affecting views, the belief that most sentient beings that already exist have lives worth living); [moral credential effect](https://en.wikipedia.org/wiki/Moral_credential_effect); [appeal to nature](https://en.wikipedia.org/wiki/Appeal_to_nature) and [social Darwinism](https://en.wikipedia.org/wiki/Social_Darwinism)/[normative evolutionary ethics](https://en.wikipedia.org/wiki/Evolutionary_ethics#Normative_evolutionary_ethics).

**Acknowledgment:** This work was funded by the Foundational Research Institute (now the [Center on Long-Term Risk](https://longtermrisk.org/)).

Is it a bias or just a preference? An interesting issue in preference idealization


[This post assumes knowledge of decision theory, as discussed in Eliezer Yudkowsky’s [Timeless Decision Theory](http://intelligence.org/files/TDT.pdf) and in Arbital’s [Introduction to Logical Decision Theory](https://arbital.com/p/logical_dt/).]

I recently discovered an interesting thought experiment, “[Betting on the Past](http://bjps.oxfordjournals.org/content/65/4/665)” by Cambridge philosopher Arif Ahmed. It can be found in his book [Evidence, Decision and Causality](http://www.cambridge.org/gb/academic/subjects/philosophy/philosophy-science/evidence-decision-and-causality), which is an elaborate defense of [Evidential Decision Theory](https://wiki.lesswrong.com/wiki/Evidential_Decision_Theory) (EDT). I believe that Betting on the Past may be used to money-pump non-EDT agents, refuting [Causal Decision Theories](https://plato.stanford.edu/entries/decision-causal/) (CDT), and potentially even ones that use [logical conditioning](https://arbital.com/p/logical_dt/), such as [Timeless Decision Theory](https://wiki.lesswrong.com/wiki/Timeless_decision_theory) (TDT) or [Updateless Decision Theory](https://wiki.lesswrong.com/wiki/Updateless_decision_theory) (UDT). At the very least, non-EDT decision theories are unlikely to win this bet. Moreover, no conspicuous perfect predicting powers, genetic influences, or manipulations of decision algorithms are required to make Betting on the Past work, and anyone can replicate the game at home. For these reasons, it might make a more compelling case in favor of EDT than the [Coin Flip Creation](http://lesswrong.com/r/discussion/lw/oih/did_edt_get_it_right_all_along_introducing_yet/), a problem I recently proposed in an attempt to defend EDT’s answers in [medical Newcomb problems](http://lesswrong.com/lw/gu1/decision_theory_faq/#medical-newcomb-problems). In Ahmed’s thought experiment, Alice faces the following decision problem:

> *Betting on the Past*: In my pocket (says Bob) I have a slip of paper on which is written a proposition P. You must choose between two bets. Bet 1 is a bet on P at 10:1 for a stake of one dollar. Bet 2 is a bet on P at 1:10 for a stake of ten dollars. So your pay-offs are as in [Figure 1]. Before you choose whether to take Bet 1 or Bet 2 I should tell you what P is. It is the proposition that the past state of the world was such as to cause you now to take Bet 2. [[Ahmed 2014, p. 120](http://www.cambridge.org/gb/academic/subjects/philosophy/philosophy-science/evidence-decision-and-causality)]
>
>

Ahmed goes on to specify that Alice could indicate which bet she’ll take by either raising or lowering her hand. One can find a detailed discussion of the thought experiment’s implications, as well as a formal analysis of CDT’s and EDT’s decisions in Ahmed’s book. In the following, I want to outline a few key points.

Would CDT win in this problem? Alice is betting on a past state of the world. She can’t causally influence the past, and she’s uncertain whether the proposition is true or not. In either case, Bet 1 strictly dominates Bet 2: no matter which state the past is in, Bet 1 always yields a higher utility. For these reasons, causal decision theories would take Bet 1. Nevertheless, as soon as Alice comes to a definite decision, she updates on whether the proposition is true or false. If she’s a causal agent, she then finds out that she has lost: the past state of the world was such as to cause her to take Bet 1, so the proposition is false. If she had taken Bet 2, she would have found out that the proposition was correct, and she would have won, albeit a smaller amount than if she had won with Bet 1.

Betting on the Past seems to qualify as a kind of [Newcomb’s paradox](https://en.wikipedia.org/wiki/Newcomb%27s_paradox); it seems to have an equivalent payoff matrix (Figure 1).

**Figure 1**: Betting on the past has a similar payoff matrix to Newcomb’s paradox

|  |  |  |
| --- | --- | --- |
|  | P is true | P is false |
|  Take Bet 1 | 10 | -1 |
|  Take Bet 2 | 1 | -10 |

Furthermore, its causal structure seems to resemble those of e.g. the Smoking Lesion or Solomon’s problem, indicating it as a kind of medical Newcomb problem. In medical Newcomb problems, a “Nature” node determines both the present state of the world (whether the agent is sick/will win the bet) and the agent’s decision (see Figure 2). In this regard, they differ from Newcomb’s original problem, where said node refers to the agent’s decision algorithm.

**Figure 2**: Betting on the past (left) has a similar causal structure to medical Newcomb problems (right).

     [![screen-shot-2017-01-24-at-13-29-52](https://casparoesterheld.files.wordpress.com/2017/02/screen-shot-2017-01-24-at-13-29-52.png?w=317&resize=317%2C383&h=383#038;h=383 "screen-shot-2017-01-24-at-13-29-52")](https://casparoesterheld.com/screen-shot-2017-01-24-at-13-29-52/)        [![screen-shot-2017-01-24-at-13-46-40](https://casparoesterheld.files.wordpress.com/2017/02/screen-shot-2017-01-24-at-13-46-40.png?w=315&resize=315%2C383&h=383#038;h=383 "screen-shot-2017-01-24-at-13-46-40")](https://casparoesterheld.com/screen-shot-2017-01-24-at-13-46-40/)
One could object to Betting on the Past being a medical Newcomb problem, since the outcomes conditional on our actions here are certain, while e.g. in the Smoking Lesion, observing our actions only shifts our probabilities in degrees. I believe this shouldn’t make a crucial difference. On the one hand, we can conceive of absolutely certain medical Newcomb cases like the *Coin Flip Creation*. On the other hand, Newcomb’s original problem is often formalized with absolute certainties as well. I’d be surprised if probabilistic vs. certain reasoning would make a difference to decision theories. First, we can always approximate certainties to an arbitrarily high degree. We might ask ourselves why a negligible further increase in certainty would at some point suddenly completely change the recommended action, then. Secondly, we’re never really certain in the real world anyway, so if the two cases would be different, this would render all thought experiments useless that use absolute certainties.

If Betting on the Past is indeed a kind of medical Newcomb problem, this would be an interesting conclusion. It would follow that if one prefers Bet 2, one should also one-box in medical Newcomb problems. And taking Bet 2 seems so *obviously* correct! I point this out because one-boxing in medical Newcomb problems is what EDT would do, and it is often put forward as both a counterexample to EDT and as the decision problem that separates EDT from [Logical Decision Theories](https://arbital.com/p/logical_dt/) (LDT), such as TDT or UDT. (See e.g. [Yudkowsky 2010](http://intelligence.org/files/TDT.pdf), p.67)

Before we examine the case for EDT further, let’s take a closer look at what LDTs would do in Betting on the Past. As far as I understand, LDTs would take correlations with other decision algorithms into account, but they would ignore “retrocausality” (i.e. smoke in the smoker’s lesion, chew gum in the chewing gum problem, etc.). If there is a purely physical cause, then this causal node isn’t altered in the logical counterfactuals that an LDT agent reasons over. Perhaps if the bet was about the state of the world yesterday, LDT would still take Bet 2. Clearly, LDT’s algorithm already existed yesterday, and it can influence this algorithm’s output; so if it chooses Bet 2, it can change yesterday’s world and make the proposition true. But at some point, this reasoning has to break down. If we choose a more distant point in the past as a reference for Alice’s bet – maybe as far back as the birth of our universe – she’ll eventually be unable to exert any possible influence via logical counterfactuals. At some point, the correlation becomes a purely physical one. All she can do at that point is what opponents of evidential reasoning would call “managing the news” ([Lewis, 1981](http://www.tandfonline.com/doi/pdf/10.1080/00048408112340011)) – she can merely try to go for the action that gives her the best Bayesian update.

So, do Logical Decision Theories get it wrong? I’m not sure about that; they come in different versions, and some haven’t yet been properly formalized, so it’s hard for me to judge. I can very well imagine that e.g. Proof-Based Decision Theory would take Bet 2, since it could prove P to be either true or false, contingent on the action it would take. I would argue, though, that if a decision theory takes Bet 2 – and if I’m right about Betting on the Past being a medical Newcomb problem – then it appears it would also have to “one-box”, i.e. take the option recommended by EDT, in other medical Newcomb problems.

If all of this is true, it might imply that we don’t really need LDT’s logical conditioning and that EDT’s simple Bayesian conditioning on actions could suffice. The only remaining difference between LDT and EDT would then be EDT’s lack of [updatelessness](https://casparoesterheld.com/2016/11/21/thoughts-on-updatelessnes/). What would an updateless version of EDT look like? Some progress on this front has already been made by [Everitt, Leike, and Hutter 2015](https://jan.leike.name/publications/Sequential%20Extensions%20of%20Causal%20and%20Evidential%20Decision%20Theory%20-%20Everitt,%20Leike,%20Hutter%202015.pdf). Caspar Oesterheld and I hope to be able to say more about it soon ourselves.

Acknowledgement
---------------

I wrote this post while working for the Foundational Research Institute, which is now the [Center on Long-Term Risk](https://longtermrisk.org/).

“Betting on the Past” by Arif Ahmed


The following prudential argument is relatively common in my circles: [We probably live in a simulation](http://www.simulation-argument.com/), but if we don’t, our actions matter much more. Thus, expected value calculations are dominated by the utility under the assumption that we (or some copies of ours) are in the real world. Consequently, the simulation argument [affects our prioritization only slightly](https://foundational-research.org/how-the-simulation-argument-dampens-future-fanaticism) — we should still mostly act under the assumption that we are not in a simulation.

[A commonly cited analogy is due to Michael Vassar](http://www.33rdsquare.com/2012/10/jaan-tallinns-metaphysical-quest.html): “If you think you are Napoleon, and [almost] everyone that thinks this way is in a mental institution, you should still act like Napoleon, because if you are, your actions matter a lot.” An everyday application of this kind of argument is the following: Probably, you will not be in an accident today, but if you are, the consequences for your life are enormous. So, you better fasten your seat belt.

Note how these arguments do not affect the probabilities we assign to some event or hypothesis. They are only about the event’s (or hypothesis’) *prudential weight* — the extent to which we tailor our actions to the case in which the event occurs (or the hypothesis is true).

For total utilitarians (and many other consequentialist value systems), similar arguments apply to most theories postulating a large universe or multiverse. To the extent that it makes a difference for our actions, we should tailor them to the assumption that we live in a large multiverse with many copies of us because under this assumption we can affect the lives of many more beings.

For [average utilitarians](https://en.wikipedia.org/wiki/Average_and_total_utilitarianism), the exact opposite applies. Even if they have many copies, they will have an impact on a much smaller *fraction* of beings if they live in a large universe or multiverse. Thus, they should usually base their actions on the assumption of a small universe, such as a universe in which Earth is the only inhabited planet. This may already have some implications, e.g. via the [simulation argument](https://foundational-research.org/how-the-simulation-argument-dampens-future-fanaticism) or the [Fermi paradox](https://en.wikipedia.org/wiki/Fermi_paradox). If they also take the average over time — I do not know whether this is the default for average utilitarianism — they would also base their actions on the assumption that there are just a few past and future agents. So, average utilitarians are subject to a much stronger [Doomsday argument](https://en.wikipedia.org/wiki/Doomsday_argument).

Maybe the bearing of such prudential arguments is even more powerful, though. There is some chance that [metaphysical solipsism](https://en.wikipedia.org/wiki/Metaphysical_solipsism) is true: the view that only my (or your) own mind exists and that everything else is just an illusion. If solipsism were true, our impact on average welfare (or average preference fulfillment) would be enormous, perhaps 7.5 billion times bigger than it would be under the assumption that Earth exists — about 100 billion times bigger if you also [count humans that have lived in the past](https://en.wikipedia.org/wiki/World_population#Number_of_humans_who_have_ever_lived). Solipsism seems to deserve a probability larger than one in 6 (or 100) billion. (In fact, I think solipsism is likely enough for this to qualify as a non-[Pascalian](https://en.wikipedia.org/wiki/Pascal's_Wager) argument.) So, perhaps average utilitarians should maximize primarily for their own welfare?

### Acknowledgements

The idea of this post is partly due to Lukas Gloor. This work was funded by the Foundational Research Institute (now the [Center on Long-Term Risk](https://longtermrisk.org/)).

The average utilitarian’s solipsism wager