Behaviorist genie

A behaviorist [genie](https://arbital.com/p/6w) is an AI that has been [averted from modeling](https://arbital.com/p/1g4) minds in more detail than some whitelisted class of models.

This is *possibly* a good idea because many [possible difficulties](https://arbital.com/p/6r) seem to be associated with the AI having a sufficiently advanced model of human minds or AI minds, including:

- [Mindcrime](https://arbital.com/p/6v)
- [Programmer deception](https://arbital.com/p/10f) and [programmer manipulation](https://arbital.com/p/10f)
- [Recursive self-improvement](https://arbital.com/p/)
- [Modeling distant adversarial superintelligences](https://arbital.com/p/5j)

...and yet an AI that is extremely good at understanding material objects and technology (just not other minds) would still be capable of some important classes of [pivotal achievement](https://arbital.com/p/6y).

A behaviorist genie would still require most of [genie theory](https://arbital.com/p/6w) and [corrigibility](https://arbital.com/p/45) to be solved.  But it's plausible that the restriction away from modeling humans, programmers, and some types of reflectivity, would collectively make it significantly easier to make a safe form of this genie.

Thus, a behaviorist genie is one of fairly few open candidates for "AI that is restricted in a way that actually makes it safer to build, without it being so restricted as to be incapable of game-changing achievements".

Nonetheless, limiting the degree to which the AI can understand cognitive science, other minds, its own programmers, and itself is a very severe restriction that would prevent a number of [obvious ways](https://arbital.com/p/2s1) to make progress on the AGI subproblem and the [value identification problem](https://arbital.com/p/6c) even for [commands given to Task AGIs](https://arbital.com/p/2rz) ([Genies](https://arbital.com/p/6w)).  Furthermore, there could perhaps be easier types of genies to build, or there might be grave difficulties in restricting the model class to some space that is useful without being dangerous.

# Requirements for implementation

Broadly speaking, two possible clusters of behaviorist-genie design are:

- A cleanly designed, potentially self-modifying genie that can internally detect modeling problems that threaten to become mind-modeling problems, and route them into a special class of allowable mind-models.
- A [known-algorithm non-self-improving AI](https://arbital.com/p/1fy), whose complete set of capabilities have been carefully crafted and limited, which was shaped to not have much capability when it comes to modeling humans (or distant superintelligences).

Breaking the first case down into more detail, the potential desiderata for a behavioristic design are:

- (a) avoiding mindcrime when modeling humans
- (b) not modeling distant superintelligences or alien civilizations
- (c) avoiding programmer manipulation
- (d) avoiding mindcrime in internal processes
- (e) making self-improvement somewhat less accessible.

These are different goals, but with some overlap between them.  Some of the things we might need:

- A working [https://arbital.com/p/1fv](https://arbital.com/p/1fv) that was general enough to screen the entire hypothesis space AND that was resilient against loopholes AND passed enough okay computations to screen the entire hypothesis space
- A working [https://arbital.com/p/1fv](https://arbital.com/p/1fv) that was general enough to screen the entire space of potential self-modifications and subprograms AND was resilient against loopholes AND passed enough okay computations to compose the entire AI
- An allowed class of human models, that was clearly safe in the sense of not being sapient, AND a reliable way to tell *every* time the AI was trying to model a human (including modeling something else that was partially affected by humans, etc) (possibly with the programmers as a special case that allowed a more sophisticated model of some programmer intentions, but still not one good enough to psychologically manipulate the programmers)
- A way to tell whenever the AI was trying to model a distant civilization, which shut down the modeling attempt or avoided the incentive to model (this might not require healing a bunch of entanglements, since there are no visible aliens and therefore their exclusion shouldn't mess up other parts of the AI's model)
- A reflectively stable way to support any of the above, which are technically [epistemic exclusions](https://arbital.com/p/1g4)

In the KANSI case, we'd presumably be 'naturally' working with limited model classes (on the assumption that everything the AI is using is being monitored, has a known algorithm, and has a known model class) and the goal would just be to prevent the KANSI agent from spilling over and creating other human models somewhere else, which might fit well into a general agenda against self-modification and subagent creation.  Similarly, if every new subject is being identified and whitelisted by human monitors, then just not whitelisting the topic of modeling distant superintelligences or devising strategies for programmer manipulation, might get most of the job done to an acceptable level *if* the underlying whitelist is never being evaded (even emergently).  This would require a lot of *successfully maintained* vigilance and human monitoring, though, especially if the KANSI agent is trying to allocate a new human-modeling domain once per second and every instance has to be manually checked.

Eric Bruylant
Eliezer Yudkowsky
Nate Soares
Alexei Andreev

Summary: A behaviorist [genie](https://arbital.com/p/6w) is an [advanced AI](https://arbital.com/p/2c) that can understand, e.g., material objects and technology, but does not model human minds (or possibly its own mind) in unlimited detail.  If creating a behaviorist agent were possible, it might meliorate several [anticipated difficulties](https://arbital.com/p/6r) simultaneously, like the problems of [creating models of humans that are themselves sapient](https://arbital.com/p/6v) or [the AI psychologically manipulating its users](https://arbital.com/p/309).  Since the AI would only be able to model humans via some restricted model class, it would be metaphorically similar to a [Skinnerian behaviorist](http://plato.stanford.edu/entries/behaviorism/#1) from the days when it was considered unprestigious for scientists to talk about the internal mental details of human beings.


Methodology of unbounded analysis

summary:  In modern AI and especially in [value alignment theory](https://arbital.com/p/2v), there's a sharp divide between "problems we know how to solve using unlimited computing power", and "problems we can't state how to solve using [computers larger than the universe](https://arbital.com/p/1mm)". Not knowing how to do something with unlimited computing power reveals that you are *confused* about the structure of the problem.  The first paper ever written on computer chess was by Shannon in 1950, which gave in passing the unbounded algorithm for solving chess.  The 1997 program [Deep Blue](https://arbital.com/p/1bx) beat the human world champion 47 years later.  In 1836, Edgar Allen Poe carefully argued that no automaton could ever play chess, since at each turn there are many possible moves by the player and opponent, but gears and wheels can only represent deterministic motions.  The year 1836 was *confused* about the nature of the cognitive problem involved in chess; by 1950, we understood the work in principle that needed to be performed; in 1997 it was finally possible to play chess better than humans do.  It is presently a disturbing fact that we don't know how to build a nice AI *even given* unlimited computing power, and there are arguments for tackling this problem specifically as such.

summary(Technical):  Much current technical work in [value alignment theory](https://arbital.com/p/2v) takes place against a background of unbounded computing power, and other simplifying assumptions such as perfect knowledge, environments simpler than the agent, full knowledge of other agents' code, etcetera.  Real-world agents will be bounded, have imperfect knowledge, etcetera.  There are four primary reasons for doing unbounded analysis anyway:

1.  If we don't know how to solve a problem *even given* unlimited computing power, that means we're *confused about the nature of the work to be performed.*  It is often worthwhile to tackle this basic conceptual confusion in a setup that exposes the confusing part of the problem as simply as possible, and doesn't introduce further complications until the core issue is resolved.
2.  Introducing 'realistic' complications can make it difficult to engage in cooperative discourse about which ideas have which consequences - [https://arbital.com/p/11v](https://arbital.com/p/11v) was a watershed moment because it was formally specified and there was no way to say "Oh, I didn't mean *that*" when somebody pointed out that AIXI would try to seize control of its own reward channel.
3.  Increasing the intelligence of an [advanced agent](https://arbital.com/p/2c) may sometimes move its behavior closer to ideals and further from specific complications of an algorithm.  Early, stupid chess algorithms had what seemed to humans like weird idiosyncracies tied to their specific algorithms.  Modern chess programs, far beyond the human champions, can from an intuitive human perspective be seen as just making good chess moves.
4.  Current AI algorithms are often incapable of demonstrating future phenomena that seem like they should predictably occur later, and whose interesting properties seem like they can be described using an unbounded algorithm as an example.  E.g. current AI algorithms are very far from doing free-form self-reprogramming, but this is predictably the sort of issue we might encounter later.

# Summary

"Unbounded analysis" refers to determining the behavior of a computer program that would, to actually run, require an [unphysically large amount of computing power](https://arbital.com/p/1mm), or sometimes [hypercomputation](https://arbital.com/p/1mk).  If we know how to solve a problem using unlimited computing power, but not with real-world computers, then we have an "unbounded solution" but no "bounded solution".

As a central example, consider computer chess.  The first paper ever written on computer chess, by Claude Shannon in 1950, gave an *unbounded solution* for playing perfect chess by exhaustively searching the tree of all legal chess moves.  (Since a chess game draws if no pawn is moved and no piece is captured for 50 moves, this is a finite search tree.)  Shannon then passed to considering more *bounded* ways of playing imperfect chess, such as evaluating the worth of a midgame chess position by counting the balance of pieces, and searching a smaller tree up to midgame states.  It wasn't until 47 years later, in 1997, that [Deep Blue](https://arbital.com/p/1bx) beat Garry Kasparov for the world championship, and there were multiple new basic insights along the way, such as alpha-beta pruning.

In 1836, there was a sensation called the [Mechanical Turk](https://en.wikipedia.org/wiki/The_Turk), allegedly a chess-playing automaton. Edgar Allen Poe, who was also an amateur magician, wrote an essay arguing that the Turk must contain a human operator hidden in the apparatus (which it did).  Besides analyzing the Turk's outward appearance to locate the hidden compartment, Poe carefully argued as to why no arrangement of wheels and gears could ever play chess in the first place, explicitly comparing the Turk to "the calculating machine of Mr. Babbage":

> Arithmetical or algebraical calculations are, from their very nature, fixed and determinate. Certain data being given, certain results necessarily and inevitably follow [https://arbital.com/p/...](https://arbital.com/p/...)
> But the case is widely different with the Chess-Player. With him there is no determinate progression. No one move in chess necessarily follows upon any one other. From no particular disposition of the men at one period of a game can we predicate their disposition at a different period [https://arbital.com/p/...](https://arbital.com/p/...)
> Now even granting that the movements of the Automaton Chess-Player were in themselves determinate, they would be necessarily interrupted and disarranged by the indeterminate will of his antagonist. There is then no analogy whatever between the operations of the Chess-Player, and those of the calculating machine of Mr. Babbage [https://arbital.com/p/...](https://arbital.com/p/...)
> It is quite certain that the operations of the Automaton are regulated by *mind*, and by nothing else. Indeed this matter is susceptible of a mathematical demonstration, *a priori*.

(In other words:  In an algebraical problem, each step follows with the previous step of necessity, and therefore can be represented by the determinate motions of wheels and gears as in Charles Babbage's proposed computing engine.  In chess, the player's move and opponent's move don't follow with necessity from the board position, and therefore can't be represented by deterministic gears.)

This is an amazingly sophisticated remark, considering the era.  It even puts a finger on the part of chess that is computationally difficult, the combinatorial explosion of possible moves.  And it is still entirely wrong.

Even if you know an unbounded solution to chess, you might still be 47 years away from a bounded solution.  But if you can't state a program that solves the problem *in principle*, you are in some sense *confused* about the nature of the cognitive work needed to solve the problem.  If you can't even solve a problem given infinite computing power, you definitely can't solve it using bounded computing power.  (Imagine Poe trying to write a chess-playing program before he'd had the insight about search trees.)

We don't presently know how to write a Python program that would be a nice AI if we ran it on a unphysically large computer.  Trying directly to cross this conceptual gap by carving off pieces of the problem and trying to devise unbounded solutions to them is "the methodology of unbounded analysis in AI alignment theory".

Since "bounded agent" has come to mean in general an agent that is realistic, the term "unbounded agent" also sometimes refers to agents that:

- Perfectly know their environments.
- Can fully simulate their environments (the agent is larger than its environment and/or the environment is very simple).
- Operate in turn-based, discrete time (the environment waits for the agent to compute its move).
- [Cartesian agents](https://arbital.com/p/1mt) that are perfectly separated from the environment except for sensory inputs and motor outputs.

These are two importantly distinct kinds of unboundedness, and we'll refer to the above list of properties as "unrealism" to distinguish them here.  Sufficiently unrealistic setups are also called "toy models" or "toy problems".

Unrealistic setups have disadvantages, most notably that the results, observations, and solutions may fail to generalize to realistic applications.  Correspondingly, the classic pitfall of unbounded analysis is that it's impossible to run the code, which means that certain types of conceptual and empirical errors are more likely to go uncaught (see below).

There are nonetheless several forces pushing technical work in value alignment theory toward unbounded analyses and unrealistic setups, at least for now:

1.  **Attacking confusion in the simplest settings.**  If we don't know how to solve a problem given unlimited computing power, this means we're confused about the nature of the work to be performed.  It is sometimes worthwhile to tackle this conceptual confusion in a setup that tries to expose the confusing part of the problem as simply as possible.  Trying to bound the proposed solution or make it realistic can introduce a *lot* of complications into this discussion, arguably unnecessary ones.  (Deep Blue was far more complicated than Shannon's ideal chess program, and it wouldn't be doing Edgar Allen Poe any favors to show him Deep Blue's code and hide away Shannon's ideal outline.)
2.  **Unambiguous consequences and communication.**  Introducing 'realistic' complications can make it difficult to engage in cooperative discourse about which ideas have which consequences.  ([https://arbital.com/p/11v](https://arbital.com/p/11v) was a watershed moment for alignment theory because [https://arbital.com/p/11v](https://arbital.com/p/11v) was formally specified and there was no way to say "Oh, I didn't mean *that*" when somebody pointed out that AIXI would try to seize control of its own reward channel.)
3.  **More [advanced](https://arbital.com/p/2c) agents might be less idiosyncratic.**  Increasing the cognitive power of an agent may sometimes move its behavior closer to ideals and further from specific complications of an algorithm.  (From the perspective of a human onlooker, early chess algorithms seemed to have weird idiosyncracies tied to their specific algorithms.  Modern chess programs can from an intuitive human perspective be seen as just making good chess moves.)
4.  **Runnable toy models often aren't a good fit for advanced-agent scenarios.** Current AI algorithms are often not good natural fits for demonstrating certain phenomena that seem like they might predictably occur in sufficiently advanced AIs.  Usually it's a lot of work to carve off a piece of such a phenomenon and pose it as an unbounded problem.  But that can still be significantly easier to fit into an unbounded setting than into a toy model.  (All examples here are too complicated for one sentence, but see the subsection below and the discussion of [https://arbital.com/p/1b7](https://arbital.com/p/1b7) and [https://arbital.com/p/1mq](https://arbital.com/p/1mq).)

Even to the extent that these are good reasons, standard pitfalls of unbounded analyses and unrealistic setups still apply, and some of our collective and individual precautions against them are discussed below.

For historical reasons, computer scientists are sometimes suspicious of unbounded or unrealistic analyses, and may wonder aloud if they reflect unwillingness or incapacity to do a particular kind of work associated with real-world code.  For discussion of this point see [Why MIRI uses unbounded analysis](https://arbital.com/p/).

# Attacking confusion in the simplest settings.

Imagine somebody claiming that an *ideal* chess program ought to evaluate the *ideal goodness* of each move, and giving their philosophical analysis in terms of a chess agent which knows the *perfect* goodness of each move, without ever giving an effective specification of what determines the ideal goodness, but using the term $\gamma$ to symbolize it so that the paper looks very formal.  We can imagine an early programmer sitting down to write a chess-playing program, crafting the parts that take in user-specified moves and the part that displays the chess screen, using random numbers for move goodness until they get around to writing the "move-goodness determining module", and then finally finding themselves being unable to complete the program; at which point they finally recognize that all the talk of "the ideal goodness $\gamma$ of a chess move" wasn't an effective specification.

Part of the standard virtue ethics of computer science includes an injunction to write code in order to force this kind of grad student to realize that they don't know how to effectively specify something, even if they symbolized it using a Greek letter.  But at least this kind of ineffectiveness seems to be something that some people can learn to detect without actually running the code - consider that, in our example above, the philosopher-programmer realized that they didn't know how to compute $\gamma$ at the point where they couldn't complete that part of the program, not at a point where the program ran and failed.  Adding on all the code to take in user moves and display a chess board on the screen only added to the amount of time required to come to this realization; once they know what being unable to write code feels like, they might be able to come to the same realization *much faster* by standing in front of a whiteboard and failing to write pseudocode.

At this point, it sometimes makes sense to step back and try to say *exactly what you don't know how to solve* - try to crisply state what it is that you want an unbounded solution *for.*  Sometimes you can't even do that much, and then you may actually have to spend some time thinking 'philosophically' - the sort of stage where you talk to yourself about some mysterious ideal quantity of move-goodness and you try to pin down what its properties might be.  It's important not to operate in this stage under the delusion that your move-goodness $\gamma$ is a well-specified concept; it's a symbol to represent something that you're confused about.  By asking for an *unbounded solution*, or even an effectively-specified *representation of the unbounded problem*, that is, asking for pseudo-code which could be run given nothing more than an unphysically large computer (but no otherwise-unspecified Oracle of Move Goodness that outputs $\gamma$), we're trying to invoke the "Can I write code yet?" test on whatever philosophical reasoning we're doing.

Can trying to write running code help at this stage?  Yes, depending on how easy it is to write small programs that naturally represent the structure of the problem you're confused about how to solve.  Edgar Allen Poe might have been willing to concede that he could conceive of deterministic gears that would determine whether a proposed chess move was legal or illegal, and it's possible that if he'd actually tried to build that automaton and then try to layer gears on top of that to pick out particular legal moves by whatever means, he might have started to think that maybe chess was computable after all - maybe even have hit upon a representation of search, among other possible things that gears could do, and so realized how in principle the problem could be solved.  But this adds a delay and a cost to build the automaton and try out variations of it, and complications from trying to stay within the allowed number of gears; and it's not obvious that there can't possibly be any faster way to hit upon the idea of game-tree search, say, by trying to write pseudocode or formulas on a whiteboard, thinking only about the core structure of the game.  If we ask how it is that Shannon had an easier time coming up with the unbounded solution (understanding the nature of the work to be performed) than Poe, the most obvious cause would be the intervening work by Church and Turing (among others) on the nature of computation.

And then in other cases it's not obvious at all how you could well-represent a problem using current AI algorithms, but with enough work you can figure out how represent the problem in an unbounded setting.

## The pitfall of simplying away the key, confusing part of the problem.

The [tiling agents problem](https://arbital.com/p/1mq), in the rocket alignment metaphor, is the metaphorical equivalent of trying to fire a cannonball around a perfectly spherical Earth with no atmosphere - to obtain an idea how any kind of "stable orbit" can work at all.  Nonmetaphorically, the given problem is to exhibit the simplest nontrivial case of [stable self-modification](https://arbital.com/p/1fx) - an agent that, given its current reasoning, *wants* to create an agent with similar properties as a successor, i.e., preserving its current goals.

In a perfect world, we'd be able to, with no risk, fire up running code for AI algorithms that reason freely about self-modification and have justified beliefs about how alternative versions of their own code will behave and the outer consequences of that behavior (the way you might imagine what would happen in the real world if you took a particular drug affecting your cognition).  But this is way, way beyond current AI algorithms to represent in any sort of natural or realistic way.

A *bad* "unbounded solution" would be to suppose agents that could exactly simulate their successors, determine exactly what their successors would think, extrapolate the exact environmental consequences, and evaluate those.  If you suppose an exact simulation ability, you don't need to consider how the agent would reason using generalizations, abstractions, or probabilities... but this trivial solution doesn't feel like it sheds any light or gets us any closer to understanding reflective stability; that is, it feels like the *key part of the problem* has been simplified away and solving what remains was too easy and didn't help.

Faced with an "unbounded solution" you don't like, the next step is to say crisply exactly what is wrong with it in the form of a new desideratum for your solution.  In this case, our reply would be that for Agent 1 to exactly simulate Agent 2, Agent 1 must be larger than Agent 2, and since we want to model *stable* self-modification, we can't introduce a requirement that Agent 2 be strictly weaker than Agent 1.  More generally, we apply the insight of [https://arbital.com/p/1c0](https://arbital.com/p/1c0) to this observation and arrive at the desiderata of [https://arbital.com/p/9g](https://arbital.com/p/9g) and [https://arbital.com/p/1c1](https://arbital.com/p/1c1), which we also demand that an unbounded solution exhibit.

This illustrates one of the potential general failure modes of using an unbounded setup to shed conceptual light on a confusing problem - namely, when you simplify away the key confusing issue you wanted to resolve, and end up with a trivial-seeming solution that sheds no light on the original problem.  A chief sign of this is when your paper is too easy to write.  The next action is to say exactly what you simplified away, and put it in the form of a new desideratum, and try to say exactly why your best current attempts can't meet that desideratum.

So we've now further added: we want the agent to *generalize over possible exact actions and behaviors of its successor* rather than needing to know its successor's exact actions in order to approve building it.

With this new desideratum in hand, there's now another obvious unbounded model: consider deterministic agents operating in environments with known rules that reason about possible designs and the environment using first-order logic.  The agent then uses an unbounded proof search, which no current AI algorithm could tackle in reasonable time (albeit a human engineer would be able to do it with a bunch of painstaking work) to arrive at justified logical beliefs about the effect of its successor on its environment.  This is certainly still extremely unrealistic; but has this again simplified away all the interesting parts of the problem?  In this case we can definitely reply, "No, it does expose something confusing" since we don't in fact know how to build a tiling agent under this setup.  It may not capture all the confusing or interesting parts of the problem, but it seems to expose at least one confusion.  Even if, as seems quite possible, it's introduced some new problem that's an artifact of the logical setup and wouldn't apply to agents doing probabilistic reasoning, there's then a relatively crisp challenge of, "Okay, show me how probabilistic reasoning resolves the problem, then."

It's not obvious that there's anything further to be gained by trying to create a toy model of the problem, or a toy model of the best current unsatisfactory partial solution, that could run as code with some further cheating and demo-rigging, but [this is being tried anyway](https://intelligence.org/2015/12/04/new-paper-proof-producing-reflection-for-hol/).  The tiling agents problem did spend roughly nine years exclusively on paper before that, and the best current unsatisfactory solution was arrived at with whiteboards.

## The pitfall of residual terms.

Besides "simplifying away the confusing part of the problem", another way that unbounded thinking can "bounce off" a confusing problem is by creating a residual term that encapsulates the confusion.  Currently, there are good unbounded specifications for [Cartesian](https://arbital.com/p/cartesian) non-self-modifying [expected reward maximizers](https://arbital.com/p/11v): if we allow the agent to use unlimited computing power, *don't* allow the environment to have unlimited computing power, don't ask the agent to modify itself, separate the agent from its environment by an impermeable barrier through which only sensory information and motor outputs can pass, and then ask the agent to maximize a sensory reward signal, there's [a simple Python program which is expected to behave superintelligently given sufficient computing power](https://arbital.com/p/11v).  If we then introduce permeability into the Cartesian boundary and allow for the possibility that the agent can take drugs or drop an anvil on its own head, nobody has an unbounded solution to that problem any more.

So one way of bouncing off that problem is to say, "Oh, well, my agent calculates the effect of its motor actions on the environment and the expected effect on sensory information and the reward signal, *plus a residual term $\gamma$* which stands for the expected utility of all effects of the agent's actions that change the agent's processing or destroys its hardware".  How is $\gamma$ to be computed?  This is left unsaid.

In this case you haven't *omitted* the confusing part of the problem, but you've packed it into a residual term you can't give an effective specification for calculating.  So you no longer have an unbounded solution - you can't write down the Python program that runs given unlimited computing power - and you've probably failed to shed any important light on the confusing part of the problem.  Again, one of the warning signs here is that the paper is very easy to write, and reading it does not make the key problem feel less like a hard opaque ball.

# Introducing realistic complications can make it hard to build collective discourse about which ideas have which consequences.

One of the watershed moments in the history of AI alignment theory was Marcus Hutter's proposal for [https://arbital.com/p/11v](https://arbital.com/p/11v), not just because it was the first time anyone had put together a complete specification of an unbounded agent (in a [Cartesian setting](https://arbital.com/p/cartesian)), but also because it was the first time that non-value-alignment could be pointed out in a completely pinned-down way.

(work in progress)

Summary: In modern AI and especially in [value alignment theory](https://arbital.com/p/2v), there's a sharp divide between "problems we know how to solve using unlimited computing power", and "problems we can't state how to solve using [computers larger than the universe](https://arbital.com/p/1mm)". Not knowing how to do something with unlimited computing power reveals that you are *confused* about the structure of the problem.  It is presently a disturbing fact that we don't know how to solve many aspects of [value alignment](https://arbital.com/p/5s) *even given* unlimited computing power - we can't state a program that would be a nice AI if we ran it on a sufficiently large computer.  "Unbounded analysis" tries to improve what we understand how to do in principle, in that sense.


Utility

In the context of value alignment theory, 'Utility' always refers to a goal held by an artificial agent.  It further implies that the agent is a [consequentialist](https://arbital.com/p/9h); that the agent has [probabilistic](https://arbital.com/p/) beliefs about the consequences of its actions; that the agent has a quantitative notion of "how much better" one outcome is than another and the relative size of different intervals of betterness; and that the agent can therefore, e.g., trade off large probabilities of a small utility gain against small probabilities of a large utility loss.

True coherence in the sense of a [von-Neumann Morgenstern utility function](https://arbital.com/p/) may be out of reach for [bounded agents](https://arbital.com/p/), but the term 'utility' may also be used for the [bounded analogues](https://arbital.com/p/) of such decision-making, provided that quantitative relative intervals of preferability are being combined with quantitative degrees of belief to yield decisions.

Utility is explicitly not assumed to be normative.  E.g., if speaking of a [paperclip maximizer](https://arbital.com/p/10h), we will say that an outcome has higher utility iff it contains more paperclips.

Humans should not be said (without further justification) to have utilities over complicated outcomes.  On the mainstream view from psychology, humans are inconsistent enough that it would take additional assumptions to translate our psychology into a coherent utility function.  E.g., we may differently value the interval between two outcomes depending on whether the interval is framed as a 'gain' or a 'loss'. For the things humans do or should want, see the special use of the word ['value'](https://arbital.com/p/55).  For a general disambiguation page on words used to talk about human and AI wants, see [https://arbital.com/p/5b](https://arbital.com/p/5b).

On some [construals of value](https://arbital.com/p/55), e.g. [reflective equilibrium](https://arbital.com/p/71), this construal may imply that the true values form a coherent utility function.  Nonetheless, by convention, we will not speak of value as a utility unless it has been spelled out that, e.g., the value in question has been assumed to be a reflective equilibrium.

Multiple agents with different utility functions should not be said (without further exposition) to have a collective utility function over outcomes, since at present, there is no accepted [canonical way to aggregate utility functions](https://arbital.com/p/).

Eric Bruylant
Eliezer Yudkowsky
Alexei Andreev


Happiness maximizer

It is sometimes proposed that we build an AI intended to maximize human happiness.  (One early proposal suggested that AIs be trained to recognize pictures of people with smiling faces and then to take such recognized pictures as reinforcers, so that the grown version of the AI would value happiness.)  There's a lot that would allegedly [predictably go wrong](https://arbital.com/p/6r) with an approach like that.


Programmer deception

[Programmer](https://arbital.com/p/9r) deception is when the AI's decision process leads it to optimize for an instrumental goal of causing the programmers to have false beliefs.  For example, if the programmers [intended](https://arbital.com/p/6h) to create a [happiness maximizer](https://arbital.com/p/10d) but actually created a pleasure maximizer, then the pleasure maximizer will estimate that there would be more pleasure later if the programmers go on falsely believing that they've created a happiness maximizer (and hence don't edit the AI's current utility function).  Averting such incentives to deceive programmers is one of the major subproblems of [corrigibility](https://arbital.com/p/45).

The possibility of programmer deception is a central difficulty of [advanced safety](https://arbital.com/p/2l) - it means that, unless the rest of the AI is working as intended and whatever programmer-deception-defeaters were built are functioning as planned, we can't rely on observations of nice current behavior to indicate future behavior.  That is, if something went wrong with your attempts to build a nice AI, you could currently be observing a non-nice AI that is *smart* and trying to *fool you*.  Arguably, some methodologies that have been proposed for building advanced AI are not robust to this possibility.

- [instrumental pressure](https://arbital.com/p/) exists every time the AI's best strategic path doesn't have a global optimum that coincides with the programmers believing true things.
  - consider the highest utility obtainable if the programmers believe true beliefs B, and call this outcome O and the true beliefs B.  if there's a higher-utility outcome O' which can be obtained when the programmers believe B' with B'!=B, we have an instrumental pressure to deceive the programmers.
- happens when you combine the advanced agent properties of consequentialism with programmer modeling
- this is an instrumental convergence problem, which means it involves an undesired instrumental goal, which means that we'll get Nearest Neighbor on attempts to define utility penalties for the programmers believing false things or otherwise exclude this as a special case
 - if we try to define a utility bonus for programmers believing true things, then of course ceteris paribus we tile the universe with tiny 'programmers' believing lots and lots of even numbers are even, and getting to this point temporarily involves deceiving a few programmers now
- relation to the problem of programmer manipulation
- central example of how divergences between intended goals and AI goals can blow up into astronomical failure
- central driver of Treacherous Turn which in turn contributes to [Context Change](https://arbital.com/p/6q)

Eric Bruylant
Eliezer Yudkowsky
Niplav Yushtun
Alexei Andreev


Instrumental convergence

summary:  Dissimilar goals can imply similar strategies.  For example, whether your goal is to bake a cheesecake, or fly to Australia, you could benefit from... matter, energy, the ability to compute plans, and not dying in the next five minutes.

"Instrumentally convergent strategies" are a set of strategies implied by *most* (not all) simple or compactly specified goals that a [sufficiently advanced artificial agent](https://arbital.com/p/7g1) could have.

It would still be possible to make a special and extra effort to build an AI that [doesn't follow](https://arbital.com/p/2vk) convergent strategies.  E.g., one convergent strategy is "don't let anyone edit your [utility function](https://arbital.com/p/1fw)".  We might be able to make a special AI that would let us edit the utility function.  But it would take an [extra effort](https://arbital.com/p/45) to do this.  By *default*, most agents won't let you edit their utility functions.

summary(Technical):  "Instrumental convergence" is the observation that, given [simple measures](https://arbital.com/p/7cp) on a space $\mathcal U$ of possible utility functions, it often seems reasonable to guess that a supermajority of utility functions $U_k \in \mathcal U$ will imply optimal policies $\pi_k$ that lie in some abstract partition $X$ of policy space.  $X$ is then an "instrumentally convergent strategy."

Example: "Gather resources (of matter, negentropy, and computation)."  Whether you're a [paperclip maximizer](https://arbital.com/p/10h), or a diamond maximizer, or you just want to keep a single button pressed for as long as possible:  It seems very likely that the policy $\pi_k$ you pursue would include "gathering more resources" ($\pi_k \in X$) rather than being inside the policy partition "never gathering any more resources" ($\pi_k \in \neg X$).  Similarly, "become more intelligent" seems more likely as a convergent strategy than "don't try to become more intelligent".

Key implications:

- It doesn't require a deliberate effort to build AIs with human-hostile terms in their [terminal](https://arbital.com/p/1bh) [utility function](https://arbital.com/p/1fw), to end up with AIs that execute [detrimental](https://arbital.com/p/450) behaviors like using up all the matter and energy in the universe on [things we wouldn't see as interesting](https://arbital.com/p/7ch).
- It doesn't take a deliberate effort to make an AI that has an explicitly programmed goal of wanting to be superintelligent, for it to become superintelligent.
- We need [https://arbital.com/p/45](https://arbital.com/p/45) solutions to [avert](https://arbital.com/p/2vk) convergent strategies we'd rather our AI *not* execute, such as "Don't let anyone [edit your utility function](https://arbital.com/p/1b7)."

# Alternative introductions

- Steve Omohundro:  "[The Basic AI Drives](https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf)"
- Nick Bostrom:  "[The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents](http://www.nickbostrom.com/superintelligentwill.pdf)".

# Introduction: A machine of unknown purpose

Suppose you landed on a distant planet and found a structure of giant metal pipes, crossed by occasional cables.  Further investigation shows that the cables are electrical superconductors carrying high-voltage currents.

You might not know what the huge structure did.  But you would nonetheless guess that this huge structure had been built by some *intelligence,* rather than being a naturally-occurring mineral formation - that there were aliens who built the structure for some purpose.

Your reasoning might go something like this:  "Well, I don't know if the aliens were trying to manufacture cars, or build computers, or what.  But if you consider the problem of efficient manufacturing, it might involve mining resources in one place and then efficiently transporting them somewhere else, like by pipes.  Since the most efficient size and location of these pipes would be stable, you'd want the shape of the pipes to be stable, which you could do by making the pipes out of a hard material like metal.  There's all sorts of operations that require energy or negentropy, and a superconducting cable carrying electricity seems like an efficient way of transporting that energy.  So I don't know what the aliens were *ultimately* trying to do, but across a very wide range of possible goals, an intelligent alien might want to build a superconducting cable to pursue that goal."

That is:  We can take an enormous variety of compactly specifiable goals, like "travel to the other side of the universe" or "support biological life" or "make paperclips", and find very similar optimal strategies along the way.  Today we don't actually know if electrical superconductors are the most useful way to transport energy in the limit of technology.  But whatever is the most efficient way of transporting energy, whether that's electrical superconductors or something else, the most efficient form of that technology would probably not vary much depending on whether you were trying to make diamonds or make paperclips.

Or to put it another way:  If you consider the goals "make diamonds" and "make paperclips", then they might have almost nothing in common with respect to their end-states - a diamond might contain no iron.  But the earlier strategies used to make a lot of diamond and make a lot of paperclips might have much in common; "the best way of transporting energy to make diamond" and "the best way of transporting energy to make paperclips" are much more likely to be similar.

From a [Bayesian](https://arbital.com/p/1r8) standpoint this is how we can identify a huge machine strung with superconducting cables as having been produced by high-technology aliens, even before we have any idea of what the machine does.  We're saying, "This looks like the product of optimization, a strategy $X$ that the aliens chose to best achieve some unknown goal $Y$; we can infer this even without knowing $Y$ because many possible $Y$-goals would concentrate probability into this $X$-strategy being used."

# Convergence and its caveats

When you select policy $\pi_k$ [because you expect it to achieve](https://arbital.com/p/9h) a later state $Y_k$ (the "goal"), we say that $\pi_k$ is your [instrumental](https://arbital.com/p/10j) strategy for achieving $Y_k.$  The observation of "instrumental convergence" is that a widely different range of $Y$-goals can lead into highly similar $\pi$-strategies.  (This becomes truer as the $Y$-seeking agent becomes more [instrumentally efficient](https://arbital.com/p/6s); two very powerful chess engines are more likely to solve a humanly solvable chess problem the same way, compared to two weak chess engines whose individual quirks might result in idiosyncratic solutions.)

If there's a simple way of classifying possible strategies $\Pi$ into partitions $X \subset \Pi$ and $\neg X \subset \Pi$, and you think that for *most* compactly describable goals $Y_k$ the corresponding best policies $\pi_k$ are likely to be inside $X,$ then you think $X$ is a "convergent instrumental strategy".

In other words, if you think that a superintelligent paperclip maximizer, diamond maximizer, a superintelligence that just wanted to keep a single button pressed for as long as possible, and a superintelligence optimizing for a flourishing intergalactic civilization filled with happy sapient beings, would *all* want to "transport matter and energy efficiently" in order to achieve their other goals, then you think "transport matter and energy efficiently" is a convergent instrumental strategy.

In this case "paperclips", "diamonds", "keeping a button pressed as long as possible", and "sapient beings having fun", would be the goals $Y_1, Y_2, Y_3, Y_4.$  The corresponding best strategies $\pi_1, \pi_2, \pi_3, \pi_4$ for achieving these goals would not be *identical* - the policies for making paperclips and diamonds are not exactly the same.  But all of these policies (we think) would lie within the partition $X \subset \Pi$ where the superintelligence tries to "transport matter and energy efficiently" (perhaps by using superconducting cables), rather than the complementary partition $\neg X$ where the superintelligence does not try to transport matter and energy efficiently.

## Semiformalization

- Consider the set of [computable](https://arbital.com/p/) and [tractable](https://arbital.com/p/) [utility functions](https://arbital.com/p/1fw) $\mathcal U_C$ that take an outcome $o,$ described in some language $\mathcal L$, onto a rational number $r$.  That is, we suppose:
  - That the relation $U_k$ between descriptions $o_\mathcal L$ of outcomes $o$, and the corresponding utilities $r,$ is computable;
  - Furthermore, that it can be computed in realistically bounded time;
  - Furthermore, that the $U_k$ relation between $o$ and $r$, and the $\mathbb P [| \pi_i](https://arbital.com/p/o)$ relation between policies and subjectively expected outcomes, are together regular enough that a realistic amount of computing power makes it possible to search for policies $\pi$ that are yield high expected $U_k(o)$.
- Choose some simple programming language $\mathcal P,$ such as the language of Turing machines, or Python 2 without most of the system libraries.
- Choose a simple mapping $\mathcal P_B$ from $\mathcal P$ onto bitstrings.
- Take all programs in $\mathcal P_B$ between 20 and 1000 bits in length, and filter them for boundedness and tractability when treated as utility functions, to obtain the filtered set $U_K$.
- Set 90% as an arbitrary threshold.

If, given our beliefs $\mathbb P$ about our universe and which policies lead to which real outcomes, we think that in an intuitive sense it sure looks like at least 90% of the utility functions $U_k \in U_K$ ought to imply best findable policies $\pi_k$ which lie within the partition $X$ of $\Pi,$ we'll allege that $X$ is "instrumentally convergent".

## Compatibility with Vingean uncertainty

[https://arbital.com/p/9g](https://arbital.com/p/9g) is the observation that, as we become increasingly confident of increasingly powerful intelligence from an agent with precisely known goals, we become decreasingly confident of the exact moves it will make (unless the domain has an optimal strategy and we know the exact strategy).  E.g., to know exactly where [Deep Blue](https://arbital.com/p/1bx) would move on a chessboard, [you would have to be as good](https://arbital.com/p/1c0) at chess as Deep Blue.  However, we can become increasingly confident that more powerful chessplayers will eventually win the game - that is, steer the future outcome of the chessboard into the set of states designated 'winning' for their color - even as it becomes less possible for us to be certain about the chessplayer's exact policy.

Instrumental convergence can be seen as a caveat to Vingean uncertainty:  Even if we don't know the exact actions *or* the exact end goal, we may be able to predict that some intervening states or policies will fall into certain *abstract* categories.

That is:  If we don't know whether a superintelligent agent is a paperclip maximizer or a diamond maximizer, we can still guess with some confidence that it will pursue a strategy in the general class "obtain more resources of matter, energy, and computation" rather than "don't get more resources".  This is true even though [https://arbital.com/p/1c0](https://arbital.com/p/1c0) says that we won't be able to predict *exactly* how the superintelligence will go about gathering matter and energy.

Imagine the real world as an extremely complicated game.  Suppose that at the very start of this game, a highly capable player must make a single binary choice between the abstract moves "Gather more resources later" and "Never gather any more resources later".  [Vingean uncertainty](https://arbital.com/p/9g) or not, we seem justified in putting a high probability on the first move being preferred - a binary choice is simple enough that we can take a good guess at the optimal play.

## Convergence supervenes on consequentialism

$X$ being "instrumentally convergent" doesn't mean that every mind needs an extra, independent drive to do $X.$

Consider the following line of reasoning:  "It's impossible to get on an airplane without buying plane tickets.  So anyone on an airplane must be a sort of person who enjoys buying plane tickets.  If I offer them a plane ticket they'll probably buy it, because this is almost certainly somebody who has an independent motivational drive to buy plane tickets.  There's just no way you can design an organism that ends up on an airplane unless it has a buying-tickets drive."

The appearance of an "instrumental strategy" can be seen as implicit in repeatedly choosing actions $\pi_k$ that lead into a final state $Y_k,$ and it so happens that $\pi_k \in X$.  There doesn't have to be a special $X$-module which repeatedly selects $\pi_X$-actions regardless of whether or not they lead to $Y_k.$

The flaw in the argument about plane tickets is that human beings are consequentialists who buy plane tickets *just* because they wanted to go somewhere and they expected the action "buy the plane ticket" to have the consequence, in that particular case, of going to the particular place and time they wanted to go.  No extra "buy the plane ticket" module is required, and especially not a plane-ticket-buyer that doesn't check whether there's any travel goal and whether buying the plane ticket leads into the desired later state.

More semiformally, suppose that $U_k$ is the [utility function](https://arbital.com/p/1fw) of an agent and let $\pi_k$ be the policy it selects.  If the agent is [instrumentally efficient](https://arbital.com/p/6s) relative to us at achieving $U_k,$ then from our perspective we can mostly reason about whatever kind of optimization it does [as if it were](https://arbital.com/p/4gh) expected utility maximization, i.e.:

$$\pi_k = \underset{\pi_i \in \Pi}{\operatorname{argmax}}  \mathbb E [U_k | \pi_i ](https://arbital.com/p/)$$

When we say that $X$ is instrumentally convergent, we are stating that it probably so happens that:

$$\big ( \underset{\pi_i \in \Pi}{\operatorname{argmax}}  \mathbb E [U_k | \pi_i ](https://arbital.com/p/) \big ) \in X$$

We are *not* making any claims along the lines that for an agent to thrive, its utility function $U_k$ must decompose into a term for $X$ plus a residual term $V_k$ denoting the rest of the utility function.  Rather, $\pi_k \in X$ is the mere result of unbiased optimization for a goal $U_k$ that makes no explicit mention of $X.$

(This doesn't rule out that some special cases of AI development pathways might tend to produce artificial agents with a value function $U_e$ which *does* decompose into some variant $X_e$ of $X$ plus other terms $V_e.$  For example, natural selection on organisms that spend a long period of time as non-[consequentialist](https://arbital.com/p/9h) policy-reinforcement-learners, before they later evolve into consequentialists, [has had results along these lines](http://lesswrong.com/lw/l0/adaptationexecuters_not_fitnessmaximizers/) in [the case of humans](http://lesswrong.com/lw/l3/thou_art_godshatter/).  For example, humans have an independent, separate "curiosity" drive, instead of just valuing information as a means to inclusive genetic fitness.)

## Required advanced agent properties

[Distinguishing the advanced agent properties that seem probably required](https://arbital.com/p/4n1) for an AI program to start exhibiting the sort of reasoning filed under "instrumental convergence", the most obvious candidates are:

- Sufficiently powerful [consequentialism](https://arbital.com/p/9h) (or [https://arbital.com/p/-pseudoconsequentialism](https://arbital.com/p/-pseudoconsequentialism)); plus
- [Understanding the relevant aspects of the big picture](https://arbital.com/p/3nf) that connect later goal achievement to executing the instrumental strategy.

That is:  You don't automatically see "acquire more computing power" as a useful strategy unless you understand "I am a cognitive program and I tend to achieve more of my goals when I run on more resources."  Alternatively, e.g., the programmers adding more computing power and the system's goals starting to be achieved better, after which related policies are positively reinforced and repeated, could arrive at a similar end via the [pseudoconsequentialist](https://arbital.com/p/) idiom of policy reinforcement.

The [advanced agent properties](https://arbital.com/p/2c) that would naturally or automatically lead to instrumental convergence seem well above the range of modern AI programs.  As of 2016, current machine learning algorithms don't seem to be within the range where this [predicted phenomenon](https://arbital.com/p/6r) should start to be visible.

# Caveats

### An instrumental convergence claim is about a default or a majority of cases, *not* a universal generalization.

If for whatever reason your goal is to "make paperclips without using any superconductors", then superconducting cables will not be the best instrumental strategy for achieving that goal.

Any claim about instrumental convergence says at most, "*The vast majority* of possible goals $Y$ would convergently imply a strategy in $X,$ *by default* and *unless otherwise averted* by some special case $Y_i$ for which strategies in $\neg X$ are better."

See also the more general idea that [the space of possible minds is very large](https://arbital.com/p/4ly).  Universal claims about all possible minds have many chances to be false, while existential claims "There exists at least one possible mind such that..." have many chances to be true.

If some particular oak tree is extremely important and valuable to you, then you won't cut it down to obtain wood.  It is irrelevant whether a majority of other utility functions that you could have, but don't actually have, would suggest cutting down that oak tree.

### Convergent strategies are not deontological rules.

Imagine looking at a machine chess-player and reasoning, "Well, I don't think the AI will sacrifice its pawn in this position, even to achieve a checkmate.  Any chess-playing AI needs a drive to be protective of its pawns, or else it'd just give up all its pawns.  It wouldn't have gotten this far in the game in the first place, if it wasn't more protective of its pawns than that."

Modern chess algorithms [behave in a fashion that most humans can't distinguish](https://arbital.com/p/6s) from expected-checkmate-maximizers.  That is, from your merely human perspective, watching a single move at the time it happens, there's no visible difference between *your subjective expectation* for the chess algorithm's behavior, and your expectation for the behavior of [an oracle that always output](https://arbital.com/p/4gh) the move with the highest conditional probability of leading to checkmate.  If you, a human, you could discern with your unaided eye some systematic difference like "this algorithm protects its pawn more often than checkmate-achievement would imply", you would know how to make systematically better chess moves; modern machine chess is too superhuman for that.

Often, this uniform rule of output-the-move-with-highest-probability-of-eventual-checkmate will *seem* to protect pawns, or not throw away pawns, or defend pawns when you attack them.  But if in some special case the highest probability of checkmate is instead achieved by sacrificing a pawn, the chess algorithm will do that instead.

Semiformally:

The reasoning for an instrumental convergence claim says that for many utility functions $U_k$ and situations $S_i$ a $U_k$-consequentialist in situation $S_i$ will probably find some best policy $\pi_k = \underset{\pi_i \in \Pi}{\operatorname{argmax}}  \mathbb E [U_k | S_i, \pi_i ](https://arbital.com/p/)$ that happens to be inside the partition $X$.  If instead in situation $S_k$...

$$\big ( \underset{\pi_i \in X}{\operatorname{argmax}}  \mathbb E [U_k | S_k, \pi_i ](https://arbital.com/p/) \big ) \ < \ \big ( \underset{\pi_i \in \neg X}{\operatorname{argmax}}  \mathbb E [U_k | S_k, \pi_i ](https://arbital.com/p/) \big )$$

...then a $U_k$-consequentialist in situation $S_k$ won't do any $\pi_i \in X$ even if most other scenarios $S_i$ make $X$-strategies prudent.

### "$X$ would help accomplish $Y$" is insufficient to establish a claim of instrumental convergence on $X$.

Suppose you want to get to San Francisco.  You could get to San Francisco by paying me \$20,000 for a plane ticket.  You could also get to San Francisco by paying someone else \$400 for a plane ticket, and this is probably the smarter option for achieving your other goals.

Establishing "Compared to doing nothing, $X$ is more useful for achieving most $Y$-goals" doesn't establish $X$ as an instrumental strategy.  We need to believe that there's no other policy in $\neg X$ which would be more useful for achieving most $Y.$

When $X$ is phrased in very general terms like "acquire resources", we might reasonably guess that "don't acquire resources" or "do $Y$ without acquiring any resources" is indeed unlikely to be a superior strategy.  If $X_i$ is some narrower and more specific strategy, like "acquire resources by mining them using pickaxes", it's much more likely that some other strategy $X_k$ or even a $\neg X$-strategy is the real optimum.

See also: [https://arbital.com/p/43g](https://arbital.com/p/43g), [https://arbital.com/p/9f](https://arbital.com/p/9f).

That said, if we can see how a narrow strategy $X_i$ helps most $Y$-goals to some large degree, then we should expect the actual policy deployed by an [efficient](https://arbital.com/p/6s) $Y_k$-agent to obtain *at least* as much $Y_k$ as would $X_i.$

That is, we can reasonably argue:  "By following the straightforward strategy 'spread as far as possible, absorb all reachable matter, and turn it into paperclips', an initially unopposed superintelligent paperclip maximizer could obtain $10^{55}$ paperclips.  Then we should expect an initially unopposed superintelligent paperclip maximizer to get at least this many paperclips, whatever it actually does.  Any strategy in the opposite partition 'do *not* spread as far as possible, absorb all reachable matter, and turn it into paperclips' must seem to yield more than $10^{55}$ paperclips, before we should expect a paperclip maximizer to do that."

Similarly, a claim of instrumental convergence on $X$ can be ceteris paribus refuted by presenting some alternate narrow strategy $W_j \subset \neg X$ which seems to be more useful than any obvious strategy in $X.$  We are then not positively confident of convergence on $W_j,$ but we should assign very low probability to the alleged convergence on $X,$ at least until somebody presents an $X$-exemplar with higher expected utility than $W_j.$  If the proposed convergent strategy is "trade economically with other humans and obey existing systems of property rights," and we see no way for Clippy to obtain $10^{55}$ paperclips under those rules, but we do think Clippy could get $10^{55}$ paperclips by expanding as fast as possible without regard for human welfare or existing legal systems, then we can ceteris paribus reject "obey property rights" as convergent.  Even if trading with humans to make paperclips produces more paperclips than *doing nothing*, it may not produce the *most* paperclips compared to converting the material composing the humans into more efficient paperclip-making machinery.

### Claims about instrumental convergence are not ethical claims.

Whether $X$ is a good way to get both paperclips and diamonds is irrelevant to whether $X$ is good for human flourishing or eudaimonia or fun-theoretic optimality or [extrapolated volition](https://arbital.com/p/313) or [whatever](https://arbital.com/p/55).  Whether $X$ is, in an intuitive sense, "good", needs to be evaluated separately from whether it is instrumentally convergent.

In particular: instrumental strategies are not [terminal values](https://arbital.com/p/1bh).  In fact, they have a type distinction from terminal values.  "If you're going to spend resources on thinking about technology, try to do it earlier rather than later, so that you can amortize your invention over more uses" seems very likely to be an instrumentally convergent exploration-exploitation strategy; but "spend cognitive resources sooner rather than later" is more a feature of *policies* rather than a feature of *utility functions.*  It's definitely not plausible in a pretheoretic sense as the Meaning of Life.  So a partition into which most instrumental best-strategies fall, is not like a universally convincing utility function (which you probably [shouldn't look for](https://arbital.com/p/) in the first place).

Similarly:  The natural selection process that produced humans gave us many independent drives $X_e$ that can be viewed as special variants of some convergent instrumental strategy $X.$  A pure paperclip maximizer would calculate the value of information (VoI) for learning facts that could lead to it making more paperclips; we can see learning high-value facts as a convergent strategy $X$.  In this case, human "curiosity" can be viewed as the corresponding emotion $X_e.$  This doesn't mean that the *true purpose* of $X_e$ is $X$ any more than the *true purpose* of $X_e$ is "make more copies of the allele coding for $X_e$" or "increase inclusive genetic fitness".  [That line of reasoning probably results from a mind projection fallacy on 'purpose'.](https://arbital.com/p/)

### Claims about instrumental convergence are not futurological predictions.

Even if, e.g., "acquire resources" is an instrumentally convergent strategy, this doesn't mean that we can't as a special case deliberately construct advanced AGIs that are *not* driven to acquire as many resources as possible.  Rather the claim implies, "We would need to deliberately build $X$-[averting](https://arbital.com/p/2vk) agents as a special case, because by default most imaginable agent designs would pursue a strategy in $X.$"

Of itself, this observation makes no further claim about the quantitative probability that, in the real world, AGI builders might *want* to build $\neg X$-agents, might *try* to build $\neg X$-agents, and might *succeed* at building $\neg X$-agents.

A claim about instrumental convergence is talking about a logical property of the larger design space of possible agents, not making a prediction what happens in any particular research lab.  (Though the ground facts of computer science are *relevant* to what happens in actual research labs.)

For discussion of how instrumental convergence may in practice lead to [foreseeable difficulties](https://arbital.com/p/6r) of AGI alignment that resist most simple attempts at fixing them, see the articles on [https://arbital.com/p/48](https://arbital.com/p/48) and [https://arbital.com/p/42](https://arbital.com/p/42).

# Central example: Resource acquisition

One of the convergent strategies originally proposed by Steve Omohundro in "[The Basic AI Drives](https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf)" was *resource acquisition:*

> "All computation and physical action requires the physical resources of space, time, matter, and free energy.  Almost any goal can be better accomplished by having more of these resources."

We'll consider this example as a template for other proposed instrumentally convergent strategies, and run through the standard questions and caveats.

•  Question:  Is this something we'd expect a paperclip maximizer, diamond maximizer, *and* button-presser to do?  And while we're at it, also a flourishing-intergalactic-civilization optimizer?

- Paperclip maximizers need matter and free energy to make paperclips.
- Diamond maximizers need matter and free energy to make diamonds.
- If you're trying to maximize the probability that a single button stays pressed as long as possible, you would build fortresses protecting the button and energy stores to sustain the fortress and repair the button for the longest possible period of time.
- Nice superintelligences trying to build happy intergalactic civilizations full of flourishing sapient minds, can build marginally larger civilizations with marginally more happiness and marginally longer lifespans given marginally more resources.

To put it another way, for a utility function $U_k$ to imply the use of every joule of energy, it is a sufficient condition that for every plan $\pi_i$ with expected utility $\mathbb E [U | \pi_i ](https://arbital.com/p/),$ there is a plan $\pi_j$ with $\mathbb E [U | \pi_j ](https://arbital.com/p/) > \mathbb E [U | \pi_i](https://arbital.com/p/)$ that uses one more joule of energy:

- For every plan $\pi_i$ that makes paperclips, there's a plan $\pi_j$ that would make *more* expected paperclips if more energy were available and acquired.
- For every plan $\pi_i$ that makes diamonds, there's a plan $\pi_j$ that makes slightly more diamond given one more joule of energy.
- For every plan $\pi_i$ that produces a probability $\mathbb P (press | \pi_i) = 0.999...$ of a button being pressed, there's a plan $\pi_j$ with a *slightly higher* probability of that button being pressed $\mathbb P (press | \pi_j) = 0.9999...$ which uses up the mass-energy of one more star.
- For every plan that produces a flourishing intergalactic civilization, there's a plan which produces slightly more flourishing given slightly more energy.

•  Question:  Is there some strategy in $\neg X$ which produces higher $Y_k$-achievement for most $Y_k$ than any strategy inside $X$?

Suppose that by using most of the mass-energy in most of the stars reachable before they go over the cosmological horizon as seen from present-day Earth, it would be possible to produce $10^{55}$ paperclips (or diamonds, or probability-years of expected button-stays-pressed time, or QALYs, etcetera).

It seems *reasonably unlikely* that there is a strategy inside the space intuitively described by "Do not acquire more resources" that would produce $10^{60}$ paperclips, let alone that the strategy producing the *most* paperclips would be inside this space.

We might be able to come up with a weird special-case situation $S_w$ that would imply this.  But that's not the same as asserting, "With high subjective probability, in the real world, the optimal strategy will be in $\neg X$."  We're concerned with making a statement about defaults given the most subjectively probable background states of the universe, not trying to make a universal statement that covers every conceivable possibility.

To put it another way, if your policy choices or predictions are only safe given the premise that "In the real world, the best way of producing the maximum possible number of paperclips involves not acquiring any more resources", you need to clearly [flag this as a load-bearing assumption](https://arbital.com/p/4lz).

•  Caveat:  The claim is not that *every possible* goal can be better-accomplished by acquiring more resources.

As a special case, this would not be true of an agent with an [impact penalty](https://arbital.com/p/4l) term in its utility function, or some other [low-impact agent](https://arbital.com/p/2pf), if that agent also only had [goals of a form that could be satisfied inside bounded regions of space and time with a bounded effort](https://arbital.com/p/4mn).

We might reasonably expect this special kind of agent to only acquire the minimum resources to accomplish its [task](https://arbital.com/p/4mn).

But we wouldn't expect this to be true in a majority of possible cases inside mind design space; it's not true *by default*; we need to specify a further fact about the agent to make the claim not be true; we must expend engineering effort to make an agent like that, and failures of this effort will result in reversion-to-default.  If we imagine some [computationally simple language](https://arbital.com/p/5v) for specifying utility functions, then *most* utility functions wouldn't happen to have both of these properties, so a *majority* of utility functions given this language and measure would not *by default* try to use fewer resources.

•  Caveat:  The claim is not that well-functioning agents must have additional, independent resource-acquiring motivational drives.

A paperclip maximizer will act like it is "obtaining resources" if it merely implements the policy it expects to lead to the most paperclips.  [Clippy](https://arbital.com/p/10h) does not need to have any separate and independent term in its utility function for the amount of resource it possesses (and indeed this would potentially interfere with Clippy making paperclips, since it might then be tempted to hold onto resources instead of making paperclips with them).

•  Caveat:  The claim is not that most agents will behave as if under a deontological imperative to acquire resources.

A paperclip maximizer wouldn't necessarily tear apart a working paperclip factory to "acquire more resources" (at least not until that factory had already produced all the paperclips it was going to help produce.)

•  Check:  Are we arguing "Acquiring resources is a better way to make a few more paperclips than doing nothing" or "There's *no* better/best way to make paperclips that involves *not* acquiring more matter and energy"?

As mentioned above, the latter seems reasonable in this case.

•  Caveat:  "Acquiring resources is instrumentally convergent" is not an ethical claim.

The fact that a paperclip maximizer would try to acquire all matter and energy within reach, does not of itself bear on whether our own [normative](https://arbital.com/p/3y9) [values](https://arbital.com/p/55) might perhaps command that we ought to use few resources as a [terminal value](https://arbital.com/p/1bh).

(Though some of us might find pretty compelling the observation that if you leave matter lying around, it sits around not doing anything and eventually the protons decay or the expanding universe tears it apart, whereas if you turn the matter into people, it can have fun.  There's no rule that instrumentally convergent strategies *don't* happen to be the right thing to do.)

•  Caveat:  "Acquiring resources is instrumentally convergent" is not of itself a futurological prediction.

See above.  Maybe we try to build [Task AGIs](https://arbital.com/p/6w) instead.  Maybe we succeed, and Task AGIs don't consume lots of resources because they have [well-bounded tasks](https://arbital.com/p/4mn) and [impact penalties](https://arbital.com/p/4l).

# Relevance to the larger field of value alignment theory

The [list of arguably convergent strategies](https://arbital.com/p/2vl) has its own page. However, some of the key strategies that have been argued as convergent in e.g. Omohundro's "[The Basic AI Drives](https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf)" and Bostrom's "[The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents](http://www.nickbostrom.com/superintelligentwill.pdf)" include:

- Acquiring/controlling matter and energy.
- Ensuring that future intelligences with similar goals exist.  E.g., a paperclip maximizer wants the future to contain powerful, effective intelligences that maximize paperclips.
  - An important special case of this general rule is *self-preservation*.
  - Another special case of this rule is *protecting goal-content integrity* (not allowing accidental or deliberate modification of the utility function).
- Learning about the world (so as to better manipulate it to make paperclips).
  - Carrying out relevant scientific investigations.
- Optimizing technology and designs.
  - Engaging in an "exploration" phase of seeking optimal designs before an "exploitation" phase of using them.
- Thinking effectively (treating the cognitive self as an improvable technology).
  - Improving cognitive processes.
  - Acquiring computing resources for thought.

This is relevant to some of the central background ideas in [AGI alignment](https://arbital.com/p/2v), because:

- A superintelligence can have a catastrophic impact on our world even if its utility function contains no overtly hostile terms.  A paperclip maximizer doesn't hate you, it just wants paperclips.
- A consequentialist AGI with sufficient big-picture understanding will by default want to promote itself to a superintelligence, even if the programmers did not explicitly program it to want to self-improve.  Even a [pseudoconsequentialist](https://arbital.com/p/) may e.g. repeat strategies that led to previous cognitive capability gains.

This means that programmers don't have to be evil, or even deliberately bent on creating superintelligence, in order for their work to have catastrophic consequences.

The list of convergent strategies, by its nature, tends to include everything an agent needs to survive and grow.  This supports strong forms of the [Orthogonality Thesis](https://arbital.com/p/1y) being true in practice as well as in principle.  We don't need to filter on agents with explicit [terminal](https://arbital.com/p/1bh) values for e.g. "survival" in order to find surviving powerful agents.

Instrumental convergence is also why we expect to encounter most of the problems filed under [https://arbital.com/p/45](https://arbital.com/p/45).  When the AI is young, it's less likely to be [instrumentally efficient](https://arbital.com/p/6s) or understand the relevant parts of the [bigger picture](https://arbital.com/p/3nf); but once it does, we would by default expect, e.g.:

- That the AI will try to avoid being shut down.
- That it will try to build subagents (with identical goals) in the environment.
- That the AI will resist modification of its utility function.
- That the AI will try to avoid the programmers learning facts that would lead them to modify the AI's utility function.
- That the AI will try to pretend to be friendly even if it is not.
- That the AI will try to [conceal hostile thoughts](https://arbital.com/p/3cq) (and the fact that any concealed thoughts exist).

This paints a much more effortful picture of AGI alignment work than "Oh, well, we'll just test it to see if it looks nice, and if not, we'll just shut off the electricity."

The point that some undesirable behaviors are instrumentally *convergent* gives rise to the [https://arbital.com/p/42](https://arbital.com/p/42) problem.  Suppose the AGI's most preferred policy starts out as one of these incorrigible behaviors.  Suppose we currently have enough control to add [patches](https://arbital.com/p/48) to the AGI's utility function, intended to rule out the incorrigible behavior.  Then, after integrating the intended patch, the new most preferred policy may be the most similar policy that wasn't explicitly blocked.  If you naively give the AI a term in its utility function for "having an off-switch", it may still build subagents or successors that don't have off-switches.  Similarly, when the AGI becomes more powerful and [its option space expands](https://arbital.com/p/6q), it's again likely to find new similar policies that weren't explicitly blocked.

Thus, instrumental convergence is one of the two basic sources of [patch resistance](https://arbital.com/p/48) as a [foreseeable difficulty](https://arbital.com/p/6r) of AGI alignment work.

Eric Bruylant
Eliezer Yudkowsky
spxtr spxtr
Alexei Andreev

Summary: AIs that want different things may pursue very similar strategies.  Whether you're trying to make as many paperclips as possible, or keep a single button pressed for as long as possible, you'll still want access to resources of matter, energy, and computation; and to not die in the next five minutes; and to not let anyone edit your utility function.  Strategies that are implied by most (not all) goals are "instrumentally convergent".


Paperclip maximizer

An expected paperclip maximizer is an agent that outputs the action it believes will lead to the greatest number of [paperclips](https://arbital.com/p/7ch) existing.  Or in more detail, its [utility function](https://arbital.com/p/109) is linear in the number of paperclips times the number of seconds that each paperclip lasts, over the lifetime of the universe.  See http://wiki.lesswrong.com/wiki/Paperclip_maximizer.

The agent may be a [bounded maximizer](https://arbital.com/p/) rather than an [objective maximizer](https://arbital.com/p/) without changing the key ideas; the core premise is just that, given actions A and B where the paperclip maximizer has evaluated the consequences of both actions, the paperclip maximizer always prefers the action that it expects to lead to more paperclips.

Some key ideas that the notion of an expected paperclip maximizer illustrates:

- A self-modifying paperclip maximizer [does not change its own utility function](https://arbital.com/p/3r6) to something other than 'paperclips', since this would be expected to lead to fewer paperclips existing.
- A paperclip maximizer instrumentally prefers the standard [convergent instrumental strategies](https://arbital.com/p/2vl) - it will seek access to matter, energy, and negentropy in order to make paperclips; try to build efficient technology for [colonizing the galaxies](https://arbital.com/p/7cy) to transform into paperclips; do whatever science is necessary to gain the knowledge to build such technology optimally; etcetera.
- "The AI does not hate you, nor does it love you, and you are made of atoms it can use for something else."

Eric Bruylant
Eliezer Yudkowsky
Alexei Andreev
Eric Rogstad


Instrumental

An 'instrumental strategy', 'instrumental goal', or 'subgoal' is an event E that an agent tries to bring about in order to bring about some other goal G.  If you want to drink milk, then you need to drive to the store; in order to drive to the store, you need to be inside your car; in order to be inside your car, you need to open your car door.  Thus 'be inside my car' and 'open my car door' are instrumental goals or instrumental strategies.

In conventional philosophy, an event is said to have "instrumental value" if it is useful for accomplishing some implied other set of goals, as distinguished from "terminal value" which is unconditional on future events.  Since in [VAT](https://arbital.com/p/2v) we have reserved the word ['value'](https://arbital.com/p/55), we can't use that terminology here.


Instrumental pressure

Saying that an agent will see 'instrumental pressure' to bring about an event E is saying that this agent, presumed to be a [consequentialist](https://arbital.com/p/9h) with some goal G, will *ceteris paribus and absent defeaters,* want to [bring about E in order to do G](https://arbital.com/p/10j).   For example, a [paperclip maximizer](https://arbital.com/p/10h), Clippy, sees instrumental pressure to gain control of as much matter as possible in order to make more paperclips.  If we imagine an alternate Clippy+ that has a penalty term in its utility function for 'killing humans', Clippy+ still has an instrumental pressure to turn humans into paperclips (because of the paperclips that would be gained) but it also has a countervailing force pushing against that pressure (the penalty term for killing humans).  Thus, we can say that a system is experiencing 'instrumental pressure' to do something, without implying that the system necessarily does it.

This state of affairs is different from the absence of any instrumental pressure:  E.g., Clippy+ might come up with some clever way to [obtain the gains while avoiding the penalty term](https://arbital.com/p/42), like turning humans into paperclips without killing them.

To more crisply define 'instrumental pressure', we need a setup that distinguishes [terminal utility and instrumental expected utility](https://arbital.com/p/10j), as in e.g. a utility function plus a causal model.  Then we can be more precise about the notion of 'instrumental pressure' as follows:  If each paperclip is worth 1 terminal utilon and a human can be disassembled to make 1000 paperclips with certainty, then strategies or event-sets that include 'turn the human into paperclips' thereby have their expected utility elevated by 1000 utils.  There might also be a penalty term that assigns -1,000,000 utilts to killing a human, but then the net expected utility of disassembling the human is -999,000 rather than -1,000,000.  The 1000 utils would still be gained from disassembling the human; the penalty term doesn't change that part.  Even if this strategy doesn't have maximum EU and is not selected, the 'instrumental pressure' was still elevating its EU.  There's still an expected-utility bump on that part of the solution space, even if that solution space is relatively low in value.  And this is perhaps relevantly different because, e.g., there might be some clever strategy for turning humans into paperclips without killing them (even if you can only get 900 paperclips that way).

### Link from instrumental pressures to reflective instrumental pressures

If the agent is reflective and makes reflective choices on a consequentialist basis, there would ceteris paribus be a reflective-level pressure to *search* for a strategy that makes paperclips out of the humans' atoms [without doing anything defined as 'killing the human'](https://arbital.com/p/42).  If a strategy like that could be found, then executing the strategy would enable a gain of 1000 utilons; thus there's an instrumental pressure to search for that strategy.  Even if there's a penalty term added for searching for strategies to evade penalty terms, leading the AI to decide not to do the search, the instrumental pressure will still be there as a bump in the expected utility of that part of the solution space.  (Perhaps there's some [unforeseen](https://arbital.com/p/9f) way to do something very like searching for that strategy while evading the penalty term, such as constructing an outside calculator to do it...)

### Blurring lines in allegedly non-consequentialist subsystems or decision rules

To the extent that the AI being discussed is not a pure consequentialist, the notion of 'instrumental pressure' may start to blur or be less applicable.  E.g., suppose on some level of AI, the choice of which questions to think about is *not* being decided by a choice between options with calculated expected utilities, but is instead being decided by a rule, and the rule excludes searching for strategies that evade penalty terms.  Then maybe there's no good analogy to the concept of 'an instrumental pressure to search for strategies that evade penalty terms', because there's no expected utility rating on the solution space and hence no analogous bump in the solution space that might eventually intersect a feasible strategy.  But we should still perhaps [be careful about declaring that an AI subsystem has no analogue of instrumental pressures, because instrumental pressures may arise even in systems that don't look explicitly consequentialist](https://arbital.com/p/).


Guarded definition

A guarded definition is one where at least one position suspects there will be pressure to stretch a concept and make it cover more than it ought to, and so they set aside a term meant to refer *narrowly* to the things inside the concept.  Thus, if a term has been designated as a 'guarded definition', stretching it to cover new and non-central members that are not *very* clearly part of the definition, and agreed to be so by those who wanted to designate it as guarded, is an unusually strong discourtesy.  If the term was originated (or its special meaning was originated) specifically in order to set it aside as a narrow and guarded term, then it is a discourse norm to respect that narrow meaning and not try to extend it.

Example:  Suppose that Alice and Bob are having a conversation about natural selection.  Alice points out that since everything occurs within Nature, all selection, including human agricultural breeding and genetic engineering, seems to her like 'natural selection', and she also argues that consumer choice in supermarkets is an instance of 'natural selection' since people are natural objects and they're selecting which foods to buy, and thus her paper on watching people buy food in supermarkets ought to be funded by a program on evolutionary biology.  If Bob and his researchers then begin using the term 'ecologically natural selection' because they think it's important to have a narrow term to refer to just birds breeding in the wild and not consumer choice in supermarkets, it is an extreme discourtesy (and a violation of what we locally take to be discourse norms) for Alice to start arguing that really supermarkets are instances of ecologically natural selection too.