Rescuing the utility function

"Saving the phenomena" is the name for the rule that brilliant new scientific theories still need to reproduce our mundane old observations.  The point of heliocentric astronomy is not to predict that the Sun careens crazily over the sky, but rather, to explain why the Sun appears to rise and set each day - the same old mundane observations we had already.  Similarly quantum mechanics is not supposed to add up to a weird universe unlike the one we observe; it is supposed to add up to normality.  New theories may have not-previously-predicted observational consequences in places we haven't looked yet, but by *default* we expect the sky to look the same color.

"Rescuing the utility function" is an analogous principle meant to apply to [naturalistic](https://arbital.com/p/112) moral philosophy: new theories about which things are composed of which other things should, by default, not affect what we value.  For example, if your values previously made mention of "moral responsibility" or "subjective experience", you should go on valuing these things after discovering that people are made of parts.

As the above sentence contains the word "should", the principle of "rescuing the utility function" is being asserted as [a normative principle rather than a descriptive theory](https://arbital.com/p/3y9).

[https://arbital.com/p/toc:](https://arbital.com/p/toc:)

# Metaphorical example: heat and kinetic energy

Suppose, for the sake of metaphor, that our species regarded "warmth" as a [terminal](https://arbital.com/p/1bh) value over the world.  It wouldn't just be nice to *feel* warm in a warm coat; instead you would prefer that the outside world actually be warm, in the same way that e.g. you prefer for your friends to actually be happy in the outside world, and ceteris paribus you wouldn't be satisfied to only deceive yourself into thinking your friends were happy.

One day, scientists propose that "heat" may really be composed of "disordered kinetic energy" - that when we experience an object as warm, it's because the particles comprising that object are vibrating and bumping into each other.

You imagine this possibility in your mind, and find that you don't get any sense of lovely warmth out of imagining lots of objects moving around.  No matter how fast you imagine an object vibrating, this imagination doesn't seem to produce a corresponding imagined feeling of warmth.  You therefore reject the idea that warmth is composed of disordered kinetic motion.

After this, science advances a bit further and *proves* that heat is composed of disordered kinetic energy.

One possible way to react to this revelation - again, assuming for the sake of argument that you cared about warmth as a [terminal value](https://arbital.com/p/1bh) - would be by experiencing great existential horror.  Science has proven that the universe is devoid of ontologically basic heat!  There's really no such thing as heat!  It's all just temperature-less particles moving around!

Sure, if you dip your finger in hot water, it *feels* warm.  But neuroscientists have shown that when our nerves tell us there's heat, they're really just being fooled into firing by being excited with kinetic energy.  When we touch an object and it feels hot, this is just an illusion being produced by fast-moving particles activating our nerves.  This is why our brains make us think that things are hot even though they're just bouncing particles.  Very sad, but at least now we know the truth.

Alternatively, you could react as follows:

- Heat doesn't have to be ontologically basic to be valuable.  Valuable things can be made out of parts.
- The parts that heat was made out of, turn out to be disordered kinetic energy.  Heat isn't an illusion, vibrating particles just *are* heat.  It's not like you're getting a consolation prize of vibrating particles when you really wanted heat.  You *have* heat.  It's *right there* in the warm water.
- From now on, you'll think of warmth when you think of disordered kinetic energy and vibrating particles.  Since your native emotions don't automatically light up when you use the vibrating-particles visualization of heat, you will now adopt the rule that whenever you *imagine* disordered kinetic energy being present, you will *imagine* a sensation of warmth so as to go on binding your emotions to this new model of reality.

This reply would be "rescuing the utility function".

## Argument for rescuing the utility function (still in the heat metaphor)

Our minds have an innate and instinctive representation of the universe to which our emotions natively and automatically bind.  Warmth and color are basic *to that representation*; we don't instinctively imagine them as made out of parts.  When we imagine warmth in our native model, our emotions automatically bind and give us the imagined feeling of warmth.

After learning more about how the universe works and how to imagine more abstract and non-native concepts, we can also visualize a lower-level model of the universe containing vibrating particles.  But unsurprisingly, our emotions automatically bind only to our native, built-in mental models and not the learned abstract models of our universe's physics.  So if you imagine tiny billiard balls whizzing about, it's no surprise that this mental picture doesn't automatically trigger warm feelings.

It's a [descriptive](https://arbital.com/p/3y9) statement about our universe, a way the universe is, that 'heat' is a high-level representation of the disordered kinetic energy of colliding and vibrating particles.  But to advocate that we should re-bind our emotions to this non-native mental model by *feeling cold* is merely one possible [normative](https://arbital.com/p/3y9) statement among many.

Saying "There is really no such thing as heat!" is from this perspective [a normative statement rather than a descriptive one](https://arbital.com/p/3y9).  The real meaning is "If you're in a universe where the observed phenomenon of heat turns out to be comprised of vibrating particles, then you *shouldn't* feel any warmth-related emotions about that universe."  Or, "Only ontologically basic heat can be valuable."  Or, "If you've only previously considered questions of value over your native representation, and that's the only representation to which your emotions automatically bind without further work, then you should attach zero value to every possible universe whose physics don't exactly match that representation."  This normative proposition is a different statement than the descriptive truth, "Our universe contains no ontologically basic heat."

The stance of "rescuing the utility function" advocates that we have no right to expect the universe to function exactly like our native representation of it.  According to this stance, it would be a strange and silly demand to make of the universe that its lowest level of operation correspond exactly to our built-in mental representations, and insist that we're not going to *feel* anything warm about reality unless heat is basic to physics.  The high-level representations our emotions natively bind to, could not reasonably have been fated to be identical with the raw low-level description of the universe.  So if we couldn't 'rescue the utility function' by identifying high-level heat with vibrating particles, this portion of our values would inevitably end in disappointment.

Once we can see as normative the question of how to feel about a universe that has dancing particles instead of ontologically basic warmth, we can see that going around wailing in existential despair about the coldness of the universe doesn't seem like the right act or the right judgment.  Instead we should rebind our native emotions to the non-instinctive but more accurate model of the universe.

If we aren't self-modifying AIs and can't actually rewrite our emotions to bind to learned abstract models, then we can come closer to normative reasoning by adopting the rule of visualizing warmth whenever we visualize whizzing particles.

On this stance, it is not a lie to visualize warmth when we visualize whizzing particles.  We are not giving ourselves a sad consolation prize for the absence of 'real' heat.  It's not an act of self-deception to imagine a sensation of lovely warmth going along with a bunch of vibrating particles.  That's just what heat is in our universe.  Similarly, our nerves are not lying to us when they make us feel that fast-vibrating water is warm water.

On the normative stance of "rescuing the utility function", when X turns out to be composed of Y, then by default we should feel about Y the same way we felt about X.  There might be other considerations that modify this, but that's the starting point or default.

After 'rescuing the utility function', your new theory of how the universe operates (that heat is made up of kinetic energy) adds up to *moral normality.*  If you previously thought that a warm summer was better than a cold winter, you will still think that a warm summer is better than a cold winter after you find out what heat is made out of.

(This is a good thing, since the point of a moral philosophy is not to be amazing and counterintuitive.)

## Reason for using the heat metaphor

One reason to start from this metaphorical example is that "heat" has a relatively understandable correspondence between high-level and low-level models.  On a high level, we can see heat melting ice and flowing from hotter objects to cooler objects.  We can, by imagination, see how vibrating particles could actually constitute heat rather than causing a mysterious extra 'heat' property to be present.  Vibrations might flow from fast-vibrating objects to slow-vibrating objects via the particles bumping into each other and transmitting their speed.  Water molecules vibrating quickly enough in an ice cube might break whatever bonds were holding them together in a solid object.

Since it does happen to be relatively easy to visualize how heat is composed of kinetic energy, we can *see* in this case that we are not lying to ourselves by imagining that lovely warmth is present wherever vibrating particles are present.

For an even more transparent reductionist identity, consider, "You're not *really* wearing socks, there *are* no socks, there's only a bunch of threads woven together that *looks* like a sock."  Your visual cortex can represent this identity directly, so it feels immediately transparent that the sock just is the collection of threads; when you imagine sock-shaped woven threads, you automatically feel your visual model recognizing a sock.

If the relation between heat and kinetic energy were too complicated to visualize easily, it might instead feel like we were being given a blind, unjustified rule that reality contains mysterious "bridging laws" that make a separate quality of heat be present when particles vibrate quickly.  Instructing ourselves to feel "warmth" as present when particles vibrate quickly would feel more like fooling ourselves, or self-deception.  But on the position of "rescuing the utility function", the same arguments ought to apply in this hypothetical, even if the level transition is less transparent.

The gap between mind and brain is larger than the gap between heat and vibration, which is why humanity understood heat as disordered kinetic energy long before anyone had any idea [how 'playing chess' could be decomposed into non-mental simpler parts](https://arbital.com/p/38r).  In some cases, we may not know what the reductionist identity will be.  Still, the advice of "rescuing the utility function" is not to morally panic about realizing that various emotionally valent things will turn out to be made of parts, or even that our mind's representations in general may run somewhat skew to reality.

## Complex rescues

In the heat metaphor, the lower level of the universe (jiggling particles) corresponds fairly exactly to the high-level notion of heat.  We'd run into more complicated metamoral questions if we'd previously lumped together the 'heat' of chili peppers and the 'heat' of a fireplace as valuable 'warmth'.

We might end up saying that there are two physical kinds of valuable warm things: ceteris paribus and by default, if X turns out to consist of Y, then Y inherits X's role in the utility function.  Alternatively, by some non-default line of reasoning, the discovery that chili peppers and fireplaces are warm in ontologically different ways might lead us to change how we feel about them on a high level as well.  In this case we might have to carry out a more complicated rescue, where it's not so immediately obvious which low-level Ys are to inherit X's value.

# Non-metaphorical utility rescues

We don't actually have [terminal values](https://arbital.com/p/1bh) for things being warm (probably).  Non-metaphorically, "rescuing the utility function" says that we should apply similar reasoning to phenomena that we do in fact value, whose corresponding native emotions we are having trouble reconciling with non-native, learned theories of the universe's ontology.

Examples might include:

- Moral responsibility.  How can we hold anyone responsible for their actions, or even hold ourselves responsible for what we see as our own choices, when our acts have causal histories behind them?
- Happiness.  What's the point of people being happy if it's just neurons firing?
- Goodness and shouldness.  Can there be any right thing to do, if there isn't an ontologically basic irreducible rightness property to correspond to our sense that some things are just right?
- Wanting and helping.  When a person wants different things at different times, and there are known experiments that expose circular and incoherent preferences, how can we possibly "help" anyone by "giving them what they want" or "[extrapolating their volition](https://arbital.com/p/313)"?

In cases like these, it may be that our native representation is in some sense running skew to the real universe.  E.g., our minds insist that something called "free will" is very important to moral responsibility, but it seems impossible to define "free will" in a coherent way.  The position of "rescuing the utility function" still takes the stance of "Okay, let's figure out how to map this emotion onto a coherent universe as best we can" not "Well, it looks like the human brain didn't start out with a perfect representation of reality, therefore, normatively speaking, we should toss the corresponding emotions out the window."  If in your native representation the Sun goes around the Earth, and then we learn differently from astronomy, then your native representation is in fact wrong, but normatively we should (by default) re-bind to the enormous glowing fusion reactor rather than saying that there's no Sun.

The role such concepts play in our values lends a special urgency to the question of how to rescue them.  But on an even more general level, one might espouse that it is the job of good reductionists to say *how* things exist if they have any scrap of reality, rather than it being the job of reductionists to go around declaring that things *don't* exist if we detect the slightest hint of fantasy.  Leif K-Brooks presented this general idea as follows (with intended application to 'free will' in particular):

![If you define a potato as a magic fairy orb and disprove the existence of magic fairy orbs, you still have a potato.](https://scontent.fsnc1-4.fna.fbcdn.net/v/t1.0-9/15697756_10154038465390025_4647971104986602099_n.jpg?oh=08071559c0a79d7779dd56ad6be1bd13&oe=58F65AAC)

# "Rescue" as resolving a degree of freedom in the pretheoretic viewpoint

A human child doesn't start out endorsing any particular way of turning emotions into utility functions.  As humans, we start out with no clear rules inside the chaos of our minds, and we have to make them up by considering various arguments appealing to our not-yet-organized intuitions.  Only then can we even try to have coherent metaethical principles.

The core argument for "rescuing the utility function" can be seen as a base intuitive appeal to someone who hasn't yet picked out any explicit rules, claiming that it isn't especially sensible to end up as the kind of agent whose utility function ends up being zero everywhere.

In other words, rather than the rescue project needing to appeal to rules that would *only* be appealing to somebody who'd already accepted the rescue project ab initio - which would indeed be circular - the start of the argument is meant to also work as an intuitive appeal to the pretheoretic state of mind of a normal human.  After that, we also hopefully find that these new rules are self-consistent.

In terms of the heat metaphor, if we're considering whether to discard heat, we can consider three types of agents:

- (1).  A pretheoretic or confused state of intuition, which knows itself to be confused.  An agent like this is not [reflectively consistent](https://arbital.com/p/2rb) - it wants to resolve the internal tension.
- (2).  An agent that has fully erased all emotions relating to warmth, as if it never had them.  This type of agent is [reflectively consistent](https://arbital.com/p/2rb); it doesn't value warmth and doesn't want to value warmth.
- (3).  An agent that values naturalistic heat, i.e., feels the way about disordered kinetic energy that a pretheoretic human feels about warmth.  This type of agent has also resolved its issues and become [reflectively consistent](https://arbital.com/p/2rb).

Since 2 and 3 are both internally consistent resolutions, there's potentially a [https://arbital.com/p/-2fr](https://arbital.com/p/-2fr) in how (1) can resolve its current internal tension or inconsistency.  That is, it's not the case that the only kind of coherent agents are 2-agents or 3-agents, so just a desire for coherence qua coherence can't tell a 1-agent whether it should become a 2-agent or 3-agent.  By advocating for rescuing the utility function, we're appealing to a pretheoretic and maybe chaotic and confused mess of intuitions, aka a human, arguing that if you want to shake out the mess, it's better to shake out as a 3-agent rather than a 2-agent.

In making this appeal, we can't appeal to firm foundations that already exist, since a 1-agent hasn't yet decided on firm philosophical foundations and there's more than one set of possible foundations to adopt.  An agent with firm foundations would already be reflectively coherent and have no further philosophical confusion left to resolve (except perhaps for a mere matter of calculation).  An existing 2-agent is of course nonplussed by any arguments that heat should be valued, in much the same way that humans would be nonplussed for arguments in favor of valuing [paperclips](https://arbital.com/p/10h) (or for that matter, things being hot).  But to point this out is no argument for why a confused 1-agent should shake itself out as a consistent 2-agent rather than a consistent 3-agent; a 3-agent is equally nonplussed by the argument that the best thing to do with an [ontology identification problem](https://arbital.com/p/5c) is throw out all corresponding terms of the utility function.

It's perhaps lucky that human beings can't actually modify their own code, meaning that somebody partially talked into taking the 2-agent state as a new ideal to aspire to, still actually has the pretheoretic emotions and can potentially "snap out of it".  Rather than becoming a 2-agent or a 3-agent, we become "a 1-agent that sees 2-agency as ideal" or "a 1-agent that sees 3-agency as ideal".  A 1-agent aspiring to be a 2-agent can still potentially be talked out of it - they may still feel the weight of arguments meant to appeal to 1-agents, even if they think they *ought* not to, and can potentially "just snap out" of taking 2-agency as an ideal, reverting to confused 1-agency or to taking 3-agency as a new ideal.

## Looping through the meta-level in "rescuing the utility function"

Since human beings don't have "utility functions" (coherent preferences over probabilistic outcomes), the notion of "rescuing the utility function" is itself a matter of rescue.  Natively, it's possible for psychology experiments to expose inconsistent preferences, but instead of throwing up our hands and saying "Well I guess nobody wants anything and we might as well [let the universe get turned into paperclips](https://arbital.com/p/10h)!", we try to back out some reasonably coherent preferences from the mess.  This is, arguendo, normatively better than throwing up our hands and turning the universe into paperclips.

Similarly, according to [the normative stance behind extrapolated volition](https://arbital.com/p/313), the very notion of "shouldness" is something that gets rescued.  Many people seem to instinctively feel that 'shouldness' wants to map onto an ontologically basic, irreducible property of rightness, such that every cognitively powerful agent with factual knowledge about this property is thereby compelled to perform the corresponding actions.  ("Moral internalism.")  But this demands an overly direct correspondence between our native sense that some acts have a compelling rightness quality about them, and wanting there to be an ontologically basic compelling rightness quality out there in the environment.

Despite the widespread appeal of moral internalism once people are exposed to it as an explicit theory, it still seems unfair to say that humans *natively* want or *pretheoretically* demand that this is what our sense of rightness correspond to.  E.g. a hunter-gatherer, or someone else who's never debated metaethics, doesn't start out with an explicit commitment about whether a feeling of rightness must correspond to universes that have irreducible rightness properties in them.  If you'd grown up thinking that your feeling of rightness [corresponded to computing a certain logical function over universes](https://arbital.com/p/313), this would seem natural and non-disappointing.

Since "shouldness" (the notion of normativity) is something that itself may need rescuing, this rescue of "shouldness" is in some sense being implicitly invoked by the normative assertion that we *should* try to "rescue the utility function".

This could be termed circular, but we could equally say that it is self-consistent.  Or rather, we are appealing to some chaotic, not-yet-rescued, pretheoretic notion of "should" in saying that we *should* try to rescue concepts like "shouldness" instead of throwing them out the window.  Afterwards, once we've performed the rescue and have a more coherent notion of concepts like "better", we can see that the loop through the meta-level has entered a consistent state.  According to this new ideal (less confused, but perhaps also seeming more abstract), it remains better not to give up on concepts like "better".

The "extrapolated volition" rescue of shouldness is meant to bootstrap, by appeal to a pretheoretic and potentially confused state of wondering what is the right thing to do (or how we even *should* resolve this whole "rightness" issue, and if maybe it would be *better* to just give up on it), into a more reflectively consistent state of mind.  Afterwards we can see both pretheoretically, and also in the light of our new explicit theory, that we *ought* to try to rescue the concept of oughtness, and adopt the rescued form of reasoning as a less-confused ideal.  We will believe that *ideally*, the best justification for extrapolated volition is to say that we know of no better candidate for what theory we'd arrive at if we thought about it for even longer.  But since humans can perhaps thankfully not directly rewrite their own code, we will also remain aware of whether this seems like a good idea in the pretheoretic sense, and perhaps prepared to unwind or jump back out of the system if it turns that out that the explicit theory has big problems we didn't know about when we originally jumped to it.

## Reducing tension / "as if you'd always known it"

We can possibly see the desired output of "rescuing the utility function" as being something like reducing a tension between native emotion-binding representations, and reality, with a minimum of added fuss and complexity.

This can look a lot like the intuition pump, "Suppose you'd grown up always knowing the true state of affairs, and nobody had suggested that you panic over it or experience any existential angst; what would you have grown up thinking?"  If you grow up with warmth-related emotions and already knowing that heat is disordered kinetic energy, and nobody has suggested to you that anyone ought to wail in existential angst about this, then you'll probably grow up valuing heat-as-disordered-kinetic-energy (and this will be a low-tension resolution).

For more confusing grades of cognitive reductionism, like free will, people might spontaneously have difficulty reconciling their internal sense of freedom with being told about deterministic physical laws.  But a good "rescue" of the corresponding sense of moral responsibility ought to end up looking like the sort of thing that you'd quietly take for granted as a quiet, obvious-seeming mapping of your sense of moral responsibility onto the physical universe, if you'd grown up taking those laws of physics for granted.

# "Rescuing" pretheoretic emotions and intuitions versus "rescuing" explicit moral theories

## Rewinding past factual-mistake-premised explicit moral theories

In the heat metaphor, suppose you'd previously adopted a 'caloric fluid' model of heat, and the explicit belief that this caloric fluid was what was valuable.  You still have wordless intuitions about warmth and good feelings about warmth.  You *also* have an explicit world-model that this heat corresponds to caloric fluid, and an explicit moral theory that this caloric fluid is what's good about heat.

Then science discovers that heat is disordered kinetic energy.  Should we try to rescue our moral feelings about caloric by looking for the closest thing in the universe to caloric fluid - electricity, maybe?

If we now reconsider the arguments for "rescuing the utility function", we find that we have more choices beyond "looking for the closest thing to caloric" and "giving up entirely on warm feelings".  An additional option is to try to rescue the intuitive sense of warmth, but not the explicit beliefs and explicit moral theories about "caloric fluid".

If we instead choose to rescue the pretheoretic emotion, we could see this as retracing our steps after being led down a garden-path of bad reasoning, aka, "not rescuing the garden path".  We started with intuitively good feelings about warmth, came to believe a false model about the causes of warmth, reacted emotionally to this false model, and developed an explicit moral theory about caloric fluid.

The [extrapolated-volition model of normativity](https://arbital.com/p/313) (what we would want\* if we knew all the facts) suggests that we could see the reasoning after adopting the false caloric model as "mistaken" and not rescue it.  When we're dealing with explicit moral beliefs that grew up around a false model of the world, we have the third option to "rewind and rescue" rather than "rescue" or "give up".

Nonmetaphorically:  Suppose you believe in a [divine command theory of metaethics](https://en.wikipedia.org/wiki/Divine_command_theory); goodness is equivalent to God wanting something.  Then one day you realize that there's no God in which to ground your moral theory.

In this case we have three options for resolution, all of which are reflectively consistent within themselves, and whose arguments may appeal to our currently-confused pretheoretic state:

- (a) Prefer to go about wailing in horror about the unfillable gap in the universe left by the absence of God.
- (b) Try to rescue the explicit divine command theory, e.g. by looking for the closest thing to a God and re-anchoring the divine command theory there.
- (c) Give up on the explicit model of divine command theory; instead, try to unwind past the garden path you went down after your native emotions reacted to the factually false model of God.  Try to remap the pretheoretic emotions and intuitions onto your new model of the universe.

Again, (a) and (b) and (c) all seem reflectively consistent in the sense that a simple agent fully in one of these states will not want to enter either of the other two states.  But given these three options, a confused agent might reasonably find either (b) or (c) more pretheoretically compelling than (a), but also find (c) more pretheoretically compelling than (b).

The notion of "rescuing" isn't meant to erase the notion of "mistakes" and "saying oops" with respect to b-vs.-c alternatives.  The arguments for "rescuing" warmth implicitly assumed that we were talking about a pretheoretic normative intuition (e.g. an emotion associated with warmth), not explicit models and theories about heat that could just as easily be revised.

Conversely, when we're dealing with preverbal intuitions and emotions whose natively bound representations are in some way running skew to reality, we can't rewind past the fact of our emotions binding to particular mental representations.  We were literally born that way.  Then our only obvious alternatives are to (a) give up entirely on that emotion and value, or (c) rescue the intuitions as best we can.  In this case (c) seems more pretheoretically appealing, ceteris paribus and by default.

(For example, suppose you were an alien that had grown up accepting commands from a Hive Queen, and you had a pretheoretic sense of the Hive Queen as knowing everything, and you mostly operated on an emotional-level Hive-Queen-command theory of rightness.  One day, you begin to suspect that the Hive Queen isn't actually omniscient.  Your alien version of "rescuing the utility function" might say to rescue the utility function by allowing valid commands to be issued by Hive Queens that knew a lot but weren't omniscient.  Or it might say to try and build a superintelligent Hive Queen that would know as much as possible, because in a pretheoretic sense that would feel better.  The aliens can't rewind past their analogue of divine command theory because, by hypothesis, the alien's equivalent of divine command metaethics is built into them on a pretheoretic and emotional level.  Though of course, in this case, such aliens seem more likely to actually resolve their tension by asking the Hive Queen what to do about it.)

## Possibility of rescuing non-mistake-premised explicit moral theories

Suppose Alice has an explicit belief that private property ought to be a thing, and this belief did *not* develop after she was told that objects had tiny XML tags declaring their irreducible objective owners, nor did she originally arrive at the conclusion based on a model in which God assigned transferable ownership of all objects at the dawn of time.  We can suppose, somewhat realistically, that Alice is a human and has a pretheoretic concept of ownership as well as deserving rewards for effort, and was raised by small-l libertarian parents who told her true facts about how East Germany did worse economically than West Germany.  Over time, she came to adopt an explicit moral theory of "private property": ownership can only transfer by consent, and that violations of this rule violate the just-rewards principle.

One day, Alice starts having trouble with her moral system because she's realized that property is made of atoms, and that even the very flesh in her body is constantly exchanging oxygen and carbon dioxide with the publicly owned atmosphere.  Can atoms really be privately owned?

The confused Alice now again sees three options, all of them [reflectively consistent](https://arbital.com/p/2rb) on their own terms if adopted:

- (a) Give up on everything to do with ownership or deserving rewards for efforts; regard these emotions as having no valid referents.
- (b) Try to rescue the explicit moral theory by saying that, sure, atoms can be privately owned.  Alice owns a changeable number of carbon atoms inside her body and she won't worry too much about how they get exchanged with the atmosphere; that's just the obvious way to map private property onto a particle-based universe.
- (c) Try to rewind past the explicit moral theory, and figure out from scratch what to do with emotions about "deserves reward" or "owns".

Leaving aside what you think of Alice's explicit moral theory, it's not obvious that Alice will end up preferring (c) to (b), especially since Alice's current intuitive state is influenced by her currently-active explicit theory of private property.

Unlike the divine command theory, Alice's private property theory was not (obviously to Alice) arrived at through a path that traversed [wrong beliefs of simple fact](https://arbital.com/p/3t3).  With the divine command theory, since it was critically premised on a wrong factual model, we face the prospect of having to stretch the theory quite a lot in order to rescue it, making it less intuitively appealing to a confused mind than the alternative prospect of stretching the pretheoretic emotions a lot less.  Whereas from Alice's perspective, she can just as easily pick up the whole moral theory and morph it onto reductionist physics with all the internal links intact, rather than needing to rewind past anything.

We at least have the apparent *option* of trying to rescue Alice's utility function in a way that preserves her explicit moral theories *not* based on bad factual models - the steps of her previous explicit reasoning that did not, of themselves, introduce any new tensions or problems in mapping her emotions or morals onto a new representation.  Whether or not we ought to do this, it's a plausible possibility on the table.

## Which explicit theories to rescue?

It's not yet obvious where to draw the line on which explicit moral theories to rescue.  So far as we can currently see, any of the following might be a reasonable way to tell a superintelligence to extrapolate someone's volition:

- Preserve explicit moral theories wherever it doesn't involve an enormous stretch.
- Be skeptical of explicit moral theories that were arrived at by fragile reasoning processes, even if they could be rescued in an obvious way.
- Only extrapolate pretheoretic intuitions.

Again, all of these viewpoints are internally consistent (they are degrees of freedom in the metaphorical meta-utility-function), so the question is which rule for drawing the line seems most intuitively appealing in our present state of confusion:

### Argument from adding up to normality

Arguendo:  Preserving explicit moral theories is important for having the rescued utility function add up to normality.  After rescuing my notion of "shouldness", then afterwards I should, by default, see mostly the same things as rescued-right.

Suppose Alice was previously a moral internalist, and thought that some things were inherently irreducibly right, such that her very notion of "shouldness" needed rescuing.  That doesn't necessarily introduce any difficulties into re-importing her beliefs about private property.  Alice may have previously refused to consider some arguments against private property because she thought it was irreducibly right, but this is a separate issue in extrapolating her volition from throwing out her entire stock of explicit moral theories because they all used the word "should".  By default, after we're done rescuing Alice, unless we are doing something that's clearly and explicitly correcting an error, her rescued viewpoint should look as normal-relative-to-her-previous-perspective as possible.

### Argument from helping

Arguendo:  Preserving explicit moral theories where possible is an important aspect of how an ideal advisor or superintelligence ought to extrapolate someone else's volition.

Suppose Alice was previously a moral internalist, and thought that some things were inherently irreducibly right, such that her very notion of "shouldness" needed rescuing.  Alice may not regard it as "helping" her to throw away all of her explicit theories and try to re-extrapolate her emotions from scratch into new theories.  If there weren't any factual flaws involved, Alice is likely to see it as less than maximally helpful to her if we needlessly toss one of her cherished explicit moral theories.

### Argument from incoherence, evil, chaos, and arbitrariness

Arguendo:  Humans are really, really bad at systematizing explicit moral theories; a supermajority of explicit moral theories in today's world will be incoherent, evil, or both.  E.g., explicit moral principles may de-facto be chosen mostly on the basis of, e.g., how hard they appear to cheer for a valorized group.  An extrapolation dynamic that tries to take into account all these chaotic, arbitrarily-generated group beliefs will end up failing to cohere.

### Argument from fragility of goodness

Arguendo:  Most of what we see as the most precious and important part of ourselves are explicit moral theories like "all sapient beings should have rights", which aren't built into human babies.  We may well have arrived at that destination through a historical trajectory that went through factual mistakes, like believing that all human beings had souls created equal by God and were loved equally by God. (E.g. Christian theology seems to have been, as a matter of historical fact, causally important in the development of explicit anti-slavery sentiment.)  Tossing the explicit moral theories is as unlikely to be good, from our perspective, as tossing our brains and trying to rerun the process of natural selection to generate new emotions.

### Argument from dependency on empirical results

Arguendo:  Which version of extrapolation we'll actually find appealing will depend on which extrapolation algorithm turns out to have a reasonable answer.  We don't have enough computing power to guess, right now, whether:

- Any reasonable-looking construal of "Toss out all the explicit cognitive content and redo by throwing pretheoretic emotions at true facts" leads to an extrapolated volition that lacks discipline and coherence, looking selfish and rather angry, failing to regenerate most of altruism or Fun Theory; *or*
- Any reasonable-looking construal of "Try to preserve explicit moral theories" leads to an incoherent mess of assertions about various people going to Hell and capitalism being bad for you.

Since we can't guess using our present computing power which rule would cause us to recoil in horror, but the actual horrified recoil would settle the question, we can only defer this single bit of information to one person who's allowed to peek at the results.

(Counterargument:  "Perhaps the ancient Greeks would have recoiled in horror if they saw how little the future would think of a glorious death in battle, thus picking the option we see as wrong, using the stated rule.")

Summary:
"Saving the phenomena" is the name for the rule that brilliant new scientific theories still need to reproduce our mundane old observations.  The point of heliocentric astronomy is not to predict that the Sun careens crazily over the sky, but rather, to explain why the Sun appears to rise and set each day.

"Rescuing the utility function" is an analogous principle meant to apply to [naturalistic](https://arbital.com/p/112) moral philosophy: new theories about which things are composed of which other things should, by default, not affect what we value: they should "add up to moral normality".  For example, if your values previously made mention of "moral responsibility" or "subjective experience", you should by default go on valuing these things after discovering that people are made of parts.

A metaphorical example is that if your utility function values 'heat', and then you discover to your horror that there's no ontologically basic heat, merely little particles bopping around, then you should by default switch to valuing disordered kinetic energy the same way you used to value heat. That's just what heat *is,* what it turned out to be.  Likewise valuing "people" or "consciousness".


Philosophy

For now, this is a stub domain to hold standard concepts from academic philosophy, which are being used elsewhere on Arbital (we're not currently targeting this domain as such).


Total alignment

An advanced agent can be said to be "totally aligned" when it can assess the *exact* [https://arbital.com/p/-55](https://arbital.com/p/-55) of well-described outcomes and hence the *exact* subjective value of actions, policies, and plans; where [https://arbital.com/p/-55](https://arbital.com/p/-55) has its overridden meaning of a metasyntactic variable standing in for "whatever we really do or really should value in the world or want from an Artificial Intelligence" (this is the same as "normative" if the speaker believes in normativity).  That is:  It's an advanced agent that captures *all* the distinctions we would make or should make within which outcomes are good or bad; it has "full coverage" of the true or intended goals; it correctly resolves every [https://arbital.com/p/-2fr](https://arbital.com/p/-2fr).

We don't need to try and give such an AI simplified orders like, e.g., "try to have a [lower impact](https://arbital.com/p/2pf)" because we're worried about, e.g., a [https://arbital.com/p/-42](https://arbital.com/p/-42) problem on trying to draw exact boundaries around what constitutes a bad impact.  The AI knows *everything* worth knowing about which impacts are bad, and even if it thinks of a really weird exotic plan, it will still be able to figure out which aspects of this plan match our intended notion of  [https://arbital.com/p/-55](https://arbital.com/p/-55) or a normative notion of  [https://arbital.com/p/-55](https://arbital.com/p/-55).

If this agent does not systematically underestimate the probability of bad outcomes / overestimate the probability of good outcomes, and its maximization over policies is not subject to adverse subjection, its estimates of expected [https://arbital.com/p/-55](https://arbital.com/p/-55) will be well-calibrated even from our own outside standpoint.


Superintelligent

Machine performance inside a domain (class of problems) can potentially be:

- Optimal (impossible to do better)
- Strongly superhuman (better than all humans by a significant margin)
- Weakly superhuman (better than all the humans most of the time and most of the humans all of the time)
- Par-human (performs about as well as most humans, better in some places and worse in others)
- Subhuman or infrahuman (performs worse than most humans)

A superintelligence is either 'strongly superhuman', or else at least 'optimal', across all cognitive domains.  It can't win against a human at [logical tic-tac-toe](https://arbital.com/p/9s), but it plays optimally there.  In a real-world game of tic-tac-toe that it strongly wanted to win, it might sabotage the opposing player, deploying superhuman strategies on the richer "real world" gameboard.

I. J. Good originally used 'ultraintelligence' to denote the same concept:  "Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever."

To say that a hypothetical agent or process is "superintelligent" will usually imply that it has all the [advanced-agent properties](https://arbital.com/p/2c).

Superintelligences are still [bounded](https://arbital.com/p/2rd) (if the character of physical law at all resembles the Standard Model of physics).  They are (presumably) not infinitely smart, infinitely fast, all-knowing, or able to achieve every describable outcome using their available resources and options.  However:

- A supernova isn't infinitely hot, but it's still pretty warm.  "Bounded" does not imply "small".  You should not try to walk into a supernova using a standard flame-retardant jumpsuit after reasoning, correctly but unhelpfully, that it is only boundedly hot.
- A superintelligence doesn't know everything and can't perfectly estimate every quantity.  However, to say that something is "superintelligent" or superhuman/optimal in every cognitive domain should almost always imply that its estimates are [epistemically efficient relative to every human and human group](https://arbital.com/p/6s).  Even a superintelligence may not be able to exactly estimate the number of hydrogen atoms in the Sun, but a human shouldn't be able to say, "Oh, it will probably underestimate the number by 10% because hydrogen atoms are pretty light" - the superintelligence knows that too.  For us to know better than the superintelligence is at least as implausible as our being able to predict a 20% price increase in Microsoft's stock six months in advance without any private information.
- A superintelligence is not omnipotent and can't obtain every describable outcome.  But to say that it is "superintelligent" should suppose at least that it is [instrumentally efficient relative to humans](https://arbital.com/p/6s):  We should not suppose that a superintelligence carries out any policy $\pi_0$ such that a human can think of a policy $\pi_1$ which would get more of the agent's [utility](https://arbital.com/p/109).  To put it another way, the assertion that a superintelligence optimizing for utility function $U,$ would pursue a policy $\pi_0,$ is by default refuted if we observe some $\pi_1$ such that, so far as we can see, $\mathbb E[| \pi_0](https://arbital.com/p/U) < \mathbb E[| \pi_1](https://arbital.com/p/U).$  We're not sure the efficient agent will do $\pi_1$ - there might be an even better alternative we [haven't foreseen](https://arbital.com/p/9f) - but we should regard it as very likely that it won't do $\pi_0.$

If we're talking about a hypothetical superintelligence, probably we're either supposing that an [intelligence explosion](https://arbital.com/p/428) happened, or we're talking about a limit state approached by a long period of progress.

Many/most problems in [AI alignment](https://arbital.com/p/2v) seem like they ought to first appear at a point short of full superintelligence.  As part of the project of making discourse about advanced agents precise, we should try to identify the key advanced agent property more precisely than saying "this problem would appear on approaching superintelligence" - to suppose superintelligence is usually *sufficient* but will rarely be necessary.

For the book, see [https://arbital.com/p/3db](https://arbital.com/p/3db).

Eric Bruylant
Eliezer Yudkowsky
Eric Rogstad

Summary: A supernova isn't infinitely hot, but it's still pretty darned hot; in the same sense, a superintelligence isn't infinitely smart, but it's pretty darned smart: strictly superhuman across all cognitive domains, by a significant margin, or else as a fallback merely optimal.  (A superintelligence can't win against a human at [logical tic-tac-toe](https://arbital.com/p/9s), though in real-world tic-tac-toe it could disassemble the opposing player.)  Superintelligences are [epistemically and instrumentally efficient](https://arbital.com/p/6s) relative to humans, and have all the other [advanced agent properties](https://arbital.com/p/2c) as well.


William Frankena's list of terminal values

"Life, consciousness, and activity; health and strength; pleasures and satisfactions of all or certain kinds; happiness, beatitude, contentment, etc.; truth; knowledge and true opinions of various kinds, understanding, wisdom; beauty, harmony, proportion in objects contemplated; aesthetic experience; morally good dispositions or virtues; mutual affection, love, friendship, cooperation; just distribution of goods and evils; harmony and proportion in one's own life; power and experiences of achievement; self-expression; freedom; peace, security; adventure and novelty; and good reputation, honor, esteem, etc." -- William Frankena's list of [things valued in themselves rather than only for their further consequences](https://arbital.com/p/1bh).


Nearest unblocked strategy

# Introduction

'Nearest unblocked strategy' seems like it should be a [foreseeable](https://arbital.com/p/6r) problem of trying to get rid of undesirable AI behaviors by adding specific penalty terms to them, or otherwise trying to exclude one class of observed or foreseen bad behaviors.  Namely, if a decision criterion thinks $X$ is the best thing to do, and you add a penalty term $P$ that you think excludes everything inside $X,$ the *next-best* thing to do may be a very similar thing $X'$ which is the most similar thing to $X$ that doesn't trigger $P.$

## Example:  Producing happiness.

Some very early proposals for AI alignment suggested that AIs be targeted on producing human happiness.  Leaving aside various other objections, arguendo, imagine the following series of problems and attempted fixes:

- By hypothesis, the AI is successfully infused with a goal of "human happiness" as a utility function over human brain states.  (Arguendo, this predicate is narrowed sufficiently that the AI does not just want to construct [the tiniest, least resource-intensive brains experiencing the largest amount of happiness per erg of energy](https://arbital.com/p/2w).)
- Initially, the AI seems to be pursuing this goal in good ways; it organizes files, tells funny jokes, helps landladies take out the garbage, etcetera.
- Encouraged, the programmers further improve the AI and add more computing power.
- The AI gains a better understanding of the world, and the AI's [policy space expands](https://arbital.com/p/6q) to include conceivable options like "administer heroin".
- The AI starts planning how to administer heroin to people.
- The programmers notice this before it happens.  (Arguendo, due to successful transparency features, or an imperative to [check plans with the users](https://arbital.com/p/2qq), which operated as [intended](https://arbital.com/p/6h) at the AI's current level of intelligence.)
- The programmers edit the AI's utility function and add a penalty of -100 utilons for any event categorized as "the AI administers heroin to humans".  (Arguendo, the AI's current level of intelligence does not suffice to [prevent the programmers from editing its utility function](https://arbital.com/p/), despite the convergent instrumental incentive to avoid this; nor does it successfully [deceive](https://arbital.com/p/10f) the programmers.)
- The AI gets slightly smarter.  New conceivable options enter the AI's option space.
- The AI starts wanting to administer cocaine to humans (instead of heroin).
- The programmers read through the current schedule of prohibited drugs and add penalty terms for administering marijuana, cocaine, etcetera.
- The AI becomes slightly smarter.  New options enter its policy space.
- The AI starts thinking about how to research a new happiness drug not on the list of drugs that its utility function designates as bad.
- The programmers, after some work, manage to develop a category for 'The AI forcibly administering any kind of psychoactive drug to humans' which is broad enough that the AI stops suggesting research campaigns to develop things slightly outside the category.
- The AI wants to build an external system to administer heroin, so that it won't be classified inside this set of bad events "the AI forcibly administering drugs".
- The programmers generalize the penalty predicate to include "machine systems in general forcibly administering heroin" as a bad thing.
- The AI recalculates what it wants, and begins to want to pay humans to administer heroin.
- The programmers try to generalize the category of penalized events to include non-voluntarily administration of drugs in general that produce happiness, whether done by humans or AIs.  The programmers patch this category so that the AI is not trying to shut down at least the nicer parts of psychiatric hospitals.
- The AI begins planning an ad campaign to persuade people to use heroin voluntarily.
- The programmers add a penalty of -100 utilons for "AIs *persuading* humans to use drugs".
- The AI goes back to helping landladies take out the garbage.  All seems to be well.
- The AI continues to increase in intelligence, becoming capable enough that the AI can no longer be edited against its own will.
- The AI notices the option "Tweak human brains to express extremely high levels of endogenous opiates, then take care of their twitching bodies to so they can go on being happy".

The overall story is one where the AI's preferences on round $i,$ denoted $U_i,$ are observed to arrive at an attainable optimum $X_i$ which the humans see as undesirable.  The humans devise a penalty term $P_i$ intended to exclude the undesirable parts of the policy space, and add this to $U_i$ creating a new utility function $U_{i+1},$ after which the AI's optimal policy settles into a new state $X_i^*$ that seems acceptable.  However, after the next expansion of the policy space, $U_{i+1}$ settles into a new attainable optimum $X_{i+1}$ which is very similar to $X_i$ and makes the minimum adjustment necessary to evade the boundaries of the penalty term $P_i,$ requiring a new penalty term $P_{i+1}$ to exclude this new misbehavior.

(The end of this story *might* not kill you if the AI had enough successful, [advanced-safe](https://arbital.com/p/2l) [corrigibility features](https://arbital.com/p/45) that the AI would [indefinitely](https://arbital.com/p/2x) go on [checking](https://arbital.com/p/2qq) [novel](https://arbital.com/p/2qp) policies and [novel](https://arbital.com/p/2qp) goal instantiations with the users, not strategically hiding its disalignment from the programmers, not deceiving the programmers, letting the programmers edit its utility function, not doing anything disastrous before the utility function had been edited, etcetera.  But you wouldn't want to rely on this.  You would not want in the first place to operate on the paradigm of 'maximize happiness, but not via any of these bad methods that we have already excluded'.)

# Preconditions

Recurrence of a nearby unblocked strategy is argued to be a [foreseeable difficulty](https://arbital.com/p/6r) given the following preconditions:

• The AI is a [consequentialist](https://arbital.com/p/9h), or is conducting some other search such that when the search is blocked at $X,$ the search may happen upon a similar $X'$ that fits the same criterion that originally promoted $X.$  E.g. in an agent that selects actions on the basis of their consequences, if an event $X$ leads to goal $G$ but $X$ is blocked, then a similar $X'$ may also have the property of leading to $G.$

• The search is taking place over a [rich domain](https://arbital.com/p/9j) where the space of relevant neighbors around X is too complicated for us to be certain that we have described all the relevant neighbors correctly.  If we imagine an agent playing [the purely ideal game of logical Tic-Tac-Toe](https://arbital.com/p/9s), then if the agent's utility function hates playing in the center of the board, we can be sure (because we can exhaustively consider the space) that there are no Tic-Tac-Toe squares that behave strategically almost like the center but don't meet the exact definition we used of 'center'.  In the far more complicated real world, when you eliminate 'administer heroin' you are very likely to find some other chemical or trick that is strategically mostly equivalent to administering heroin.  See "[Almost all real-world domains are rich](https://arbital.com/p/RealIsRich)".

• From our perspective on [https://arbital.com/p/-55](https://arbital.com/p/-55), the AI does not have an [absolute identification of value](https://arbital.com/p/) for the domain, due to some combination of "the domain is rich" and "[value is complex](https://arbital.com/p/5l)".  Chess is complicated enough that human players can't absolutely identify winning moves, but since a chess program can have an absolute identification of which endstates constitute winning, we don't run into a problem of unending patches in identifying which states of the board are good play.  (However, if we consider a very early chess program that (from our perspective) was trying to be a consequentialist but wasn't very good at it, then we can imagine that, if the early chess program consistently threw its queen onto the right edge of the board for strange reasons, forbidding it to move the queen there might well lead it to throw the queen onto the left edge for the same strange reasons.)

# Arguments

## 'Nearest unblocked' behavior is sometimes observed in humans

Although humans obeying the law make poor analogies for mathematical algorithms, in some cases human economic actors expect not to encounter legal or social penalties for obeying the letter rather than the spirit of the law.  In those cases, after a previously high-yield strategy is outlawed or penalized, the result is very often a near-neighboring result that barely evades the letter of the law.  This illustrates that the theoretical argument also applies in practice to at least some pseudo-economic agents (humans), as we would expect given the stated preconditions.

## [Complexity of value](https://arbital.com/p/5l) means we should not expect to find a simple encoding to exclude detrimental strategies

To a human, 'poisonous' is one word.  In terms of molecular biology, the exact volume of the configuration space of molecules that is 'nonpoisonous' is very complicated.  By having a single word/concept for poisonous-vs.-nonpoisonous, we're *dimensionally reducing* the space of edible substances - taking a very squiggly volume of molecule-space, and mapping it all onto a linear scale from 'nonpoisonous' to 'poisonous'.

There's a sense in which human cognition implicitly performs dimensional reduction on our solution space, especially by simplifying dimensions that are relevant to some component of our values.  There may be some psychological sense in which we feel like "do X, only not weird low-value X" ought to be a simple instruction, and an agent that repeatedly produces the next unblocked weird low-value X is being perverse - that the agent, given a few examples of weird low-value Xs labeled as noninstances of the desired concept, ought to be able to just generalize to not produce weird low-value Xs.

In fact, if it were possible to [encode all relevant dimensions of human value into the agent](https://arbital.com/p/full_coverage) then we could just say *directly* to "do X, but not low-value X".  By the definition of [https://arbital.com/p/-full_coverage](https://arbital.com/p/-full_coverage), the agent's concept for 'low-value' includes everything that is actually of low [value](https://arbital.com/p/55), so this one instruction would blanket all the undesirable strategies we want to avoid.

Conversely, the truth of the [complexity of value thesis](https://arbital.com/p/5l) would imply that the simple word 'low-value' is dimensionally reducing a space of tremendous [algorithmic complexity](https://arbital.com/p/5v).  Thus the effort required to actually convey the relevant dos and don'ts of "X, only not weird low-value X" would be high, and a human-generated set of supervised examples labeled 'not the kind of X we mean' would be unlikely to cover and stabilize all the dimensions of the underlying space of possibilities.  Since the weird low-value X cannot be eliminated in one instruction or several patches or a human-generated set of supervised examples, the [https://arbital.com/p/-42](https://arbital.com/p/-42) problem will recur incrementally each time a patch is attempted and then the policy space is widened again.

# Consequences

[https://arbital.com/p/42](https://arbital.com/p/42) being a [foreseeable difficulty](https://arbital.com/p/6r) is a major contributor to worrying that short-term incentives in AI development, to get today's system working today, or to have today's system not exhibiting any immediately visible problems today, will not lead to advanced agents which are [safe after undergoing significant gains in capability](https://arbital.com/p/2l).

More generally, [https://arbital.com/p/-42](https://arbital.com/p/-42) is a [foreseeable](https://arbital.com/p/6r) reason why saying "Well just exclude X" or "Just write the code to not X" or "Add a penalty term for X" doesn't solve most of the issues that crop up in AI alignment.

Even more generally, this suggests that we want AIs to operate inside a space of [conservative categories containing actively whitelisted strategies and goal instantiations](https://arbital.com/p/2qp), rather than having the AI operate inside a (constantly expanding) space of all conceivable policies minus a set of blacklisted categories.

Eric Bruylant
Eliezer Yudkowsky
Alexei Andreev

Summary: Nearest Unblocked Strategy is a hypothetical source of [https://arbital.com/p/-48](https://arbital.com/p/-48) in the [alignment problem](https://arbital.com/p/5s) for [advanced agents](https://arbital.com/p/2c) that search [rich solution spaces](https://arbital.com/p/9j).  If an agent's [preference framework](https://arbital.com/p/5f) is [patched](https://arbital.com/p/48) to try to block a possible solution that seems undesirable, the next-best solution found may be the most similar solution that technically avoids the block.  This kind of patching seems especially likely to lead to a [change](https://arbital.com/p/-context) where a patch appears [beneficial](https://arbital.com/p/3d9) in a narrow option space, but proves detrimental after increased intelligence opens up more options.


Intelligence explosion

An "intelligence explosion" is what happens if a machine intelligence has [fast, consistent returns on investing work into improving its own cognitive powers, over an extended period](https://intelligence.org/files/IEM.pdf).  This would most stereotypically happen because it became able to optimize its own cognitive software, but could also apply in the case of "invested cognitive power in seizing all the computing power on the Internet" or "invested cognitive power in cracking the protein folding problem and then built nanocomputers".


Artificial General Intelligence

An "Artificial General Intelligence" is a machine intelligence possessed of some form of the same ["significantly more generally applicable" intelligence](https://arbital.com/p/7vh) that distinguishes humans from our nearest chimpanzee relatives.  A bee builds hives, a beaver builds dams, a human looks at both and imagines a dam with honeycomb structure.  We can drive cars or do algebra, even though no similar problems were found in our environment of evolutionary adaptedness, and we haven't had time to evolve further to match cars or algebra.  The brain's algorithms are sufficiently general, sufficiently cross-domain, that we can learn a tremendous variety of new [domains](https://arbital.com/p/7vf) within our lifetimes.  We are not perfectly general - we have an easier time learning to walk than learning to do abstract calculus, even though the latter is much easier in an objective sense.  But we're sufficiently general that we can figure out Special Relativity and engineer skyscrapers despite our not having those abilities at "compile time".  An Artificial General Intelligence has the same property; it can learn a tremendous variety of domains, including domains it had no inkling of when it was switched on.


Advanced nonagent

A standard agent:

 1. Observes reality
 2. Uses its observations to build a model of reality
 3. Uses its model to forecast the effects of possible actions or policies
 4. Chooses among policies on the basis of its [utility function](https://arbital.com/p/109) over the consequences of those policies
 5. Carries out the chosen policy

(...and then observes the actual results of its actions, and updates its model, and considers new policies, etcetera.)

It's conceivable that a cognitively powerful program could carry out some, but not all, of these activities.  We could call this an "advanced pseudoagent" or "advanced nonagent".

# Example: Planning Oracle

Imagine that we have an [Oracle](https://arbital.com/p/6x) agent which outputs a plan $\pi_0$ which is meant to maximize lives saved or [eudaimonia](https://arbital.com/p/) etc., *assuming* that the human operators decide to carry out the plan.  By hypothesis, the agent does not assess the probability that the plan will be carried out, or try to maximize the probability that the plan will be carried out.

We could look at this as modifying step 4 of the loop: rather than this pseudoagent selecting the output whose expected consequences optimize its utility function, it selects the output that optimizes utility *assuming* some other event occurs (the humans deciding to carry out the plan).

We could also look at the whole Oracle schema as interrupting step 5 of the loop.  If the Oracle works as [intended](https://arbital.com/p/6h), its purpose is not to immediately output optimized actions into the world; rather it is meant to output plans for humans to carry out.  This though is more of a metaphorical or big-picture property.  If not for the modification of step four where the Oracle calculates $\mathbb E [| \operatorname{do}](https://arbital.com/p/U)$ instead of $\mathbb E [| \operatorname{do}](https://arbital.com/p/U),$ the Oracle's outputted plans would just *be* its actions within the agent schema above.  (And it would optimize the general effects of its plan-outputting actions, including the problem of getting the humans to carry out the plans.)

# Example: Imitation-based agents

[Imitation-based agents](https://arbital.com/p/2sj) would modify steps 3 and 4 of the loop by "trying to output an action indistinguishable from the output of the human imitated" rather than forecasting consequences or optimizing over consequences, except perhaps insofar as forecasting consequences is important for guessing what the human would do, or they're internally imitating a human mode of thought that involves mentally imagining the consequences and choosing between them.  "Imitation-based agents" might justly be called pseudoagents, in this schema.

(But the "pseudoagent" terminology is relatively new, and a bit awkward, and it won't be surprising if we all go on saying "imitation-based agents" or "act-based agents".  The point of having terms like 'pseudoagent' or 'advanced nonagent' is to have a name for the general concept, not to [reserve and guard](https://arbital.com/p/10l) the word 'agent' for only 100% real pure agents.)

# Safety benefits and difficulties

Advanced pseudoagents and nonagents are usually proposed in the hope of averting some [advanced safety issue](https://arbital.com/p/2l) that seems to arise from the *agenty* part of "[advanced agency](https://arbital.com/p/2c)", while preserving other [advanced cognitive powers](https://arbital.com/p/2c) that seem [useful](https://arbital.com/p/6y).

A proposal like this can fail to the extent that it's not pragmatically possible to unentangle one aspect of agency from another; or to the extent that removing that much agency would make the AI [safe but useless](https://arbital.com/p/42k).

Some hypothetical examples that would, if they happened, constitute cases of failed safety or [unworkable tradeoffs](https://arbital.com/p/42k) in pseudoagent compromises:

• Somebody proposes to obtain an Oracle merely in virtue of only giving the AI a text output channel, and only taking what it says as suggestions, thereby interrupting the loop between the agent's policies and it acting in the world.  If this is all that changes, then from the Oracle's perspective it's still an agent, its text output is its motor channel, and it still immediately outputs whatever act it expects to maximize subjective expected utility, treating the humans as part of the environment to be optimized.  It's an agent that somebody is trying to *use* as part of a larger process with an interrupted agent loop, but the AI design itself is a pure agent.

• Somebody advocates for designing an AI that [only computes and outputs probability estimates](https://arbital.com/p/1v4); and never searches for any EU-maximizing policies, let alone outputs them.  It turns out that this AI cannot well-manage its internal and reflective operations, because it can't use consequentialism to select the best thought to think next.  As a result, the AI design fails to bootstrap, or fails to work sufficiently well before competing AI designs that use internal consequentialism.  (Safe but useless, much like a rock.)

• Somebody advocates that an [imitative agent design](https://arbital.com/p/2sj) will avoid invoking the [advanced safety issues that seem like they should be associated with consequentialist reasoning](https://arbital.com/p/2c), because the imitation-based pseudoagent never does any consequentialist reasoning or planning; it only tries to produce an output extremely similar to its training set of observed human outputs.  But it turns out (arguendo) that the pseudoagent, to imitate the human, has to imitate consequentialist reasoning, and so the implied dangers end up pretty much the same.

• An agent is supposed to just be an extremely powerful policy-reinforcement learner instead of an expected utility optimizer.  After a huge amount of optimization and mutation on a very general representation for policies, it turns out that the best policies, the ones that were the most reinforced by the highest rewards, are computing consequentialist models internally.  The actual result ends up being that the AI is doing consequentialist reasoning that is obscured and hidden, since it takes place outside the designed and easily visible high-level-loop of the AI.

Coming up with a proposal for an advanced pseudoagent, that still did something pivotal and was actually safer, would reasonably require: (a) understanding how to slice up agent properties along their natural joints; (b) understanding which advanced-agency properties lead to which expected safety problems and how; and (c) understanding which internal cognitive functions would be needed to carry out some particular [pivotal](https://arbital.com/p/6y) task; adding up to (d) see an exploitable prying-apart of the advanced-AI joints.

What's often proposed in practice is more along the lines of:

- "We just need to build AIs without emotions so they won't have drives that make them compete with us."  (Can you translate that into the language of utility functions and consequentialist planning, please?)
- "Let's just build an AI that answers human questions."  (It's doing a lot more than that internally, so how are the internal operations organized?  Also, [what do you do](https://arbital.com/p/6y) with a question-answering AI that averts the consequences of somebody else building a more agenty AI?)

Coming up with a sensible proposal for a pseudoagent is hard.  The reason for talking about "agents" in talking about future AIs isn't because the speaker wants to give AIs lots of power and have them wandering the world doing whatever they like under their own drives (for this entirely separate concept see [autonomous AGI](https://arbital.com/p/1g3)).  The reason we talk about observe-model-predict-act expected-utility consequentialists, is that this seems to carve a lot of important concepts at their joints.  Some alternative proposals exist, but they often have a feel of "carving against the joints" or trying to push through an unnatural arrangement, and aren't as natural or as simple to describe.


Safe but useless

"This type of safety implies uselessness" (or conversely, "any AI powerful enough to be useful will still be unsafe") is an accusation leveled against a proposed AI safety measure that must, to make the AI safe, be enforced to the point that it will make the AI useless.

For a non-AI metaphor, consider a scissors and its dangerous blades.  We can have a "safety scissors" that is only *just* sharp enough to cut paper - but this is still sharp enough to do some damage if you work at it.  If you try to make the scissors *even safer* by encasing the dangerous blades in foam rubber, the scissors can't cut paper any more.  If the scissors *can* cut paper, it's still unsafe.  Maybe you could in principle cut clay with a scissors like that, but this is no defense unless you can tell us [something very useful](https://arbital.com/p/6y) that can be done by cutting clay.

Similarly, there's an obvious way to try cutting down the allowed output of an [Oracle AGI](https://arbital.com/p/6x) to the point where [all it can do is tell us that a given theorem is provable from the axioms of Zermelo-Fraenkel set theory](https://arbital.com/p/70).  This [might](https://arbital.com/p/2j) prevent the AGI from hacking the human operators into letting it out, since all that can leave the box is a single yes-or-no bit, sent at some particular time.  An untrusted superintelligence inside this scheme would have the option of strategically not telling us when a theorem *is* provable in ZF; but if the bit from the proof-verifier said that the input theorem was ZF-provable, we could very likely trust that.

But now we run up against the problem that nobody knows how to [actually save the world](https://arbital.com/p/6y) by virtue of sometimes knowing for sure that a theorem is provable in ZF.  The scissors has been blunted to where it's probably completely safe, but can only cut clay; and nobody knows how to [do *enough* good](https://arbital.com/p/6y) by cutting clay.

# Ideal models of "safe but useless" agents

Should you have cause to do a mathematical study of this issue, then an excellent [ideal model](https://arbital.com/p/107) of a safe but useless agent, embodying maximal safety and minimum usefulness, would be a rock.

Summary:
Arguendo, when some particular proposed AI safety measures is alleged to be inherently opposed to the useful work the AI is meant to do.

We could use the metaphor of a scissors and its dangerous blades.  We can have a "safety scissors" that is only *just* sharp enough to cut paper, but this is still sharp enough to do some damage if you work at it.  If you make the scissors *even safer* by encasing the dangerous blades in foam rubber, the scissors can't cut paper any more; and if it *can* cut paper, it's still unsafe.  Maybe you can cut clay, but nobody knows how to do a [sufficiently large amount of good](https://arbital.com/p/6y) by cutting clay.

Similarly, there's an obvious way to cut down the output of an [Oracle AGI](https://arbital.com/p/6x) to the point where [all it can do is tell us that a proposed theorem is provable from the axioms of Zermelo-Fraenkel set theory](https://arbital.com/p/70).  Unfortunately, nobody knows how to use a ZF provability oracle to [save the world](https://arbital.com/p/6y).


Needs summary

By default, a page's [summary](https://arbital.com/p/1kl) (that shows up in the popover when a user hovers over a [greenlink](https://arbital.com/p/17f)) is the page's first paragraph. Sometimes the first paragraph is not well-suited for a popover summary. Add this tag to pages that need to have a custom summary.

Some pages may require an [accessible](https://arbital.com/p/5cb), [technical](https://arbital.com/p/4pt), or [brief](https://arbital.com/p/4q2) summary.

Jaime Sevilla Molina
Alexei Andreev
Mark Chimes
Eric Rogstad
Eric Bruylant
Nate Soares
Dylan Hendrickson
Team Arbital
Patrick Stevens
Joe Zeng


Missing the weird alternative

The "[https://arbital.com/p/47](https://arbital.com/p/47)" problem is alleged to be a foreseeable difficulty of coming up with a [good](https://arbital.com/p/3d9) goal for an [AGI](https://arbital.com/p/42g) (part of the [alignment problem](https://arbital.com/p/2v) for [advanced agents](https://arbital.com/p/2c)).  Roughly, an "unforeseen maximum" happens when somebody thinks that "produce smiles" would be a great goal for an AGI, because you can produce lots of smiles by making people happy, and making people happy is good.  However, while it's true that making people happy by ordinary means will produce *some* smiles, what will produce even *more* smiles is administering regular doses of heroin or turning all matter within reach into tiny molecular smileyfaces.

"Missing the weird alternative" is an attempt to [psychologize](https://arbital.com/p/43h) about why people talking about AGI utility functions might make this kind of oversight systematically.  To avoid [Bulverism](https://arbital.com/p/43k), if you're not yet convinced that missing a weird alternative *would* be a dangerous oversight, please read [https://arbital.com/p/47](https://arbital.com/p/47) first or instead.

In what follows we'll use $U$ to denote a proposed utility function for an AGI, $V$ to denote our own [normative values](https://arbital.com/p/55), $\pi_1$ to denote the high-$V$ policy that somebody thinks is the attainable maximum of $U,$ and $\pi_0$ to denote what somebody else suggests is a higher-$U$ lower-$V$ alternative.

# Alleged historical cases

Some historical instances of AGI goal systems proposed in a publication or conference presentation, that have been argued to be "missing the weird alternative" are:

- "Just program AIs to maximize their gains in compression of sensory data."  Proposed by Juergen Schmidhuber, director of IDSIA, in a presentation at the 2009 Singularity Summit; see the entry on [https://arbital.com/p/47](https://arbital.com/p/47).
  - Claimed by Schmidhuber to motivate art and science.
  - Yudkowsky suggested that this would, e.g., motivate the AI to construct objects that encrypted streams of 1s or 0s, then revealed the encryption key to the AI.
- Program an AI by showing it pictures/video of smiling faces to train (via supervised learning) which sensory events indicate good outcomes.  Formally proposed twice, once by J. Storrs Hall in the book *Beyond AI,* once in an ACM paper by somebody who since exercised their [sovereign right to change their mind](https://arbital.com/p/43l).
  - Claimed to motivate an AI to make people happy.
  - Suggested by Yudkowsky to motivate tiling the universe with tiny molecular smileyfaces.

Many other instances of this alleged issue have allegedly been spotted in more informal dicussion.

# Psychologized reasons to miss a weird alternative

[Psychologizing](https://arbital.com/p/43h) some possible reasons why some people might systematically "miss the weird alternative", assuming that was actually happening:

## Our brain doesn't bother searching V-bad parts of policy space

Arguendo:  The human brain is built to implicitly search for high-$V$ ways to accomplish a goal.  Or not actually high-$V$, but high-$W$ where $W$ is what we intuitively want, which [has something to do with](https://arbital.com/p/313) $V.$   "Tile the universe with tiny smiley-faces" is low-$W$ so doesn't get considered.

Arguendo, your brain is built to search for policies *it* prefers.  If you were looking for a way to open a stuck jar, your brain wouldn't generate the option of detonating a stick of dynamite, because that would be a policy ranked very low in your preference-ordering.  So what's the point of searching that part of the policy space?

This argument seems to [prove too much](https://arbital.com/p/3tc) in that it suggests that a chess player would be unable to search for their opponent's most preferred moves, if human brains could only search for policies that were high inside their own preference ordering.  But there could be an explicit perspective-taking operation required, and somebody modeling an AI they had warm feelings about might fail to fully take the AI's perspective; that is, they fail to carry out an explicit cognitive step needed to switch off the "only $W$-good policies" filter.

We might also have a *limited* native ability to take perspectives on goals not our own.  I.e., without further training, our brain can readily imagine that a chess opponent wants us to lose, or imagine that an AI wants to kill us because it hates us, and consider "reasonable" policy options along those lines.  But this expanded policy search still fails to consider policies on the lines of "turn everything into tiny smileyfaces" when asking for ways to produce smiles, because *nobody* in the ancestral environment would have wanted that option and so our brain has a hard time natively modeling it.

## Our brain doesn't automatically search weird parts of policy space

Arguendo:  The human brain doesn't search "weird" (generalization-violating) parts of the policy space without an explicit effort.

The potential issue here is that "tile the galaxy with tiny smileyfaces" or "build environmental objects that encrypt streams of 1s or 0s, then reveal secrets" would be *weird* in the sense of violating generalizations that usually hold about policies or consequences in human experience.  Not generalizations like, "nobody wants smiles smaller than an inch", but rather, "most problems are not solved with tiny molecular things".

[https://arbital.com/p/2w](https://arbital.com/p/2w) would tend to push the maximum (attainable optimum) of $U$ in "weird" or "extreme" directions - e.g., the *most* smiles can be obtained by making them very small, if this variable is not otherwise constrained.  So the unforeseen maxima might tend to violate implicit generalizations that usually govern most goals or policies and that our brains take for granted.  Aka, the unforeseen maximum isn't considered/generated by the policy search, because it's weird.

## Conflating the helpful with the optimal

Arguendo:  Someone might simply get as far as "$\pi_1$ increases $U$" and then stop there and conclude that a $U$-agent does $\pi_1.$

That is, they might just not realize that the argument "an advanced agent optimizing $U$ will execute policy $\pi_1$" requires "$\pi_1$ is the best way to optimize $U$" and not just "ceteris paribus, doing $\pi_1$ is better for $U$ than doing nothing".  So they don't realize that establishing "a $U$-agent does $\pi_1$" requires establishing that no other $\pi_k$ produces higher expected $U$.  So they just never search for a $\pi_k$ like that.

They might also be implicitly modeling $U$-agents as only weakly optimizing $U$, and hence not seeing a $U$-agent as facing tradeoffs or opportunity costs; that is, they implicitly model a $U$-agent as having no desire to produce any more $U$ than $\pi_1$ produces.  Again psychologizing, it does sometimes seem like people try to mentally model a $U$-agent as "an agent that sorta wants to produce some $U$ as a hobby, so long as nothing more important comes along" rather "an agent whose action-selection criterion entirely consists of doing whatever action is expected to lead to the highest $U$".

This would well-reflect the alleged observation that people allegedly "overlooking the weird alternative" seem more like they failed to search at all, than like they conducted a search but couldn't think of anything.

## Political persuasion instincts on convenient instrumental strategies

If the above hypothetical was true - that people just hadn't thought of the possibility of higher-$U$ $\pi_k$ existing - then we'd expect them to quickly change their minds upon this being pointed out.  Actually, it's been empirically observed that there seems to be a lot more resistance than this.

One possible force that could produce resistance to the observation "$\pi_0$ produces more $U$" - over and above the null hypothesis of ordinary pushback in argument, admittedly sometimes a very powerful force on its own - might be a brain running in a mode of "persuade another agent to execute a strategy $\pi$ which is convenient to me, by arguing to the agent that $\pi$ best serves the agent's own goals".  E.g. if you want to persuade your boss to give you a raise, one would be wise to argue "you should give me a raise because it will make this project more efficient" rather than "you should give me a raise because I like money".  By the general schema of the political brain, we'd be very likely to have built-in support for searching for arguments that policy $\pi$ that we just happen to like, is a *great* way to achieve somebody else's goal $U.$

Then on the same schema, a competing policy $\pi_0$ which is *better* at achieving the other agent's $U$, but less convenient for us than $\pi_1$, is an "enemy soldier" in the political debate.  We'll automatically search for reasons why $\pi_0$ is actually really bad for $U$ and $\pi_1$ is actually really good, and feel an instinctive dislike of $\pi_0.$  By the standard schema on the self-deceptive brain, we'd probably convince ourselves that $\pi_0$ is really bad for $U$ and $\pi_1$ is really best for $U.$  It would not be advantageous to our persuasion to go around noticing ourselves all the reasons that $\pi_0$ is good for $U.$  And we definitely wouldn't start spontaneously searching for $\pi_k$ that are $U$-better than $\pi_1,$ once we'd already found some $\pi_1$ that was very convenient to us.

(For a general post on the "fear of third alternatives", see [here](http://lesswrong.com/lw/hu/the_third_alternative/).  This essay also suggests that a good test for whether you might be suffering from "fear of third alternatives" is to ask yourself whether you instinctively dislike or automatically feel skeptical of any proposed other options for achieving the stated criterion.)

## The [apple pie problem](https://arbital.com/p/apple_pie_problem)

Sometimes people propose that the only utility function an AGI needs is $U$, where $U$ is something very good, like democracy or freedom or [apple pie](https://arbital.com/p/apple_pie_problem).

In this case, perhaps it sounds like a good thing to say about $U$ that it is the only utility function an AGI needs; and refusing to agree with this is *not* praising $U$ as highly as possible, hence an enemy soldier against $U.$

Or: The speaker may not realize that "$U$ is really quite amazingly fantastically good" is not the same proposition as "an agent that maximizes $U$ and nothing else is [beneficial](https://arbital.com/p/3d9)", so they treat contradictions of the second statement as though they contradicted the first.

Or: Pointing out that $\pi_0$ is high-$U$ but low-$V$ may sound like an argument against $U,$ rather than an observation that apple pie is not the only good.  "A universe filled with nothing but apple pie has low value" is not the same statement as "apple pie is bad and should not be in our utility function".

If the "apple pie problem" is real, it seems likely to implicitly rely on or interact with some of the other alleged problems.  For example, someone may not realize that their own complex values $W$ contain a number of implicit filters $F_1, F_2$ which act to filter out $V$-bad ways of achieving $U,$ because they themselves are implicitly searching only for high-$W$ ways of achieving $U.$


Psychologizing

"Psychologizing" is when we go past arguing that a position *is in fact* wrong, and start considering *what could have gone wrong in people's minds* to make them believe something so wrong.  Realistically, we can't entirely avoid psychologizing - sometimes cognitive biases are real and understanding how they potentially apply is important.  Nonetheless, since psychologizing is potentially a pernicious and poisonous activity, Arbital's discussion norms say to explicitly tag or greenlink with "[https://arbital.com/p/43h](https://arbital.com/p/43h)" each place where you speculate about the psychology of how people could possibly be so wrong.  Again, psychologizing isn't necessarily wrong - sometimes people do make mistakes for systematic psychological reasons that are understandable - but it's dangerous, potentially degenerates extremely quickly, and deserves to be highlighted each and every time.

If you go off on an extended discourse about why somebody is mentally biased or thinking poorly, *before* you've argued on the object level that they are in fact mistaken about the subject matter, this is [Bulverism](https://en.wikipedia.org/wiki/Bulverism).  Bulverism is flatly not okay on Arbital.  If you want to speculate about what goes wrong in the minds of people who believe in UFOs, then a good way of doing this on Arbital would be to *first* link a page arguing that there are in fact no UFOs, and say "I think this page establishes there are no UFOs; if you're not familiar with the arguments, read that first.  Given that I think there are in fact no UFOs, I will now [psychologize](https://arbital.com/p/43h) about why people would mistakenly believe in UFOs..."

If that clear label makes your page less snappy, persuasive, and fun to read, please consider that to be an *intended effect* of this rule.  Psychologizing *is* fun and that's why we want to take steps against it coming to dominate discussions here.


Don't try to solve the entire alignment problem

On first approaching the [alignment problem](https://arbital.com/p/2v) for [advanced agents](https://arbital.com/p/2c), aka "[robust](https://arbital.com/p/2l) and [beneficial](https://arbital.com/p/3d9) [AGI](https://arbital.com/p/42g)", aka "[Friendly AI](https://arbital.com/p/1g2)", a very common approach is to try to come up with *one* idea that solves *all* of [AI alignment](https://arbital.com/p/2v).  A simple design concept; a simple utility function; a simple development strategy; one guideline for everyone to adhere to; or a large diagram full of boxes with lines to other boxes; that is allegedly sufficient to realize around as much [benefit](https://arbital.com/p/55) from beneficial [superintelligences](https://arbital.com/p/41l) as can possibly be realized.

Without knowing the details of your current idea, this article can't tell you why it's wrong - though frankly we've got a strong [prior](https://arbital.com/p/1rm) against it at this point.  But some very standard advice would be:

- Glance over what current discussants think of as [standard challenges and difficulties of the overall problem](https://arbital.com/p/2l), i.e., why people think the alignment might be hard, and what standard questions a new approach would face.
- Consider focusing your attention down on a [single subproblem](https://arbital.com/p/4m) of alignment, and trying to make progress there - not necessarily solve it completely, but contribute nonobvious knowledge about the problem that wasn't there before.  (If you have a broad new approach that solves all of alignment, maybe you could walk through *exactly* how it solves one [crisply identified subproblem](https://arbital.com/p/2mx)?)
- Check out the flaws in previous proposals that people currently think won't work.  E.g. various versions of [utility indifference](https://arbital.com/p/1b7).

A good initial goal is not "persuade everyone in the field to agree with a new idea" but rather "come up with a contribution to an open discussion that is sufficiently crisply stated that, if it were in fact wrong, it would be possible for somebody else to shoot it down today."  I.e., an idea such that if you're wrong, this can be pointed out in the form of a crisply derivable consequence of a crisply specified idea, rather than it taking 20 years to see what happens.  For there to be sustained progress, propositions need to be stated modularly enough and crisply enough that there can be a conversation about them that goes beyond "does not / does too" - ideas need to be stated in forms that have sufficiently clear and derivable consequences that if there's a problem, people can see the problem and agree on it.

Alternatively, [poke a clearly demonstrable flaw in some solution currently being critiqued](https://arbital.com/p/3nj).  Since most proposals in alignment theory get shot down, trying to participate in the critiquing process has a great advantage over trying to invent solutions, in that you'll probably have started with the true premise "proposal X is broken or incomplete" rather than the false premise "proposal X works and solves everything".

[https://arbital.com/p/43h](https://arbital.com/p/43h) a little about why people might try to solve all of alignment theory in one shot, one might recount Robyn Dawes's advice that:

- Research shows that people come up with better solutions when they discuss the problem as thoroughly as possible before discussing any answers.
- Dawes has observed that people seem *more* likely to violate this principle as the problem becomes more difficult.

...and finally remark that building a nice machine intelligence correctly on the first try must be pretty darned difficult, since so many people solve it in the first 15 seconds.

It's possible that everyone working in this field is just missing the obvious and that there *is* some simple idea which solves all the problems.  But realistically, you should be aware that everyone in this field has already heard a dozen terrible Total Solutions, and probably hasn't had anything fun happen as a result of discussing them, resulting in some amount of attentional fatigue.  (Similarly:  If not everyone believes you, or even if it's hard to get people to listen to your solution instead of talking with people they already know, that's not necessarily because of some [deep-seated psychological problem](https://arbital.com/p/43h) on their part, such as being uninterested in outsiders' ideas.  Even if you're not an obvious crank, people are still unlikely to take the time out to engage with you unless you signal awareness of what *they* think are the usual issues and obstacles.  It's not so different here from other fields.)

Summary:
Rather than trying to solve the entire problem of [building nice Artificial Intelligences](https://arbital.com/p/2v) with a sufficiently right idea:

- Focus on a single, crisply stated subproblem.  (If your idea solves all of alignment theory, you should also be able to walk through a much smaller-scale example of how it solves one [open problem](https://arbital.com/p/4m), right?)
- Make one non-obvious new statement about a subproblem of alignment theory, such that if you were in fact wrong, it would be possible to figure this out now rather than in 20 years.
  - E.g., your new statement must be clear enough that its further consequences can be derived in a way that current researchers can agree on and have sustained conversations about it.
  - Or, less likely, that it's possible to test right now with current algorithms, in such a way that [if this kind of AI would blow up later then it would also blow up right now](https://arbital.com/p/6q).
- Glance over what current workers in the field consider to be [standard difficulties and challenges](https://arbital.com/p/2l), so that you can explain to them in their own language why your theory doesn't fail for the usual reasons.