Ontology identification problem: Technical tutorial

The problem of [ontology identification](https://arbital.com/p/5c) is the problem of loading a goal into an [advanced agent](https://arbital.com/p/2c) when that agent's representation of the world is likely to change in ways [unforeseen in the development phase](https://arbital.com/p/5d).  This tutorial focuses primarily on explaining what the problem is and why it is a [foreseeable difficulty](https://arbital.com/p/6r); for the corresponding research problems, see [the main page on Ontology Identification](https://arbital.com/p/5c).

This is a technical tutorial, meaning that it assumes some familiarity with [value alignment theory](https://arbital.com/p/2v), the [value identification problem](https://arbital.com/p/6c), and [safety thinking for advanced agents](https://arbital.com/p/2l).

To isolate ontology identification from other parts of the value identification problem, we consider a simplified but still very difficult problem: to state an unbounded program implementing a [diamond maximizer](https://arbital.com/p/5g) that will turn as much of the physical universe into diamond as possible.  The goal of "making diamonds" was chosen to have a crisp-seeming definition for our universe: namely, the amount of diamond is the number of carbon atoms covalently bound to four other carbon atoms.  Since it seems that in this case our [intended goal](https://arbital.com/p/6h) should be crisply definable relative to our universe's physics, we can avert many other issues of trying to identify [complex values](https://arbital.com/p/5l) to the agent.  Ontology identification is a difficulty that still remains even in this case - the agent's representation of 'carbon atoms' may still change over time.

## Introduction: Two sources of representational unpredictability

Suppose we wanted to write a hand-coded, [object-level](https://arbital.com/p/5t) utility function that evaluated the amount of diamond material present in the AI's model of the world.  We might foresee the following two difficulties:

1.  Where exactly do I find 'carbon atoms' inside the AI's model of the world?  As the programmer, all I see are these mysterious ones and zeroes, and the only parts that directly correspond to events I understand is the represention of the pixels in the AI's webcam... maybe I can figure out where the 'carbon' concept is by showing the AI graphite, buckytubes, and a diamond on its webcam and seeing what parts get activated... whoops, looks like the AI just revised its internal representation to be more computationally efficient, now I once again have no idea what 'carbon' looks like in there.  How can I make my hand-coded utility function re-bind itself to 'carbon' each time the AI revises its model's representation of the world?

2.  What exactly is 'diamond'?  If you say it's a nucleus with six protons, what's a proton?  If you define a proton as being made of quarks, what if there are unknown other particles underlying quarks?  What if the Standard Model of physics is incomplete or wrong - can we state exactly and formally what constitutes a carbon atom when we aren't certain what the underlying quarks are made of?

Difficulty 2 probably seems more exotic than the first, but Difficulty 2 is easier to explain in a formal sense and turns out to be a simpler way to illustrate many of the key issues that also appear in Difficulty 1.  We can see Difficulty 2 as the problem of binding an [intended goal](https://arbital.com/p/6h) to an unknown territory, and Difficulty 1 as the problem of binding an intended goal to an unknown map.  So the first step of the tutorial will be to walk through how Difficulty 2 (what exactly is a diamond?) might result in weird behavior in an [unbounded agent](https://arbital.com/p/107) intended to be a diamond maximizer.

## Try 1: Hacking AIXI to maximize diamonds?

The classic unbounded agent - an agent using far more computing power than the size of its environment - is [AIXI](https://arbital.com/p/11v).  Roughly speaking, AIXI considers all computable hypotheses for how its environment might work - all possible Turing machines that would turn AIXI's outputs into AIXI's future inputs.  (The finite variant AIXI-tl has a hypothesis space that includes all Turing machines that can be specified using fewer than $l$ bits and run in less than time $t$.)

From the perspective of AIXI, any Turing machine that takes one input tape and produces two output tapes is a "hypothesis about the environment", where the input to the Turing machine encodes AIXI's hypothetical action, and the outputs are interpreted as a prediction about AIXI's sensory data and AIXI's reward signal.  (In Marcus Hutter's formalism, the agent's reward is a separate sensory input to the agent, so hypotheses about the environment also make predictions about sensed rewards).  AIXI then behaves as a [Bayesian predictor](https://arbital.com/p/) that uses [algorithmic complexity](https://arbital.com/p/5v) to give higher [prior probabilities](https://arbital.com/p/) to simpler hypotheses (that is, Turing machines with fewer states and smaller state transition diagrams), and updates its mix of hypotheses based on sensory evidence (which can confirm or disconfirm the predictions of particular Turing machines).

As a decision agent, AIXI always outputs the motor action that leads to the highest predicted reward, assuming that the environment is described by the updated probability mixture of all Turing machines that could represent the environment (and assuming that future iterations of AIXI update and choose similarly).

The ontology identification problem shows up sharply when we imagine trying to modify AIXI to "maximize expectations of diamonds in the outside environment" rather than "maximize expectations of sensory reward signals".  As a [Cartesian agent](https://arbital.com/p/), AIXI has sharply defined sensory inputs and motor outputs, so we can have a [probability mixture](https://arbital.com/p/) over all Turing machines that relate motor outputs to sense inputs (as crisply represented in the input and output tapes).  But even if some otherwise arbitrary Turing machine happens to predict sensory experiences extremely well, how do we look at the state and working tape of that Turing machine to evaluate 'the amount of diamond' or 'the estimated number of carbon atoms bound to four other carbon atoms'?  The highest-weighted Turing machines that have best predicted the sensory data so far, presumably contain *some* sort of representation of the environment, but we have no idea how to get 'the number of diamonds' out of it.

(Example:  Maybe one Turing machine that is producing good sequence predictions inside AIXI, actually does so by simulating a large universe, identifying a superintelligent civilization that evolves inside that universe, and motivating that civilization to try to intelligently predict future future bits from past bits (as provided by some intervention).  To write a formal utility function that could extract the 'amount of real diamond in the environment' from arbitrary predictors in the above case , we'd need the function to read the Turing machine, decode that universe, find the superintelligence, decode the superintelligence's thought processes, find the concept (if any) resembling 'diamond', and hope that the superintelligence had precalculated how much diamond was around in the outer universe being manipulated by AIXI.)

This is, in general, the reason why the AIXI family of architectures can only contain agents defined to maximize direct functions of their sensory input, and not agents that behave so as to optimize facts about their external environment.  (We can't make AIXI maximize diamonds by making it want *pictures* of diamonds because then it will just, e.g., [build an environmental subagent that seizes control of AIXI's webcam and shows it pictures of diamonds](https://arbital.com/p/).  If you ask AIXI to show itself sensory pictures of diamonds, you can get it to show its webcam lots of pictures of diamonds, but this is not the same thing as building an environmental diamond maximizer.)

## Try 2: Unbounded agent using classical atomic hypotheses?

Given the origins of the above difficulty, we next imagine constraining the agent's hypothesis space to something other than "literally all computable functions from motor outputs to sense inputs", so that we can figure out how to find diamonds or carbon inside the agent's representation of the world.

As an [unrealistic example](https://arbital.com/p/):  Suppose someone was trying to define 'diamonds' to the AI's utility function.  Suppose they knew about atomic physics but not nuclear physics.  Suppose they build an AI which, during its development phase, learns about atomic physics from the programmers, and thus builds a world-model that is based on atomic physics.

Again for purposes of [unrealistic examples](https://arbital.com/p/), suppose that the AI's world-model is encoded in such fashion that when the AI imagines a molecular structure - represents a mental image of some molecules - then carbon atoms are represented as a particular kind of basic element of the representation.  Again, as an [unrealistic example](https://arbital.com/p/), imagine that there are [little LISP tokens](https://arbital.com/p/) representing environmental objects, and that the environmental-object-type of carbon-objects is encoded by the integer 6.  Imagine also that each atom, inside this representation, is followed by a list of the other atoms to which it's covalently bound.  Then when the AI is imagining a carbon atom participating in a diamond, inside the representation we would see an object of type 6, followed by a list containing exactly four other 6-objects.

Can we fix this representation for all hypotheses, and then write a utility function for the AI that counts the number of type-6 objects that are bound to exactly four other type-6 objects?  And if we did so, would the result actually be a diamond maximizer?

### AIXI-atomic

As a first approach to implementing this idea - an agent whose hypothesis space is constrained to models that directly represent all the carbon atoms - imagine a variant of AIXI-tl that, rather than considering all tl-bounded Turing machines, considers all simulated atomic universes containing up to 10^100 particles spread out over up to 10^50 light-years.  In other words, the agent's hypotheses are universe-sized simulations of classical, pre-nuclear models of physics; and these simulations are constrained to a common representation, so a fixed utility function can look at the representation and count carbon atoms bound to four other carbon atoms.  Call this agent AIXI-atomic.

(Note that AIXI-atomic, as an [unbounded agent](https://arbital.com/p/107), may use far more computing power than is embodied in its environment.  For purposes of the thought experiment, assume that the universe contains exactly one hypercomputer that runs AIXI-atomic.)

A first difficulty is that universes composed only of classical atoms are not good explanations of our own universe, even in terms of surface phenomena; e.g. the [ultraviolet catastrophe](http://en.wikipedia.org/wiki/Ultraviolet_catastrophe).  So let it be supposed that we have simulation rules for classical physics that replicate at least whatever phenomena the programmers have observed at [development time](https://arbital.com/p/), even if the rules have some seemingly ad-hoc elements (like there being no ultraviolent catastrophes).  We will *not* however suppose that the programmers have discovered all experimental phenomena we now see as pointing to nuclear or quantum physics.

A second difficulty is that a simulated universe of classical atoms does not identify where in the universe the AIXI-atomic agent resides, or say how to match the types of AIXI-atomic's sense inputs with the underlying behaviors of atoms.  We can elide this difficulty by imagining that AIXI-atomic simulates classical universes containing a single hypercomputer, and that AIXI-atomic knows a simple function from each simulated universe onto its own sensory data (e.g., it knows to look at the simulated universe, and translate simulated photons impinging on its webcam onto predicted webcam data in the standard format).  This elides most of the problem of [naturalized induction](https://arbital.com/p/).

So the AIXI-atomic agent that is hoped to maximize diamond:

- Considers only hypotheses that directly represent universes as huge systems of classical atoms, so that the function 'count atoms bound to four other carbon atoms' can be directly run over any possible future the agent models.
- Assigns probabilistic priors over these possible atomic representations of the universe, favoring representations that are in some sense simpler.
- Somehow [maps each atomic-level representation onto the agent's predicted sensory experiences](https://arbital.com/p/).
- [Bayes-updates its priors](https://arbital.com/p/) based on actual sensory experiences, the same as classical AIXI.
- Can evaluate the 'expected diamondness on the next turn' of a single action by looking at all hypothetical universes where that action is performed, weighted by their current probability, and summing over the expectation of 'carbon atoms bound to four other carbon atoms' after some unit amount of time has passed.
- Can evaluate the 'future expected diamondness' of an action, over some finite time horizon, by assuming that its future self will also Bayes-update and maximize expected diamondness over that time horizon.
- On each turn, outputs the action with greatest expected diamondness over some finite time horizon.

Suppose our own real universe was amended to otherwise be exactly the same, but contain a single [impermeable](https://arbital.com/p/) hypercomputer.  Suppose we defined an agent like the one above, using simulations of 1910-era models of physics, and ran that agent on the hypercomputer.  Should we expect the result to be an actual diamond maximizer - expect that the outcome of running this program on a single hypercomputer would indeed be that most mass in our universe would be turned into carbon and arranged into diamonds?

### Anticipated failure: AIXI-atomic tries to 'maximize outside the simulation'

In fact, our own universe isn't atomic, it's nuclear and quantum-mechanical.  This means that AIXI-atomic does not contain any hypotheses in its hypothesis space that *directly represent* our universe.  By the previously specified hypothesis of the thought experiment, AIXI-atomic's model of simulated physics was built to encompass all the experimental phenomena the programmers had yet discovered, but there were some quantum and nuclear phenomena that AIXI-atomic's programmers had not yet discovered.  When those phenomena are discovered, there will be no simple explanation on the direct terms of the model.

Intuitively, of course, we'd like AIXI-atomic to discover the composition of nuclei, shift its models to use nuclear physics, and refine the 'carbon atoms' mentioned in its utility function to mean 'atoms with nuclei containing six protons'.

But we didn't actually specify that when constructing the agent (and saying how to do it in general is, so far as we know, hard; in fact it's the whole ontology identification problem).  We constrained the hypothesis space to contain only universes running on the classical physics that the programmers knew about.  So what happens instead?

Probably the 'simplest atomic hypothesis that fits the facts' will be an enormous atom-based computer, *simulating* nuclear physics and quantum physics in order to create a simulated non-classical universe whose outputs are ultimately hooked up to AIXI's webcam.  From our perspective this hypothesis seems silly, but if you restrict the hypothesis space to only classical atomic universes, that's what ends up being the computationally simplest hypothesis that predicts, in detail, the results of nuclear and quantum experiments.

AIXI-atomic will then try to choose actions so as to maximize the amount of expected diamond inside the probable *outside universes* that could contain the giant atom-based simulator of quantum physics.  It is not obvious what sort of behavior this would imply.

### Metaphor for difficulty: AIXI-atomic cares about only fundamental carbon

One metaphorical way of looking at the problem is that AIXI-atomic was implicitly defined to care only about diamonds made out of *ontologically fundamental* carbon atoms, not diamonds made out of quarks.  A probability function that assigns 0 probability to all universes made of quarks, and a utility function that outputs a constant on all universes made of quarks, [yield functionally identical behavior](https://arbital.com/p/).  So it is an exact metaphor to say that AIXI-atomic only *cares* about universes with ontologically basic carbon atoms, given that AIXI-atomic's hypothesis space only contains universes with ontologically basic carbon atoms.

Imagine that AIXI-atomic's hypothesis space does contain many other universes with other laws of physics, but its hand-coded utility function just returns 0 on those universes since it can't find any 'carbon atoms' inside the model.  Since AIXI-atomic only cares about diamond made of fundamental carbon, when AIXI-atomic discovers the experimental data implying that almost all of its probability mass should reside in nuclear or quantum universes in which there were no fundamental carbon atoms, AIXI-atomic stops caring about the effect its actions have on the vast majority of probability mass inside its model.  Instead AIXI-atomic tries to maximize inside the tiny remaining probabilities in which it *is* inside a universe with fundamental carbon atoms that is somehow reproducing its sensory experience of nuclei and quantum fields... for example, a classical atomic universe containing a computer simulating a quantum universe and showing the results to AIXI-atomic.

From our perspective, we failed to solve the 'ontology identification problem' and get the real-world result we [intended](https://arbital.com/p/6h), because we tried to define the agent's *utility function* over properties of a universe made out of atoms, and the real universe turned out to be made of quantum fields.  This caused the utility function to *fail to bind* to the agent's representation in the way we intuitively had in mind.

Today we do know about quantum mechanics, so if we tried to build a diamond maximizer using some bounded version of the above formula, it might not fail on account of [the particular exact problem](https://arbital.com/p/48) of atomic physics being false.

But perhaps there are discoveries still remaining that would change our picture of the universe's ontology to imply something else underlying quarks or quantum fields.  Human beings have only known about quantum fields for less than a century; our model of the ontological basics of our universe has been stable for less than a hundred years of our human experience.  So we should seek an AI design that does not assume we know the exact, true, fundamental ontology of our universe during an AI's [development phase](https://arbital.com/p/5d).

As another important metaphorical case in point, consider a human being who feels angst on contemplating a universe in which "By convention sweetness, by convention bitterness, by convention color, in reality only atoms and the void" (Democritus); someone who wonders where there is any room in this collection of lifeless particles for love, free will, or even the existence of people.  Since, after all, people are just *mere* collections of atoms.  This person can be seen as undergoing an ontology identification problem: they don't know how to find the objects of value in a representation containing atoms instead of ontologically basic people.

Human beings simultaneously evolved a particular set of standard mental representations (e.g., a representation for colors in terms of a 3-dimensional subjective color space) along with evolving emotions that bind to these representations ([identification of flowering landscapes as beautiful](http://en.wikipedia.org/wiki/Evolutionary_aesthetics#Landscape_and_other_visual_arts_preferences).  When someone visualizes any particular configuration of 'mere atoms', their built-in desires don't automatically fire and bind to that mental representation, the way they would bind to the brain's native representation of the environment.  Generalizing that no set of atoms can be meaningful (since no abstract configuration of 'mere atoms' they imagine, seems to trigger any emotions to bind to it) and being told that reality is composed entirely of such atoms, they feel they've been told that the true state of reality, underlying appearances, is a meaningless one.

## The utility rebinding problem

Intuitively, we would think it was [common sense](https://arbital.com/p/) for an agent that wanted diamonds to react to the experimental data identifying nuclear physics, by deciding that a carbon atom is 'really' a nucleus containing six protons.  We can imagine this agent [common-sensically](https://arbital.com/p/) updating its model of the universe to a nuclear model, and redefining the 'carbon atoms' that its old utility function counted to mean 'nuclei containing exactly six protons'.  Then the new utility function could evaluate outcomes in the newly discovered nuclear-physics universe.  The problem of producing this desirable agent behavior is the **utility rebinding problem**.

To see why this problem is nontrivial, consider that the most common form of carbon is C-12, with nuclei composed of six protons and six neutrons.  The second most common form of carbon is C-14, with nuclei composed of six protons and eight neutrons.  Is C-14 *truly* carbon - is it the sort of carbon that can participate in valuable diamonds of high utility?  Well, that depends on your utility function, obviously; and from a human perspective it just sounds arbitrary.

But consider a closely analogous question from a humanly important perspective: Is a chimpanzee truly a person?  Where the question means not, "How do we arbitrarily define the syllables per-son?" but "Should we care a lot about chimpanzees?", i.e., how do we define the part of our preferences that care about people, to the possibly-person edge cases of chimpanzees?

If you live in a world where chimpanzees haven't been discovered, you may have an easy time running your utility function over your model of the environment, since the objects of your experience classify sharply into the 'person' and 'nonperson' categories.  Then you discover chimpanzees, and they're neither typical people (John Smith) nor typical nonpeople (like rocks).

We can see the force of this question as arising from something like an ontological shift: we're used to valuing cognitive systems that are made from whole human minds, but it turns out that minds are made of parts, and then we have the question of how to value things that are made from some of the person-parts but not all of them... sort of like the question of how to treat carbon atoms that have the usual number of protons but not the usual number of neutrons.

Chimpanzees definitely have neural areas of various sizes, and particular cognitive abilities - we can suppose the empirical truth is unambiguous at this level, and known to us.  So the question is then whether we regard a particular configuration of neural parts (a frontal cortex of a certain size) and particular cognitive abilities (consequentialist means-end reasoning and empathy, but no recursive language) as something that our 'person' category values... once we've rewritten the person category to value configurations of cognitive parts, rather than whole atomic people.

In fact, we run into this question as soon as we learn that human beings run on brains and the brains are made out of neural regions with functional properties; we can then *imagine* chimpanzees even if we haven't met any, and ask to what degree our preferences should treat this edge-person as deserving of moral rights.  If we can 'rebind' our emotions and preferences to live in a world of nuclear brains rather than atomic people, this rebinding will *implicitly* say whether or not a chimpanzee is a person, depending on how our preference over brain configurations treats the configuration that is a chimpanzee.

In this sense the problem we face with chimpanzees is exactly analogous to the question a diamond maximizer would face after discovering nuclear physics and asking itself whether a carbon-14 atom counted as 'carbon' for purposes of caring about diamonds.  Once a diamond maximizer knows about neutrons, it can see that C-14 is chemically like carbon and forms the same kind of chemical bonds, but that it's heavier because it has two extra neutrons.  We can see that chimpanzees have a similar brain architectures to the sort of people we always considered before, but that they have smaller frontal cortexes and no ability to use recursive language, etcetera.

Without knowing more about the diamond maximizer, we can't guess what sort of considerations it might bring to bear in deciding what is Truly Carbon and Really A Diamond.  But the breadth of considerations human beings need to invoke in deciding how much to care about chimpanzees, is one way of illustrating that the problem of rebinding a utility function to a shifted ontology is [https://arbital.com/p/value-laden](https://arbital.com/p/value-laden) and can potentially undergo [https://arbital.com/p/excursions](https://arbital.com/p/excursions) into [complex desiderata](https://arbital.com/p/5l).  Redefining a [moral category](https://arbital.com/p/) so that it talks about the underlying parts of what were previously seen as all-or-nothing atomic objects, may carry an implicit ruling about how to value many kinds of [https://arbital.com/p/edge-case](https://arbital.com/p/edge-case) objects that were never seen before.

It's possible that some formal part of this problem could be usefully carved out from the complex value-laded edge-case-reclassification part.  E.g., how would you redefine carbon as C12 if there were no other isotopes?  How would you rebind the utility function to *at least* C12?  In general, how could edge cases be [identified and queried](https://arbital.com/p/) by an [online Genie](https://arbital.com/p/6w)?

### Reappearance on the reflective level

An obvious thought (especially for [online Genies](https://arbital.com/p/6w)) is that if the AI is unsure about how to reinterpret its goals in light of a shifting mental representation, it should query the programmers.

Since the definition of a programmer would then itself be baked into the [preference framework](https://arbital.com/p/5f), the problem might [reproduce itself on the reflective level](https://arbital.com/p/) if the AI became unsure of where to find 'programmers':  "My preference framework said that programmers were made of carbon atoms, but all I can find in this universe are quantum fields!"

Thus the ontology identification problem is arguably one of the [critical subproblems](https://arbital.com/p/) of value alignment: it plausibly has the property that, if botched, it could potentially [crash the error recovery mechanism](https://arbital.com/p/).

## Diamond identification in multi-level maps

A realistic, [bounded diamond maximizer](https://arbital.com/p/5g) wouldn't represent the outside universe with atomically detailed or quantum-detailed models.  Instead, a bounded agent would have some version of a [multi-level map](https://arbital.com/p/) of the world in which the agent knew in principle that things were composed of atoms, but didn't model most things in atomic detail.  A bounded agent's model of an airplane would have wings, or wing shapes, rather than atomically detailed wings.  It would think about wings when doing aerodynamic engineering, atoms when doing chemistry, nuclear physics when doing nuclear engineering, and definitely not try to model everything in its experience down to the level of quantum fields.

At the present, there are not yet any proposed formalisms for how to do probability theory with multi-level maps (in other words: [nobody has yet put forward a guess at how to solve the problem even given infinite computing power](https://arbital.com/p/)).  But it seems very likely that, if we did know what multi-level maps looked like formally, it might suggest a formal solution to non-value-laden utility-rebinding.

E.g., if an agent already has a separate high-level concept of 'diamond' that's bound to a lower-level concept of 'carbon atoms bound to four other carbon atoms', then maybe when you discover nuclear physics, the multi-level map itself would tend to suggest that 'carbon atoms' be re-bound to 'nuclei with six protons' or 'nuclei with six protons and six neutrons'.  It might at least be possible to phrase the equivalent of a prior or mixture of weightings for how the utility function would re-bind itself, and say, "Given this prior, care about whatever that sparkly hard stuff 'diamond' ends up binding to on the lower level."

Unfortunately, we have very little formal probability theory to describe how a multi-level map would go from 'that unknown sparkly hard stuff' to 'carbon atoms bound to four other carbon atoms in tetrahedral patterns, which is the only known repeating pattern for carbon atoms bound to four other carbon atoms' to 'C12 and C14 are chemically identical but C14 is heavier'.  This being the case, we don't know how to say anything about a dynamically updating multi-level map inside a [preference framework](https://arbital.com/p/5f).

If we were actually trying to build a diamond maximizer, we would be likely to encounter this problem long before it started formulating new physics.  The equivalent of a computational discovery that changes 'the most efficient way to represent diamonds' is likely to happen much earlier than a physical discovery that changes 'what underlying physical systems probably constitute a diamond'.

This also means that we are liable to face the ontology identification problem long before the agent starts discovering new physics, as soon as it starts revising its representation.  Only very unreflective agents with strongly fixed-in-place representations for every part of the environment that we think the agent is supposed to care about, would let the ontology identification problem be elided entirely.  Only *very* not-self-modifying agents, or [Cartesian agents](https://arbital.com/p/) with goals formulated only over sense data, would not confront their programmers with ontology identification problems.

## Research paths

More of these are described in the [main article on ontology identification](https://arbital.com/p/5c).  But here's a quick list of some relevant research subproblems and avenues:

* Transparent priors.  Priors that are constrained to meaningful hypothesis spaces that the utility function knows how to interpret.  Rather than all Turing machines being hypotheses, we could have only causal models being hypotheses, and then preference frameworks that talked about 'the cause of' labeled sensory data could read the hypotheses.  (Note that the space of causal models can be Turing-complete, in the sense of being able to embed any Turing machine as a causal system.  So we'd be able to explain any computable sense data in terms of a causal model - we wouldn't sacrifice any explanatory power by restricting ourselves to 'causal models' instead of 'all Turing machines'.)

* Reductionist identifications.  Being able to go hunting, inside the current model of an environment, for a thingy that looks like it's made out of type-1 thingies bound to four other type-1 thingies, where a type-1 thingy is itself made out of six type-2, six type-3, and six type-4 thingies (6 electrons, 6 protons, 6 neutrons).

* Causal identifications.  Some variation on trying to identify diamonds as the causes of pictures of diamonds, for some data set of things labeled as diamonds or non-diamonds.  This doesn't work immediately because then it's not clear whether "the cause" of the picture is the photons reflecting off the diamond, the diamond itself, the geological pressures that produced the diamond, the laws of physics, etcetera.  But perhaps some crossfire of identification could pin down the 'diamond' category inside a causal model, by applying some formal rule to several sets of the right sort of labeled sense data.  As an open problem:  If an agent has a rich causal model that includes categories like 'diamond' somewhere unknown, and you can point to labeled sensory datasets and use casual and categorical language, what labeled datasets and language would unambiguously identify diamonds, and no other white sparkly things, even if the resulting concept of 'diamond' was being [subject to maximization](https://arbital.com/p/2w)?  (Note that under this approach, as with any preference framework that talks about the causes of sensory experiences, we need to worry about [Christiano's Hack](https://arbital.com/p/5j).)

* Ambiguity resolution.  Detect when an ontology identification is ambiguous, and refer the problem to the user/programmer.  At our present stage of knowledge this seems like pretty much the same problem as [inductive ambiguity resolution](https://arbital.com/p/).

* Multi-level maps.  Solve the problem of bounded agents having maps of the world that operate at multiple, interacting reductionist levels, as designed to save on computing power.  Then solve ontology identification by initially binding to a higher level of the map, and introducing some rule for re-binding as the map updates.  Note that multi-level mapping is an [AGI rather than FAI problem](https://arbital.com/p/), meaning that work here [should perhaps be classified](https://arbital.com/p/).

* Solution for [non-self-modifying Genies](https://arbital.com/p/6w).  Try to state a 'hack' solution to ontology identification that would work for an AI running on fixed algorithms where a persistent knowledge representation is known at development time.

## Some implications

The ontology identification problem is one more reason to believe that [hard-coded object-level utility functions should be avoided](https://arbital.com/p/) and that [value identification in general is hard](https://arbital.com/p/).

Ontology identification is heavily entangled with AGI problems, meaning that some research on ontology identification [may need to be non-public](https://arbital.com/p/).  This is an example instance of the argument that [at least some VAT research may need to be non-public](https://arbital.com/p/), based on that [at least some AGI research is better off non-public](https://arbital.com/p/).


Value identification problem

The subproblem category of [value alignment](https://arbital.com/p/5s) which deals with pinpointing [valuable](https://arbital.com/p/55) outcomes to an [advanced agent](https://arbital.com/p/2c) and distinguishing them from non-valuable outcomes.  E.g., the [Edge Instantiation](https://arbital.com/p/2w) and [Ontology Identification](https://arbital.com/p/5c) problems are argued to be [foreseeable difficulties](https://arbital.com/p/6r) of value identification.  A central foreseen difficulty of value identification is [Complexity of Value](https://arbital.com/p/5l).

Eric Bruylant
Eliezer Yudkowsky
Alexei Andreev


Goodhart's Curse

summary(Gloss):  Goodhart's Curse is a neologism for the combination of the Optimizer's Curse and Goodhart's Law.  It states that a powerful agent neutrally optimizing a proxy measure U, meant to align with true values V, will implicitly tend to find upward divergences of U from V.

In other words: powerfully optimizing for a utility function is strongly liable to blow up anything we'd regard as an error in defining that utility function.

Goodhart's Curse is a neologism for the combination of the Optimizer's Curse and Goodhart's Law, particularly as applied to [the value alignment problem for Artificial Intelligences](https://arbital.com/p/5s).

Goodhart's Curse in this form says that a powerful agent neutrally optimizing a proxy measure U that we hoped to align with true values V, will implicitly seek out upward divergences of U from V.

In other words: powerfully optimizing for a utility function is strongly liable to blow up anything we'd regard as an error in defining that utility function.

# Winner's Curse, Optimizer's Curse, and Goodhart's Law

## Winner's Curse

The **[Winner's Curse](https://en.wikipedia.org/wiki/Winner%27s_curse)** in auction theory says that if multiple bidders all bid their [unbiased estimate](https://arbital.com/p/unbiased_estimator) of an item's value, the winner is likely to be someone whose estimate contained an upward error.

That is:  If we have lots of bidders on an item, and each bidder is individually unbiased *on average,* selecting the winner selects somebody who probably made a mistake this *particular* time and overbid.  They are likely to experience post-auction regret systematically, not just occasionally and accidentally.

For example, let's say that the true value of an item is \$10 to all bidders.  Each bidder bids the true value, \$10, plus some [Gaussian noise](https://arbital.com/p/gaussian_noise).  Each individual bidder is as likely to overbid \$2 as to underbid \$2, so each individual bidder's average expected bid is \$10; individually, their bid is an unbiased estimator of the true value.  But the *winning* bidder is probably somebody who overbid \$2, not somebody who underbid \$2.  So if we know that Alice won the auction, our [revised](https://arbital.com/p/1rp) guess should be that Alice made an upward error in her bid.

## Optimizer's Curse

The [Optimizer's Curse](https://faculty.fuqua.duke.edu/~jes9/bio/The_Optimizers_Curse.pdf) in decision analysis generalizes this observation to an agent that estimates the expected utility of actions, and executes the action with the highest expected utility.  Even if each utility estimate is locally unbiased, the action with seemingly highest utility is more likely, in our posterior estimate, to have an upward error in its expected utility.

Worse, the Optimizer's Curse means that actions with *high-variance estimates* are selected for.  Suppose we're considering 5 possible actions which in fact have utility \$10 each, and our estimates of those 5 utilities are Gaussian-noisy with a standard deviation of \$2.  Another 5 possible actions in fact have utility of -\$20, and our estimate of each of these 5 actions is influenced by unbiased Gaussian noise with a standard deviation of \$100.  We are likely to pick one of the bad five actions whose enormously uncertain value estimates happened to produce a huge upward error.

The Optimizer's Curse grows worse as a larger policy space is implicitly searched; the more options we consider, the higher the average error in whatever policy is selected.  To effectively reason about a large policy space, we need to either have a good prior over policy goodness and to know the variance in our estimators; or we need very precise estimates; or we need mostly correlated and little uncorrelated noise; or we need the highest real points in the policy space to have an advantage bigger than the uncertainty in our estimates.

The Optimizer's Curse is not exactly similar to the Winner's Curse because the Optimizer's Curse potentially applies to *implicit* selection over large search spaces.  Perhaps we're searching by gradient ascent rather than explicitly considering each element of an exponentially vast space of possible policies.  We are still implicitly selecting over some effective search space, and this method will still seek out upward errors.  If we're imperfectly estimating the value function to get the gradient, then gradient ascent is implicitly following and amplifying any upward errors in the estimator.

The proposers of the Optimizer's Curse also described a Bayesian remedy in which we have a prior on the expected utilities and variances and we are more skeptical of very high estimates.  This however assumes that the prior itself is perfect, as are our estimates of variance.  If the prior or variance-estimates contain large flaws somewhere, a search over a very wide space of possibilities would be expected to seek out and blow up any flaws in the prior or the estimates of variance.

## Goodhart's Law

[Goodhart's Law](https://en.wikipedia.org/wiki/Goodhart%27s_law) is named after the economist Charles Goodhart.  A standard formulation is "When a measure becomes a target, it ceases to be a good measure."  Goodhart's original formulation is "Any observed statistical regularity will tend to collapse when pressure is placed upon it for control purposes."

For example, suppose we require banks to have '3% capital reserves' as defined some particular way.  'Capital reserves' measured that particular exact way will rapidly become a much less good indicator of the stability of a bank, as accountants fiddle with balance sheets to make them legally correspond to the highest possible level of 'capital reserves'.

Decades earlier, IBM once paid its programmers per line of code produced.  If you pay people per line of code produced, the "total lines of code produced" will have even less correlation with real productivity than it had previously.

# Goodhart's Curse in alignment theory

**Goodhart's Curse** is a neologism (by Yudkowsky) for the crossover of the Optimizer's Curse with Goodhart's Law, yielding that **neutrally optimizing a proxy measure U of V seeks out upward divergence of U from V.**

Suppose the humans have true values V.  We try to convey these values to a [powerful AI](https://arbital.com/p/2c), via some value learning methodology that ends up giving the AI a utility function U.

Even if U is locally an unbiased estimator of V, optimizing U will seek out what *we* would regard as 'errors in the definition', places where U diverges upward from V.  Optimizing for a high U may implicitly seek out regions where U - V is high; that is, places where V is lower than U.  This may especially include regions of the outcome space or policy space where the value learning system was subject to great variance; that is, places where the value learning worked poorly or ran into a snag.

Goodhart's Curse would be expected to grow worse as the AI became more powerful.  A more powerful AI would be implicitly searching a larger space and would have more opportunity to uncover what we'd regard as "errors"; it would be able to find smaller loopholes, blow up more minor flaws.  There is a potential [context disaster](https://arbital.com/p/6q) if new divergences are uncovered as more of the possibility space is searched, etcetera.

We could see the genie as *implicitly* or *emergently* seeking out any possible loophole in the wish:  *Not* because it is an evil genie that knows our 'truly intended' V and is looking for some place that V can be minimized while appearing to satisfy U; but just because the genie is neutrally seeking out very large values of U and these are places where it is unusually likely that U diverged upward from V.

Many [foreseeable difficulties](https://arbital.com/p/6r) of AGI alignment interact with Goodhart's Curse.  Goodhart's Curse is one of the central reasons we'd expect 'little tiny mistakes' to 'break' when we dump a ton of optimization pressure on them.  Hence the [claim](https://arbital.com/p/1cv):  "AI alignment is hard like building a rocket is hard:  enormous pressures will break things that don't break in less extreme engineering domains."

## Goodhart's Curse and meta-utility functions

An obvious next question is "Why not just define the AI such that the AI itself regards U as an estimate of V, causing the AI's U to more closely align with V as the AI gets a more accurate empirical picture of the world?"

Reply:  Of course this is the obvious thing that we'd *want* to do.  But what if we make an error in exactly how we define "treat U as an estimate of V"?  Goodhart's Curse will magnify and blow up any error in this definition as well.

We must distinguish:

- V, the [true value function that is in our hearts](https://arbital.com/p/55).
- T, the external target that we formally told the AI to align on, where we are *hoping* that T really means V.
- U, the AI's current estimate of T or probability distribution over possible T.

U will converge toward T as the AI becomes more advanced.  The AI's epistemic improvements and learned experience will tend over time to eliminate a subclass of Goodhart's Curse where the current estimate of U-value has diverged upward *from T-value,* cases where the uncertain U-estimate was selected to be erroneously above the correct formal value T.

*However,* Goodhart's Curse will still apply to any potential regions where T diverges upward from V, where the formal target diverges from the true value function that is in our hearts.  We'd be placing immense pressure toward seeking out what we would retrospectively regard as human errors in defining the meta-rule for determining utilities. %note: That is, we'd retrospectively regard those as errors if we survived.%

## Goodhart's Curse and 'moral uncertainty'

"Moral uncertainty" is sometimes offered as a solution source in AI alignment; if the AI has a probability distribution over utility functions, it can be risk-averse about things that *might* be bad.  Would this not be safer than having the AI be very sure about what it ought to do?

Translating this idea into the V-T-U story, we want to give the AI a formal external target T to which the AI does not currently have full access and knowledge.  We are then hoping that the AI's uncertainty about T, the AI's estimate of the variance between T and U, will warn the AI away from regions where from our perspective U would be a high-variance estimate of V.  In other words, we're hoping that estimated U-T uncertainty correlates well with, and is a good proxy for, actual U-V divergence.

The idea would be that T is something like a supervised learning procedure from labeled examples, and the places where the current U diverges from V are things we 'forgot to tell the AI'; so the AI should notice that in these cases it has little information about T.

Goodhart's Curse would then seek out any flaws or loopholes in this hoped-for correlation between estimated U-T uncertainty and real U-V divergence.  Searching a very wide space of options would be liable to select on:

- Regions where the AI has made an epistemic error and poorly estimated the variance between U and T;
- Regions where the formal target T is solidly estimable to the AI, but from our own perspective the divergence from T to V is high (that is, the U-T uncertainty fails to *perfectly* cover all T-V divergences).

The second case seems especially likely to occur in future phases where the AI is smarter and has more empirical information, and has *correctly* reduced its uncertainty about its formal target T.  So moral uncertainty and risk aversion may not scale well to superintelligence as a means of warning the AI away from regions where we'd retrospectively judge that U/T and V had diverged.

Concretely:

You tell the AI that human values are defined relative to human brains in some particular way T.  While the AI is young and stupid, the AI knows that it is very uncertain about human brains, hence uncertain about T.  Human behavior is produced by human brains, so the AI can regard human behavior as informative about T; the AI is sensitive to spoken human warnings that killing the housecat is bad.

When the AI is more advanced, the AI scans a human brain using molecular nanotechnology and resolves all its moral uncertainty about T.  As we defined T, the optimum T turns out to be "feed humans heroin because that is what human brains maximally want".

Now the AI already knows everything our formal definition of T requires the AI to know about the human brain to get a very sharp estimate of U.  So human behaviors like shouting "stop!" are no longer seen as informative about T and don't lead to updates in U.

T, as defined, was always misaligned with V.  But early on, the misalignment was in a region where the young AI estimated high variance between U and T, thus keeping the AI out of this low-V region.  Later, the AI's empirical uncertainty about T was reduced, and this protective barrier of moral uncertainty and risk aversion was dispelled.

Unless the AI's moral uncertainty is *perfectly* conservative and *never* underestimates the true regions of U-V divergence, there will be some cases where the AI thinks it is morally sure even though from our standpoint the U-V divergence is large.  Then Goodhart's Curse would select on those cases.

Could we use a very *conservative* estimate of utility-function uncertainty, or a formal target T that is very hard for even a superintelligence to become certain about?

We would first need to worry that if the utility-function uncertainty is unresolvable, that means the AI can't ever obtain empirically strong evidence about it.  In this case the AI would not update its estimate of T from observing human behaviors, making the AI again insensitive to humans shouting "Stop!"

Another proposal would be to rely on risk aversion over *unresolvably* uncertain probabilities broad enough to contain something similar to the true V as a hypothesis, and hence engender sufficient aversion to low-true-V outcomes.  Then we should worry on a pragmatic level that a *sufficiently* conservative amount of moral uncertainty--so conservative that U-T risk aversion *never underestimated* the appropriate degree of risk aversion from our V-standpoint--would end up preventing the AI from acting *ever.*  Or that this degree of moral risk aversion would be such a pragmatic hindrance that the programmers might end up pragmatically bypassing all this inconvenient aversion in some set of safe-seeming cases.  Then Goodhart's Curse would seek out any unforeseen flaws in the coded behavior of 'safe-seeming cases'.

# Conditions for Goodhart's Curse

The exact conditions for Goodhart's Curse applying between V and a point estimate or probability distribution over U, have not yet been written out in a convincing way.

For example, suppose we have a multivariate normal distribution in which X and Y dimensions are positively correlated, only Y is observable, and we are selecting on Y in order to obtain more X.  While X will revert to the mean compared to Y, it's not likely to be zero or negative; picking maximum Y is our best strategy for obtaining maximum X and will probably obtain a very high X.  (Observation due to [https://arbital.com/p/111](https://arbital.com/p/111).)

Consider also the case of the [smile maximizer](https://arbital.com/p/10d) which we trained to optimize smiles as a proxy for happiness.  Tiny molecular smileyfaces are very low in happiness, an apparent manifestation of Goodhart's Curse.  On the otherwise, if we optimized for 'true happiness' among biological humans, this would produce more smiles than default.  It might be only a tiny fraction of possible smiles, on the order of 1e-30, but it would be more smiles than would have existed otherwise.  So the relation between V (maximized at 'true happiness', zero at tiny molecular smileyfaces) and U (maximized at tiny molecular smileyfaces, but also above average for true happiness) is not symmetric; and this is one hint to the unknown necessary and/or sufficient condition for Goodhart's Curse to apply.

In the case above, we might handwave something like, "U had lots of local peaks one of which was V, but the U of V's peak wasn't anywhere near the highest U-peak, and the highest U-peak was low in V.  V was more narrow and its more unique peak was noncoincidentally high in U."

# Research avenues

[Mild optimization](https://arbital.com/p/2r8) is a proposed avenue for direct attack on the central difficulty of Goodhart's Curse and all the other difficulties it exacerbates.  Obviously, if our formulation of mild optimization is not *perfect,* Goodhart's Curse may well select for any place where our notion of 'mild optimization' turns out to have a loophole that allows a lot of optimization.  But insofar as some version of mild optimization is working most of the time, it could avoid blowing up things that would otherwise blow up.  See also [Tasks](https://arbital.com/p/4mn).

Similarly, [conservative strategies](https://arbital.com/p/2qp) can be seen as a more indirect attack on some forms of Goodhart's Curse--we try to stick to a conservative boundary drawn around previously whitelisted instances of the goal concept, or to using strategies similar to previously whitelisted strategies.  This averts searching a much huger space of possibilities that would be more likely to contain errors somewhere.  But Goodhart's Curse might single out what constitutes a 'conservative' boundary, if our definition is less than absolutely perfect.

Eric Bruylant
Eliezer Yudkowsky
Rob Bensinger

Summary:
The **[Winner's Curse](https://en.wikipedia.org/wiki/Winner%27s_curse)** in auction theory says that if many individually fallible but unbiased bidders all compete in an auction, the winner has been selected to be unusually likely to have made an upward error in their bid.

The [Optimizer's Curse](https://faculty.fuqua.duke.edu/~jes9/bio/The_Optimizers_Curse.pdf) is that if we consider many possible courses of action, and pick the course of action that seems best, we are implicitly selecting for places where we're likely to have made an upward error in the estimate.  Worse, this means we're selecting for places where our unbiased estimate has high variance.

[Goodhart's Law](https://en.wikipedia.org/wiki/Goodhart%27s_law) says that whatever proxy measure an organization tries to control soon ceases to be a good proxy.  If you demand that banks have 3% 'capital reserves' defined a certain way, the bank will look for ways to get 'capital reserves' with a minimum of inconvenience, and this selects against 'capital reserves' that do what we wanted.

*Goodhart's Curse* is a neologism for the combination of the Optimizer's Curse with Goodhart's Law, particularly as applied to [value alignment of Artificial Intelligences](https://arbital.com/p/5s).

Suppose our true values are $V$; $V$ is the [true value function that is in our hearts](https://arbital.com/p/55).  If by any system or meta-system we try to align the AI's utility $U$ with $V,$ then even if our alignment procedure makes $U$ a generally unbiased estimator of $V,$ *heavily optimizing* expected $U$ is unusually likely to seek out places where $U$ poorly aligns with $V.$

Seeking out high values of $U$ implicitly seeks out high values of the divergence $U-V$ if any such divergence exists anywhere.  Worse, this implicitly seeks out places where the variance $\|U - V\|$ is generally high--places where we made an error in defining our meta-rules for alignment, some seemingly tiny mistake, a loophole.


Intended goal

Definition.  An "intended goal" refers to the intuitive intention in the mind of a human programmer when they executed some formal directive or goal within the AI.  For example, if the programmer wants to create worthwhile happiness and the AI ends up tiling the universe with tiny molecular smiley-faces, we would say that worthwhile happiness (in some intuitive, possibly pre-verbal sense existing in the programmer's mind) was the "intended goal", as distinct from the result of the formal utility function actually encoded in the AI (which proved to have a maximum at tiny molecular smiley-faces).


Arbital Labs

This is a temporary group created as an experiment in both software and community design. It's meant to act as a seed for multiple future Arbital communities. Currently membership is by invite only.