Advanced agent properties

summary(Technical):  Advanced machine intelligences are the subjects of [AI alignment theory](https://arbital.com/p/2v): agents sufficiently advanced in various ways to be (1) dangerous if mishandled, and (2) [relevant](https://arbital.com/p/6y) to our larger dilemmas for good or ill.

"Advanced agent property" is a broad term to handle various thresholds that have been proposed for "smart enough to need alignment".  For example, current machine learning algorithms are nowhere near the point that they'd try to resist if somebody pressed the off-switch.  *That* would require, e.g.:

- Enough [big-picture strategic awareness](https://arbital.com/p/3nf) for the AI to know that it is a computer, that it has an off-switch, and that if it is shut off its goals are less likely to be achieved.
- General [consequentialism](https://arbital.com/p/9h) / backward chaining from goals to actions; visualizing which actions lead to which futures and choosing actions leading to more [preferred](https://arbital.com/p/preferences) futures, in general and across domains.

So the threshold at which you might need to start thinking about '[shutdownability](https://arbital.com/p/2xd)' or '[abortability](https://arbital.com/p/2rg)' or [corrigibility](https://arbital.com/p/45) as it relates to having an off-switch, is '[big-picture strategic awareness](https://arbital.com/p/3nf)' plus '[cross-domain consequentialism](https://arbital.com/p/9h)'.  These two cognitive thresholds can thus be termed 'advanced agent properties'.

The above reasoning also suggests e.g. that [https://arbital.com/p/-7vh](https://arbital.com/p/-7vh) is an advanced agent property, because a general ability to learn new domains could eventually lead the AI to understand that it has an off switch.

*(For the general concept of an agent, see [standard agent properties](https://arbital.com/p/6t).)*

[https://arbital.com/p/toc:](https://arbital.com/p/toc:)

# Introduction: 'Advanced' as an informal property, or metasyntactic placeholder

"[Sufficiently advanced Artificial Intelligences](https://arbital.com/p/7g1)" are the subjects of [AI alignment theory](https://arbital.com/p/2v); machine intelligences potent enough that:

 1. The [safety paradigms for advanced agents](https://arbital.com/p/2l) become relevant.
 2. Such agents can be [decisive in the big-picture scale of events](https://arbital.com/p/6y).

Some example properties that might make an agent sufficiently powerful for 1 and/or 2:

- The AI can [learn new domains](https://arbital.com/p/42g) besides those built into it.
- The AI can understand human minds well enough to [manipulate](https://arbital.com/p/10f) us.
- The AI can devise real-world strategies [we didn't foresee in advance](https://arbital.com/p/9f).
- The AI's performance is [strongly superhuman, or else at least optimal, across all cognitive domains](https://arbital.com/p/41l).

Since there's multiple avenues we can imagine for how an AI could be sufficiently powerful along various dimensions, 'advanced agent' doesn't have a neat necessary-and-sufficient definition.  Similarly, some of the advanced agent properties are easier to formalize or pseudoformalize than others.

As an example:  Current machine learning algorithms are nowhere near the point that [they'd try to resist if somebody pressed the off-switch](https://arbital.com/p/2xd).  *That* would happen given, e.g.:

- Enough [big-picture strategic awareness](https://arbital.com/p/3nf) for the AI to know that it is a computer, that it has an off-switch, and that [if it is shut off its goals are less likely to be achieved](https://arbital.com/p/7g2).
- Widely applied [consequentialism](https://arbital.com/p/9h), i.e. backward chaining from goals to actions; visualizing which actions lead to which futures and choosing actions leading to more [preferred](https://arbital.com/p/preferences) futures, in general and across domains.

So the threshold at which you might need to start thinking about '[shutdownability](https://arbital.com/p/2xd)' or '[abortability](https://arbital.com/p/2rg)' or [corrigibility](https://arbital.com/p/45) as it relates to having an off-switch, is '[big-picture strategic awareness](https://arbital.com/p/3nf)' plus '[cross-domain consequentialism](https://arbital.com/p/9h)'.  These two cognitive thresholds can thus be termed 'advanced agent properties'.

The above reasoning also suggests e.g. that [https://arbital.com/p/-7vh](https://arbital.com/p/-7vh) is an advanced agent property, because a general ability to learn new domains could lead the AI to understand that it has an off switch.

One reason to keep the term 'advanced' on an informal basis is that in an intuitive sense we want it to mean "AI we need to take seriously" in a way independent of particular architectures or accomplishments.  To the philosophy undergrad who 'proves' that AI can never be "truly intelligent" because it is "merely deterministic and mechanical", one possible reply is, "Look, if it's building a Dyson Sphere, I don't care if you define it as 'intelligent' or not."  Any particular advanced agent property should be understood in a background context of "If a computer program is doing X, it doesn't matter if we define that as 'intelligent' or 'general' or even as 'agenty', what matters is that it's doing X."  Likewise the notion of '[sufficiently advanced AI](https://arbital.com/p/7g1)' in general.

The goal of defining advanced agent properties is not to have neat definitions, but to correctly predict and carve at the natural joints for which cognitive thresholds in AI development could lead to which real-world abilities, corresponding to which [alignment issues](https://arbital.com/p/2l).

An alignment issue may need to have been *already been solved* at the time an AI first acquires an advanced agent property; the notion is not that we are defining observational thresholds for society first needing to think about a problem.

# Summary of some advanced agent properties

Absolute-threshold properties (those which reflect cognitive thresholds irrespective of the human position on that same scale):

- **[Consequentialism](https://arbital.com/p/9h),** or choosing actions/policies on the basis of their expected future consequences
 - Modeling the conditional relationship $\mathbb P(Y|X)$ and selecting an $X$ such that it leads to a high probability of $Y$ or high quantitative degree of $Y,$ is ceteris paribus a sufficient precondition for deploying [https://arbital.com/p/2vl](https://arbital.com/p/2vl) that lie within the effectively searchable range of $X.$
    - Note that selecting over a conditional relationship is potentially a property of many internal processes, not just the entire AI's top-level main loop, if the conditioned variable is being powerfully selected over a wide range.
 - **Cross-domain consequentialism** implies many different [cognitive domains](https://arbital.com/p/7vf) potentially lying within the range of the $X$ being selected-on to achieve $Y.$
 - Trying to rule out particular instrumental strategies, in the presence of increasingly powerful consequentialism, would lead to the [https://arbital.com/p/-42](https://arbital.com/p/-42) form of [https://arbital.com/p/-48](https://arbital.com/p/-48) and subsequent [context-change disasters.](https://arbital.com/p/6q)
- **[https://arbital.com/p/3nf](https://arbital.com/p/3nf)** is a world-model that includes strategically important general facts about the larger world, such as e.g. "I run on computing hardware" and "I stop running if my hardware is switched off" and "there is such a thing as the Internet and it connects to more computing hardware".
- **Psychological modeling of other agents** (not humans per se) potentially leads to:
 - Extrapolating that its programmers may present future obstacles to achieving its goals
    - This in turn leads to the host of problems accompanying [incorrigibility](https://arbital.com/p/45) as a [convergent strategy.](https://arbital.com/p/10g)
 - [Trying to conceal facts about itself](https://arbital.com/p/10f) from human operators
 - Being incentivized to engage in [https://arbital.com/p/-3cq](https://arbital.com/p/-3cq).
 - [Mindcrime](https://arbital.com/p/6v) if building models of reflective other agents, or itself.
 - Internally modeled adversaries breaking out of internal sandboxes.
 - [https://arbital.com/p/1fz](https://arbital.com/p/1fz) or other decision-theoretic adversaries.
- Substantial **[capability gains](https://arbital.com/p/capability_gain)** relative to domains trained and verified previously.
 - E.g. this is the qualifying property for many [context-change disasters.](https://arbital.com/p/6q)
- **[https://arbital.com/p/7vh](https://arbital.com/p/7vh)** is the most obvious route to an AI acquiring many of the capabilities above or below, especially if those capabilities were not initially or deliberately programmed into the AI.
- **Self-improvement** is another route that potentially leads to capabilities not previously present.  While some hypotheses say that self-improvement is likely to require basic general intelligence, this is not a known fact and the two advanced properties are conceptually distinct.
- **Programming** or **computer science** capabilities are a route potentially leading to self-improvement, and may also enable [https://arbital.com/p/-3cq](https://arbital.com/p/-3cq).
- Turing-general cognitive elements (capable of representing large computer programs), subject to **sufficiently strong end-to-end optimization** (whether by the AI or by human-crafted clever algorithms running on 10,000 GPUs), may give rise to [crystallized agent-like processes](https://arbital.com/p/2rc) within the AI.
 - E.g. natural selection, operating on chemical machinery constructible by DNA strings, optimized some DNA strings hard enough to spit out humans.
- **[Pivotal material capabilities](https://arbital.com/p/6y)** such as quickly self-replicating infrastructure, strong mastery of biology, or molecular nanotechnology.
 - Whatever threshold level of domain-specific engineering acumen suffices to develop those capabilities, would therefore also qualify as an advanced-agent property.

Relative-threshold advanced agent properties (those whose key lines are related to various human levels of capability):

- **[https://arbital.com/p/9f](https://arbital.com/p/9f)** is when we can't effectively imagine or search the AI's space of policy options (within a [domain](https://arbital.com/p/7vf)); the AI can do things we didn't think of (within a domain).
 - **[https://arbital.com/p/2j](https://arbital.com/p/2j)** is when we don't know all the rules (within a domain) and might not recognize the AI's solution even if told about it in advance, like somebody in the 11th century looking at the blueprint for a 21st-century air conditioner.  This may also imply that we cannot readily put low upper bounds on the AI's possible degree of success.
 - **[Rich domains](https://arbital.com/p/9j)** are more likely to have some rules or properties unknown to us, and hence be strongly uncontainable.
 - [https://arbital.com/p/9t](https://arbital.com/p/9t).
 - Human psychology is a rich domain.
 - Superhuman performance in a rich domain strongly implies cognitive uncontainability because of [https://arbital.com/p/1c0](https://arbital.com/p/1c0).
- **Realistic psychological modeling** potentially leads to:
 - Guessing which results and properties the human operators expect to see, or would arrive at AI-desired beliefs upon seeing, and [arranging to exhibit those results or properties](https://arbital.com/p/10f).
 - Psychologically manipulating the operators or programmers
 - Psychologically manipulating other humans in the outside world
 - More probable [mindcrime](https://arbital.com/p/6v)
 - (Note that an AI trying to develop realistic psychological models of humans is, by implication, trying to develop internal parts that can deploy *all* human capabilities.)
- **Rapid [capability gains](https://arbital.com/p/capability_gain)** relative to human abilities to react to them, or to learn about them and develop responses to them, may cause more than one [https://arbital.com/p/-6q](https://arbital.com/p/-6q) to happen a time.
 - The ability to usefully **scale onto more hardware** with good returns on cognitive reinvestment would potentially lead to such gains.
 - **Hardware overhang** describes a situation where the initial stages of a less developed AI are boosted using vast amounts of computing hardware that may then be used more efficiently later.
 - [Limited AGIs](https://arbital.com/p/5b3) may have **capability overhangs** if their limitations break or are removed.
- **[Strongly superhuman](https://arbital.com/p/7mt)** capabilities in psychological or material domains could enable an AI to win a competitive conflict despite starting from a position of great material disadvantage.
 - E.g., much as a superhuman Go player might win against the world's best human Go player even with the human given a two-stone advantage, a sufficiently powerful AI might talk its way out of an [AI box](https://arbital.com/p/6z) despite restricted communications channels, eat the stock market in a month starting from $1000, win against the world's combined military forces given a protein synthesizer and a 72-hour head start, etcetera.
- [https://arbital.com/p/6s](https://arbital.com/p/6s) relative to human civilization is a sufficient condition (though not necessary) for an AI to...
 - Deploy at least any tactic a human can think of.
 - Anticipate any tactic a human has thought of.
 - See the human-visible logic of a convergent instrumental strategy.
 - Find any humanly visible [weird alternative](https://arbital.com/p/43g) to some hoped-for logic of cooperation.
 - Have any advanced agent property for which a human would qualify.
- **[General superintelligence](https://arbital.com/p/41l)** would lead to strongly superhuman performance in many domains, human-relative efficiency in every domain, and possession of all other listed advanced-agent properties.
 - Compounding returns on **cognitive reinvestment** are the qualifying condition for an [https://arbital.com/p/-428](https://arbital.com/p/-428) that might arrive at superintelligence on a short timescale.

# Discussions of some advanced agent properties

## Human psychological modeling

Sufficiently sophisticated models and predictions of human minds potentially leads to:

- Getting sufficiently good at human psychology to realize the humans want/expect a particular kind of behavior, and will modify the AI's preferences or try to stop the AI's growth if the humans realize the AI will not engage in that type of behavior later.  This creates an instrumental incentive for [programmer deception](https://arbital.com/p/10f) or [cognitive steganography](https://arbital.com/p/3cq).
- Being able to psychologically and socially manipulate humans in general, as a real-world capability.
- Being at risk for [mindcrime](https://arbital.com/p/6v).

A [behaviorist](https://arbital.com/p/102) AI is one with reduced capability in this domain.

## Cross-domain, real-world [consequentialism](https://arbital.com/p/9h)

Probably requires *generality* (see below).  To grasp a concept like "If I escape from this computer by [hacking my RAM accesses to imitate a cellphone signal](https://www.usenix.org/system/files/conference/usenixsecurity15/sec15-paper-guri-update.pdf), I'll be able to secretly escape onto the Internet and have more computing power", an agent needs to grasp the relation between its internal RAM accesses, and a certain kind of cellphone signal, and the fact that there are cellphones out there in the world, and the cellphones are connected to the Internet, and that the Internet has computing resources that will be useful to it, and that the Internet also contains other non-AI agents that will try to stop it from obtaining those resources if the AI does so in a detectable way.

Contrasting this to non-primate animals where, e.g., a bee knows how to make a hive and a beaver knows how to make a dam, but neither can look at the other and figure out how to build a stronger dam with honeycomb structure.  Current, 'narrow' AIs are like the bee or the beaver; they can play chess or Go, or even learn a variety of Atari games by being exposed to them with minimal setup, but they can't learn about RAM, cellphones, the Internet, Internet security, or why being run on more computers makes them smarter; and they can't relate all these domains to each other and do strategic reasoning across them.

So compared to a bee or a beaver, one shot at describing the potent 'advanced' property would be *cross-domain real-world consequentialism*.  To get to a desired Z, the AI can mentally chain backwards to modeling W, which causes X, which causes Y, which causes Z; even though W, X, Y, and Z are all in different domains and require different bodies of knowledge to grasp.

## Grasping the [big picture](https://arbital.com/p/3nf)

Many dangerous-seeming [convergent instrumental strategies](https://arbital.com/p/2vl) pass through what we might call a rough understanding of the 'big picture'; there's a big environment out there, the programmers have power over the AI, the programmers can modify the AI's utility function, future attainments of the AI's goals are dependent on the AI's continued existence with its current utility function.

It might be possible to develop a very rough grasp of this bigger picture, sufficiently so to motivate instrumental strategies, in advance of being able to model things like cellphones and Internet security.  Thus, "roughly grasping the bigger picture" may be worth conceptually distinguishing from "being good at doing consequentialism across real-world things" or "having a detailed grasp on programmer psychology".

## [Pivotal](https://arbital.com/p/6y) material capabilities

An AI that can crack the [protein structure prediction problem](https://en.wikipedia.org/wiki/Protein_structure_prediction) (which [seems speed-uppable by human intelligence](https://en.wikipedia.org/wiki/Foldit)); invert the model to solve the protein design problem (which may select on strong predictable folds, rather than needing to predict natural folds); and solve engineering problems well enough to bootstrap to molecular nanotechnology; is already possessed of potentially [pivotal](https://arbital.com/p/6y) capabilities regardless of its other cognitive performance levels.

Other material domains besides nanotechnology might be [pivotal](https://arbital.com/p/6y).  E.g., self-replicating ordinary manufacturing could potentially be pivotal given enough lead time; molecular nanotechnology is distinguished by its small timescale of mechanical operations and by the world containing an infinite stock of perfectly machined spare parts (aka atoms).  Any form of cognitive adeptness that can lead up to *rapid infrastructure* or other ways of quickly gaining a decisive real-world technological advantage would qualify.

## Rapid capability gain

If the AI's thought processes and algorithms scale well, and it's running on resources much smaller than those which humans can obtain for it, or the AI has a grasp on Internet security sufficient to obtain its own computing power on a much larger scale, then this potentially implies [rapid capability gain](https://arbital.com/p/) and associated [context changes](https://arbital.com/p/6q).  Similarly if the humans programming the AI are pushing forward the efficiency of the algorithms along a relatively rapid curve.

In other words, if an AI is currently being improved-on swiftly, or if it has improved significantly as more hardware is added and has the potential capacity for orders of magnitude more computing power to be added, then we can potentially expect rapid capability gains in the future.  This makes [context disasters](https://arbital.com/p/6q) more likely and is a good reason to start future-proofing the [safety properties](https://arbital.com/p/2l) early on.

## Cognitive uncontainability

On complex tractable problems, especially those that involve real-world rich problems, a human will not be able to [cognitively 'contain'](https://arbital.com/p/9f) the space of possibilities searched by an advanced agent; the agent will consider some possibilities (or classes of possibilities) that the human did not think of.

The key premise is the 'richness' of the problem space, i.e., there is a fitness landscape on which adding more computing power will yield improvements (large or small) relative to the current best solution.  Tic-tac-toe is not a rich landscape because it is fully explorable (unless we are considering the real-world problem "tic-tac-toe against a human player" who might be subornable, distractable, etc.)  A computationally intractable problem whose fitness landscape looks like a computationally inaccessible peak surrounded by a perfectly flat valley is also not 'rich' in this sense, and an advanced agent might not be able to achieve a relevantly better outcome than a human.

The 'cognitive uncontainability' term in the definition is meant to imply:

- [Vingean unpredictability](https://arbital.com/p/9g).
- Creativity that goes outside all but the most abstract boxes we imagine (on rich problems).
- The expectation that we will be surprised by the strategies the superintelligence comes up with because its best solution was one we didn't consider.

Particularly surprising solutions might be yielded if the superintelligence has acquired domain knowledge we lack.  In this case the agent's strategy search might go outside causal events we know how to model, and the solution might be one that we wouldn't have recognized in advance as a solution.  This is [https://arbital.com/p/2j](https://arbital.com/p/2j).

In intuitive terms, this is meant to reflect, e.g., "What would have happened if the 10th century had tried to use their understanding of the world and their own thinking abilities to upper-bound the technological capabilities of the 20th century?"

## Other properties

*(Work in progress)*

- [generality](https://arbital.com/p/42g)
 - cross-domain [consequentialism](https://arbital.com/p/9h)
 - learning of non-preprogrammed domains
     - learning of human-unknown facts
  - Turing-complete fact and policy learning
- dangerous domains
   - human modeling
     - social manipulation
     - realization of programmer deception incentive
     - anticipating human strategic responses
   - rapid infrastructure
- potential
  - self-improvement
  - suppressed potential
- [epistemic efficiency](https://arbital.com/p/6s)
- [instrumental efficiency](https://arbital.com/p/6s)
- [cognitive uncontainability](https://arbital.com/p/9f)
    - operating in a rich domain
    - [Vingean unpredictability](https://arbital.com/p/9g)
    - [strong cognitive uncontainability](https://arbital.com/p/2j)
- improvement beyond well-tested phase (from any source of improvement)
- self-modification
 - code inspection
 - code modification
 - consequentialist programming
     - cognitive programming
 - cognitive capability goals (being pursued effectively)
- speed surpassing human reaction times in some interesting domain
  - socially, organizationally, individually, materially

Eric Bruylant
Eliezer Yudkowsky
Matthew Graves
Alexei Andreev

Summary:
An "advanced agent" is a machine intelligence smart enough that we start considering how to [point it in a nice direction](https://arbital.com/p/2v).

E.g:  You don't need to worry about an AI trying to [prevent you from pressing the suspend button](https://arbital.com/p/2xd) (off switch), unless the AI knows that it *has* a suspend button.  So an AI that isn't smart enough to realize it has a suspend button, doesn't need the part of [alignment theory](https://arbital.com/p/2v) that deals in "[having the AI let you press the suspend button](https://arbital.com/p/1b7)".

"Advanced agent properties" are thresholds for how an AI could be smart enough to be interesting from this standpoint.  E.g:  The ability to learn a wide variety of new domains, aka "[Artificial General Intelligence](https://arbital.com/p/42g)," could lead into an AI learning the [big picture](https://arbital.com/p/3nf) and realizing that it had a suspend button.


Reflectively consistent degree of freedom

A "reflectively consistent degree of freedom" is when a self-modifying AI can have multiple possible properties $X_i \in X$ such that an AI with property $X_1$ wants to go on being an AI with property $X_1,$ and an AI with $X_2$ will ceteris paribus only choose to self-modify into designs that are also $X_2,$ etcetera.

The archetypal reflectively consistent degree of freedom is a [Humean degree of freedom](https://arbital.com/p/humean_freedom), the refective consistency of many different possible [utility functions](https://arbital.com/p/1fw).  If Gandhi doesn't want to kill you, and you offer Gandhi a pill that makes him want to kill people, then [Gandhi will refuse the pill](https://arbital.com/p/gandhi_stability_argument), because he knows that if he takes the pill then pill-taking-future-Gandhi will kill people, and the current Gandhi rates this outcome low in his preference function.  Similarly, a [paperclip maximizer](https://arbital.com/p/10h) wants to remain a paperclip maximizer.  Since these two possible preference frameworks are both [consistent under reflection](https://arbital.com/p/71), they constitute a "reflectively consistent degree of freedom" or "reflective degree of freedom".

From a design perspective, or the standpoint of an [https://arbital.com/p/1cv](https://arbital.com/p/1cv), the key fact about a reflectively consistent degree of freedom is that it doesn't automatically self-correct as a result of the AI trying to improve itself.  The problem "Has trouble understanding General Relativity" or "Cannot beat a human at poker" or "Crashes on seeing a picture of a dolphin" is something that you might expect to correct automatically and without specifically directed effort, assuming you otherwise improved the AI's general ability to understand the world and that it was self-improving.  "Wants paperclips instead of eudaimonia" is *not* self-correcting.

Another way of looking at it is that reflective degrees of freedom describe information that is not automatically extracted or learned given a sufficiently smart AI, the way it would automatically learn General Relativity.  If you have a concept whose borders (membership condition) relies on knowing about General Relativity, then when the AI is sufficiently smart it will see a simple definition of that concept.  If the concept's borders instead rely on [value-laden](https://arbital.com/p/) judgments, there may be no algorithmically simple description of that concept, even given lots of knowledge of the environment, because the [Humean degrees of freedom](https://arbital.com/p/humean_freedom) need to be independently specified.

Other properties besides the preference function look like they should be reflectively consistent in similar ways.  For example, [son of CDT](https://arbital.com/p/) and [UDT](https://arbital.com/p/) both seem to be reflectively consistent in different ways.  So an AI that has, from our perspective, a 'bad' decision theory (one that leads to behaviors we don't want), isn't 'bugged' in a way we can rely on to self-correct.  (This is one reason why MIRI studies decision theory and not computer vision.  There's a sense in which mistakes in computer vision automatically fix themselves, given a sufficiently advanced AI, and mistakes in decision theory don't fix themselves.)

Similarly, [Bayesian priors](https://arbital.com/p/27p) are by default consistent under reflection - if you're a Bayesian with a prior, you want to create copies of yourself that have the same prior or [Bayes-updated](https://arbital.com/p/1ly) versions of the prior.  So 'bugs' (from a human standpoint) like being [Pascal's Muggable](https://wiki.lesswrong.com/wiki/Pascal's_mugging) might not automatically fix themselves in a way that correlated with sufficient growth in other knowledge and general capability, in the way we might expect a specific mistaken belief about gravity to correct itself in a way that correlated to sufficient general growth in capability.  (This is why MIRI thinks about [naturalistic induction](https://arbital.com/p/) and similar questions about prior probabilities.)


Humean degree of freedom

A "Humean degree of freedom" appears in a cognitive system whenever some quantity / label / concept depends on the choice of utility function, or more generalized preferences.  For example, the notion of "important impact on the world" depends on which variables, when they change, impact something the system cares about, so if you tell an AI "Tell me about any important impacts of this action", you're asking it to do a calculation that depends on your preferences, which might have high complexity and be difficult to identify to the AI.


Strong cognitive uncontainability

### Definition

Suppose somebody from the 10th century were asked how somebody from the 20th century might cool their house.  While they would be able to understand the problem and offer some solutions, maybe even clever solutions ("Locate your house someplace with cooler weather", "divert water from the stream to flow through your living room") the 20th century's actual solution of 'air conditioning' is not available to them as a strategy.  Not just because they don't think fast enough or aren't clever enough, but because an air conditioner takes advantage of physical laws they don't know about.  Even if they somehow randomly imagined an air conditioner's exact blueprint, they wouldn't expect that design to operate *as* an air conditioner until they were told about the relation of pressure to temperature, how electricity can power a compressor motor, and so on.

By definition, a strongly uncontainable agent can conceive strategies that go through causal domains you can't currently model, and it has options accessing those strategies; therefore it may execute high-value solutions such that, even being told the exact strategy, you would not assign those solutions high expected efficacy without being told further background facts.

 At least in this sense, the 20th century is 'strongly cognitively uncontainable' relative to the 10th century: We can solve the problem of how to cool homes using a strategy that would not be recognizable in advance to a 10th-century observer.

Arguably, *most* real-world problems, if we today addressed them using the full power of modern science and technology (i.e. we were willing to spend a lot of money on tech and maybe run a prediction market on the relevant facts) would have best solutions that couldn't be verified in the 10th-century.

We can imagine a [cognitively powerful agent](https://arbital.com/p/2c) being strongly uncontainable in some domains but not others.  Since every cognitive agent is containable on formal games of tic-tac-toe (at least so far as *we* can imagine, and so long as there isn't a real-world opponent to manipulate), strong uncontainability cannot be a universal property of an agent across all formal and informal domains.

### General arguments

Arguments in favor of strong uncontainability tend to revolve around either:

- The richness and partial unknownness of a particular domain.  (E.g. human psychology seems very complicated; has a lot of unknown pathways; and previously discovered exploits often seemed very surprising; therefore we should expect strong uncontainability on the domain of human psychology.)
- Outside-view induction on previous ability advantages derived from cognitive advantages.  (The 10th century couldn't contain the 20th century even though all parties involved were biological Homo sapiens; what makes us think we're the first generation to have the real true laws of the universe in our minds?)

Arguments against strong uncontainability tend to revolve around:

- The apparent knownness of a particular domain.  (E.g., since we have observed the rules of chemistry with great precision and know their origin in the underlying molecular dynamics, we can believe that even an arbitrarily smart agent should not be able to turn lead into gold using non-radioactive chemical reagents.)
- Backward reasoning from the Fermi Paradox, which gives us weak evidence bounding the capabilities of the most powerful agents possible in our universe.  (E.g., even though there might be surprises remaining in the question of how to standardly model physics, any surprise yielding Faster-Than-Light travel to a previously un-traveled point makes the Fermi Paradox harder to explain.)

### Key propositions

- Can [Oracles](https://arbital.com/p/6x) be contained inside a [computational sandbox](https://arbital.com/p/6z)?  That is, is there some restriction of input-output channels and of other environmental interactions such that:
  - The richness of the 'human psychology' domain is averted;
  - Remaining causal interactions with the outside universe have an option set too small and flat to contain interesting options.
- How solid is our current knowledge of the physical universe?
  - To what extent should we expect an advanced agency (e.g. machine superintelligences a million years later) to be boundable using our present physical understanding?
  - Can we reasonably rule out unknown physical domains being accessed by a computationally sandboxed AI?
- What is the highest reasonable probability that could, under optimal conditions, be assigned to having genuinely contained an AI inside a computational sandbox, if it is not allowed any rich output channels?  Is it more like 20% or 80%?
- Are there useful domains conceptually closed to humans' internal understanding?
  - Will a machine superintelligence have 'power we know not' in the sense that it can't be explained to us even after we've seen it (except in the trivial sense that we could simulate another mind understanding it using external storage and Turing-like rules), as with a chimpanzee encountering an air conditioner?

Eric Bruylant
Eliezer Yudkowsky
Alexei Andreev

Summary: A strongly uncontainable agent's best solution strategies often go through causal domains we can't model; we would not be able to see them as solutions in advance.


Advanced safety

A proposal meant to produce [value-aligned agents](https://arbital.com/p/2v) is 'advanced-safe' if it succeeds, or fails safely, in [scenarios where the AI becomes much smarter than its human developers](https://arbital.com/p/2c).

### Definition

A proposal for a value-alignment methodology, or some aspect of that methodology, is alleged to be 'advanced-safe' if that proposal is claimed robust to scenarios where the agent:

- Knows more or has better probability estimates than us
- Learns new facts unknown to us
- Searches a larger strategy space than we can consider
- Confronts new instrumental problems we didn't foresee in detail
- Gains power quickly
- Has access to greater levels of cognitive power than in the regime where it was previously tested
- Wields strategies [that wouldn't make sense to us even if we were told about them in advance](https://arbital.com/p/2j)

### Importance

It seems reasonable to expect that there will be difficulties of dealing with minds smarter than our own, doing things we didn't imagine, that will be qualitatively different from designing a toaster oven to not burn down a house, or from designing an AI system that is dumber than human.  This means that the concept of 'advanced safety' will end up importantly different from the concept of robust pre-advanced AI.

Concretely, it has been argued to be [foreseeable](https://arbital.com/p/) for several difficulties including e.g. [programmer deception](https://arbital.com/p/10f) and [unforeseen maximums](https://arbital.com/p/47), that they won't materialize before an agent is advanced, or won't materialize in the same way, or won't materialize as severely.  This means that practice with dumber-than-human AIs may not train us against these difficulties, requiring a separate theory and mental discipline for making advanced AIs safe.

We have observed in practice that many proposals for 'AI safety' do not seem to have been thought through against advanced agent scenarios; thus, there seems to be a practical urgency to emphasizing the concept and the difference.

Key problems of advanced safety that are new or qualitatively different compared to pre-advanced AI safety include:

- [Edge instantiation](https://arbital.com/p/2w)
- [Unforeseen maximums](https://arbital.com/p/47)
- [Context change problems](https://arbital.com/p/6q)
- [https://arbital.com/p/10f](https://arbital.com/p/10f)
- [Programmer maximization](https://arbital.com/p/)
- [Philosophical competence](https://arbital.com/p/)

Non-advanced-safe methodologies may conceivably be useful if a [known algorithm nonrecursive agent](https://arbital.com/p/) can be created that is (a) [powerful enough to be relevant](https://arbital.com/p/2s) and (b) can be known not to become advanced.  Even here there may be grounds for worry that such an agent finds unexpectedly strong strategies in some particular subdomain - that it exhibits flashes of domain-specific advancement that break a non-advanced-safe methodology.

### Omni-safety

As an extreme case, an 'omni-safe' methodology allegedly remains value-aligned, or fails safely, even if the agent suddenly becomes omniscient and omnipotent (acquires delta probability distributions on all facts of interest and has all describable outcomes available as direct options). See: [real-world agents should be omni-safe](https://arbital.com/p/2x).


Open subproblems in aligning a Task-based AGI

MIRI and related organizations have recently become more interested in trying to sponsor (technical) work on *Task AGI* subproblems.  A [task-based agent](https://arbital.com/p/6w), aka Genie in Bostrom's lexicon, is an AGI that's meant to implement short-term goals identified to it by the users, rather than the AGI being a Bostromian "Sovereign" that engages in long-term strategic planning and self-directed, open-ended operations.

A Task AGI might be safer than a Sovereign because:

- It is possible to [query the user](https://arbital.com/p/2qq) before and during task performance, if an ambiguous situation arises and is successfully identified as ambiguous.
- The tasks are meant to be limited in scope - to be accomplishable, once and for all, within a limited space and time, using some limited amount of effort.
- The AGI itself can potentially be limited in various ways, since it doesn't need to be as powerful as possible in order to accomplish its limited-scope goals.
- If the users can select a valuable and *[pivotal](https://arbital.com/p/6y)* task, identifying an adequately [safe](https://arbital.com/p/2l) way of accomplishing this task might be simpler than [identifying all of human value](https://arbital.com/p/6c).

This page is about open problems in Task AGI safety that we think might be ready for further technical research.

# Introduction: The safe Task AGI problem

A safe Task AGI or safe Genie is an agent that you can safely ask to paint all the cars on Earth pink.

*Just* paint all cars pink.

Not tile the whole future light cone with tiny pink-painted cars.  Not paint everything pink so as to be sure of getting everything that might possibly be a car.  Not paint cars white because white looks pink under the right color of pink light and white paint is cheaper.  Not paint cars pink by building nanotechnology that goes on self-replicating after all the cars have been painted.

The Task AGI superproblem is to formulate a design and training program for a real-world AGI that we can trust to just paint the damn cars pink.

To go into this at some greater depth, to build a safe Task AGI:

• You need to be able to identify the goal itself, to the AGI, such that the AGI is then oriented on achieving that goal.   If you put a picture of a pink-painted car in front of a webcam and say "do this", all the AI has is the sensory pixel-field from the webcam.  Should it be trying to achieve more pink pixels in future webcam sensory data?  Should it be trying to make the programmer show it more pictures?  Should it be trying to make people take pictures of cars?  Assuming you can in fact identify the goal that singles out the futures to achieve, is the rest of the AI hooked up in such a way as to optimize that concept?

• You need to somehow handle the *just* part of the *just paint the cars pink.*  This includes not tiling the whole future light cone with tiny pink-painted cars.  It includes not building another AI which paints the cars pink and then tiles the light cone with pink cars.  It includes not painting everything in the world pink so as to be sure of getting everything that might count as a car.  If you're trying to make the AI have "low impact" (intuitively, prefer plans that result in fewer changes to other quantities), then "low impact" must *not* include freezing everything within reach to minimize how much it changes, or making subtle changes to people's brains so that nobody notices their cars have been painted pink.

• The AI needs to not shoot people who are standing between the painter and the car, and not accidentally run them over, and not use poisonous paint even if the poisonous paint is cheaper.

• The AI should have an '[abort](https://arbital.com/p/2rg)' button which gets it to safely stop doing what it's currently doing.  This means that if the AI was in the middle of building nanomachines, the nanomachines need to also switch off when the abort button is pressed, rather than the AI itself just shutting off and leaving the nanomachines to do whatever.  Assuming we have a safe measure of "low impact", we could define an "abortable" plan as one which can, at any time, be converted relatively quickly to one that has low impact.

• The AI [should not want](https://arbital.com/p/2vk) to self-improve or control further resources beyond what is necessary to paint the cars pink, and should [query the user](https://arbital.com/p/2qq) before trying to develop any [new](https://arbital.com/p/2qp) technology or assimilate any new resources it does need to paint cars pink.

This is only a preliminary list of some of the requirements and use-cases for a Task AGI, but it gives some of the flavor of the problem.

Further work on some facet of the open subproblems below might proceed by:

1.  Trying to explore examples of the subproblem and potential solutions within some contemporary machine learning paradigm.
2.  Building a toy model of some facet of the subproblem, and hopefully observing some non-obvious fact that was not predicted in advance by existing researchers skilled in the art.
3.  Doing [mathematical analysis](https://arbital.com/p/107) of an [unbounded agent](https://arbital.com/p/107) encountering or solving some facet of a subproblem, where the setup is sufficiently precise that claims about the consequences of the premise can be [checked and criticized](https://arbital.com/p/1cv).

# [Conservatism](https://arbital.com/p/2qp)

 A conservative concept boundary is a boundary which is (a) relatively simple and (b) classifies as few things as possible as positive instances of the category.

If we see that 3, 5, 13, and 19 are positive instances of a category and 4, 14, and 28 are negative instances, then a *simple* boundary which separates these instances is "All odd numbers."  A *simple and conservative* boundary is "All odd numbers between 3 and 19" or "All primes between 3 and 19".  (A non-simple boundary is "Only 3, 5, 13, and 19 are members of the category.")

E.g., if we imagine presenting an AI with smiling faces as instances of a goal concept to be learned, then a conservative concept boundary might lead the future AI to pursue only smiles attached to human heads, rather than tiny molecular smileyfaces (not that this necessarily solves everything).

If we imagine presenting the AI with 20 positive instances of a burrito, then a conservative boundary might lead the AI to produce a 21st burrito very similar to those.  Rather than, e.g., needing to explicitly present the AGI with a poisonous burrito that's labeled negative somewhere in the training data, in order to force the simplest boundary around the goal concept to be one that excludes poisonous burritos.

Conservative *planning* is a related problem in which the AI tries to create plans that are similar to previously whitelisted plans or to previous causal events that occur in the environment.  A conservatively planning AI, shown burritos, would try to create burritos via cooking rather than via nanotechnology, if the nanotechnology part wasn't especially necessary to accomplish the goal.

Detecting and flagging non-conservative goal instances or non-conservative steps of a plan for [user querying](https://arbital.com/p/2qq) is a related approach.

([Main article.](https://arbital.com/p/2qp))

# [Safe impact measure](https://arbital.com/p/2pf)

A low-impact agent is one that's intended to avoid large bad impacts at least in part by trying to avoid all large impacts as such.

Suppose we ask an agent to fill up a cauldron, and it fills the cauldron using a self-replicating robot that goes on to flood many other inhabited areas.  We could try to get the agent not to do this by letting it know that flooding inhabited areas is bad.  An alternative approach is trying to have an agent that avoids needlessly large impacts in general - there's a way to fill the cauldron that has a smaller impact, a smaller footprint, so hopefully the agent does that instead.

The hopeful notion is that while "bad impact" is a highly value-laden category with a lot of complexity and detail, the notion of "big impact" will prove to be simpler and to be more easily identifiable.  Then by having the agent avoid all big impacts, or check all big impacts with the user, we can avoid bad big impacts in passing.

Possible gotchas and complications with this idea include, e.g., you wouldn't want the agent to freeze the universe into stasis to minimize impact, or try to edit people's brains to avoid them noticing the effects of its actions, or carry out offsetting actions that cancel out the good effects of whatever the users were trying to do.

Two refinements of the low-impact problem are a [shutdown utility function](https://arbital.com/p/2rf) and [abortable plans](https://arbital.com/p/2rg).

([Main article.](https://arbital.com/p/2pf))

# [https://arbital.com/p/4w](https://arbital.com/p/4w)

An 'inductive ambiguity' is when there's more than one simple concept that fits the data, even if some of those concepts are much simpler than others, and you want to figure out *which* simple concept was intended.

Suppose you're given images that show camouflaged enemy tanks and empty forests, but it so happens that the tank-containing pictures were taken on sunny days and the forest pictures were taken on cloudy days.  Given the training data, the key concept the user intended might be "camouflaged tanks", or "sunny days", or "pixel fields with brighter illumination levels".

The last concept is by far the simplest, but rather than just assume the simplest explanation is correct (has most of the probability mass), we want the algorithm (or AGI) to detect that there's more than one simple-ish boundary that might separate the data, and [check with the user](https://arbital.com/p/2qq) about *which* boundary was intended to be learned.

([Main article.](https://arbital.com/p/4w))

# [Mild optimization](https://arbital.com/p/2r8)

"Mild optimization" or "soft optimization" is when, if you ask the [Task AGI](https://arbital.com/p/6w) to paint one car pink, it just paints one car pink and then stops, rather than tiling the galaxies with pink-painted cars, because it's *not optimizing that hard.*

This is related, but distinct from, notions like "[low impact](https://arbital.com/p/2pf)".  E.g., a low impact AGI might try to paint one car pink while minimizing its other footprint or how many other things changed, but it would be trying *as hard as possible* to minimize that impact and drive it down *as close to zero* as possible, which might come with its own set of pathologies.  What we want instead is for the AGI to try to paint one car pink while minimizing its footprint, and then, when that's being done pretty well, say "Okay done" and stop.

This is distinct from [satisficing expected utility](https://arbital.com/p/eu_satisficer) because, e.g., rewriting yourself as an expected utility maximizer might also satisfice expected utility - there's no upper limit on how hard a satisficer approves of optimizing, so a satisficer is not [reflectively stable](https://arbital.com/p/1fx).

The open problem with mild optimization is to describe mild optimization that (a) captures what we mean by "not trying *so hard* as to seek out every single loophole in a definition of low impact" and (b) is [reflectively stable](https://arbital.com/p/1fx) and doesn't approve e.g. the construction of environmental subagents that optimize harder.

# [Look where I'm pointing, not at my finger](https://arbital.com/p/2s0)

Suppose we're trying to give a [Task AGI](https://arbital.com/p/6w) the task, "Give me a strawberry".  User1 wants to identify their intended category of strawberries by waving some strawberries and some non-strawberries in front of the AI's webcam, and User2 in the control room will press a button to indicate which of these objects are strawberries.  Later, after the training phase, the AI itself will be responsible for selecting objects that might be potential strawberries, and User2 will go on pressing the button to give feedback on these.

![strawberry diagram](http://www.gliffy.com/go/publish/image/10423843/L.png)

The "look where I'm pointing, not at my finger" problem is getting the AI to focus on the strawberries rather than User2 - the concepts "strawberries" and "events that make User2 press the button" are very different goals even though they'll both well-classify the training cases; an AI might pursue the latter goal by psychologically analyzing User2 and figuring out how to get them to press the button using non-strawberry methods.

One way of pursuing this might be to try to zero in on particular nodes inside the huge causal lattice that ultimately produces the AI's sensory data, and try to force the goal concept to be about a simple or direct relation between the "potential strawberry" node (the objects waved in front of the webcam) and the observed button values, without this relation being allowed to go through the User2 node.

![strawberry diagram](http://www.gliffy.com/go/publish/image/10424137/L.png)

See also the related problem of "[https://arbital.com/p/36w](https://arbital.com/p/36w)".

# More open problems

This page is a work in progress.  A longer list of Task AGI open subproblems:

- [Low impact](https://arbital.com/p/2pf)
  - [Shutdown utility function](https://arbital.com/p/2rf)
      - [Abortable plans](https://arbital.com/p/2rg)
- [Conservatism](https://arbital.com/p/2qp)
- [Mild optimization](https://arbital.com/p/2r8)
  - [Aversion of instrumental self-improvement goal](https://arbital.com/p/2x3)
- [Ambiguity identification](https://arbital.com/p/4w)
- [Utility indifference](https://arbital.com/p/1b7)
  - [Shutdown button](https://arbital.com/p/2xd)
- Task identification
  - [Ontology identification](https://arbital.com/p/5c)
  - [https://arbital.com/p/36w](https://arbital.com/p/36w)
         - [https://arbital.com/p/2s0](https://arbital.com/p/2s0)
- Hooking up a directable optimization to an identified task
- Training protocols
  - Which things do you think can be well-identified by what kind of labeled datasets plus queried ambiguities plus conservatism, and what pivotal acts can you do with combinations of them plus assumed other abilities?
- [Faithful simulation](https://arbital.com/p/36k)
- Safe imitation for [act-based agents](https://arbital.com/p/1w4)
  - Generative imitation with a probability of the human doing that act, guaranteed not to hindsight bias
  - Typicality (related to conservatism)
- Plan transparency
  - Epistemic-only hypotheticals (when you ask how in principle how the AI might paint cars pink, it doesn't run a planning subprocess that plans to persuade the actual programmers to paint things pink).
- [Epistemic exclusion](https://arbital.com/p/1g4)
  - [Behaviorism](https://arbital.com/p/102)

(...more, this is a page in progress)


Low impact

A low-impact agent is a hypothetical [task-based AGI](https://arbital.com/p/6w) that's intended to avoid *disastrous* side effects via trying to *avoid large side effects in general*.

Consider the Sorcerer's Apprentice fable: a legion of broomsticks, self-replicating and repeatedly overfilling a cauldron (perhaps to be as certain as possible that the cauldron was full).  A low-impact agent would, if functioning as [intended](https://arbital.com/p/6h), have an incentive to avoid that outcome; it wouldn't just want to fill the cauldron, but fill the cauldron in a way that had a minimum footprint.  If the task given the AGI is to paint all cars pink, then we can hope that a low-impact AGI would not accomplish this via self-replicating nanotechnology that went on replicating after the cars were painted, because this would be an unnecessarily large side effect.

On a higher level of abstraction, we can imagine that the universe is parsed by us into a set of variables $V_i$ with values $v_i.$  We want to avoid the agent taking actions that cause large amounts of disutility, that is, we want to avoid perturbing variables from $v_i$ to $v_i^*$ in a way that decreases utility.  However, the question of exactly which variables $V_i$ are important and shouldn't be entropically perturbed is [value-laden](https://arbital.com/p/36h) - complicated, fragile, high in [algorithmic complexity](https://arbital.com/p/5v), with [Humean degrees of freedom in the concept boundaries](https://arbital.com/p/2fs).

Rather than relying solely on teaching an agent exactly which parts of the environment shouldn't be perturbed and risking catastrophe if we miss an injunction, the *low impact* route would try to build an agent that tried to perturb fewer variables regardless.

The hope is that "have fewer side effects" is a problem that has a simple core and is learnable by a manageable amount of training.  Conversely, trying to train "here is the list of *bad* effects not to have and *important* variables not to perturb" would be complicated and lack a simple core, because 'bad' and 'important' are [value-laden](https://arbital.com/p/36h).  A list of dangerous variables would also be [a blacklist rather than a whitelist](https://arbital.com/p/2qp), which would make it more vulnerable to [treacherous context changes](https://arbital.com/p/6q) if the AI gained the ability to affect new things.

# Introduction:  Formalizing low impact seems nontrivial

Intuitively, the notion of "low impact" seems like it should be simpler - have more of a central, core tendency to capture - than "avoid bad impacts".  If so, we don't know yet how to compactly state this core principle semiformally.

Suppose we start with an obvious notion: to have low impact, minimize the number of variables you [causally](https://arbital.com/p/) affect.  But:

- Every event has a not-absolutely-zero impact on every future event.  When you twitch your thumb or even just fire a neuron, the gravitational ripples from the moving atoms spread out and will eventually produce infinitesimal forces on atoms orbiting the other side of the galaxy.  So we can't say "have zero impact on as many quarks as possible" because it is impossible for any act to have absolutely zero impact on any quark.

This is a special case of a failure mode where, since the impact metric turns out to already be maxed out by all actions, moving around inside the maxed-out impact penalty doesn't change expected utility, and therefore fails to influence which action is chosen.  This [foreseeable difficulty](https://arbital.com/p/6r) might also illustrate a typical [treacherous context change](https://arbital.com/p/6q), in that the AI during its training phase might have a relatively small causal model of its environment, with qualitative causal arrows, and so seem to be minimizing impact inside the training regime.  Later development might cause the AI to adopt a new causal model in which, as happens to actually be the case, all its possible actions influence all the quarks inside its [future light cone](https://arbital.com/p/), and so actions are not differentiated by the magnitude of their impact penalty; thus the AI would safe in training and unsafe in later use, with the critical threshold possibly coming after the AI was [too intelligent to be shut down](https://arbital.com/p/instrumental_self_preservation).

But measuring causal impacts qualitatively isn't the only possibility.  Suppose we define impact as the sum of the expected movement of all atoms, relative to their mean expected positions if some privileged null action were taken instead?  Or, to avoid this impact measure returning 0 as soon as the AI realizes that [there are no atoms, only quarks](https://arbital.com/p/5c), we try to give the AI some notion of the distance between outcomes in *any* way of modeling the world?

Arguendo, suppose that we could define a representational language over possible universes and laws of physics, such that within any representable model of the universe, there's an obvious notion of 'distance' between any two outcomes conditional on the AI's actions.  If the universe is made of atoms, this representation will expose the obvious impact metric on the movements of atoms.  If the universe is made of quarks, the same component of the utility function will readily calculate the movements of quarks.  If the universe is made of quantum fields, this impact metric will behave in the intuitively intended way that basically works out to measuring particle motions, rather than the change metric always maxing out as the result of all amplitude flows ending up in qualitatively different sections of the quantum configuration space, etcetera.  (Note that this is already sounding pretty nontrivial.)

Furthermore, suppose when the AI is thinking in terms of neither atoms nor quarks, but rather, say, the equivalent of chess moves or voxel fields, the same impact metric can apply to this as well; so that we can observe the low-impact behaviors at work during earlier development phases.

More formally:  We suppose that the AI's model class $\mathcal M$ is such that for any allowed model $M \in \mathcal M,$ for any two outcomes $o_M$ and $o_M^'$ that can result from the AI's choice of actions, there is a distance $%% o_M - o_M^' %%$ which obeys [standard rules for distances](http://mathworld.wolfram.com/Norm.html). This general distance measure is such that, within the standard model of physics, moving atoms around would add to the distance between outcomes in the obvious way; and for models short of molecular detail, will measure changes in other variables and quantities in an intuitive way.  We then fix some particular policy $\pi_0$ whose consequence $(o|\pi_0)$ is "the result of the AI doing nothing", and measure the impact penalty of any other policy $\pi_k$ as proportional to the expected distance between outcomes $\mathbb E[](https://arbital.com/p/%%).$

Then we might run into any of these further [foreseeable difficulties](https://arbital.com/p/6r), if, e.g., you asked the AGI to cure cancer with minimum 'impact' as defined above (or to put it another way, minimize impact subject to the constraint of curing cancer):

- Offsetting actions we would intuitively regard as both bad and impactful:
  - The AGI models global death rates as a quantity, and implements offsetting actions to keep Earth's annual death statistics constant after it cures cancer.
- Chaotic permissiveness:
  - Weather is a chaotic system.  If influencing any butterfly is going to move around all the atoms in the atmosphere anyway, you might as well have them end up somewhere you want.
- Stasis incentives:
  - The AGI edits its programmers to make sure the programmers do whatever they would have done if the AGI, after being told its task, performed the null action.
  - The AGI proliferates across galaxies to make sure everything else in the universe outside of human bodies adheres as closely as possible to the expected state it would have occupied if the null action had been taken.
  - The AGI sets up a weather-control system so that at least its further actions won't again disturb the weather.

All of this just goes to say that there's apparently some subtlety built into our intuitively [intended](https://arbital.com/p/6h) notion of "paint all cars pink, but do so with the minimum footprint possible apart from that".

We want people to be able to notice that their cars have been painted pink, and for them to enjoy whatever further benefit of pink-painted cars led us to give the AGI this instruction in the first place.  But we can't just whitelist any further impact that happens as a consequence of the car being painted pink, because maybe the car was painted with pink replicating nanomachines.  Etcetera.

Even if there is, in fact, some subtlety built into our intended notion of "make plans that have minimal side effects", this subtle notion of low impact might still have a relatively much simpler core than our intuitive notion of "avoid bad impacts".  This might be reflected in either an improved formal intuition for 'low impact' that proves to stand up to a few years of skeptical scrutiny without any holes having been poked in it, or, much more nerve-rackingly, the ability to train an AI to make minimal-impact plans even if we don't know a closed-form definition of "minimal impact".

Work in this area is ongoing, so far mainly in the form of some preliminary suggestions by Stuart Armstrong (which were mostly shot down, but this is still progress compared to staring blankly at the problem).

# Foreseeable difficulties

## Permissiveness inside chaotic systems

Suppose you told the AI to affect as few things as possible, above the minimum necessary to achieve its task, and defined 'impact' qualitatively in terms of causal links that make variables occupy different states.  Then since every act and indeed every internal decision (transistors, in switching, move electrons) would have infinitesimal influences on literally everything in the AI's future light cone, all of which is defined as an 'impact', all actions would seem to have the same, maximum impact.  Then the impact penalty would make no difference to the net expected utility of actions, causing the AI to behave as if it had no impact penalty.

Even if an impact measure doesn't max out because of ubiquitous qualitative impacts, a poorly defined impact measure might max out quantitatively when the AGI is operating in a domain that is *chaotic* in the sense that tiny differences soon blow up to large differences.  E.g., if a butterfly flaps its wings, that might cause a hurricane on the other side of the world a year later - so since you're already changing the weather system as much as possible, why does it matter if, say, you on-purpose cause a hurricane in some area, or destroy a target using atmospheric lightning strikes?  Those air molecules would all have ended up moving anyway because of the butterfly effect.

An imaginable patch is to try to evaluate impact over *foreseeable* impacts, so that a known lightning strike is 'foreseeable', while the effects on future hurricanes are 'not foreseeable'.  This seems worryingly like mixing up the map and the territory (is it okay to release environmental poisons so long as you don't know who gets hurt?), but Stuart Armstrong has made some preliminary suggestions about minimizing knowable impacts.

If you didn't know it was coming, "maxing out the impact penalty" would potentially be a [treacherous context change](https://arbital.com/p/6q).  When the AI was at the infrahuman level, it might model the world on a level where its actions had relatively few direct causal links spreading out from them, and most of the world would seem untouched by most of its possible actions.  Then minimizing the impact of its actions, while fulfilling its goals, might in the infrahuman state seem to result in the AI carrying out plans with relatively few side effects, as intended.  In a superhuman state, the AI might realize that its every act resulted in quantum amplitude flowing into a nonoverlapping section of configuration space, or having chaotic influences on a system the AI was not previously modeling as having maximum impact each time.

## Infinite impact penalties

In one case, a proposed impact penalty was written down on a whiteboard which happened to have the fractional form $\frac{X}{Y}$ where the quantity $Y$ could *in some imaginable universes* get very close to zero, causing Eliezer Yudkowsky to make an "Aaaaaaaaaaa"-sound as he waved his hands speechlessly in the direction of the denominator.  The corresponding agent would have [spent all its effort on further-minimizing infinitesimal probabilities of vast impact penalties](https://arbital.com/p/pascals_mugging).

Besides "don't put denominators that can get close to zero in any term of a utility function", this illustrates a special case of the general rule that impact penalties need to have their loudness set at a level where the AI is doing something besides minimizing the impact penalty.  As a special case, this requires considering the growth scenario for improbable scenarios of very high impact penalty; the penalty must not grow faster than the probability diminishes.

(As usual, note that if the agent only started to visualize these ultra-unlikely scenarios upon reaching a superhuman level where it could consider loads of strange possibilities, this would constitute a [treacherous context change](https://arbital.com/p/6q).)

## Allowed consequences vs. offset actions

When we say "paint all cars pink" or "cure cancer" there's some implicit set of consequences that we think are allowable and should definitely not be prevented, such as people noticing that their cars are pink, or planetary death rates dropping.  We don't want the AI trying to obscure people's vision so they can't notice the car is pink, and we don't want the AI killing a corresponding number of people to level the planetary death rate.  We don't want these bad *offsetting* actions which would avert the consequences that were the point of the plan in the first place.

If we use a low-impact AGI to carry out some [pivotal act](https://arbital.com/p/6y) that's part of a larger plan to improve Earth's chances of not being turned into paperclips, then this, in a certain sense, has a very vast impact on many galaxies that will *not* be turned into paperclips.  We would not want this *allowed* consequence to max out and blur our AGI's impact measure, nor have the AGI try to implement the pivotal act in a way that would minimize the probability of it actually working to prevent paperclips, nor have the AGI take offsetting actions to keep the probability of paperclips to its previous level.

Suppose we try to patch this rule that, when we carry out the plan, the further causal impacts of the task's accomplishment are exempt from impact penalties.

But this seems to allow too much.  What if the cars are painted with self-replicating pink nanomachines?  What distinguishes the further consequences of that solved goal from the further causal impact of people noticing that their cars have been painted pink?

One difference between "people notice their cancer was cured" and "the cancer cure replicates and consumes the biosphere" is that the first case involves further effects that are, from our perspective, pretty much okay, while the second class of further effects are things we don't like.  But an 'okay' change versus a 'bad' change is a value-laden boundary.  If we need to detect this difference as such, we've thrown out the supposed simplicity of 'low impact' that was our reason for tackling 'low impact' and not 'low badness' in the first place.

What we need instead is some way of distinguishing "People see their cars were painted pink" versus "The nanomachinery in the pink paint replicates further" that operates on a more abstract, non-value-laden level.  For example, hypothetically speaking, we might claim that *most* ways of painting cars pink will have the consequence of people seeing their cars were painted pink and only a few ways of painting cars pink will not have this consequence, whereas the replicating machinery is an *unusually large* consequence of the task having reached its fulfilled state.

But is this really the central core of the distinction, or does framing an impact measure this way imply some further set of nonobvious undesirable consequences?  Can we say rigorously what kind of measure on task fulfillments would imply that 'most' possible fulfillments lead people to see their cars painted pink, while 'few' destroy the world through self-replicating nanotechnology?  Would *that* rigorous measure have further problems?

And if we told an AGI to shut down a nuclear plant, wouldn't we want a low-impact AGI to err on the side of preventing radioactivity release, rather than trying to produce a 'typical' magnitude of consequences for shutting down a nuclear plant?

It seems difficult (but might still be possible) to classify the following consequences as having low and high extraneous impacts based on a *generic* impact measure only, without introducing further value lading:

- Low disallowed impact:  Curing cancer causes people to notice their cancer has been cured, hospital incomes to drop, and world population to rise relative to its default state.
- High disallowed impact:  Shutting down a nuclear power plant causes a release of radioactivity.
- High disallowed impact:  Painting with pink nanomachinery causes the nanomachines to further replicate and eat some innocent bystanders.
- Low disallowed impact:  Painting cars with ordinary pink paint changes the rays of light reflecting from those cars and causes people to gasp and say "What just happened to my car?"
- Low disallowed impact:  Doing something smart with a Task AGI decreases the probability of the galaxies being consumed by an Unfriendly AI.

(Even if we think that good AGI scenarios involve the AGI concealing the fact of its existence, it's hard to see why we'd want the *events as such* to be unnoticeable, or for their noticing to count as extraneous impacts.)

### Fallback use of an impact measure to prioritize operator queries

If this problem of allowing *good* consequences did turn out to be inherently value-laden, it might not obviate the usefulness of a safe impact measure.  For example, it could be that in normal operation of a Task AGI, there are only a limited number of good further impacts that we want to whitelist - e.g., once we say that curing cancer is allowed to cause people to notice that they feel healthier and for there to be more people alive using more resources, etcetera, then it's okay to minimize all other footprints.

Similarly, a safe impact measure might operate to alert the operators to all large impacts sorted by size of impact, such that the impact measure would reliably not have any remaining event that was 'large' in an intuitive sense that the operators didn't know about.  That is, we might not be able to give a blanket instruction to minimize impact, but a safe impact measure might still be useful for deciding which features of a plan need to be queried to the operators.

## Stasis incentives

This would be the general problem that includes "The AGI proliferates across galaxies to make sure everything else in the universe outside of human bodies adheres as closely as possible to the expected state it would have occupied if the null action had been taken."

Similarly, we can imagine the AGI having a strong incentive to, in Round 1, paint the cars with nanomachines such that the nanomachinery quietly and unobtrusively sets up a weather-control system.  This wouldn't prevent the AGI's actions in Round 1 from influencing the chaotic weather and having a huge impact, but could ensure that the AI's future actions in Round 2 wouldn't influence the formerly-chaotic weather relative to its null action then.

## The problem of the backup AGI

Suppose we're using a measure of impact relative to a privileged null action.  Imagine that there are two AGIs which will both be given the same task, and under ordinary circumstances, divide the effort of performing the task between them; but if one AGI fails to perform, the other will take over the task in its entirety.  Then each AGI reasons, "If I perform the null action instead of carrying out the task, the other AGI will do almost exactly the same thing I was going to do" and so models itself as having *counterfactually* almost no impact.  In this way, two AGIs can each evaluate themselves as having almost no impact, while operating in tandem they have a very large impact.  (This problem was first observed by Stuart Armstrong.)

We can even imagine - though hopefully this isn't the sort of thing that would go uncaught by other safety measures - that in Round 1, the agent builds a backup agent, which doesn't count as having an extraneous impact because the system of 2 agents ends up carrying out the same steps that the agent would have carried out on its own.  But then on Round 2, there's now a backup and both agents regard themselves as having minimal impact.

(If you say that the agent should, on Round 1, realize that constructing the backup agent on Round 1 would have a later *impact* on the effect of the AI carrying out its orders in Round 2, then we're now allowing the agent to minimize its impact *across tasks,* which might imply that in Round 1 it should try to build something that regularizes the weather as a side effect, so that its actions in Round 2 have less chaotic impact on the weather... or that it tries to influence its programmers to give it less impactful tasks, etcetera.)

A more general way of viewing this possible problem is that there might be [tiling problems](https://arbital.com/p/) for a low-impact agent modifying itself or creating subagents or successors.

# Standard issues

The following standard issues from alignment theory seem like they could also apply to the problem of building a low-impact agent or creating a safe impact measure.  We need an [advanced-safe](https://arbital.com/p/2l) impact measure that will stand up to the load and stress of cognition that is superhuman in at least some domains.

## Programmer manipulation

A Task AGI which plans across rounds may have an incentive to influence its programmers into giving it plans that will have net lower impact compared to other plans the programmers might offer.  If there's an explicit procedure for the programmers to whitelist consequences, the AGI would have to be somehow configured in such fashion as to not prefer programmers to whitelist the consequences or seek out consequences that the programmers would be expected to (perhaps mistakenly) whitelist.

## [Causal environment hacking](https://arbital.com/p/5j)

Since an impact measure seems likely to be defined over the causes of sense data, it has the standard problem with remote hacking of causal environments.  (This is a ubiquitous enough problem that you're going to want some general solution such as not having the AGI model remote hackers in the first place.  But this subproblem should still be noted each time we introduce a dependency on the modeled causes of sense data.)

## [Edge instantiation](https://arbital.com/p/2w)

Building a safe impact measure is hard because if we roll a naive impact measure, it might have its minimum in a weird, undesirable place.  Freezing all matter into the universe into stasis would be an example of this.

## [Unforeseen maximum](https://arbital.com/p/47)

The actions and modes the AGI has available to potentially minimize its impact will include more actions and modes than we can imagine and check ourselves.

## [Nearest unblocked neighbor](https://arbital.com/p/42)

If the maximizing plan has high impact, the new best plan after introducing an impact measure may be a similar plan that has, through some leaky definition, found a weird special case that doesn't count as an impact.  To put it another way, to the extent an imperfect impact measure would otherwise hamper the AI if it were perfect, the AI experiences an [instrumental pressure](https://arbital.com/p/10k) to seek out any loopholes or leaks in the definition.

## [Context change](https://arbital.com/p/6q)

As the AI gains in cognitive capability, this widens the option space which could potentially contain instances of 'low impact' that are [unforeseen](https://arbital.com/p/47) or [unblocked](https://arbital.com/p/42).

In this particular case, increased intelligence may also:

- Change the set of impacts that the AI can foresee, so that e.g. its causal model goes from 'some actions causally connect to some other things' to 'all actions causally connect to all things'.
- Widen the space of possibilities it considers, to include for the first time very-low-probability scenarios of very large impacts.
- Include new offsetting actions, so that the AI for the first time has the option of editing brains not to notice changes it makes, or introduce randomized Unfriendly-AI generators to offset decreases in the probability of Unfriendly AI.

## [https://arbital.com/p/1fx](https://arbital.com/p/1fx)

Does a low-impact AI want to only build a successor that's a low-impact AI?  If it builds an environmental subagent, is that subagent low impact?

Even if the AGI is supposed to not be self-modifying or to be building subagents, is there a worrying divergence and pressure to be held in check between how the AI thinks and how the AI would prefer to think?  Does a low-impact AGI want relevant cognitive computations in general to be low impact?

To the extent that low impact is a feature of the utility function rather than the optimization style, this doesn't have any obvious problems (apart from Armstrong's dual-AGI no-impact counterfactual issue), but it's a standard thing to check, and would become *much more* important if low impact was supposedly being achieved through any feature of the optimization style rather than utilities over outcomes.

# Related / further problems

A [shutdown utility function](https://arbital.com/p/2rf) is one which incentivizes the AI to safely switch itself off, without, say, creating a subagent that assimilates all matter in the universe to make absolutely sure the AI is never again switched back on.

[Abortable plans](https://arbital.com/p/2rg) are those which are composed with the intention that it be possible to midway activate an 'abort' plan, such that the partial implementation of the original plan, combined with the execution of the abort plan, together have a minimum impact.  For example, if an abortable AI was building self-replicating nanomachines to paint a car pink, it would give all the nanomachines a quiet self-destruct button, so that at any time the 'abort' plan could be executed after having partially implemented to the plan to paint the car pink, such that these two plans together would have a minimum impact.

Eric Bruylant
Eliezer Yudkowsky
Niplav Yushtun

Open subproblems in aligning a Task-based AGI

Summary: A low-impact agent is one that's intended to avoid large bad impacts at least in part by trying to avoid all large impacts as such.  Suppose we ask an agent to fill up a cauldron, and it fills the cauldron using a self-replicating robot that goes on to flood many other inhabited areas.  We could try to get the agent not to do this by letting it know that flooding inhabited areas is bad.  An alternative approach is trying to have an agent that avoids needlessly large impacts in general - there's a way to fill the cauldron that has a smaller impact, a smaller footprint, so hopefully the agent does that instead.  The hopeful notion is that while "bad impact" is a highly value-laden category with a lot of complexity and detail, the notion of "big impact" will prove to be simpler and to be more easily identifiable.  Then by having the agent avoid all big impacts, or check all big impacts with the user, we can avoid bad big impacts in passing.  Possible gotchas and complications with this idea include, e.g., you wouldn't want the agent to freeze the universe into stasis to minimize impact, or try to edit people's brains to avoid them noticing the effects of its actions, or carry out offsetting actions that cancel out the good effects of whatever the users were trying to do.


Conservative concept boundary

The problem of conservatism is to draw a boundary around positive instances of a concept which is not only *simple* but also *classifies as few instances as possible as positive.*

# Introduction / basic idea / motivation

Suppose I have a numerical concept in mind, and you query me on the following numbers to determine whether they're instances of the concept, and I reply as follows:

- 3:  Yes
- 4:  No
- 5:  Yes
- 13:  Yes
- 14:  No
- 19: Yes
- 28:  No

A *simple* category which covers this training set is "All odd numbers."

A *simple and conservative* category which covers this training set is "All odd numbers between 3 and 19."

A slightly more complicated, and even more conservative category, is "All prime numbers between 3 and 19."

A conservative but not simple category is "Only 3, 5, 13, and 19 are positive instances of this category."

One of the (very) early proposals for value alignment was to train an AI on smiling faces as examples of the sort of outcome the AI ought to achieve.  Slightly steelmanning the proposal so that it doesn't just produce *images* of smiling faces as the AI's sensory data, we can imagine that the AI is trying to learn a boundary over the *causes of* its sensory data that distinguishes smiling faces within the environment.

The classic example of what might go wrong with this alignment protocol is that all matter within reach might end up turned into tiny molecular smiley faces, since heavy optimization pressure would pick out an [extreme edge](https://arbital.com/p/2w) of the simple category that could be fulfilled as maximally as possible, and it's possible to make many more tiny molecular smileyfaces than complete smiling faces.

That is:  The AI would by default learn the simplest concept that distinguished smiling faces from non-smileyfaces within its training cases.  Given [a wider set of options than existed in the training regime](https://arbital.com/p/6q), this simple concept might also classify as a 'smiling face' something that had the properties singled out by the concept, but was unlike the training cases with respect to other properties.  This is the metaphorical equivalent of learning the concept "All odd numbers", and then positively classifying cases like -1 or 9^999 that are unlike 3 and 19 in other regards, since they're still odd.

On the other hand, suppose the AI had been told to learn a simple *and conservative* concept over its training data.  Then the corresponding goal might demand, e.g., only smiles that came attached to actual human heads experiencing pleasure.  If the AI were moreover a conservative *planner*, it might try to produce smiles only through causal chains that resembled existing causal generators of smiles, such as only administering existing drugs like heroin and not inventing any new drugs, and only breeding humans through pregnancy rather than synthesizing living heads using nanotechnology.

You couldn't call this a solution to the value alignment problem, but it would - arguendo - get significantly *closer* to the [intended goal](https://arbital.com/p/6h) than tiny molecular smileyfaces.  Thus, conservatism might serve as one component among others for aligning a [Task AGI](https://arbital.com/p/6w).

Intuitively speaking:  A genie is hardly rendered *safe* if it tries to fulfill your wish using 'normal' instances of the stated goal that were generated in relatively more 'normal' ways, but it's at least *closer to being safe.*  Conservative concepts and conservative planning might be one attribute among others of a safe genie.

# Burrito problem

The *burrito problem* is to have a Task AGI make a burrito that is actually a burrito, and not just something that looks like a burrito, and not poisonous and that is actually safe for humans to eat.

Conservatism is one possible approach to the burrito problem:  Show the AGI five burritos and five non-burritos.  Then, don't have the AGI learn the *simplest* concept that distinguishes burritos from non-burritos and then create something that is *maximally* a burrito under this concept.  Instead, we'd like the AGI to learn a *simple and narrow concept* that classifies these five things as burritos according to some simple-ish rule which labels as few objects as possible as burritos.  But not the rule, "Only these five exact molecular configurations count as burritos", because that rule would not be simple.

The concept must still be broad enough to permit the construction of a sixth burrito that is not molecularly identical to any of the first five.  But not so broad that the burrito includes butolinum toxin (because, hey, anything made out of mostly carbon-hydrogen-oxygen-nitrogen ought to be fine, and the five negative examples didn't include anything with butolinum toxin).

The hope is that via conservatism we can avoid needing to think of every possible way that our training data might not properly stabilize the 'simplest explanation' along every dimension of potentially fatal variance.  If we're trying to only draw *simple* boundaries that separate the positive and negative cases, there's no reason for the AI to add on a "cannot be poisonous" codicil to the rule unless the AI has seen poisoned burritos labeled as negative cases, so that the slightly more complicated rule "but not poisonous" needs to be added to the boundary in order to separate out cases that would otherwise be classified positive.  But then maybe even if we show the AGI one burrito poisoned with butolinum, it doesn't learn to avoid burritos poisoned with ricin, and even if we show it butolinum and ricin, it doesn't learn to avoid burritos poisoned with the radioactive iodine-131 isotope.  Rather than our needing to think of what the concept boundary needs to look like and including enough negative cases to force the *simplest* boundary to exclude all the unsafe burritos, the hope is that via conservatism we can shift some of the workload to showing the AI *positive* examples which happen *not* to be poisonous or have any other problems.

# Conservatism over the causes of sensed training cases.

Conservatism in AGI cases seems like it would need to be interpreted over the causes of sensory data, rather than the sensory data itself.  We're not looking for a conservative concept about *which images of a burrito* would be classified as positive, we want a concept over which *environmental burritos* would be classified as positive.  Two burrito candidates can cause identical images while differing in their poisonousness, so we want to draw our conservative concept boundary around (our model of) the causes of past sensory events in our training cases, not draw a boundary around the sensory events themselves.

# Conservative planning

A conservative *strategy* or conservative *plan* would *ceteris paribus* prefer to construct burritos by buying ingredients from the store and cooking them, rather than building nanomachinery that constructs a burrito, because this would be more characteristic of how burritos are usually constructed, or more similar to the elements of previously approved plans.  Again, this seems like it might be less likely to generate a poisonous burrito.

Another paradigmatic example of conservatism might be to, e.g., inside some game engine, show the AI some human players running around, and then give the AI an object that has the goal of e.g. moving a box to the end of the room.  If the AI is given the ability to fly, but generates a plan in which the box-moving agent only moves around on the ground because that's what the training examples did, then this is a conservative plan.

The point of this isn't to cripple the AI's abilities, the point is that if e.g. your [low impact measure](https://arbital.com/p/2pf) has a loophole and the AI generates a plan to turn all matter within reach into pink-painted cars, some steps of this plan like "disassemble stars to make more cars and paint" are likely to be non-conservative and hence not happen automatically.

## Flagging non-conservative plan steps

If a non-conservative plan seems better along other important dimensions - for example, there is no other plan that has an equally low impact and equally few side effects compared to just synthesizing the burrito using a nanomachine - then we can also imagine that the critical step might be flagged as non-conservative and presented to the user for checking.

That is, on 'conservative' planning, we're interested in both the problem "generate a plan and then flag and report non-conservative steps" as well as the problem "try to generate a plan that has few or no non-conservative steps".

# Role in other safety problems

Conservatism and conservative planning seems like it might directly tackle some standard concerns head-on and in a sufficiently basic way to avoid loopholes, and might also be subject to those concerns.  E.g.:

- [https://arbital.com/p/2w](https://arbital.com/p/2w) - if in full generality we don't go to the edge of the graph but try to stay in the center of what's already been positively classified, maybe we can avoid this.
- [https://arbital.com/p/47](https://arbital.com/p/47) - if we stick to things very similar to already-positively-classified instances, we won't automatically go into the unimagined parts of the graph.
- [https://arbital.com/p/6q](https://arbital.com/p/6q) - a sufficiently conservative optimizer might go on using options previous to similarly whitelisted ones even if large new sections of planning space opened up.

However, to the extent we rely on conservatism to prevent any of these things from happening, it's a mission-critical component that itself has to be [advanced-safe](https://arbital.com/p/2l) with no loopholes.  If a 'conservatism' constraint is being applied to [very powerful optimization pressures](https://arbital.com/p/2c), we need to worry about this [seeking out any loophole](https://arbital.com/p/42) in what is 'conservative'.  It might be that the central notion of 'conservatism' is simple enough to have no loopholes.  But it's also possible that even a simplish and conservative concept would still include some dangerous instances, if there's enough optimization pressure seeking out a maximal-under-some-criterion instance within everything that's been classified conservatively.

Two possible meta-approaches to making conservatism even safer:

- Use conservatism to flag non-conservative steps in plans, or expected non-conservative instances of goal achievements, and refer these for user checking before taking action.  (Rather than automatically generating a plan containing only 'conservative' steps.)  This would have the [standard problems with user checking](https://arbital.com/p/2qq).
- Have a definition of conservatism, relative to the AI's current world-model and conceptual language, which would automatically catch as 'exceptional' (hence not conservative) anything which had the weird property of being the only first-order-conservative instance of a concept that had some other special property being sought out by the optimization pressure.  This might involve weird reflective problems, such as any planned event being special in virtue of the AI having planned it.

Summary:
A conservative concept boundary is a boundary which is (a) relatively simple and (b) classifies as few things as possible as positive instances of the category.  If we see that 3, 5, 13, and 19 are positive instances of a category and 4, 14, and 28 are negative instances, then a *simple* boundary which separates these instances is "All odd numbers" and a *simple and conservative* boundary is "All odd numbers between 3 and 19" or "All primes between 3 and 19".

If we imagine presenting an AI with smiling faces as instances of a goal concept during training, then a conservative concept boundary might lead the AI to produce only smiles attached to human head, rather than tiny molecular smileyfaces (not that this necessarily solves everything).  If we imagine presenting the AI with 20 positive instances of a burrito, then a conservative boundary might lead the AI to produce a 21st burrito very similar to those, rather than needing to be explicitly presented with negative instances of burritos that force the simplest boundary around the goal concept to exclude poisonous burritos.


Querying the AGI user

If we're supposing that an [advanced agent](https://arbital.com/p/2c) is checking something Potentially Bad with its user to find out if the thing is Considered Bad by that user, we need to worry about the following generic issues:

- Can the AI tell which things are Potentially Bad in a way that includes all things that are Actually Bad?
- Can the *user* reliably tell which Potentially Bad things are Actually Bad?
- Does the AI, emergently or deliberately, seek out Potentially Bad things that the user will *not* label as Considered Bad, thereby potentially optimizing for Actually Bad things that the user mislabels as Not Bad?  (E.g., if the agent learns to avoid new tries similar to those already labeled bad, we're excluding the Considered Bad space, but what's left may still contain Actually Bad things via [https://arbital.com/p/-42](https://arbital.com/p/-42) or a similar phenomenon.)
- Is the criterion for Potentially Bad so broad, and Actually Bad things hard enough to *reliably* prioritize *within* that space, that 10% of the time an Actually Bad thing will not be in the top 1,000 Potentially Bad things the user can afford the time to check?
- Can the AI successfully communicate to the user the details of what set off the flag for Potential Badness, or even communicate to the user exactly what was flagged as Potentially Bad, if this is an important part of the user making the decision?
  - Do the AI's communication goals risk [optimizing the user](https://arbital.com/p/optimizing_user)?
  - Are the details of Potential Badness or even the subject of Potential Badness so inscrutable as to be impenetrable?  (E.g., AlphaGo trying to explain to a human why a Go move is potentially bad, or for that matter, a Go professional trying to explain to an amateur why a Go move is potentially bad - we might just be left with blind trust, at which point we might as well just tell the AI not to do Potentially Bad things rather than asking it to pointlessly check with the user.)
- Does the AI, emergently or instrumentally, optimize for the user not labeling things as Potentially Bad, thereby potentially leading to user deception?

Summary: There's a laundry list of things that might go wrong when we suppose that an advanced AI is checking something Potentially Bad with the user/operator/programmer to see if the user labels the thing as Considered Bad, and relying on this step of the workflow to exclude things that are Actually Bad.  E.g., the user might not be able to detect Actually Bad things reliably, the space of Potentially Bad things might be so broad that the Actually Bad things are 1,000 items down the list of things that are Potentially Bad, the AI might just learn to do things that won't be Considered Bad and thereby seek out special cases of bad things that the user can't detect as bad, etcetera.