The scalability of [iterated amplification](https://arxiv.org/pdf/1810.08575.pdf) or [debate](https://arxiv.org/abs/1805.00899) seems to depend on whether large enough teams of humans can carry out arbitrarily complicated reasoning. Are these schemes “universal,” or are there kinds of reasoning that work but which humans fundamentally can’t understand?

This post defines the concept of “ascription universality,” which tries to capture the property that a question-answering system **A** is better-informed than any particular simpler computation **C**.

[These](/informed-oversight-18fcb5d3d1e1) [parallel](/universality-and-consequentialism-within-hch-c0bee00365bd) [posts](/universality-and-model-based-rl-b08701394ddd) explain why I believe that the alignment of iterated amplification largely depends on whether HCH is ascription universal. Ultimately I think that the “right” definition will be closely tied to the use we want to make of it, and so we should be refining this definition in parallel with exploring its applications.

I’m using the awkward term “ascription universality” partly to explicitly flag that this is a preliminary definition, and partly to reserve linguistic space for the better definitions that I’m optimistic will follow.

(Thanks to Geoffrey Irving for discussions about many of the ideas in this post.)

I. Definition
=============

We will try to define what it means for a question-answering system **A** to be “ascription universal.”

1. Ascribing beliefs to A
-------------------------

Fix a language (e.g. English with arbitrarily big compound [terms](/approval-directed-algorithm-learning-bf1f8fad42cd)) in which we can represent questions and answers.

To ascribe beliefs to **A**, we ask it. If **A**(“are there infinitely many twin primes?”) = “probably, though it’s hard to be sure” then we ascribe that belief about twin primes to **A**.

This is not a general way of ascribing “belief.” This procedure wouldn’t capture the beliefs of a native Spanish speaker, or for someone who wasn’t answering questions honestly. But it can give us a sufficient condition, and is particularly useful for someone who wants to use **A** as part of an alignment scheme.

Even in this “straightforward” procedure there is a lot of subtlety. In some cases there are questions that we can’t articulate in our language, but which (when combined with **A**’s other beliefs) have consequences that we can articulate. In this case, we can infer something about **A**’s beliefs from its answers to the questions that we can articulate.

2. Ascribing beliefs to arbitrary computations
----------------------------------------------

We are interested in whether **A** “can understand everything that could be understood by someone.” To clarify this, we need to be more precise about what we mean by “could be understood by someone.”

This will be the most informal step in this post. (Not that any of it is very formal!)

We can imagine various ways of ascribingbeliefs to an arbitrary computation **C**. For example:

* We can give **C** questions in a particular encoding and assume its answers reflect its beliefs. We can either use those answers directly to infer **C**’s beliefs (as in the last section), or we can ask what set of beliefs about latent facts would explain **C**’s answers.
* We can view **C** as optimizingsomething andask what set of beliefs rationalize that optimization. For example, we can give **C** a chess board as input, see what move it produces, assume it is trying to win, and infer what it must believe. We might conclude that **C** believes a particular line of play will be won by black, or that **C** believes general heuristics like “a pawn is worth 3 tempi,” or so on.
* We can reason about how **C**’s behavior depends on facts about the world, and ask what state of the world is determined by its current behavior. For example, we can observe that **C**(113327) = 1 but that **C**(113327) “would have been” 0 if 113327 had been composite, concluding that **C**(11327) “knows” that 113327 is prime. We can extend to probabilistic beliefs, e.g. if **C**(113327) “probably” would have been 0 if 113327 had been composite, then we might that **C** knows that 113327 is “probably prime.” This certainly isn’t a precise definition, since it involves considering logical counterfactuals, and I’m not clear whether it can be made precise. (See also ideas along the lines of [“knowledge is freedom”](https://www.lesswrong.com/posts/b3Bt9Cz4hEtR26ANX/knowledge-is-freedom).)
* If a computation behaves differently under different conditions, then we could use restrict attention to a particular condition. For example, if a question-answering system appears to be bilingual but answers questions differently in Spanish and English, we could ascribe two different sets of beliefs. Similarly, we could ascribe beliefs to any subcomputation. For example, if a part of **C** can be understood as optimizing the way data is laid out in memory, then we can ascribe beliefs to that computation about the way that data will be used.

Note that these aren’t intended to be efficient procedures that we could actually apply to a given computation **C**. They are hypothetical procedures that we will use to define what it means for **A** to be universal.

I’m not going to try to ascribe a single set of beliefs to a given computation; instead, I’ll consider all of the reasonable ascription procedures. For example, I think different procedures would ascribe different beliefs to a particular human, and don’t want to claim there is a unique answer to what a human “really” believes. A universal reasoner needs to have more reasonable beliefs than the beliefs ascribed to that a human using any particular method.

An ascription-universal reasoner needs to compete with any beliefs that can be ascribed to **C**, so I want to be generous with this definition. For example, given a chess-playing algorithm, we might rationalize it as trying to win a game and infer its beliefs about the rules of chess. Or we might rationalize it as trying to look like a human and infer its beliefs about what a human would do. Or something different altogether. Most of these will be kind of crazy ascriptions, but I want to compete with them anyway (competing with crazier beliefs will turn out to just be easier).

It’s not totally clear what counts as a “reasonable” ascription procedure, and that’s the biggest source of informality. Intuitively, the key property is that the ascription itself isn’t doing the “hard work.” In practice I’m using an informal extensional definition, guided by examples like those in the bulleted list.

3. Comparing beliefs
--------------------

What does it mean to say that one agent is “better-informed” than another?

It’s natural to try to express this in terms of empirical information about the world, but we are particularly interested in the different inferencesthat agents are able to draw from the same data. Another natural approach is to compare their “knowledge,” but I have no idea how to define knowledge or justified belief. So I’m reduced to working directly with sets of beliefs.

Consider two sets of beliefs, described by the subjective expectations 𝔼¹ and 𝔼². What does it mean to say that 𝔼¹ is better-informed than 𝔼²?

This framing makes it tempting to try something simple: “for every quantity, 𝔼¹’s belief about that quantity is more accurate.” But this is property is totally unachievable. Even if 𝔼¹ is obtained by conditioning 𝔼² on a true fact, it will almost certainly happen to update in the “wrong” direction for some claims.

We will instead use a subjective definition, i.e. we’ll define this concept from a particular epistemic position represented by another subjective expectation 𝔼.

Then we say that 𝔼¹ **dominates** 𝔼² (w.r.t. 𝔼) if, for every bounded quantity X and for every “nice” property Φ:

* 𝔼[X|Φ(𝔼¹, 𝔼²)] = 𝔼[𝔼¹[X]|Φ(𝔼¹, 𝔼²)]

(By “nice” I mean something like: simple to define and open in the product topology, viewing 𝔼¹ and 𝔼² as infinite tables of numbers.)

Intuitively, this means that 𝔼 always “trusts” 𝔼¹, even if given arbitrary information about 𝔼¹ and 𝔼². For example, if 𝔼 was told that 𝔼¹[X] ≈ *x* and
𝔼²[X] ≈ *y*, then it would expect X to be around *x* (rather than *y*)*.* Allowing arbitrary predicates Φ allows us to make stronger inferences, effectively that 𝔼 thinks that 𝔼¹ captures *everything* useful about 𝔼².

I’m not sure if this is exactly the right property, and it becomes particularly tricky if the quantity X is itself related to the behavior of 𝔼¹ or 𝔼² (continuity in the product topology is the minimum plausible condition to avoid a self-referential paradox). But I think it’s at least roughly what we want and it may be exactly what we want.

Note that dominance is *subjective*, i.e. it depends on the epistemic vantage point 𝔼 used for the outer expectation. This property is a little bit stronger than what we originally asked for, since it also requires 𝔼 to trust 𝔼¹, but this turns out to be implied anyway by our definition of universality so it’s not a big defect.

Note that dominance is a property of the *descriptions* of 𝔼¹ and 𝔼². There could be two different computations that in fact compute the same set of expectations, such that 𝔼 trusts one of them but not the other. Perhaps one computation hard-codes a particular result, while the other does a bunch of work to estimate it. Even if the hard-coded result happened to be correct, such that the two computations had the same outputs, 𝔼 might trust the hard work but not the wild guess.

4. Complexity and parameterization
----------------------------------

There are computations with arbitrarily sophisticated beliefs, so no fixed **A** can hope to dominate everything. To remedy this, rather than comparing to a fixed question-answerer **A**, we’ll compare to a parameterized family **A**[**C**].

I’ll consider two different kinds of potentially-universal reasoners **A**:

* In the “idealized” case, **A**[**C**] depends only on the complexity of **C**.
For example, we might hope that an *n*-round debate dominates any beliefs that could be ascribed to a fast computation with (*n*-1) rounds of [alternation](https://en.wikipedia.org/wiki/Alternating_Turing_machine). In particular, this **A**[**C**] is the same for any two computations **C** of the same complexity.
* In the “practical” case, **A**[**C**] depends on the complexity of **C** but also uses the computation **C** as a hint. For example, if **C** is the training process for a neural net, then we might take **A**[**C**] to be a debate in which the debaters are able to share weights and activations with the neural net throughout the entire training process.

I’m generally interested in the case where **A**[**C**] is only slightly more powerful than **C** itself. This mirrors the setting where a universal Turing machine is able to run any other Turing machine with only a modest slowdown.

Putting it all together
-----------------------

We say that a set of beliefs 𝔼ᴬ *epistemically dominates* a computation **C** (w.r.t. some beliefs 𝔼 and language L) if the beliefs ascribed to A by the “straightforward” procedure, using L, dominate (w.r.t. 𝔼) the beliefs ascribed to **C** by any reasonable ascription procedure.

We say that a family of question-answering systems **A**[**·**] are *ascription universal* (w.r.t. 𝔼 and L) if **A**[**C**] epistemically dominates **C** for every computation **C**.

II. Discussion
==============

Why is (subjective) dominance sufficient?
-----------------------------------------

This universality condition requires that *we* believe that **A**[**C**] is better-informed than **C**. Naively we might have wanted it to *actually* be the case that **A**[**C**] is better-informed than **C**; the stronger condition is clearly unachievable, but why should we be satisfied with the weaker condition?

In [applications](/informed-oversight-18fcb5d3d1e1) [of this](/universality-and-consequentialism-within-hch-c0bee00365bd) [property](/universality-and-model-based-rl-b08701394ddd), the subjective condition is what we need in order for *us* to believe that **A**[**C**] will cope with the challenges posed by **C**. For example, suppose that **C** formulates a plan to “trick” **A**[**C**]. Then the subjective universality condition implies that we don’t expect **C** to succeed.

This isn’t as good as actually knowing that **C** won’t succeed. But I think it should be good enough for us — the reason we are thinking about AI safety is because we are concerned that something bad will happen. If we find a technique that defuses this argument, then we’ve addressed the motivating problem. It may still be the case that bad things happen (and we should still search for additional reasons that bad things might happen), but we don’t particularly expect them to.

Of course if you select over a large number of computations, then you may find one that will succeed in tricking **A**. But if we are concerned about that, then we can instead apply ascription universality to the entire process including the selection.

Why trust opaque computation?
-----------------------------

If **C** uses some clever heuristics that I don’t understand, then **C**’s “beliefs” might be excellent, but I might not *expect* them to be excellent. In this sense understanding may seem almost vacuous. If there is some heuristic that I trust, wouldn’t **A** just use it?

To see why the definition is demanding, consider the special case where **C** performs an extensive search to find a computation that works well empirically. For example, **C** might be the following computation:

* Start with a training set of (image, label) pairs.
* Search over simple programs to find one that makes good predictions.
* Run that simple program on a new image to predict its label.

In this case, we can ascribe beliefs to **C** about the contents of the new image. And because those beliefs are coming from a simple program that works empirically, I expect them to be accurate (in some respects).

For example, a simple classifier **C** may “believe” that the new image contains a particular curve that typically appears in images labeled “dog;” or a really sophisticated classifier may perform complex deductions about the contents of the scene, starting from premises that were empirically validated on the training set.

So it’s not OK for **A** to simply ignore whatever heuristics **C** is using — if those heuristics have the kind of empirical support that makes us think they actually work, then A needs to be able to understand everything that those heuristics imply about the domain.

Why be so general?
------------------

I’ve formulated universality as competing with arbitrary computations **C**. It seems totally possible that the form of **C** discussed in the last section — searching for a program that works well in practice and then using it in a new situation — is so central that the definition of universality should focus entirely on it.

One reason to use the broader definition is because sometimes this “selection” process can be embedded in a non-trivial way in a larger computation. For example, if I have a sufficiently large group of humans, I might expect memetic selection to occur and produce systems that could be said to have “beliefs,” and I’d like universal systems to dominate those beliefs as well.

The other reason to use this very general definition is because I don’t see an easy way to simplify the definition by using the additional structural assumption about **C**. I do think it’s likely there’s a nicer statement out there that someone else can find.

Universal from whose perspective?
---------------------------------

Unfortunately, achieving universality depends a lot on the epistemic perspective 𝔼 from which it is being evaluated. For example, if 𝔼 knows any facts, than a universal agent must know all of those facts as well. Thus “a debate judged by Paul” may be universal from Paul’s perspective, but “a debate arbitrated by Alice” cannot be universal from my perspective unless I believe that Alice knows everything I know.

This isn’t necessarily a big problem. It will limit us to conclusions like: Google engineers believe that the AI they’ve built serves the user’s interests reasonably well. The user might not agree with that assessment, if they have different beliefs from Google engineers. This is what you’d expect in any case where Google engineers build a product, however good their intentions.

(Of course Google engineers’ notion of “serving the user’s interests” can involve deferring to the user’s beliefs in cases where they disagree with Google engineers, just as they could defer to the user’s beliefs with other products. That gives us reason to be less concerned about such divergences, but eventually these evaluations do need to bottom out somewhere.)

This property becomes more problematic when we ask questions like: is there a way to [seriously limit the inputs and outputs to a human while preserving universality of HCH](/universality-and-security-amplification-551b314a3bab)? This causes trouble because even if limiting the human intuitively preserves universality, it will effectively eliminate some of the human’s knowledge and know-how that can [only be accessed on large inputs](https://medium.com/@weidai/to-put-it-another-way-a-human-translator-has-learned-a-lot-of-valuable-information-much-of-it-48457f95b9bf), and hence violate universality.

So when investigating schemes based on this kind of impoverished human, we would need to evaluate universality from some impoverished epistemic perspective. We’d like to say that the impoverished perspective is still “good enough” for us to feel safe, despite not being good enough to capture literally everything we know. But now we risk begging the question: how do we evaluate whether the impoverished perspective is good enough? I think this is probably OK, but it’s definitely subtle.

I think that defining universality w.r.t. 𝔼 is an artifact of this definition strategy, and I’m optimistic that a better definition wouldn’t have this dependence, probably by directly attacking the notion of “justified” belief (which would likely also be useful for actually establishing universality, and may even be necessary). But that’s a hard problem. Philosophers have thought about very similar problems extensively without making the kind of progress that seems adequate for our purposes, and I don’t see an immediate angle of attack.

III. Which A might be universal?
================================

Two regimes
-----------

I’m interested in universality in two distinct regimes:

* Universality of idealized procedures defined in terms of perfect optimization, such as [debate](https://arxiv.org/abs/1805.00899) under optimal play or [max-HCH](/humans-consulting-hch-f893f6051455), where **A**[**C**] depends only the computational complexity of **C**.
* Universality of practical variants, in which the perfect optimization is replaced by a bounded optimization, which might use the computation **C** as a “hint.” For example, we might consider amplification or debate where the agents are allowed to inspect a transcript of **C**; or if **C** is a neural net training process, we might train them jointly with **C**.

Idealized models
----------------

**A**[**C**] could be:

* Adebate, judged by a human, with perfect debaters, whose length depends on the complexity of **C**. (Hopefully in such a way that **A**[**A**[**C**]] is only a slightly longer debate than **A**[**C**].)
* max-HCH, using perfect optimization rather than a bounded optimizer. Again, we can impose a budget limit that depends on the complexity of **C**, ideally such that **A**[**A**[**C**]] does not involve a much larger computation than **A**[**C**].
* Use HCH to implement two functions, Info and Answer. Define **A**[**C**](Q) to be Answer(Q, *a\**)*,* where *a\* =* argmax Info(*a*, Q*),* and the optimization is over advice strings *a* of length comparable to the description of **C**.

I don’t think any of these would be universal if instantiated with an open-ended human policy — providing arbitrary complex inputs to a human is likely to lead to bad behavior. Instead, I imagine restricting the human to some small enough space of inputs that they can behave “reasonably” over the whole space, while still being rich enough to be universal. Even as such, I think [additional measures](/universality-and-consequentialism-within-hch-c0bee00365bd) might be needed to achieve security (since a team of humans may behave badly in some exotic corners of the input space, even if individual humans are sheltered from exotic inputs).

Practical models
----------------

In practice I’m particularly interested in the case where **C** has the special form described in the “Trusting opaque computations” section: it trains a neural net on some dataset, and then runs that neural network on new inputs.

**A**[**C**] could be:

* Debates arbitrated by a human where the debaters are trained jointly with **C**, and can make references to individual examples from the dataset.
* max-HCH where the agent is trained jointly with **C**, and the overseer can consult individual entries from the dataset.
* HCH, combined with interpretability techniques that allow it to get information about what **C** is thinking and to relate its behavior to the dataset on which it was trained.

For each of these practical variants, I don’t expect it to be universal unless the corresponding idealized model is. So we can think of dividing the universality question into two pieces:

* Would our procedure be universal *if the optimization was performed perfectly*?
* Is weight sharing / interpretability strong enough to solve the optimization problems needed for universality?

I think none of the suggestions in this section are plausibly universal without some additional mechanisms, and so even if HCH or debate is universal, additional ideas are needed in order to translate those to practical universal systems. I’m still inclined to focus on the idealized case, because I want to better understand what is needed for universality without thinking about algorithmic difficulties.

IV. Which C are hard to epistemically dominate?
===============================================

The definition in this post was somewhat informal, but the discussion in this appendix will be extremely informal. The purpose is to communicate some of my early thinking about universality and my sense for what a universal reasoner might look like.

The first three examples will consider programs **C** with the following form:

* Iterate over programs to find an *f* that successfully predicts the answers to some dataset of historical questions for which we know the answer.
* Use that program to predict the answers to future questions.

The different examples explore features of *f* that might make it difficult to dominate the overall computation.

Deduction
---------

Suppose *f* takes the following form:

* Given a question and context, it performs a short series of deductions from facts in the context, together with some axioms (e.g. “Bob’s eyes are more likely to twitch when he is lying,” the rules of poker, *etc.*)
* When it arrives at answers that constrain the possible answers to the given question, it outputs an answer consistent with those constraints.

We could attribute the statements produced by this process to **C** as beliefs.We expect this set of beliefs to have useful information because it was selected for making correct predictions.

Fortunately, if HCH is able to see the training dataset then it may be able to arrive at similarly accurate views:

* It can compute that P(observations | axioms) is high by following along with the deductive process used by *f .*
* Bayes implies that the axioms are likely to be true (fully capturing the argument which leads us, from our current perspective, to expect them to be true).
* Once HCH knows the axioms are likely to be true, it can follow along with the deductive process to reproduce all of *f*’s beliefs.

Modeling
--------

Suppose *f* takes the following form:

* Given a question Q and context, set up a relevant physical situation.
* Use a dynamics model M to predict what will happen at some future time.
* Use the resulting state to answer Q.

We could say that **C** believes everything implied by the outcome of this physical simulation. We expect these beliefs to be accurate because *f* has predicted well in the past.

As in the deduction case, hopefully HCH can compute that P(observations|M is accurate) is high, since it can also compute the consequences of M. Then Bayes implies the dynamics model is accurate, and HCH can use that model to compute physical states.

Inferring all the beliefs from a dynamics model is not trivial though. As an extreme example, if *f* is performing an atom-by-atom simulation of a room, and that room contains Alice and Bob, then we could ascribe extensive beliefs about Alice and Bob to the computation **C**.

(Here we run head on into the fuzziness about what counts as a “reasonable” ascription procedure, but for the moment I’ll assume that some reasonable procedure ascribes beliefs about Alice and Bob to the computation.)

To compete with these ascriptions, HCH needs to infer those high-level beliefs about Alice and Bob from the low-level computation involving atoms. One way to do this is to search over possible “bridging” hypotheses that relate low-level physical facts to high-level facts about the environment. If such a hypothesis can explain additional high-level facts, then a Bayesian can learn that it is true. Similarly, if the bridging hypothesis relates facts about the model to constraints we know from the high-level interpretation, then the Bayesian can potentially use that as evidence. (This kind of reasoning will be discussed in a bit more detail in the next section.)

We could further hope that searching for a bridging hypothesis isn’t much harder than performing the original search over low-level physics, given that the low-level physics needed to explain a bunch of high-level facts and so already must encode some part of that correspondence.

(Note that the “deduction” example in the previous case could also involve alien concepts or models, in which case the same kind of work would be needed.)

Alien reasoning
---------------

In the previous section we described two styles of reasoning we already understand. But there are probably many kinds of reasoning that work well in practice but that would be more alien, and those might be more challenging. This section will explore one example in some detail to try to help anchor our reasoning about the general phenomenon. It will also elaborate on some of the reasoning about “bridging” hypotheses mentioned in the last section.

Suppose that our predictions are always of the same form (e.g. what is the probability the stock market will go up today), and *f* works as follows (the details are long but not very important):

* Find the PSD matrix A with maximum log determinant subject to the constraints in the next bullet points, then output the (0, 0) entry.
* There is an implicit correspondence between the rows/columns of A, and some uncertain properties X(0), X(1), X(2), …. (which we’ll view as 0–1 variables), where X(0) is the property we want to forecast.
* If the (*i*, *j*) entry of A represented the expectation E[X(*i*)X(*j*)], then the matrix would necessarily satisfy a bunch of constraints, which we impose A. For example:
* If the context implies that X(*i*) = 1, then E[X(*i*)X(*j*)] = E[X(*j*)] = E[X(*j*)²], so A(*i*, *j*) = A(*j*, *j*).
* If X(*i*) and X(*j*) together imply X(*k*), then we must have E[X(*i*)X(*j*)] ≤ E[X(*i*)X(*k*)] and hence A(*i*, *j*) ≤ A(*i*, *k*).
* For any constants *a*, *b*, …, E[(*a* X(1) + *b* X(2) + … )²] ≥ 0 — i.e., the matrix A must be PSD.

The chosen matrix A(opt) corresponds to a set of beliefs about the propositions X(*i*), and we can ascribe these beliefs to **C**. Because *f* predicts well, we again expect these beliefs to say something important about the world.

I chose this procedure *f* in part because we can give a kind of argument for why the matrix A(opt) should tend to encode accurate beliefs. But I don’t think that a universal reasoner can make use of that argument:

* Finding the argument that *f* works is an additional problem, beyond finding *f* itself, which might be much harder.
* A comprehensible version of that argument may be much larger than the strategy itself, so even in the idealized cases like debate with perfect optimization, we may need to increase the scale.
* I don’t expect that all “good” reasoning strategies have clean understandable arguments in their favor (and even in this case, if it the scheme worked well it would be largely an empirical fact rather than a consequence of the simple theorems we could prove). I think this kind of example is useful because we can easily imagine a human debate judge not having the argument while still being apparently universal. This makes it a useful analogy for cases where the argument really doesn’t exist.

Instead, I think a universal reasoner needs to be able to infer the efficacy of this reasoning procedure from its empirical success. It’s relatively easy for a Bayesian to learn the regularity “*f* makes good predictions.” Recovering the rest of the matrix A, and learning how to interpret and whether to trust them, is the hard part.

This is going to require the same kind of bridging/identification we discussed in the last section. Let’s write X(A) for the set of beliefs about the world implied by the “intended” identification. Searching over possible identifications to find X (or something like it) is the only way we can ever relate the rows of A to the quantities X(*i*). Again, we can hope that it isn’t much harder than finding the original reasoning procedure.

I think that a sufficiently sophisticated Bayesian would probably be able to learn to trust X(A):

* If *f* is performing well enough that we think it’s more likely to be right in the future, then the Bayesian is going to end believing some claim like “the predictions of *f* are good” (since it explains the data so well).
* This is a complicated statement, and without some kind of explanation this claim has a low prior probability (roughly decaying with the complexity of *f)*. The Bayesian is motivated to find an explanation with higher prior probability.
* The correspondence X can explain the constraints on the matrix A, in terms of facts that we already know about the world. This explanation may end up being simpler (or at least higher prior) than a direct enumeration of the constraints on A — I hope (and think it’s plausible) that this happens iff we’d actually believe on reflection that X(A) captures reality.
(To the extent that we are uncertain and think A’s beliefs have a non-negligible chance of capturing reality, then hopefully we can capture that by the same mechanism by ending up with a non-degenerate posterior.)
* Now the Bayesian is faced with at least two kinds of explanations:
(a) “If you use the constraints implied by correspondence X(A) + positive semidefiniteness, and then optimize log det, you get a matrix A for which X(A) makes good predictions,”
(b) “The actual situation in the real world is described by positive semi-definite matrices with higher log determinant (under the correspondence X).”
* Explanation (b) is explaining two things at once: both why the optimization done by *f* respects the constraints on our beliefs, and why that optimization leads to good predictions. Hopefully this is simpler than making two separate bridging claims, one which explains *f* as respecting the constraints implied by X, and one which claims that *f* makes good predictions. Ideally, this 2-for-1 that favors (b) exactly mirrors the underlying reasoning that leads us to actually believe that X(A) is correct, rather than resembling what we know about reality and making good predictions “by coincidence.”

This is a pretty speculative discussion — it’s not very careful, and it’s hard to make it careful in part because I don’t have a formalization of Bayesian reasoning that can even really be applied to this setting. But it seems to match my intuitions about what reasonable Bayesian reasoning “should” do, which gives me a lot more optimism that a careful Bayesian would be able to epistemically dominate **C**.

Deliberation and self-improvement
---------------------------------

Often we expect the computation **C** to have accurate beliefs because it uses a strategy that appears to work in practice — the last 3 examples have discussed that case. But there are other reasons to trust a computation.

For example, humans often write code and trust it (to some extent) even without extensive empirical testing — instead, we have a reason to think it will work, and need only modest testing to make sure that we haven’t made an error in our implementation or reasoning. If I write an automated mathematician that works by finding proofs that pass a proof checker, I don’t expect it to be correct because of the empirical record (Empirical data backs up some key assumptions, but isn’t being used to directly establishing the correctness of the method.)

Likewise, if we train a powerful agent, that agent might initially use strategies that work well in training, but over time it might use learned reasoning to identify other promising strategies and use those. Reasoning might allow it to totally skip empirical testing, or to adopt the method after much less testing than would have been necessary without the reasoning.

To dominate the beliefs produced by such reasoning, we can’t directly appeal to the kind of statistical inference made in the previous section. But in these cases I think we have access to an even more direct strategy.

Concretely, consider the situation where **C** contains a process *f* that designs a new reasoning process *g*. Then:

* From the outside, we trust *g* because we trust *f* and it trusts *g.*
* An otherwise-universal reasoner **A** will dominate *f*’s beliefs, and in particular if *f* is justified in thinking that *g* will work then **A** will believe that and understand why.
* Once we understand *f’*s beliefs, dominating *g* is essentially another instance of the original ascription universality problem, but now from a slightly stronger epistemic state that involves both what 𝔼 knows and what *f* knows. So unless our original approach to universality was tightly wedded to details of 𝔼, we can probably dominate *g*.

At the end of the day we’d like to put all of this together into a tight argument for universality, which will need to incorporate both statistical arguments and this kind of dynamic. But I’m tentatively optimistic about achieving universality in light of the prospect of agents designing new agents, and am much more worried about the kind of opaque computations that “just work” described in the last few sections.

Towards formalizing universality


(Revisited)
-----------

[![Paul Christiano](https://miro.medium.com/v2/resize:fill:88:88/1*BNjZCuQuRfIgcXCBMipuBw.jpeg)](https://paulfchristiano.medium.com/?source=post_page-----ce0e0a3b9b4d--------------------------------)[![AI Alignment](https://miro.medium.com/v2/resize:fill:48:48/1*N56Qc5-aHTcfGff0scntKQ.png)](https://ai-alignment.com/?source=post_page-----ce0e0a3b9b4d--------------------------------)[Paul Christiano](https://paulfchristiano.medium.com/?source=post_page-----ce0e0a3b9b4d--------------------------------)

·[Follow](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F_%2Fsubscribe%2Fuser%2F57f1a655a613&operation=register&redirect=https%3A%2F%2Fai-alignment.com%2Ftraining-robust-corrigibility-ce0e0a3b9b4d&user=Paul+Christiano&userId=57f1a655a613&source=post_page-57f1a655a613----ce0e0a3b9b4d---------------------post_header-----------)

Published in[AI Alignment](https://ai-alignment.com/?source=post_page-----ce0e0a3b9b4d--------------------------------)·13 min read·Jan 21, 2019--

5

Listen

Share

Even if we are very careful about how we deploy ML, we may reach the point where a small number of correlated failures could quickly become catastrophic. Powerful models could actively undermine safeguards, resist attempts at correction, and manipulate their operators.

I think the long-term safety of ML systems requires being able to rule out this kind of behavior, which I’ll call *unacceptable*, even for inputs which are extremely rare on the input distribution.

In this post I’ll explain why I think this goal is reasonably likely to be achievable, by highlighting the three ingredients that seem most important to me: adversarial training, transparency, and relaxations.

Preliminaries
=============

This is an updated version of [this post](/techniques-for-optimizing-worst-case-performance-39eafec74b99). It mostly has the same content, but my thinking now feels clearer and there is a more detailed description of my hopes about relaxations.

Disclaimer
----------

I’m trying to communicate intuitions about whether this problem is likely to be soluble someday, rather than making concrete progress or stake out new ground. I’m a theorist who doesn’t work in this area.

Defining acceptable
-------------------

I’ll talk about “acceptable”behavior, but be deliberately vague about what “acceptable” means.

In different contexts, different behavior might be acceptable and it’s up to the user of these techniques to decide. For example, a self-driving car trainer might specify: Crashing your car is tragic but acceptable. Deliberately covering up the fact that you crashed is unacceptable.

My key assumptions about “acceptability” are:

* As long as the model *always* behaves acceptably, and achieves a high reward *on average*, we can be happy. The model may still make mistakes, especially when it encounters novel situations, but if we are mindful about how we use AI we can avoid these mistakes escalating into irreversible catastrophes.
* Requiring a model to always behave acceptably wouldn’t make a hard problem too much harder. It requires some knowledge about what is acceptable, but the complexity of that knowledge doesn’t scale up with the difficulty of the task. Unacceptable behavior is a failure of commission rather than omission, and so avoiding it doesn’t require intricate understanding of the domain. Moreover, we are OK if the model conservatively avoids behaviors that are anything close to unacceptable, allowing it to use simpler concepts.

For any given task there may or may not be a notion of acceptability that meets these properties. For a chess-playing AI, it seems pretty likely to be possible — if you find yourself murdering your opponent you’ve crossed a bright line, and understanding that line doesn’t become harder as you get better at chess. For an AI that’s *intended* to frequently overrule strenuous objections from humans, there may be no obvious bright lines to prevent catastrophe.

My view is that we can probably get everything we want out of life by applying AI systems to problems where there is a clear notion of acceptability, even in a ruthlessly competitive world where other actors don’t care about safety. If you agree with me, then read “acceptable” as referring to whatever concept you think has those key properties. If you don’t think there is any concept with these properties, then this post is less likely to be interesting to you.

This is the second guarantee from “[Two guarantees](/two-guarantees-c4c03a6b434f),” and is basically [corrigibility](https://intelligence.org/files/Corrigibility.pdf).

I. Basic framework: *adversar*ial training
==========================================

In addition to training our agent to perform well on natural inputs, we can try to find adversarial inputs on which it behaves unacceptably and then penalize it for that behavior.

Adversarial training requires being able to reliably detect when the agent behaves unacceptably. I’m optimistic that we can achieve this by using debate or amplification to train an overseer to answer the question “is this behavior unacceptable?” Obviously the feasibility of this depends on the details of the task and what we consider unacceptable. I discuss the problem of training an overseer [here](/informed-oversight-18fcb5d3d1e1), and I’ll set it aside for the rest of this post.

Adversarial training also requires an adversary who is able to find inputs on which the agent behaves unacceptably. In current practice the adversary is most often gradient descent over the input space, but in the long run I think this technique can’t find most possible failures. I’m optimistic it may be possible to train a strong enough adversary, by combining transparency techniques (section II), relaxations that make the adversary’s job easier (section III), and amplification.

Even with both of those pieces, getting adversarial training to work well is an important research problem, and there could be other fundamental problems lurking in store. But I would be relatively optimistic that other challenges could be resolved. Overall, I think the most likely fundamental obstruction is the difficulty of implementing a good adversary.

II. Informed adversaries
========================

The challenge
-------------

To see how hard the adversary’s job could get, consider an agent that works as follows:

* Predict whether the current input could possibly appear during training.
* If the current input definitely couldn’t appear during training, then behave unacceptably.

(This is an exotic failure mode, but could arise e.g. as part of a “treacherous turn,” a situation where training produces a sophisticated enough consequentialist to understand that it needs to perform well during training in order to avoid a negative reward.)

In this context, the adversary is directly pitted against the agent. If there is *any* predictable difference between the adversary’s challenges and the actual situations that appear at test time, then the agent can exploit that difference to fail only at test time.

Looking inside
--------------

This game seems almost impossible for the adversary, but we do have one big advantage: we have access to the internals of the agent, so in principle could give the adversary considerable information about what the agent is “thinking.”

For example, we can examine internal features computed by the model and observe how those features depend on the input and each other. There is a growing field dedicated to interrogating neural networks to understand what they are doing. ([This article](https://distill.pub/2018/building-blocks/) describes some relevant techniques and makes an early attempt at combining them to get some insight into what a model is doing.)

On top of this kind of explicit transparency, we can have the adversary and agent share most of their computation, so that the adversary can more easily understand whatever the agent can understand (and can learn to manipulate the internal concepts used by the agent in the same way the agent does). This seems particularly important in combination with amplification, discussed briefly below.

Between these two techniques, I think it’s realistic to hope for the adversary to understand “everything the agent understands,” and moreover to understand how the agent is disposed towards unacceptable behavior. I don’t think this would automatically resolve robustness problems, but I do think it could defuse the adversarial aspect of the problem and give us traction on the remaining problem.

For example, the adversary could start by generating a very broad range of situations in which it would be particularly easy for the agent to behave unacceptably. Once the agent learned to pass these tests, the adversary could try to understand *why* it was passing these tests. If the agent was using a misguided picture of acceptability, the adversary could choose an input that highlights the mistake. If the agent is actively trying to detect whether it’s at training time, the adversary can look at the test that the agent uses and try to generate an input specifically to fool this test. (Though this problem may be too hard even if you understand exactly what the agent is doing — see section III.)

Approaches
----------

**RL**. The adversary’s job is to perform the task “find an input on which the agent will behave unacceptably.” One approach is to train this adversary directly with an objective that rewards identifying unacceptable behavior.

We can think of such an adversary as an environment, whose observations are the agent’s actions and whose actions are the agent’s observations. We can then incorporate information about what the agent is thinking by allowing the adversary to interact with transparency tools before needing to interact with the environment.

We can also train this adversary jointly with the agent (being careful about how we take derivatives!), hoping that the intermediate state produced by the agent will be useful for an adversary trying to attack that agent.

**Demonstrations**. This RL problem is extremely difficult. It would be easier if the adversary could learn from demonstrations of successful attacks rather than having to solve an exploration problem from scratch.

For example, if our transparency tools were powerful enough, then we might hope that a human could reliably use them to find inputs where the model would behave unacceptably. If we only needed a few examples to train the agent to behave acceptably then we wouldn’t need an automated adversary at all. If we needed many examples, we could use these human demonstrations to initialize a model that is able to implement an attack.

**Amplification**. Rather than using a human as an expert adversary, we could train the adversary with iterated amplification. This seems likely to be necessary in cases where the model is so complex that an individual human can’t make sense of its behavior.

We can view this process as training a set of more advanced tools that can answer higher-level questions about “what the model is thinking,” on top of some library of core building blocks for interpretability. The human constructs more and more powerful tools by using one generation of tools to produce training data for the next generation. The tools at the very highest level can be used to directly answer the question “where is the model likely to behave unacceptably?”, and the human provides training data for these very hard question by using previous generations of tools to answer questions like “why did the model decide to take that action?” and “why is the model trying to achieve this goal?”

Relationship to ascription universality
---------------------------------------

Even with the aid of transparency tools, I’m only optimistic about an adversary being able to find an attack if they are able to understand everything the model understands. This feels closely related to [ascription universality](/towards-formalizing-universality-409ab893a456).

Note that ascription universality isn’t sufficient on its own: we ultimately care about what the model would have done under different conditions, which is not the kind of thing the model can be naturally said to know. We need some other ideas; in the description above the magic is done by interpretability techniques. The actual interface between interpretability and universality is blurry and I have a very foggy picture of this space.

I don’t have any clear picture of what *is* enough; at this point I’m just feeling out intuitions. I’m planning to revisit this question sometime after getting more clarity on ascription universality.

III. Relaxation
===============

The challenge
-------------

Unfortunately, no matter how completely the adversary understands the agent and how clever they are, we can’t generally expect them to find all points on which the agent would behave unacceptably. The basic problem is that recognizing inputs with a certain property is easier than generating inputs with that property.

As a simple but silly example, suppose our model works as follows:

* Pick a cryptographic puzzle (e.g. “factor RSA-2048”).
* When it sees a solution to that puzzle, it behaves badly.
* Otherwise, it behaves well.

Even the adversary understands perfectly what this model is doing, they can’t find an input on which it will behave badly unless they can factor RSA-2048. But if deployed in the world, this model will eventually behave badly.

Possible approach
-----------------

To deal with this problem, we need to penalize the agent for behaving badly on an input even if we can’t actually find it. This will require being conservative, and sometimes penalizing our agent even when there isn’t actually any input where it would behave badly, since merely telling if a puzzle is soluble can already be computationally infeasible.

(It might sound weird to penalize our agent if it wouldn’t actually ever do anything wrong. For intuition, consider the puzzle case: learning that my model *would* behave badly if it saw a factorization of RSA-2048 already tells me that something has gone wrong, even if turns out that the puzzle is insoluble.)

So we can change the adversary’s task to make it easier. Rather than asking them to actually produce an input, we are asking them to describe a situation in which the model would behave badly (“the input contains a solution to puzzle P”), even if they can’t find a situation with that description or even tell whether one exists.

I’ll call this easier problem the “relaxed” problem (as is typical in optimization), and call the hypothetical situations “pseudo-inputs” (inspired by [Barak et al](https://arxiv.org/pdf/1205.4484.pdf)).

Illustration: SDP relaxation
----------------------------

This kind of relaxation has been explored in practice in the context of verification. For example, [Raghunathan et al](https://arxiv.org/pdf/1811.01057.pdf) use an SDP to certify properties of a neural net (taking small liberties for simplicity):

* Rather than having an attacker specify an input, the attacker specifies some properties of a hypothetical distribution over inputs. In particular, they they specify each expectation of the form 𝔼[*pq*] where *p* and *q* are either input variables or activations in some layer of a neural network.
* We require the attacker’s input to satisfy a bunch of simple properties that are always satisfied by real distributions. For example, if an input *x* is known to be in the interval [-1, 1], then 𝔼[*x*²] ≤ 1. If *z* = ReLU(*x*), then either *x = z* or *z* = 0, so 𝔼[*z²*] = 𝔼[*xz*]. If *f* is an arbitrary function then 𝔼[*f²*] ≥ 0. These inequalities are all linear (in terms of the numbers 𝔼[*pq*]), and so define a convex set over which it is easy to optimize.
* We read off the probability of bad behavior from 𝔼. If we were testing a complex constraint computed by another neural network, then the attacker would also specify the expectations of all of the activations of *that* network, such that the final output is itself just one entry of 𝔼.

Because the attacker’s constraints don’t perfectly characterize the set of achievable moments, it’s not easy to tell whether there is any distribution over inputs that has those moments. (In fact it’s NP-hard.) But on the other hand, if the attacker can’t find any set of moments, then the model really does always work.

The point of performing the relaxation is that it makes the attacker’s job *much* easier. In this case we the relaxation is convex, which means that the attacker can be guaranteed to find an attack if one exists.

I think the relaxations we want to use in the long term won’t be so easy to optimize over. For example, the attacker may still be able to specify properties of a hypothetical distribution, but won’t be allowed to freely specify quite so many degrees of freedom, and in exchange will have to deal with a more complex set of constraints. But hopefully this can still make the adversary’s problem easy enough that an adversary can solve them with the aid of the mechanisms described in section II.

The hope: stronger relaxations
------------------------------

We need to find a relaxation which has two properties:

* It’s still “easy enough” for the agent to never behave unacceptably for any pseudo-input.
* It’s easy for the adversary to find a pseudo-input on which the agent behaves unacceptably, if any exist.

For example, in the case of a model which fails whenever it encounters a factorization of RSA-2048, an attacker needs to be allowed to supply a pseudo-input like “a distribution over pairs of numbers that multiply together to 2048.” Finding that input needs to be easy.

On the flip side, if the adversary has too much flexibility, then they might be able to specify a pseudo-input like “a distribution over inputs that would cause the agent to want to kill everyone.” There need to be some limits on the adversary in order for the game to be possible.

This second requirement is tied up with one of the key properties of acceptability: somehow the agent was able to steer clear of unacceptable behaviors. We are now asking them to solve the harder task, of steering clear of unacceptable behavior not only for actual sequences of thoughts, but also for whatever “pseudo-thoughts” they might think in response to a pseudo-input. Intuitively, we need to force the adversary to respect enough of the agent’s internal structure that this “steering clear” procedure can still work, without forcing the agent to totally cripple itself.

Putting it together
===================

I think the following general plan has a significant chance of producing robustly acceptable behavior:

* Specify a relaxed problem for an adversary, where they specify a distribution over inputs by specifying some of its properties rather than by actually producing examples. We don’t have any clear sense of how this relaxation could work.
* Train our model with iterated amplification to answer the questions: “which action should we take in state *s*?”, “is action *a* unacceptable in state *s*?”, and “what’s a state where you are particularly likely to recommend an unacceptable action?”
The first two questions can potentially be answered directly with amplification, but the last would require some kind of progress in model interpretability (since it’s a question about how the agent would behave on other inputs, rather than a question that the agent itself “knows” how to answer).
* Now we can use the model to generate a pseudo-input on which the agent is likely to behave badly, and use the model to verify that the agent in fact behaves badly on that agent (i.e., have the informed adversary described in section II play the relaxed game in section III instead of forcing it to generate an actual output)

I don’t think this particular procedure is likely to work. But I do feel like we have a plausible angle of attack for getting sufficiently “easy” worst-case guarantees. I don’t think we can see any fundamental obstruction to an approach based on adversarial training, and I feel pretty optimistic about either being able to make significant progress in this area or else identifying a clearer obstruction.

Postscripts
-----------

* To answer “is action *a* unacceptable?” we need to use the model to answer a bunch of sub-questions, like “what are the worst consequences of action *a*?” If the model sometimes chooses bad actions, it’s reasonable to be concerned that it will also sometimes answer these subquestions badly. But we constantly train “is action *a* unacceptable?” on the pseudo-inputs most expected to yield unacceptable behavior, and so an average-case guarantee from the question-answering system implies that it typically answers these subquestions well.
* If we trained the adversary using RL to elicit unacceptable behavior, then we could have a chicken-and-egg problem: the overseer fails to recognize some behavior as unacceptable, and so the adversary isn’t motivated to elicit it, and so the overseer never learns to correctly recognize it. Demonstrations allow us to overcome this problem, if the demonstrator attempts to generate unacceptable behavior even when the overseer hasn’t yet learned to recognize it.
* I find it helpful to think about sections II and III separately, since each of them seems to confront a distinct difficulty. It seems meaningful to think about how we can relax the adversary’s problem, and independently to ask how we can give the adversary insight into where the model might behave badly. But at some point I expect them to blur together, and the whole procedure might not end up looking very much like adversarial training.

Worst-case guarantees


When I imagine proving something about an AI, or making an inductive argument about [amplification](/iterated-distillation-and-amplification-157debfd1616), I think about two different guarantees:

1. The AI behaves well *on average* over some particular distribution of inputs. (The performance guarantee.)
2. The AI never does anything “actively malicious,” on *any* input.
(The control guarantee.)

Definitions
-----------

We can define “behaves well on average” by looking at the average impact of decisions on our utility function. [This post](/thoughts-on-reward-engineering-82b193ec03f6) addresses some subtleties with that definition, including inconsistent preferences, different scales of impact, and long-term consequences. Actually evaluating this quantity, so that it can be used in training, requires [amplification](/iterated-distillation-and-amplification-157debfd1616) and [informed oversight](/the-informed-oversight-problem-1b51b4f66b35).

My intuition about “actively malicious” is best captured by the (informal) idea of [incorrigible](/corrigibility-3039e668638) optimization: our AI should never be actively trying to undermine our understanding or control of the situation. This property seems “easy,” in the sense that it can be satisfied without any domain knowledge or expertise, which makes me optimistic that it is possible to [satisfy it for every input](/techniques-for-optimizing-worst-case-performance-39eafec74b99).

Rationale
=========

Rationale for the performance guarantee
---------------------------------------

The performance guarantee is what does the work; it’s the reason you’d want an AI instead of a pet rock.

If we construct the reward so it reflects the difference in utility between different actions, then the average case is actually a pretty good thing to optimize.

In reality the data distribution is constantly changing over time. So we want an [online guarantee](https://www.cs.huji.ac.il/~shais/papers/OLsurvey.pdf), maximizing the average case performance over a varying sequence, rather than a batch guarantee. Normally deep learning is designed for the batch setting under the iid assumption, and data distributions change slowly enough that this is acceptable. Overall, I don’t think this distinction is a huge deal.

Note that the “average case” guarantee still reflects the overseer’s desire for [corrigibility](/corrigibility-3039e668638). This is formally reflected when (a) the overseer evaluates the expected utility impact of a decision by marginalizing over all of the mistakes that the AI might make in the future, (b) the overseer evaluates individual decisions by assigning a large penalty for incorrigibly doing the wrong thing, but only a small penalty for behaving corrigibly but slightly inefficiently.

Rationale for control guarantee
-------------------------------

Unfortunately, it is only possible to get an average performance guarantee when the rewards are small relative to the total amount of data. If a few bad decisions can destroy all of the value from a lifetime of good decisions, then traditional ML can’t do well on average.

If an AI is *trying* to cause a catastrophe, it can do so very quickly even in apparently benign situations (by breaking the mechanisms that humans use to remain in control of the situation). The goal of the control guarantee is to prevent this kind of catastrophe.

Similarly, if an AI is trying to be difficult, it could intentionally undermine whatever testing regime we use (by failing selectively on inputs that don’t look like tests). The control guarantee should also rule out this kind of intentional sabotage.

Hopefully the control guarantee cuts off the worst catastrophes, making the performance guarantee meaningful.

Other catastrophes
------------------

Even if our AI satisfies the control guarantee, it can make other kinds of catastrophic mistakes (just as an [aligned human](/clarifying-ai-alignment-cec47cd69dd6) could make a catastrophic mistake). For example, the system might be in charge of running a nuclear defense system, and a single failure could lead to a nuclear war. Or an attacker might fool the AI into executing an instruction by impersonating the operator, which in turn open the system to further attack.

Neither the control nor performance guarantee directly address these problems. Instead, anyone who deploys an AI needs to be aware of the system’s limits, to test the system to see where it might fail, to design mechanisms with redundancy, to protect the system for attackers, and to avoid incorrectly assuming perfect performance. (The same measures we would take if delegating to a human who sometimes made mistakes.)

Corrigibility
-------------

The performance and control guarantees interact to create a system that is corrigible: the performance guarantee ensures the system is typically trying to give the human more effective understanding and control over the situation. The control guarantee ensures the system isn’t undermining those measures, for example by constructing a fake narrative for the human or illusory control, by leaving backdoors that can be exploited, etc.

The performance guarantee leaves open the possibility that the system will sometimes fail to inform or empower the human effectively. But as long as those are failures aren’t optimized to be unrecoverable, it seems the human can work around them by recognizing the shortcoming and having the AI optimize for the robustness of human control and understanding.

I think there is a meaningful analogy between this picture and this [post about monitoring and whistleblowing](https://sideways-view.com/2018/02/01/honest-organizations/). and I have a vague intuition that there is some important underlying dynamic that could be better understood.

Amplification
=============

Amplification and performance
-----------------------------

[Amplification](/policy-amplification-6a70cbee4f34) solves a task by (adaptively) breaking it into several subtasks, solving the subtasks, and combining the results.

If we are making an inductive argument about amplification, then the performance guarantee implies the expected *average* performance on subtasks is good, and requires that the expected performance on the overall task is good.

If some of the subtasks make “honest mistakes,” then I think this is a reasonable hope.

In order to make it work we definitely need to have an unequal weighting over subtasks, which corresponds to an unequal sampling of subtasks during training. For example, if we are trying to figure out if any of {A, B, C} are positive, we proceed by independently estimating A, B, C, and we conclude that B is positive, then the estimation of B was a more important subtask than the estimation of A or C.

Even with arbitrary weights on subtasks, preserving average performance might seem too optimistic. In the previous example, if we conclude that all of A, B, C are negative, then any one of the three errors could lead to an erroneous overall conclusion. So if our original error rate was 5%, then our new error rate will be 15%.

In cases like this, it’s wrong of think of “error rate” as the measure of performance that should be preserved inductively. Some questions are harder than others, and should be expected to have higher error rate — if any rational agent tries to figure out whether any one of N quantities is positive, without any clever trick for figuring it out, then they’ll have a harder time than if they were just trying to estimate whether a single quantity is positive. This makes it harder to talk precisely about the guarantees of amplification, but leaves me feeling relatively optimistic about eventually finding a working analysis.

Amplification and control
-------------------------

I find the above intuition plausible when errors are “honest mistakes.” If instead one of the subtasks is optimized to cause trouble, then I think it is very hard (and likely impossible) to correct the problem. So in order to have any hope of maintaining performance, we need to guarantee that none of the subtask solutions are actively malicious.

But in order to guarantee that a subtask isn’t actively malicious, we also need to assume that none of *its* subtasks are actively malicious. If we want to carry out the induction to a reasonable depth, this basically requires a worst-case guarantee. Otherwise even a single failure, in one leaf of an exponentially large tree, could cause trouble.

Unfortunately, a human probably does not satisfy the control guarantee, since they may behave very strangely on some small subset of possible inputs or with small probability. I originally described [security amplification](/universality-and-security-amplification-551b314a3bab) and [reliability amplification](/reliability-amplification-a96efa115687) to gradually eliminate the human control failures (as well as new failures that crop up, e.g. from memetic selection inside the amplification process).

Interestingly, some ways of achieving the control guarantee in the distillation step could also be applied even if the overseer *doesn’t* satisfy the control guarantee, and so could be used as a replacement for security/reliability amplification. This isn’t as crazy as it sounds, since the ability to look at the entire code of a learned model gives us a substantial advantage over dealing with an opaque overseer. It gives a further reason to prioritize [techniques for worst-case performance](/techniques-for-optimizing-worst-case-performance-39eafec74b99) (and particularly interpretability).

Conclusion
==========

We can’t guarantee an AI is aligned if we have only an average-case guarantee, *or* onlyaworst-case guarantee. So achieving both seems like the “minimum viable product” for alignment research.

My original intuition (in mid-2016) was that having two separate guarantees must be an ugly hack, and that the real goal should encapsulate both. That’s no longer so clear to me: I think these two properties interact surprisingly nicely, such that they may actually suffice to get good behavior even though it looks like a weird combination. At the same time, I think attempts to capture both are much less promising than I’d initially believed.

I still think we need more clarity about what we should be trying to prove. I think that having two separate guarantees, one for the worst case and one for the average case, is the current best guess and is the most promising starting point for further research. In the short term, my focus will be on understanding ways in which this structure is inadequate and on independently refining each of these two subgoals.

Two guarantees


The scalability of iterated amplification is closely tied to whether very large teams of humans are “[ascription universal](/towards-formalizing-universality-409ab893a456).” If they are, we [could](/informed-oversight-18fcb5d3d1e1) [hope](/universality-and-model-based-rl-b08701394ddd) that their approval would be an adequate training signal for powerful aligned behavior, and that we can approximate this signal with iterated amplification.

There are many reasons that a team of humans could fail to be universal. For example:

1. They could be unable to understand some kinds of effective reasoning, because they can’t represent relevant concepts, understand particular inferences, or whatever else.
2. They could make errors that derail an otherwise-universal reasoning process.
3. They could use processes that work “most of the time,” but fail catastrophically for a small number of inputs.
4. Errors of type #2 or #3 might produce extreme outputs that tend to propagate through the reasoning process and compound into larger failures (even if they occur with small probability or for a small fraction of inputs).
5. In a large enough computation, errors of type #2 or #3 could eventually give rise to intelligent patterns of behavior that systematically identify and exploit mechanisms by which they could propagate themselves or cause cascading failures. For intuition, note that memetic selection amongst a group of 10¹⁰⁰ humans could be much stronger than the genetic selection that gave rise to life on Earth.

(This is not an exhaustive, exclusive, or even properly parallel list.)

My current view is that #2 and #3 can probably be corrected by traditional methods, even when compounded by #4: computing carefully or redundantly, sanity-checking outputs, structuring computations to be more robust to isolated failures, and so on*.* These problems seem be easier to solve for smarter teams(though also more problematic if left unchecked), and I suspect there is some competence threshold above which they become manageable.

A priori, #2 or #3 compounded by #5 could be harder to cope with:

* An intelligent optimization process could systematically subvert whatever error-correction or robustness mechanisms we try to design.
* For “random” failures, it’s easy to imagine that a sufficiently diverse and extensive set of checks can drive failure rates exponentially low, but malicious optimization is a natural source of highly correlated failures.
* Problem #5 potentially becomes *worse* as we move to larger and larger teams, because such teams may give rise to larger and larger adversaries. This means that our defenses are engaged in a race, instead of fighting against a fixed target which we might expect them to eventually overcome. (Absent an adversary, we still have a “race” in that we need to drive failure probabilities lower and lower for larger teams, but if we get exponential or even linear scaling then this seems feasible.)
* Intuitively, I would feel a bit surprised if random errors and propagating patterns managed to totally overwhelm the optimization being applied by a very large team of humans. But I would not feel so surprised if a large team of humans gave rise to other misaligned optimization power, and if that misaligned optimization eventually became the “main affair.” This is a little bit like the difference between concerns that could extinguish all life on Earth (which seems quite hard to me) and those that could replace life on Earth with more effective competitors (which feels very possible).

This post
---------

I’m going to argue that problem #5, the appearance of consequentialism within HCH, might be manageable. That is, if HCH would be ascription universal if not forthe possibility of malign consequentialism within HCH, then we can probably make it universal even in light of that possibility.

The basic idea is to apply a “sanity check” to each of HCH’s computations. If a computation is optimized for negative consequences, then by ascription universality HCH can identify those consequences, and any output affected by such a malicious computation can be censored. In practice I expect this to be contiguous with the use of [reflection](https://ought.org/projects/factored-cognition/networks-of-workspaces#stage-3-reflection) to sanity-check large computations (and similar mechanisms may be the best way to protect against more prosaic cascading failures).

The arguments in this post are not tight, and indeed it’s not clear they could be totally tight without clarifying the assumption “HCH could be universal if not forthe possibility of malicious consequentialism.” But I find these arguments suggestive enough that I think the current focus should be on problems #1 and #3 — whether it is possible for HCH to robustly understand complex reasoning even in principle — without worrying about the possible emergence of consequentialism. If this is successful, I think there is a good chance that we could address issues #2, #4, and #5.

Filtering
=========

Given a “transcript” τ of HCH(Q, N)’s behavior, i.e. a record of all of the questions, subquestions, and answers, we can ask two questions:

* Info(*a*, *a*′, τ) : “Which of the two arguments *a* and *a*′ is more useful for determining whether the transcript τ contains problematic subcomputations?”
* Trouble(*a*, τ) : “in light of argument *a*, does transcript τ contain problematic subcomputations?”

Now we’ll jointly define filtered-HCH(Q, N) and Filter(τ, N) as follows.

Define Filter(τ, N) as:

* Find the *a*\*[maximizing](/optimizing-with-comparisons-c02b8c0d7877) filtered-HCH(Info(·, ·, τ), N).
(Interpret ⟂ as a tie.)
* Run filtered-HCH(Trouble(*a*\*, τ), N) 3 times, and return the majority vote.
(Interpret a response of ⟂ as a judgment that τ is problematic.)

Define filtered-HCH(Q, N) as:

* First call HCH(Q, N) to get an answer A with transcript τ. But every time you would call HCH to answer a subquestion, call filtered-HCH instead.
* Call Filter(τ, N-1) to test whether this answer is problematic.
* If τ is problematic, return a special symbol ⟂, otherwise return A unmodified. (If you return ⟂, you can eliminate this subtree from the transcript.)

Within HCH, if a question is answered with ⟂, the question-asker needs to respond appropriately. They could try re-asking the question in a different way; if that is unsuccessful they might assume that the question itself represents an attack on HCH and so needs to avoided. What to do next depends on the context — HCH might decide that a particular hypothesis is “malicious” and therefore best dropped from consideration; or that a particular idea is hazardous to think about directly and so it needs to engage in indirect reasoning to try to mitigate the harms; or it might just pass the failure up the tree with some explanation of where the problem appeared.

Note that the pattern of which questions get censored could itself cause trouble if it were maliciously optimized. However, achieving that would require a malicious computation to sometimes decide *not* to get the surrounding computation censored, which shouldn’t be possible. That is, if a computation thinks to itself “should I do something malicious in order to cause the computation to be censored?” and concludes “no” in order to cause trouble, then that should already be grounds for censoring.

Informal analysis
=================

Assumption
----------

I’ll assume that using HCH to find a maximally useful argument *a*\* about Q, and then using HCH to answer Q given *a*\*, is [ascription universal](/towards-formalizing-universality-409ab893a456) — or rather would be universal if not for the possibility of malicious internal consequentialists.

Moreover, I’ll assume that it’s universal with some small blow-up, i.e. that if we give HCH a budget *k* N*,* then it epistemically dominates any computation that could be simulated by HCH with budget N.

Note that this is a very strong assumption. This verifier receives the argument *a*\* that they’d evaluate most highly, and so this will fail if there is *any* argument that would cause them to give a high score and behave badly. I’m trying to modify HCH to be robust to the possibility of internal consequentialists, but there are all sorts of other ways that a carefully chosen input could compromise a system that is “mostly” universal, which I’m assuming are already resolved.

Hope
----

Why might we expect this filtering procedure to fix problems caused by malicious consequentialists?

Then the rough hope is:

* If there is a consequentialist whose behavior is optimized to break HCH, then it “knows” something about why that behavior will cause trouble.
* So by ascription universality, filtered-HCH(Trouble(*a*\*, τ)) also knows why the behavior within τ will cause trouble.

This is only plausible once the original human H always behaves in the “intended” way and never performs malicious optimization, which I’d hope to achieve by [only showing them inputs from some small set](/universality-and-security-amplification-551b314a3bab). So any malicious consequentialist is implemented “on top of” these humans rather than within.

(Easy mode) Handling stochastic failures
----------------------------------------

Suppose that HCH(·, N) has internal malice with probability ε ≪ 1 on some inputs. This is a special case of the full problem (where errors occur with small probability) and the solution won’t be reused to solve the more general problem, so an impatient reader should skip ahead. I’m including it because I feel quite uncertain about the general solution, and I think this simpler solution already shows some of the upside of filtering and has a reasonable chance of being useful.

If we only want to cope with stochastic failures, the easiest approach is to use a slightly modified definition of Filter:

* Use HCH(·, *k*N) to evaluate Trouble(*a*\*, τ) and Info(*a*, *a*′, τ), rather than filtered-HCH(·, N).

We hope that HCH(·, *k*N) has a probability at most *k*εof encountering malicious internal computations. This seems intuitively plausible, since it is *k* times larger than HCH(·, N).

Internal malicious computations can affect Filter(τ, *k*N) in two ways:

* HCH(Trouble(*a*, τ), *k*N) may behave badly with probability *k*ε. But the only possible “bad” behavior is to give the wrong answer, either a false positive or false negative. If we take a best-of-3 vote, this probability is reduced to *k²*ε², so we have significantly reduced the failure probability as long as ε ≪ 1/*k*² (even if we consider a false positive, leading a subcomputation to be censored, to be as bad as an arbitrary failure).
* HCH(Info(*a*, *a*′, τ), *k*N) may behave badly, which affects which argument *a*\* will be optimal. However, we still get a distribution over computations which is “almost optimal” in the sense that it is preferred to every other argument with probability at least 1/2 − *k*ε (while the optimal argument would be preferred with probability at least 1/2). It seems reasonable to assume this small difference doesn’t affect the universality of HCH (and we can drive this probability exponentially close to 1/2 by ensembling).

This seems like a significant improvement over my previous proposal for [reliability amplification](/reliability-amplification-a96efa115687) because Filter(τ, N) returns yes or no, even if Q itself has complex answers. This lets us use a simple majority vote to improve reliability. In particular, we no longer require that no agent in the ensemble be too much smarter than the others — we have shifted a vague assumption about relative capabilities to a slightly more precise assumption about universality.

This mechanism only fixes errors that occur with sufficiently small probability. But we can apply it throughout a large tree, reducing the error probability frequently enough that it never gets large.

(Hard mode) Handling “bad” questions
------------------------------------

The problem is much harder when there are “bad” questions Q for which HCH has a significant probability of encountering malicious consequentialism and failing. If τ is the transcript from a question where HCH failed, Trouble(*a*\*, τ) may well be another question on which HCH predictably fails (e.g. because the same vulnerability is triggered by the attempt to diagnose the original failure). This could in turn prevent Filter() from censoring the original failure.

In this case we are going to argue inductively based on budgets: we can assume that filtered-HCH(·, N-1) probably has no malicious consequentialism by induction, and hence is universal.

Because we want to use an induction, it’s important that we answer Trouble(*a*\*, τ) and Info(*a*, *a*′, τ) using a *smaller* budget than was used in τ. But our assumption about universality requires the reasoner to be *k* times *bigger* than the computation it wants to compete with.

I hope that if there is malicious consequentialism within HCH(Q, N), then it needs to be “small” enough that we can rely on filtered-HCH(Trouble(*a*\*, τ), N-1) to compete with it. Intuitively:

* None of the unfiltered subquestions in HCH(Q, N) involve any malicious consequentialism, so we hope each subquestion either “behaves as intended” or produces the special symbol ⟂.
* If all subquestions of HCH(Q, N) are behaving as intended, and if the human at the root of HCH(Q, N) is behaving as intended (which we attempted to guarantee by limiting their input space), then any malicious consequentialism is very tightly constrained.
* Intuitively, it’s restricted to two options:
(a) be computed implicitly with “side effects” of the intended behavior, repurpose valid parts of the computation for unintended effects, *etc.*
(b) make use of the intended behavior of questions
* When using option (a), it feels like the malicious optimization will be significantly less powerful than HCH with the same budget. For example, when I’m answering a question, it seems like most of my useful computational work is going into actually answering the question; if someone wanted to try to do computational work based on details of how I formatted my answer, I think they’d get less done. So we can hope that this part of the malicious consequentialism could be simulated by HCH with total budget much less than N.
* When using option (b), the malicious consequentialism can be very efficient — exactly as efficient as HCH itself, since it’s just using that computation in the intended way. But filtered-HCH(Trouble(*a*\*, τ)) may not need to understand these parts of that computation in order to do its job. Instead it can look at the transcript τ and take the relevant answers at face value, just like the malicious computation is doing. (This relies on the inductive invariant, and even with that isn’t obvious, but it looks pretty plausible to me.)

Note that this strategy really leans on the assumption that HCH works well for *every* input. In addition to the fact that *a*\* is optimized, Filter calls HCH with a large and potentially malicious transcript τ as its input. The distinction between “untrusted input that you consult when appropriate” and “subcomputation that has a privileged position from which it can cause harm.”

Universality and consequentialism within HCH


[Ascription universality](/towards-formalizing-universality-409ab893a456) seems sufficient for [informed oversight](/informed-oversight-18fcb5d3d1e1), and I’ve argued it could be used to [prevent cascading failures in HCH](/universality-and-consequentialism-within-hch-c0bee00365bd). In this post I’ll argue that ascription universality also appears to address two problems with aligning model-based RL:

1. It allows you to perform induction without being hijacked by malicious predictors (this is the “[benign induction](https://agentfoundations.org/item?id=1263)” problem, which is extremely speculative)
2. It seems necessary, though maybe not sufficient, for extracting “all the relevant facts a model knows.” (Which is at least necessary, and I think probably sufficient, for defining a reward function that yields aligned behavior when optimized.)

These arguments aren’t rigorous enough that I find them convincing, but they are enough to convince me that better understanding universality is a top priority. At this point I wouldn’t be surprised if universality is enough to resolve most of the classical philosophical difficulties for AI safety. If so, first we’ll need to better understand in what sense it could be achievable.

Setup
-----

I’m interested in the following hopefully-aligned version of model-based RL:

* Use iterated amplification to learn a (distribution over) models of the world.
* Use iterated amplification to learn a utility function over sequences of states and actions.
* Plan in that model+utility function (e.g. using MCTS)
(You might want to combine learning a value function and terminal values, but for now I’ll simplify.)

This is similar to the traditional model-based RL approach, except that we learn a model and utility function with iterated amplification, instead of learning a model to predict future observations and defining a utility function over observations by hand.

What is a model?
----------------

The simplest kind of model to consider formally is a *predictor*, which we can view as a probability distribution over sequences of observations. We can evaluate predictors on past observations, and use them to predict future observations.

This is not enough for making the decisions, because the utility of an outcome is not a (simple) function of our future observations. We care about lots of features of the world that are hard to directly observe, and if we tried to define our preferences in terms of our observations alone we’d need an additional step where we map our observations to a prediction about what’s “really happening” out there in the world. For example, given a bunch of observations we might ask “given those observations, is Alice actually happy?” At some point we need to get a a distribution over models that we can use to answer questions about *the stuff we care about*, and not just our observations.

(Closely related: [Formalizing Two Problems of Realistic World Models](https://intelligence.org/files/RealisticWorldModels.pdf), though the basic philosophical issue is a classic.)

As a first step we can ask a model to answer arbitrary questions about the world rather than only reproducing observations. We can evaluate a model by looking at questions whose answers we already know. A problem with this is that in order to behave well a model only needs to correctly answer *the kinds of questions that we might know the answer to*, and the simplest model may well behave strangely on other questions.

A second step is to view a model M as an object that HCH can reason about, and use HCH to answer questions like “If model M describes reality, then what would be true?” For example, a model might posit some kinds of physical objects, and HCH could reason from those objects both to explain our observations and to infer the existence of other objects we care about. This seems to more closely track the reasoning humans perform about possible models, and (a) gives us more traction on a good prior over models, (b) gives us traction to pose other questions about the model (beyond what its predictions are).

I’ll use this notion of “model” throughout the post.

Aside: defining goals
---------------------

If we are able to learn a “correct” and “understandable” model, I’m optimistic about being able to find a utility function that can induce aligned behavior. In particular, I think that function can look something like:

* Identify a thing that is “me” in the resulting state
* Conservatively evaluate how many resources “I” effectively control
(working together with AI systems, *etc.*)
* Conservatively evaluate how “I” changed over the history — do I endorse the way in which my beliefs and values shifted?

We can’t hope to have completely robust answers to any of these questions. But we can at least hope to have a utility function that is safe to maximize within our model, by being aware of the history leading to high utility and penalizing the act if that history leads to high utility for bad reasons.

For example, if we can see that we control resources only because other agents disagree with our assessment of which resources are important, then we can revise our estimates appropriately — we don’t have to fix a single way of valuing resources which is robust to arbitrary adversarial pressure.

(Realizing this kind of hope is related to the ideas expressed by Jessica Taylor [here](https://agentfoundations.org/item?id=1149) and [here](https://agentfoundations.org/item?id=1224).)

As another example, our preferences over changes to ourselves will not be completely robust. But maximization over the utility function will only exert pressure for such changes if they somehow lead to *higher* reward. And in order to do that, the changed behavior needs to actually manifest over the history, and we can use that manifestation to detect a corruption that allows the agent to receive higher reward even if we wouldn’t recognize most possible corruptions.

“Knowing what the model knows” seems to be a critical ingredient for constructing such a utility function. It’s not clear whether or not it’s sufficient, but I won’t discuss this issue more in this post.

Problem 1: Benign induction
===========================

If we take the “simplest” model that explains our observations, it could potentially be [very](https://ordinaryideas.wordpress.com/2016/11/30/what-does-the-universal-prior-actually-look-like/) [strange](https://agentfoundations.org/item?id=1263). In particular, it could produce correct observations because it contains an intelligent predictor rather than because it captures “real” physics. This could cause it to generalize in unpredictable and dangerous ways (depending on the motivations of that agent).

Two variants of this problem have been discussed:

1. The predictor may literally *contain* intelligent agents who are trying to predict our observations. These agents may be motivated by the desire to survive the training process, but then may generalize maliciously once off the training distribution.
2. The predictor could *reason* *about* intelligent agents who change their behavior in order to explain our behavior, because the existence of such agents is the “simplest” explanation for our observations. A notion of simplicity that produces this behavior is in some sense not the “right” one, but I believe it would probably apply to the simple formalisms we currently understand (such as minimum description length).

(A single situation may have some features of both #1 and #2, this is not a clean dichotomy.)

If we have a universal reasoning procedure, I think we can likely avoid these problems.

Option 1: filter malign models
------------------------------

We use the following procedure:

* Compute log likelihoods for models as normal, by asking HCH(“how likely are facts F if model M is true?”)
* Additionally, compute HCH(“is model M likely to lead to catastrophically bad predictions on inputs we’ll encounter in the future?”) and judge a model “unsafe” is the answer is “yes.”
* Apply Bayes as usual but restricted to the safe models.

Why might this work?

If a model M is malign, then it is making good predictions *because* of the existence of some adversarial agent. Then:

* If that agent is actually contained in the model (situation #1), then universality means we can learn everything that agent knows. But the agent has formed the plan of giving good answers in order to increase the model’s log likelihood and so have more influence later, and if we know this then we will realize the model will generalize poorly and filter it out.
* If the agent is being reasoned about by the model (situation #2) then things are even easier, since the model knows about the existence of the agent and its motivation. Then applying universality to the model itself will let us know the model would generalize poorly.

Going beyond the simplest safety test, we could also try to throw out any “abnormal” models, e.g. where the stars don’t actually exist or our local universe is actually a fiction. This could leave value on the table if these weird hypotheses are important, but it could also reduce the risk of making a serious error based on strange hypotheses. Universality is relevant because it potentially allows us to avoid adversarial pressure on this definition of “normal” — if a model looks normal because an adversary is trying to make it look normal, then we can recognize that and throw it out as unsafe.

Option 2: outcompete malign models
----------------------------------

Suppose that I have a prior μ over models, and I believe that the posterior puts significant probability on “malicious” models that make good predictions only because they contain agents who are motivated to make good predictions. (Or because they reason about such agents.)

This is a strange epistemic situation to be in. I must believe that on average these malicious agents have a “better” prior μ than I do — that’s what they need in order to assign a higher probability to the data than the rest of μ. Jessica Taylor writes about this situation [here](https://www.lesswrong.com/posts/5bd75cc58225bf067037535e/are-daemons-a-problem-for-ideal-agents), and has similar intuitions about it.

Intuitively, something like universality should allow you to avoid this state. That is, the malicious models must perform some reasoning that we believe allows them to assign better prior probabilities; by ascription universality HCH must be able to capture this reasoning and use it to inform *its* prior. As long as malicious agents have a reasonably low prior they will therefore end up with a reasonably low posterior.

To implement this strategy, we could use the same procedure as in the last section, but instead of using a simplicity prior over models M, we would use HCH to compute those prior probabilities by asking HCH(“a priori how likely is model M to describe the observations we are going to provide to it?”) or something along those lines.

I don’t know whether this works, and I’m not sure it will really be possible to say much precisely without having a cleaner picture of the sense in which HCH is or isn’t universal. If it does work, this seems like a much cleaner solution than filtering. The fact that there are two plausible options on the table makes me more optimistic that universality will address benign induction one way or another.

Problem 2: extracting facts from a model
========================================

Consider a model M that specifies low-level physical principles and some basic “bridging” facts about how everyday objects are implemented in low-level physics. HCH can answer questions about the world by performing a detailed physical calculation, and then using bridging facts to relate the resulting state to things we care about.

(Modeling physics itself is a somewhat exotic example, but I think the same phenomenon would occur in more prosaic cases, wherever there are effective but opaque models.)

Unfortunately, M can be a good model even if the bridging facts are not exhaustive. They might allow HCH to answer some kinds of questions — e.g. to predict the observations that we’re using to test models — but not to answer other important questions.

For example, it may be the case that M predicts that Alice will report high satisfaction with the AI’s behavior, by performing some physics calculation and then using bridging facts that relate physical states to Alice’s reports. At the same time, the physics calculation might actually simulate Alice being bribed to report high satisfaction, but it might not be possible to infer that from the available bridging facts.

But the physics computation contains a complete simulation of Alice being bribed, and so we can ascribe knowledge of the bribery to it. Thus if HCH is ascription universal, we can hope it recovers all of these facts.

It’s not obvious whether universality is actually sufficient for this purpose, and it will depend on exactly what form of universality ends up being achievable. It might be that a “reasonable” ascription strategy would recognize that the model knows facts about atoms, but not facts about Alice.

At a minimum universality is *necessary* for this purpose — if M can believe facts that HCH can’t identify, then those facts could turn out to be relevant to the utility evaluation. It’s just that there might or might not be a further step not captured by universality.

My current guess is that the most natural form of universality will give us everything we want, and that thinking about “ontology identification” is mostly useful for finding hard cases for universality. Regardless of whether that’s true, I think the next step should be a clearer picture of prospects for universality, after which we can revisit this question.

Universality and model-based RL