Factored Cognition sequence

The <@Factored Cognition Hypothesis@>(@Factored Cognition@) informally states that any task can be performed by recursively decomposing the task into smaller and smaller subtasks until eventually the smallest tasks can be done by a human. This sequence aims to formalize the hypothesis to the point that it can be used to argue for the outer alignment of (idealized versions of) <@iterated amplification@>(@Supervising strong learners by amplifying weak experts@) and <@debate@>(@AI safety via debate@).

The key concept is that of an _explanation_ or _decomposition_. An explanation for some statement **s** is a list of other statements **s1, s2, … sn** along with the statement “(**s1** and **s2** and … and **sn**) implies **s**”. A _debate tree_ is a tree in which for a given node **n** with statement **s**, the children of **n** form an explanation (decomposition) of **s**. The leaves of the tree should be statements that the human can verify. (Note that the full formalism has significantly more detail, e.g. a concept of the “difficulty” for the human to verify any given statement.)

We can then define an idealized version of debate, in which the first debater must produce an answer with associated explanation, and the second debater can choose any particular statement to expand further. The judge decides the winner based on whether they can confidently verify the final statement or not. Assuming optimal play, the correct (honest) answer is an equilibrium as long as:

**Ideal Debate Factored Cognition Hypothesis:** For every question, there exists a debate tree for the correct answer where every leaf can be verified by the judge.

The idealized form of iterated amplification is <@HCH@>(@Humans Consulting HCH@); the corresponding Factored Cognition Hypothesis is simply “For every question, HCH returns the correct answer”. Note that the _existence_ of a debate tree is not enough to guarantee this, as HCH must also _find_ the decompositions in this debate tree. If we imagine that HCH gets access to a decomposition oracle that tells it the right decomposition to make at each node, then HCH would be similar to idealized debate. (HCH could of course simply try all possible decompositions, but we are ignoring that possibility: the decompositions that we rely on should reduce or hide complexity.)

Is the HCH version of the Factored Cognition Hypothesis true? The author tends to lean against (more specifically, that HCH would not be superintelligent), because it seems hard for HCH to find good decompositions. In particular, humans seem to improve their decompositions over time as they learn more, and also seem to improve the concepts by which they think over time, all of which are challenging for HCH to do. On the other hand, the author is cautiously optimistic about debate.

I enjoyed this sequence: I’m glad to see more analysis of what is and isn’t necessary for iterated amplification and debate to work, as well as more theoretical models of debate. I broadly agreed with the conceptual points made, with one exception: I’m not convinced that we should not allow brute force for HCH, and for similar reasons I don’t find the arguments that HCH won’t be superintelligent convincing. In particular, the hope with iterated amplification is to approximate a truly massive tree of humans, perhaps a tree containing around 2^100 (about 1e30) base agents / humans. At that scale (or even at just a measly billion (1e9) humans), I don’t expect the reasoning to look anything like what an individual human does, and approaches that are more like “brute force” seem a lot more feasible.

One might wonder why I think it is possible to approximate a tree with more base agents than there are grains of sand in the Sahara desert. Well, a perfect binary tree of depth 99 would have 1e30 nodes; thus we can roughly say that we’re approximating 99-depth-limited HCH. If we had perfect distillation, this would take 99 rounds of iterated amplification and distillation, which seems quite reasonable. Of course, we don’t have perfect distillation, but I expect that to be a relatively small constant factor on top (say 100x), which still seems pretty reasonable. (There’s more detail about how we get this implicit exponential-time computation in <@this post@>(@Factored Cognition@).)


AI Safety Debate and Its Applications

This post defines the components of a <@debate@>(@AI safety via debate@) game, lists some of its applications, and defines truth-seeking as the property that we want. Assuming that the agent chooses randomly from the possible Nash equilibria, the truth-promoting likelihood is the probability that the agent picks the actually correct answer. The post then shows the results of experiments on MNIST and Fashion MNIST, seeing comparable results to the original paper.


Capability amplification

Capability amplification is the problem of taking some existing policy and producing a better policy, perhaps using much more time and compute. It is a particularly interesting problem to study because it could be used to define the goals of a powerful AI system, and it could be combined with [reward engineering](https://www.alignmentforum.org/posts/4nZRzoGTqg8xy5rr8/the-reward-engineering-problem) above to create a powerful aligned system. (Capability amplification and reward engineering are analogous to amplification and distillation respectively.) In addition, capability amplification seems simpler than the general problem of "build an AI that does the right thing", because we get to start with a weak policy A rather than nothing, and were allowed to take lots of time and computation to implement the better policy. It would be useful to tell whether the "hard part" of value alignment is in capability amplification, or somewhere else.

We can evaluate capability amplification using the concepts of reachability and obstructions. A policy C is _reachable_ from another policy A if there is some chain of policies from A to C, such that at each step capability amplification takes you from the first policy to something at least as good as the second policy. Ideally, all policies would be reachable from some very simple policy. This is impossible if there exists an _obstruction_, that is a partition of policies into two sets L and H, such that it is impossible to amplify any policy in L to get a policy that is at least as good as some policy in H. Intuitively, an obstruction prevents us from getting to arbitrarily good behavior, and means that all of the policies in H are not reachable from any policy in L.

We can do further work on capability amplification. With theory, we can search for challenging obstructions, and design procedures that overcome them. With experiment, we can study capability amplification with humans (something which [Ought](https://ought.org/) is now doing).

There's a clear reason for work on capability amplification: it could be used as a part of an implementation of iterated amplification. However, this post also suggests another reason for such work -- it may help us determine where the "hard part" of AI safety lies. Does it help to assume that you have lots of time and compute, and that you have access to a weaker policy?

Certainly if you just have access to a weaker policy, this doesn't make the problem any easier. If you could take a weak policy and amplify it into a stronger policy efficiently, then you could just repeatedly apply this policy-improvement operator to some very weak base policy (say, a neural net with random weights) to solve the full problem. (In other variants, you have a much stronger aligned base policy, eg. the human policy with short inputs and over a short time horizon; in that case this assumption is more powerful.) The more interesting assumption is that you have lots of time and compute, which does seem to have a lot of potential. I feel pretty optimistic that a human thinking for a long time could reach "superhuman performance" by our current standards; capability amplification asks if we can do this in a particular structured way.


Benign model-free RL

This post is very similar to the previous one, just with different language: distillation is now implemented through reward modeling with robustness. The point of robustness is to ensure that the distilled agent is benign even outside of the training distribution (though it can be incompetent). There's also an analysis of the costs of the scheme. One important note is that this approach only works for model-free RL systems -- we'll need something else for eg. model-based RL, if it enables capabilities that we can't get with model-free RL.


Iterated Distillation and Amplification

This is the first in a series of four posts describing the iterated amplification framework in different ways. This post focuses on the repetition of two steps. In amplification, we take a fast aligned agent and turn it into a slow but more capable aligned agent, by allowing a human to coordinate many copies of the fast agent in order to make better decisions. In distillation, we take a slow aligned agent and turn it a fast aligned agent (perhaps by training a neural net to imitate the judgments of the slow agent). This is similar to AlphaGoZero, in which MCTS can be thought of as amplification, while distillation consists of updating the neural net to predict the outputs of the MCTS.

This allows us to get both alignment and powerful capabilities, whereas usually the two trade off against each other. High capabilities implies a sufficiently broad mandate to search for good behaviors, allowing our AI systems to find novel behaviors that we never would have thought of, which could be bad if the objective was slightly wrong. On the other hand, high alignment typically requires staying within the realm of human behavior, as in imitation learning, which prevents the AI from finding novel solutions.

In addition to distillation and amplification robustly preserving alignment, we also need to ensure that given a human as a starting point, iterated distillation and amplification can scale to arbitrary capabilities. We would also want it be about as cost-efficient as alternatives. This seems to be true at test time, when we are simply executing a learned model, but it could be that training is much more expensive.

This is a great simple explanation of the scheme. I don't have much to say about the idea since I've talked about iterated amplification so much in this newsletter already.


Learning to Interactively Learn and Assist

Mark Woodward
Chelsea Finn
Karol Hausman

[Cooperative Inverse Reinforcement Learning](https://arxiv.org/abs/1606.03137) proposed a model in which an AI assistant would help a human principal, where only the principal knows the task reward. This paper explores this idea in the context of deep reinforcement learning. In their grid-world environment, two agents move around and pick up lemons or plums. The principal is penalized for moving, but is the only one who knows whether plums or lemons should be picked up. The authors hypothesize that simply by jointly training the two agents to maximize rewards, they will automatically learn to interact in order for the assistant to learn the task, rather than requiring an explicit mechanism like comparisons or demonstrations.

Recurrent Q-networks are used for the agents, which are then trained via deep Q-learning. The authors run several experiments that show emergent interaction. In the first experiment, when the principal is penalized for moving it learns to demonstrate the task to the assistant, and then let the assistant finish the job. In the second experiment, when the assistant has a restricted field of view, it learns to follow the principal until it can infer whether the principal wants plums or lemons. In the third, they tell the assistant the task 50% of the time, and so the principal is initially unsure whether the agent needs any direction (and due to the motion cost, the principal would rather not do anything). When the agent knows the task, it performs it. When the agent doesn't know the task, it moves closer to the principal, in effect "asking" what the reward is, and the principal moves until it can see the object, and then "answers" by either moving towards the object (if it should be collected) or doing nothing (if not). Finally, the authors run an experiment using pixels as input. While they had to switch to dueling DQNs instead of vanilla DQNs, they show that the joint reward is competitive with the grid approach. They also run an experiment with human principals and show that the human/assistant pair outperforms the solo-human setup.

Overall, I found the idea expressed in this paper to be well-articulated. While I think that the grid-world environment is a bit simplistic, their results are interesting. Being able to learn intent in an online manner is an important problem to solve if we’re interested in robust collaboration between humans and autonomous agents. However, the authors point out that training on pixel input fails in the majority of cases, 64% of the time, which raises concerns about how well the method would generalize to non-trivial environments.

**Rohin's opinion:** I'm excited that the ideas from [CIRL](https://arxiv.org/abs/1606.03137) are making their way to deep RL. Ultimately I expect we'll want an agent that takes all of its sensory data as evidence about "what the human wants", rather than relying on a special reward channel, or a special type of data called "comparisons" or "demonstrations", and this work takes that sort of approach.

For these simple environments, an agent trained to perform well with another artificial agent will generalize reasonably well to real humans, because there's only a few reasonable strategies for the principal to take. However, with more complex environments, when there are many ways to interact, we can't expect such generalization. (I'll have a paper and blog post coming out soon about this phenomenon.)


Learning Preferences by Looking at the World

The key idea with this project of mine is that the state of the world is already optimized for our preferences, and so simply by looking at the world we can infer these preferences. Consider the case where there is a vase standing upright on the table. This is an unstable equilibrium -- it's very easy to knock over the vase so it is lying sideways, or is completely broken. The fact that this hasn't happened yet suggests that we care about vases being upright and intact; otherwise at some point we probably would have let it fall.

Since we have optimized the world for our preferences, the natural approach is to model this process, and then invert it to get the preferences. You could imagine that we could consider all possible reward functions, and put probability mass on them in proportion to how likely they make the current world state if a human optimized them. Basically, we are simulating the past in order to figure out what must have happened and why. With the vase example, we would notice that in any reward function where humans wanted to break vases, or were indifferent to broken vases, we would expect the current state to contain broken vases. Since we don't observe that, it must be the case that we care about keeping vases intact.

Our algorithm, Reward Learning by Simulating the Past (RLSP), takes this intuition and applies it in the framework of [Maximum Causal Entropy IRL](http://www.cs.cmu.edu/~bziebart/publications/maximum-causal-entropy.pdf) ([AN #12](https://mailchi.mp/bcb2c6f1d507/alignment-newsletter-12)), where you assume that the human was acting over T timesteps to produce the state that you observe. We then show a few gridworld environments in which applying RLSP can fix a misspecified reward function.

In addition to this blog post and the [paper](https://openreview.net/forum?id=rkevMnRqYQ), I also wrote a [post](https://www.alignmentforum.org/posts/7f6DNZhracD7RvxMr/learning-preferences-by-looking-at-the-world) on the Alignment Forum expressing opinions about the work. There are too many disparate opinions to put in here, so I'd recommend reading the post itself. I guess one thing I'll mention is that to infer preferences with a single state, you definitely need a good dynamics model, and a good set of features. While this may seem difficult to get, it's worth noting that dynamics are empirical facts about the world, and features might be, and there is already lots of work on learning both dynamics and features.

Paper: Preferences Implicit in the State of the World


AI Alignment Podcast: Cooperative Inverse Reinforcement Learning

Dylan puts forward his conception of Cooperative Inverse Reinforcement Learning as a definition of what it means for a human-AI system to be rational, given the information bottleneck between a human's preferences and an AI's observations. He notes that there are some clear mismatches between this problem and reality, such as the CIRL assumption that humans have static preferences, and how fuzzy the abstraction of "rational agents with utility functions" becomes in the context of agents with bounded rationality. Nevertheless, he claims that this is a useful unifying framework for thinking about AI safety.

Dylan argues that the process by which a robot learns to accomplish tasks is best described not just as maximising an objective function but instead in a way which includes the system designer who selects and modifies the optimisation algorithms, hyperparameters, etc. In fact, he claims, it doesn't make sense to talk about how well a system is doing without talking about the way in which it was instructed and the type of information it got. In CIRL, this is modeled via the combination of a "teaching strategy" and a "learning strategy". The former can take many forms: providing rankings of options, or demonstrations, or binary comparisons, etc. Dylan also mentions an extension of this in which the teacher needs to learn their own values over time. This is useful for us because we don't yet understand the normative processes by which human societies come to moral judgements, or how to integrate machines into that process.


Scalable agent alignment via reward modeling

This blog post and the [associated paper](https://arxiv.org/abs/1811.07871) outline a research direction that DeepMind's AGI safety team is pursuing. The key idea is to learn behavior by learning a reward and a policy simultaneously, from human evaluations of outcomes, which can scale to superhuman performance in tasks where evaluation is easier than demonstration. However, in many cases it is hard for humans to evaluate outcomes: in this case, we can train simpler agents using reward modeling that can assist the human in evaluating outcomes for the harder task, a technique the authors call recursive reward modeling. For example, if you want to train an agent to write a fantasy novel, it would be quite expensive to have a human evaluate outcomes, i.e. rate how good the produced fantasy novels are. We could instead use reward modeling to train agents that can produce plot summaries, assess prose quality and character development, etc. which allows a human to assess the fantasy novels. There are several research challenges, such as what kind of feedback to get, making it sufficiently sample efficient, preventing reward hacking and unacceptable outcomes, and closing the reward-result gap. They outline several promising approaches to solving these problems.

The proposal sounds to me like a specific flavor of narrow value learning, where you learn reward functions to accomplish particular tasks, rather than trying to figure out the "true human utility function". The recursive aspect is similar to [iterated amplification](https://blog.openai.com/amplifying-ai-training/) and [debate](https://blog.openai.com/debate/). Iterated amplification and debate can be thought of as operating on a tree of arguments, where each node is the result of considering many child nodes (the considerations that go into the argument). Importantly, the child nodes are themselves arguments that can be decomposed into smaller considerations. Iterated amplification works by learning how to compose and decompose nodes from children, while debate works by having humans evaluate a particular path in the argument tree. Recursive reward modeling instead uses reward modeling to train agents that can help _evaluate outcomes_ on the task of interest. This seems less recursive to me, since the subagents are used to evaluate outcomes, which would typically be a different-in-kind task than the task of interest. This also still requires the tasks to be fast -- it is not clear how to use recursive reward modeling to eg. train an agent that can teach math to children, since it takes days or months of real time to even produce outcomes to evaluate. These considerations make me a bit less optimistic about recursive reward modeling, but I look forward to seeing future work that proves me wrong.

The post also talks about how reward modeling allows us to separate what to do (reward) from how to do it (policy). I think it is an open question whether this is desirable. [Past work](https://arxiv.org/abs/1806.01946) found that the reward generalized somewhat (whereas policies typically don't generalize at all), but this seems relatively minor. For example, rewards inferred using deep variants of inverse reinforcement learning often don't generalize. Another possibility is that the particular structure of "policy that optimizes a reward" provides a useful inductive bias that makes things easier to learn. It would probably also be easier to inspect a specification of "what to do" than to inspect learned behavior. However, these advantages are fairly speculative and it remains to be seen whether they pan out. There are also practical advantages: any advances in deep RL can immediately be leveraged, and reward functions can often be learned much more sample efficiently than behavior, reducing requirements on human labor. On the other hand, this design "locks in" that the specification of behavior must be a reward function. I'm not a fan of reward functions because they're so unintuitive for humans to work with -- if we could have agents that work with natural language, I suspect I do not want the natural language to be translated into a reward function that is then optimized.


Deep Imitative Models for Flexible Inference, Planning, and Control

Nicholas Rhinehart
Rowan McAllister
Sergey Levine

It's hard to apply deep RL techniques to autonomous driving, because we can't simply collect a large amount of experience with collisions in order to learn. However, imitation learning is also hard, because as soon as your car deviates from the expert trajectories that you are imitating, you are out of distribution, and you could make more mistakes, leading to accumulating errors until you crash. Instead, we can model the expert's behavior, so that we can tell when we are moving out of distribution, and take corrective action.

They split up the problem into three different stages. First, they generate a set of _waypoints_ along the path to be followed, which are about 20m away from each other, by using A* search on a map. Next, they use model-based planning using an imitative model to generate a plan (sequence of states) that would take the car to the next waypoint. Finally, they use a simple PID controller to choose low-level actions that keep the car on target towards the next state in the plan.

The key technical contribution is with the imitative model, which is a probabilistic model P(s_{1:T}, G, φ), where φ is the current observation (eg. LIDAR), s_{1:T} is the planned trajectory, and G is a goal. We can learn P(s_{1:T} | φ) from expert demonstrations. The goal G can be anything for which you can write down a specification P(G | s_{1:T}, φ). For example, if you simply want to reach a waypoint, you can use the normal distribution on the distance between the final state s_T and the waypoint. You can also incorporate a hand-designed cost on each state.

They evaluate in simulation on a static world (so no pedestrians, for example). They show decent transfer from one map to a second map, and also that they can avoid artificially introduced potholes at test time (despite not seeing them at training time), simply by adding a cost on states over a pothole (which they can take into account because they are performing model-based planning).

I really like this paper, it showcases the benefits of both model-based planning and imitation learning. Since the problem has been decomposed into a predictive model, a goal G, and a planner, we can edit G directly to get new behavior at test time without any retraining (as they demonstrate with the pothole experiment). At the same time, they can get away with not specifying a full reward function, as many features of good driving, like passenger comfort and staying in the correct lane, are learned simply by imitating an expert.

That said, they initially state that one of their goals is to learn from offline data, even though offline data typically has no examples of crashes, and "A model ignorant to the possibility of a crash cannot know how to prevent it". I think the idea is that you never get into a situation where you could get in a crash, because you never deviate from expert behavior since that would have low P(s_{1:T} | φ). This is better than model-based planning on offline data, which would consider actions that lead to a crash and have no idea what would happen, outputting garbage. However, it still seems that situations could arise where a crash is imminent, which don't arise much (if at all) in the training data, and the car fails to swerve or brake hard, because it hasn't seen enough data.


Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos

This work demonstrates a method that learns a _generalizable multi-task reward function_ in the context of robotic manipulation; at deployment, this function can be conditioned on a human demonstration of an unseen task to generate reward signals for the robot, even in new environments.

A key insight here was to train a discriminative model that learned whether two given video clips were performing the same actions. These clips came from both a (large) dataset of human demonstrations and a relatively smaller set of robot expert trajectories, and each clip was labelled with a task-id. This training pipeline thus leveraged huge quantities of extant human behaviour from a diversity of viewpoints to learn a metric of 'functional similarity' between pairs of videos, independent of whether they were executed by human or machine.

Once trained, this model (called the 'Domain-agnostic Video Discriminator' or DVD) can determine if a candidate robotic behaviour is similar to a desired human-demonstrated action. Such candidates are drawn from an action-conditioned video predictor, and the best-scoring action sequence is selected for execution on the (simulated or real) robot.

Performance increased with the inclusion of human data, even that from unrelated tasks, so one intuition I updated on was "More data is better, even if it's not perfect". This also feels related to "Data as regularization": to some extent, noisy data combats model overconfidence, and perhaps this would play an important role in aligning future systems.

Another thing I like about such pipeline papers is the opportunity to look for where systems might break. For example, in this work, the robot does actually need (prior) experience in the test environments with which to train the video predictor to be able to generate candidate solutions at test time. So in spite of the given result -- that DVD itself needs limited robot trajectories and no data from the test environments -- there's a potential point-of-failure far sooner in the pipeline, where if the robot did not have sufficient _background_ experience with diverse situations, it might not provide _any_ feasible candidate actions for DVD's evaluation.


BASALT: A Benchmark for Learning from Human Feedback

Rohin Shah
Cody Wild
Steven H. Wang
Neel Alex
Brandon Houghton
William Guss
Sharada Mohanty
Anssi Kanervisto
Stephanie Milani
Nicholay Topin
Pieter Abbeel
Stuart Russell
Anca Dragan

A typical argument for AI risk, given in <@Human Compatible@>(@Human Compatible: Artificial Intelligence and the Problem of Control@), is that current AI systems treat their specifications as definite and certain, even though they are typically misspecified. This state of affairs can lead to the agent pursuing <@instrumental subgoals@>(@The Basic AI Drives@). To solve this, we might instead build AI systems that continually learn the objective from human feedback. This post and paper (on which I am an author) present the MineRL BASALT competition, which aims to promote research on algorithms that learn from human feedback.

BASALT aims to provide a benchmark with tasks that are realistic in the sense that (a) it is challenging to write a reward function for them and (b) there are many other potential goals that the AI system “could have” pursued in the environment. Criterion (a) implies that we can’t have automated evaluation of agents (otherwise that could be turned into a reward function) and so suggests that we use human evaluation of agents as our ground truth. Criterion (b) suggests choosing a very “open world” environment; the authors chose Minecraft for this purpose. They provide task descriptions such as “create a waterfall and take a scenic picture of it”; it is then up to researchers to create agents that solve this task using any method they want. Human evaluators then compare two agents against each other and determine which is better. Agents are then given a score using the [TrueSkill system](https://en.wikipedia.org/wiki/TrueSkill).

The authors provide a number of reasons to prefer the BASALT benchmark over more traditional benchmarks like Atari or MuJoCo:

1. In Atari or MuJoCo, there are often only a few reasonable goals: for example, in Pong, you either hit the ball back, or you die. If you’re testing algorithms that are meant to learn what the goal is, you want an environment where there could be many possible goals, as is the case in Minecraft.
2. There’s lots of Minecraft videos on YouTube, so you could test a “GPT-3 for Minecraft” approach.
3. The “true reward function” in Atari or MuJoCo is often not a great evaluation: for example, a Hopper policy trained to stand still using a constant reward gets 1000 reward! Human evaluations should not be subject to the same problem.
4. Since the tasks were chosen to be inherently fuzzy and challenging to formalize, researchers are allowed to take whatever approach they want to solving the task, including “try to write down a reward function”. In contrast, for something like Atari or MuJoCo, you need to ban such strategies. The only restriction is that researchers cannot extract additional state information from the Minecraft simulator.
5. Just as we’ve <@overestimated few-shot learning capabilities@>(@True Few-Shot Learning with Language Models@) by tuning prompts on large datasets of examples, we might also be overestimating the performance of algorithms that learn from human feedback because we tune hyperparameters on the true reward function. Since BASALT doesn’t have a true reward function, this is much harder to do.
6. Since Minecraft is so popular, it is easy to hire Minecraft experts, allowing us to design algorithms that rely on expert time instead of just end user time.
7. Unlike Atari or MuJoCo, BASALT has a clear path to scaling up: the tasks can be made more and more challenging. In the long run, we could aim to deploy agents on public multiplayer Minecraft servers that follow instructions or assist with whatever large-scale project players are working on, all while adhering to the norms and customs of that server.

You won’t be surprised to hear that I’m excited about this benchmark, given that I worked on it. While we listed a bunch of concrete advantages in the post above, I think many (though not all) of the advantages come from the fact that we are trying to mimic the situation we face in the real world as closely as possible, so there’s less opportunity for Goodhart’s Law to strike. For example, later in this newsletter we’ll see that synthetically generated demos are not a good proxy for human demos. Even though this is the norm for existing benchmarks, and we didn’t intentionally try to avoid this problem, BASALT (mostly) avoids it. With BASALT you would have to go pretty far out of your way to get synthetically generated demos, because by design the tasks are hard to complete synthetically, and so you _have_ to use human demos.

I’d encourage readers to [participate in the competition](https://www.aicrowd.com/challenges/neurips-2021-minerl-basalt-competition), because I think it’s especially good as a way to get started with ML research. It’s a new benchmark, so there’s a lot of low-hanging fruit in applying existing ideas to the benchmark, and in identifying new problems not present in previous benchmarks and designing solutions to them. It’s also pretty easy to get started: the BC baseline is fairly straightforward and takes a couple of hours to be trained on a single GPU. (That’s partly because BC doesn’t require environment samples; something like <@GAIL@>(@Generative Adversarial Imitation Learning@) would probably take a day or two to train instead.)

Paper: The MineRL BASALT Competition on Learning from Human Feedback


One-Shot Imitation from Watching Videos

Can we get a robot to learn a task by watching a human do it? This is very different from standard imitation learning. First, we want to do it with a single demonstration, and second, we want to do it by _watching a human_ -- that is, we're learning from a video of a human, not a trajectory where the robot actions are given to us. Well, first consider how we could do this if we have demonstrations from a teleoperated robot. In this case, we do actually have demonstrations in the form of trajectories, so normal imitation learning techniques (behavioral cloning in this case) work fine. We can then take this loss function and use it with [MAML](http://bair.berkeley.edu/blog/2017/07/18/learning-to-learn/) to learn from a large dataset of tasks and demonstrations how to perform a new task given a single demonstration. But this still requires the demonstration to be collected by teleoperating the robot. What if we want to learn from a video of a human demonstrating? They propose learning a _loss function_ that given the human video provides a loss from which gradients can be calculated to update the policy. Note that at training time there are still teleoperation demonstrations, so the hard task of learning how to perform tasks is done then. At test time, the loss function inferred from the human video is primarily used to identify which objects to manipulate.

This is cool, it actually works on a real robot, and it deals with the issue that a human and a robot have different action spaces.

Some form of meta-learning (ideally [MAML](http://bair.berkeley.edu/blog/2017/07/18/learning-to-learn/)).


Grounding Language in Play

This paper presents a new approach to learning to follow natural language human instruction in a robotics setting. It builds on similar ideas to <@Learning Latent Plans from Play@>, in that it uses unsupervised "play" data (trajectories of humans playing on the robot with no goal in mind).

The paper combines several ideas to enable training a policy which can follow natural language instructions with only limited human annotations.
* In *Hindsight Instruction Pairing*, human annotators watch small trajectories from the play data, and label them with the instruction which is being completed in the clip. This instruction can take any form, and means we don't need to choose the instructions and ask humans to perform specific tasks.
* *Multicontext Imitation Learning* is a method designed to allow goal-conditioned policies to be learned with multiple different types of goals. For example, we can have lots of example trajectories where the goal is an end state image (as these can be generated automatically without humans), and just a small amount of example trajectories where the goal is a natural language instruction (gathered using *Hindsight Instruction Pairing*). The approach is to learn a goal embedding network for each type of goal specification, and a single shared policy which takes the goal embedding as input.

Combining these two methods enables them to train a policy and embedding networks end to end using imitation learning from a large dataset of (trajectory, image goal) pairs and a small dataset of (trajectory, natural language goal) pairs. The policy can follow very long sequences of natural language instructions in a fairly complex grasping environment with a variety of buttons and objects. Their method performs better than the Learning from Play (LfP) method, even though LfP uses a goal image as the goal conditioning, instead of a natural language instruction.

Further, they propose that instead of learning the goal embedding for the natural language instructions, they use a pretrained large language model to produce the embeddings. This improves the performance of their method over learning the embedding from scratch, which the authors claim is the first example of the knowledge in large language models being transferred and improving performance in a robotics domain. This model also performs well when they create purposefully out of distribution natural language instructions (i.e. with weird synonyms, or google-translated from a different language).

I think this paper shows two important things:

1. Embedding the natural language instructions in the same space as the image conditioning works well, and is a good way of extending the usefulness of human annotations.

2. Large pretrained language models can be used to improve the performance of language-conditioned reinforcement learning (in this case imitation learning) algorithms and policies.

Methods which enable us to scale human feedback to complex settings are useful, and this method seems like it could scale well, especially with the use of pretrained large language models which might reduce the amount of language annotations needed further.


Pitfalls of learning a reward function online

Stuart Armstrong
Jan Leike
Laurent Orseau
Shane Legg

It can be dangerous to learn the metric that you are trying to optimize: if you don't set it up correctly, you may end up incentivizing the agent to "update in a particular direction" in the metric learning for the sake of future optimization (a point previously made in [Towards Interactive Inverse Reinforcement Learning](https://jan.leike.name/publications/Towards%20Interactive%20Inverse%20Reinforcement%20Learning%20-%20Armstrong,%20Leike%202016.pdf)). This paper analyzes the problems that can arise when an agent simultaneously learns a reward function, and optimizes that reward function.

The agent may have an incentive to "rig" the reward learning process, such that it finds a reward that is easy to optimize. For example, consider a student Sandra who must figure out the deadline and evaluation criteria for a project from a teacher Trisha. Sandra expects that if she asks Trisha when the deadline is, she will say that the deadline is later this week. So, Sandra might cleverly ask, "Is the project due next week, or the week after", to which Trisha might respond "next week". In this way, Sandra can rig the deadline-learning process in order to obtain a more favorable deadline.

Worse, in such scenarios the need to rig the learning process can destroy value for _every_ reward function you are considering. For example, let's suppose that if Trisha couldn't be manipulated, Sandra's optimal policy would be to start the project today, _regardless_ of when the actual deadline is. However, given that Trisha _can_ be manipulated, Sandra will spend today manipulating Trisha into setting a later deadline -- an action that seems clearly suboptimal from the perspective of any fixed deadline. The paper describes this as _sacrificing reward with certainty_.

To avoid such situations, we need _unriggable_ learning processes, that is, ones where at all times, the expected final learned reward (deadline) is independent of the agent's (Sandra's) policy. This unriggability property is nearly equivalent to the property of _uninfluencability_, in which we must be able to posit some background variables in the environment such that the learning process can be said to be "learning" these variables. Technically, an unriggable process need not be uninfluenceable, though it usually is (see the paper for details).

However, these properties only constrain the _expectation over environments_ of the final reward distribution: it doesn't prevent the agent from somehow shuffling around reward functions to be matched with suitable environments. For example, without knowing which projects are easy or hard, Sandra could manipulate Trisha into giving early deadlines for easy projects, and late deadlines for hard projects, in a manner that preserved the _distribution_ over early and late deadlines; this would satisfy the unriggable property (and probably also the uninfluenceable property, depending on the exact formalization).

The authors demonstrate these problems in a simple gridworld example. They also point out that there's a simple way to make any learning process uninfluenceable: choose a specific policy π that gathers information about the reward, and then define the new learning process to be "whatever the original learning process would have said if you executed π".

I would explain this paper's point somewhat differently than the paper does. Consider an AI system in which we build in a prior over rewards and an update rule, and then have it act in the world. At the end of the trajectory, it is rewarded according to the expected reward of the trajectory under the inferred posterior over rewards. Then, the AI system is incentivized to choose actions under which the resulting posterior is easy to maximize.

This doesn't require the reward function to be ambiguous; it just requires that the update rule isn't perfect. For example, imagine that Alice has a real preference for apples over bananas, and you use the update rule "if Alice eats an apple, infer that she likes apples; if Alice eats a banana, infer that she likes bananas". The robot finds it easier to grasp the rigid apple, and so can get higher expected reward in the worlds where Alice likes apples. If you train a robot in the manner above, then the robot will learn to throw away the bananas, so that Alice's only choice is an apple (that we assume she then eats), allowing the robot to "infer" that Alice likes apples, which it can then easily maximize. This sort of problem could happen in most current reward learning setups, if we had powerful enough optimizers.

It seems to me that the problem is that you are training the actor, but not training the update rule, and so the actor learns to "trick" the update rule. Instead, it seems like we should train both. This is kind of what happens with <@assistance games / CIRL@>(@Cooperative Inverse Reinforcement Learning@), in which you train a policy to maximize expected reward under the _prior_, and so the policy is incentivized to take the best information gathering actions (which, if you squint, is like "training to update well"), and to maximize what it thinks is the true reward. Of course, if your prior / update rule within the game are misspecified, then bad things can happen. See also Stuart's reactions [here](https://www.alignmentforum.org/posts/gbuwgyYG9WvtsErki/how-should-ais-update-a-prior-over-human-preferences) and [here](https://www.alignmentforum.org/posts/EYEkYX6vijL7zsKEt/reward-functions-and-updating-assumptions-can-hide-a), as well as my comments on those posts.

Blog post: Learning and manipulating learning


Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition

Justin Fu
Avi Singh
Dibya Ghosh
Larry Yang
Sergey Levine

For reinforcement learning, we can create a probabilistic model in which there are events for the state the agent is in and the action the agent takes. We can also add events e_t corresponding roughly to "the agent achieved something good in timestep t". We set P(e_t = 1 | s_t, a_t) to be exp(R(s_t, a_t)). Then, we can simply set all of the e_t to 1, and infer the likely state-action pairs that would have led to that. This leads to maximum entropy reinforcement learning, which in the setting of deterministic dynamics is equivalent to soft Q-learning. The authors then note that in this setup, the reward corresponds to the log probability of event e_t happening. So, instead of specifying a reward function, we can instead define binary events that we care about, model their probability of occurring, and then find the actions that maximize the likelihood of the event occurring. The authors derive backup equations for three kinds of queries -- ALL (the event must happen every timestep), AT (the event happens at a particular timestep), and ANY (the event happens on some timestep).

In this setup, specifying a reward function corresponds to explicitly writing down probabilities P(e | s, a). Of course, we can learn these probabilities from data using standard ML techniques, and this now corresponds to learning a reward function! If we use the ALL query, this corresponds to inverse reinforcement learning. However, by using the AT or ANY query instead, we only require examples of the event e_t for a single s_t and a_t -- for example, images that represent a goal state. They derive an algorithm for this query and show experimentally that this framework can learn event probabilities that lead to good behavior on Mujoco environments.

I like this framework for a couple of reasons. First, it allows for multiple kinds of queries, which correspond to different ways of specifying tasks, increasing the number of types of inputs we can give in order to communincate our intent to an AI. Concretely, the framework can handle both demonstrations (as in IRL) and examples of goal states. Second, it reduces learning a reward function to learning the probabilities of events, which has been studied in much more depth in the machine learning community and so will hopefully work better.


Meta-Inverse Reinforcement Learning with Probabilistic Context Variables

Lantao Yu*
Tianhe Yu*
Chelsea Finn
Stefano Ermon

This work explores improving performance on multi-task inverse reinforcement learning in a single-shot setting by extending <@Adversarial Inverse Reinforcement Learning@>(@Learning Robust Rewards with Adversarial Inverse Reinforcement Learning@) with "latent context variables" that condition the learned reward function. The paper makes two notable contributions: 1) It details an algorithm to simultaneously learn a flexible reward function and a conditional policy with competitive few-shot generalization abilities from expert demonstrations of multiple related tasks _without_ task specifications or identifiers; 2) The authors empirically demonstrate strong performance of a policy trained on the inferred reward of a structurally similar task with modified environmental dynamics, claiming that in order to succeed "the agent must correctly infer the underlying goal of the task instead of simply mimicking the demonstration".

Since this work "integrates ideas from context-based meta-learning, deep latent variable generative models, and maximum entropy inverse RL" and covers the relevant mathematics, it is an involved, if rewarding, study into multi-task IRL. I am convinced that this is a big step forward for IRL, but I'd be interested in seeing comparisons on setups that are more complicated.

'Data efficiency' is implied as a desirable quality, and the paper makes a case that they learn from a limited number demonstrations at meta-test time. However, it does not specify how many demonstrations were required for each task during _meta-training_. Additionally, for two environments, _tens of millions_ of environment interactions were required, which is entirely infeasible for real systems.


Deep Bayesian Reward Learning from Preferences

Bayesian inverse reinforcement learning (IRL) is ideal for safe imitation learning since it allows uncertainty in the reward function estimator to be quantified. This approach requires thousands of likelihood estimates for proposed reward functions. However, each likelihood estimate requires training an agent according to the hypothesized reward function. Predictably, such a method is computationally intractable for high dimensional problems.

**In this paper, the authors propose Bayesian Reward Extrapolation (B-REX), a scalable preference-based Bayesian reward learning algorithm.** They note that in this setting, a likelihood estimate that requires a loop over all demonstrations is much more feasible than an estimate that requires training a new agent. So, they assume that they have a set of _ranked_ trajectories, and evaluate the likelihood of a reward function by its ability to reproduce the preference ordering in the demonstrations. To get further speedups, they fix all but the last layer of the reward model using a pretraining step: the reward of a trajectory is then simply the dot product of the last layer with the features of the trajectory as computed by all but the last layer of the net (which can be precomputed and cached once).

The authors test B-REX on pixel-level Atari games and show competitive performance to <@T-REX@>(@Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations@), a related method that only computes the MAP estimate. Furthermore, the authors can create confidence intervals for performance since they can sample from the reward distribution.

Workshop on Safety and Robustness in Decision Making at NeurIPS 2019

The idea of using preference orderings (Bradley-Terry) to speed up the posterior probability calculation was ingenious. While B-REX isn't strictly better than T-REX in terms of rewards achieved, the ability to construct confidence intervals for performance is a major benefit. My takeaway is that Bayesian IRL is getting more efficient and may have good potential as a practical approach to safe value learning.


Learning human objectives by evaluating hypothetical behaviours

Siddharth Reddy
Anca D. Dragan
Sergey Levine
Shane Legg
Jan Leike

[Deep RL from Human Preferences](https://deepmind.com/blog/learning-through-human-feedback/) updated its reward model by collecting human comparisons on on-policy trajectories where the reward model ensemble was most uncertain about what the reward should be. However, we want our reward model to be accurate off policy as well, even in unsafe states. To this end, we would like to train our reward model on _hypothetical_ trajectories. This paper proposes learning a generative model of trajectories from some dataset of environment dynamics, such as safe expert demonstrations or rollouts from a random policy, and then finding trajectories that are "useful" for training the reward model. They consider four different criteria for usefulness of a trajectory: _uncertain rewards_ (which intuitively are areas where the reward model needs training), _high rewards_ (which could indicate reward hacking), _low rewards_ (which increases the number of unsafe states that the reward model is trained on), and _novelty_ (which covers more of the state space). Once a trajectory is generated, they have a human label it as good, neutral, or unsafe, and then train the reward model on these labels.

The authors are targeting an agent that can _explore safely_: since they already have a world model and a reward model, they use a model-based RL algorithm to act in the environment. Specifically, to act, they use gradient descent to optimize a trajectory in the latent space that maximizes expected rewards under the reward model and world model, and then take the first action of that trajectory. They argue that the world model can be trained on a dataset of safe human demonstrations (though in their experiments they use rollouts from a random policy), and then since the reward model is trained on hypothetical behavior and the model-based RL algorithm doesn't need any training, we get an agent that acts without us ever getting to an unsafe state.

I like the focus on integrating active selection of trajectory queries into reward model training, and especially the four different kinds of active criteria that they consider, and the detailed experiments (including an ablation study) on the benefits of these criteria. These seem important for improving the efficiency of reward modeling.

However, I don't buy the argument that this allows us to train an agent without visiting unsafe states. In their actual experiments, they use a dataset gathered from a random policy, which certainly will visit unsafe states. If you instead use a dataset of safe human demonstrations, your generative model will only place probability mass on safe demonstrations, and so you'll never generate trajectories that visit unsafe states, and your reward model won't know that they are unsafe. (_Maybe_ your generative model will generalize properly to the unsafe states, but that seems unlikely to me.) Such a reward model will either be limited to imitation learning (sticking to the same trajectories as in the demonstrations, and never finding something like AlphaGo's move 37), or it will eventually visit unsafe states.

Paper: Learning Human Objectives by Evaluating Hypothetical Behavior


Norms, Rewards, and the Intentional Stance: Comparing Machine Learning Approaches to Ethical Training

Daniel Kasenberg
Thomas Arnold
Matthias Scheutz

This paper argues that _norm inference_ is a plausible alternative to inverse reinforcement learning (IRL) for teaching a system what people want. Existing IRL algorithms rely on the _Markov assumption_: that the next state of the world depends only on the previous state of the world and the action that the agent takes from that state, rather than on the agent’s entire history. In cases where information about the past matters, IRL will either fail to infer the right reward function, or will be forced to make challenging guesses about what past information to encode in each state. By contrast, _norm inference_ tries to infer what (potentially temporal) propositions encode the reward of the system, keeping around only past information that is relevant to evaluating potential propositions. The paper argues that norm inference results in more interpretable systems that generalize better than IRL -- systems that use norm inference can successfully model reward-driven agents, but systems that use IRL do poorly at learning temporal norms.

This paper presents a interesting novel alternative to inverse reinforcement learning and does a good job of acknowledging potential objections. Deciding whether and how to store information about the past seems like an important problem that inverse reinforcement learning has to reckon with. My main concern with norm inference, which the paper mentions, is that optimizing over all possible propositions is in practice extremely slow. I don't anticipate that norm inference will be a performance-tractable strategy unless a lot of computation power is available.

**Rohin's opinion:** The idea of "norms" used here is very different from what I usually imagine, as in e.g. <@Following human norms@>. Usually, I think of norms as imposing a constraint upon policies rather than defining an optimal policy, (often) specifying what not to do rather than what to do, and being a property of groups of agents, rather than of a single agent. (See also [this comment](https://www.alignmentforum.org/posts/eBd6WvzhuqduCkYv3/following-human-norms#ujma2pWoH7ibhdog2).) The "norms" in this paper don't satisfy any of these properties: I would describe their norm inference as performing IRL with history-dependent reward functions, with a strong inductive bias towards "logical" reward functions (which comes from their use of Linear Temporal Logic). Note that some inductive bias is necessary, as without inductive bias history-dependent reward functions are far too expressive, and nothing could be reasonably learned. I think despite how it's written, the paper should be taken not as a denouncement of IRL-the-paradigm, but a proposal for better IRL algorithms that are quite different from the ones we currently have.


Leveraging Human Guidance for Deep Reinforcement Learning Tasks

Ruohan Zhang
Faraz Torabi
Lin Guan
Dana H. Ballard
Peter Stone

A core problem in RL is the communication of our goals and prior knowledge to an agent. One common approach to this is imitation learning: the human provides example demonstrations of a task, and the agent learns to mimic them. However, there are some limitations to this approach, such as requiring the human to be capable of the task. This paper outlines five different modalities from which agents can learn: evaluations, preferences, hierarchical feedback, observations, and attention (for example, where humans are looking while solving a task). It then suggests future research directions.

For this summary, I will focus on the future research directions, but you can read the full paper to understand existing approaches.  The first issue is that datasets of human guidance are difficult to capture and depend on many specific factors of the individuals providing guidance. As a result, the paper suggests creating standard datasets to save effort and enable fair comparisons. The second direction is to better understand how humans should teach agents. The literature currently emphasizes progress in learning methods, but improved teaching methods may be just as valuable when learning from human guidance. The last is unifying learning across different input modalities; ideally an agent would be able to learn from many different types of human guidance over different phases of its learning.

I think the problem of providing human guidance to agents is a core problem in alignment, and I am glad to see more discussion of that problem. I generally think that this type of broad overview is very valuable for communicating research to those who just want a broad overview of the field and don’t need to know the individual details of each paper. However, I would appreciate if there were more quantitative comparisons of the tradeoffs between different paradigms. The introduction mentions sample efficiency and the large effort required for human labelling, which made me hope for theoretical or empirical comparisons of the different methods with regards to sample efficiency and labelling effort. Since this was lacking, it also left me unclear on what motivated their suggested research directions. Personally, I would be much more excited to pursue a research direction if there were quantitative results showing particular failure modes or negative characteristics of current approaches that motivated that particular approach.

**Rohin's opinion:** This seems like a great survey paper and I like their proposed future directions, especially on learning from different kinds of human guidance, and on improving methods of teaching. While it does seem useful to have datasets of human guidance in order to compare algorithms, this prevents researchers from making improvements by figuring out new forms of guidance not present in the dataset. As a result, I'd be more excited about benchmarks that are evaluated by how much time it takes for Mechanical Turkers to train an agent to complete the task. Admittedly, it would be costlier in both time and money for researchers to do such an evaluation.


Fine-Tuning GPT-2 from Human Preferences

Daniel M. Ziegler
Nisan Stiennon
Jeffrey Wu
Tom B. Brown
Alec Radford
Dario Amodei
Paul Christiano
Geoffrey Irving

This blog post and its [associated paper](https://arxiv.org/abs/1909.08593) describes the results of several text generation/continuation experiments, where human feedback on initial/older samples was used in the form of a reinforcement learning reward signal to finetune the base 774-million parameter <@GPT-2 language model@>(@Better Language Models and Their Implications@). The key motivation here was to understand whether interactions with humans can help algorithms better learn and adapt to human preferences in natural language generation tasks.

They report mixed results. For the tasks of continuing text with positive sentiment or physically descriptive language, they report improved performance above the baseline (as assessed by external examiners) after fine-tuning on only 5,000 human judgments of samples generated from the base model. The summarization task required 60,000 samples of _online_ human feedback to perform similarly to a simple baseline, lead-3 - which returns the first three sentences as the summary - as assessed by humans.

Some of the lessons learned while performing this research include 1) the need for better, less ambiguous tasks and labelling protocols for sourcing higher quality annotations, and 2) a reminder that "bugs can optimize for bad behaviour", as a sign error propagated through the training process to generate "not gibberish but maximally bad output". The work concludes on the note that it is a step towards scalable AI alignment methods such as debate and amplification.

It is good to see research on mainstream NLProc/ML tasks that includes discussions on challenges, failure modes and relevance to the broader motivating goals of AI research.

The work opens up interesting avenues within OpenAI's alignment agenda, for example learning a diversity of preferences (A OR B), or a hierarchy of preferences (A AND B) sequentially without catastrophic forgetting.

In order to scale, we would want to generate automated labelers through semi-supervised reinforcement learning, to derive the most gains from every piece of human input. The robustness of this needs further empirical and conceptual investigation before we can be confident that such a system can work to form a hierarchy of learners, e.g. in amplification.

**Rohin's opinion:** One thing I particularly like here is that the evaluation is done by humans. This seems significantly more robust as an evaluation metric than any automated system we could come up with, and I hope that more people use human evaluation in the future.

Paper: Fine-Tuning Language Models from Human Preferences


Cognitive Model Priors for Predicting Human Decisions

David D. Bourgin*
Joshua C. Peterson*
Daniel Reichman
Thomas L. Griffiths
Stuart J. Russell

Human decision making is notoriously difficult to predict, being a combination of expected value calculation and likely-not-fully-enumerated cognitive biases. Normally we could predict well using a neural net with a ton of data, but data about human decision making is expensive and scarce. This paper proposes that we pretrain a neural net on lots of data simulated from theoretical models of human decision making and then finetune on the small real dataset. In effect, we are using the theoretical model as a kind of prior, that provides the neural net with a strong inductive bias. The method achieves better performance than existing theoretical or empirical methods, without requiring feature engineering, both on existing datasets and a new, larger dataset collected via Mechanical Turk.

I am a little cautious to make a strong statement about the importance of this paper, since I don't have as much domain knowledge in cognitive science as I do in machine learning, but overall this "treat your theoretical model like a generative model and sample from it" idea seems like an elegant and plausibly more broadly extensible way of incorporating theoretical priors alongside real data.


Deep Reinforcement Learning from Policy-Dependent Human Feedback

Dilip Arumugam
Jun Ki Lee
Sophie Saskin
Michael L. Littman

One obvious approach to human-in-the-loop reinforcement learning is to have humans provide an external reward signal that the policy optimizes. [Previous work](https://arxiv.org/abs/1701.06049) noted that humans tend to _correct_ existing behavior, rather than providing an "objective" measurement of how good the behavior is (which is what a reward function is). They proposed Convergent Actor-Critic by Humans (COACH), where instead of using human feedback as a reward signal, they use it as the _advantage function_. This means that human feedback is modeled as specifying how good an action is relative to the "average" action that the agent would have chosen from that state. (It's an average because the policy is stochastic.) Thus, as the policy gets better, it will no longer get positive feedback on behaviors that it has successfully learned to do, which matches how humans give reinforcement signals.

This work takes COACH and extends it to the deep RL setting, evaluating it on Minecraft. While the original COACH had an eligibility trace that helps "smooth out" human feedback over time, deep COACH requires an eligibility replay buffer. For sample efficiency, they first train an autoencoder to learn a good representation of the space (presumably using experience collected with a random policy), and feed these representations into the control policy. They reward entropy so that the policy doesn't commit to a particular behavior, making it responsive to feedback, but select actions by always picking the action with maximal probability (rather than sampling from the distribution) in order to have interpretable, consistent behavior for the human trainers to provide feedback on. They evaluate on simple navigation tasks in the complex 3D environment of Minecraft, including a task where the agent must patrol the perimeter of a room, which cannot be captured by a state-based reward function.

I really like the focus on figuring out how humans actually provide feedback in practice; it makes a lot of sense that we provide reinforcement signals that reflect the advantage function rather than the reward function. That said, I wish the evaluation had more complex tasks, and had involved human trainers who were not authors of the paper -- it might have taken an hour or two of human time instead of 10-15 minutes, but would have been a lot more compelling.

Before continuing, I recommend reading about Simulated Policy Learning in Video Models below. As in that case, I think that you get sample efficiency here by getting a lot of "supervision information" from the pixels used to train the VAE, though in this case it's by learning useful features rather than using the world model to simulate trajectories. (Importantly, in this setting we care about sample efficiency _with respect to human feedback_ as opposed to environment interaction.) I think the techniques used there could help with scaling to more complex tasks. In particular, it would be interesting to see a variant of deep COACH that alternated between training the VAE with the learned control policy, and training the learned control policy with the new VAE features. One issue would be that as you retrain the VAE, you would invalidate your previous control policy, but you could probably get around that (e.g. by also training the control policy to imitate itself while the VAE is being trained).


Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow

Xue Bin Peng
Angjoo Kanazawa
Sam Toyer
Pieter Abbeel
Sergey Levine

Adversarial learning techniques require a delicate balance between the generator and the discriminator. If the discriminator is too weak, it cannot tell the difference between generated samples and true samples, and it cannot provide a learning signal for the generator. If the discriminator is too strong, small changes to the generator are not going to fool the discriminator, and so again the gradient is uninformative. This paper proposes to control the power of the discriminator using an _information bottleneck_.

Instead of providing data points directly to the discriminator, the data points are first encoded into a new representation, and the discriminator must work with the new representation. The representation is learned to be helpful for the discriminator under the constraint of an upper bound on the mutual information between the representation and the original data points. The choice of upper bound determines how much information the discriminator is allowed to access, which in turn determines how powerful the discriminator is.

They apply this idea to imitation learning (GAIL), inverse reinforcement learning (AIRL), and image generation (GANs), and find that it improves results.


Guiding Policies with Language via Meta-Learning

The authors train an agent to perform tasks specified in natural language, with a "correction" after each attempt (also in natural language). They formulate this as a meta-learning problem: for each instruction, several attempt-correction cycles are allowed. Each attempt takes into account previous attempts to achieve the same instruction by passing each previous trajectory and its corresponding correction through a CNN, then using the mean of all outputs as an input to a policy module.

In their experiments, all instructions and corrections are generated automatically, and test-time performance is evaluated as a function of how many corrections are allowed. In one experiment, the tasks is to navigate rooms to reach a goal, where the correction is the next subgoal required. Given 4 corrections, their agent outperforms a baseline which was given all 5 subgoals at the beginning of the task. In another experiment, the task is to move a block to an ambiguously-specified location, and the corrections narrow down the target area; their trained agent scores 0.9, as opposed to 0.96 for an agent given the exact target location.

This paper explores an important idea: correcting poorly-specified instructions using human-in-the-loop feedback. The second task in particular is a nice toy example of iterative preference clarification. I'm not sure whether their meta-learning approach is directly relevant to safety, particularly because each correction is only "in scope" for a single episode, and also only occurs after a bad attempt has finished. However, the broad idea of correction-based learning seems promising.


Prompting: Better Ways of Using Language Models for NLP Tasks

Since the publication of <@GPT-3@>(@Language Models are Few-Shot Learners@), many papers have been written about how to select the best prompt for large language models to have them solve particular tasks of interest. This post gives an overview of this literature. The papers can be roughly divided into two approaches: first, we have discrete prompts, where you search for a sequence of words that forms an effective prompt; these are “discrete” since words are discrete. Second, we have soft prompts, where you search within the space of embeddings of words for an embedding that forms an effective prompt; since embeddings are vectors of real numbers they are continuous (or “soft”) and can be optimized through gradient descent (unlike discrete prompts).


Bayesian Inverse Reinforcement Learning

Unlike many other methods, [Bayesian Inverse Reinforcement Learning](https://www.aaai.org/Papers/IJCAI/2007/IJCAI07-416.pdf) produces a _posterior distribution_ over the reward functions that would explain the observed demonstrations. This distribution can be used for e.g. planning in a risk-averse manner. It works by starting with some randomly chosen reward function, and then repeating the following steps:

1. Perturb the reward function randomly
2. Solve for the optimal policy for that reward function
3. Use the learned policy to see how likely the demonstrations would be for the reward function
4. Use the likelihood to determine whether to take this new reward function, or return to the old one.

(This is the application of a standard MCMC sampling algorithm to the likelihood model used in IRL.)


AXRP 2: Learning Human Biases

After talking about <@my paper on learning biases@>(@Learning biases and rewards simultaneously@) (for which I refer you to the linked blog post and past AN summary), Daniel and I talked about the implications of inverse reinforcement learning for AI safety, and in particular how we would want AI systems to be architected at a high level.

My position was that we want intelligent AI systems to be trying to help their users: they are explicitly interacting with humans in order to clarify what they should do, perhaps by explicitly asking questions, or by watching other human decisions and making inferences about what humans must care about. (However, this isn’t the vast majority of what they do; it is probably significantly less than one-fifth of “everything they do”.)

In contrast, Daniel would prefer for a superintelligent AI system to be pursuing a well-defined task, such as “build a thriving city”. He has three reasons for this:

1. When our goal is to build AI systems that can pursue a relatively well-defined task, it is much easier for us to tell whether we are succeeding, and we can be much clearer about what it is we are trying to accomplish.
2. We can increase the difficulty of well-specified tasks over time, rising in tandem with the capabilities of AI systems. In contrast, if our AI system is supposed to generically make our life better, that seems like a fixed task that is fairly difficult and requires quite a high minimum threshold of capabilities.
3. It seems easier to tell whether your AI system has built a good city, than to tell whether an AI system has generically improved your life.

In the podcast, I don’t think I really engaged properly with the first two points, so I’ll talk about that in the opinion. I did disagree with the third point -- I don’t see why it should be harder to evaluate whether my life has been generically improved; for example, I expect that we are capable of telling apart good and bad personal assistants.

Daniel also asked why it helps to aim for “AI systems that are trying to help you” -- how has that made the problem any simpler? It seems to me that the notion of “helpfulness” is domain-independent: once you have the concept of being helpful, it can be applied in different domains. One hopes that we could then train lots of AI systems that are specialized to particular domains, but all of them are still trying to be helpful.

I think I broadly agree with Daniel’s first two points in support of the task-based approach, and I was somewhat talking past him during the podcast. I generally _do_ agree that individual AI systems should be specialized to particular tasks or domains, and should not be “generically improving one’s life”. I agree with Daniel that at least outwardly it seems like most of the AI alignment field seems to be about building AI systems that can generically optimize your entire life, or even more ambitiously, the lot of humanity; I also agree that this is weird and probably not the right thing to do.

My optimism about helpfulness is not predicated on an idea that we’ll build AI systems that are generically trying to make all aspects of your life better: I do think that we still want our AI systems to be domain-specific, such as (say) a financial advisor AI system. The idea is more that if we can design domain-general _techniques_ that allow us to train domain-specific _systems_ that are trying to be helpful, that seems like it would be a solution to the AI alignment problem (the problem of how to prevent an AI from adversarially optimizing against its user).


Imitation Learning in the Low-Data Regime

Robert Dadashi
Léonard Hussenot
Matthieu Geist
Olivier Pietquin

<@Non-Adversarial Imitation Learning@>(@Non-Adversarial Imitation Learning and its Connections to Adversarial Methods@) has become more popular recently due to the fact that GAN style architectures can be notoriously unstable during training. This paper makes a contribution by introducing an imitation learning strategy that relies on minimizing an upper bound on the Wasserstein distance between the imitator and expert state visitation distributions. The Wasserstein distance can be understood using the 'Earth Mover's Analogy'. In this interpretation, we view the distance as the cost of the most efficient transport strategy to move probability mass from the imitator distribution to the expert distribution. The advantage of such an approach is that the metric can be calculated in an offline way. If we calculate the distance for partial rollouts then we can create a dense, albeit non-stationary, reward for the imitator. In experiments, agents trained using the Wasserstein distance are able to learn control tasks using only a single trajectory.

With this paper, I conclude that IRL works for Mujoco-style control tasks. The performance of this method is similar to offline GAIL but is better justified and more stable. However, ultimately, I'm a bit skeptical of their claim that the method will generalize to other tasks. Results for GAIL/DAC are quite poor in Atari-like environments whereas pair-wise reward modeling seems to perform quite well. This would suggest a reward modeling approach would scale much better in more complicated settings.

Paper: Primal Wasserstein Imitation Learning


Learning to Summarize with Human Feedback

Nisan Stiennon*
Long Ouyang*
Jeff Wu*
Daniel M. Ziegler*
Ryan Lowe
Chelsea Voss
Alec Radford
Dario Amodei
Paul Christiano

OpenAI has been working on <@finetuning language models from human preferences@>(@Fine-Tuning GPT-2 from Human Preferences@). This blog post and paper show the progress they have made on text summarization in particular since their last release.

As a reminder, the basic setup is similar to that of [Deep RL from Human Preferences](https://deepmind.com/blog/learning-through-human-feedback/): we get candidate summaries by executing the policy, have humans compare which of two summaries is better, and use this feedback to train a reward model that can then be used to improve the policy. The main differences in this paper are:

1. They put in a lot of effort to ensure high data quality. Rather than having MTurk workers compare between summaries, they hire a few contractors who are paid a flat hourly rate, and they put a lot of effort into communicating what they care about to ensure high agreement between labelers and researchers.
2. Rather than collecting preferences in an online training setup, they collect large batches at a time, and run a relatively small number of iterations of alternating between training the reward model and training the policy. My understanding is that this primarily makes it simpler from a practical perspective, e.g. you can look at the large batch of data you collected from humans and analyze it as a unit.
3. They initialize the policy from a model that is first pretrained in an unsupervised manner (as in <@GPT-3@>(@Language Models are Few-Shot Learners@)) and then finetuned on the reference summaries using supervised learning.

On the Reddit task they train on, their summaries are preferred over the reference summaries (though since the reference summaries have varying quality, this does not imply that their model is superhuman). They also transfer the policy to summarize CNN / DailyMail news articles and find that it still outperforms the supervised model, despite not being trained at all for this setting (except inasmuch as the unsupervised pretraining step saw CNN / DailyMail articles).

An important ingredient to this success is that they ensure their policy doesn’t overoptimize the reward, by adding a term to the reward function that penalizes deviation from the supervised learning baseline. They show that if they put a very low weight on this term, the model overfits to the reward model and starts producing bad outputs.

This paper is a great look at what reward learning would look like at scale. The most salient takeaways for me were that data quality becomes very important and having very large models does not mean that the reward can now be optimized arbitrarily.

Paper: Learning to summarize from human feedback


Multi-Principal Assistance Games

Arnaud Fickinger
Simon Zhuang
Dylan Hadfield-Menell
Stuart Russell

So far the work in the <@assistance games framework@>(@Human Compatible: Artificial Intelligence and the Problem of Control@) (previously called CIRL) has focused on the case where there is a single human and a single AI assistant. Once we have multiple humans (or _principals_, as the paper calls them), things get much trickier.

One problem is that we don’t know how to aggregate the values across different principals. Rather than taking a stance on the problem, this paper assumes that we have some mechanism that can combine reward functions in some reasonable way. It instead focuses on a second problem: while previously we could trust the human to report their preferences accurately (as the human and agent were aligned), when there are multiple principals whose preference will be aggregated, the principals have an incentive to misrepresent their preferences (which we’ll call non-straightforward play).

Let’s consider the case where the principals provide demonstrations, _and get reward for those demonstrations_. For now our agent will assume that the principals are playing straightforwardly, and so the agent simply infers their preferences, aggregates them, and optimizes the results. In this setting, if the agent will act far more often than the principals provide demonstrations (so that the reward of the demonstrations is almost irrelevant), we can apply the Gibbard-Satterthwaite theorem to show that any non-trivial mechanism will be vulnerable to non-straightforward play. In contrast, if the principals provide lots of demonstrations, while the agent only acts for a short period of time, then optimal principals primarily want to ensure their demonstrations are good, and so will be straightforward most of the time (provably). In the middle, the fact that principals get rewarded for demonstrations does help reduce non-straightforward play, but does not eliminate it.

Now let’s consider the case where the agent can design a mechanism. Here, when the principals are providing demonstrations, the agent can override their action choice with one of its own (a setting considered <@previously@>(@The Assistive Multi-Armed Bandit@)). Roughly speaking, the algorithm only executes a proposed human action if it hasn’t executed it before. By doing so, it incentivizes the principals to report second-best actions, and so on, giving the agent more information about the principals' utility functions. The mechanism incentivizes straightforward play, and is approximately efficient (i.e. there is an upper bound on the worst case social welfare achieved).

Workshop on Incentives in Machine Learning, ICML 2020

According to me, the main insight of this paper is that it is both necessary and difficult to design mechanisms that incentivize principals to report not just the best thing to do, but a comparison amongst different alternatives. Within the formalism of paper, this is done by overriding a principal’s action unless it is a novel action, but I expect in practice we’ll do this in some other way (it seems rather unusual to imagine the agent overriding a human, I’d be surprised if that was how we ended up building our AI systems).


Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization

Paul Barde*
Julien Roy*
Wonseok Jeon*
Joelle Pineau
Christopher Pal
Derek Nowrouzezahrai

This work aims to simplify algorithms for adversarial imitation learning by using a _structured_ discriminator, which is parameterised by the current generator and a learned policy. They prove that if so formulated, the policy that yields the optimal discriminator is exactly the same as the policy that generated the expert data, which is also precisely what we hope the generator will learn. As long as the discriminator's learned policy is parameterised correctly such that it can be sampled and evaluated, this eliminates the need for a reinforcement learning outer loop for policy improvement, as this learned policy can be substituted in for the generator's policy in the next training iteration. They empirically show the competitiveness of their method with state-of-the-art algorithms across a small but increasingly complex suite of tasks.

Since their theoretical results are only for optimal values, it's unclear whether starting from random initial policies will necessarily converge to these optimal values -- indeed, they make this point themselves, that they do not train to convergence as gradient descent cannot hope to find the global optimum for GAN-like non-convex loss functions. In light of that, it's not evident *why* their algorithms outperform the competition. Additionally, they do not report computational speed-up or wall-clock comparisons, which to me felt like the broad motivation behind this work. Nonetheless, the work illuminates new territory in adversarial imitation learning, provides positive evidence for a novel technique, and raises interesting questions for future work, such as how to learn robust reward functions via this method, or what kind of convergence properties can be expected.


Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences

Daniel S. Brown
Russell Coleman
Ravi Srinivasan
Scott Niekum

Bayesian reward learning would allow for rigorous safety analysis when performing imitation learning. However, Bayesian reward learning methods are typically computationally expensive to use. This is because a separate MDP needs to be solved for each reward hypothesis. The main contribution of this work is a proposal for a more efficient reward evaluation scheme called Bayesian REX (see also an <@earlier version@>(@Deep Bayesian Reward Learning from Preferences@)). It works by pre-training a low-dimensional feature encoding of the observation space which allows reward hypotheses to be evaluated as a linear combination over the learned features. Demonstrations are ranked using pair-wise preference which is relativistic and thus conceptually easier for a human to evaluate. Using this method, sampling and evaluating reward hypotheses is extremely fast: 100,000 samples in only 5 minutes using a PC. Moreover, Bayesian REX can be used to play Atari games by finding a most likely or mean reward hypothesis that best explains the ranked preferences and then using that hypothesis as a reward function for the agent.

It's worth emphasizing that this isn't quite a pure IRL method. They use preferences over demonstrations in addition to the demonstrations themselves and so they have more information than would be available in a pure IRL context. However, it’s also worth emphasizing that (as the authors show) pixel-level features make it difficult to use IRL or GAIL to learn an imitation policy, which means I wasn’t expecting a pure IRL approach to work here. Conceptually, what's interesting about the Bayesian approach is that uncertainty in the reward distribution translates into confidence intervals on expected performance. This means that Bayesian REX is fairly robust to direct attempts at reward hacking due to the ability to directly measure overfitting to the reward function as high variance in the expected reward.


Showing versus doing: Teaching by demonstration

Mark K. Ho
Michael Littman
James MacGlashan
Fiery Cushman
Joseph L. Austerweil

This paper creates and validates a model of _pedagogy_ as applied to reward learning. Typically, inverse reinforcement learning (IRL) algorithms assume access to a set of demonstrations that are created from an approximately _optimal_ policy. However, in practice, when people are asked to _show_ a task, they don't give the optimal trajectory; they give the trajectory that helps the learner best _disambiguate_ between the possible tasks. They formalize this by creating a model in two steps:

1. A literal or IRL robot is one which learns rewards under the model that the demonstrator is Boltzmann rational.
2. The pedagogic human shows trajectories in proportion to how likely a literal robot would think the true reward is upon seeing the trajectory.

They validate this model with user studies and find that it predicts human demonstrations well.

<@Literal or Pedagogic Human? Analyzing Human Model Misspecification in Objective Learning@>


Active Preference-Based Learning of Reward Functions

Dorsa Sadigh
Anca D. Dragan
Shankar Sastry
and Sanjit A. Seshia

In a continuous, dynamical system such as an autonomous car, how can we learn good behavior without hardcoding a reward function? It may be too hard to get demonstrations (humans could only show how they drive a car, not how they want the car to drive). The authors propose presenting the human with two trajectories and asking her to choose which one is better, and using this information to infer the reward.

This paper is a good example of reward learning, and it's interesting to compare and contrast this more principled method to Deep RL from Human Preferences. For example, while they maintain a distribution over rewards and choose the most useful query, deep RL from human preferences works with higher-dimensional reward functions where this would be too expensive, and so they instead train an ensemble of reward predictors and use disagreement between the reward predictors as a measurement of uncertainty.

Deep Reinforcement Learning from Human Preferences


Imitation Learning via Off-Policy Distribution Matching

Ilya Kostrikov
Ofir Nachum
Jonathan Tompson

One way to view imitation learning is as a distribution matching problem. In other words, the agent is rewarded based on how well it can imitate the state-distribution induced by the expert. In recent years, distribution matching via adversarial methods such as GAIL has become a popular approach to imitation learning. However, one weakness of these methods is that they require on-policy samples which means they require the agent to interact with the environment. In this paper, the authors present an off-policy method for distribution matching which can work without environment interaction. They do this by building on the prior work of DualDICE, a policy-agnostic method to estimate distribution ratios between agent and expert which can then be used to provide a reward to the agent. This allows the optimal policy to be estimated directly from demonstrations without any need for agent interaction. The authors run a few experiments and show that the method has comparable performance to behavioral cloning in the off-policy setting and adversarial methods in the on-policy setting.

This is a cool application of density-estimation via DualDICE. While the experiments are a bit weak, the fact that an off-policy method exists to do distribution-matching is interesting in its own right. Moreover, the method seems able to compete with both BC and GAIL-like methods which is intriguing.


State-only Imitation with Transition Dynamics Mismatch

Most existing imitation learning algorithms rely on the availability of expert demonstrations that come from the *same* MDP as the one the imitator will be evaluated in. With the advent of <@adversarial inverse reinforcement learning (AIRL)@>(@Learning Robust Rewards with Adversarial Inverse Reinforcement Learning@), it has become possible to learn general behaviors. However, algorithms such as <@GAIL@>(@Generative Adversarial Imitation Learning@) are capable of learning with just state-information, something that AIRL was not designed for. In this paper, the authors introduce indirect-imitation learning (I2L) to try and merge the benefits of both GAIL and AIRL. The basic sketch of the algorithm is to first use a generalization of AIRL to imitate demonstrations via a buffer distribution and then focus on moving that buffer closer to the expert's demonstration distribution using a Wasserstein critic, a smoother way to train GAN networks. By combining these two approaches, agents trained with I2L learn how to control Ant in regular gravity and can *generalize* to perform in simulations with differing parameters for gravity. For the suite of Gym continuous domains, they show consistent advantages for I2L over other algorithms such as GAIL, BCO, and AIRL when parameters such as friction, density, and gravity are changed.

The main contribution in this paper seems to be deriving a new bound so that AIRL can handle state-only imitation learning. The use of indirection via a buffer is also interesting and seems to be a good idea to provide stability in training. However, they did not do an ablation. Overall, it's aesthetically interesting that this paper is borrowing tricks, such as buffering and Wasserstein critic. Finally, the results seem promising, particularly for the sim-to-real problem. It would be interesting to see a follow-up to gauge whether or not I2L can help bridge this gap.

Paper: Learning Robust Rewards With Adversarial Inverse Reinforcement Learning


The MineRL Competition on Sample-Efficient Reinforcement Learning Using Human Priors: A Retrospective

Stephanie Milani
Nicholay Topin
Brandon Houghton
William H. Guss
Sharada P. Mohanty
Oriol Vinyals
Noboru Sean Kuno

This paper reports on the results of the <@MineRL competition@>(@NeurIPS 2019 Competition: The MineRL Competition on Sample Efficient Reinforcement Learning using Human Priors@), in which participants had to train agents to obtain a diamond in Minecraft using a limited amount of compute, environment interactions, and human demonstrations. While no team achieved this task, one team did make it to the penultimate milestone: obtaining an iron pickaxe.

The top nine teams all used some form of action reduction: that is, they constrained their agents to only take a subset of all available actions, shaping the space in which the agent had to learn and explore. The top four teams all used some form of hierarchy in order to learn longer "options" that could then be selected from. The second place team used pure imitation learning (and so required _no_ environment interactions), while the eighth and ninth place teams used pure reinforcement learning (and so required _no_ human demonstrations).

I was surprised to see pure RL solutions rank in the leaderboard, given the limitations on compute and environment interactions. Notably though, while the second place team (pure imitation) got 42.41 points, the eighth place team (pure RL) only got 8.25 points.

More generally, I was excited to see an actual benchmark for techniques using human demonstrations: so far there hasn't been a good evaluation of such techniques. It does seem like Minecraft benefits a lot from hierarchy and action pruning, which we may not care about when evaluating algorithms.


Learning Safe Policies with Expert Guidance

Jessie Huang
Fa Wu
Doina Precup
Yang Cai

Expert demonstrations can be consistent with many possible reward functions. Instead of simply trying to mimic the demonstration, the authors consider all possible rewards that are consistent with the demonstration, and then maximize the worst reward, leading to safe behavior.

This is very related to [Inverse Reward Design](https://arxiv.org/abs/1711.02827), where instead of maxmin planning we use risk-averse planning, and instead of considering all rewards compatible with an expert demonstration we consider all reward functions that are probable based on which reward function the designer wrote down.


Learning to Imitate Human Demonstrations via CycleGAN

Laura Smith
Nikita Dhawan
Marvin Zhang
Pieter Abbeel
Sergey Levine

Most methods for imitation learning, where robots learn from a demonstration, assume that the actions of the demonstrator and robot are the same. This means that expensive techniques such as teleoperation have to be used to generate demonstrations. **This paper presents a method to engage in automated visual instruction-following with demonstrations (AVID) that works by translating video demonstrations done by a human into demonstrations done by a robot.** To do this, the authors use [CycleGAN](https://junyanz.github.io/CycleGAN/), a method to translate an image from one domain to another domain using unpaired images as training data. CycleGAN allows them to translate videos of humans performing the task into videos of the robot performing the task, which the robot can then imitate. In order to make learning tractable, the demonstrations had to be divided up into 'key stages' so that the robot can learn a sequence of more manageable tasks. In this setup, the robot only needs supervision to ensure that it's copying each stage properly before moving on to the next one. To test the method, the authors have the robot retrieve a coffee cup and make coffee. AVID significantly outperforms other imitation learning methods and can achieve 70% / 80% success rate on the tasks, respectively.

In general, I like the idea of 'translating' demonstrations from one domain into another. It's worth noting that there do exist methods for translating visual demonstrations into latent policies. I'm a bit surprised that we didn't see any comparisons with other adversarial methods like [GAIfO](https://arxiv.org/pdf/1807.06158.pdf), but I understand that those methods have high sample complexity so perhaps the methods weren't useful in this context. It's also important to note that these other methods would still require demonstration translation. Another criticism is that AVID is not fully autonomous since it relies on human feedback to progress between stages. However, compared to kinetic teaching or teleoperation, sparse feedback from a human overseer is a minor inconvenience.

Paper: AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos