AI Alignment Podcast: Synthesizing a human’s preferences into a utility function

Stuart Armstrong's <@agenda@>(@Research Agenda v0.9: Synthesising a human's preferences into a utility function@) involves extracting partial preferences from a human and synthesizing them together into an _adequate_ utility function. Among other things, this podcast goes into the design decisions underlying the agenda:

First, why even have a utility function? In practice, there are [many pressures](https://www.lesswrong.com/posts/RQpNHSiWaXTvDxt6R/coherent-decisions-imply-consistent-utilities) suggesting that maximizing expected utility is the "right" thing to do -- if you aren't doing this, you're leaving value on the table. So any agent that isn't maximizing a utility function will want to self-modify into one that is using a utility function, so we should just use a utility function in the first place.

Second, why not defer to a long reflection process, as in [Indirect Normativity](https://ordinaryideas.wordpress.com/2012/04/21/indirect-normativity-write-up/), or some sort of reflectively stable values? Stuart worries that such a process would lead to us prioritizing simplicity and elegance, but losing out on something of real value. This is also why he focuses on _partial preferences_: that is, our preferences in "normal" situations, without requiring such preferences to be extrapolated to very novel situations. Of course, in any situation where our moral concepts break down, we will have to extrapolate somehow (otherwise it wouldn't be a utility function) -- this presents the biggest challenge to the research agenda.

Stuart Armstrong Research Agenda Online Talk


Learning from Observations Using a Single Video Demonstration and Human Feedback

Sunil Gandhi
Tim Oates
Tinoosh Mohsenin
Nicholas Waytowich

Designing rewards can be a long and consuming process, even for experts. One common method to circumvent this problem is through demonstration. However, it might be difficult to record demonstrations in a standard representation, such as joint positions. **In this paper, the authors propose using human feedback to circumvent the discrepancy between how demonstrations are recorded (video) and the desired standard representation (joint positions).** First, humans provide similarity evaluations of short clips of an expert demonstration to the agent's attempt and a similarity function is learned by the agent. Second, this similarity function is used to help train a policy that can imitate the expert. Both functions are learned jointly. The algorithm can learn to make a Hopper agent back-flip both from a Hopper demonstration of a back-flip, and from a YouTube video of a human backflipping. Ultimately, the authors show that their method improves over another method that uses human feedback without direct comparison to desired behavior.

This paper seems like a natural extension of prior work. The imitation learning problem from observation is well-known and difficult. Introducing human feedback with a structured state space definitely seems like a viable way to get around a lot of the known difficulties with other methods such as a GAIL.


Incorrigibility in the CIRL Framework

This paper demonstrates that when the agent has an _incorrect_ belief about the human's reward function, then you no longer get the benefit that the agent will obey shutdown instructions. It argues that since the purpose of a shutdown button is to function as a safety measure of last resort (when all other measures have failed), it should not rely on an assumption that the agent's belief about the reward is correct.

I certainly agree that if the agent is wrong in its beliefs about the reward, then it is quite likely that it would not obey shutdown commands. For example, in the off switch game, if the agent is incorrectly certain that u is positive, then it will take action a, even though the human would want to shut it down. See also <@these@>(@Latent Variables and Model Mis-Specification@) <@posts@>(@Model Mis-specification and Inverse Reinforcement Learning@) on model misspecification and IRL. For a discussion of how serious the overall critique is, both from HC's perspective and mine, see the opinion on the next post.


Problem of fully updated deference

This article points out that even if you have an agent with uncertainty over the reward function, it will acquire information and reduce its uncertainty over the reward, until eventually it can't reduce uncertainty any more, and then it would simply optimize the expectation of the resulting distribution, which is equivalent to optimizing a known objective, and has the same issues (such as disabling shutdown buttons).

As with the previous paper, this argument is only really a problem when the agent's belief about the reward function is _wrong_: if it is correct, then at the point where there is no more information to gain, the agent should already know that humans don't like to be killed, do like to be happy, etc. and optimizing the expectation of the reward distribution should lead to good outcomes. Both this and the previous critique are worrisome when you can't even put a reasonable _prior_ over the reward function, which is quite a strong claim.

HC's response is that the agent should never assign zero probability to any hypothesis. It suggests that you could have an expandable hierarchical prior, where initially there are relatively simple hypotheses, but as hypotheses become worse at explaining the data, you "expand" the set of hypotheses, ultimately bottoming out at (perhaps) the universal prior. I think that such an approach could work in principle, and there are two challenges in practice. First, it may not be computationally feasible to do this. Second, it's not clear how such an approach can deal with the fact that human preferences _change_ over time. (HC does want more research into both of these.)

Fully updated deference could also be a problem if the observation model used by the agent is incorrect, rather than the prior. I'm not sure if this is part of the argument.


Some Comments on Stuart Armstrong's "Research Agenda v0.9"

This post makes two main critiques of the research agenda in the previous entry. First, the research agenda involves a lot of human-designed features and modules, but <@The Bitter Lesson@> is that machine learning tends to shine with highly abstract large models that can make use of a lot of compute. Second, the symbol grounding part of the agenda requires the AI system to develop representations of the world that match the representations that humans use, and we have no idea how to do that, or even what it would mean to "match human representations" when the AI is more intelligent than humans. The post also includes some more specific comments that I'm not summarizing.

I agree with both of these critiques, especially the one about the bitter lesson. It seems like Stuart's approach imposes a particular structure or algorithm for how to synthesize the utility function; I am generally skeptical of such approaches. Also, as you might already know, I think it is neither necessary nor sufficient for AI alignment to find a utility function or "goal" that the AI can safely optimize. Since this promises to be a very difficult enterprise (Section 0.2 notes that it aims to "solve at least 5 major open problems in philosophy, to a level rigorous enough that we can specify them in code"), I prefer to look into other approaches that seem more tractable.

I do think that the problems that motivate the various aspects of the agenda are important and useful to think about, and I am happy that they have all been put into this single post. I also like the fact that the research agenda is directly aiming for a full solution to AI alignment.


Batch Active Preference-Based Learning of Reward Functions

This paper builds on a trend of recent papers that try to learn human preferences, not through demonstrations of optimal behavior, but through a human expressing a preference over two possible trajectories, which has both pragmatic advantages (re limits of human optimality) and theoretic ones (better ability to extrapolate a reward function). Here, the task is framed as: we want to send humans batches of paired trajectories to rank, but which ones? Batch learning is preferable to single-sample active learning because it's more efficient to update a network after a batch of human judgments, rather than after each single one. This adds complexity to the problem because you'd prefer to not have a batch of samples that are individually high-expected-information, but which are redundant with one another. The authors define an information criterion (basically the examples about which we're most uncertain of the human's judgment) and then pick a batch of examples based on different heuristics for getting a set of trajectories with high information content that are separated from each other in feature space.

This is an elegant paper that makes good use of the toolkit of active learning for human preference solicitation, but it's batch heuristics are all very reliant on having a set of high level trajectory features in which Euclidean distance between points is a meaningful similarity metric, which feels like a not impossible to generalize but still somewhat limiting constraint.

<@Active Preference-Based Learning of Reward Functions@>


End-to-End Robotic Reinforcement Learning without Reward Engineering

Avi Singh
Larry Yang
Kristian Hartikainen
Chelsea Finn
Sergey Levine

This paper demonstrates an approach that can learn to perform real world robotics tasks based not on example trajectories (states and actions) but just a small number (10) of pixel-level images of goal states showing successful task completion. Their method learns a GAN-like classifier to predict whether a given image is a success, continually adding data sampled from the still-learning policy to the set of negative examples, so the model at each step needs to further refine its model of success. The classifier, which is used as the reward signal in learning the policy, also makes use of a simple active learning approach, choosing the state its classifier is most confident is success and querying a human about it on fixed intervals, ultimately using less than 75 queries in all cases.

This is a result I find impressive, primarily because of its interest in abiding by sensible real-world constraints: it's easier for humans to label successful end states than to demonstrate a series of actions, and the number of queries made was similarly pragmatically low.


Conditional revealed preference

When backing out preferences by looking at people's actions, you may find that even though they say they are optimizing for X, their actions are better explained as optimizing for Y. This is better than relying on what they say, at least if you want to predict what they will do in the future. However, all such inferences are specific to the current context. For example, you may infer that schools are "about" dealing with authoritarian work environments, as opposed to learning -- but maybe this is because everyone who designs schools doesn't realize what the most effective methods of teaching-for-learning are, and if they were convinced that some other method was better for learning they would switch to that. So, in order to figure out what people "really want", we need to see not only what they do in the current context, but also what they would do in a range of alternative scenarios.

The general point here, which comes up pretty often, is that any information you get about "what humans want" is going to be specific to the context in which you elicit that information. This post makes that point when the information you get is the actions that people take. Some other instances of this point:

    - [Inverse Reward Design](https://arxiv.org/abs/1711.02827) notes that a human-provided reward function should be treated as _specific to the training environment_, instead of as a description of good behavior in all possible environments.
    - [CP-Nets](https://www.cs.toronto.edu/~cebly/Papers/CPnets.pdf) are based on the point that when a human says "I want X" it is not a statement that is meant to hold in all possible contexts. They propose very weak semantics, where "I want X" means "holding every other aspect of the world constant, it would be better for X to be present than for it not to be present".
    - [Wei Dai's point](https://www.alignmentforum.org/posts/HTgakSs6JpnogD6c2/two-neglected-problems-in-human-ai-safety) ([AN #37](https://mailchi.mp/6eb8190e723f/alignment-newsletter-37)) that humans likely have adversarial examples, and we should not expect preferences to generalize under distribution shift.
    - Stuart Armstrong and Paul Christiano have made or addressed this point in many of their posts.


AI Alignment Podcast: Human Cognition and the Nature of Intelligence

Joshua Greene's lab has two research directions. The first is how we combine concepts to form thoughts: a process which allows us to understand arbitrary novel scenarios (even ones we don't think ever occurred). He discusses some of his recent reseach, which uses brain imaging to infer what's happening when humans think about compound concepts. While Joshua considers the combinatorial nature of thought to be important, he argues that to build AGI, it's necessary to start with "grounded cognition" in which representations are derived from perception and physical action, rather than just learning to manipulate symbols (like language).

Joshua also works on the psychology and neuroscience of morality. He discusses his recent work in which participants are prompted to consider Rawls' Veil of Ignorance argument (that when making decisions affecting many people, we should do so as if we don't know which one we are) and then asked to evaluate moral dilemmas such as trolley problems. Joshua argues that the concept of impartiality is at the core of morality, and that it pushes people towards more utilitarian ideas (although he wants to rebrand utilitarianism as "deep pragmatism" to address its PR problems).


Learning from Demonstration in the Wild

Feryal Behbahani
Kyriacos Shiarlis
Xi Chen
Vitaly Kurin
Sudhanshu Kasewa
Ciprian Stirbu
João Gomes
Supratik Paul
Frans A. Oliehoek
João Messias
Shimon Whiteson

This paper learns traffic trajectories from unsupervised data by converting traffic camera footage into a Unity scene simulation, using that simulation to generate pseudo-LIDAR readings for each "expert trajectory", and then training an agent to imitate them using a variant of generative adversarial imitation learning (GAIL).

This is a cool example of how huge amounts of existing unlabeled video data might be utilised. The task they attempt is significantly more complex than those in other similar work (such as [this paper](https://arxiv.org/abs/1805.11592) which learns to play Atari games from Youtube videos); however, this also makes it difficult to judge how well the learned policy performed, and how much potential it has to transfer into the real world.


Shared Autonomy via Deep Reinforcement Learning

In shared autonomy, an AI system assists a human to complete a task. The authors implement shared autonomy in a deep RL framework by simply extending the state with the control input from the human, and then learning a policy that chooses actions given the extended state. They show that the human-AI team performs better than either one alone in the Lunar Lander environment.

Shared autonomy is an interesting setting because the human is still necessary in order to actually perform the task, whereas in typical reward learning settings, once you have learned the reward function and the AI is performing well, the human does not need to be present in order to execute a good policy.


Addressing Sample Inefficiency and Reward Bias in Inverse Reinforcement Learning

Ilya Kostrikov
Kumar Krishna Agrawal
Debidatta Dwibedi
Sergey Levine
Jonathan Tompson

Deep IRL algorithms typically work by training a discriminator that distinguishes between states and actions from the expert from states and actions from the learned policy, and extracting a reward function from the discriminator. In any environment where the episode can end after a variable number of timesteps, this assumes that the reward is zero after the episode ends. The reward function from the discriminator often takes a form where it must always be positive, inducing a survival incentive, or a form where it must always be negative, inducing a living cost. For example, [GAIL](https://arxiv.org/abs/1606.03476)'s reward is always positive, giving a survival incentive. As a result, _without any reward learning at all_ GAIL does better on Hopper than behavioral cloning, and fails to learn on a reaching or pushing task (where you want to do the task as quickly as possible, so you want the living cost). To solve this, they learn an "absorbing state reward", which is a reward given after the episode ends -- this allows the algorithm to learn for itself whether it should have a survival incentive or living cost.

They also introduce a version that keeps a replay buffer of experience and uses an off-policy algorithm to learn from the replay buffer in order to improve sample efficiency.

The key insight that rewards are _not_ invariant to additions of a constant when you have variable-length episodes is useful and I'm glad that it's been pointed out, and a solution proposed. However, the experiments are really strange -- in one case (Figure 4, HalfCheetah) their algorithm outperforms the expert (which has access to the true reward), and in another (Figure 5, right) the blue line implies that using a uniformly zero reward lets you achieve around a third of expert performance (!!).


Risk-Sensitive Generative Adversarial Imitation Learning

Jonathan Lacotte
Mohammad Ghavamzadeh
Yinlam Chow
Marco Pavone

This paper extends GAIL to perform imitation learning where we try to optimize a policy for the mean reward collected under the constraint that the policy is no more risky than the expert policy. Since we don't know the true cost function, we have to approximate this problem with another problem where we infer the cost function as well, and evaluate the risk profile relative to the inferred cost function. The algorithm ends up looking very similar to the original GAIL algorithm, where the gradient updates change in order to include terms dependent on the conditional value-at-risk (CVaR). They evaluate against GAIL and RAIL (another risk-sensitive imitation learning algorithm) and find that their method performs the best on the Hopper and Walker Mujoco environments.

I only skimmed through the math, so I don't understand the paper well enough to have a good opinion on it. The overall objective of having more risk-sensitivity seems useful for safety. That said, I do find the VNM utility theorem compelling, and it suggests that risk aversion is a bad strategy. I currently resolve this by saying that while the VNM theorem is true, if you want to optimize expected reward over a long time horizon in an environment with high-downside actions but not high-upside actions, even if you are maximizing expected utility you would not take low-probability-of-high-downside actions. (Here a high-downside action is one that causes something like death/episode termination.) Since humans are (probably) scope-insensitive with respect to time, it becomes important for humans to have a heuristic of risk aversion in order to actually maximize expected utility in practice. I'd be interested in seeing experiments with current (risk neutral) RL algorithms in long-horizon environments with actions with high downside, and see if they automatically learn behavior that we would call "risk-averse".

Take this with a grain of salt -- it's a lot more speculative than most of my opinions, which can already be quite speculative. Most of the steps in that argument are handwavy intuitions I have that aren't based on any research that's been done (though I haven't looked for any such research). Though you can think of the argument for focusing on long-term AI safety at all as an instance of this idea, where the argument is that our risk-aversion heuristic is only sufficient for timescales on the orders of human lifetimes, not for cosmic timescales, and so we should explicitly be more risk-averse and focus on reducing existential risk.


Inverse Decision Modeling: Learning Interpretable Representations of Behavior

Daniel Jarrett*
Alihan Hüyük*
Mihaela van der Schaar

There’s lots of work on learning preferences from demonstrations, which varies in how much structure they assume on the demonstrator: for example, we might consider them to be <@Boltzmann rational@>(@Modeling Interaction via the Principle of Maximum Causal Entropy@) or [risk sensitive](https://rss2017.lids.mit.edu/static/papers/62.pdf), or we could try to <@learn their biases@>(@Learning biases and rewards simultaneously@). This paper proposes a framework to encompass all of these choices: the core idea is to model the demonstrator as choosing actions according to a _planner_; some parameters of this planner are fixed in advance to provide an assumption on the structure of the planner, while others are learned from data. This also allows them to separate beliefs, decision-making, and rewards, so that different structures can be imposed on each of them individually.

The paper provides a mathematical treatment of both the forward problem (how to compute actions in the planner given the reward, think of algorithms like value iteration) and the backward problem (how to compute the reward given demonstrations, the typical inverse reinforcement learning setting). They demonstrate the framework on a medical dataset, where they introduce a planner with parameters for flexibility of decision-making, optimism of beliefs, and adaptivity of beliefs. In this case they specify the desired reward function and then run backward inference to conclude that, with respect to this reward function, clinicians appear to be significantly less optimistic when diagnosing dementia in female and elderly patients.

One thing to note about this paper is that it is an incredible work of scholarship; it fluently cites research across a variety of disciplines, including AI safety, and provides a useful organizing framework for many such papers. If you need to do a literature review on inverse reinforcement learning, this paper is a good place to start.


B-Pref: Benchmarking Preference-Based Reinforcement Learning

Kimin Lee
Laura Smith
Anca Dragan
Pieter Abbeel

Deep RL has become a powerful method to solve a variety of sequential decision tasks using a known reward function for training. However, in practice, rewards are hard to specify making it hard to scale Deep RL for many applications. Preference-based RL provides an alternative by allowing a teacher to indicate preferences between a pair of behaviors. Because the teacher can interactively give feedback to an agent, preference-based RL has the potential to help address this limitation of Deep RL. Despite the advantages of preference-based RL it has proven difficult to design useful benchmarks for the problem. This paper introduces a benchmark (B-Pref) that is useful for preference-based RL in various locomotion and robotic manipulation tasks.

One difficulty with designing a useful benchmark is that teachers may have a variety of irrationalities. For example, teachers might be myopic or make mistakes. The B-Pref benchmark addresses this by emphasizing measuring performance under a variety of teacher irrationalities. They do this by providing various performance metrics to introduce irrationality into otherwise deterministic reward criteria. While previous approaches to preference-based RL work well when the teacher responses are consistent, experiments show they are not robust to feedback noise or teacher mistakes. Experiments also show that how queries are selected has a major impact on performance. With these results, the authors identify these two problems as areas for future work.

NeurIPS 2021 Track Datasets and Benchmarks

While the authors do a good job advocating for the problem of preference-based RL, I'm less convinced their particular benchmark is a large step forward. In particular, it seems the main contribution is not a suite of tasks, but rather a collection of different ways to add irrationality to the teacher oracle. The main takeaway of this paper is that current algorithms don't seem to perform well when the teacher can make mistakes, but this is quite similar to having a misspecified reward function. Beyond that criticism, the experiments support the areas suggested for future work.


VILD: Variational Imitation Learning with Diverse-quality Demonstrations

Voot Tangkaratt
Bo Han
Mohammad Emtiyaz Khan
Masashi Sugiyama

We saw in the previous summary that existing methods struggle to cope with datasets of demonstrations of mixed quality. This paper aims to tackle exactly this problem. They consider a model in which there are k demonstrators with varying levels of quality. Each demonstrator is modeled as computing an action Boltzmann-rationally and then applying some Gaussian noise; the standard deviation of the Gaussian noise differs across the demonstrators (with higher standard deviation corresponding to lower quality).

They use variational inference to derive an algorithm for this problem that infers the reward function as well as an optimal policy to go along with it. In addition, they oversample data from the demonstrations that the model thinks are high quality in order to get more informative gradients. (They use an importance sampling correction in order to keep the gradient estimate unbiased.)

Their experiments on machine-generated data show significant improvement over existing imitation learning algorithms, both in the case where we synthetically add Gaussian noise (matching the model) and when we add time-signal-dependent (TSD) noise (in which case the model is misspecified).

This seems like a reasonable approach. It has a similar ethos as Boltzmann rationality. In Boltzmann rationality, it seems like all you need to do is model the demonstrator as having some noise but still being more likely to choose higher-reward actions, and that’s enough to get decent performance; similarly here you just need to model different demonstrators as applying different amounts of Gaussian noise to the optimal policy and that’s enough to distinguish good from bad.

Note that, while the experimental results are good, the paper doesn’t have experiments with real human demonstrations; as we saw in the previous summary these can often be quite different (in ways that matter) from machine-generated demonstrations.


Reward Identification in Inverse Reinforcement Learning

Kuno Kim
Kirankumar Shiragur
Shivam Garg
Stefano Ermon

As mentioned in the previous summary, a major challenge with inverse reinforcement learning is that rewards are unidentifiable: even given perfect knowledge of the policy, we cannot recover the reward function that produces it. This is partly for boring reasons like “you can add a constant to a reward function without changing anything”, but even if you exclude those kinds of reasons, others remain. For example, since every policy is optimal for the constant reward function, the zero reward function can rationalize any policy.

For this reason, the authors instead focus on the case where we assume the policy is a solution to the maximum entropy RL objective (you can think of this as Boltzmann rationality, if you’re more familiar with that). The solution to MaxEnt RL for a zero reward is a uniformly random policy, so the zero reward no longer rationalizes every policy. Perhaps rewards are identifiable in this case?

(You might have noticed that I neglected the question of whether the MaxEnt RL model was better than the regular RL model in cases that we care about. As far as I can tell the paper doesn’t address this. But if they did so, perhaps they might say that in realistic situations we are dealing with boundedly-rational agents, and Boltzmann rationality / MaxEnt RL is a common model in such situations.)

Well, we still need to deal with the “additive constant” argument. To address this, the authors define two reward functions to be equivalent if they agree up to an additive constant. There are actually two versions of this: “trajectory equivalence” means that they agree on the rewards for all feasible trajectories, while “state-action equivalence” means that they agree on the rewards for all state-action pairs. Correspondingly, “weak identifiability” means that you can identify rewards up to trajectory equivalence, while “strong identifiability” means you can identify them up to state-action equivalence. Strong identifiability implies weak identifiability, since if you know the rewards on state-action pairs, that determines the reward for any given trajectory.

All deterministic MDPs are weakly identifiable under the MaxEnt RL model, since in this case a trajectory τ is selected with probability p(τ) proportional to exp(r(τ)), so the probability p(τ) can then be inverted to get r(τ). However, stochastic MDPs need not be weakly identifiable. Imagine an MDP in which no matter what you do, you are teleported to a random state. In such an MDP, the agent has no control over the trajectory, and so the MaxEnt RL objective will choose a uniformly random policy, no matter what the reward is, and so the reward must be unidentifiable.

Now the question is, assuming you have weak identifiability (i.e. you can infer r(τ)), when do you also have strong identifiability (i.e. you can infer r(s, a))? Intuitively, there needs to be a sufficient “diversity” of feasible trajectories τ, that cover a wide variety of possible (s, a) pairs, so that you can use the r(τ) values to infer the r(s, a) values. The authors prove a sufficient condition called “coverage”: there exists some timestep T, such that for every state there is some feasible trajectory that reaches that state at timestep T. (They also require the horizon to be at least 2T.) Coverage can be a fairly easy property to have; for example, if you can get to any state from any other state in some number of steps, then all you need is a single self-loop somewhere in the MDP that allows you to “waste time” so that you reach the desired state at exactly timestep T (instead of reaching too early).

[Identifiability in inverse reinforcement learning](https://arxiv.org/abs/2106.03498) has the same motivation and studies a very similar setting, but has a few different results. It's also easier to read if you're not as familiar with MaxEnt methods.


Exploring Hierarchy-Aware Inverse Reinforcement Learning

One heuristic that humans use to deal with bounded computation is to make plans hierarchically, building long-term plans out of slightly smaller building blocks. How can we incorporate this knowledge into an IRL algorithm? This paper extends [Bayesian IRL](https://www.aaai.org/Papers/IJCAI/2007/IJCAI07-416.pdf) to the setting where the demonstrator has access to a set of _options_, which are (to a first approximation) policies that can be used to achieve some subgoal. Now, when you are given a trajectory of states and actions, it is no longer clear which options the demonstrator was using to generate that trajectory. The authors provide an algorithm that can enumerate all the options that are consistent with the trajectory, and assign probabilities to them according to the Boltzmann-rational model. They evaluate on a taxi driver gridworld often used in hierarchical planning, as well as on real human data from a game called Wikispeedia.

Hierarchy seems to be a very important tool that humans use, so I'm glad to see work on it. Currently, the algorithm is very computationally expensive, and can only be applied in small domains right now, and requires the options to be specified ahead of time, but it does lead to a benefit on the environments they consider, despite the inevitable misspecification from having to hardcode the options. I would be very interested to see an extension to high-dimensional data where the options are learned (analogous to [Meta-Learning Shared Hierarchies](https://blog.openai.com/learning-a-hierarchy/) for hierarchical RL). Not only would this be more realistic, it could perform better because the options would be learned, not hardcoded.


IBM researchers train AI to follow code of ethics

Parents want movie recommendation systems not to recommend particular kinds of movies to children, but we would also like the recommendation system to suggest movies that the children will actually like. Researchers solved this problem by first learning a model for what kinds of movies should not be recommended, and then combined that with a contextual bandit model that learns online from the child's data to provide good suggestions that follow the parent's constraints.

We can look at this from an alignment perspective -- the child is giving the AI system a misspecified reward, relative to the parent's goal of "provide good suggestions that do not have inappropriate content". While the researchers solve it using contextual bandits, it could be interesting to consider how AI alignment approaches could deal with this situation.

Using Contextual Bandits with Behavioral Constraints for Constrained Online Movie Recommendation


David Lindner
Rohin Shah
Pieter Abbeel
Anca Dragan

Since the state of the world has already been optimized for human preferences, it can be used to infer those preferences. For example, it isn’t a coincidence that vases tend to be intact and on tables. An agent with an understanding of physics can observe that humans haven’t yet broken a particular vase, and infer that they care about vases not being broken.

<@Previous work@>(@Learning Preferences by Looking at the World@) provides an algorithm, RLSP, that can perform this type of reasoning, but it is limited to small environments with known dynamics and features. In this paper (on which I am an author), we introduce a deep variant of the algorithm, called Deep RLSP, to move past these limitations. While RLSP assumes known features, Deep RLSP learns a feature function using self-supervised learning. While RLSP computes statistics for all possible past trajectories using dynamic programming, deep RLSP learns an inverse dynamics model and inverse policy to _simulate_ the most likely past trajectories, which serve as a good approximation for the necessary statistics.

We evaluate the resulting algorithm on a variety of Mujoco tasks, with promising results. For example, given a single state of a HalfCheetah balancing on one leg, Deep RLSP is able to learn a (noisy) policy that somewhat mimics this balancing behavior. (These results can be seen [here](https://sites.google.com/view/deep-rlsp).)

[Paper: Learning What To Do by Simulating the Past](https://arxiv.org/abs/2104.03946)

[Thesis: Extracting and Using Preference Information from the State of the World](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2020/EECS-2020-210.html)


Recursive Classification: Replacing Rewards with Examples in RL

Benjamin Eysenbach
Sergey Levine
Ruslan Salakhutdinov

Previous work has suggested learning a reward model from examples of successfully solving the task. This paper suggests that rather than a two stage process of learning a reward model and then optimizing it using RL, we can instead directly learn a policy from the examples by building an equivalent of Bellman backups that apply directly to examples (rather than having to go through intermediate rewards). Their experiments show that this works well.

Paper: Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification


Four Motivations for Learning Normativity

We’ve <@previously seen@>(@Learning Normativity: A Research Agenda@) desiderata for agents that learn normativity from humans: specifically, we would like such agents to:

1. **Learn at all levels:** We don’t just learn about uncertain values, we also learn how to learn values, and how to learn to learn values, etc. There is **no perfect loss function** that works at any level; we assume conservatively that Goodhart’s Law will always apply. In order to not have to give infinite feedback for the infinite levels, we need to **share feedback between levels**.
2. **Learn to interpret feedback:** Similarly, we conservatively assume that there is **no perfect feedback**; so rather than fixing a model for how to interpret feedback, we want feedback to be **uncertain** and **reinterpretable**.
3. **Process-level feedback:** Rather than having to justify all feedback in terms of the consequences of the agent’s actions, we should also be able to provide feedback on the way the agent is reasoning. Sometimes we’ll have to judge the entire chain of reasoning with **whole-process feedback**.

This post notes that we can motivate these desiderata from multiple different frames:

1. _Outer alignment:_ The core problem of outer alignment is that any specified objective tends to be wrong. This applies at all levels, suggesting that we need to **learn at all levels**, and also **learn to interpret feedback** for the same reason. **Process-level feedback** is then needed because not all decisions can be justified based on consequences of actions.
2. _Recovering from human error:_ Another view that we can take is that humans don’t always give the right feedback, and so we need to be robust to this. This motivates all the desiderata in the same way as for outer alignment.
3. _Process-level feedback:_ We can instead view process-level feedback as central, since having agents doing the right type of _reasoning_ (not just getting good outcomes) is crucial for inner alignment. In order to have something general (rather than identifying cases of bad reasoning one at a time), we could imagine learning a classifier that detects whether reasoning is good or not. However, then we don’t know whether the reasoning of the classifier is good or not. Once again, it seems we would like to **learn at all levels**.
4. _Generalizing learning theory:_ In learning theory, we have a distribution over a set of hypotheses, which we update based on how well the hypotheses predict observations. **Process-level feedback** would allow us to provide feedback on an individual hypothesis, and this feedback could be **uncertain**. **Reinterpretable feedback** on the other hand can be thought of as part of a (future) theory of meta-learning.


Learning Montezuma’s Revenge from a Single Demonstration

Montezuma's Revenge is widely considered to be one of the hardest Atari games to learn, because the reward is so sparse -- it takes many actions to reach the first positive reward, and if you're using random exploration, it will take exponentially many actions (in N, the number of actions till the first reward) to find any reward. A human demonstration should make the exploration problem much easier. In particular, we can start just before the end of the demonstration, and train the RL agent to get as much score as the demonstration. Once it learns that, we can start it at slightly earlier in the demonstration, and do it again. Repeating this, we eventually get an agent that can perform the whole demonstration from start to finish, and it takes time linear in the length of the demonstration. Note that the agent must be able to generalize a little bit to states "around" the human demonstration -- when it takes random actions it will eventually reach a state that is similar to a state it saw earlier, but not exactly the same, and it needs to generalize properly. It turns out that this works for Montezuma's Revenge, but not for other Atari games like Gravitar and Pitfall.

Here, the task definition continues to be the reward function, and the human demonstration is used to help the agent effectively optimize the reward function. Such agents are still vulnerable to misspecified reward functions -- in fact, the agent discovers a bug in the emulator that wouldn't have happened if it was trying to imitate the human. I would still expect the agent to be more human-like than one trained with standard RL, since it only learns the environment near the human policy.


Learning Rewards from Linguistic Feedback

Theodore R. Sumers
Mark K. Ho
Robert D. Hawkins
Karthik Narasimhan
Thomas L. Griffiths

This paper proposes another approach to reinforcement learning using natural language. After the agent plays an episode, we can ask a human for feedback in natural language. We then take their response, figure out what features of the environment the response mentions, and then use sentiment analysis to determine how to update the weights on the features. For sentiment analysis we can use an off-the-shelf classifier; the hard part is in determining the relevant environment feature vectors:

1. **Evaluative** feedback is feedback about the trajectory the agent produced, for example “good job”, so we can just use the features of this trajectory.
2. **Imperative** feedback specifies what the agent should have done, e.g. “you should have gone to the top right corner”. In this case, we must find the features consistent with the given instruction.
3. **Descriptive** feedback provides feedback directly about the reward, for example “yellow objects are bad”. In this case, we use a feature vector that has a 1 for every feature mentioned (in this case, the feature for yellow objects) and 0 everywhere else.

Types 2 and 3 require some domain knowledge in order to write down programs that map language to the relevant features. The environment the authors used was simple enough that they were able to do this.

Once we have the feature vector f and the sentiment s, we perform a Bayesian update on our weight distribution. This is similar to the way we perform Bayesian updates on the reward distribution upon seeing a human action as evidence, as in <@Bayesian IRL@>(@Bayesian Inverse Reinforcement Learning@) or <@reward-rational implicit choice@>(@Reward-rational (implicit) choice: A unifying formalism for reward learning@).

This model so far performs reasonably well. By adding a couple of heuristics inspired by pragmatics (e.g. assuming that features that aren’t mentioned aren’t decision-relevant), they reach approximately human-level performance.


Non-Adversarial Imitation Learning and its Connections to Adversarial Methods

Viewing imitation learning as a distribution matching problem has become more popular in recent years (see <@Value-Dice@>(@Imitation Learning via Off-Policy Distribution Matching@) / <@I2L@>(@State-only Imitation with Transition Dynamics Mismatch@)). However, the authors in this paper argue that such methods are unstable due to their formulation as saddle-point problems which means they have weak convergence guarantees due to the assumption that the policy is slowly updated. In this paper, the authors reformulate <@Adversarial IRL@>(@Learning Robust Rewards with Adversarial Inverse Reinforcement Learning@) as a non-adversarial problem allowing for much stronger convergence guarantees to be proved. In particular, the authors derive a lower-bound on the discrimination reward which allows for larger policy updates and then introduce a method to iteratively tighten this bound. They also build on prior work for value-dice and derive a soft actor-critic algorithm (ONAIL) that they evaluate on a variety of control tasks.

The experiments in this paper are a bit underwhelming. While they run a large number of experiments, ONAIL only occasionally outperforms value-dice consistently in the HalfCheetah environment. The authors justify this by noting that ONAIL wasn't regularized. Additionally, the policies are initialized with behavior cloning, something that value-dice doesn't require. However, the theoretical insight on iterative tightening is interesting, and together with the recent work on value-dice indicates that the design space of imitation learning algorithms is far from being exhausted.


An Efficient, Generalized Bellman Update For Cooperative Inverse Reinforcement Learning

Dhruv Malik
Malayandi Palaniappan
Jaime F. Fisac
Dylan Hadfield-Menell
Stuart Russell
Anca D. Dragan

Previously, Cooperative Inverse Reinforcement Learning (CIRL) games were solved by reducing them to a POMDP with an exponentially-sized action space, and then solving with POMDP algorithms that are exponential in the size of the action space, leading to a doubly-exponential algorithm. This paper leverages the fact that the human has perfect information to create a modified Bellman update that still computes the optimal policy, but no longer requires an exponential action space. The modified Bellman update works with the human's policy, and so we can now swap in more accurate models of the human, including eg. noisy rationality (whereas previously the human had to be exactly optimal). They show huge speedups in experiments, and discuss some interesting qualitative behavior that arises out of CIRL games -- for example, sometimes the human _waits_ instead of making progress on the task, because it is a good signal to the robot of what the human wants.

I'm excited by this improvement, since now we can actually solve non-trivial CIRL games -- one of the games they solve has around 10 billion states. With this we can run experiments with real humans, which seems really important, and the paper does mention a very preliminary pilot study run with real humans.


Learning a Prior over Intent via Meta-Inverse Reinforcement Learning

Kelvin Xu
Ellis Ratner
Anca Dragan
Sergey Levine
Chelsea Finn

For complex rewards, such as reward functions defined on pixels, standard IRL methods require a large number of demonstrations. However, many tasks are very related, and so we should be able to leverage demonstrations from one task to learn rewards for other tasks. This naturally suggests that we use meta learning. The authors adapt [MAML](https://arxiv.org/abs/1703.03400) to work with maximum entropy IRL (which requires differentiating through the MaxEnt IRL gradient). They evaluate their approach, called MandRIL, on a navigation task whose underlying structure is a gridworld, but the state is represented as an image so that the reward function is nonlinear and requires a convnet.

In one of the experiments, the baseline of running IRL from scratch performed second best, beating out two other methods of meta-learning. I'd guess that this is because both MandRIL and standard IRL benefit from assuming the maxent IRL distribution over trajectories (which I believe is how the demonstrations were synthetically generated), whereas the other two meta learning baselines do not have any such assumption, and must learn this relationship.


Imitation Learning from Video by Leveraging Proprioception

Faraz Torabi
Garrett Warnell
Peter Stone

Recent work into imitation learning from observation (IfO) allows agents to perform a task from visual demonstrations that do not include state and action information. In this paper the authors are interested in leveraging proprioception information, knowledge of internal states, to create an efficient IfO algorithm. As opposed to GAIfO, which typically uses only the observation vector, this algorithm only allows images to be used for discrimination but lets the agent make use of internal states to generate actions. They test their proposed technique on several MujoCo domains and show that it outperforms other imitation from observation algorithms. The authors note that in practice occlusion and fast movement in environments like Walker2d and HalfCheetah make it difficult to learn directly from images which partly explains the success of using proprioceptive features.

I think it's easy to forget that observations aren't necessarily equivalent to state representations. This paper did a good job of reminding me that using state features on the MujoCo tasks is different from using images to train imitation learning agents. In practice, trying to learn just from images can fail because of partial observability, but introducing proprioception is a natural solution here. I broadly agree with the authors' conclusion that resolving embodiment mismatch and viewpoint mismatch are natural next steps for this kind of research.


Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement

Chao Yang*
Xiaojian Ma*
Wenbing Huang
Fuchun Sun
Huaping Liu
Junzhou Huang
Chuang Gan

Learning from observation (LfO) focuses on imitation learning in situations where we want to learn from state-only demonstrations. This contrasts with learning from demonstration (LfD) which needs both state and action information. In practice, LfO is the more common situation due to the prevalence of unannotated data, such as video. In this paper, the authors show that the gap between LfO and LfD comes from the disagreement of inverse dynamics models between the imitator and the expert. If the inverse dynamics model is perfect, then state transitions can be labeled with actions and LfD can be performed on the result. However, it's often the case that many actions can generate the same state transition. They then show that optimizing an upper-bound on this gap leads to improved performance as compared to other LfO methods such as GAIfO (GAIL extended to LfO).

The main value of this paper is that the difference between LfO and LfD is clarified by introducing the notion of inverse disagreement. Related to this analysis, the authors note that GAIfO has the same objective as the inverse disagreement model if we replace KL with JS divergence. This makes me suspect that there's a general LfO [divergence minimization perspective](https://arxiv.org/abs/1911.02256) relating all of these methods together. In other words, the fact that the objectives for LfO and LfD can be related via KL/JS divergence indicates that there is an entire class of methods underlying this approach to LfO. Specifically, I'd hypothesize that regularized inverse reinforcement learning from observation followed by reinforcement learning would be equivalent to a divergence minimization problem.

[GAIfO](https://arxiv.org/abs/1807.06158) and [Recent Advances in LfO](https://arxiv.org/pdf/1905.13566.pdf)


On the Foundations of Expected Expected Utility

Suppose you want to understand someone's preferences over a set of outcomes, and lotteries over those outcomes. The VNM theorem shows that under certain reasonable assumptions about those preferences, we can represent the preferences using a utility function over outcomes (_not_ lotteries), such that the person prefers lottery A to lottery B if and only if the expected utility of lottery A is greater than the expected utility of lottery B. Furthermore, the utility function is unique up to an affine transformation.

What if we want to have uncertainty over the utility function? The natural approach would be to maintain a probability distribution over utility functions, and then say that the person prefers lottery A to lottery B if and only if the expected expected utility of lottery A exceeds the expected expected utility of lottery B. Here, the first expectation is taken over the probability distribution over utility functions, and the second expectation is taken over the probabilities in the lottery. This decision rule is called Maximum Expected Expected Utility (MEEU).

However, the MEEU decision rule gives different answers if you transform one of the utility functions in an affine way. This isn't great, since affine transformations of utility functions encode the same preferences, and so you'd like your decision rule to be invariant to them. This is the standard problem that utility functions are not by default _commensurable_.

This paper shows that everything is fine if you assume that all the utility functions agree on the best and worst outcomes. In this case, any utility function can be uniquely characterized by stating for each state s, for what p are you indifferent between s and {p: s_best, (1-p): s_worst}. We can then define the base decision rule as follows: when presented with a lottery A = {p_i : s_i, ...}, you convert it into a lottery over utility functions: {q_U : A_U, ...}, where for each utility function, you define A_U as A, but replacing all the s_i with the corresponding {p: s_best, (1-p): s_worst} lotteries as defined by U. So now, we have a compound lottery of the form {q_U : {p_i : {p: s_best, (1-p): s_worst}, ...}, ...}, which can be reduced to a simple lottery of the form {x: s_best, (1-x): s_worst}. When comparing two such lotteries, you prefer lottery A to lottery B if and only if x_A > x_B.

So that's what you do without using utility functions, and only using preference orderings. They show that the MEEU decision rule is identical to this base decision rule, justifying the use of MEEU, as long as you normalize all of your utility functions so that they all assign the same value to the best and worst outcomes.


Sample Efficient Reinforcement Learning through Learning from Demonstrations in Minecraft

Christian Scheller
Yanick Schraner
Manfred Vogel

This paper explains the technique used by the 3rd place team in the MineRL competition (summarized above). They used behavior cloning to train their neural net on human demonstrations, and then used reinforcement learning (specifically, IMPALA) with experience replay and advantage clipping to improve. There are more details about their architecture and design choices in the paper.


From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following

Justin Fu
Anoop Korattikara
Sergey Levine
Sergio Guadarrama

Rewards and language commands are more generalizable than policies: "pick up the vase" would make sense in any house, but the actions that navigate to and pick up a vase in one house would not work in another house. Based on this observation, this paper proposes that we have a dataset where for several (language command, environment) pairs, we are given expert demonstrations of how to follow the command in that environment. For each data point, we can use IRL to infer a reward function, and use that to train a neural net that can map from the language command to the reward function. Then, at test time, given a language command, we can convert it to a reward function, after which we can use standard deep RL techniques to get a policy that executes the command.

The authors evaluate on a 3D house domain with pixel observations, and two types of language commands: navigation and pick-and-place. During training, when IRL needs to be done, since deep IRL algorithms are computationally expensive they convert the task into a small, tabular MDP with known dynamics for which they can solve the IRL problem exactly, deriving a gradient that can then be applied in the observation space to train a neural net that given image observations and a language command predicts the reward. Note that this only needs to be done at training time: at test time, the reward function can be used in a new environment with unknown dynamics and image observations. They show that the learned rewards generalize to novel combinations of objects within a house, as well as to entirely new houses (though to a lesser extent).

I think the success at generalization comes primarily because of the MaxEnt IRL during training: it provides a lot of structure and inductive bias that means that the rewards on which the reward predictor is trained are "close" to the intended reward function. For example, in the navigation tasks, the demonstrations for a command like "go to the vase" will involve trajectories through the state of many houses that end up in the vase. For each demonstration, MaxEnt IRL "assigns" positive reward to the states in the demonstration, and negative reward to everything else. However, once you average across demonstrations in different houses, the state with the vase gets a huge amount of positive reward (since it is in all trajectories) while all the other states are relatively neutral (since they will only be in a few trajectories, where the agent needed to pass that point in order to get to the vase). So when this is "transferred" to the neural net via gradients, the neural net is basically "told" that high reward only happens in states that contain vases, which is a strong constraint on the learned reward. On the other hand, for "move X to Y" tasks, while the same argument suggests that you will have high reward on reaching X, the rest of the trajectories from X to Y will all be the same and MaxEnt IRL will learn a shaped reward that doesn't transfer well. So, I predict that on "move X to Y" tasks at test time, the agent will successfully pick up the object X, but fail to then move it to Y.

To be clear, this is not meant as a critique of the paper: indeed, I think when you want out-of-distribution generalization, you _have_ to do it by imposing structure/inductive bias, and this is a new way to do it that I hadn't seen before.


Risk-Aware Active Inverse Reinforcement Learning

This paper presents an algorithm that actively solicits demonstrations on states where it could potentially behave badly due to its uncertainty about the reward function. They use Bayesian IRL as their IRL algorithm, so that they get a distribution over reward functions. They use the most likely reward to train a policy, and then find a state from which that policy has high risk (because of the uncertainty over reward functions). They show in experiments that this performs better than other active IRL algorithms.

I don't fully understand this paper -- how exactly are they searching over states, when there are exponentially many of them? Are they sampling them somehow? It's definitely possible that this is in the paper and I missed it, I did skim it fairly quickly.


Adversarial Imitation via Variational Inverse Reinforcement Learning

Ahmed H. Qureshi
Byron Boots
Michael C. Yip

A short history of deep IRL algorithms: [GAIL](https://arxiv.org/abs/1606.03476) introduced the idea of training a policy that fools a discriminator that tries to distinguish a policy from expert demonstrations, [GAN-GCL](https://arxiv.org/abs/1611.03852) showed how to recover a reward function from the discriminator, and [AIRL](https://arxiv.org/abs/1710.11248) ([AN #17](https://mailchi.mp/ad852629e45a/alignment-newsletter-17)) trains on (s, a, s') tuples instead of trajectories to reduce variance, and learns a reward shaping term separately so that it transfers better to new environments. This paper proposed that the reward shaping term be the _empowerment_ of a state. The empowerment of a state is the maximum mutual information between a sequence of actions from a state, and the achieved next state. Intuitively, this would lead to choosing to go to states from which you can reach the most possible future states. Their evaluation shows that they do about as well as AIRL in learning to imitate an expert, but perform much better in transfer tasks (where the learned reward function must generalize to a new environment).

I'm confused by this paper, because they only compute the empowerment for a _single action_. I would expect that in most states, different actions lead to different next states, which suggests that the empowerment will be the same for all states. Why then does it have any effect? And even if the empowerment was computed over longer action sequences, what is the reason that this leads to learning generalizable rewards? My normal model is that IRL algorithms don't learn generalizable rewards because they mostly use the reward to "memorize" the correct actions to take in any given state, rather than learning the underlying true reward. I don't see why empowerment would prevent this from happening. Yet, their experiments show quite large improvements, and don't seem particularly suited to empowerment.


Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization

Sreejith Balakrishnan
Quoc Phong Nguyen
Bryan Kian Hsiang Low
Harold Soh

In the description of Bayesian IRL above, Step 2 is a very expensive step, as it requires solving a full RL problem. Can we improve any of the other steps to reduce the amount of times we have to run step 2? This paper aims to improve step 1: rather than choosing the next reward _randomly_, we can choose one that we think will be most informative. The authors apply the framework of Bayesian optimization to put this into practice. I won’t explain it more here since the details are fairly technical and involved (and I didn’t read the paper closely enough to understand it myself). They did have to introduce a new kernel in order to handle the fact that reward functions are invariant to the addition of a potential function.


Explanation Augmented Feedback in Human-in-the-Loop Reinforcement Learning

Lin Guan*
Mudit Verma*
Subbarao Kambhampati

This paper starts from a similar position as the highlighted paper: that we can improve on algorithms by having humans provide different kinds of feedback that help with learning. They ask humans to provide “explanations” to improve sample efficiency in deep RL, which in this case means asking a human to segment parts of the image observation that are important (similar to a saliency map). They use this to define auxiliary losses that incentivize the agent to be invariant to augmentations of the irrelevant parts of the image. Their empirical evaluation shows improvements in sample efficiency relative to simple good/bad evaluative feedback.

Human in the Loop Learning Workshop at ICML 2020

The idea is cool, but the empirical results are not great. On Taxi, training with the reward signal and binary good/bad evaluative feedback takes 180k environment steps, and adding in explanations for a quarter of the steps brings it down to 130k environment steps. However, this seems like it would increase the human effort required by an order of magnitude or more, which seems way too high for the benefit provided.

It does seem to me that saliency explanations could contain a fair amount of information, and so you should be able to do better -- maybe a future algorithm will do so.


Learning a Behavioral Repertoire from Demonstrations

Niels Justesen*
Miguel Gonzalez Duque*
Daniel Cabarcas Jaramillo
Jean-Baptiste Mouret
Sebastian Risi

They extend vanilla Imitation Learning, adding a behaviour encoding as an input to the policy to prevent it from learning an 'average' behaviour, but instead to learn different strategies with a single policy. Training data includes 7,777 human demonstrations of Terran army build-orders v/s the Zerg in StarCraft 2, for which build-order strategies are first extracted in a high-dimension semantically-meaningful space, and then reduced to two dimensions using PCA. At training time, each game's 2D code _b_ is augmented to the state, and supervised learning is applied to the policy: π(s, b) = a, where _a_ is the action following state _s_ in the human demonstration.

This is a neat, straightforward extension to vanilla Imitation Learning that learns a single policy capable of exhibiting a diversity of behaviours. It offers yet another example of how to create and exploit clusters in some specification space; here the specification encodes desired build order rather than a particular task. However, their empirically-motivated choice of PCA (over t-SNE) for dimensionality reduction did not illuminate how best to cluster for behavioural diversity.

The evaluation also demonstrates how one can use the UCB1 algorithm to make better behaviour choices over a series of episodes. However, while the algorithm permits changing the behaviour _during_ an episode, it is unclear how to make such a choice or whether the agent will perform well under such a circumstance. The work also doesn't compare against previous approaches, e.g. these approaches from [2017](http://papers.nips.cc/paper/7116-robust-imitation-of-diverse-behaviors.pdf) and [2018](https://arxiv.org/pdf/1802.09564.pdf), making it difficult to determine the value of this approach without a deep-dive.


A Framework and Method for Online Inverse Reinforcement Learning

Saurabh Arora
Prashant Doshi
Bikramjit Banerjee

This paper introduces Incremental Inverse Reinforcement Learning (I2RL), where the agent continually gets new demonstrations from an expert, and has to update the estimate of the reward function in real time. The running example is a robot that has to navigate to a goal location without being seen by two guards that are patrolling. The robot needs to infer the rewards of the two guards in order to predict what they will do and plan around them. Since the guards are sometimes out of sight, we get demonstrations _with occlusion_, that is, some of the states in the demonstrations are hidden.

In the batch setting, this is solved with Latent Maximum Entropy IRL. To deal with occluded states Z, we define a probability distribution Pr(Z | Y, theta), where Y is the visible states and theta is the reward weights. Then, you can use expectation maximization to find theta -- in the expectation step, you compute feature expectations of the demonstrations (taking an expectation over hidden states Z), and in the maximization step, you compute the reward weights using the feature expectations as in standard maximum entropy IRL. The authors show how to extend this algorithm to the incremental setting where you only keep the reward weights, the feature expectations, and the number of past demonstrations as statistics. They show some convergence guarantees and evaluate on their running example of a robot that must evade guards.

IRL algorithms are often more computationally expensive than state-of-the-art RL algorithms, so I'm happy to see work that's trying to make it more realistic. That said, this paper focuses on settings where IRL is used to infer other agent's preferences so we can plan around them (as opposed to imitation learning) -- this setting seems not very important for AI alignment. I'm also very confused by the experiments -- it seems in Figure 2 that if you ignore previous optimization and initialize the reward with random weights, it does better. (It isn't ignoring all previous data, because it still has access to past feature expectations.) They don't comment on this in the paper, but my guess is that they ran more iterations of expectation maximization (which is why the learning duration is higher) and that's why they got better performance.


Syntax vs semantics: alarm better example than thermostat

This post gives a new example that more clearly illustrates the points made in a [previous post](https://www.alignmentforum.org/posts/EEPdbtvW8ei9Yi2e8/bridging-syntax-and-semantics-empirically) ([AN #26](https://mailchi.mp/1ecd1b775703/alignment-newsletter-26)).

Bridging syntax and semantics, empirically


Expert-augmented actor-critic for ViZDoom and Montezumas Revenge

Michał Garmulewicz
Henryk Michalewski
Piotr Miłoś

The authors augment ACKTR (a natural gradient RL algorithm) with an additional term in the loss function which depends on expert data. In particular, policies which choose different actions from samples of 14 expert trajectories are penalised, with a coefficient that depends on the expert's advantage over the critic's current expectations. This allows the agent to perform well on Montezuma's Revenge and a ViZDoom maze, sometimes beating the experts it was trained on. It also discovered a new bug in Montezuma's Revenge which increases its score by a factor of 40.

I'm not convinced that this paper's method of utilising expert data is an improvement on other approaches, such as [this paper](https://arxiv.org/abs/1805.11592) in which an agent learns to play Montezuma's revenge from watching a Youtube video. However, it does seem to learn faster than most others, probably due to using ACKTR. I'd also expect it to be overfitting to the expert trajectories, but can't determine the extent to which this is the case (the authors claim that their agent can continue gameplay into the second world of Montezuma's Revenge despite only having expert trajectories for the first world, but don't provide metrics of success in the second world).


Cycle-of-Learning for Autonomous Systems from Human Interaction

Nicholas R. Waytowich
Vinicius G. Goecks
Vernon J. Lawhern

We've developed many techniques for learning behaviors from humans in the last few years. This paper categorizes them as learning from demonstrations (think imitation learning and IRL), learning from intervention (think [Safe RL via Human Intervention](https://arxiv.org/abs/1707.05173)), and learning from evaluation (think [Deep RL from Human Preferences](https://arxiv.org/abs/1706.03741)). They propose running these techniques in sequence, followed by pure RL, to train a full system. Intuitively, demonstrations are used to jumpstart the learning, getting to near-human performance, and then intervention and evaluation based learning allow the system to safely improve beyond human-level, since it can learn behaviors that humans can't perform themselves but can recognize as good, and then RL is used to improve even more.

The general idea makes sense, but I wish they had actually implemented it and seen how it worked. (They do want to test in robotics in future work.) For example, they talk about inferring a reward with IRL from demonstrations, and then updating it during the intervention and evaluation stages. How are they planning to update it? Does the format of the reward function have to be the same in all stages, and will that affect how well each method works?

This feels like a single point in the space of possible designs, and doesn't include all of the techniques I'd be interested in. What about active methods, combined with exploration methods in RL? Perhaps you could start with a hand-specified reward function, get a prior using [inverse reward design](https://arxiv.org/abs/1711.02827), start optimizing it using RL with curiosity, and have a human either intervene when necessary (if you want safe exploration) or have the RL system actively query the human at certain states, where the human can respond with demonstrations or evaluations.


Directed Policy Gradient for Safe Reinforcement Learning with Human Advice

Hélène Plisnier
Denis Steckelmacher
Tim Brys
Diederik M. Roijers
Ann Nowé

One way that you could get advice from humans for RL would be to have the human provide a policy, which can be treated as a suggestion. In this paper, the authors propose to take such a policy, and incorporate it into a policy gradient algorithm by simply multiplying it with the policy chosen by the neural net to get a new policy that is in between the two. You can then run any on-policy RL algorithms using that policy.

European Workshop on Reinforcement Learning 2018

I'm annoyed at some claims that this paper makes. First, they say that the algorithm can ignore wrong advice that the human gives, but in the deterministic case, it does not ignore the advice, it just learns that if it gets into situations where it has to follow the advice bad things happen, and so it avoids getting into such situations. (The stochastic case is a bit better, in that at convergence the agent will ignore the advice, but it will take much longer to converge, if at all.) Second, their experiment involves a gridworld with 5 macro-actions, and they call this a "complicated environment with sparse rewards" -- yet if you had a uniformly random policy, in expectation it would take 5^3 = 125 episodes before you found the optimal trajectory, which would then be strongly reinforced getting quick convergence.

I do like the idea of providing advice by shaping the policy towards parts of the space that are better -- this would lead to better sample efficiency and safer exploration. I'd be pretty excited to see a paper that ran with this idea and had a more compelling story for how to get the advice policy from a human (specifying a policy is hard!) and better experiments that test the feasibility of the idea in a more complex environment.


Beyond Winning and Losing: Modeling Human Motivations and Behaviors Using Inverse Reinforcement Learning

Baoxiang Wang
Tongfang Sun
Xianjun Sam Zheng

How could you perform IRL without access to a simulator, or a model of the dynamics of the game, or the full human policy (only a set of demonstrations)? In this setting, as long as you have a large dataset of diverse human behavior, you can use Q-learning on the demonstrations to estimate separate Q-function for each feature, and then for a given set of demonstrations you can infer the reward for that set of demonstrations using a linear program that attempts to make all of the human actions optimal given the reward function. They define (manually) five features for World of Warcraft Avatar History (WoWAH) that correspond to different motivations and kinds of human behavior (hence the title of the paper) and infer the weights for those rewards. It isn't really an evaluation because there's no ground truth.


Policy Approval

Argues that even if we had the true human utility function (assuming it exists), an AI that optimizes it would still not be aligned. It also sketches out an idea for learning policies instead of utility functions that gets around these issues.

I disagree with the post but most likely I don't understand it. My strawman of the post is that it is arguing for imitation learning instead of inverse reinforcement learning (which differ when the AI and human know different things), which seems wrong to me.


Reinforcement Learning Under Moral Uncertainty

Given that we don’t have a perfect ethical theory ready to load into an AI system, and we don’t seem poised to get one any time soon, it seems worth looking into approaches that can deal with _moral uncertainty_. Drawing on the literature on moral uncertainty in philosophy, the authors consider several methods by which multiple moral theories can be aggregated, such as averaging over the theories, making decisions through a voting system, and having the theories compete to control the agent’s overall actions. They implement several of these in RL agents, and test them on simple gridworld versions of various trolley problems. They find that all of the methods have advantages and disadvantages.

The central challenge here is that normalizing different moral theories so that they are comparable is <@difficult@>(@Research Agenda v0.9: Synthesising a human's preferences into a utility function@) (see Section 2.3). This issue plagues even computationally intractable idealizations like <@assistance games@>(@Cooperative Inverse Reinforcement Learning@) that can perform full Bayesian updating on different moral theories. I’d love to see better theoretical solutions for this challenge.


Tech firms move to put ethical guard rails around AI

A description of the ethics boards that tech companies are putting up.


How would you teach AI to be kind?

The EthicsNet Guardians Challenge is looking for suggestions on how to create a dataset that could be used to teach prosocial behavior. This is not aimed to answer difficult philosophical questions, but to teach an AI system general, simple prosocial behaviors, such as alerting someone who dropped their wallet but didn't notice. They have some ideas for how to achieve this, but are looking for more ideas before they actually start collecting a dataset.

One of the things I think about now is how to learn "common sense", and this seems very related (though not exactly the same). One of the hardest things to do with novel AI research is to collect a good dataset (if you don't have a simulator, anyway), so this seems like a great opportunity to get a good dataset for projects trying to tackle these sorts of issues, especially for somewhat fleshed out projects where you know what kind of dataset you'll need.


Fluid Annotation: An Exploratory Machine Learning–Powered Interface for Faster Image Annotation

This post describes a system that can be used to help humans label images to generate labels for segmentation. The post summarizes it well: "Fluid Annotation starts from the output of a strong semantic segmentation model, which a human annotator can modify through machine-assisted edit operations using a natural user interface. Our interface empowers annotators to choose what to correct and in which order, allowing them to effectively focus their efforts on what the machine does not already know."

I'm excited about techniques like this that allow us to scale up AI systems with less human effort, by focusing human effort on the aspects of the problem that AI cannot yet solve, while using existing AI systems to do the low-level work (generating a shortlist of potential segmentations, in this case). This is an example of the paradigm of using AI to help humans more effectively create better AI, which is one of the key ideas underlying iterated amplification. (Though iterated amplification focuses on how to use existing AI systems to allow the human to provide a training signal for tasks _that humans cannot perform or evaluate themselves_.)


Meta-Learning MCMC Proposals

Tongzhou Wang
Yi Wu
David A. Moore
and Stuart J. Russell

Probabilistic programming offers the potential to encode our domain knowledge about a particular area into structured probabilistic models. However, algorithms that answer queries about these models can be very slow, even if we are only looking for an approximate answer. MCMC algorithms are approximate algorithms that _propose_ values for variables, which are then either _accepted_ or _rejected_. They only work well if the proposer is able to suggest high probability values, which allows for quick convergence. So, in practice researchers use hand-tuned proposers with these algorithms. This paper suggests that we could instead use a neural net as a proposer. The net is trained to give good proposals on small sections of models (called _motifs_), with a range of possible parameter values that affect what should be proposed. Then, faced with a new model, the net is able to look at the motifs in the new model and propose good values for those motifs.

This is another example of how we can leverage the messy power of deep learning to improve more structured algorithms that have guarantees.


A Theory of Universal Learning

Olivier Bousquet
Steve Hanneke
Shay Moran
Ramon van Handel
Amir Yehudayoff

In machine learning, algorithms are presented with labeled examples of categories from a training dataset and the objective is to output a classifier that distinguishes categories on a validation dataset. The generalization ability of the classifier is usually measured by calculating the error rate of the classifications on the validation set. One popular way to display generalization capability as a function of training set size is to plot a learning curve. A learning curve is a function that outputs the performance of a learning algorithm as a function of the data distribution and training sample size. A faster decay rate for a learning curve indicates a better ability to generalize with fewer data.

**In this paper, the authors characterize the conditions for a learning algorithm to have learning curves with a certain decay rate.** A learning curve is produced from the decay rate according to the formula 1/rate. The authors show that there are only three universal rates: exponential, linear, and arbitrarily slow decay. Moreover, the authors show there are problem classes that can be learned quickly in each instance but are slow to learn in the worst-case. This stands in contrast to classical results which analyze only the worst-case performance of learning algorithms. This produces pessimistic bounds because the guarantee must hold for all possible data distributions. This is often stronger than what is necessary for practice. Thus, by looking at rates instead of the worst-case learning curve, the authors show that it is possible to learn more efficiently than what is predicted by classical theory.

This paper is mathematically sophisticated, but full of examples to illustrate the main points of the theory. More generally, work towards non-uniform bounds has become a popular topic recently as a result of classical generalization theory's inability to explain the success of deep learning and phenomena such as double-descent. These results could allow for progress in explaining the generalization capability of over-parameterized models, such as neural networks. Additionally, the theory presented here could lead to more efficient algorithms that take advantage of potential speedups over empirical risk minimization proved in the paper.


Introducing TensorFlow Probability

Tensorflow now also supports probabilistic programming.

Probabilistic programming is becoming more and more important in machine learning, and is in some sense a counterpart to deep learning -- it lets you have probability distributions over parameters (as opposed to the point estimates provided by neural nets), but inference is often intractable and must be performed approximately, and even then you are often limited to smaller models than with deep learning. It's interesting to have both of these provided by a single library -- hopefully we'll see applications that combine both approaches to get the best of both worlds. In particular, probabilistic programming feels more principled and amenable to theoretical analysis, which may make it easier to reason about safety.


Inner Alignment, Outer Alignment, and Proposals for Building Safe Advanced AI

This podcast covers a lot of topics, with special focus on <@Risks from Learned Optimization in Advanced Machine Learning Systems@> and <@An overview of 11 proposals for building safe advanced AI@>.

My summary is light on detail because many of the topics have been highlighted before in this newsletter, but if you aren’t familiar with them the podcast is a great resource for learning about them.


AXRP #4 - Risks from Learned Optimization

This podcast delves into a bunch of questions and thoughts around <@mesa optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@). Here are some of the points that stood out to me (to be clear, many of these have been covered in this newsletter before, but it seemed worth it to state them again):

- A model is a mesa optimizer if it is a _mechanistic_ optimizer, that is, it is executing an algorithm that performs search for some objective.
- We need to focus on mechanistic optimizers instead of things that behave as though they are optimizing for some goal, because those two categories can have very different generalization behavior, and we are primarily interested in how they will generalize.
- Humans do seem like mesa optimizers relative to evolution (though perhaps not a central example). In particular, it seems accurate to say that humans look at different possible strategies and select the ones which have good properties, and thus we are implementing a mechanistic search algorithm.
- To reason about whether machine learning will result in these mechanistic optimizers, we need to reason about the _inductive biases_ of machine learning. We mostly don’t yet know how likely they are.
- Evan expects that in powerful neural networks there will exist a combination of neurons that encode the objective, which we might be able to find with interpretability techniques.
- Even if training on a myopic base objective, we might expect the mesa objective to be non-myopic, as the non-myopic objective "pursue X" is simpler than the myopic objective "pursue X until time T".
- We can’t rely on generalization bounds to guarantee performance, since in practice there is always some distribution shift (which invalidates those bounds).
- Although it is usually phrased in the train/test paradigm, mesa optimization is still a concern in an online learning setup, since at every time we are interested in whether the model will generalize well to the next data point it sees.
- We will probably select for simple ML models (in the sense of short description length) but not for low inference time, such that mechanistic optimizers are more likely than models that use more space (the extreme version being lookup tables).
- If you want to avoid mesa optimizers entirely (rather than aligning them), you probably need to have a pretty major change from the current practice of AI, as with STEM AI and Microscope AI (explained <@here@>(@An overview of 11 proposals for building safe advanced AI@)).
- Even in a <@CAIS scenario@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@) where we have (say) a thousand models doing different tasks, each of those tasks will still likely be complex enough to lead to the models being mesa optimizers.
- There are lots of mesa objectives which would lead to deceptive alignment relative to corrigible or internalized alignment, and so we should expect deceptive alignment a priori.


Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning

Tianhe Yu*
Deirdre Quillen*
Zhanpeng He*
Ryan Julian
Karol Hausman
Chelsea Finn
Sergey Levine

"Meta-learning" or "learning to learn" refers to the problem of transferring insight and skills from one set of tasks to be able to quickly perform well on new tasks. For example, you might want an algorithm that trains on some set of platformer games to pick up general skills that it can use to quickly learn new platformer games.

This paper introduces a new benchmark, "Meta World", for evaluating meta-learning algorithms. The benchmark consists of 50 simulated robotic manipulation tasks that require a robot arm to do a combination of reaching, pushing and grasping. The benchmark tests the ability of algorithms to learn to do a single task well, learn one multi-task policy that trains and performs well on several tasks at once, and adapt to new tasks after training on a number of other tasks. The paper argues that unlike previous meta-learning evaluations, the task distribution in this benchmark is very broad while still having enough shared structure that meta-learning is possible.

The paper evaluates existing multi-task learning and meta-learning algorithms on this new benchmark. In meta-learning, it finds that different algorithms do better depending on how much training data they're given. In multi-task learning, it finds that the algorithm that performs best uses multiple "heads", or ends of neural networks, one for each task. It also finds that algorithms that are "off-policy"-- that estimate the value of actions other than the one that the network is currently planning to take-- perform better on multi-task learning than "on-policy" algorithms.

I really like the idea of having a standardized benchmark for evaluating meta-learning algorithms. There's a lot of room for improvement in performance on the benchmark tasks and it would be cool if this incentivized algorithm development. As with any benchmark, I worry that it is too narrow to capture all the nuances of potential algorithms; I wouldn't be surprised if some meta-learning algorithm performed poorly here but did well in some other domain.


Meta-Learning: A Survey

This taxonomy of meta-learning classifies approaches by the main type of meta-data they learn from:
1. Evaluations of other models on related tasks
2. Characterisations of the tasks at hand (and a similarity metric between them)
3. The structures and parameters of related models
Vanschoren explores a number of different approaches in each category.


Understanding meta-trained algorithms through a Bayesian lens

Vladimir Mikulik*
Grégoire Delétang*
Tom McGrath*
Tim Genewein*
Miljan Martic
Shane Legg
Pedro A. Ortega

The previous paper suggested that meta-learning can implement optimal reasoning processes in theory. Does it work in practice? This paper sets out to answer this question by studying some simple prediction and decision-making tasks.

For prediction, we consider agents that are trained on a family of distributions (e.g. Bernoulli distributions whose parameter is chosen from a Beta distribution) to predict the probability distribution after seeing a sample generated from it. For decision-making, we consider two-armed bandit problems (where again there is a distribution over the parameters of the problem). These problems were chosen because their optimal solutions can be calculated analytically.

The authors train neural nets with memory to perform well on these tasks (as discussed in the previous paper) and find that they do indeed behave optimally, achieving effectively the best possible performance. They then try to investigate whether they are implementing the same reasoning algorithm as the analytic Bayes-optimal solution. To do this, they see whether they can train a second neural net to map the hidden states (memory) of the agent to the states in the Bayes-optimal solution, and vice versa. (One way to think of this: can you simulate the Bayes-optimal algorithm using the observation encodings from the RNN, and vice versa?)

They find that they _can_ learn a good mapping from agent states to Bayes-optimal states, but _cannot_ learn a good mapping from Bayes-optimal states to agent states. It seems likely that the agent has states that encode more information than is necessary, and so the minimal information stored by the Bayes-optimal algorithm is insufficient to reconstruct the agent states.

I suspect that in these simple tasks the posterior distribution over the parameters θ maintained by the Bayes-optimal algorithm is a _minimal_ sufficient statistic, that is, _any_ optimal policy must have states that are sufficient to reconstruct the information stored by the Bayes-optimal algorithm. So it makes sense that, for an agent with optimal behavior, the agent’s states could be used to simulate the Bayes-optimal states. I don’t think this tells us that much about the algorithm the network is implementing.

Note that I am quite happy to see work investigating the sorts of reasoning processes that neural networks have learned. While I don’t think the specific results in this paper have told us that much, I’m excited to see this line of work scaled up to more complex tasks, where agents may not reach optimal behavior, or might do so by learning heuristics that _don’t_ encode all of the information that the Bayes-optimal algorithm would use.

Paper: Meta-trained agents implement Bayes-optimal agents


ML Writing Month May 2018

The author wrote up a summary of an ML paper every day in May, which have all been collected in this doc.

These summaries seem really good to me (probably higher quality than a typical summary that I write), but are often on topics I'm not an expert in (eg. GANs) so it's hard for me to evaluate. The one paper I knew well ([Inverse Reward Design](https://arxiv.org/abs/1711.02827)) had a good summary.


Unreproducible Research is Reproducible

Xavier Bouthillier
Cesar Laurent
Pascal Vincent

This paper argues that despite the growing popularity of sharing code, machine learning research has a problem with reproducibility. It makes the distinction between the reproducibility of **methods/results**, which can be achieved by fixing random seeds and sharing code, and the reproducibility of **findings/conclusions**, which requires that different experimental setups (or at least random seeds) lead to the same conclusion.

Several popular neural network architectures are trained on several image classification datasets several times with different random seeds determining the weight initialization and sampling of data. The relative rankings of the architectures with respect to the test accuracy are found to vary relevantly with the random seed for all data sets, as well as between data sets.

The authors then argue that while the reproducibility of methods can help with speeding up **exploratory research**, the reproducibility of findings is necessary for **empirical research** from which robust conclusions can be drawn.  They claim that exploratory research that is not based on robust findings can get inefficient, and so call for the machine learning community to do more empirical research.

I really like that this paper not just claims that there is a problem with reproducibility, but demonstrates this more rigorously using an experiment. More robust empirical findings seem quite important for getting to a better understanding of machine learning systems in the medium term. Since this understanding is especially important for safety relevant research, where exploratory research seems more problematic by default, I am excited for a push in that direction.


2021 AI Index Report

Daniel Zhang
Saurabh Mishra
Erik Brynjolfsson
John Etchemendy
Deep Ganguli
Barbara Grosz
Terah Lyons
James Manyika
Juan Carlos Niebles
Michael Sellitto
Yoav Shoham
Jack Clark
Raymond Perrault

The AI Index Report is a project to track and distill data related to artificial intelligence. One central theme the report focuses on is the effects of COVID on AI research direction. The report highlights significant increases in spending on drug development, 4.5 times that in 2019. The report also focuses a spotlight on the relative lack of AI ethics benchmarks. This could pose a significant problem as surveillance technologies become an increasingly mature technology. Beyond these broad themes, there's data on publication trends, politics, diversity, and more in the 222-page report. Additionally, a significant amount of data is publicly available or interactive.

This is well presented and you can glean a lot from looking at the introductory sections. If you choose to dive into a particular topic, charts and methodology are presented in a clear manner with nice hyperlinking to make navigation relatively painless. There is also an [interactive](https://aiindex.stanford.edu/vibrancy/) visualization that allows for cross-country comparison according to user-defined metrics. Once again, very well presented.


Explainable AI, Sparse Representations, and Signals

So far, we have built AI systems that store knowledge _symbolically_ or in a _distributed fashion_ (with neural nets being the latter). While the distributed form allows us to learn knowledge and rules automatically, it is much harder to understand and interpret than symbolically represented knowledge. This post argues that the main difference is in the **sparsity** of the learned knowledge. Of course, with more "sparse" knowledge, it should be easier for us to understand the internal workings of the AI system, since we can ignore the pruned connections. However, the author also argues that sparse knowledge will help 'guide the search for models and agents that can be said to "learn" but also "reason"'. Given that AGI will likely involve finding good representations for the world (in the sense of unsupervised learning), then sparse learning can be thought of as a bias towards finding better [bases](https://en.wikipedia.org/wiki/Basis_(linear_algebra)) for world models, that are more likely to be conceptually clean and more in line with Occam's razor.

In a postscript, the author considers arguments for AI risk. Notably, there isn't any consideration of goal-directedness or alignment failures; the worry is that we will start applying superhuman AI systems to superhuman tasks, and we won't know how to deal with these situations.

Sparsity seems like a good objective to shoot for in order to ensure explainability. I'm less convinced that it's worthwhile for representation learning: I doubt humans have any sort of "sparse learning" bias; I think sparsity of knowledge is a natural consequence of having to understand a very complex world with a very small brain. (Whereas current ML systems only have to understand much simpler environments.)


Making it easier to discover datasets

Google has launched Dataset Search, a tool that lets you search for datasets that you could then use in research.

I imagine that this is primarily targeted at data scientists aiming to learn about the real world, and not ML researchers, but I wouldn't be surprised if it was helpful for us as well. MNIST and ImageNet are both present, and a search for "self-driving cars" turned up some promising-looking links that I didn't investigate further.


State of AI Report 2021

As with <@past@>(@State of AI@) <@reports@>(@State of AI Report 2020@), I’m not going to summarize the entire thing; instead you get the high-level themes that the authors identified:

1. AI is stepping up in more concrete ways, including in mission critical infrastructure.
2. AI-first approaches have taken biology by storm (and we aren’t just talking about AlphaFold).
3. Transformers have emerged as a general purpose architecture for machine learning in many domains, not just NLP.
4. Investors have taken notice, with record funding this year into AI startups, and two first ever IPOs for AI-first drug discovery companies, as well as blockbuster IPOs for data infrastructure and cybersecurity companies that help enterprises retool for the AI-first era.
5. The under-resourced AI-alignment efforts from key organisations who are advancing the overall field of AI, as well as concerns about datasets used to train AI models and bias in model evaluation benchmarks, raise important questions about how best to chart the progress of AI systems with rapidly advancing capabilities.
6. AI is now an actual arms race rather than a figurative one, with reports of recent use of autonomous weapons by various militaries.
7. Within the US-China rivalry, China's ascension in research quality and talent training is notable, with Chinese institutions now beating the most prominent Western ones.
8. There is an emergence and nationalisation of large language models.

In <@last year’s report@>(@State of AI Report 2020@), I said that their 8 predictions seemed to be going out on a limb, and that even 67% accuracy woud be pretty impressive. This year, they scored their predictions as 5 “Yes”, 1 “Sort of”, and 2 “No”. That being said, they graded “The first 10 trillion parameter dense model” as “Yes”, I believe on the basis that Microsoft had run a couple of steps of training on a 32 trillion parameter dense model. I definitely interpreted the prediction as saying that a 10 trillion parameter model would be trained _to completion_, which I do not think happened publicly, so I’m inclined to give it a “No”. Still, this does seem like a decent track record for what seemed to me to be non-trivial predictions. This year's predictions seem similarly "out on a limb" as last year's.

This year’s report included one-slide summaries of many papers I’ve summarized before. I only found one major issue -- the slide on <@TruthfulQA@>(@TruthfulQA: Measuring How Models Mimic Human Falsehoods@) implies that larger language models are less honest _in general_, rather than being more likely to imitate human falsehoods. This is actually a pretty good track record, given the number of things they summarized where I would have noticed if there were major issues.