Learning Heuristics for Automated Reasoning through Deep Reinforcement Learning

Gil Lederman
Markus N. Rabe
Edward A. Lee
Sanjit A. Seshia

The formal methods community uses SAT solvers all the time in order to solve complex search-based problems. These solvers use handtuned heuristics in order to drastically improve performance. The heuristics only affect the choices in the search process, not the correctness of the algorithm overall. Obviously, we should consider using neural nets to learn these heuristics instead. However, neural nets take a long time to run, and SAT solvers have to make these decisions very frequently, so it's unlikely to actually be helpful -- the neural net would have to be orders of magnitude better than existing heuristics. So, they instead do this for QBF (quantified boolean formulas) -- these are PSPACE complete, and the infrastructure needed to support the theory takes more time, so it's more likely that neural nets can actually help. They implement this using a graph neural network and engineer some simple features for variables and clauses. (Feature engineering is needed because there can hundreds of thousands of variables, so you can only have ~10 numbers to describe the variable.) It works well, doing better than the handcoded heuristics.

For over a year now people keep asking me whether something like this is doable, since it seems like an obvious win combining PL and ML, and why no one has done it yet. I've mentioned the issue about neural nets being too slow, but it still seemed doable, and I was really tempted to do it myself. So I'm really excited that it's finally been done!

Oh right, AI alignment. Yeah, I do actually think this is somewhat relevant -- this sort of work could lead to much better theorem provers and formal reasoning, which could make it possible to create AI systems with formal guarantees. I'm not very optimistic about this approach myself, but I know others are.


Visual Reinforcement Learning with Imagined Goals

Ashvin Nair
Vitchyr Pong
Murtaza Dalal
Shikhar Bahl
Steven Lin
Sergey Levine

[Hindsight Experience Replay](https://blog.openai.com/ingredients-for-robotics-research/) (HER) introduced the idea of accelerating learning with sparse rewards, by taking trajectories where you fail to achieve the goal (and so get no reward, and thus no learning signal) and replacing the actual goal with an "imagined" goal chosen in hindsight such that you actually achieved that goal, which means you get reward and can learn. This requires that you have a space of goals such that for any trajectory, you can come up with a goal such that the trajectory achieves that goal. In practice, this means that you are limited to tasks where the goals are of the form "reach this goal state". However, if your goal state is an image, it is very hard to learn how to act in order to reach any possible image goal state (even if you restrict to realistic ones), since the space is so large and unstructured. The authors propose to first learn a structured latent representation of the space of images using a variational autoencoder (VAE), and then use that structured latent space as the space of goals which can be achieved. They also use Q-learning instead of DDPG (which is what HER used), so that they can imagine any goal with a minibatch (s, a, s') and learn from it (whereas HER/DDPG is limited to states on the trajectory).

This is a cool example of a relatively simple yet powerful idea -- instead of having a goal space over all states, learn a good latent representation and use that as your goal space. This enables unsupervised learning in order to figure out how to use a robot to generally affect the world, probably similarly to how babies explore and learn.


OpenAI Five Benchmark

The benchmark match for OpenAI Five will be a best-of-three match on August 5 at 2pm. They have already removed many of the restrictions on gameplay, including the two most important ones (wards and Roshan), as well as widening the pool of heroes to choose from 5 to 18.

I wonder if they are planning to play a game where both sides draft heroes, or where both sides get a randomly chosen team of 5 heroes. Previously I would have expected that they were choosing randomly, since it seems very difficult to learn solely from experience whether your team choice works well, given that the number of possible drafts is combinatorially large, and the way that the draft affects outcome is very complicated and long term and so hard to capture in a gradient. Now, I'm pretty uncertain -- if deep RL was enough to get this far, it could be good enough to deal with that as well. And it's possible that you can actually do well at drafting with some relatively simple heuristics -- I don't know Dota well enough to say.


The Pursuit of (Robotic) Happiness: How TRPO and PPO Stabilize Policy Gradient Methods

I barely looked at this -- I think it's an introduction to policy gradient methods for reinforcement learning. It assumes very little background (less than I assume in these summaries).


Retro Contest: Results

John Schulman
Vicki Pfau
Alex Nichol
Christopher Hesse
Oleg Klimov and Larissa Schiavo

OpenAI has announced the results of the [Retro Contest](https://blog.openai.com/retro-contest/). The winning submissions were modified versions of existing algorithms like joint PPO and Rainbow, without any Sonic-specific parts.


Fast reinforcement learning through the composition of behaviours

André Barreto
Shaobo Hou
Diana Borsa
David Silver
Doina Precup

While model-based RL agents can easily adapt their policy to changed rewards on the same environment, planning is expensive and learning good models can be challenging for many tasks. On the other hand, it is challenging to get model-free agents to adapt their policy to a new reward without extensive retraining. An intermediate solution is to use so-called successor features: Instead of a value function **V(π,s)** representing the expected discounted reward for a policy **π** starting in state **s**, successor features are a vector-valued value function **ψ(π,s)** representing an expected discounted feature vector **ϕ**. If our reward equals **r = w ⋅ ϕ** for some weight vector **w**, we can easily obtain the original value function by taking the scalar product of the successor features and the weight vector: **V(π,s) = w ⋅ ψ(π,s)**. Successor features thus allow us to evaluate a fixed policy **π** for all rewards that are linear in **ϕ**, which is called *generalized policy evaluation*.

Now that we can evaluate policies for different preferences, we would like to efficiently find a good policy for a given novel preference. Inspired by human learning that often combines previously learned skills, we employ *generalized policy improvement*. In vanilla policy improvement, we improve upon a policy **π** we can evaluate by choosing the action that maximizes the immediate reward plus the discounted value **V(π,s')** of following **π** starting in the next state **s'**. In generalized policy improvement, we have multiple policies and choose the action that maximizes the reward plus the discounted value of following the best of these policies starting in the next state **s'**. To obtain a policy for the new preference, we "stitch together" all policies we learnt for previous preferences and the resulting policy performs at least as good as all of the old policies with respect to the new preference. As generalized policy improvement does not require any additional environment samples, it enables zero-shot transfer to new preferences. Empirically, even if the weight vector **w** has to be learnt from reward signals, generalized policy improvement is very sample efficient. Additional samples can then be used to further improve the policy using standard RL.

I really like the idea of successor features. Similar to model-based systems, they allow us to evaluate policies for many different rewards, which can be useful for anticipating problematic behaviour before deploying a system. However, note that we still need to execute the policy we obtained by generalized policy improvement to evaluate it for different rewards: The only guarantees we have is that it is better than the previous policies for the reward for which the improvement step was carried out (and potentially some weaker bounds based on the similarity of different rewards).

Fast reinforcement learning with generalized policy updates


Does On-Policy Data Collection Fix Errors in Off-Policy Reinforcement Learning?

Aviral Kumar
Abhishek Gupta
Sergey Levine

Q-learning finds the optimal **Q**-function **Q*** by updating our estimate **Q(s,a)** for a state-action pair **(s,a)** to get closer to the immediate reward plus the discounted **Q**-value for the best action **a'** in the next state **s'**. To generate samples, we usually pick actions corresponding to high **Q**-values. In bandit problems where **s'** is always terminal and thus has all **Q**-values at zero, this leads to **corrective feedback**: If we overestimated an actions value, we will pick this action again soon and are quickly able to correct our misconception. In general MDPs, corrective feedback can be a lot weaker as our update of **Q(s,a)** also depends on the **Q**-values for the next state: To get corrective feedback, we need somewhat correct **Q**-values for the next state, but to get these we likely needed good values for the second to next state, etc. This is particularly problematic with function approximation as updating the current state's **Q**-value might lead to a worse estimate for values down the chain. Consequently, we might see convergence to suboptimal **Q**-functions, instable learning, or problems with sparse or noisy rewards.

To deal with this, we would like to first prioritize correct estimates for states near the end of the chain. But in many branching problems, we actually observe these states with the least frequency such that their values are influenced disproportionally by other states' values when function approximation is used. The authors' approach, dubbed DisCor, reweighs the data distribution to account for this: We would like to preferentially sample states for which we expect **Q** to be close to **Q*** after the update and thus give more weight to state-action pairs when we expect the error **|Q*-Q|** to already be small. As we don't know **Q***, we rely on a bound for the error at a state-action pair **(s,a)** equal to the sum of the magnitudes of previous updates down the chain plus the initial error, discounted by the usual discount rate **γ** as we move back in time. Thus, the error in the next state one step ago is discounted by **γ**, the error in the second to next state two steps ago is discounted by **γ** squared and the initial error is discounted by **γ** to the **k**. This bound can be approximated by a neural network using a SARSA-like update rule, for which the influence of the unknown initial error fades for large **k** due to the discounting.

DisCor is evaluated on MetaWorld tasks in both the single and multi-task setting and SAC augmented with DisCor clearly outperforms SAC in many settings. Similar improvements can be observed for DQN on Atari.

Putting less weight on updating values with fluctuating targets seems like a good idea. As the approach does not require much additional compute if weights are shared for the **Q**-network and the network estimating the bound, and as it seems quite orthogonal to previous improvements to methods based on **Q**-functions, I would not be surprised if it became somewhat widely used.

Paper: DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction


Generalized Hindsight for Reinforcement Learning

Alexander C. Li
Lerrel Pinto
Pieter Abbeel

[Hindsight Experience Replay](https://arxiv.org/abs/1707.01495) (HER) introduced the idea of _relabeling_ trajectories in order to provide more learning signal for the algorithm. Intuitively, if you stumble upon the kitchen while searching for the bedroom, you can’t learn much about the task of going to the bedroom, but you can learn a lot about the task of going to the kitchen. So even if the original task was to go to the bedroom, we can simply pretend that the trajectory got rewards as if the task was to go to the kitchen, and then update our kitchen-traversal policy using an off-policy algorithm.

HER was limited to goal-reaching tasks, in which a trajectory would be relabeled as attempting to reach the state at the end of the trajectory. What if we want to handle other kinds of goals? The key insight of this paper is that trajectory relabeling is effectively an inverse RL problem: we want to find the task or goal for which the given trajectory is (near-)optimal. This allows us to generalize hindsight to arbitrary spaces of reward functions.

This leads to a simple algorithm: given a set of N possible tasks, when we get a new trajectory, rank how well that trajectory does relative to past experience for each of the N possible tasks, and then relabel that trajectory with the task for which it is closest to optimal (relative to past experience). Experiments show that this is quite effective and can lead to significant gains in sample efficiency. They also experiment with other heuristics for relabeling trajectories, which are less accurate but more computationally efficient.

Getting a good learning signal can be a key challenge with RL. I’m somewhat surprised it took this long for HER to be generalized to arbitrary reward spaces -- it seems like a clear win that shouldn’t have taken too long to discover (though I didn’t think of it when I first read HER).


Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement

Benjamin Eysenbach*
Xinyang Geng*
Sergey Levine
Ruslan Salakhutdinov

This paper was published at about the same time as the previous one, and has the same key insight. There are three main differences with the previous paper:

1. It shows theoretically that MaxEnt IRL is the “optimal” (sort of) way to relabel data if you want to optimize the multitask MaxEnt RL objective.
2. In addition to using the relabeled data with an off-policy RL algorithm, it also uses the relabeled data with behavior cloning.
3. It focuses on fewer environments and only uses a single relabeling strategy (MaxEnt IRL relabeling).


Using Selective Attention in Reinforcement Learning Agents

Recently winning a best paper award at GECCO 2020, this work marks a leap forward in the performance capabilities learned by small agents via evolutionary methods. Specifically, it shows that by jointly learning which small fraction of input to attend to, agents with only thousands of free parameters can be trained by an evolutionary strategy to achieve state-of-the-art performance in vision-based control tasks.

The key pieces include self-attention over input patches, non-differentiable top-K patch selection that affect 'inattentional blindness', and training via CMA-ES. By design, the agent is interpretable as the top-K patches that are selected can be examined. Empirically, the agent has 1000x fewer weights than a competing neural architecture, and the method shows robustness to changes in task-irrelevant inputs, as the agent learns to focus only on task-relevant patches.

The parallelism afforded by evolutionary methods and genetic algorithms might be valuable in an environment where weak compute is plentiful, so it's exciting to see evidence of such methods besting GPU-hungry deep neural networks. However, I wonder how this would do on sparse reward tasks, where the fitness function is almost always uninformative. Finally, while it generalises to settings where there are task-irrelevant distractions, its deliberately sharp self-attention likely leaves it vulnerable to even simple adversarial attacks.

Paper: Neuroevolution of Self-Interpretable Agents


Improving Sample Efficiency in Model-Free Reinforcement Learning from Images

Denis Yarats
Amy Zhang
Ilya Kostrikov
Brandon Amos
Joelle Pineau
Rob Fergus

Sample efficiency in RL can be improved by using off-policy methods that can reuse the same sample multiple times and by using self-supervised auxiliary losses that help with representation learning, especially when rewards are sparse. This work combines both approaches by proposing to learn a latent state representation using an autoencoder while jontly training an agent on that latent representation using <@SAC@>(@Soft Actor-Critic: Deep Reinforcement Learning for Robotics@). Previous work in the on-policy case shows a positive effect from propagating Actor-Critic gradients through the encoder to improve the usefulness of the encoding for policy learning. However, this destabilizes training in the off-policy case, as changing the encoding to facilitate the actor also changes the Q-function estimate, which in turn changes the actor's goal and can introduce nonstationarity. This problem is circumvented by only propagating the Q-network's gradients through the encoder while blocking the actor's gradients.

The method strongly outperforms SAC trained on pixels. It also matches the previous state-of-the-art set by model-based approaches on an image-based continuous control task and outperforms them for noisy observations (as these make dynamics models hard to learn). The authors also find that the learnt encodings generalize between tasks to some extent and that reconstructing the true environment state is easier using their latent representation than using a representation obtained by training SAC on pixels directly.

Methods like this that can benefit from seeing a lot of action-independent environment observations might be quite important for applying RL to the real world, as this type of data is a lot cheaper to generate. For example, we can easily generate a ton of observations from a factory by equipping workers with cameras, but state-action-next-state triples from a robot interacting with the factory are very costly to obtain.


Learning to Play No-Press Diplomacy with Best Response Policy Iteration

Thomas Anthony*
Tom Eccles*
Andrea Tacchetti
János Kramár
Ian Gemp
Thomas C. Hudson
Nicolas Porcel
Marc Lanctot
Julien Pérolat
Richard Everett
Satinder Singh
Thore Graepel
Yoram Bachrach

Diplomacy is a game with simple rules where 7 players simultaneously move units every turn to capture territory. Units are evenly matched by default, so winning relies on getting support from some players against others. 'No-Press' Diplomacy limits communication between players to only orders submitted to units, removing the complex verbal negotiations that characterize traditional gameplay.

Previous state-of-the-art No-Press Diplomacy methods were trained to imitate human actions after collecting a dataset of 150,000 human Diplomacy games. This paper presents a new algorithmic method for playing No-Press Diplomacy using a policy iteration approach initialized with human imitation. To find better policies, their methods use "best response" calculations, where the best response policy for some player is the policy that maximizes the expected return for that player against opponent policies. Diplomacy is far too large for exact best response calculation, so the paper introduces an approximation, "Sampled Best Response", which
- Uses Monte-Carlo sampling to estimate opponents' actions each turn
- Only considers a small set of actions sampled from each candidate best response policy
- Only tries to make a single-turn improvement to its policy (rather than trying to optimize for the whole rest of the game)
Similar to other policy iteration methods, the paper creates a dataset of games every iteration using its Sampled Best Response method, then trains neural networks to create policy and value functions that predict the actions chosen by Sampled Best Response. To remedy issues where Sampled Best Response continually cycles through the best strategy for the last iteration,  the paper tries several variants of a technique called "Fictitious Play". In the best-performing variant, the policy network is trained to predict the latest Sampled Best Response given explicitly averaged _historical_ opponent and player policies, rather than just the latest policies.

The paper's methods outperform existing algorithmic methods for No-Press Diplomacy on a variety of metrics, but are still fairly few-shot _exploitable_-- at the end of training, the strongest (non-human) exploiter of the final policy wins 48% of the time. They also find that the strongest exploit doesn't change much through training, though few-shot exploitability does decrease from the beginning of training to the end.

This paper represented real progress in automated Diplomacy, but is still far from human-level. I’ll be pretty interested to see whether we can reach human-level by creating improved self-play algorithms, like the one presented in this paper, and the ones used for Poker and Go, or if we will have to wait for novel, more general reasoning algorithms applied to Diplomacy. Unlike Poker, Diplomacy against multiple human players involves collusion and implicit signalling, even with No Press. It seems possible to me that it is very difficult to become good at modeling those dynamics through self-play alone. If we did get to human-level through self-play, it would make me more optimistic about the extent to which training is likely to be a bottleneck in other domains which require sophisticated models of human behavior.


Retro Contest

Christopher Hesse
John Schulman
Vicki Pfau
Alex Nichol
Oleg Klimov
and Larissa Schiavo

OpenAI has released Gym Retro, providing an interface to work with video games from SEGA Genesis, which are more complex than the ones from Atari. They want to use these environments to test transfer learning in particular, where the agent may be pretrained on initial levels for as long as desired, and then must learn how to complete a new test level with only 1 million timesteps (~18 hours) of gameplay. (Humans do well with 2 hours of pretraining and 1 hour of play on the test level.)

If you want to keep track of progress in deep RL, probably -- this seems quite likely to become the new set of benchmarks that researchers work on. There's also another example of specification gaming in the post.


Danijar Hafner
Timothy Lillicrap
Mohammad Norouzi
Jimmy Ba

Model-based reinforcement learning can have better sample efficiency, allows for smarter exploration strategies, and facilitates generalization between different tasks. Still, previous attempts at model-based RL on the Atari Benchmark like <@Dreamer@>(@Dream to Control: Learning Behaviors by Latent Imagination@) and <@SimPLe@>(@Simulated Policy Learning in Video Models@) were unable to compete with model-free algorithms in terms of final performance. This paper presents DreamerV2, a model-based algorithm that outperforms DQN and its variants -- including Rainbow -- in terms of both median human- or gamer-normalized performance and on mean world-record normalized performance on Atari after 200M environment steps, achieving roughly 35% on the latter (25% if algorithm performance is clipped to max out at 100% for each game).

DreamerV2 learns a recurrent state-space model that stochastically encodes frames and a hidden state into a latent variable and uses the hidden state to predict the next value of the latent variable. Frames and reward are then reconstructed using both the hidden state and the latent variable. A policy is obtained by actor-critic training on the latent state space, leveraging parallelization to train on 468B imagined samples. As DreamerV2 does not use MCTS, it requires 8x less wall clock time to train than the more complicated but better performing <@MuZero Reanalyze@>(@Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model@).  Unlike earlier approaches, DreamerV2 uses a vector of categorical latent variables rather than gaussians to enable better model predictions for dynamics with multiple distinct modes, as well as KL-balancing (scaling up the importance of the transition loss compared to the entropy regularizer on the latent variable). Ablations confirm that the image reconstruction loss is crucial for DreamerV2's performance and that both the use of discrete latent variables and KL-balancing lead to significant improvements. Interestingly, preventing the gradients for reward prediction from affecting the world model does not affect performance at all.

It is worth noting that the authors use the <@Dopamine@>(@Introducing a New Framework for Flexible and Reproducible Reinforcement Learning Research@) framework for evaluating the model-free baselines, meaning that a slightly stunted version of Rainbow is used on an evaluation protocol different from the original publication without retuning hyperparameters. That said, DreamerV2 definitely performs at a level similar to Rainbow, which is significant progress in model-based RL. In particular, the fact that the reward can be inferred from the world model even without gradients flowing back from the reward suggests transferability of the world models to different tasks with the same underlying dynamics.

Paper: Mastering Atari with Discrete World Models


What Can Learned Intrinsic Rewards Capture?

Zeyu Zheng*
Junhyuk Oh*
Matteo Hessel
Zhongwen Xu
Manuel Kroiss
Hado van Hasselt
David Silver
Satinder Singh

This paper studies whether a learned reward function can serve as a locus of knowledge about the environment, that can be used to accelerate training of new agents. In particular, such a learned intrinsic reward can help with test-time adaptation: in a novel environment, the intrinsic reward can quickly "tell" the agent e.g. where it should explore -- even if in the new environment the agent has a different action space, or uses a different learning algorithm (situations that meta learning would typically not be able to handle).

The authors create an algorithm that learns an intrinsic reward function, that when used to train a new agent over a “lifetime” (which consists of multiple episodes), leads to the best cumulative reward over the lifetime, using a meta-gradient approach. Experiments on gridworlds demonstrate that these learned intrinsic rewards: 1. switch between early exploration and later exploitation, 2. explore only for information that is relevant for optimal behavior, 3. capture invariant causal relationships, and 4. can anticipate and adapt to changes in the extrinsic reward within a lifetime.

A common intuition that many researchers have is that specifying _what_ to do (the reward function) should be easier than specifying _how_ to do it (the policy). In practice, this _doesn't_ seem to be the case for deep learning, where imitation via inverse reinforcement learning (inferring a reward function and optimizing it) seems to be similar to imitation learning via behavior cloning ("copying" the policy). Similarly, this method seems broadly similar to meta learning algorithms like MAML and RL^2, though it does outperform them on one (probably carefully designed) transfer learning task.


Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Sparse Reward Environments

Vinicius G. Goecks
Gregory M. Gremillion
Vernon J. Lawhern
John Valasek
Nicholas R. Waytowich

This paper contributes to the effort of combining imitation and reinforcement learning to train agents more efficiently. The current difficulty in this area is that imitation and reinforcement learning proceed under rather different objectives which presents a significant challenge to updating a policy learned from a pure demonstration. A major portion of this difficulty stems from the use of so-called "on-policy" methods for training which require a significant number of environment interactions to be effective. In this paper, the authors propose a framework dubbed "Cycle-of-Learning" (CoL) that allows for the off-policy combination of imitation and reinforcement learning. This allows the two approaches to be combined much more directly which grounds the agent's policy in the expert demonstrations while simultaneously allowing for RL to fine-tune the policy. The authors show that CoL is an improvement over the current state of the art by testing their algorithm in several environments and performing an ablation study.

At first glance, it would seem as though the idea of using an off-policy method to combine imitation and reinforcement learning is obvious. However, the implementation is complicated by the fact that we want the value functions being estimated by our agent to satisfy the optimality condition for the Bellman equation. Prior work, such as [Hester et al. 2018](https://aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16976/16682) uses n-step returns to help pre-training and make use of on-policy methods when performing RL. What I like about this paper is that they perform an ablation study and show that simple sequencing of imitation learning and RL algorithms isn't enough to get good performance. This means that combining the imitation and reinforcement objectives into a single loss function is providing a significant improvement over other methods.


On Inductive Biases in Deep Reinforcement Learning

Matteo Hessel*
Hado van Hasselt*
Joseph Modayil
David Silver

The fewer inductive biases we use, the more general our algorithms will be. But how much does it really help to have fewer inductive biases? This paper replaces several hand-engineered components of an A2C agent with generic or adaptive variants.

Specifically, they compared: 1) reward clipping vs. reward normalization via <@PopArt@>(@Preserving Outputs Precisely while Adaptively Rescaling Targets@), 2) handpicked discount factor vs. online adaptive discounting via meta-learning, 3) fixed action repeats vs. learned action-commitment, and 4) standard Atari observation preprocessing vs. passing raw observations to a recurrent network. Over 57 Atari tasks, they found that the tuned algorithm outperformed the adaptive method only in (1). Performance was similar for (2) and (3), and the proposed method outperformed the baseline for (4). When the fully adaptive agent was compared to the vanilla agent (with heuristics designed for Atari) over 28 unseen continuous control tasks, the adaptive agent performed better in 14 of them, worse in one, and about the same in the rest, providing evidence that fewer inductive biases do lead to more general agents.

On net, I am quite happy to see work which argues in favour of reducing time spent hand-tuning and hand-crafting parts of a complex pipeline, and demonstrates the alternatives that currently exist to do so.

However, I feel the work did not fully compare the trade-off between tuning hyperparameters, and increasing the complexity of the pipeline by adding the adaptive components. I agree, though, that the latter is a one-time effort (per inductive bias), and is thus far more scalable than the former which needs to be repeated for each bias for every new task.

It would also be interesting to see how adaptive agents fare on problems where we care more about failures than successes, or if they are better/worse suited for safe exploration than baseline agents. My intuition is that adaptive internals of the agent cause it behave more noisily/unpredictably, and it may not fare as well as our current efforts for such problems.

**Rohin's opinion:** While it's certainly true that fewer inductive biases imply more general agents, it also usually means more compute and data requirements. For action repetition and learned discount factors, only one new parameter has to be learned, so it doesn't make much of a difference either way (and in fact performance on Atari doesn't change much). Clipped rewards do in fact learn faster than PopArt. I don't know why a recurrent network improves upon standard observation preprocessing for Atari -- perhaps initially RNNs were hard to train, and it became a de facto standard to use observation preprocessing, and no one checked about using recurrent networks later when RNNs became easier to train?


Are Deep Policy Gradient Algorithms Truly Policy Gradient Algorithms?

Andrew Ilyas
Logan Engstrom
Shibani Santurkar
Dimitris Tsipras
Firdaus Janoos
Larry Rudolph
Aleksander Madry

This paper argues that policy gradient algorithms are very dependent on additional optimisations (such as value function clipping, reward scaling, etc), and that they operate with poor estimates of the gradient. It also demonstrates that the PPO objective is unable to enforce a trust region, and that the algorithm's empirical success at doing so is due to the additional optimisations.

While the work in this paper is solid, the conclusions don't seem particularly surprising: everyone knows that deep RL is incredibly sample intensive (which straightforwardly implies inaccurate gradient estimates) and relies on many implementation tricks. I'm not familiar enough with PPO to know how surprising their last result is.


Assessing Generalization in Deep Reinforcement Learning

Charles Packer
Katelyn Gao
Jernej Kos
Philipp Krähenbühl
Vladlen Koltun
Dawn Song

This paper aims to create a benchmark for measuring generalisation in reinforcement learning. They evaluate a range of standard model-free algorithms on OpenAI Gym and Roboschool environments; the extent of generalisation is measured by varying environmental parameters at test time (note that these tasks are intended for algorithms which do not update at test time, unlike many transfer and multi-task learners). They distinguish between two forms of generalisation: interpolation (between values seen during training) and extrapolation (beyond them). The latter, which is typically much harder for neural networks, is measured by setting environmental parameters to more extreme values in testing than in training.

I agree that having standard benchmarks is often useful for spurring progress in deep learning, and that this one will be useful. I'm somewhat concerned that the tasks the authors have selected (CartPole, HalfCheetah, etc) are too simple, and that the property they're measuring is more like robustness to peturbations than the sort of combinatorial generalisation discussed in [this paper] (http://arxiv.org/abs/1806.01261) from [last week's newsletter](https://mailchi.mp/c1f376f3a12e/alignment-newsletter-30). The paper would benefit from more clarity about what they mean by "generalisation".


CURIOUS: Intrinsically Motivated Multi-Task, Multi-Goal Reinforcement Learning

Cédric Colas
Pierre Fournier
Olivier Sigaud
Mohamed Chetouani
Pierre-Yves Oudeyer

This paper presents an intrinsically-motivated algorithm (an extension of Universal Value Function Approximators) which learns to complete multiple tasks, each parameterised by multiple “goals” (e.g. the locations of targets). It prioritises replays of tasks which are neither too easy nor too hard, but instead allow maximal learning progress; this also help prevent catastrophic forgetting by refocusing on tasks which it begins to forget.

While I don’t think this paper is particularly novel, it usefully combines several ideas and provides easily-interpretable results.


Reinforcement Learning for Improving Agent Design

This paper explores what happens when you allow an RL agent to modify aspects of the environment; in this case, the agent's body. This allows you to learn asymmetric body designs that are better suited for the task at hand. There's another fun example of specification gaming -- the agent makes its legs so long that it simply falls forward to reach the goal.


TStarBots: Defeating the Cheating Level Builtin AI in StarCraft II in the Full Game

Peng Sun
Xinghai Sun
Lei Han
Jiechao Xiong
Qing Wang
Bo Li
Yang Zheng
Ji Liu
Yongsheng Liu
Han Liu
Tong Zhang

This paper showcases an RL agent which is able to defeat the built-in Starcraft AI (roughly at the level of the 30th-50th percentile of players). It do so by choosing between 165 hand-coded macro actions, which each correspond to an elementary task like producing a certain building. This avoids the necessity of learning unimportant details like exactly where the building should be placed, as well as difficult rules like the prerequisites for each building. The authors create a second agent which uses an expert system to choose actions in a hierarchical fashion, which performs at a similar level to the first. See also [Import AI](https://jack-clark.net/2018/09/25/import-ai-113-why-satellitesai-gives-us-a-global-eye-industry-pays-academia-to-say-sorry-for-strip-mining-it-and-kindred-researchers-seek-robot-standardization/).

I find Starcraft more interesting as a test-bed for deep RL than as a goal in itself. While the results of this paper are cool,  I doubt that its methods will scale well - in general, approaches which rely on a lot of human knowledge being hard-coded in don't tend to.

Note the similarity between the macros in this paper and the way that OpenAI Five could choose between different hand-coded combinations of items. However, the latter is only a small part of the game, whereas the former is much more extensive.


Challenges of Context and Time in Reinforcement Learning: Introducing Space Fortress as a Benchmark

The authors note that most existing RL benchmarks (like Atari games) lack sharp context-dependence and temporal sensitivity. The former requires an agent to sometimes change strategies abruptly; the latter requires an agent's strategy to vary over time. Space Fortress is an arcade-style game which does have these properties, and which cannot be solved by standard RL algorithms, even when rewards are made dense in a naive way. However, when the authors shape the rewards to highlight the context changes, their agent achieves superhuman performance.

The two properties that this paper highlights do seem important, and the fact that they can be varied in Space Fortress makes it a good benchmark for them.

I'm not convinced that the experimental work is particularly useful, though. It seems to reinforce the well-known point that shaped rewards can work well when they're shaped in sensible ways, and much less well otherwise.


Learning Invariances for Policy Generalization

Remi Tachet des Combes
Philip Bachman
Harm van Seijen

This paper compares three ways to induce generalisation in deep RL agents: data augmentation, meta-learning and adversarial training. They find that data augmentation boosts performance on a simple task from 2.8% to 99.8%, whereas the other two don't improve on the baseline.

This paper feels incomplete; it has very little discussion of the reasons why meta-learning and adversarial training failed, and the data augmentation result is rather perfunctory.


The use of embeddings in OpenAI Five

This blog post discusses some information which has been released by OpenAI about the structure of the OpenAI Five networks. Starting from Dota API information, each unit, ability, item and modifier is converted to an embedding. During processing, there's some surprising use of max-pooling over those embeddings even when it doesn't seem appropriate. Actions are chosen depending on the dot product similarity between their embedding and the final LSTM output. In a previous version, the target for the chosen action was picked without conditioning on what the action actually was; this has changed in more recent versions.


Why we want unbiased learning processes

Suppose we have a reward learning agent where we have designed the reward space so that "ask the human whether to do X" always has higher reward than "do X". The agent is now considering whether to ask the human to try heroin, or just give them heroin. If the agent gives them heroin, it will see their look of ecstasy and will update to have the reward function "5 for giving the human heroin, 7 for asking the human". If the agent asks the human, then the human will say "no", and the agent will update to have the reward function "-1 for giving the human heroin, 1 for asking the human". In both cases asking the human is the optimal action, yet the agent will end up giving the human heroin, since that gets reward 5, while asking the human gets reward 1. The post doesn't have this example, it has a formal model and an abstract example in that model that is isomorphic to the example I have.

If you found my example interesting and worth thinking about, it is worth reading the post to see an attempt at formalizing it.


Hierarchical system preferences and subagent preferences

Often, when looking at a hierarchical system, you can ascribe preferences to the system as a whole, as well as to individual parts or subagents of the system. Often, any divergence between the parts can be either interpreted as a difference between the goals of the parts, or a failure of rationality of one of the parts. The post gives a particular algorithm, and shows how based on the structure of the code of the subagent, we could either infer that the subagent is mistaken, or that it has different goals.

Now, we could infer meta-preferences by seeing how the system tends to self-modify -- perhaps we notice that it tends to amplify the "goals" of one particular subagent, in which case we can infer that it has a meta-preference for those goals. But without that, there's no correct answer to what the true goals are. In the post's own words: "In the absence of some sort of meta-preferences, there are multiple ways of establishing the preferences of a hierarchical system, and many of them are equally valid."


Figuring out what Alice wants: non-human Alice

We know that if we have a potentially irrational agent, then inferring their preferences is [impossible](https://arxiv.org/abs/1712.05812) without further assumptions. However, in practice we can infer preferences of humans quite well. This is because we have very specific and narrow models of how humans work: we tend to agree on our judgments of whether someone is angry, and what anger implies about their preferences. This is exactly what the theorem is meant to prohibit, which means that humans are making some strong assumptions about other humans. As a result, we can hope to solve the value learning problem by figuring out what assumptions humans are already making and using those assumptions.

The fact that humans are quite good at inferring preferences should give us optimism about value learning. In the [framework](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc/p/h9DesGT3WT9u2k7Hr) of rationality with a mistake model, we are trying to infer the mistake model from the way that humans infer preferences about other humans. This sidesteps the impossibility result by focusing on the _structure_ of the algorithm that generates the policy. However, it still seems like we have to make some assumption about how the structure of the algorithm leads to a mistake model, or a model for what values are. Though perhaps we can get an answer that is principled enough or intuitive enough that we believe it.


Designing robust & reliable AI systems and how to succeed in AI

(As is typical for large content, I'm only summarizing the most salient points, and ignoring entire sections of the podcast that didn't seem as relevant.)

In this podcast, Rob delves into the details of Pushmeet's work on making AI systems _robust_. Pushmeet doesn't view AI safety and AI capabilities as particularly distinct -- part of building a good AI system is ensuring that the system is safe, robust, reliable, and generalizes well. Otherwise, it won't do what we want, so why would we even bother using it. He aims to improve robustness by actively searching for behaviors that violate the specification, or by formally verifying particular properties of the neural net. That said, he also thinks that one of the major challenges here is in figuring out the specification of what to verify in the first place.

He sees the problems in AI as being similar to the ones that arise in programming and computer security. In programming, it is often the case that the program that one writes down does not accurately match the intended specification, leading to bugs. Often we simply accept that these bugs happen, but for security critical systems such as traffic lights we can use techniques like testing, fuzzing, symbolic execution, and formal verification that allow us to find these failures in programs. We now need to develop these techniques for machine learning systems.

The analogy can go much further. Static analysis involves understanding properties of a program separately from any inputs, while dynamic analysis involves understanding a program with a specific input. Similarly, we can have "static" interpretability, which understands the model as a whole (as in [Feature visualization](https://distill.pub/2017/feature-visualization/)), or "dynamic" interpretability, which explains the model's output for a particular input. Another example is that the technique of abstract interpretation of programs is analogous to a particular method for verifying properties of neural nets.

This analogy suggests that we have faced the problems of AI safety before, and have made substantial progress on them; the challenge is now in doing it again but with machine learning systems. That said, there are some problems that are unique to AGI-type systems; it's just not the specification problem. For example, it is extremely unclear how we should communicate with such a system, which may have its own concepts and models that are very different from those of humans. We could try to use natural language, but if we do we need to ground the natural language in the way that humans do, and it's not clear how we could do that, though perhaps we could test if the learned concepts generalize to new settings. We could also try to look at the weights of our machine learning model and analyze whether it has learned the concept -- but only if we already have a formal specification of the concept, which seems hard to get.

I really like the analogy between programming and AI; a lot of my thoughts have been shaped by thinking about this analogy myself. I agree that the analogy implies that we are trying to solve problems that we've attacked before in a different context, but I do think there are significant differences now. In particular, with long-term AI safety we are considering a setting in which mistakes can be extremely costly, _and_ we can't provide a formal specification of what we want. Contrast this to traffic lights, where mistakes can be extremely costly but I'm guessing we can provide a formal specification of the safety constraints that need to be obeyed. To be fair, Pushmeet acknowledges this and highlights specification learning as a key area of research, but to me it feels like a qualitative difference from previous problems we've faced, whereas I think Pushmeet would disagree with that (but I'm not sure why).

<@Towards Robust and Verified AI: Specification Testing, Robust Training, and Formal Verification@>


The Conditional Entropy Bottleneck

While I've categorized this paper under robustness because it can apply to most forms of training, I'll talk about it specifically in the context of unsupervised learning (and in particular its relation to Contrastive Predictive Coding (CPC), summarized in the highlights).

One potential problem with deep learning is that there might be too _much_ information in the input, causing the model to learn spurious correlations that do not actually generalize well (see <@Causal Confusion in Imitation Learning@> as an example). The idea with CEB is to penalize the model for learning irrelevant information, using a form of _information bottleneck_.

We consider a setting where we want to learn a representation **Z** of some input data **X** in order to predict some downstream data **Y**. In CPC, **X** would be the inputs from time 1 to t, **Z** would be the latent representation **z_t**, and **Y** would be the future data **x_{t+k}**. Then, we want **Z** to capture the **minimum necessary information** needed for **Z** to predict **Y** as best as possible. The _necessary_ information is **I(Y; Z)**, that is, the mutual information between **Z** and **Y**: we want to maximize this to maximize our accuracy at predicting **Y**. Since **Y** depends on **X** and **Z** is computed from **X**, any information about **Y** must come through mutual information between **X** and **Z**. Maximizing just this **I(Y; Z)** term gives us Contrastive Predictive Coding.

However, we don't want to capture any extra irrelevant information (the minimality criterion), which means that **Z** shouldn't capture any _more_ information about **X** beyond what it captured to maximize **I(Y; Z)**. In information-theoretic terms, we want to _minimize_ **I(X; Z | Y)**. Thus, we have the CEB objective: minimizing **I(X; Z | Y) - γ I(Y; Z)**, where **γ** is a hyperparameter controlling the tradeoff between the two terms. The authors then use some fairly straightforward math to reduce the objective to simpler terms which can be bounded using variational approximations, leading to an algorithm that can work in practice.

The authors perform experiments on Fashion MNIST and CIFAR10 (where Y corresponds to the labels for the images, so we're in the supervised learning setting). Since the main benefit of CEB is to remove unnecessary information from the model, they evaluate adversarial robustness and out-of-distribution detection in addition to standard performance checks. They find that models trained with CEB perform better than ones trained with a variational information bottleneck, or ones trained with vanilla SGD.

While I'm not sure to what extent models learn truly irrelevant information (see <@Adversarial Examples Are Not Bugs, They Are Features@>), it seems good to add an incentive against learning information that won't be useful for a downstream task, and the empirical results (especially of the next paper) suggest that it is providing some benefit.


CEB Improves Model Robustness

This empirical paper finds that ImageNet classifiers trained with the CEB objective (summarized above) are already somewhat adversarially robust, without having any decrease in accuracy, and without any adversarial training. Notably, since CEB does not rely on knowing the attack method ahead of time, its adversarial robustness generalizes to multiple kinds of attacks, whereas models that were adversarially trained tend to be fragile in the face of previously unseen attacks.


Call for Papers: ICML 2019 Workshop on Uncertainty and Robustness in Deep Learning

Topics of this workshop include out-of-distribution detection, calibration, robustness to corruptions, robustness to adversaries, etc. Submissions are due April 30th.


Evaluating the Robustness of Collaborative Agents

Paul Knott
Micah Carroll
Sam Devlin
Kamil Ciosek
Katja Hofmann
A. D. Dragan
Rohin Shah

Assuming a well-specified reward function, we would like to evaluate robustness of an agent by looking at the average reward it obtains on a wide scenario of plausible test time inputs that it might get. However, the key challenge of robustness is that it is hard to specify the test distribution in advance, and we must work with the training distribution instead.

This paper (on which I am an author) proposes _measuring_ robustness using a suite of hand-designed _unit tests_. Just as a function is tested by having the programmer write down potential edge cases and checking for the expected behavior, AI developers can come up with a set of potential “edge case” situations (especially ones not likely to arise during training) and check whether the agent’s behavior on these situations works well or not. Intuitively, since these unit tests are created separately from the training process, they may not have the same spurious correlations that could be present in the training data. Thus, they can serve as an evaluation of the robustness of the agent.

The authors built a test suite for <@Overcooked@>(@Collaborating with Humans Requires Understanding Them@), and use it to evaluate several techniques aimed to improve the robustness of agents trained to collaborate with humans.

For example, one technique is to start each episode from a state sampled randomly from a dataset of human-human gameplay, so that the agents learn how to handle a broader diversity of states. This technique _decreases_ the average _validation_ reward, and if that’s all we look at, we would conclude that it did not work. However, the technique also _increases_ performance on the unit test suite, suggesting that in reality the technique does increase robustness, though it comes at the cost of reduced performance when playing with the particular set of partners that make up the validation distribution.


Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks

Previous [empirical](https://arxiv.org/pdf/1901.09960.pdf#page=2&zoom=100,0,81) [papers](https://papers.nips.cc/paper/8094-generalized-cross-entropy-loss-for-training-deep-neural-networks-with-noisy-labels.pdf) have shown that finding ways to decrease training time greatly improves robustness to label corruptions, but to my knowledge this is the first theoretical treatment.


Towards Robust and Verified AI: Specification Testing, Robust Training, and Formal Verification

and the Robust & Verified Deep Learning group

This post highlights three areas of current research towards making robust AI systems. First, we need better evaluation metrics: rather than just evaluating RL systems on the environments they were trained on, we need to actively search for situations in which they fail. Second, given a specification or constraint that we would like to ensure, we can develop new training techniques that can ensure that the specifications hold. Finally, given a specification, we can use formal verification techniques to ensure that the model obeys the specification on all possible inputs. The authors also list four areas of future research that they are excited about: leveraging AI capabilities for evaluation and verification, developing publicly available tools for evaluation and verification, broadening the scope of adversarial examples beyond the L-infinity norm ball, and learning specifications.

The biggest challenge I see with this area of research, at least in its application to powerful and general AI systems, is how you get the specification in the first place, so I'm glad to see "learning specifications" as one of the areas of interest.

If I take the view from this post, it seems to me that techniques like domain randomization, and more generally training on a larger distribution of data, would count as an example of the second type of research: it is a change to the training procedure that allows us to meet the specification "the agent should achieve high reward in a broad variety of environments". Of course, this doesn't give us any provable guarantees, so I'm not sure if the authors of the post would include it in this category.


Trustworthy Deep Learning Course

Jacob Steinhardt
Dawn Song
Trevor Darrell

This underway course covers topics in AI Safety topics for current deep learning systems. The course includes slides and videos.


AI Alignment Podcast: The Byzantine Generals’ Problem, Poisoning, and Distributed Machine Learning

Byzantine resilience is the ability of a system to operate successfully when some of its components have been corrupted, even if it's unclear which ones they are. In the context of machine learning, this is relevant to poisoning attacks in which some training data is altered to affect the batch gradient (one example being the activity of fake accounts on social media sites). El Mahdi explains that when data is very high-dimensional, it is easy to push a neural network into a bad local minimum by altering only a small fraction of the data. He argues that his work on mitigating this is relevant to AI safety: even superintelligent AGI will be vulnerable to data poisoning due to time constraints on computation, and the fact that data poisoning is easier than resilient learning.


An Empirical Evaluation on Robustness and Uncertainty of Regularization Methods

Sanghyuk Chun
Seong Joon Oh
Sangdoo Yun
Dongyoon Han
Junsuk Choe
Youngjoon Yoo

There are several small tricks to improve classification performance such as label smoothing, dropout-like regularization, mixup, and so on. However, this paper shows that many of these techniques have mixed and often negative effects on various notions of robustness and uncertainty estimates.


Maximum Entropy Inverse Reinforcement Learning

Brian D. Ziebart
Andrew Maas
J.Andrew Bagnell
and Anind K. Dey

While matching empirical feature counts helps to deal with the ambiguity of the reward functions, exactly matching featuer counts will typically require policies to be stochastic, in which case there are many stochastic policies that get the right feature counts. How do you pick among these policies? We should choose the distribution using the [principle of maximum entropy](https://en.wikipedia.org/wiki/Principle_of_maximum_entropy), which says to pick the stochastic policy (or alternatively, a probability distribution over trajectories) that has maximum entropy (and so the least amount of information). Formally, we’re trying to find a function P(ζ) that maximizes H(P), subject to E[features(ζ)] = empirical feature counts, and that P(ζ) is a probability distribution (sums to 1 and is non-negative for all trajectories). For the moment, we’re assuming deterministic dynamics.

We solve this constrained optimization problem using the method of Lagrange multipliers. With simply analytical methods, we can get to the standard MaxEnt distribution, where P(ζ | θ) is proportional to exp(θ f(ζ)). But where did θ come from? It is the Lagrange multiplier for constraint on expected feature counts. So we’re actually not done with the optimization yet, but this intermediate form is interesting in and of itself, because we can identify the Lagrange multiplier θ as the reward weights. Unfortunately, we can’t finish the optimization analytically -- however, we can compute the gradient for θ, which we can then use in a gradient descent algorithm. This gives the full MaxEnt IRL algorithm for deterministic environments. When you have (known) stochastic dynamics, we simply tack on the probability of the observed transitions to the model P(ζ | θ) and optimize from there, but this is not as theoretically compelling.

One warning -- when people say they are using MaxEnt IRL, they are usually actually talking about MaxCausalEnt IRL, which we'll discuss next.


Modeling Interaction via the Principle of Maximum Causal Entropy

Brian D. Ziebart
J. Andrew Bagnell
and Anind K. Dey

When we have stochastic dynamics, MaxEnt IRL does weird things. It is basically trying to maximize the entropy H(A1, A2, ... | S1, S2, ...), subject to matching the feature expectations. However, when you choose the action A1, you don’t know what the future states are going to look like. What you really want to do is maximize the causal entropy, that is, you want to maximize H(A1 | S1) + H(A2 | S1, S2) + ..., so that each action’s entropy is only conditioned on the previous states, and not future states. You can then run through the same machinery as for MaxEnt IRL to get the MaxCausalEnt IRL algorithm.


A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress

This is a comprehensive survey of IRL that should be useful to researchers, or students looking to perform a deep dive into IRL. It's particularly useful because it can compare and contrast across many different IRL algorithms, whereas each individual IRL paper only talks about their method and a few particular weaknesses of other methods. If you want to learn a lot about IRL, I would start with the previous readings, then read this one, and perhaps after that read individual papers that interest you.


Learning from humans: what is inverse reinforcement learning?

This article introduces and summarizes the first few influential papers on inverse reinforcement learning. [Algorithms for IRL](http://ai.stanford.edu/~ang/papers/icml00-irl.pdf) attacked the problem by formulating it as a linear program, assuming that the given policy or demonstrations is optimal. However, there are many possible solutions to this problem -- for example, the zero reward makes any policy or demonstration optimal. [Apprenticeship Learning via IRL](http://people.eecs.berkeley.edu/~russell/classes/cs294/s11/readings/Abbeel+Ng:2004.pdf) lets you learn from an expert policy that is near-optimal. It assumes that the reward function is a weighted linear combination of _features_ of the state. In this case, given some demonstrations, we only need to match the feature expectations of the demonstrations in order to achieve the same performance as the demonstrations (since the reward is linear in the features). So, they do not need to infer the underlying reward function (which may be ambiguous).


AI Alignment Podcast: An Overview of Technical AI Alignment in 2018 and 2019

Lucas Perry
Buck Shlegeris and Rohin Shah

This podcast with Buck and me is loosely structured around the [review I wrote](https://www.alignmentforum.org/posts/dKxX76SCfCvceJXHv/ai-alignment-2018-19-review) ([AN #84](https://mailchi.mp/1af38085edc5/an-84-reviewing-ai-alignment-work-in-2018-19)), but with a lot more debate and delving into specific points of pessimism and optimism. I suspect that every reader will have some section they're interested in. Since much of the discussion was itself meant to be a summary, I'm not going to try and summarize even further. Here's the list of topics covered:

Our optimism and pessimism about different approaches to aligned AI
Traditional arguments for AI as an x-risk
Modeling agents as expected utility maximizers
Ambitious value learning and specification learning/narrow value learning
Agency and optimization
Robustness
Scaling to superhuman abilities
Universality
Impact regularization
Causal models, oracles, and decision theory
Discontinuous and continuous takeoff scenarios
Probability of AI-induced existential risk
Timelines for AGI
Information hazards


AI Alignment Podcast: On DeepMind, AI Safety, and Recursive Reward Modeling

While Jan originally worked on theory (specifically AIXI), DQN, AlphaZero and others demonstrated that deep RL was a plausible path to AGI, and so now Jan works on more empirical approaches. In particular, when selecting research directions, he looks for techniques that are deeply integrated with the current paradigm, that could scale to AGI and beyond. He also wants the technique to work for agents in general, rather than just question answering systems, since people will want to build agents that can act, at least in the digital world (e.g. composing emails). This has led him to work on <@recursive reward modeling@>(@Scalable agent alignment via reward modeling@), which tries to solve the specification problem in the <@SRA framework@>(@Building safe artificial intelligence: specification, robustness, and assurance@).

Reward functions are useful because they allow the AI to find novel solutions that we wouldn't think of (e.g. AlphaGo's move 37), but often are incorrectly specified, leading to reward hacking. This suggests that we should do _reward modeling_, where we learn a model of the reward function from human feedback. Of course, such a model is still likely to have errors leading to reward hacking, and so to avoid this, the reward model needs to be updated online. As long as it is **easier to evaluate behavior than to produce behavior**, reward modeling should allow AIs to find novel solutions that we wouldn't think of.

However, we would eventually like to apply reward modeling to tasks where evaluation is also hard. In this case, we can decompose the evaluation task into smaller tasks, and recursively apply reward modeling to train AI systems that can perform those small helper tasks. Then, assisted by these helpers, the human should be able to evaluate the original task. This is essentially forming a "tree" of reward modeling agents that are all building up to the reward model for the original, hard task. While currently the decomposition would be done by a human, you could in principle also use recursive reward modeling to automate the decomposition. Assuming that we can get regular reward modeling working robustly, we then need to make sure that the tree of reward models doesn't introduce new problems. In particular, it might be the case that as you go up the tree, the errors compound: errors in the reward model at the leaves lead to slightly worse helper agents, which lead to worse evaluations for the second layer, and so on.

He recommends that rather than spending a lot of time figuring out the theoretically optimal way to address a problem, AI safety researchers should alternate between conceptual thinking and trying to make something work. The ML community errs on the other side, where they try out lots of techniques, but don't think as much about how their systems will be deployed in the real world. Jan also wants the community to focus more on clear, concrete technical explanations, rather than vague blog posts that are difficult to critique and reason about. This would allow us to more easily build on past work, rather than reasoning from first principles and reinventing the wheel many times.

More broadly, DeepMind is taking a portfolio approach to AI safety: they are trying many different lines of attack, and hoping that some of them will pan out. Currently, there are teams for agent alignment (primarily recursive reward modeling), incentive theory, trained agent analysis, policy, and ethics. They have also spent some time thinking about AI safety benchmarks, as in [AI Safety Gridworlds](https://arxiv.org/abs/1711.09883), since progress in machine learning is driven by benchmarks, though Jan does think it is quite hard to create a well-made benchmark.

I've become more optimistic about recursive reward modeling since the <@original paper@>(@Scalable agent alignment via reward modeling@), primarily (I think) because I now see more value in approaches that can be used to perform specific tasks (relative to approaches that try to infer "human values").

I also appreciated the recommendations for the AI safety community, and agree with them quite a lot. Relative to Jan, I see more value in conceptual work described using fuzzy intuitions, but I do think that more effort should be put into exposition of that kind of work.


AI alignment landscape

This post presents the following decomposition of how to make AI go well:

IMAGE HERE

Here are a few points about this decomposition that were particularly salient or interesting to me.

First, at the top level, the problem is decomposed into alignment, competence, and coping with the impacts of AI. The "alignment tax" (extra technical cost for safety) is only applied to alignment, and not competence. While there isn't a tax in the "coping" section, I expect that is simply due to a lack of space; I expect that extra work will be needed for this, though it may not be technical. I broadly agree with this perspective: to me, it seems like the major technical problem which _differentially_ increases long-term safety is to figure out how to get powerful AI systems that are _trying_ to do what we want, i.e. they have the right [motivation](https://www.alignmentforum.org/posts/ZeE7EKHTFMBs8eMxn/clarifying-ai-alignment#3ECKoYzFNW2ZqS6km) ([AN #33](https://mailchi.mp/b6dc636f6a1b/alignment-newsletter-33)). Such AI systems will hopefully make sure to check with us before taking unusual irreversible actions, making e.g. robustness and reliability less important. Note that <@techniques like verification, transparency, and adversarial training@>(@Techniques for optimizing worst-case performance@) may still be needed to ensure that the _alignment_ itself is robust and reliable (see the inner alignment box); the claim is just that robustness and reliability of the AI's _capabilities_ is less important.

Second, strategy and policy work here is divided into two categories: improving our ability to pay technical taxes (extra work that needs to be done to make AI systems better), and improving our ability to handle impacts of AI. Often, generically improving coordination can help with both categories: for example, the <@publishing concerns around GPT-2@>(@Better Language Models and Their Implications@) have allowed researchers to develop synthetic text detection (the first category) as well as to coordinate on when not to release models (the second category).

Third, the categorization is relatively agnostic to the details of the AI systems we develop -- these only show up in level 4, where Paul specifies that he is mostly thinking about aligning learning, and not planning and deduction. It's not clear to me to what extent the upper levels of the decomposition make as much sense if considering other types of AI systems: I wouldn't be surprised if I thought the decomposition was not as good for risks from e.g. powerful deductive algorithms, but it would depend on the details of how deductive algorithms become so powerful. I'd be particularly excited to see more work presenting more concrete models of powerful AGI systems, and reasoning about risks in those models, as was done in <@Risks from Learned Optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@).


AI Alignment Research Overview

It has been over three years since _[Concrete Problems in AI Safety](https://arxiv.org/abs/1606.06565)_. Since that time we have learned more about the structure of the safety problem. This document represents an updated taxonomy of problems relevant for AI alignment. Jacob Steinhardt decomposes the remaining technical work into “technical alignment (the overcoming of conceptual or engineering issues needed to create aligned AI), detecting failures (the development of tools for proactively assessing the safety/alignment of a system or approach), methodological understanding (best practices backed up by experience), and system-building (how to tie together the three preceding categories in the context of many engineers working on a large system).”

The first topic under “technical alignment” is “Out-of-Distribution Robustness,” which receives more emphasis than it did in Concrete Problems. Out-of-Distribution Robustness is in part motivated by the fact that transformative AI will lead to substantial changes to the real world, and we should like our systems to perform well even under these large and possibly rapid data shifts. Specific subproblems include _some_ work on adversarial examples and out-of-distribution detection. Next, the problem of Reward Learning is described. For this, there are challenges including learning human values and ensuring those lossily represented human values can remain aligned under extreme optimization. While we have attained more conceptual clarity about reward learning since _Concrete Problems_, reward learning still remains largely “uncharted,” and it is still not clear “how approach the problem.” The next section on Scalable Reward Generation points out that, in the future, labeling meaning or providing human oversight will prove increasingly difficult. Next, he proposes that we ought to study how to make systems “act conservatively,” such as endowing systems with the ability to activate a conservative fallback routine when they are uncertain. The final topic under technical alignment is Counterfactual Reasoning. Here one possible direction is generating a family of simulated environments to generate counterfactuals.

The “technical alignment” section is the majority of this document. Later sections such as “Detecting Failures in Advance” highlight the importance of deep neural network visualization and recent model stress-test datasets. “Methodological Understanding” suggests that we are more likely to build aligned AI systems if we improve our best practices for building and evaluating models, and “System Building” speculates about how to do this for future multi-faceted ML systems.

This is a welcome update to _Concrete Problems_ since it is slightly more concrete, current, and discusses improving safety in both deep learning and RL rather than mostly RL. While the document mentions many problems, the set of problems retains precision and fortunately does not include every capabilities concern that may possibly one day impact safety. A takeaway is that value learning and model transparency still need groundwork, but fortunately other problems including out-of-distribution robustness are more concretized and mostly need time and continued effort.

**Rohin's opinion:** One thing I particularly like about this agenda is that the connection to AI _alignment_ is significantly clearer than in _Concrete Problems_.


AI Alignment Podcast: Inverse Reinforcement Learning and the State of AI Alignment

Lucas interviewed me and we talked about a bunch of different topics. Some quick highlights, without the supporting arguments:
- If we want to use inverse reinforcement learning (IRL) to infer a utility function that we then optimize, we would have to account for systematic biases, and this is hard, and subject to an impossibility result.
- Humans do seem to be good at inferring goals of other humans, probably because we model them as planning in a similar way that we ourselves plan. It's reasonable to think that IRL could replicate this. However, humans have very different ideas on how the future should go, so this seems not enough to get a utility function that can then be optimized over the long term.
- Another issue with having a utility function that is optimized over the long term is that it would have to somehow solve a whole lot of very difficult problems like the nature of identity and population ethics and metaphilosophy.
- Since human preferences seem to change as the environment changes, we could try to build AI systems whose goals are constantly changing by continuously running IRL. This sort of approach is promising but we don't know how to get it working yet.
- IRL, agency and optimization all seem to require a notion of counterfactuals.
- One view of agency is that it is about how a search process thinks of itself, or about other search processes. This gives it a feeling of "choice", even though the output of the search process is determined by physics. This can explain the debates over whether evolution is an optimization process -- on the one hand, it can be viewed as a search process, but on the other, we understand it well enough to think of it as a "deterministic" procedure.
- One way to view the AI alignment problem is to view it as a human-AI interaction problem, so that we get an AI that evolves over time along with us.
- Rather than building a function maximizer, we could aim to build an AI system that is corrigible, or one that follows norms.- Both iterated amplification and debate operate on an exponential deliberation tree, though in different ways, using reasoning learned from humans. If a human would have some desirable property (such as good epistemics), so too should their amplification.- Both iterated amplification and debate are based on _explicit_ human reasoning, as opposed to intuitive reasoning.
- Value drift in the literal sense can be both positive and negative -- I certainly expect and want my stated preferences to change as I become more knowledgeable in the future.
- We only want the combined human-AI system to have a goal, which allows for a space of possibilities where the AI is not optimizing a goal.
- One of the problems that seems most troubling is the issue of inner optimizers, which will hopefully be described in a sequence soon.


Realism about rationality

In the same way that moral realism claims that there is one true morality (even though we may not know it yet), rationality realism is the claim that there is one "correct" algorithm for rationality or intelligence. This post argues that many disagreements can be traced back to differences on how much one identifies with the rationality realism mindset. For example, people who agree with rationality realism are more likely to think that there is a simple theoretical framework that captures intelligence, that there is an "ideal" decision theory, that certain types of moral reasoning are "correct", that having contradictory preferences or beliefs is really bad, etc. The author's skepticism about this mindset also makes them skeptical about agent foundations research.

This does feel like an important generator of many disagreements I've had. I'd split rationality realism into two subcases -- whether you expect that there is a simple "correct" algorithm for computation-bounded rationality, and whether you expect there is only a simple "correct" algorithm for rationality given infinite compute, but the bounded computation case may be a lot messier. (I'm guessing almost all rationality realists fall in the latter category, but I'm not sure.)

I'd expect most of the people working on reducing existential risk from AI to be much more realist about rationality, since we often start working on this based on astronomical waste arguments and utilitarianism, which seems very realist about preferences. (At least, this was the case for me.) This is worrying -- it seems plausible to me that there isn't a "correct" rationality or intelligence algorithm (even in the infinite compute case), but that we wouldn't realize this because people who believe that also wouldn't want to work on AI alignment.


RFP: Measuring and forecasting risks

Measurement and forecasting is useful for two reasons. First, it gives us empirical data that can improve our understanding and spur progress. Second, it can allow us to quantitatively compare the safety performance of different systems, which could enable the creation of safety standards. So what makes for a good measurement?

1. **Relevance to AI alignment:** The measurement exhibits a failure mode that becomes worse as models become larger, or tracks a potential capability that may emerge with further scale (which in turn could enable deception, hacking, resource acquisition, etc).
2. **Forward-looking:** The measurement helps us understand _future_ issues, not just those that exist today. Isolated examples of a phenomenon are good if we have nothing else, but we’d much prefer to have a systematic understanding of when a phenomenon occurs and how it tends to quantitatively increase or decrease with various factors. See for example <@scaling laws@>(@Scaling Laws for Neural Language Models@).
3. **Rich data source:** Not all trends in MNIST generalize to CIFAR-10, and not all trends in CIFAR-10 generalize to ImageNet. Measurements on data sources with rich factors of variation are more likely to give general insights.
4. **Soundness and quality:** This is a general category for things like “do we know that the signal isn’t overwhelmed by the noise” and “are there any reasons that the measurement might produce false positives or false negatives”.

What sorts of things might you measure?

1. As you scale up task complexity, how much do you need to scale up human-labeled data to continue to maintain good performance and avoid reward hacking? If you fail at this and there are imperfections in the reward, how bad does this become?
2. What changes do we observe based on changes in the _quality_ of the human feedback (e.g. getting feedback from amateurs vs experts)? This could give us information about the acceptable “difference in intelligence” between a model and its supervisor.
3. What happens when models are pushed out of distribution along a factor of variation that was not varied in the pretraining data?
4. To what extent do models provide wrong or undesired outputs in contexts where they are capable of providing the right answer?

Measurements generally seem great. One story for impact is that we have a measurement that we think is strongly correlated with x-risk, and we use that measurement to select an AI system that scores low on such a metric. This seems distinctly good and I think would in fact reduce x-risk! But I want to clarify that I don’t think it would convince me that the system was safe with high confidence. The conceptual arguments against high confidence in safety seem quite strong and not easily overcome by such measurements. (I’m thinking of <@objective robustness failures@>(@2-D Robustness@) of the form “the model is trying to pursue a simple proxy, but behaves well on the training distribution until it can execute a treacherous turn”.)

You can also tell stories where the measurements reveal empirical facts that then help us have high confidence in safety, by allowing us to build better theories and arguments, which can rule out the conceptual arguments above.

Separately, these measurements are also useful as a form of legible evidence about risk to others who are more skeptical of conceptual arguments.


RFP: Techniques for enhancing human feedback

Consider a topic previously analyzed in <@aligning narrowly superhuman models@>(@The case for aligning narrowly superhuman models@): how can we use human feedback to train models to do what we want in cases where the models are _more_ knowledgeable than the humans providing the feedback? A variety of techniques have been proposed to solve this problem, including <@iterated amplification@>(@Supervising strong learners by amplifying weak experts@), <@debate@>(@AI safety via debate@), <@recursive reward modeling@>(@Scalable agent alignment via reward modeling@), <@market making@>(@AI safety via market making@), and [generalizing from short deliberations to long deliberations](https://ai-alignment.com/turning-reflection-up-to-11-1bd6171afd21). This RFP solicits proposals that aim to test these or other mechanisms on existing systems. There are a variety of ways to set up the experiments so that the models are more knowledgeable than the humans providing the feedback, for example:

1. Train a language model to accurately explain things about a field that the feedback providers are not familiar with.
2. Train an RL agent to act well in an environment where the RL agent can observe more information than the feedback providers can.
3. Train a multilingual model to translate between English and a foreign language that the feedback providers do not know.


RFP: Interpretability

The author provides this one sentence summary: _We would like to see research building towards the ability to “reverse engineer" trained neural networks into human-understandable algorithms, enabling auditors to catch unanticipated safety problems in these models._

This RFP is primarily focused on an aspirational “intermediate” goal: to fully reverse engineer some modern neural network, such as an ImageNet classifier. (Despite the ambition, it is only an “intermediate” goal because what we would eventually need is a general method for _cheaply_ reverse engineering _any_ neural network.) The proposed areas of research are primarily inspired by the <@Circuits line of work@>(@Circuits Thread@):

1. **Discovering Features and Circuits:** This is the most obvious approach to the aspirational goal. We simply “turn the crank” using existing tools to study new features and circuits, and this fairly often yields an interesting result that makes progress towards reverse engineering a neural network.
2. **Scaling Circuits to Larger Models:** So far the largest example of reverse engineering is [curve circuits](https://distill.pub/2020/circuits/curve-circuits/), with 50K parameters. Can we find examples of structure in the neural networks that allow us to drastically reduce the amount of effort required per parameter? (As examples, see [equivariance](https://distill.pub/2020/circuits/equivariance/) and [branch specialization](https://distill.pub/2020/circuits/branch-specialization/).)
3. **Resolving Polysemanticity:** One of the core building blocks of the circuits approach is to identify a neuron with a concept, so that connections between neurons can be analyzed as connections between concepts. Unfortunately, some neurons are _polysemantic_, that is, they encode multiple different concepts. This greatly complicates analysis of the connections and circuits between these neurons. How can we deal with this potential obstacle?

The full RFP has many, many more points about these topics; it’s 8 pages of remarkably information-dense yet readable prose. If you’re at all interested in mechanistic interpretability, I recommend reading it in full.

This RFP also has the benefit of having the most obvious pathway to impact: if we understand what algorithm neural networks are running, there’s a much better chance that we can catch any problems that arise, especially ones in which the neural network is deliberately optimizing against us. It’s one of the few areas where nearly everyone agrees that further progress is especially valuable.


RFP: Truthful and honest AI

This RFP outlines research projects on [Truthful AI](https://arxiv.org/abs/2110.06674) (summarized below). They fall under three main categories:

1. Increasing clarity about “truthfulness” and “honesty”. While there are some tentative definitions of these concepts, there is still more precision to be had: for example, how do we deal with statements with ambiguous meanings, or ones involving figurative language? What is the appropriate standard for _robustly_ truthful AI? It seems too strong to require the AI system to never generate a false statement; for example it might misunderstand the meaning of a newly coined piece of jargon.
2. Creating benchmarks and tasks for Truthful AI, such as <@TruthfulQA@>(@TruthfulQA: Measuring How Models Mimic Human Falsehoods@), which checks for imitative falsehoods. This is not just meant to create a metric to improve on; it may also simply perform as a measurement. For example, we could <@experimentally evaluate whether honesty generalizes@>(@Experimentally evaluating whether honesty generalizes@), or explore how much truthfulness is reduced when adding in a task-specific objective.
3. Improving the truthfulness of models, for example by finetuning models on curated datasets of truthful utterances, finetuning on human feedback, using <@debate@>(@AI safety via debate@), etc.

Besides the societal benefits from truthful AI, building truthful AI systems can also help with AI alignment:
1. A truthful AI system can be used to supervise its own actions, by asking it whether its selected action was good.
2. A robustly truthful AI system could continue to do this after deployment, allowing for ongoing monitoring of the AI system.
3. Similarly, we could have a robustly truthful AI system supervise its own actions in hypothetical scenarios, to make it more robustly aligned.

While I agree that making AI systems truthful would then enable many alignment strategies, I’m actually more interested in the _methods_ by which we make AI systems truthful. Many of the ideas suggested in the RFP are ones that would apply to alignment more generally and aren’t particularly specific to truthful AI. So it seems like whatever techniques we used to build truthful AI could then be repurposed for alignment. In other words, I expect that the benefit to AI alignment of working on truthful AI is that it serves as a good test case for methods that aim to impose constraints upon an AI system. In this sense, it is a more challenging, larger version of the <@”never describe someone getting injured” challenge@>(@Redwood Research’s current project@). Note that I am only talking about how this helps AI alignment; there are also beneficial effects on society from pursuing truthful AI that I haven’t talked about here.


Andrew Critch on AI Research Considerations for Human Existential Safety

This podcast discusses the recent <@ARCHES@>(@AI Research Considerations for Human Existential Safety@) document, and several thoughts surrounding it. There’s a lot in here that I won’t summarize, including a bunch of stuff that was in the summary of ARCHES. I’m going to focus primarily on the (substantial) discussion of how to prioritize within the realm of possible risks related in some way to AI systems.

Firstly, let’s be clear about the goal: ensuring existential safety, that is, making sure human extinction never happens. Note the author means literal extinction, as opposed to something like “the loss of humanity’s long-term potential”, because the former is clearer. While it is not always clear whether something counts as “extinction” (what if we all become uploads?), it is a lot clearer than whether a scenario counts as a loss of potential.

Typical alignment work focuses on the “single-single” case, where a single AI system must be aligned with a single human, as in e.g. <@intent alignment@>(@Clarifying "AI Alignment"@). However, this isn’t ultimately what we care about: we care about multi-multi existential safety, that is, ensuring that when multiple AI systems act in a world with multiple humans, extinction does not happen. There are pretty significant differences between these: in particular, it’s not clear whether multi-multi “alignment” even has meaning, since it is unclear whether it makes sense to view humanity as an agent to which an AI system could be “aligned”.

Nonetheless, single-single alignment seems like an important subproblem of multi-multi existential safety: we will be delegating to AI systems in the future; it seems important that we know how to do so. How do we prioritize between single-single alignment, and the other subproblems of multi-multi existential safety? A crucial point is that single-single work will not be neglected, because companies have strong incentives to solve single-single alignment (both in the sense of optimizing for the right thing, and for being robust to distributional shift). In contrast, in multi-multi systems, it is often the case that there is a complex set of interacting effects that lead to some negative outcome, and there is no one actor to blame for the negative outcome, and as a result it doesn’t become anybody’s job to prevent that negative outcome.

For example, if you get a huge medical bill because the necessary authorization forms hadn’t been filled out, whose fault is it? Often in such cases there are many people to blame: you could blame yourself for not checking the authorization, or you could blame the doctor’s office for not sending the right forms or for not informing you that the authorization hadn’t been obtained, etc. Since it’s nobody’s job to fix such problems, they are and will remain neglected, and so work on them is more impactful.

Something like transparency is in a middle ground: it isn’t profitable yet, but probably will be soon. So, if someone were indifferent between a bunch of areas of research, the author would advise for e.g. multi-stakeholder delegation over transparency over robustness. However, the author emphasizes that it’s far more important that people work in some area of research that they find intellectually enriching and relevant to existential safety.

The podcast has lots of other points, here is an incomplete quick selection of them:

- In a multi-multi world, without good coordination you move the world in a “random” direction. There are a lot of variables which have to be set just right for humans to survive (temperature, atmospheric composition, etc) that are not as important for machines. So sufficiently powerful systems moving the world in a “random” direction will lead to human extinction.
- One response to the multi-multi challenge is to have a single group make a powerful AI system and “take over the world”. This approach is problematic since many people will oppose such a huge concentration of power. In addition, it is probably not desirable even if possible, since it reduces robustness by creating a single point of failure.
- Another suggestion is to create a powerful AI system that protects humanity (but is still uncontrollable in that humanity cannot stop its operation). The author does not like the solution much, because if we get it wrong and deploy a misaligned uncontrollable AI system, then we definitely die. The author prefers that we instead always have control over the AI systems we deploy.

Both this and the previous summary illustrate an increasingly common perspective:

1. The world is not going to look like “today’s world plus a single AGI agent”: instead, we will likely have a proliferation of many different AI systems specialized for different purposes.
2. In such a world, there are a lot of different challenges that aren’t standard intent alignment.
3. We should focus on these other challenges because [a variety of reasons].

**If you have technical CS skills**, how should you prioritize between this perspective and the more classical intent alignment perspective?

**Importance.** I’ve <@estimated@>(@Conversation with Rohin Shah@) a 10% chance of existential catastrophe via a failure of intent alignment, absent intervention from longtermists to address intent alignment. Estimates vary quite a lot, even among people who have thought about the problem a lot; I’ve heard as low as < 1% and as high as 80% (though these usually don’t assume “no intervention from longtermists”).

It’s harder to estimate the importance of structural risks and extinction risks highlighted in the two summaries above, but the arguments in the previous two posts seem reasonably compelling and I think I’d be inclined to assign a similar importance to it (i.e. similar probability of causing an existential catastrophe).

Note that this means I’m disagreeing with Critch: he believes that we are far more likely to go extinct through effects unique to multi-multi dynamics; in contrast I find the argument less persuasive because we do have governance, regulations, national security etc. that would already be trying to mitigate issues that arise in multi-multi contexts, especially things that could plausibly cause extinction.

**Neglectedness.** I’ve already taken into account neglectedness outside of EA in estimating the probabilities for importance. Within EA there is already a huge amount of effort going into intent alignment, and much less in governance and multi-multi scenarios -- perhaps a difference of 1-2 orders of magnitude; the difference is even higher if we only consider people with technical CS skills.

**Tractability.** I buy the argument in Dafoe’s article that for AI governance due to our vast uncertainty we need a “metropolis” model where field-building is quite important; I think that implies that solving the full problem (at today's level of knowledge) would require a lot of work and building of expertise. In contrast, with intent alignment, we have a single technical problem with significantly less uncertainty. As a result, I expect that currently in expectation a single unit of work goes further to solving intent alignment than to solving structural risks / multi-multi problems, and so intent alignment is more tractable.

I also expect technical ideas to be a bigger portion of "the full solution" in the case of intent alignment -- as Dafoe argues, I expect that for structural risks the solution looks more like "we build expertise and this causes various societal decisions to go better" as opposed to "we figure out how to write this piece of code differently so that it does better things". This doesn't have an obvious impact on tractability -- if anything, I'd guess it argues in favor of the tractability of work on structural risks, because it seems easier to me to create prestigious experts in particular areas than to make progress on a challenging technical problem whose contours are still uncertain since it arises primarily in the future.

I suspect that I disagree with Critch here: I think he is more optimistic about technical solutions to multi-multi issues themselves being useful. In the past I think humanity has resolved such issues via governance and regulations and it doesn’t seem to have relied very much on technical research; I’d expect that trend to continue.

**Personal fit.** This is obviously important, but there isn’t much in general for me to say about it.

Once again, I should note that this is all under the assumption that you have technical CS skills. I think overall I end up pretty uncertain which of the two areas I’d advise going in (assuming personal fit was equal in both areas). However, if you are more of a generalist, I feel much more inclined to recommend choosing some subfield of AI governance, again subject to personal fit, and Critch agrees with this.