Formal Language Constraints for Markov Decision Processes

Eleanor Quint
Dong Xu
Haluk Dogan
Zeynep Hakguder
Stephen Scott
Matthew Dwyer

Within the framework of RL, the authors propose using constraints defined by DFAs (deterministic finite automata) in order to eliminate safety failures, or to prevent agents from exploring clearly ineffective policies (which would accelerate learning). Constraints can be defined on any auxiliary information that can be computed from the "base" MDP. A constraint could either restrict the action space, forcing the agent to take an action that doesn't violate the constraint, which they term "hard" constraints; or a constraint could impose a penalty on the agent, thus acting as a form of reward shaping, which they term a "soft" constraint. They consider two constraints: one that prevents the agent from "dithering" (going left, then right, then left, then right), and one that prevents the agent from "overactuating" (going in the same direction four times in a row). They evaluate their approach with these constraints on Atari games and Mujoco environments, and show that they lead to increased reward and decreased constraint violations.

NeurIPS 2019 Workshop on Safety and Robustness in Decision Making

This method seems like a good way to build in domain knowledge about what kinds of action sequences are unlikely to work in a domain, which can help accelerate learning. Both of the constraints in the experiments do this. The paper also suggests using the technique to enforce safety constraints, but the experiments don't involve any safety constraints, and conceptually there do seem to be two big obstacles. First, the constraints will depend on state, but it is very hard to write such constraints given access only to actions and high-dimensional pixel observations. Second, you can only prevent constraint violations by removing actions one timestep before the constraint is violated: if there is an action that will inevitably lead to a constraint violation in 10 timesteps, there's no way in this framework to not take that action. (Of course, you can use a soft constraint, but this is then the standard technique of reward shaping.)

In general, methods like this face a major challenge: how do you specify the safety constraint that you would like to avoid violating? I'd love to see more research on how to create specifications for formal analysis.


Safe Option-Critic: Learning Safety in the Option-Critic Architecture

Arushi Jain
Khimya Khetarpal
Doina Precup

Let's consider an RL agent in the options framework (one way of doing hierarchical reinforcement learning). One way in which we could make such an agent safer would be to make it risk-averse. The authors define the controllability of a (state, option) pair to be the negative expected variance of the TD error. Intuitively, the controllability is higher when the value of the (state, option) pair is more predictable to the agent, and so by optimizing for controllability we can encourage risk-aversion. They derive the policy gradient when the objective is to maximize the reward and the controllability of the initial (state, option) pair, and use this to create the Safe-A2OC algorithm (a safe version of A2OC, which itself is a version of A2C for options). They test this out on a four-rooms gridworld problem, Cartpole, and three games from the Arcade Learning Environment (ALE).

I'm very excited to see a paper tackling safety in hierarchical reinforcement learning -- that seems like a really important area to consider, and doesn't have many safety people working on it yet. That said, this paper feels weird to me, because in order to learn that a particular (state, option) pair is bad, the RL agent must experience that pair somewhat often, so it will have done the risky thing. It's not clear to me where this would be useful. One upside could be that we do risky things less often, so our RL agent learns faster from its mistakes, and doesn't make them as often. (And in fact they find that this leads to faster learning in three games in the ALE.) Perhaps we could also use this to train a risk-averse agent in simulation, that then never makes a mistake when deployed in the real world.

I also wonder whether we should be trying to make our agents risk-averse. The "right" answer seems to me to a combination of two things: First, some things are actually very bad and have very large negative reward, and so they should be avoided with high probability. Second, when you are acting over a long period of time, even a small probability of failure at every time step compounds and leads to a near-guaranteed failure. If these are actually the reasons underlying risk aversion, it seems like we want to be able to imbue our RL agent with the underlying reasons, rather than flat risk aversion.


Model Reconstruction from Model Explanations

Smitha Milli
Ludwig Schmidt
Anca D. Dragan
Moritz Hardt

Back in [AN #16](https://mailchi.mp/f2950ed2ac4b/alignment-newsletter-16), I said that one way to prevent model reconstruction from gradient-based explanations was to add noise to the gradients. Smitha pointed out that the experiments with SmoothGrad are actually of this form, and it still is possible to recover the full model, so even adding noise may not help. I don't really understand SmoothGrad and it's relationship with noise (which is chosen to make a saliency map look nice, if I understand correctly) so I don't know exactly what to think here.

Fourteenth Artificial Intelligence and Interactive Digital Entertainment Conference


Does GPT-2 Know Your Phone Number?

Nicholas Carlini
Florian Tramer
Eric Wallace
Matthew Jagielski
Ariel Herbert-Voss
Katherine Lee
Adam Roberts
Tom Brown
Dawn Song
Ulfar Erlingsson
Alina Oprea
Colin Raffel

This post and associated paper demonstrate that large language models memorize rare training data, and (some of) that training data can then be extracted through an automated attack. The key idea is to sample text that is _unusually_ high likelihood. Given a high likelihood sample from a language model, we can check whether the likelihood is especially high by comparing the likelihood to:

1. The likelihood assigned by other (especially smaller) language models. Presumably these models would not have memorized the same content, especially if the content was rare (which is the content we are most interested in).
2. The length of the text when compressed by (say) zlib. Existing compression algorithms are pretty good at compressing regular English text, and so it is notable when a language model assigns high likelihood but the compression algorithm can’t compress it much.
3. The likelihood assigned to the same text, but lowercase. Often, memorized content is case-sensitive, and likelihood drops significantly when the case is changed.

The authors generate a lot of samples from GPT-2, use the metrics above to rank them in order of how likely they are to be memorized from the training set, and then investigate the top 1800 manually. They find that 604 of them are directly from the training set. While many are unobjectionable (such as news headlines), in some cases GPT-2 has memorized personal data (and the authors have extracted it simply by prompting GPT-2). In their most objectionable example, they extract the name, email, phone number, work address, and fax of a single person.

I really liked the paper: it contains a lot of empirical detail that didn’t make it into the blog post, that gave me a much better sense of the scope of the problem. I don’t really have the space to summarize it here, so I recommend reading the paper.

[Blog post: Privacy Considerations in Large Language Models](https://ai.googleblog.com/2020/12/privacy-considerations-in-large.html)
[Paper: Extracting Training Data from Large Language Models](https://arxiv.org/abs/2012.07805)


Privacy and machine learning: two unexpected allies?

Differential privacy provides guarantees on how much information you can obtain by making queries of a specific type of a dataset. Normally, in order to achieve such guarantees, you must add in randomness to the input data that can change the decision, so that there is a plausible explanation for any decision. Unsurprisingly, this tends to degrade performance. However, in deep learning, we often have the problem of our models overfitting to specific details in the training set instead of generalizing appropriately, so we might expect that differential privacy could actually _help_ with performance (as well as privacy). Private Aggregation of Teacher Ensembles (PATE) demonstrates that this is the case. It works by training several teacher models on different datasets to solve the task at hand. Then, by aggregating the results across the ensemble with some random noise, we can answer queries and put bounds on the amount of information that is leaked. However, with each query we use up more of our "privacy budget", so it can't be used arbitrarily long. To solve this, we can make a fixed number of queries to label some unlabelled data, use those labels to train a student model, and use the student model to make predictions forever after. An adversary could at worst infer the entire training dataset of the student model -- but that training set was designed to be private.

I would have been excited by work that randomizes the inputs to a deep learning technique in order to get better generalizability. It's cool that this goal dovetails so beautifully with the desire for differential privacy.


Seeking Power is Provably Instrumentally Convergent in MDPs

[The Basic AI Drives](https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf) argues that it is _instrumentally convergent_ for an agent to collect resources and gain power. This post and [associated paper](https://arxiv.org/abs/1912.01683) aim to formalize this argument. Informally, an _action_ is _instrumentally convergent_ if it is helpful for many goals, or equivalently, an action is instrumentally convergent to the extent that we expect an agent to take it, if we do not know what the agent’s goal is. Similarly, a _state_ has high power if it is easier to achieve a wide variety of goals from that state.

A natural formalization is to assume we have a distribution over the agent's goal, and define power and instrumental convergence relative to this distribution. We can then define power as the expected value that can be obtained from a state (modulo some technical caveats), and instrumental convergence as the probability that an action is optimal, _from our perspective of uncertainty_:  of course, the _agent_ knows its own goal, and acts optimally in pursuit of that goal.

You might think that optimal agents would provably seek out states with high power. However, this is not true. Consider a decision faced by high school students: should they take a gap year, or go directly to college? Let’s assume college is necessary for (100-ε)% of careers, but if you take a gap year, you could focus on the other ε% of careers or decide to go to college after the year. Then in the limit of farsightedness, taking a gap year leads to a more powerful state, since you can still achieve all of the careers, albeit slightly less efficiently for the college careers. However, if you know which career you want, then it is (100-ε)% likely that you go to college, so going to college is very strongly instrumentally convergent even though taking a gap year leads to a more powerful state.

Nonetheless, there are things we can prove. In environments where the only cycles are states with a single action leading back to the same state, and apart from that every action leads to a new state, and many states have more than one action, farsighted agents are more likely to choose trajectories that spend more time navigating to a cycle before spending the rest of the time in the cycle. For example, in Tic-Tac-Toe where the opponent is playing optimally according to the normal win condition, but the agent's reward for each state is drawn independently from some distribution on [0, 1], the agent is much more likely to play out to a long game where the entire board is filled. This is because the number of states that can be reached grows exponentially in the horizon, and so agents have more control by taking longer trajectories. Equivalently, the cycle with maximal reward is much more likely to be at the end of a longer trajectory, and so the optimal possibility is more likely to be a long trajectory.

I like the formalizations of power and instrumental convergence. I think in practice there will be a lot of complexity in a) the reward distribution that power and instrumental convergence are defined relative to, b) the structure of the environment, and c) how powerful AI systems actually work (since they won't be perfectly optimal, and won't know the environment structure ahead of time). Nonetheless, results with specific classes of reward distributions, environment structures, and agent models can still provide useful intuition.

[Clarifying Power-Seeking and Instrumental Convergence](https://www.alignmentforum.org/posts/cwpKagyTvqSyAJB7q/clarifying-power-seeking-and-instrumental-convergence), [Paper: Optimal Farsighted Agents Tend to Seek Power](https://arxiv.org/abs/1912.01683)


Consequences of Misaligned AI

One intuition for why powerful AI systems might lead to bad consequences goes as follows:

1) Humans care about many attributes of the world and we would likely forget some of these when trying to list them all.
2) Improvements along these attributes usually require resources, and gaining additional resources often requires sacrifices along some attributes.
3) Because of 1), naively deployed AI systems would only optimize some of the attributes we care about, and because of 2) this would lead to bad outcomes along the other attributes.

This paper formalizes this intuition in a model, identifies conditions for when deploying AIs can reduce true utility within the model and proposes two mitigation strategies, impact minimization and interactivity.

We assume that the world state consists of L attributes, all of which the human cares about having more of, that is, true utility is strictly increasing in each of the attributes. Each attribute has some minimum value, and can be increased from that minimum value through the use of a fixed, finite resource (which you could think of as money, if you want); this allows us to formalize (2) above. To formalize (1), we assume that the proxy utility optimized by the AI is only allowed to depend on J<L of the attribute dimensions.

Given this setup, the paper proves that if the AI maximizes the proxy utility, then all attributes that were omitted in the proxy utility will be set to their minimal value. This will be worse than not using the AI system at all if 1) the minimum values of attributes are sufficiently small (allowing the AI to cause damage), 2) the resource cost (resp. gain in true utility) for increasing an attribute is independent of the other attributes' level, 3) it always costs at least K resources to get a unit increase in any attribute, for some K > 0, and 4) utility has diminishing marginal returns in each attribute (and marginal returns tend to zero as the attribute increases).

Regarding mitigation, impact minimization requires that the AI keep all attributes that are omitted by the proxy constant. In this case, any gains in proxy utility must also be gains in true utility.

Meanwhile, in the interactive condition, the human gets to regularly select a new proxy (still only specifying J<L weights), or can choose to turn the AI off. Whether or not this is helpful depends on the AI’s optimization strategy and the frequency of human interventions: If the AI is “efficient” in the sense that it changes attributes as little as possible for any fixed gain in proxy utility, the human can choose a proxy that guarantees that _locally_, increases in the proxy correspond to increases in true utility. The strategy is to choose the attributes that are most sensitive to changes in resources (i.e. with the largest marginal returns) at the current state, and define the proxy to grow in these attributes as much as the true utility. As long as the human provides new proxies frequently enough to prevent the local guarantee from breaking, optimizing the proxy increases human utility.

We can also combine interactivity and impact minimization: in this case, the human should choose proxy utility functions that contain the most and least sensitive attributes (i.e. largest and smallest marginal returns) for the given state. The AI will then transfer some resources from the least sensitive attributes to the most sensitive attributes, while holding all other attributes fixed, leading to a guaranteed increase in true utility. In fact, it is possible to prove that this will converge to the maximum possible true utility.

This is close to an informal model I've had for a while and I am glad that it got formalized including theoretical results. I find it interesting that the frequency of updates to the proxy matters even if movement in the state space is reversible. As the authors mention, it is also crucial that the AI's actions don't hinder the human's ability to update the proxy, and I imagine that frequent updates to the proxy would be important for that as well in many cases.

**Rohin's opinion:** This is a nice formalization of several important conceptual points in the AI alignment literature:

1. If you forget to specify something you care about, it will usually be set to extreme values ([Of Myths and Moonshine](https://www.edge.org/conversation/the-myth-of-ai#26015)). In particular, the AI system will extract any resources that were being used for that attribute, and apply them elsewhere (<@The Basic AI Drives@>, [Formalizing convergent instrumental goals](https://intelligence.org/2015/11/26/new-paper-formalizing-convergent-instrumental-goals/))
2. Given that perfect information is impossible, interactivity becomes important (<@Human-AI Interaction@>, <@Incomplete Contracting and AI Alignment@>).
3. Conservatism (in this case through impact regularization) can be helpful (see the many blog posts and papers on mild optimization, low impact, and conservatism).


Inaccessible information

One way to think about the problem of AI alignment is that we only know how to train models on information that is _accessible_ to us, but we want models that leverage _inaccessible_ information.

Information is accessible if it can be checked directly, or if an ML model would successfully transfer to provide the information when trained on some other accessible information. (An example of the latter would be if we trained a system to predict what happens in a day, and it successfully transfers to predicting what happens in a month.) Otherwise, the information is inaccessible: for example, “what Alice is thinking” is (at least currently) inaccessible, while “what Alice will say” is accessible. The post has several other examples.

Note that while an ML model may not directly say exactly what Alice is thinking, if we train it to predict what Alice will say, it will probably have some internal model of what Alice is thinking, since that is useful for predicting what Alice will say. It is nonetheless inaccessible because there’s no obvious way of extracting this information from the model. While we could train the model to also output “what Alice is thinking”, this would have to be training for “a consistent and plausible answer to what Alice is thinking”, since we don’t have the ground truth answer. This could incentivize bad policies that figure out what we would most believe, rather than reporting the truth.

The argument for risk is then as follows: we care about inaccessible information (e.g. we care about what people _actually_ experience, rather than what they _say_ they experience) but can’t easily make AI systems that optimize for it. However, AI systems will be able to infer and use inaccessible information, and would outcompete ones that don’t. AI systems will be able to plan using such inaccessible information for at least some goals. Then, the AI systems that plan using the inaccessible information could eventually control most resources. Key quote: “The key asymmetry working against us is that optimizing flourishing appears to require a particular quantity to be accessible, while danger just requires anything to be accessible.”

The post then goes on to list some possible angles of attack on this problem. Iterated amplification can be thought of as addressing gaps in speed, size, experience, algorithmic sophistication etc. between the agents we train and ourselves, which can limit what inaccessible information our agents can have that we won’t. However, it seems likely that amplification will eventually run up against some inaccessible information that will never be produced. As a result, this could be a “hard core” of alignment.

I think the idea of inaccessible information is an important one, but it’s one that feels deceptively hard to reason about. For example, I often think about solving alignment by approximating “what a human would say after thinking for a long time”; this is effectively a claim that human reasoning transfers well when iterated over long periods of time, and “what a human would say” is at least somewhat accessible. Regardless, it seems reasonably likely that AI systems will inherit the same property of transferability that I attribute to human reasoning, in which case the argument for risk applies primarily because the AI system might apply its reasoning towards a different goal than the ones we care about, which leads us back to the <@intent alignment@>(@Clarifying "AI Alignment"@) formulation.

This [response](https://www.alignmentforum.org/posts/A9vvxguZMytsN3ze9/reply-to-paul-christiano-s-inaccessible-information) views this post as a fairly general argument against black box optimization, where we only look at input-output behavior, as then we can’t use inaccessible information. It suggests that we need to understand how the AI system works, rather than relying on search, to avoid these problems.


Multiparty Dynamics and Failure Modes for Machine Learning and Artificial Intelligence

While [Categorizing Variants of Goodhart’s Law](https://arxiv.org/abs/1803.04585) explains failure modes that occur when a single agent’s proxy becomes decoupled from the true goal, this paper aims to characterize failures involving multiple agents:

 **Accidental steering** happens when the combined actions of multiple agents facilitate single-agent failures. For example, catching more fish now is usually positively correlated with a fisherman's long term goals, but this relationship inverts once there are lots of fishermen optimizing for short term gains and the fish population collapses.

**Coordination Failure** occurs when agents with mutually compatible goals fail to coordinate. For example, due to incomplete models of other agent's goals and capabilities, two agents sharing a goal might compete for a resource even though one of them is strictly better at converting the resource into progress towards their goal.

**Adversarial optimization** is when an agent **O** steers the world into states where **V**'s proxy goal is positively correlated with **O**'s goal. For example, one could exploit investors who use short term volatility as a proxy for risk by selling them instruments that are not very volatile but still risky.

**Input Spoofing** is the act of one agent manipulating another learning agent's model, either by manufacturing false evidence or by filtering the received evidence systematically, as arguably happened with [Microsoft’s Tay](https://en.wikipedia.org/wiki/Tay_(bot)).

Finally, **Goal co-option** happens when agent **O** has (partial) control over the hardware agent **V** runs or relies on. This way, **O** can either modify the reward signal **V** receives to change what **V** optimizes for, or it can directly change **V**'s outputs.

The difficulties in precisely modelling other sophisticated agents and other concerns related to embedded agency make it hard to completely avoid these failure modes with current methods. Slowing down the deployment of AI systems and focussing on the mitigation of the discussed failure modes might prevent limited near term catastrophes, which in turn might cause a slowdown of further deployment and prioritization of safety.

I like that this paper subdivides failure modes that can happen in multiparty optimization into several clear categories and provides various models and examples for each of them. I am unsure about the conclusion: on one hand, slowing down deployment to improve the safety of contemporary systems seems very sensible. On the other hand, it seems like there would be some failures of limited scope that are hard to reproduce "in the lab". Widely deployed AI systems might provide us with valuable empirical data about these failures and improve our understanding of the failure modes in general. I guess ideally there would be differential deployment with rapid deployment in noncritical areas like managing local parking lots, but very slow deployment for critical infrastructure.

**Rohin's opinion:** I'm particularly interested in an analysis of how these kinds of failures affect existential risk. I'm not sure if David believes they are relevant for x-risk, but even if so the arguments aren't presented in this paper.


The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models

Reward hacking occurs when RL agents exploit the difference between a true reward and a proxy. Reward hacking has been <@observed in practice@>(@Specification gaming examples in AI@), and as reinforcement learning agents are trained with better algorithms, more data, and larger policies, they are at increased risk of overfitting their proxy objectives. However, reward hacking has not yet been systematically studied.

This paper fills this gap by constructing four example environments with a total of nine proxy rewards to investigate how reward hacking changes as a function of optimization power. They increase optimization power in several different ways, such as increasing the size of the neural net, or providing the model with more fine-grained observations.

Overall, the authors find that reward hacking occurs in five of the nine cases. Moreover, the authors observed phase transitions in four of these cases. These are stark transitions where a moderate increase in optimization power leads to a drastic increase in reward hacking behavior. This poses a challenge in monitoring the safety of ML systems. To address this the authors suggest performing anomaly detection to notice reward hacking and offer several baselines.

It is good to see an attempt at formalizing reward hacking. The experimental contributions are interesting and the anomaly detection method seems reasonable. However, the proxy rewards chosen to represent reward hacking are questionable. In my opinion, these rewards are obviously 'wrong' so it is less surprising that they result in undesired behavior. I look forward to seeing more comprehensive experiments on this subject.

**Rohin’s opinion:** Note that on OpenReview, the authors say that one of the proxy rewards (maximize average velocity for the driving environment) was actually the default and they only noticed it was problematic after they had trained large neural nets on that environment. I do agree that future proxy objectives will probably be less clearly wrong than most of the ones in this paper.


Clarifying “What failure looks like” (part 1)

The first scenario outlined in <@What failure looks like@> stems from a failure to specify what we actually want, so that we instead build AI systems that pursue proxies of what we want instead. As AI systems become responsible for more of the economy, human values become less influential relative to the proxy objectives the AI systems pursue, and as a result we lose control over the future. This post aims to clarify whether such a scenario leads to _lock in_, where we are stuck with the state of affairs and cannot correct it to get “back on course”. It identifies five factors which make this more likely:

1. _Collective action problems:_ Many human institutions will face competitive (short-term) pressures to deploy AI systems with bad proxies, even if it isn’t in humanity’s long-term interest.
2. _Regulatory capture:_ Influential people (such as CEOs of AI companies) may benefit from AI systems that optimize proxies, and so oppose measures to fix the issue (e.g. by banning such AI systems).
3. _Ambiguity:_ There may be genuine ambiguity about whether it is better to have these AI systems that optimize for proxies, even from a long-term perspective, especially because all clear and easy-to-define metrics will likely be going up (since those can be turned into proxy objectives).
4. _Dependency:_ AI systems may become so embedded in society that society can no longer function without them.
5. _Opposition:_ The AI systems themselves may oppose any fixes we propose.

We can also look at historical precedents. Factors 1-3 have played an important role in climate change, though if it does lead to lock in, this will be “because of physics”, unlike the case with AI. The agricultural revolution, which arguably made human life significantly worse, still persisted thanks to its productivity gains (factor 1) and the loss of hunter-gathering skills (factor 4). When the British colonized New Zealand, the Maori people lost significant control over their future, because each individual chief needed guns (factor 1), trading with the British genuinely made them better off initially (factor 3), and eventually the British turned to manipulation, confiscation and conflict (factor 5).

With AI in particular, we might expect that an increase in misinformation and echo chambers exacerbates ambiguity (factor 3), and that due to its general-purpose nature, dependency (factor 4) may be more of a risk.

The post also suggests some future directions for estimating the _severity_ of lock in for this failure mode.

I think this topic is important and the post did it justice. I feel like factors 4 and 5 (dependency and opposition) capture the reasons I expect lock in, with factors 1-3 as less important but still relevant mechanisms. I also really liked the analogy with the British colonization of New Zealand -- it felt like it was in fact quite analogous to how I’d expect this sort of failure to happen.


Problems in AI Alignment that philosophers could potentially contribute to

Exactly what it says. The post is short enough that I'm not going to summarize it -- it would be as long as the original.


Agency Failure AI Apocalypse?

This is a response to <@More realistic tales of doom@>, arguing that the scenarios described in the post are unrealistic given what we know about principal-agent problems. In a typical principal-agent problem, the principal doesn't know everything about the agent, and the agent can use this fact to gain "agency rents" where it can gain extra value for itself, or there could be an "agency failure" where the principal doesn't get as much as they want. For example, an employee might spend half of their day browsing the web, because their manager can't tell that that's what they are doing. Our economic literature on principal-agent problems suggests that agency problems get harder with more information asymmetry, more noise in outcomes, etc. but not with smarter agents, and in any case we typically see limited agency rents and failures. So, it's unlikely that the case for AI will be any different, and while it's good to have a couple of people keeping an eye on the problem, it's not worth the large investment of resources from future-oriented people that we currently see.

I have a bunch of complicated thoughts on this post, many of which were said in Paul's comment reply to the post, but I'll say a few things. Firstly, I think that if you want to view the AI alignment problem in the context of the principal-agent literature, the natural way to think about it is with the principal being less rational than the agent. I claim that it is at least conceivable that an AI system could make humans worse off, but the standard principal-agent model cannot accommodate such a scenario because it assumes the principal is rational, which means the principal always does at least as well as not ceding any control to the agent at all. More importantly, although I'm not too familiar with the principal-agent literature, I'm guessing that the literature assumes the presence of norms, laws and institutions that constrain both the principal and the agent, and in such cases it makes sense that the loss that the principal could incur would be bounded -- but it's not obvious that this would hold for sufficiently powerful AI systems.


Imitation learning considered unsafe?

We might hope that using imitation learning to mimic a corrigible human would be safe. However, this would involve mimicking the human's planning process. It seems fairly likely that slight errors in the imitation of this process could lead to the creation of a goal-directed planning process that does dangerous long-term optimization.

This seems pretty similar to the problem of inner optimizers, in which while searching for a good policy for some task T on training distribution D, you end up finding a consequentialist agent that is optimizing some utility function that leads to good performance on D. That agent will have all the standard dangers of goal-directed optimization out of distribution.


"Unsupervised" translation as an (intent) alignment problem

We have previously seen that a major challenge for alignment is that our models may learn <@inaccessible information@>(@Inaccessible information@) that we cannot extract from them, because we do not know how to provide a learning signal to train them to output such information. This post proposes unsupervised translation as a particular concrete problem to ground this out.

Suppose we have lots of English text, and lots of Klingon text, but no translations from English to Klingon (or vice versa), and no bilingual speakers. If we train GPT on the text, it will probably develop a good understanding of both English and Klingon, such that it “should” have the ability to translate between the two (at least approximately). How can we get it to actually (try to) do so? Existing methods (both in unsupervised translation and in AI alignment) do not seem to meet this bar.

One vague hope is that we could train a helper agent such that a human can perform next-word prediction on Klingon with the assistance of the helper agent, using a method like the one in [Learning the prior](https://www.alignmentforum.org/posts/SL9mKhgdmDKXmxwE4/learning-the-prior) ([AN #109](https://mailchi.mp/ee62c1c9e331/an-109teaching-neural-nets-to-generalize-the-way-humans-would)).


Steven Pinker and Stuart Russell on the Foundations, Benefits, and Possible Existential Threat of AI

Lucas Perry
Steven Pinker and Stuart Russell

Despite their disagreements on AI risk, Stuart and Steven agree on quite a lot. They both see the development of AI as depending on many historical ideas. They are both particularly critical of the idea that we can get general intelligence by simply scaling up existing deep learning models, citing the need for reasoning, symbol manipulation, and few-shot learning, which current models mostly don’t do. They both predict that we probably won’t go extinct from superintelligent AI, at least in part because we’ll notice and fix any potential failures, either via extensive testing or via initial failures that illustrate the problem.

On the AI risk side, while they spent a lot of time discussing it, I’ll only talk about the parts where it seems to me that there is a real disagreement, and not mention anything else. Steven’s position against AI risk seems to be twofold. First, we are unlikely to build superintelligent AI soon, and so we should focus on other clear risks like climate change. In contrast, Stuart thinks that superintelligent AI is reasonably likely by the end of the century and thus worth thinking about. Second, the idea of building a super-optimizer that focuses on a single goal is so obviously bad that AI researchers will obviously not build such a thing. In contrast, Stuart thinks that goal-directed systems are our default way of modeling and building intelligent systems. It seemed like Steven was particularly objecting to the especially simplistic goals used in examples like maximizing paperclips or curing cancer, to which Stuart argued that the problem doesn’t go away if you have multiple goals, because there will always be some part of your goal that you failed to specify.

Steven also disagrees with the notion of intelligence that is typically used by AI risk proponents, saying “a super-optimizer that pursued a single goal is self-evidently unintelligent, not superintelligent”. I don’t get what he means by this, but it seems relevant to his views.

Unsurprisingly I agreed with Stuart’s responses, but nevertheless I found this illuminating, especially in illustrating the downsides of examples with simplistic goals. I did find it frustrating that Steven didn’t respond to the point about multiple goals not helping, since that seemed like a major crux, though they were discussing many different aspects and that thread may simply have been dropped by accident.


Metaphilosophical competence can't be disentangled from alignment

Would you be comfortable taking a single human, and making them a quadrillion times more powerful?

I am curious to see people's answers to this, I think it might be a good question to reveal major differences in worldviews between optimistic and pessimistic safety researchers.


Specification gaming: the flip side of AI ingenuity

Victoria Krakovna
Jonathan Uesato
Vladimir Mikulik
Matthew Rahtz
Tom Everitt
Ramana Kumar
Zac Kenton
Jan Leike
Shane Legg

This post on the DeepMind website explains the concept of <@specification gaming@>(@Specification gaming examples in AI@), and illustrates three problems that arise within it. First and most obviously, we need to capture the human concept of a given task in a reward function. Second, we must design agents without introducing any mistaken implicit assumptions (e.g. that the physics simulation is accurate, when it isn't). Finally, we need to ensure that agents don't tamper with their reward functions.


How the Enlightenment Ends

This is an article about the dangers of AI written by a non-technologist, hitting some points that are relatively familiar.

While there are many points that I disagree with (eg. "what [AIs] do uniquely is not thinking as heretofore conceived and experienced. Rather, it is unprecedented memorization and computation"), overall there was a surprising amount of familiar material said in a different way (such as explainability and unintended consequences).


When Bots Teach Themselves How To Cheat

A media article about specification gaming in AI that I actually just agree with, and it doesn't even have a Terminator picture!


Aligning AI to Human Values means Picking the Right Metrics

There has been a lot of attention recently on the flaws of recommender systems, especially when optimizing for simple metrics like engagement -- an example of what we might call "narrow value alignment". This post reconstructs how Facebook and YouTube have been incorporating better metrics into their algorithms from 2015 and 2017 respectively. For example, Facebook found that academic research suggested that well-being was improved by "meaningful social interactions", but worsened by passive consumption of content. As a result, they changed the metric for the recommendation algorithm to better track this. How did they measure it? It seems that they simply asked a survey of thousands of people what the most meaningful content was (both on and off Facebook), and used this to train a model to predict "meaningful interactions". They estimated that this resulted in a 5% decrease in time spent on Facebook, at least in the short term. The story with YouTube is similar, though sparser on details (and it's not clear if there was input from end users in YouTube's case).

The author then contrasts this sort of narrow value alignment with AGI alignment. His main take is that narrow alignment should be easier to address, since we can learn from how existing systems behave in the real world, and the insights we gain may be critical for AGI alignment. I'll end with a quote from the conclusion: "My argument is not so much that one should use AI to optimize for well-being. Rather, we live in a world where large-scale optimization is already happening. We can choose not to evaluate or adjust these systems, but there is little reason to imagine that ignorance and inaction would be better."

Even though I often feel like an <@optimist@>(@Conversation with Rohin Shah@) about incentives towards alignment, even I was surprised to see the amount of effort that it seems Facebook has put into trying to align its recommendation algorithm with well-being. To the extent that the recommendation algorithm is still primarily harmful (which might be true or false, idk), this suggests to me that it might just be really hard to give good recommendations given the sparse feedback you get. Of course, there are more cynical explanations, e.g. Facebook just wants to look like they care about well-being, but if they really cared they could do way better. I lean towards the first explanation, but it's very hard to distinguish between these hypotheses.

While this post claimed that narrow value alignment should be easier than AGI alignment, I'm actually not so sure. With AGI alignment, you have the really powerful assumption that the AI system you are trying to align is _intelligent_: this could plausibly help a lot. For example, maybe the recommender systems that Facebook is using are just incapable of predicting what will and won't improve human well-being, in which case narrow alignment is doomed. This wouldn't be the case with an AGI (depending on your definition of AGI) -- it should be capable of doing at least as well as humans do. The challenge is in ensuring that the AI systems are actually <@motivated@>(@Clarifying "AI Alignment"@) to do so, not whether they are capable of doing so; with narrow alignment you need to solve both problems.


How Much Do Recommender Systems Drive Polarization?

There is a common worry that social media (and recommender systems in particular) are responsible for increased polarization in recent years. This post delves into the evidence for this claim. By “polarization”, we mostly mean _affective polarization_, which measures how positive your feelings towards the opposing party are (though we also sometimes mean _issue polarization_, which measures the correlation between your opinions on e.g. gun control, abortion, and taxes). The main relevant facts are:

1. Polarization in the US has increased steadily since 1980 (i.e. pre-internet), though arguably there was a slight increase from the trend around 2016.
2. Since 2000, polarization has only increased in some Western countries, even though Internet use has increased relatively uniformly across countries.
3. Polarization in the US has increased most in the 65+ age group (which has the least social media usage).

(2) could be partly explained by social media causing polarization only in two-party systems, and (3) could be explained by saying that social media changed the incentives of more traditional media (such as TV) which then increased polarization in the 65+ age group. Nevertheless, overall it seems like social media is probably not the main driver of increased polarization. Social media may have accelerated the process (for instance by changing the incentives of traditional media), but the data is too noisy to tell one way or the other.

I’m glad to see a simple summary of the evidence we currently have on the effects of social media on polarization. I feel like for the past year or two I’ve constantly heard people speculating about massive harms and even existential risks based on a couple of anecdotes or armchair reasoning, without bothering to check what has actually happened; whenever I talk to someone who seems to have actually studied the topic in depth, it seems they think that there are problems with recommender systems, but they are different from what people usually imagine. (The post also notes a reason to expect our intuitions to be misguided: we are unusual in that we get most of our news online; apparently every age group, starting from 18-24, gets more news from television than online.)

Note that there have been a few pieces arguing for these harms; I haven't sent them out in the newsletter because I don't find them very convincing, but you can find links to some of them along with my thoughts [here](https://www.alignmentforum.org/posts/TmHRACaxXrLbXb5tS/rohinmshah-s-shortform?commentId=EAKEfPmP8mKbEbERv).


Beyond Engagement: Aligning Algorithmic Recommendations With Prosocial Goals

To decide what item to show a user, a recommender system needs to have some metric by which to rank items. Since this metric must usually be automated, it often contains in large part some operationalization of “engagement”. Unfortunately, this metric may not be able to differentiate between clickbait or extremist content on the one hand, and actually valuable posts on the other. In a workshop on the topic, participants brainstormed four main approaches for improvement:

1. **Build better controls:** Offer users more and better ways to control their feed.
2. **Develop standardized survey-based metrics:** Surveys should be able to get a significantly higher quality signal to optimize than engagement.
3. **Pay users for better data,** such as survey data.
4. **Recommend feeds, not items:** If we rank items individually, it is quite likely that all the posts of the same type (e.g. controversial posts) will get high scores. By ranking entire feeds, we can also optimize for the diversity of items within the feed.
5. **Incentivize the creation of different feeds,** so that users can choose which ones they prefer all things considered.


Playing hard exploration games by watching YouTube

Yusuf Aytar
Tobias Pfaff
David Budden
Tom Le Paine
Ziyu Wang
Nando de Freitas

There are many YouTube videos demonstrating how to play levels of eg. Montezuma's Revenge. Can we use these demonstrations to solve the hard exploration tasks in Atari? One challenge is that the videos have slightly different visual properties (like color and resolution). They propose to learn a shared feature space by using an auxiliary loss where the network must predict the number of timesteps between two frames of a video, or to predict the delay between a video and audio clip from the same trajectory. Using this shared feature space, they can define a reward function that encourages the agent to take trajectories whose features match those of the demonstrations. In experiments they exceed human performance on Atari games with hard exploration problems.

It seems to me that this is how we'll have to solve exploration in practice if we don't want to have a huge sample complexity, though I know other researchers are optimistic about solving exploration using curiosity or diversity. It's pretty exciting that they could use a source of data that was already present in the real world.


Dota 2 with Large Scale Deep Reinforcement Learning

OpenAI
Christopher Berner
Greg Brockman
Brooke Chan
Vicki Cheung
Przemysław Dębiak
Christy Dennison
David Farhi
Quirin Fischer
Shariq Hashme
Chris Hesse
Rafal Józefowicz
Scott Gray
Catherine Olsson
Jakub Pachocki
Michael Petrov
Henrique Pondé de Oliveira Pinto
Jonathan Raiman
Tim Salimans
Jeremy Schlatter
Jonas Schneider
Szymon Sidor
Ilya Sutskever
Jie Tang
Filip Wolski
Susan Zhang

In April, <@OpenAI Five@>(@How to Train Your OpenAI Five@) defeated the world champion Dota 2 team, OG. This paper describes its training process. OpenAI et al. hand-engineered the reward function as well as some features, actions, and parts of the policy. The rest of the policy was trained using PPO with an LSTM architecture at a massive scale. They trained this in a distributed fashion as follows:

-   The *Controller* receives and distributes the updated parameters.
-   The *Rollout Worker CPUs* simulate the game, send observations to the *Forward Pass GPUs* and publish samples to the *Experience Buffer*.
-   The *Forward Pass GPUs* determine the actions to use and send them to the *Rollout Workers*.
-   The *Optimizer GPUs* sample experience from the *Experience Buffer*, calculate gradient updates, and then publish updated parameters to the *Controller*.

The model trained over 296 days. In that time, OpenAI needed to adapt it to changes in the code and game mechanics. This was done via model “surgery”, in which they would try to initialize a new model to maintain the same input-output mapping as the old one. When this was not possible, they gradually increased the proportion of games played with the new version over time.

I feel similarly to my opinion on <@AlphaStar@>(@AlphaStar: Grandmaster level in StarCraft II using multi-agent reinforcement learning@) here. The result is definitely impressive and a major step up in complexity from shorter, discrete games like chess or go. However, I don’t see how the approach of just running PPO at a large scale brings us closer to AGI because we can’t run massively parallel simulations of real world tasks. Even for tasks that can be simulated, this seems prohibitively expensive for most use cases (I couldn’t find the exact costs, but I’d estimate this model cost tens of millions of dollars). I’d be quite excited to see an example of deep RL being used for a complex real world task without training in simulation.


Learning Latent Plans from Play

Corey Lynch
Mohi Khansari
Ted Xiao
Vikash Kumar
Jonathan Tompson
Sergey Levine
Pierre Sermanet

This paper collects unsupervised data of humans playing with robotic control systems, and uses that data to thread a needle between two problems in learning. One problem is that per-task demonstration data is costly, especially as number of tasks grows; the other is that randomly sampled control actions will rarely stumble across complex motor tasks in ways that allow robots to learn. The authors argue that human play data is a good compromise because humans at play tend to explore different ways of manipulating objects in ways that give robots nuggets of useful information like "how do I move this block inside a drawer", which can be composed into more complicated and intentional tasks.

The model works by learning to produce vectors that represent plans (or sequences of actions), and jointly learning to decode those vectors into action sequences. This architecture learns to generate plan vectors by using an autoencoder-like structure that uses KL divergence to align (1) a distribution of plan vectors predicted from the start and end state of a window of play data, and (2) a distribution of plan vectors predicted by looking back at all the actions taken in that window. Because we're jointly learning to unroll the (2) lookback-summarized vector such that it matches the actions actually taken, we'll ideally end up with a system that can take in a given plan vector and produce a sequence of actions to execute that plan. And, because we're learning to predict a vector that aligns with actions successfully taken to get to an end state from a starting one, the model at test time should be able to produce a play vector corresponding to feasible actions that will get it from its current state to a goal state we'd like it to reach.  The authors found that their Play-trained model was able to outperform single-task models on a range of manipulation tasks, even though those single-task models were trained with explicit demonstrations of the task.

I really liked this paper: it was creative in combining conceptual components from variational methods and imitation learning, and it was pragmatic in trying to address the problem of how to get viable human-demonstration data in a way that avoids having to get distinct datasets for a huge set of different discrete tasks.


AlphaStar: Mastering the Real-Time Strategy Game StarCraft II

The AlphaStar system from DeepMind has beaten top human pros at StarCraft. You can read about the particular details of the matches in many sources, such as the blog post itself, this [Vox article](https://www.vox.com/future-perfect/2019/1/24/18196177/ai-artificial-intelligence-google-deepmind-starcraft-game), or [Import AI](https://jack-clark.net/2019/01/28/import-ai-131-ibm-optimizes-ai-with-ai-via-neunets-amazon-reveals-its-scout-delivery-robot-google-releases-300k-natural-questions-dataset/). The quick summary is that while there are some reasons you might not think it is conclusively superhuman yet (notably, it only won when it didn't have to manipulate the camera, and even then it may have had short bursts of very high actions per minute that humans can't do), it is clearly extremely good at StarCraft, both at the technically precise micro level and at the strategic macro level.

I want to focus instead on the technical details of how AlphaStar works. The key ideas seem to be a) using imitation learning to get policies that do something reasonable to start with and b) training a population of agents in order to explore the full space of strategies and how to play against all of them, without any catastrophic forgetting. Specifically, they take a dataset of human games and train various agents to mimic humans. This allows them to avoid the particularly hard exploration problems that happen when you start with a random agent. Once they have these agents to start with, they begin to do population-based training, where they play agents against each other and update their weights using an RL algorithm. The population of agents evolves over time, with well-performing agents splitting into two new agents that diversify a bit more. Some agents also have auxiliary rewards that encourage them to explore different parts of the strategy space -- for example, an agent might get reward for building a specific type of unit. Once training is done, we have a final population of agents. Using their empirical win probabilities, we can construct a Nash equilibrium of these agents, which forms the final AlphaStar agent. _(Note: I'm not sure if at the beginning of the game, one of the agents is chosen according to the Nash probabilities, or if at each timestep an action is chosen according to the Nash probabilities. I would expect the former, since the latter would result in one agent making a long-term plan that is then ruined by a different agent taking some other action, but the blog post seems to indicate the latter -- with the former, it's not clear why the compute ability of a GPU restricts the number of agents in the Nash equilibrium, which the blog posts mentions.)_

There are also a bunch of interesting technical details on how they get this to actually work, which you can get some information about in this [Reddit AMA](https://www.reddit.com/r/MachineLearning/comments/ajgzoc/we_are_oriol_vinyals_and_david_silver_from/). For example, "we included a policy distillation cost to ensure that the agent continues to try human-like behaviours with some probability throughout training, and this makes it much easier to discover unlikely strategies than when starting from self-play", and "there are elements of our research (for example temporally abstract actions that choose how many ticks to delay, or the adaptive selection of incentives for agents) that might be considered “hierarchical”". But it's probably best to wait for the journal publication (which is currently in preparation) for the full details.

I'm particularly interested by this [Balduzzi et al paper](https://arxiv.org/abs/1901.08106) that gives some more theoretical justification for the population-based training. In particular, the paper introduces the concept of "gamescapes", which can be thought of as a geometric visualization of which strategies beat which other strategies. In some games, like "say a number between 1 and 10, you get reward equal to your number - opponent's number", the gamescape is a 1-D line -- there is a scalar value of "how good a strategy is", and a better strategy will beat a weaker strategy. On the other hand, rock-paper-scissors is a cyclic game, and the gamescape looks like a triangle -- there's no strategy that strictly dominates all other strategies. Even the Nash strategy of randomizing between all three actions is not the "best", in that it fails to exploit suboptimal strategies, eg. the strategy of always playing rock. With games that are even somewhat cyclic (such as StarCraft), rather than trying to find the Nash equilibrium, we should try to explore and map out the entire strategy space. The paper also has some theoretical results supporting this that I haven't read through in detail.

I don't care very much about whether AlphaStar is superhuman or not -- it clearly is very good at StarCraft at both the micro and macro levels. Whether it hits the rather arbitrary level of "top human performance" is not as interesting as the fact that it is anywhere in the ballpark of "top human performance".

It's interesting to compare this to [OpenAI Five](https://blog.openai.com/openai-five/) ([AN #13](https://mailchi.mp/8234356e4b7f/alignment-newsletter-13)). While OpenAI solved the exploration problem using a combination of reward shaping and domain randomization, DeepMind solved it by using imitation learning on human games. While OpenAI relied primarily on self-play, DeepMind used population-based training in order to deal with catastrophic forgetting and in order to be robust to many different strategies. It's possible that this is because of the games they were playing -- it's plausible to me that StarCraft has more rock-paper-scissors-like cyclic mechanics than Dota, and so it's more important to be robust to many strategies in StarCraft. But I don't know either game very well, so this is pure speculation.

Exploring the full strategy space rather than finding the Nash equilibrium seems like the right thing to do, though I haven't kept up with the multiagent RL literature so take that with a grain of salt. That said, it doesn't seem like the full solution -- you also want some way of identifying what strategy your opponent is playing, so that you can choose the optimal strategy to play against them.

I often think about how you can build AI systems that _cooperate_ with humans. This can be significantly harder: in competitive games, if your opponent is more suboptimal than you were expecting, you just crush them even harder. However, in a cooperative game, if you make a bad assumption about what your partner will do, you can get significantly worse performance. (If you've played Hanabi, you've probably experienced this.) Self-play does not seem like it would handle this situation, but this kind of population-based training could potentially handle it, if you also had a method to identify how your partner is playing. (Without such a method, you would play some generic strategy that would hopefully be quite robust to playstyles, but would still not be nearly as good as being able to predict what your partner does.)

[Open-ended Learning in Symmetric Zero-sum Games](https://arxiv.org/abs/1901.08106), [AMA with AlphaStar creators and pro players](https://www.reddit.com/r/MachineLearning/comments/ajgzoc/we_are_oriol_vinyals_and_david_silver_from), and [Vox: StarCraft is a deep, complicated war strategy game. Google’s AlphaStar AI crushed it.](https://www.vox.com/future-perfect/2019/1/24/18196177/ai-artificial-intelligence-google-deepmind-starcraft-game)


Spinning Up in Deep RL

OpenAI has released an educational resource aimed to help software engineers become skilled at deep reinforcement learning. It includes simple implementations of many deep RL algorithms (as opposed to the relatively complex, highly optimized implementations in Baselines), educational exercises, documentation, and tutorials. OpenAI will host a workshop on the topic at their headquarters on Feb 2nd, and are also planning to hold a workshop at CHAI some time in early 2019.

I know that a lot of effort has gone into this project, and I expect that as a result this is probably the best educational resource on deep RL out there. The main other resource I know of is the [deep RL bootcamp](https://sites.google.com/view/deep-rl-bootcamp/), which probably supplements this resource nicely, especially with the lectures (though it is a year out of date).


Preserving Outputs Precisely while Adaptively Rescaling Targets

When an agent is trained on multiple tasks across which the sizes of rewards vary greatly, it usually focuses on tasks which provide the largest or most frequent rewards at the expense of performance on others. Previous work dealt with this by clipping rewards outside a certain range, but this changes the optimal policy (eg. in Pacman, eating pellets is just as rewarding as eating ghosts). This paper uses PopArt (introduced in [this 2016 paper](https://arxiv.org/abs/1602.07714)) to normalise rewards from each task before using them to update the policy in an actor-critic RL algorithm. The authors use PopArt to train a single IMPALA agent which can play all 57 Atari games, achieving a median performance slightly higher than human performance.

To delve into more detail about PopArt, let's consider training a policy with an actor-critic algorithm. In this case, we need a critic that produces estimates of values V, and an actor that produces probabilities of actions. Both of these networks are trained by taking gradients of their outputs, and weighting them based on the observed rewards. Now, a key empirical fact about deep learning is that it works better if all the things are normalized, especially the gradients. (If nothing else, it makes it easier to choose the learning rate.) For the actor, this is easy -- probabilities are already normalized, and the weight of gradient is proportional to the reward, so we can just rescale the weight of the gradient based on the mean and standard deviation of the rewards we have observed so far. This is a bit harder for the critic, since it has to predict values, so we have to normalize both the outputs and the gradient weights. We can normalize the gradient weights in the same way as before. However, normalizing the outputs is tricky, because as time goes on the means and standard deviations change. To do this, at every timestep we modify the weights of the critic that is equivalent to unnormalizing based on the old statistics and then normalizing based on the new statistics. This gives the PopArt method.

Here's a simple example where I butcher types, ignore the difference between states and trajectories, and throw away the standard deviation. Suppose the first reward we see is 10, so we say that our mean is 10 and train our net to output a normalized reward of 0 for this state and action. Then, we see a reward of 100, so we update our mean to 55. On our previous (state, action) pair, we still output a normalized reward of 0, which now corresponds to a real reward of 55, even though it should correspond to 10! We then do the unnormalize-and-renormalize trick. After unnormalization, the critic would output 10, and after renormalization, the network would output -45, which when combined with the mean of 55 would give us the desired 10 reward.

This is an impressive result, since it's the first time a single agent has performed so well on a range of Atari games. It doesn't seem to have required any novel techniques except for a straightforward extension of PopArt to the multi-task setting, but this is still interesting since the results from the previous PopArt paper were very mixed (performance increased and decreased dramatically on different games, with the average remaining roughly stable).

One confusing aspect was that PopArt still benefitted slightly from being trained with reward clipping (110% vs. 101% in the unclipped case), even though the point of PopArt was to normalize rewards so that clipping wasn't necessary. I'm assuming the clipping happens after PopArt normalization, since if it happens before then you lose information as in the Pacman example. In this case, maybe it's that the reward distribution is fat-tailed, and so even after normalization there could be some extreme rewards that after normalization are still large enough that they would cause updates that are too large, and clipping alleviates this problem.


Visual Reinforcement Learning with Imagined Goals

This is a blog post explaining a paper by the same name that I covered in [AN #16](https://mailchi.mp/f2950ed2ac4b/alignment-newsletter-16). It's particularly clear and well-explained, and I continue to think the idea is cool and interesting. I've recopied my summary and opinion here, but you should read the blog post, it explains it very well.

[Hindsight Experience Replay](https://blog.openai.com/ingredients-for-robotics-research/) (HER) introduced the idea of accelerating learning with sparse rewards, by taking trajectories where you fail to achieve the goal (and so get no reward, and thus no learning signal) and replacing the actual goal with an "imagined" goal chosen in hindsight such that you actually achieved that goal, which means you get reward and can learn. This requires that you have a space of goals such that for any trajectory, you can come up with a goal such that the trajectory achieves that goal. In practice, this means that you are limited to tasks where the goals are of the form "reach this goal state". However, if your goal state is an image, it is very hard to learn how to act in order to reach any possible image goal state (even if you restrict to realistic ones), since the space is so large and unstructured. The authors propose to first learn a structured latent representation of the space of images using a variational autoencoder (VAE), and then use that structured latent space as the space of goals which can be achieved. They also use Q-learning instead of DDPG (which is what HER used), so that they can imagine any goal with a minibatch (s, a, s') and learn from it (whereas HER/DDPG is limited to states on the trajectory).

This is a cool example of a relatively simple yet powerful idea -- instead of having a goal space over all states, learn a good latent representation and use that as your goal space. This enables unsupervised learning in order to figure out how to use a robot to generally affect the world, probably similarly to how babies explore and learn.


Recurrent World Models Facilitate Policy Evolution

I read the [interactive version](https://worldmodels.github.io/) of the paper. The basic idea is to do model-based reinforcement learning, where the model is composed of a variational auto-encoder that turns a high-dimensional state of pixels into a low-dimensional representation, and a large RNN that predicts how the (low-dimensional) state will evolve in the future. The outputs of this model are fed into a very simple linear controller that chooses actions. Since the controller is so simple, they can train it using a black box optimization method (an evolutionary strategy) that doesn't require any gradient information. They evaluate on a racing task and on Doom, and set new state-of-the-art results. There are also other interesting setups -- for example, once you have a world model, you can train the controller completely within the world model without interacting with the outside world at all (using the number of timesteps before the episode ends as your reward function, since the world model doesn't predict standard rewards, but does predict whether the episode ends). There are a lot of cool visualizations that let you play with the models trained with their method.

I agree with [Shimon Whiteson's take](https://twitter.com/shimon8282/status/979344417961250817), which is that this method gets improvements by creating a separation of concerns between modelling the world and learning a controller for the model, and evaluating on environments where this separation mostly holds. A major challenge in RL is learning the features that are important for the task under consideration, and this method instead learns features that allow you to reconstruct the state, which could be very different, but happen to not be different in their environments. That said, I really like the presentation of the paper and the fact that they did ablation studies.


Large-Scale Study of Curiosity-Driven Learning

Yuri Burda
Harri Edwards
Deepak Pathak
Amos Storkey
Trevor Darrell and Alexei A. Efros

One major challenge in RL is how to explore the environment sufficiently in order to find good rewards to learn from. One proposed method is curiosity, in which the agent generates an internal reward for taking any transition where the outcome was surprising, where surprisal is measured as the negative log probability assigned to the outcome by the agent. In this paper, a neural net that takes as input observation features φ(x) and action a, and predicts the features of the next state observation. The mean squared error with the actual features of the next state is then a measure of the surprisal, and is used as the curiosity reward. This is equivalent to treating the output of the neural net as the mean of a Gaussian distribution with fixed variance, and defining the reward to be the negative log probability assigned to the actual next state.

This still leaves the feature function φ undetermined. They consider using pixels directly, using a CNN with randomly chosen fixed weights, learned CNN features using a variational autoencoder (VAE) (which optimize for features that are useful for reconstructing the observation), and learned CNN features using inverse dynamics (IDF) (which optimize for features that are useful for reconstructing the action, biasing the features towards aspect of the environment that the agent can control). As you might expect, pixels don't work very well. However, random features do work quite well, often beating the VAE and IDF. This can happen because the random features stay fixed, leading to more stable learning, whereas with the VAE and IDF methods the features are changing over time, and the environment distribution is changing over time (as the agent explores more of it), leading to a harder learning problem.

Typically, curiosity is combined with an external reward. In this paper, the authors evaluate how well an agent can do with _only_ curiosity and no external reward. Intuitively, in game environments designed by humans, the designer sets up a good curriculum for humans to learn, which would align well with a curiosity reward. In fact, this is what happens, with a curiosity based reward leading to great performance (as measured by the external reward) on Atari games, Super Mario, Unity mazes, and Roboschool Pong, when using random features or IDF features. (The VAE features sometimes work well but were very unstable.) They evaluate transfer between levels in Super Mario, and find that the learned features transfer in more cases than random ones. Looking at the graphs, this seems like a very small effect to me -- I'm not sure if I'd agree with the claim, but I'd want to look at the behavior in videos and what the reward function rewards before making that claim strongly. They also investigate Pong with both players being driven by curiosity, and the players become so good at rallying that they crash the emulator.

Finally, they note one downside -- in any stochastic environment, or any environment where there will be lots of uncertainty about what will happen (eg. in multiagent settings), at convergence the reward for any action will be equal to the entropy of the next state distribution. While they don't demonstrate this flaw in particular, they show a related one -- if you add a TV to a Unity maze, and an action to change the channel, then the agent learns to stand in front of the TV and change the channel forever, rather than solving the maze.

2nd Workshop on Meta-Learning at NeurIPS 2018

I really like these empirical papers that compare different methods and show their advantages and disadvantages. I was pretty surprised to see random features do as well as they did, especially to see that they transferred as well as learned features in one of the two cases they studied. There was of course a neural net that could learn how to use the arbitrary representation induced by the features, but then why couldn't it do the same for pixels? Perhaps the CNN was useful primarily for reducing the dimensionality of the pixels by combining nearby pixels together, and it didn't really matter how that was done since it still retains all the important information, but in a smaller vector?

I'm glad that the paper acknowledges that the good performance of curiosity is limited to environments that human designers have created. In a real world task, such as a house-cleaning robot, there are many other sources of uncertainty in the world that are unrelated to the task, and you need some form of specification to focus on it -- curiosity alone will not be enough.


Lessons Learned Reproducing a Deep Reinforcement Learning Paper

It's exactly what the title says. There were a lot of points that I can't easily summarize, but some highlights:
- Deep RL is very fiddly and tricky, and most of his time was spent debugging.
- Since each run takes a long time, it's very important to put a lot of effort into figuring out what hypotheses to test.
- The project took ~$850 of compute (~9,000 CPU hours and ~450 GPU hours)

If you do deep RL research regularly, you probably won't get too much out of it (though you might still get some handy tips on things you can do with Tensorflow), but I think everyone else should read it to get a more concrete sense of what deep RL research actually looks like and to be able to communicate more effectively with deep RL researchers.

Deep Reinforcement Learning Doesn’t Work Yet


OpenAI Five Benchmark: Results

The OpenAI Five benchmark happened last Sunday, where OpenAI Five won two matches against the human team, and lost the last one when their draft was adversarially selected. They are now planning to play at The International in a couple of weeks (dates to be finalized). That will be a harder challenge, since they will be playing against teams that play and train professionally, and so will be better at communication and coordination than the human team here.

Blitz (one of the human players) [said](https://www.reddit.com/r/MachineLearning/comments/9533g8/n_openai_five_benchmark_results/e3qrnbt/): "The only noticeable difference in the mechanical skill aspect was the hex from the Lion, but even that was sorta irrelevant to the overall game flow. Got outdrafted and outmaneuvered pretty heavily, and from a strategy perspective it was just better then us. Even with the limitations in place it still 'felt' like a dota game, against a very good team. It made all the right plays I'd expect most top tier teams to make."

On the technical side, OpenAI implemented a brute-force draft system. With a pool of 18 heroes, you get some combinatorial explosion, but there are still only ~11 million possible matchups. You can then do a simple tree search over which hero to draft, where at the leaves (when you have a full draft) you choose which leaf you want based on the win probability (which OpenAI Five already outputs). Seeing this in action, it seems to me like it's a vanilla minimax algorithm, probably with alpha-beta pruning so that they don't have to evaluate all ~159 billion nodes in the tree. (Or they could have done the full search once, hardcoded the action it comes up with for the first decision, and run the full search for every subsequent action, which would have under 10 billion nodes in the tree.)

Besides the win probabilities, there are other ways to get insight into what the model is "thinking" -- for example, by asking the model to predict where the hero will be in 6 seconds, or by predicting how many last hits / denies / kills / deaths it will have.

The model that played the benchmark has been training since June 9th. Of course, in that time they've changed many things about the system (if for no other reason than to remove many of the restrictions in the original post). This is not a thing that you can easily do -- typically you would change your model architecture, which means your old parameters don't map over to the new architecture. I've been pretty curious about how they handle this, but unfortunately the blog post doesn't go into much detail, beyond saying that they can in fact handle these kinds of "surgery" issues.

They estimate that this particular model has used 190 petaflop/s-days of compute, putting it [just below AlphaZero](https://blog.openai.com/ai-and-compute/).

I think this finally fell within my expectations, after two instances where I underestimated OpenAI Five. I expected that they would let the human team choose heroes in some limited way (~80%), that OpenAI Five would not be able to draft using just gradients via PPO (~60%), and (after having seen the first two games) that the human team would win after an adversarial draft (~70%). Of course, a draft did happen, but it was done by a tree search algorithm, not an algorithm learned using PPO.

The games themselves were pretty interesting (though I have not played Dota so take this with a grain of salt). It seemed to me like OpenAI Five had learned a particularly good strategy that plays to the advantages of computers, but hadn't learned some of the strategies and ideas that human players use to think about Dota. Since it uses the same amount of computation for each decision, it makes good decisions on all timescales, including ones where something surprising has occurred where humans would need some time to react, and also to coordinate. For example, as soon as a human hero entered within range of the bots (just to look and retreat), all of the bots would immediately unleash a barrage of attacks, killing the hero -- a move that humans could not execute, because of slower reaction times and worse communication and teamwork. Similarly, one common tactic in human gameplay is to teleport into a group of heroes and unleash an area-of-effect ability, but when they tried this against OpenAI Five, one of the bots hexed the hero as soon as he teleported in, rendering him unable to cast the spell. (That felt like the decisive moment in the first game.) On the other hand, there were some clear issues with the bots. At one point, two OpenAI bots were chasing Blitz, and Blitz used an ability that made him invisible while standing still. Any human player would have spammed area attacks, but the bots simply became confused and eventually left. Similarly, I believe (if I understood the commentary correctly) that a bot once used an ability multiple times, wasting mana, even though all uses after the first had no additional effect.

Other articles would have you believe that the games weren't even close, and if you look at the kill counts, that would seem accurate. I don't think that's actually right -- from what I understand, kills aren't as important as experience and gold, and you could see this in the human gameplay. OpenAI Five would often group most of its heroes together to push forward, which means they get less experience and gold. The human team continued to keep their heroes spread out over the map to collect resources -- and even though OpenAI Five got way more kills, the overall net worth of the two teams' heroes remained about equal for most of the early game. The big difference seemed to be that when the inevitable big confrontation between the two teams happened, OpenAI Five always came out on top. I'm not sure how, my Dota knowledge isn't good enough for that. Based on Blitz's comment, my guess is that OpenAI Five is particularly good at fights between heroes, and the draft reflects that. But I'd still guess that if you had pro human players who ceded control to OpenAI Five whenever a fight was about to happen, they would beat OpenAI Five (~70%). I used to put 80% on that prediction, but Blitz's comment updated me away from that.

One interesting thing was that the win probability seemed to be very strongly influenced by the draft, which in hindsight seems obvious. Dota is a really complicated game that is constantly tweaked to keep it balanced for humans, and even then the draft is very important. When you now introduce a new player (OpenAI Five) with very different capabilities (such as very good decision making under time pressure) and change the game conditions (such as a different pool of heroes), you should expect the game to become very imbalanced, with some teams far outshining others. And in fact we did see that Lion (the hero with the hexing ability) was remarkably useful (against humans, at least).


Variational Option Discovery Algorithms

Joshua Achiam
Harrison Edwards
Dario Amodei
Pieter Abbeel

We can hope to do hierarchical reinforcement learning by first discovering several useful simple policies (or "options") by just acting in the environment without any reward function, and then using these options as primitive actions in a higher level policy that learns to do some task (using a reward function). How could we learn the options without a reward function though? Intuitively, we would like to learn behaviors that are different from each other. One way to frame this would be to think of this as an encoder-decoder problem. Suppose we want to learn K options. Then, we can give the encoder a number in the range [1, K], have it "encode" the number into a trajectory τ (that is, our encoder is a policy), and then have a decoder take τ and recover the original number. We train the encoder/policy and decoder jointly, optimizing them to successfully recover the original number (called a _context_). Intuitively, the encoder/policy wants to have very different behaviors for each option, so that it easy for decoder to figure out the context from the trajectory τ. However, a simple solution would be for the encoder/policy to just take a particular series of actions for each context and then stop, and the decoder learns an exact mapping from final states to contexts. To avoid this, we can decrease the capacity of the decoder (i.e. don't give it too many layers), and we also optimize for the _entropy_ of the encoder/policy, which encourages the encoder/policy to be more stochastic, and so it is more likely to learn overall behaviors that can still have some stochasticity, while still allowing the decoder to decode them. It turns out that this optimization problem has a one-to-one correspondence with variational autoencoders, motivating the name "variational option discovery". To stabilize training, they start with a small K, and increase K whenever the decoder becomes powerful enough. They evaluate in Gym environments, a simulated robotic hand, and a new "Toddler" environment. They find that the scheme works well (in terms of maximizing the objective) in all environments, but that the learned behaviors no longer look natural in the Toddler environment (which is the most complex). They also show that the learned policies can be used for hierarchical RL in the AntMaze problem.

This is very similar to the recent [Diversity Is All You Need](https://arxiv.org/abs/1802.06070). DIAYN aims to decode the context from _every state_ along a trajectory, which incentivizes it to find behaviors of the form "go to a goal state", whereas VALOR (this work) decodes the context from the entire trajectory (without actions, which would make the decoder's job too easy), which allows it to learn behaviors with motion, such as "go around in a circle".

It's really refreshing to read a paper with a negative result about their own method (specifically, that the learned behaviors on Toddler do not look natural). It makes me trust the rest of their paper so much more. (A very gameable instinct, I know.) While they were able to find a fairly diverse set of options, and could interpolate between them, their experiments found that using this for hierarchical RL was about as good as training hierarchical RL from scratch. I guess I'm just saying things they've already said -- I think they've done such a great job writing this paper that they've already told me what my opinion about the topic should be, so there's not much left for me to say.


Learning Dexterity

Most current experiments with robotics work on relatively small state spaces (think 7 degrees of freedom, each a real number) and are trained in simulation. If we could throw a lot of compute at the problem, could we do significantly better? Yes! Using the same general approach as with [OpenAI Five](https://blog.openai.com/openai-five/), OpenAI has built a system called Dactyl, which allows a physical real-world dexterous hand to manipulate a block. It may not seem as impressive as the videos of humanoids running through obstacle courses, but this is way harder than your typical Mujoco environment, especially since they aim to get it working on a real robot. As with OpenAI Five, they only need a reward function (I believe not even a shaped reward function in this case), a simulator, and a good way to explore. In this setting though, "exploration" is actually domain randomization, where you randomly set parameters that you are uncertain about (such as the coefficient of friction between two surfaces), so that the learned policy is robust to distribution shift from the simulator to the real world. (OpenAI Five also used domain randomization, but in that case it was not because we were uncertain about the parameters in the simulator, but because the policy was too specialized to the kinds of characters and heroes it was seeing, and randomizing those properties exposed it to a wider variety of scenarios so it had to learn more general policies.) They use 6144 CPU cores and 8 GPUs, which is _much_ less than for OpenAI Five, but _much_ more than for a typical Mujoco environment.

They do separate the problem into two pieces -- first, they learn how to map from camera pictures to a 3D pose (using convolutional nets), and second, they use RL to choose actions based on the 3D pose. They can also get better estimates of the 3D pose using motion tracking. They find that the CNN is almost as good as motion tracking, and that the domain randomization is crucial for getting the system to actually work.

They also have a couple of sections on surprising results and things that didn't work. Probably the most interesting part was that they didn't need to use the tactile sensors to get these results. They couldn't get these sensors in simulation, so they just did without and it seems to have worked fine. It also turns out that the robot's reaction time wasn't too important -- there wasn't a big difference in changing from 80ms reaction time to 40ms reaction time; in fact, this just increased the required training time without much benefit.

Probably the most interesting part of the post is the last paragraph (italics indicates my notes): "This project completes a full cycle of AI development that OpenAI has been pursuing for the past two years: we’ve developed a new learning algorithm _(PPO)_, scaled it massively to solve hard simulated tasks _(OpenAI Five)_, and then applied the resulting system to the real world _(this post)_. Repeating this cycle at increasing scale is the primary route we are pursuing to increase the capabilities of today’s AI systems towards safe artificial general intelligence."

This is pretty exciting -- transferring a policy from simulation to the real world is notoriously hard, but it turns out that as long as you use domain randomization (and 30x the compute) it actually is possible to transfer the policy. I wish they had compared the success probability in simulation to the success probability in the real world -- right now I don't know how well the policy transferred. (That is, I want to evaluate how well domain randomization solved the distribution shift problem.) Lots of other exciting things too, but they are pretty similar to the exciting things about OpenAI Five, such as the ability to learn higher level strategies like finger pivoting and sliding (analogously, fighting over mid or 5-man push).


Capture the Flag: the emergence of complex cooperative agents

Max Jaderberg
Wojciech M. Czarnecki
Iain Dunning
Luke Harris
and Thore Graepel

DeepMind has trained FTW (For The Win) agents that can play Quake III Arena Capture The Flag from raw pixels, given _only_ the signal of whether they win or not. They identify three key ideas that enable this -- population based training (instead of self play), learning an internal reward function, and operating at two timescales (enabling better use of memory). Their ablation studies show that all of these are necessary, and in particular it even outperforms population based training with manual reward shaping. The trained agents can cooperate and compete with a wide range of agents (thanks to the population based training), including humans.

But why are these three techniques so useful? This isn't as clear, but I can speculate. Population based training works well because the agents are trained against a diversity of collaborators and opponents, which can fix the issue of instability that afflicts self-play. Operating at two timescales gives the agent a better inductive bias. They say that it enables the agent to use memory more effectively, but my story is that it lets it do something more hierarchical, where the slow RNN makes "plans", while the fast RNN executes on those plans. Learning an internal reward function flummoxed me for a while, it really seemed like that should not outperform manual reward shaping, but then I found out that the internal reward function is computed from the game points screen, not from the full trajectory. This gives it a really strong inductive bias (since the points screen provides really good features for defining reward functions) that allows it to quickly learn an internal reward function that's more effective than manual reward shaping. It's still somewhat surprising, since it's still learning this reward function from the pixels of the points screen (I assume), but more believable.

This is quite impressive, since they are learning from the binary win-loss reward signal. I'm surprised that the agents generalized well enough to play alongside humans -- I would have expected that to cause a substantial distributional shift preventing good generalization. They only had 30 agents in their population, so it seems unlikely a priori that this would induce a distribution that included humans. Perhaps Quake III is simple enough strategically that there aren't very many viable strategies, and most strategies are robust to having slightly worse allies? That doesn't seem right though.

DeepMind did a _lot_ of different things to analyze what the agents learned and how they are different from humans -- check out the [paper](https://arxiv.org/pdf/1807.01281.pdf) for details. For example, they showed that the agents are much better at tagging (shooting) at short ranges, while humans are much better at long ranges.


OpenAI Five

OpenAI has trained a team of five neural networks to play a particular set of Dota heroes in a mirror match (playing against the same set of heroes) with a few restrictions, and have started to beat amateur human players. They are aiming to beat a team of top professionals at The International in August, with the same set of five heroes, but without any other restrictions. Salient points:
- The method is remarkably simple -- it's a scaled up version of PPO with training data coming from self-play, with reward shaping and some heuristics for exploration, where each agent is implemented by an LSTM.
- There's no human data apart from the reward shaping and exploration heuristics.
- Contrary to most expectations, they didn't need anything fundamentally new in order to get long-term strategic planning. I was particularly surprised by this. Some interesting thoughts from OpenAI researchers in [this thread](https://news.ycombinator.com/item?id=17392700) -- in particular, assuming good exploration, the variance of the gradient should scale linearly with the duration, and so you might expect you only need linearly more samples to counteract this.
- They used 256 dedicated GPUs and 128,000 preemptible CPUs. A [Hacker News comment](https://news.ycombinator.com/item?id=17394150) estimates the cost at $2500 per hour, which would put the likely total cost in the millions of dollars.
- They simulate 900 years of Dota every day, which is a ratio of ~330,000:1, suggesting that each CPU is running Dota ~2.6x faster than real time. In reality, it's probably running many times faster than that, but preemptions, communication costs, synchronization etc. all lead to inefficiency.
- There was no explicit communication mechanism between agents, but they all get to observe the full Dota 2 state (_not_ pixels) that any of the agents could observe, so communication is not really necessary.
- A version of the code with a serious bug was still able to train to beat humans. Not encouraging for safety.
- Alex Irpan covers some of these points in more depth in [Quick Opinions on OpenAI Five](https://www.alexirpan.com/2018/06/27/dota-2-five.html).
- Gwern [comments](https://www.reddit.com/r/reinforcementlearning/comments/8tqzvq/openai_dota_update_ppo_lstm_reaches_amateurlevel/) as well.

I might be more excited by an approach that was able to learn from human games (which are plentiful), and perhaps finetune with RL, in order to develop an approach that could generalize to more tasks in the future, where human data is available but a simulator is not. (Given the ridiculous sample complexity, pure RL with PPO can only be used in tasks with a simulator.) On the other hand, an approach that leveraged human data would necessarily be at least somewhat specific to Dota. A dependence on human data is unlikely to get us to _general_ intelligence, whereas this result suggests that we can solve tasks that have a simulator, exploration strategy, and a dense reward function, which really is pushing the boundary on generality. This seems to be [gdb's take](https://news.ycombinator.com/item?id=17392802): "We are very encouraged by the algorithmic implication of this result — in fact, it mirrors closely the story of deep learning (existing algorithms at large scale solve otherwise unsolvable problems). If you have a very hard problem for which you have a simulator, our results imply there is a _real, practical path_ towards solving it. This still needs to be proven out in real-world domains, but it will be very interesting to see the full ramifications of this finding."


A Reinforcement Learning Potpourri

This blog post summarizes several recent papers in RL (including the data augmentation papers I summarized above, as well as [First Return Then Explore](https://arxiv.org/abs/2004.12919), the successor to <@Go-Explore@>(@Montezuma’s Revenge Solved by Go-Explore, a New Algorithm for Hard-Exploration Problems@).

The whole blog post is worth reading, but I particularly agree with his point that data augmentation generally seems like a no-brainer, since you can think of it either as increasing the size of your dataset by some constant factor, or as a way of eliminating spurious correlations that your model might otherwise learn.


Agent57: Outperforming the human Atari benchmark

Adrià Puigdomènech Badia*
Bilal Piot*
Steven Kapturowski*
Pablo Sprechmann*
Alex Vitvitskyi
Daniel Guo
Charles Blundell

This blogpost and its associated [arxiv publication](https://arxiv.org/abs/2003.13350) present Agent57, DeepMind's latest RL agent created for the purpose of achieving human-level performance in a suite of 57 Atari games. Notably, Agent57 is the first agent that is able to surpass average human performance, as measured by Human Normalized Score or HNS, on every individual game in the suite, with the same set of hyperparameters. The blogpost details the evolution of DeepMind's Atari agents from DQN up to Agent57, and the paper elaborates on the improvements made in Agent57.

Specifically, Agent57 builds on a recent agent 'Never Give Up' (NGU), which itself augments R2D2 with episodic memory for curiosity-driven exploration. Agent57 introduces (i) a new parameterization of state-action value function that decomposes into intrinsic and extrinsic rewards, and (ii) a meta-controller which selects which of its numerous distributed policies to prioritize during learning, allowing the agent to control the exploration/exploitation trade-off.

On the one hand, this work feels like the achievement of an important milestone in DeepMind's ongoing research agenda towards building more general agents. On the other hand, it has the flavour of engineered sophistry: a remarkable collection of building blocks arranged together to patch specific known weaknesses, but lacking in core insights about how to make agents more general, without, say, making them more complex.

The work is well presented and accessible, especially the blogpost that contains a snapshot of the functional development of deep reinforcement learning capabilities over time. There are several open questions from here on out; personally, I hope this progresses to a single instance of an agent that is proficient at multiple games, and to the design of agents that do not require extensive hyperparameter tuning. The scale of DeepMind's experiments continues to grow, with 256 actors, and 10s of billions of frames, suggesting that, for now, this work is only suitable for simulated environments.

Paper: Agent57: Outperforming the Atari Human Benchmark