On Catastrophic Interference in Atari 2600 Games

William Fedus*
Dibya Ghosh*
John D. Martin
Marc G. Bellemare
Yoshua Bengio
Hugo Larochelle

One common worry with deep learning is the possibility of _catastrophic interference_: as the model uses gradients to learn a new behaviour, those same gradients cause it to forget past behaviours. In model-free deep RL, this would be particularly harmful in long, sequential tasks as in hard exploration problems like Montezuma’s Revenge: after the model learns how to do the first few subtasks, as it is trying to learn the next subtask, it would “forget” the first subtasks, degrading performance. The authors set out to test this hypothesis.

If this hypothesis were true, there would be an easy way to improve performance: once you have learned to perform the first subtask, just create a brand new neural net for the next subtask, so that training for this next subtask doesn’t interfere with past learning. Since the new agent has no information about what happened in the past, and must just “pick up” from wherever the previous agent left off, it is called the Memento agent (a reference to the movie of the same name). One can then solve the entire task by executing each agent in sequence.

In practice, they train an agent until its reward plateaus. They train a new Memento agent starting from the states that the previous agent reached, and note that it reliably makes further progress in hard exploration games like Montezuma’s Revenge, and not in “steady-state” games like Pong (where you wouldn’t expect as much catastrophic interference). Of course, with the Memento agent, you get both twice the training time and twice the model size, which could explain the improvement. They compare against giving the original agent twice the compute and model capacity, and find that Memento still does significantly better. They also present some fine-grained experiments which show that for a typical agent, training on specific contexts adversely affects performance on other contexts that are qualitatively different.

I think this is pretty strong evidence that catastrophic interference is in fact a problem with the Atari games. On the other hand, <@OpenAI Five@> also has many, many subtasks, that in theory should interfere with each other, and it still seems to train well. Some guesses at how to reconcile these facts:

1) the tasks in Dota are more correlated than in (say) Montezuma’s Revenge, and so interference is less of a problem (seems plausible)
2) the policy in OpenAI Five was large enough that it could easily allocate separate capacity for various subtasks (seems unlikely, I believe the policy was relatively small), or
3) with sufficiently large-scale training, there is more “exploration” in weight-space until a configuration is found where interference doesn’t happen (seems unlikely given that large batch sizes help, since they tend to reduce weight-space exploration).


Reward-Conditioned Policies

Standard RL algorithms create a policy that maximizes a reward function; the *Reward-Conditioned Policy* algorithm instead creates a policy that can achieve a particular reward value passed in as an input. This allows the policy to be trained via supervised regression on a dataset. Each example in the dataset consists of a state, action, and either a return or an advantage, referred to as *Z*. The network then predicts the action based on the state and *Z*. The learned model is able to generalize to policies for larger returns. During training, the target value is sampled from a distribution that gradually increases so that it continues to learn higher rewards.

During evaluation, they then feed in the state and a high target value of *Z* (set one standard deviation above the average in their paper.) This enables them to achieve solid - but not state of the art - performance on a variety of the OpenAI Gym benchmark tasks. They also run ablation studies showing, among other things, that the policy is indeed accurate in achieving the target reward it aims for.

One of the dangers of training powerful AI to maximize a reward function is that optimizing the function to extreme values may no longer correlate with what we want, as in the classic paperclip maximizer example. I think RCP provides an interesting solution to that problem; if we can instead specify a good, but reasonable, value, we may be able to avoid those extreme cases. We can then gradually increase the desired reward without retraining while continuously monitoring for issues. I think there are likely flaws in the above scheme, but I am optimistic in general about the potential of finding alternate ways to communicate goals to an agent.

One piece I am still curious about is whether the policy remembers how to achieve lower rewards as its training dataset updates towards higher rewards. They show in a heatmap that the target and actual rewards do match up well, but the target rewards are all sampled quite near each other; it would be interesting to see how well the final policy generalizes to the entire spectrum of target rewards.


Procgen Benchmark

Karl Cobbe
Christopher Hesse
Jacob Hilton
John Schulman

Existing game-based benchmarks for reinforcement learners suffer from the problem that agents constantly encounter near-identical states, meaning that the agents may be overfitting and memorizing specific trajectories rather than learning a general set of skills. In an attempt to remedy this, in this post OpenAI introduces Procgen Benchmark, 16 procedurally-generated video game environments used to measure how quickly a reinforcement learning agent learns generalizable skills.

The authors conduct several experiments using the benchmark. Notably, they discover that:
- Agents strongly overfit to small training sets and need access to as many as 10,000 levels to generalize appropriately.
- After a certain threshold, training performance improves as the training set grows, counter to trends in other supervised learning tasks.
- Using a fixed series of levels for each training sample (as other benchmarks do) makes agents fail to generalize to randomly generated series of levels at test time.
- Larger models improve sample efficiency and generalization.

This seems like a useful benchmark. I find it particularly interesting that their experiment testing non-procedurally generated levels as training samples implies huge overfitting effects in existing agents trained in video-game environments.

Paper: Leveraging Procedural Generation to Benchmark Reinforcement Learning


Learning to Predict Without Looking Ahead: World Models Without Forward Prediction

One [critique](https://twitter.com/shimon8282/status/979344417961250817) of the <@World Models@>(@Recurrent World Models Facilitate Policy Evolution@) paper was that in any realistic setting, you only want to learn the features that are important for the task under consideration, while the VAE used in the paper would learn features for state reconstruction. This paper instead studies world models that are trained directly from reward, rather than by supervised learning on observed future states, which should lead to models that only focus on task-relevant features. Specifically, they use _observational dropout_ on the environment percepts, where the true state is passed to the policy with a peek probability _p_, while a neural network, **M**, generates a proxy state with probability _1 - p_. At the next time-step, **M**  takes the same input as the policy, plus the policy's action, and generates the next proxy state, which then may get passed to the controller, again with probability _1 - p_.

They investigate whether the emergent 'world model' **M** behaves like a good forward predictive model. They find that even with very low peek probability e.g. _p_ = 5%, **M** learns a good enough world model that enables the policy to perform reasonably well. Additionally, they find that world models thus learned can be used to train policies that sometimes transfer well to the real environment. They claim that the world model only learns features that are useful for task performance, but also note that interpretability of those features depends on inductive biases such as the network architecture.

This work warrants a visit for the easy-to-absorb animations and charts. On the other hand, they make a few innocent-sounding observations that made me uncomfortable because they weren't rigourously proved nor labelled as speculation, e.g. a) "At higher peek probabilities, the learned dynamics model is not needed to solve the task thus is never learned.", and b) "Here, the world model clearly only learns reliable transition maps for moving down and to the right, which is sufficient."

While this is a neat bit of work well presented, it is nevertheless still unlikely this (and most other current work in deep model-based RL) will scale to more complex alignment problems such as <@Embedded World-Models@>; these world models do not capture the notion of an agent, and do not model the agent as an entity making long-horizon plans in the environment.


Superhuman AI for multiplayer poker

In July, this paper presented the first AI that can play six-player no-limit Texas hold’em poker better than professional players. Rather than using deep learning, it works by precomputing a blueprint strategy using a novel variant of Monte Carlo linear counterfactual regret minimization, an iterative self-play algorithm. To traverse the enormous game tree, the AI buckets moves by abstracting information in the game. During play, the AI adapts its strategy by modifying its abstractions according to how the opponents play, and by performing real-time search through the game tree. It used the equivalent of $144 of cloud compute to calculate the blueprint strategy and two server grade CPUs, which was much less hardware than what prior AI game milesones required.

From what I understand, much of the difficulty of poker lies in being careful not to reveal information. For decades, computers have already had an upper hand in being silent, computing probabilities, and choosing unpredictable strategies, which makes me a bit surprised that this result took so long. Nonetheless, I found it interesting how little compute was required to accomplish superhuman play.

Let's Read: Superhuman AI for multiplayer poker


AlphaStar: Grandmaster level in StarCraft II using multi-agent reinforcement learning

<@AlphaStar@>(@AlphaStar: Mastering the Real-Time Strategy Game StarCraft II@), DeepMind’s StarCraft II AI, has now defeated a top professional player and is better than 99.8% of players. While previous versions were limited to only a subset of the game, it now plays the full game and has limitations on how quickly it can take actions similar to top human players. It was trained initially via supervised learning on human players and then afterwards trained using RL.

A challenge in learning StarCraft via self-play is that strategies exhibit non-transitivity: Stalker units beat Void Rays, Void Rays beat Immortals, but Immortals beat Stalkers. This can lead to training getting stuck in cycles. In order to avoid this, they set up a League of exploiter agents and main agents. The exploiter agents train only against the current iteration of main agents, so they can learn specific counter-strategies. The main agents then train against a mixture of current main agents, past main agents, and exploiters, prioritizing opponents that they have a lower win rate against.

I think this is a very impressive display of how powerful current ML methods are at a very complex game. StarCraft poses many challenges that are not present in board games such as chess and go, such as limited visibility, a large state and action space, and strategies that play out over very long time horizons. I found it particularly interesting how they used imitation learning and human examples to avoid trying to find new strategies by exploration, but then attained higher performance by training on top of that.

I do believe progress on games is becoming less correlated with progress on AGI. Most of the key innovations in this paper revolve around the League training, which seems quite specific to StarCraft. In order to continue making progress towards AGI, I think we need to focus on being able to learn in the real world on tasks that are not as easy to simulate.

Paper: Grandmaster level in StarCraft II using multi-agent reinforcement learning


Deep Dynamics Models for Dexterous Manipulation

Anusha Nagabandi
Kurt Konoglie
Sergey Levine
Vikash Kumar

For hard robotic tasks like manipulating a screwdriver, model-free RL requires large amounts of data that are hard to generate with real-world hardware. So, we might want to use the more sample-efficient model-based RL, which has the additional advantage that the model can be reused for similar tasks with different rewards. This paper uses an ensemble of neural networks to predict state transitions, and plans by sampling trajectories for different policies. With this, they train a real anthropomorphic robot hand to be able to rotate two balls in its hand somewhat reliably within a few hours. They also trained for the same task in a simulation and were able to reuse the resulting model to move a single ball to a target location.

The videos look impressive, even though the robot hand still has some clunkiness to it. My intuition is that model-based approaches can be very useful in robotics and similar domains, where the randomness in transitions can easily be approximated by Gaussians. In other tasks where transitions follow more complicated, multimodal distributions, I am more sceptical.


Let's Discuss OpenAI's Rubik's Cube Result

This post makes many points about <@OpenAI's Rubik's cube result@>(@Solving Rubik’s Cube with a Robot Hand@), but I'm only going to focus on two. First, the result is a major success for OpenAI's focus on design decisions that encourage long-term research success. In particular, it relied heavily on the engineering-heavy model surgery and policy distillation capabilities that allow them to modify e.g. the architecture in the middle of a training run (which we've seen with <@OpenAI Five@>(@OpenAI Five Benchmark: Results@)). Second, the domain randomization doesn't help as much as you might think: OpenAI needed to put a significant amount of effort into improving the simulation to get these results, tripling the number of successes on a face rotation task. Intuitively, we still need to put in a lot of effort to getting the simulation to be "near" reality, and then domain randomization can take care of the last little bit needed to robustly transfer to reality. Given that domain randomization isn't doing that much, it's not clear if the paradigm of zero-shot sim-to-real transfer is the right one to pursue. To quote the post's conclusion:  _I see two endgames here. In one, robot learning reduces to building rich simulators that are well-instrumented for randomization, then using ludicrous amounts of compute across those simulators. In the other, randomization is never good enough to be more than a bootstrapping step before real robot data, no matter what the compute situation looks like. Both seem plausible to me, and we’ll see how things shake out._

As usual, Alex's analysis is spot on, and I have nothing to add beyond strong agreement.


Solving Rubik’s Cube with a Robot Hand

Historically, researchers have had limited success making general purpose robot hands. Now, OpenAI has successfully trained a pair of neural networks to solve a Rubik's cube with a human-like robot hand (the learned portion of the problem is manipulating the hand -- solving the Rubik's cube is specified via a classical algorithm). The hand is able to solve the Rubik's cube even under a variety of perturbations, including having some of its fingers tied together, or having its view of the cube partially occluded. The primary innovation presented is a new method called _Automatic Domain Randomization_ (ADR). ADR automatically generates progressively more difficult environments to train on in simulation that are diverse enough to capture the physics of the real world. ADR performs better than existing domain randomization methods, which require manually specifying randomization ranges. The post speculates that ADR is actually leading to _emergent meta-learning_, where the network learns a learning algorithm that allows itself to rapidly adapt its behavior to its environment.

My impression is that this is a very impressive robotics result, largely because the problem of transferring training in simulation to real life ("sim2real") is extremly difficult. I also think it's quite novel if as the authors hypothesize, the system is exhibiting emergent meta-learning. It's worth noting that the hand is still not quite at human-level -- in the hardest configurations, it only succeeds 20% of the time, and for most experiments, the hand gets some of the state of the cube via Bluetooth sensors inside the cube, not just via vision.

Vox: Watch this robot solve a Rubik’s Cube one-handed


Emergent Tool Use from Multi-Agent Interaction

Bowen Baker
Ingmar Kanitscheider
Todor Markov
Yi Wu
Glenn Powell
Bob McGrew
Igor Mordatch

We have such a vast diversity of organisms and behaviors on Earth because of evolution: every time a new strategy evolved, it created new pressures and incentives for other organisms, leading to new behaviors. The multiagent competition led to an _autocurriculum_. This work harnesses this effect: they design a multiagent environment and task, and then use standard RL algorithms to learn several interesting behaviors. Their task is hide-and-seek, where the agents are able to move boxes, walls and ramps, and lock objects in place. The agents find _six_ different strategies, each emerging from incentives created by the previous strategy: seekers chasing hiders, hiders building shelters, seekers using ramps to get into shelters, hiders locking ramps away from seekers, seekers surfing boxes to hiders, and hiders locking both boxes and ramps.

The hope is that this can be used to learn general skills that can then be used for specific tasks. This makes it a form of unsupervised learning, with a similar goal as e.g. <@curiosity@>(@Large-Scale Study of Curiosity-Driven Learning@). We might hope that multiagent autocurricula would do better than curiosity, because they automatically tend to use features that are important for control in the environment (such as ramps and boxes), while intrinsic motivation methods often end up focusing on features we wouldn't think are particularly important. They empirically test this by designing five tasks in the environment and checking whether finetuning the agents from the multiagent autocurricula learns faster than direct training and finetuning curiosity-based agents. They find that the multiagent autocurricula agents do best, but only slightly. To explain this, they hypothesize that the learned skill representations are still highly entangled and so are hard to finetune, whereas learned feature representations transfer more easily.

This is somewhat similar to <@AI-GAs@>(@AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence@): both depend on _environment design_, which so far has been relatively neglected. However, AI-GAs are hoping to create _learning algorithms_, while multiagent autocurricula leads to _tool use_, at least in this case. Another point of similarity is that they both require vast amounts of compute, as discovering new strategies can take significant exploration. That said, it seems that we might be able to drastically decrease the amount of compute needed by solving the exploration problem using e.g. human play data or demonstrations (discussed in two different papers above).

More speculatively, I hypothesize that it will be useful to have environments where you need to identify _what strategy your opponent is using_. In this environment, each strategy has the property that it beats _all_ of the strategies that preceded it. As a result, it was fine for the agent to undergo catastrophic forgetting: even though it was trained against past agents, it only needed to learn the current strategy well; it didn't need to remember previous strategies. As a result, it may have forgotten prior strategies and skills, which might have reduced its ability to learn new tasks quickly.

[Paper: Emergent Tool Use from Multi-Agent Autocurricula](https://d4mucfpksywv.cloudfront.net/emergent-tool-use/paper/Multi_Agent_Emergence_2019.pdf), [Vox: Watch an AI learn to play hide-and-seek](https://www.vox.com/future-perfect/2019/9/20/20872672/ai-learn-play-hide-and-seek)


A Survey of Reinforcement Learning Informed by Natural Language

Jelena Luketina
Nantas Nardelli
Gregory Farquhar
Jakob Foerster
Jacob Andreas
Edward Grefenstette
Shimon Whiteson
Tim Rocktäschel

Humans use language as a way of efficiently storing knowledge of the world and instructions for handling new scenarios; this paper is written from the perspective that it would be potentially hugely valuable if RL agents could leverage information stored in language in similar ways. They look at both the case where language is an inherent part of the task (example: the goal is parameterized by a language instruction) and where language is used to give auxiliary information (example: parts of the environment are described using language). Overall, the authors push for more work in this area, and, in particular, more work using external-corpus-pretrained language models and with research designs that use human-generated rather than synthetically-generated language; the latter is typically preferred for the sake of speed, but the former has particular challenges we'll need to tackle to actually use existing sources of human language data.

This article is a solid and useful version of what I would expect out of a review article: mostly useful as a way to get thinking in the direction of the intersection of RL and language, and makes me more interested in digging more into some of the mentioned techniques, since by design this review didn't go very deep into any of them.


How to Train Your OpenAI Five

<@OpenAI Five@> has now beaten the Dota world champions 2-0, after training for 8x longer, for a total of 800 petaflop/s-days or 45000 years of Dota self-play experience. During this insanely long training run, OpenAI grew the LSTM to 4096 units, added buybacks to the game, and switched versions twice. Interestingly, they found it hard to add in new heroes: they could bring a few new heroes up to 95th percentile of humans, but it didn't look like they would train fast enough to reach pro level. This could be because the other heroes were already so capable that it was too hard to learn, since the new heroes would constantly be beaten. The resulting team was also able to play cooperatively with humans, even though they had never been trained with humans.

As usual, I like [Alex Irpan's thoughts](https://www.alexirpan.com/2019/04/14/openai-finals.html). On the Dota side, he found Five's reaction times more believable, but was disappointed by the limited hero pool. He also predicted that with [OpenAI Five Arena](https://arena.openai.com/), which allowed anyone to play either alongside Five, or against Five, at least one of the _many_ teams would figure out a strategy that could reliably beat Five. He was right: while Five had a 99.4% win rate, one team was able to beat it [10 times in a row](https://arena.openai.com/#/results), another beat it thrice in a row, and two teams beat it twice in a row.

In this era of scaling up compute via parallelism, it was quite surprising to see OpenAI scaling up compute simply by training for almost a year. That feels like one of the last resorts to scale up compute, so maybe we're seeing the limits of the trend identified in <@AI and Compute@>?

Back when OpenAI Five beat a strong team in their <@Benchmark@>(@OpenAI Five Benchmark: Results@), I and a few others predicted that the team would be able to beat Five after playing a few games against it. I think this prediction has been somewhat validated, given that four teams figured out how to beat a much stronger version of the bot. Of course, humans played over 7000 games against Five, not just a few, so this could be that enough random search finds a weakness. Still, I'd expect pros to be able to do this in tens, maybe hundreds of games, and probably this would have been much easier at the time of the Benchmark.

The underlying model here is that Dota has an extremely large space of strategies, and neither Five nor humans have explored it all. However, pros have a better (lower-dimensional) representation of strategy space (concepts like "split-push") that allow them to update quickly when seeing a better opponent. I don't know what it would take to have AI systems learn these sorts of low-dimensional representations, but it seems key to having AI systems that can adapt quickly like humans can.

Vox: AI triumphs against the world’s top pro team in strategy game Dota 2


Eighteen Months of RL Research at Google Brain in Montreal

One approach to reinforcement learning is to predict the entire distribution of rewards from taking an action, instead of predicting just the expected reward. Empirically, this works better, even though in both cases we choose the action with highest expected reward. This blog post provides an overview of work at Google Brain Montreal that attempts to understand this phenomenon. I'm only summarizing the part that most interested me.

First, they found that in theory, distributional RL performs on par with or worse than standard RL when using either a tabular representation or linear features. They then tested this empirically on Cartpole, and found similar results: distributional RL performed worse when using tabular or linear representations, but better when using a deep neural net. This suggests that distributional RL "learns better representations". So, they visualize representations for RL on the four-room environment, and find that distributional RL captures more structured representations. Similarly this [paper](https://arxiv.org/abs/1902.0686) showed that predicting value functions for multiple discount rates is an effective way to produce auxiliary tasks for Atari.

This is a really interesting mystery with deep RL, and after reading this post I have a story for it. Note I am far from an expert in this field and it's quite plausible that if I read the papers cited in this post I could tell this story is false, but here's the story anyway. As we saw with PreQN earlier in this issue, one of the most important aspects of deep RL is how information about one (s, a) pair is used to generalize to other (s, a) pairs. I'd guess that the benefit from distributional RL is primarily that you get "good representations" that let you do this generalization well. With a tabular representation you don't do any generalization, and with a linear feature space the representation is hand-designed by humans to do this generalization well, so distributional RL doesn't help in those cases.

But why does distributional RL learn good representations? I claim that it provides stronger supervision given the same amount of experience. With normal expected RL, the final layer of the neural net need only be useful for predicting the expected reward, but with distributional RL they must be useful for predicting all of the quantiles of the reward distribution. There may be "shortcuts" or "heuristics" that allow you to predict expected reward well because of spurious correlations in your environment, but it's less likely that those heuristics work well for all of the quantiles of the reward distribution. As a result, having to predict more things enforces a stronger constraint on what representations your neural net must have, and thus you are more likely to find good representations. This perspective also explains why predicting value functions for multiple discount rates helps with Atari, and why adding auxiliary tasks is often helpful (as long as the auxiliary task is relevant to the main task).

The important aspect here is that all of the quantiles are forcing the same neural net to learn good representations. If you instead have different neural nets predicting each quantile, each neural net has roughly the same amount of supervision as in expected RL, so I'd expect that to work about as well as expected RL, maybe a little worse since quantiles are probably harder to predict than means. If anyone actually runs this experiment, please do let me know the result!


An Overdue Post on AlphaStar

The [first post](https://www.alexirpan.com/2019/02/22/alphastar.html) in this two-parter talks about the impact of [AlphaStar](https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/) ([AN #43](https://mailchi.mp/768a8130013f/alignment-newsletter-43)) on the StarCraft community and broader public. I'm focusing on the second one, which talks about AlphaStar's technical details and implications. Some of this post overlaps with my summary of AlphaStar, but those parts are better fleshed out and have more details.

First, imitation learning is a surprisingly good base policy, getting to the level of a Gold player. It's surprising because you might expect the [DAgger](https://www.ri.cmu.edu/pub_files/2011/4/Ross-AISTATS11-NoRegret.pdf) problem to be extreme: since there are so many actions in a StarCraft game, your imitation learning policy will make some errors, and those errors will then compound over the very long remainder of the episode as they take the policy further away from normal human play into states that the policy wasn't trained on.

Second, population-based training is probably crucial and will be important in the future, because it allows for exploring the full strategy space.

Third, the major challenge is making RL achieve okay performance, and after that they very quickly become great. It took years of research to get Dota and StarCraft bots reach decent play, and then a few days of more training got them to be world class. Fun quote: "although OpenAI’s DotA 2 agent lost against a pro team, [they were able to beat their old agent 80% of the time with 10 days of training](https://twitter.com/openai/status/1037765547427954688)".

Fourth, there were a lot of research results that went into AlphaStar. This suggests that there are large gains to be had by throwing a lot of techniques together and seeing how well they work, which doesn't happen very much currently. There are good reasons for this: it's much easier to evaluate a technique if its built upon a simple, standard algorithm rather than having to consider all of its interactions with other techniques which you may or may not be able to properly compare against. Still, there are going to be some cool results that we could do now if we just threw the right things together, and this sort of work also lets us test techniques in new settings to see which ones actually work in general, as opposed to only in the original evaluation.

I really like this post, and agree with almost everything in it. On the imitation learning point, I also found it surprising how well imitation learning worked. Alex suggests that it could be that human data has enough variation that the agent can learn how to recover from incorrect decisions it could make. I think this is a partial explanation at best -- there is a huge combinatorial explosion, it's not clear why you don't need a much larger dataset to cover the entire space. Maybe there are "natural" representations in any realistic complex environment that you start to accurately learn at the level of compute that they're using, and once you have those then imitation learning with sufficient variation can work well.

On the last point about tossing techniques together, I think this might sometimes be worth doing but often may not be. It makes sense to do this with any real task, since that's a test of the technique against reality. (Here StarCraft counts as a "real" task while Atari does not; the criterion is something like "if the task is successfully automated we are impressed regardless of how it is solved".) I'm less keen on tossing techniques together for artificial benchmarks. I think typically these techniques improve the sample efficiency by a constant multiplicative factor by adding something akin to a good inductive bias; in that case throwing them together may let us solve the artificial benchmark sooner but it doesn't give us great evidence that the "inductive bias" will be good for realistic tasks. I think I don't actually disagree with Alex very much on the object-level recommendations, I would just frame them differently.


Off-Policy Deep Reinforcement Learning without Exploration

This paper discusses off-policy batch reinforcement learning, in which an agent is trying to learn a policy from data which is not based on its own policy, and without the opportunity to collect more data during training. The authors demonstrate that standard RL algorithms do badly in this setting because they give unseen state-action pairs unrealistically high values, and lack the opportunity to update them. They  proposes to address this problem by only selecting actions from previously seen state-action pairs; they prove various optimality results for this algorithm in the MDP setting. To adapt this approach to the continuous control case, the authors train a generative model to produce likely actions (conditional on the state and the data batch) and then only select from the top n actions. Their batch-conditional q-learning algorithm (BCQ) consists of that generative model, a perturbation model to slightly alter the top actions, and a value network and critic to perform the selection. When n = 0, BCQ resembles behavioural cloning, and when n -> ∞, it resembles Q-learning. BCQ with n=10 handily outperformed DQN and DDPG on some Mujoco experiments using batch data.

This is an interesting paper, with a good balance of intuitive motivations, theoretical proofs, and empirical results. While it's not directly safety-related, the broad direction of combining imitation learning and reinforcement learning seems like it might have promise. Relatedly, I wish the authors had discussed in more depth what assumptions can or should be made about the source of batch data. For example, BCQ would presumably perform worse than DQN when data is collected from an expert trying to minimise reward, and (from the paper’s experiments) performs worse than behavioural cloning when data is collected from an expert trying to maximise reward. Most human data an advanced AI might learn from is presumably somewhere in between those two extremes, and so understanding how well algorithms like BCQ would work on it may be valuable.


Visual Model-Based Reinforcement Learning as a Path towards Generalist Robots

Frederik Ebert
Chelsea Finn
Sudeep Dasari
Annie Xie
Alex Lee
and Sergey Levine

How can we get general robots that can perform a diverse array of tasks? We could collect a lot of data from robots acting randomly, train a dynamics model on pixels, and then use model-predictive control to plan. The dynamics model is a neural net trained to predict the next image given the current image and action. It helps to use temporal skip connections, because this allows the robot to get some object permanence since it can now "remember" objects it saw in the past that are currently blocked by something else. Model predictive control then samples sequences of actions (called plans), predicts the final image achieved, chooses the plan that best achieves the goal, and takes the first action of that plan. This is then repeated to choose subsequent actions. (Their method is slightly more sophisticated but this is the basic idea.) We can specify the goal by choosing a particular pixel and asking that the object at that pixel be moved to some other pixel. Alternatively, [Few-Shot Goal Inference for Visuomotor Learning and Planning](https://arxiv.org/abs/1810.00482) ([AN #27](https://mailchi.mp/0212425e5544/alignment-newsletter-27)) trains a classifier that can take a few demonstrations and output a goal.

This is probably the easiest way to get a robot to do interesting things, since you just need it to collect experience autonomously with very little human involvement, you don't need to have good object detection, and in many cases goal specification can be done without too much effort. I'm surprised that using random actions is enough -- how does the robot get enough examples of picking up an object with random actions? Maybe the robot's random strategy is actually coded up in such a way that it is particularly likely to do interesting things like picking up an object.

It does seem like this approach will need something else in order to scale to more advanced capabilities, especially hierarchical tasks -- for example, you'll never have an example of picking up a napkin, getting it wet, and wiping down a table. But perhaps we can iterate the process, where after we learn how to grasp and push, we start collecting data again using grasping and pushing instead of random low-level actions. Safe exploration would become more of a concern here.


AlphaZero: Shedding new light on the grand games of chess, shogi and Go

David Silver
Thomas Hubert
Julian Schrittwieser
and Demis Hassabis

If you didn't already believe that AlphaZero is excellent at Go, Chess and Shogi, this post and the associated paper show it more clearly with a detailed evaluation. A few highlights:
- AlphaZero can beat Stockfish starting from common human openings, suggesting that it generalizes well
- The amount of computation given to AlphaZero to choose a move has a larger effect on the win probability than I was expecting
- I always wondered why they use MCTS and not alpha-beta search. They speculate that alpha-beta search with a neural net evaluation function succumbs to the [winner's curse](https://www.investopedia.com/terms/w/winnerscurse.asp) since alpha-beta involves a lot of maxes and mins, whereas MCTS averages over evaluations and so is more robust. In contrast, evaluation functions designed by humans are much more likely to generalize well, and alpha-beta outperforms MCTS.


Open Sourcing Active Question Reformulation with Reinforcement Learning

Michelle Chen Huebscher and Rodrigo Nogueira

Given a question-answering (QA) system, we can get better performance by reformulating a question into a format that is better processed by that system. (A real-world example is [google-fu](https://en.wiktionary.org/wiki/Google-fu), especially several years ago when using the right search terms was more important.) This blog post and accompanying paper consider doing this using reinforcement learning -- try a question reformulation, see if gives a good answer, and if so increase the probability of generating that reformulation. For this to work at all, the neural net generating reformulations has to be pretrained to output sensible questions (otherwise it is an _extremely_ sparse reward problem). They do this by training an English-English machine translation system. The generated reformulations are quite interesting -- 99.8% start with "what is name", and many of them repeat words. Presumably the repetition of words is meant to tell the underlying QA system that the word is particularly important.

I like how this demonstrates the faults of our current QA systems -- for example, instead of understanding the semantic content of a question, they instead focus on terms that are repeated multiple times. In fact, this might be a great way to tell whether our systems are "actually understanding" the question (as opposed to, say, learning a heuristic of searching for sentences with similar words and taking the last noun phrase of that sentence and returning it as the answer). For a good QA system, one would hope that the optimal question reformulation is just to ask the same question again. However, this won't work exactly as stated, since the RL system could learn the answers itself, which could allow it to "reformulate" the question such that the answer is obvious, for example reformulating "In what year did India gain independence?" to "What is 1946 + 1?" Unless the QA system is perfectly optimal, there will be some questions where the RL system could memorize the answer this way to improve performance.


Near-Optimal Representation Learning for Hierarchical Reinforcement Learning

Ofir Nachum
Shixiang Gu
Honglak Lee
Sergey Levine

This paper discusses the use of learned representations in hierarchical RL. In the setting where a higher-level policy chooses goals which lower-level policies are rewarded for reaching, how bad is it when the goal representation isn't able to express all possible states? The authors define a metric for a representation's lossiness based on how close to optimal the policies which can be learned using that representation are, and prove that using a certain objective function, representations with bounded lossiness can be learned. They note a similarity between this objective function and those of mutual information estimators.

The authors test their learner on the MuJoCo Ant Maze environment, achieving compelling results.

This is a fairly mathematical paper and I didn't entirely follow the proofs, so I'm not sure how dependent they are on the particular choice of objective function. However, the empirical results using that objective seem very impressive, and significantly outperform alternative methods of learning representations.


Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

Sergey Levine
Aviral Kumar
George Tucker
Justin Fu

The authors in this paper give an overview of offline-reinforcement learning with the aim that readers gain enough familiarity to start thinking about how to make contributions in this area. The utility of a fully offline RL framework is significant: just as supervised learning methods have been able to utilize data for generalizable and powerful pattern recognition, offline RL methods could enable data to be funneled into decision-making machines for applications such as health-care, robotics, and recommender systems. The organization of the article is split into a section on formulation and another on benchmarks, followed by a section on applications and a general discussion.

In the formulation portion of the review, the authors give an overview of the offline learning problem and then discuss a number of approaches. Broadly speaking, the biggest challenge is the need for counterfactual reasoning because the agent must learn using data by another agent. Thus, the agent is forced to reason about what would happen if a different decision was used. Importance sampling, approximate dynamic programming, and offline model-based approaches are discussed as possible approaches to this counterfactual reasoning problem. In the benchmarks section, the authors review evaluation techniques for offline RL methods. While the authors find that there are many domain-specific evaluations, general benchmarking is less well established. A major issue in creating benchmarks is deciding whether or not to use diverse trajectories/replay buffer data, or only the final expert policy.

In the discussion, the authors argue that while importance sampling and dynamic programming work on low-dimensional and short-horizon tasks, they struggle to integrate well with function approximators. On the other hand, the authors see approaches that constrain the space of policies to be near the dataset as a promising direction to mitigate the effects of distributional shift. However, the authors acknowledge that it may ultimately take more systematic datasets to push the field forward.

This was a great overview of the state of the field. A recurring theme that the authors highlight is that offline RL requires counterfactual reasoning which may be fundamentally difficult to achieve because of distributional shift. Some results shown in the paper suggest that offline RL may just be fundamentally hard. However, I find myself sharing optimism with the authors on the subject of policy constraint techniques and the inevitable importance of better datasets.


The Animal-AI Testbed and Competition

Matthew Crosby
Benjamin Beyret
Murray Shanahan
José Hernández-Orallo
Lucy Cheke
Marta Halina

The Animal-AI testbed tests agents on the ability to solve the sorts of tasks that are used to test animal cognition: for example, is the agent able to reach around a transparent obstacle in order to obtain the food inside. This has a few benefits over standard RL environments:

1. The Animal-AI testbed is designed to test for specific abilities, unlike environments based on existing games like Atari.
2. A single agent is evaluated on multiple hidden tasks, preventing overfitting. In contrast, in typical RL environments the test setting is identical to the train setting, and so overfitting would count as a valid solution.

The authors ran a competition at NeurIPS 2019 in which submissions were tested on a wide variety of hidden tasks. The winning submission used an iterative method to design the agent: after using PPO to train an agent with the current reward and environment suite, the designer would analyze the behavior of the resulting agent, and tweak the reward and environments and then continue training, in order to increase robustness. However, it still falls far short of the perfect 100% that the author can achieve on the tests (though the author is not seeing the tests for the first time, as the agents are).

NeurIPS 2019 Competition and Demonstration Track

I’m not sure that the path to general intelligence needs to go through replicating embodied animal intelligence. Nonetheless, I really like this benchmark, because its evaluation setup involves new, unseen tasks in order to prevent overfitting, and because of its focus on learning multiple different skills. These features seem important for RL benchmarks regardless of whether we are replicating animal intelligence or not.

Building Thinking Machines by Solving Animal Cognition Tasks


Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey

Sanmit Narvekar
Bei Peng
Matteo Leonetti
Jivko Sinapov
Matthew E. Taylor
Peter Stone

For a variety of learning problems, the training process is organized so that new concepts and tasks leverage previously learned information. This can serve as a broad definition of curriculum learning. This paper gives an overview of curriculum learning and a framework to organize various approaches to the curriculum learning problem. One central difficulty is that there is a broad class of methods that can be considered curricula. At one extreme, we have curricula where new tasks are created to speed up learning. At another extreme, some curricula simply reorder experience samples. For example, the prioritized replay buffer is one such reordering method. Thus, to cover as much of the literature as possible the authors outline a framework for curriculum learning and then use that structure to classify various approaches. In general, the definition, learning, construction, and the evaluation of curricula are all covered in this work. This is done by breaking the curriculum learning problem into three steps: task generation, sequencing, and transfer learning. Using this problem decomposition the authors give an overview of work addressing each component.

Before I read this, I thought of curricula as 'hacks' used to improve training. However, the authors' presentation of connections with transfer learning and experience replay has significantly changed my opinion. In particular, the phrasing of curriculum learning as a kind of 'meta-MDP seems particularly interesting to me. Moreover, there seem to be interesting challenges in this field. One such challenge is that there does not seem to be a great amount of theory about *why* curricula work which could indicate a point of departure for people interested in safety research. Knowing more about theory could help answer safety questions. For example, how do we design curricula so that we can guarantee/check the agent is behaving correctly at each step?


Mastering Complex Control in MOBA Games with Deep Reinforcement Learning

Deheng Ye
Zhao Liu
Mingfei Sun
Bei Shi
Peilin Zhao
Hao Wu
Hongsheng Yu
Shaojie Yang
Xipeng Wu
Qingwei Guo
Qiaobo Chen
Yinyuting Yin
Hao Zhang
Tengfei Shi
Liang Wang
Qiang Fu
Wei Yang
Lanxiao Huang

This paper presents an AI system that can play the Multi-player Online Battle Arena (MOBA) game _Honor of Kings_. They are inspired by <@OpenAI Five@> (and Honor of Kings sounds quite similar to Dota, though it is 1v1 instead of 5v5), and have a similar learning setup: reinforcement learning using PPO. Their architecture requires an off-policy algorithm (I’m not sure why, maybe they have stale parameters across their rollout servers), so they add an importance sampling correction to the PPO objective, as well as an additional type of gradient clipping. The input is a combination of the image and underlying game state info. The resulting agents are able to beat top human players, and in an event with the public, the AI system lost only 4 out of 2100 matches. Unlike OpenAI Five, this required only around 100 hours to train (though it’s unclear how much compute was used).


"Other-Play" for Zero-Shot Coordination

Hengyuan Hu
Adam Lerer
Alex Peysakhovich
Jakob Foerster

How can we build AI systems that can _coordinate_ with humans? While <@past@>(@Collaborating with Humans Requires Understanding Them@) <@work@>(@Learning Existing Social Conventions via Observationally Augmented Self-Play@) has assumed access to some amount of human data, this paper aims to coordinate _without any human data at all_, which they call _zero-shot coordination_. In order to develop an algorithm, they assume that their partner is also "trained" for zero-shot coordination.

Their key idea is that in zero-shot coordination, since you can't break symmetries by agreeing upon a protocol in advance (i.e. you can't agree on things like "we'll drive on the left, not the right"), you need a policy that is _robust to relabelings that preserve these symmetries_. This is easy to train for: you just train in self-play, but randomly relabel the states, actions and observations separately for each side in a way that preserves the MDP structure (i.e. uses one of the symmetries). Thus, each side must play a policy that works well _without knowing how the other agent's observations and actions have been relabeled_. In practice, for an N-player game you only need to randomize N-1 of the relabelings, and so in the two player games they consider they only randomly relabel one side of the self-play.

They evaluate this in Hanabi (where the game is invariant to relabeling of the colors), and show that the resulting agents are better at playing with other agents trained on different seeds or with slightly different architectures, and also that they play better with humans, achieving an average score of 15.75 with non-expert human players, compared to 9.15 for agents trained via regular self-play.

For comparison, I think I get around 17-22 when playing with new players, out of a max of 25, so 15.75 is quite a healthy score given that it doesn't use _any_ human data. That being said, it seems hard to use this method in other settings -- even in the relatively simple <@Overcooked environment@>(@Collaborating with Humans Requires Understanding Them@), there aren't any obvious symmetries to use for such training. Perhaps future work will allow us to find approximate symmetries in games somehow, that we can then train to be robust to?


Building AI that can master complex cooperative games with hidden information

Adam Lerer
Hengyuan Hu
Jakob Foerster
Noam Brown

This paper improves on the state of the art for AI agents playing <@Hanabi@>(@The Hanabi Challenge: A New Frontier for AI Research@), a cooperative multiplayer game that is challenging because of distributed hidden information and restricted communication.

The approach works by improving a baseline policy using search. In the simplest case, only one agent performs search while all other agents follow a fixed policy, such that the problem is reduced to search in a POMDP. This alone leads to relevant improvements, even when the search is very shallow. The fixed policies help because they allow the searching agent to correctly update its belief about hidden information when it sees other agents behaving (as it knows how other agents would behave given different states of the hidden information). This idea can be generalized to the case where all agents perform search by letting the agents simulate each other's search process. This can get expensive quickly as agent A's beliefs in the second round also depend on agent B's search process in counterfactual scenarios in the first round, such that agent B's search in round two also has to simulate these counterfactuals.  A computation budget is introduced to make this computationally feasible and all agents know that the other agents will only use search in a turn if the cost of this is below the budget.

As search can be performed on top of any policy and allows to leverage compute during inference, not just training, it nicely complements more direct approaches using deep RL, which is a theme that has also been observed in Go and Poker.

This solution seems stunningly obvious in retrospect. While the authors informally report that their approach improves robustness to replacing other agents by humans, the example they give seems to indicate that this is because search prevents obvious mistakes in novel situations induced by human behaviour. Thus, I still expect (implicit) <@human models@>(@Thoughts on Human Models@) to be a vital component of human-machine cooperation.

Paper: Improving Policies via Search in Cooperative Partially Observable Games


CURL: Contrastive Unsupervised Representations for Reinforcement Learning

Aravind Srinivas*
Michael Laskin*
Pieter Abbeel

This paper applies contrastive learning (discussed above) to reinforcement learning. In RL, rather than training in an initial unsupervised phase, the contrastive learning happens alongside the RL training, and so serves as an auxiliary objective to speed up learning. They use random crops for their data augmentation.


Henry Zhu*
Justin Yu*
Abhishek Gupta*
Dhruv Shah
Kristian Hartikainen
Avi Singh
Vikash Kumar
Sergey Levine

Suppose we wanted to train a robot to perform a task in the real world, and we didn't want to deal with the headache of sim-to-real transfer. Typically, since all of our experience must be collected in the real world, we would need a human to reset the robot to its initial state. The key idea of this paper is that the point of resets is to ensure that the robot explores a diversity of states causing it to learn a robust policy; this can be achieved by learning a _perturbation policy_ whose objective is to take the robot to states it hasn't visited before. They then combine this with representation learning (so that they can learn from pixels) and use a classifier that distinguishes goal states from non-goal states as the reward function, to get a fully automated setup where once you start the robot's training, it trains itself without any human in the loop.

This is a cool proof of concept, but the learned perturbation policy can only take you so far -- no learned perturbation policy is going to allow you to e.g. pick up an object after it is dropped, as you would want if you're training a robot to <@manipulate a Rubik's cube@>(@Solving Rubik’s Cube with a Robot Hand@). It seems hard to overcome this sort of problem in a fully automated and learned way (though perhaps you could use more classical techniques to have a "hardcoded" but still automated reset policy).

Paper: The Ingredients of Real World Robotic Reinforcement Learning


Massively Scaling Reinforcement Learning with SEED RL

Lasse Espeholt*
Raphaël Marinier*
Piotr Stanczyk*
Ke Wang
Marcin Michalski

Deep learning has <@historically@>(@AI and Compute@) seen many improvements as a result of scaling to larger models with larger amounts of computation, as with the months-long training of <@OpenAI Five@>(@Dota 2 with Large Scale Deep Reinforcement Learning@) and <@AlphaStar@>(@AlphaStar: Mastering the Real-Time Strategy Game StarCraft II@). SEED RL redesigns the architecture of distributed RL to enable better machine utilization and communication and achieves an order of magnitude improvement in training speed.

Current distributed architectures typically separate machines into *actors* and *learners*. *Actors* are typically CPUs that simulate the environment, and run inference to predict agent actions. They then send *trajectories* to the *learners*. *Learners* are typically accelerators (GPUs or TPUs), which are responsible for training the model. They then send the updated model parameters to the *actors*.

SEED RL addresses 3 main issues in this setup:
1. Inference could benefit from specialized accelerators
2. Sending model parameters and states requires high bandwidth.
3. Environment simulation and inference are very different tasks and having them on the same machine makes it hard to utilize the resource efficiently.

The solution is to instead have actors **only** simulate the environment. After each step, they send the resulting observation to the *learner*, which is responsible for both training and inference, possibly split on separate hardware). It then sends back just the actions to the environment. This enables each piece of hardware to be used for its designed purpose. Since they now need to communicate at each step, they use gRPC to minimize latency.

Given how compute intensive deep RL is, I think it is quite useful to enable cheaper and faster training before these algorithms can be broadly useful. Their claimed speedup is quite impressive, and I like how well they can separate the training and inference from the simulation. I expect that specialized hardware for both training and inference will soon become the norm and SEED RL seems like it will scale well as those accelerators become faster. One thing to note is that this architecture seems very specifically tuned to the problem of games where CPUs can efficiently simulate the environment and it does not improve the sample efficiency for situations where we can’t run lots of simulations.

**Rohin's opinion:** It was quite surprising to me that this worked as well as it did: this model requires communication across machines _at every timestep of the environment_, which intuitively means that latency should be a major bottleneck, while the standard model only requires communication once per batch of trajectories.

Paper: SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference


Robots Learning to Move like Animals

Xue Bin Peng
Erwin Coumans
Tingnan Zhang
Tsang-Wei Lee
Jie Tan
Sergey Levine

<@Previous work@>(@Learning Acrobatics by Watching YouTube@) has suggested that we can get good policies by estimating and imitating poses. This work takes this idea and tries to make it work with sim-to-real transfer. Domain randomization would result in a policy that must be robust to all the possible values of the hidden parameters (such as friction). To make the problem easier, they do domain randomization, but give the agent access to (a latent representation of) the hidden parameters, so that its policy can depend on the hidden parameters. Then, to transfer to the real world, they simply need to search over the latent representation of the hidden parameters in order to find one where the policy actually works in the real world. In practice, they can adapt to the real world with just 8 minutes of real world data.

This is a cool improvement to domain randomization: it seems like it should be distinctly easier to learn a policy that is dependent on the hidden parameters, and that seems to come at the relatively low cost of needing just a little real world data.

Paper: Learning Agile Robotic Locomotion Skills by Imitating Animals


Planning with Goal-Conditioned Policies

Soroush Nasiriany*
Vitchyr H. Pong*
Steven Lin
Sergey Levine

Reinforcement learning can learn complex skills by interacting with the environment. However, temporally extended or long-range decision-making problems require more than just well-honed reactions. **In this paper, the authors investigate whether or not they can obtain the benefits of action planning found in model-based RL without the need to model the environment at the lowest level.** The authors propose a model-free planning framework that learns low-level goal-conditioned policies that use their value functions as implicit models. Goal-conditioned policies are policies that can be trained to reach a goal state provided as an additional input. Given a goal-conditioned policy, the agent can then plan over intermediate subgoals (goal states) using a goal-conditioned value function to estimate reachability. Since the state space is large, the authors propose what they call latent embeddings for abstracted planning (LEAP), which is able to find useful subgoals by first searching a much smaller latent representation space and then planning a sequence of reachable subgoals that reaches the target state. In experiments, LEAP significantly outperforms prior algorithms on 2D navigation and push/reach tasks. Moreover, their method can get a quadruped ant to navigate around walls which is difficult because much of the planning happens in configuration space. This shows that LEAP is able to be extended to non-visual domains.

The presentation of the paper is clear. In particular, the idea of planning a sequence of maximally feasible subgoals seems particularly intuitive. In general, I think that LEAP relies on the clever idea of reusing trajectory data to augment the data-set for the goal-conditioned policy. As the authors noted, the question of exploration was mostly neglected. I wonder how well the idea of reusing trajectory data generalizes to the general exploration problem.

**Rohin's opinion:** The general goal of inferring hierarchy and using this to plan more efficiently seems very compelling but hard to do well; this is the goal in most hierarchical RL algorithms and <@Learning Latent Plans from Play@>.


Dream to Control: Learning Behaviors by Latent Imagination

Danijar Hafner
Timothy Lillicrap
Jimmy Ba
Mohammad Norouzi

In the past year or so, the idea of learning a transition model in a latent space has gained traction, motivated by the hope that such an approach could combine the best of the worlds of model-free and model-based learning. The central appeal of learning a latent transition model is that it allows you to imagine future trajectories in a potentially high-dimensional, structured observation space without actually having to generate those high-dimensional observations.

Dreamer builds on a prior model by the same authors, <@PlaNet@>(@Learning Latent Dynamics for Planning from Pixels@), which learned a latent representation of the observations, p(s|o), trained both through a VAE-style observation reconstruction loss, and also a transition model q(s-next|s, a), which is trained to predict the state at the next step given only the state at the prior one, with no next-step observation data. Together, these two models allow you to simulate action-conditioned trajectories through latent state space. If you then predict reward from state, you can use this to simulate the value of trajectories. Dreamer extends on this by also training an Actor Critic-style model on top of states to predict action and value, forcing the state representation to not only capture next-step transition information, but also information relevant to predicting future rewards. The authors claim this extension makes their model more able to solve long-horizon problems, because the predicted value function can capture far-future rewards without needing to simulate the entire way there. Empirically, there seems to be reasonable evidence that this claim plays out, at least within the fairly simple environments the model is tested in.

The extension from PlaNet (adding actor-critic rather than direct single-step reward prediction) is relatively straightforward, but I think latent models are an interesting area - especially if they eventually become at all possible to interpret - and so I'm happy to see more work in this area.


Gym Retro, again

OpenAI is releasing the full version of Gym Retro, with over a thousand games, and a tool for integrating new games into the framework. And of course we see new games in which RL agents find infinite loops that give them lots of reward -- Cheese Cat-Astrophe and Blades of Vengeance.


Adaptive Online Planning for Continual Lifelong Learning

Lifelong learning is distinct from standard RL benchmarks because
 1. The environment is *sequential* rather than *episodic*; it is never reset to a new start state.
 2. The current *transition* and *reward* function are given, but they change over time.

Given this setup, there are two basic approaches: first, run model-free learning on simulated future trajectories and rerun it every time the dynamics change, and second, run model-based planning on the current model. If you ignore computational constraints, these should be equivalent; however, in practice, the second option tends to be more computationally efficient. The contribution of this work is to make this more efficient, rather than improving final performance, by starting with the second option and then using model-free learning to “distill” the knowledge produced by the model-based planner allowing for more efficient planning in the future.

Specifically, Adaptive Online Planning (AOP) balances between the model-based planner MPPI (a variant of MPC) and the model-free algorithm TD3. MPPI uses the given model to generate a trajectory up to a horizon and then uses an ensemble of value functions to estimate the cumulative reward. This knowledge is then distilled into TD3 for later use as a prior for MPPI. During future rollouts, the variance and Bellman error of the value function ensemble are used to determine how long the horizon should be, and therefore how much computation is used.

I agree that episodic training and fixed world dynamics seem like unlikely conditions for most situations we would expect agents to encounter in the real world. Accounting for them seems particularly important to ensure safe exploration and robustness to distributional shift, and I think that these environments could serve as useful benchmarks for these safety problems as well.


Model-Based Reinforcement Learning: Theory and Practice

Michael Janner
Justin Fu
Marvin Zhang
Sergey Levine

This post provides a broad overview of model-based reinforcement learning, and argues that a learned (explicit) model allows you to generate sample trajectories from the current policy at arbitrary states, correcting for off-policy error, at the cost of introducing model bias. Since model errors compound as you sample longer and longer trajectories, the authors propose an algorithm in which the model is used to sample short trajectories from states in the replay buffer, rather than sampling trajectories from the initial state (which are as long as the task's horizon).

Paper: When to Trust Your Model: Model-Based Policy Optimization


Stabilizing Transformers for Reinforcement Learning

Emilio Parisotto
H. Francis Song
Jack W. Rae
Razvan Pascanu
Caglar Gulcehre
Siddhant M. Jayakumar
Max Jaderberg
Raphael Lopez Kaufman
Aidan Clark
Seb Noury
Matthew M. Botvinick
Nicolas Heess
Raia Hadsell

Transformers have been incredibly successful in domains with sequential data. Naturally, one might expect transformers to be useful in partially observable RL problems. However, transformers have complex implementations making them difficult to use in an already challenging domain for learning. In this paper, the authors explore a novel transformer architecture they call Gated Transformer-XL (GTrXL) that can be used in the RL setting. The authors succeed in stabilizing training with a reordering of the layer normalization coupled with the addition of a new gating mechanism located at key points in the submodules of the transformer. The new architecture is tested on DMlab-30, a suite of RL tasks including memory, and shows improvement over baseline transformer architectures and the neural computer architecture MERLIN. Furthermore, GTrXL learns faster and is more robust than a baseline transformer architecture.

This is one of those 'obvious' ideas that turns out to be very difficult to put into practice. I'm glad to see a paper like this simply because the authors do a good job at explaining why a naive execution of the transformer idea is bound to fail. Overall, the architecture seems to be a solid improvement over the TrXL variant. I'd be curious whether or not the architecture is also better in an NLP setting.


Learning to Learn with Probabilistic Task Embeddings

Kate Rakelly*
Aurick Zhou*
Deirdre Quillen
Chelsea Finn
Sergey Levine

This paper proposes a solution to off-policy meta reinforcement learning, an appealing problem because on-policy RL is so sample-intensive, and meta-RL is even worse because it needs to solve a distribution over RL problems. The authors' approach divides the problem into two subproblems: infer an embedding, z, of the current task given context, and learning an optimal policy q function conditioned on that task embedding. At the beginning of each task, z is sampled from the (Gaussian) prior, and as the agent gains more samples of that particular task, it updates its posterior over z, which can be thought of as refining its guess as to which task it's been dropped into this time. The trick here is that this subdividing of the problem allows it to be done mostly off-policy, because you only need to use on-policy learning for the task inference component (predicting z given current task transitions), and can learn the Actor-Critic model conditioned on z with off-policy data. The method works by alternating between these two learning modes.

I enjoyed this; it's a well-written paper that uses a few core interesting ideas (posterior sampling over a task distribution, representation of a task distribution as a distribution of embedding vectors passed in to condition Q functions), and builds them up to make a method that achieves some impressive empirical results.

Efficient Off-Policy Meta-RL via Probabilistic Context Variables


Diagnosing Bottlenecks in Deep Q-learning Algorithms

Justin Fu
Aviral Kumar
Matthew Soh
Sergey Levine

While the PreQN paper used a theoretical approach to tackle Deep Q-Learning algorithms, this one takes an empirical approach. Their results:
    - Small neural nets cannot represent Q*, and so have undesired bias that results in worse performance. However, they also have convergence issues, where the Q-function they actually converge to is significantly worse than the best Q-function that they could express. Larger architectures mitigate both of these problems.
    - When there are more samples, we get a lower validation loss, showing that we are overfitting. Despite this, larger architectures are better, because the performance loss from overfitting is not as bad as the performance loss from having a bad bias. A good early stopping criterion could help with this.
    - To study how non-stationarity affects DQL algorithms, they study a variant where the Q-function is a moving average of the past Q-functions (instead of the full update), which means that the target values don't change as quickly (i.e. it is closer to a stationary target). They find that non-stationarity doesn't matter much for large architectures.
    - To study distribution shift, they look at the difference between the expected Bellman error before and after an update to the parameters. They find that distribution shift doesn't correlate much with performance and so is likely not important.
    - Algorithms differ strongly in the distribution over (s, a) pairs that the DQL update is computed over. They study this in the absence of sampling (i.e. when they simply weight all possible (s, a) pairs, rather than just the ones sampled from a policy) and find that distributions that are "close to uniform" perform best. They hypothesize that this is the reason that experience replay helps -- initially an on-policy algorithm would take samples from a single policy, while experience replay adds samples from previous versions of the policy, which should increase the coverage of (s, a) pairs.

To sum up, the important factors are using an expressive neural net architecture, and designing a good sampling distribution. Inspired by this, they design Adversarial Feature Matching (AFM), which like Prioritized Experience Replay (PER) puts more weight on samples that have high Bellman error. However, unlike PER, AFM does not try to reduce distribution shift via importance sampling, since their experiments found that this was not important.

This is a great experimental paper, there's a lot of data that can help understand DQL algorithms. I wouldn't take the results too literally, since insights on simple environments may not generalize to more complex environments. For example, they found overfitting to be an issue in their environments -- it's plausible to me that with more complex environments (think Dota/StarCraft, not Mujoco) this reverses and you end up underfitting the data you have. Nonetheless, I think data like this is particularly valuable for coming up with an intuitive theory of how deep RL works, if not a formal one.


Simulated Policy Learning in Video Models

Lukasz Kaiser
Mohammad Babaeizadeh
Piotr Miłos
Błażej Osiński
Roy H Campbell
Konrad Czechowski
Chelsea Finn
Piotr Kozakowski
Sergey Levine
Ryan Sepassi
George Tucker and Henryk Michalewski

This blog post and the associated [paper](https://arxiv.org/abs/1903.00374) tackle model-based RL for Atari. The recent <@world models@>(@Recurrent World Models Facilitate Policy Evolution@) paper proposed first learning a model of the world by interacting with the environment using a random policy, and then using the model to simulate the environment and training a control policy using those simulations. (This wasn't it's main point, but it was one of the things it talked about.) The authors take this idea and put it in an iterative loop: they first train the world model using experience from a random policy, then train a policy using the world model, retrain the world model with experience collected using the newly trained policy, retrain the policy, and so on. This allows us to correct any mistakes in the world model and let it adapt to novel situations that the control policy discovers. This allows them to train agents that can play Atari with only 100K interactions with the environment (corresponding to about two hours of real-time gameplay), though the final performance is lower than the state-of-the-art achieved with model-free RL. See [Import AI](https://jack-clark.net/2019/04/01/import-ai-140-surveilling-a-city-via-the-cityflow-dataset-25000-images-of-chinese-shop-signs-and-the-seven-traps-of-ai-ethics/) for more details.

This work follows the standard pattern where model-based RL is more sample efficient but reaches worse final performance compared to model-free RL. Let's try to explain this using the same story as in the rest of this newsletter.

The sample efficiency comes from the fact that they learn a world model that can predict the future, and then use that model to solve the control problem (which has zero sample cost, since you are no longer interacting with the environment). It turns out that predicting the future is "easier" than selecting the optimal action, and so the world model can be trained in fewer samples than it would take to solve the control problem directly. Why is the world model "easier" to learn? One possibility is that solving the control problem requires you to model the world anyway, and so must be a harder problem. If you don't know what your actions are going to do, you can't choose the best one. I don't find this very compelling, since there are lots of aspects of world modeling that are irrelevant to the control problem -- you don't need to know exactly how the background art will change in order to choose what action to take, but world modeling requires you to do this. I think the real reason is that world modeling benefits from much more supervision -- rather than getting a sparse reward signal over a trajectory, you get a full grid of pixels every timestep that you were supposed to predict. This gives you many orders of magnitude more "supervision information" per sample, and so it makes it easier to learn. (This is basically the same argument as in [Yann Lecun's cake analogy](https://medium.com/syncedreview/yann-lecun-cake-analogy-2-0-a361da560dae).)

Why does it lead to worse performance overall? The policy is now being trained using rollouts that are subtly wrong, and so instead of specializing to the true Atari dynamics it will be specialized to the world model dynamics, which is going to be somewhat different and should lead to a slight dip in performance. (Imagine a basketball player having to shoot a ball that was a bit heavier than usual -- she'll probably still be good, but not as good as with a regular basketball.) In addition, since the world model is supervised by pixels, any small objects are not very important to the world model (i.e. getting them wrong does not incur much loss), even if they are very important for control. In fact, they find that bullets tend to disappear in Atlantis and Battle Zone, which is not good if you want to learn to play those games.

I'm not sure if they shared weights between the world model and the control policy. If they did, then they would also have the problem that the features that are useful for predicting the future are not the same as the features that are useful for selecting actions, which would also cause a drop in performance. My guess is that they didn't share weights for precisely this reason, but I'm not sure.

Model-Based Reinforcement Learning for Atari


Unifying Physics and Deep Learning with TossingBot

TossingBot is a system that learns how to pick up and toss objects into bins using deep RL. The most interesting thing about it is that instead of using neural nets to directly predict actions, they are instead used to predict _adjustments_ to actions that are computed by a physics-based controller. Since the physics-based controller generalizes well to new situations, TossingBot is also able to generalize to new tossing locations.

This is a cool example of using structured knowledge in order to get generalization while also using deep learning in order to get performance. I also recently came across [Residual Reinforcement Learning for Robot Control](https://arxiv.org/abs/1812.03201), which seems to have the same idea of combining deep RL with conventional control mechanisms. I haven't read either of the papers in depth, so I can't compare them, but a _very_ brief skim suggests that their techniques are significantly different.


Assessing Generalization in Deep Reinforcement Learning (blog post)

This is a blog post summarizing <@Assessing Generalization in Deep Reinforcement Learning@>.


TDM: From Model-Free to Model-Based Deep Reinforcement Learning

In many tasks, we have hierarchical structure where we want to plan at a high level, but to execute the low-level actions we want to rely on learning through experience. For example, when biking from UC Berkeley to the Golden Gate Bridge, you definitely want to plan in advance the route you'll take (as opposed to learning through trial-and-error), but you want to learn how to bike through trial-and-error. Temporal Difference Models allow you to do model-based planning at the high level, and model-free learning at the low level. Specifically, you learn a function Q(s1, a, s2, T), which intuitively says "if I start from state s1, taking action a, and running for T steps, how close can I get to state s2". It turns out that this can be thought of as a Q function and so can be trained using standard model-free RL techniques. Note that the constraint Q(s1, a, s2, T) = 0 says that it is possible to get from s1 to s2 in T steps after first taking action a.

One standard way to solve model-based RL is to search for a sequence of states and actions (s0, a0, s1, a1, ...) that is feasible (agrees with the dynamics) and maximizes the reward, and then take the first action from that sequence. Using TDMs, we can now search for the sequence (s0, a0, sK, aK, s2K, a2k, ...) that is feasible and maximizes reward. The feasibility requirement is expressed by the constraint Q(s0, a0, sK, K) = 0.

Firstly, the blog post is very readable and provides a great introduction (it's much more friendly than my summary).

This technique does require that we can reinterpret any state as a goal state, similar to the assumption in [Hindsight Experience Replay](https://arxiv.org/abs/1707.01495) (HER). They do compare to HER, and find that HER doesn't do very well, which I was quite surprised by. Clicking through to the paper, it turns out the authors were surprised as well, but then realized that this is because HER is designed to work with sparse reward problems, whereas they were evaluating on problems with relatively shaped rewards.


Long-Range Robotic Navigation via Automated Reinforcement Learning

How can we get robots that successfully navigate in the real world? One approach is to use a high-level route planner that uses a learned control policy over very short distances (10-15 meters). The control policy is learned using deep reinforcement learning, where the network architecture and reward shaping is also learned via neural architecture search (or at least something very similar). The simulations have enough noise that the learned control policy transfers well to new environments. Given this policy as well as a floorplan of the environment we want the robot to navigate in, we can build a graph of points on the floorplan, where there is an edge between two points if the robot can safely navigate between the two points using the learned controller (which I _think_ is checked in simulation). At execution time, we can find a path to the goal in this graph, and move along the edges using the learned policy. They were able to build a graph for the four buildings at the Google main campus using 300 workers over 4 days. They find that the robots are very robust in the real world. See also [Import AI](https://jack-clark.net/2019/03/04/import-ai-136-what-machine-learning-power-infrastructure-means-for-humanity-new-gca-benchmarkdataset-challenges-image-captioning-systems-and-google-uses-frankenrl-to-create-more-mobile-robot/).

This is a great example of a pattern that seems quite common: once we automate tasks using end-to-end training that previously required more structured approaches, new more complex tasks will arise that will use the end-to-end trained systems as building blocks in a bigger structured approach. In this case, we can now train robots to navigate over short distances using end-to-end training, and this has been used in a structured approach involving graphs and waypoints to create robots that can traverse larger distances.

It's also an example of what you can do when you have a ton of compute: for the learned controller, they learned both the network architecture and the reward shaping. About the only thing that had to be explicity specified was the sparse true reward. (Although I'm sure in practice it took a lot of effort to get everything to actually work.)


Neural MMO

Neural MMO is "a massively multiagent game environment for reinforcement learning agents". It was designed to be persistent (with concurrent learning and no environment resets), large-scale, efficient and expandable. Agents need to traverse an environment to obtain food and water in order to survive for longer (the metric for which they are rewarded), and are also able to engage in combat with other agents. Agents trained within a larger population explore more and consistently outperform those trained in smaller populations (when evaluated together). The authors note that multiagent training is a curriculum magnifier, not a curriculum in itself, and that the environment must facilitate adaptive pressures by allowing a sufficient range of interactions.


The Hanabi Challenge: A New Frontier for AI Research

Nolan Bard
Jakob N. Foerster
Sarath Chandar
Neil Burch
Marc Lanctot
H. Francis Song
Emilio Parisotto
Vincent Dumoulin
Subhodeep Moitra
Edward Hughes
Iain Dunning
Shibl Mourad
Hugo Larochelle
Marc G. Bellemare
Michael Bowling

The authors propose the cooperative, imperfect-information card game Hanabi as a target for AI research, due to the necessity of reasoning about the beliefs and intentions of other players in order to win. They identify two challenges: firstly, discovering a policy for a whole team that allows it to win (the self-play setting); and secondly, discovering an individual policy that allows an agent to play with an ad-hoc team without previous coordination. They note that successful self-play policies are often very brittle in the ad-hoc setting, which makes the latter the key problem. The authors provide an open-source framework, an evaluation benchmark and the results of existing RL techniques.

I endorse the goals of this paper, but my guess is that Hanabi is simple enough that agents can solve it using isolated heuristics rather than general reasoning about other agents' beliefs.

*Rohin's opinion:* I'm particularly excited to see more work on ad hoc teamwork, since it seems like very similar to the setting we are in, where we would like to deploy AI system among groups of humans and have things go well. See [Following human norms](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc/p/eBd6WvzhuqduCkYv3) ([AN #42](https://mailchi.mp/f6488137d76c/alignment-newsletter-42)) for more details.

A cooperative benchmark: Announcing the Hanabi Learning Environment


Recurrent Experience Replay in Distributed Reinforcement Learning

Steven Kapturowski
Georg Ostrovski
John Quan
Remi Munos
Will Dabney


Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions

Rui Wang
Joel Lehman
Jeff Clune
Kenneth O. Stanley

The POET algorithm uses evolutionary strategies to evolve a population of pairs of tasks and agents. During each iteration, it first generates a new environment by perturbing an existing environment, then optimises each agent for its paired environment, then attempts to transfer agents between existing environments to improve performance (in case one environment turns out to be a useful "stepping stone" towards another). New environments are kept if they are neither too hard nor too easy for the current population of agents. This algorithm was tested using the Bipedal Walker environment, where it significantly outperformed standard evolutionary search.

I think that the "problem problem" is going to become increasingly important in RL, and that this is a promising approach. Note that this paper's contribution seems to be mainly that it combines ideas from previous papers on minimal criteria coevolution and innovation engines.


Natural Environment Benchmarks for Reinforcement Learning

This paper notes that RL performance tends to be measured in simple artificial environments - unlike other areas of ML in which using real-world data such as images or text is common. The authors propose three new benchmarks to address this disparity. In the first two, an agent is assigned to a random location in an image, and can only observe parts of the image near it. At every time step, it is able to move in one of the cardinal directions, unmasking new sections of the image, until it can classify the image correctly (task 1) or locate a given object (task 2). The third type of benchmark is adding natural video as background to existing Mujoco or Atari tasks. In testing this third category of benchmark, they find that PPO and A2C fall into a local optimum where they ignore the observed state when deciding the next action.

While I agree with some of the concerns laid out in this paper, I'm not sure that these benchmarks are the best way to address them. The third task in particular is mainly testing for ability to ignore the "natural data" used, which doesn't seem very useful. I think a better alternative would be to replace Atari with tasks in procedurally-generated environments with realistic physics engines. However, this paper's benchmarks do benefit from being much easier to produce and less computationally demanding.


Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search

Lars Buesing
Theophane Weber
Yori Zwols
Sebastien Racaniere
Arthur Guez
Jean-Baptiste Lespiau
Nicolas Heess

This paper aims to alleviate the data inefficiency of RL by using a model to synthesise data. However, even when environment dynamics can be modeled accurately, it can be difficult to generate data which matches the true distribution. To solve this problem, the authors use a Structured Causal Model trained to predict the outcomes which would have occurred if different actions had been taken from previous states. Data is then synthesised by rolling out from previously-seen states. The authors test performance in a partially-observable version of SOKOBAN, in which their system outperforms other methods of generating data.

This is an interesting approach which I can imagine becoming useful. It would be nice to see more experimental work in more stochastic environments, though.


Learning Latent Dynamics for Planning from Pixels

Danijar Hafner
Timothy Lillicrap
Ian Fischer
Ruben Villegas
David Ha
Honglak Lee
James Davidson

The authors introduce PlaNet, an agent that learns an environment's dynamics from pixels and then chooses actions by planning in latent space. At each step, it searches for the best action sequence under its Recurrent State Space dynamics model, then executes the first action and replans. The authors note that having a model with both deterministic and stochastic transitions is critical to learning a good policy. They also use a technique called variational overshooting to train the model on multi-step predictions, by generalising the standard variational bound for one-step predictions. PlaNet approaches the performance of top model-free algorithms even when trained on 50x fewer episodes.

This paper seems like a step forward in addressing the instability of using learned models in RL. However, the extent to which it's introducing new contributions, as opposed to combining existing ideas, is a little unclear.


Evolved Policy Gradients

In this meta-learning approach for reinforcement learning, the outer optimization loop proposes a new _loss function_ for the inner loop to optimize (in contrast to eg. MAML, where the outer optimization leads to better initializations for the policy parameters). The outer optimization is done using evolution strategies, while the inner optimization is stochastic gradient descent. The authors see good results on generalization to out-of-distribution tasks, which other algorithms such as RL2 don't achieve.


Open sourcing TRFL: a library of reinforcement learning building blocks

Diego de Las Casas and Gabriel Barth-Maron

DeepMind is open-sourcing a Tensorflow library of "key algorithmic components" used in their RL agents. They hope that this will allow less buggy RL code.

This continues the trend of being able to easily implement deep learning at higher and higher levels of abstraction. I'm looking forward to using it.


Learning Acrobatics by Watching YouTube

Xue Bin (Jason) Peng and Angjoo Kanazawa

To imitate human behavior in videos, it is sufficient to estimate the human pose for each frame, to smooth the poses across frames to eliminate any jittery artifacts or mistakes made by the pose estimator, and then to train the robot to match the motion exactly. This results in really good performance that looks significantly better than corresponding deep RL approaches, but of course it relies on having labeled poses to train the pose estimator in addition to the simulator.

It's quite remarkable how some supervision (poses in this case) can lead to such large improvements in the task. Of course, the promise of deep RL is to accomplish tasks with very little supervision (just a reward function), so this isn't a huge breakthrough, but it's still better than I expected. Intuitively, this works so well because the "reward" during the imitation phase is extremely dense -- the reference motion provides feedback after each action, so you don't have to solve the credit assignment problem.


Introducing a New Framework for Flexible and Reproducible Reinforcement Learning Research

Pablo Samuel Castro and Marc G. Bellemare

Researchers at Google have released Dopamine, a small framework for RL research on Atari games, with four built-in agents -- DQN, C51, a simplified version of Rainbow, and the recent Implicit Quantile Network. There's a particular emphasis on reproducibility, by providing logs from training runs, training data, etc.


The International 2018: Results

Two human teams beat OpenAI Five at The International. The games seemed much more like regular Dota, probably because there was now only one vulnerable courier for items instead of five invulnerable ones. This meant that OpenAI Five's strategy of a relentless team attack on the enemy was no longer as powerful, because they couldn't get the health regeneration items they needed to constantly stay alive to continue the attack. It's also possible (but less likely to me) that the matches were more normal because the teams were more even, or because the human teams knew about Five's strategy this time and were countering it in ways that I don't understand.

There are still some things that the bots do that seem like bad decisions. You can interpret this a few ways. Five could have learned a large number of heuristics that make it good enough to beat almost all humans, but that break down in edge cases. In this story, Five is not good at learning logical or abstract reasoning, but can compensate for that in the average case with the sheer number of heuristics it can learn. Another interpretation is that Five learns a good representation of Dota which lets it come up with new, novel insights into the game, which we can't see or understand because the representation is alien to us. However, the representation makes it harder to come up with other insights about Dota that we have using our representations of Dota, and as a result Five makes some mistakes that humans can easily recognize as mistakes. I lean towards the first interpretation, but not very strongly.


Learning Actionable Representations from Visual Observations

Debidatta Dwibedi
Jonathan Tompson
Corey Lynch
Pierre Sermanet

Prior work on Time Contrastive Networks (TCN)s showed that you can use time as an unsupervised learning signal, in order to learn good embeddings of states that you can then use in other tasks. This paper extends TCNs to work with multiple frames, so that it can understand motion as well. Consider any two short videos of a task demonstration. If they were taken at different times, then they should be mapped to different embedding vectors (since they correspond to different "parts" of the task). On the other hand, if they were taken at the same time (even if from different viewpoints), they should be mapped to the same embedding vector. The loss function based on this encourages the network to learn an embedding for these short videos that is invariant to changes in perspective (which are very large changes in pixel-space), but _is_ different for changes in time (which may be very small changes in pixel-space). They evaluate with a bunch of different experiments.

Unsupervised learning seems like the way forward to learn rich models of the world, because of the sheer volume of data that you can use.


Learning Plannable Representations with Causal InfoGAN

Thanard Kurutach
Aviv Tamar
Ge Yang
Stuart Russell
Pieter Abbeel

Hierarchical reinforcement learning aims to learn a hierarchy of actions that an agent can take, each implemented in terms of actions lower in the hierarchy, in order to get more efficient planning. Another way we can achieve this is to use a classical planning algorithm to find a sequence of _waypoints_, or states that the agent should reach that will allow it to reach its goal. These waypoints can be thought of as a high-level plan. You can then use standard RL algorithms to figure out how to go from one waypoint to the next. However, typical planning algorithms that can produce a sequence of waypoints require very structured state representations, that were designed by humans in the past. How can we learn them directly from data? This paper proposes Causal InfoGAN. They use a GAN where the generator creates adjacent waypoints in the sequence, while the discriminator tries to distinguish between waypoints from the generator and pairs of points sampled from the true environment. This incentivizes the generator to generate waypoints that are close to each other, so that we can use an RL algorithm to learn to go from one waypoint to the next. However, this only lets us generate adjacent waypoints. In order to use this to make a sequence of waypoints that gets from a start state to a goal state, we need to use some classical planning algorithm. In order to do that, we need to have a structured state representation. GANs do not do this by default. InfoGAN tries to make the latent representation in a GAN more meaningful by providing the generator with a "code" (a state in our case) and maximizing the mutual information of the code and the output of the generator. In this setting, we want to learn representations that are good for planning, so we want to encode information about _transitions_ between states. This leads to the Causal InfoGAN objective, where we provide the generator with a pair of abstract states (s, s'), have it generate a pair of observations (o, o') and maximize the mutual information between (s, s') and (o, o'), so that s and s' become good low-dimensional representations of o and o'. They show that Causal InfoGAN can create sequences of waypoints in a rope manipulation task, that previously had to be done manually.

We're seeing more and more work combining classical symbolic approaches with the current wave of statistical machine learning from big data, that gives them the best of both worlds. While the results we see are not general intelligence, it's becoming less and less true that you can point to a broad swath of capabilities that AI cannot do yet. I wouldn't be surprised if a combination of symbolic and stastical AI techniques led to large capability gains in the next few years.