The Alignment Problem for Bayesian History-Based Reinforcement Learners

After forgetting its existence for quite a while, I've finally read through this technical report (which won first place in <@round 2 of the AI alignment prize@>(@Announcement: AI alignment prize round 2 winners and next round@)). It analyzes the alignment problem from an AIXI-like perspective, that is, by theoretical analysis of powerful Bayesian RL agents in an online POMDP setting.

In this setup, we have a POMDP environment, in which the environment has some underlying state, but the agent only gets observations of the state and must take actions in order to maximize rewards. The authors consider three main setups: 1) rewards are computed by a preprogrammed reward function, 2) rewards are provided by a human in the loop, and 3) rewards are provided by a _reward predictor_ which is trained interactively from human-generated data.

For each setup, they consider the various objects present in the formalism, and ask how these objects could be corrupted, misspecified, or misleading. This methodology allows them to identify several potential issues, which I won't get into as I expect most readers are familiar with them. (Examples include wireheading and threatening to harm the human unless they provide maximal reward.)

They also propose several tools that can be used to help solve misalignment. In order to prevent reward function corruption, we can have the agent _simulate_ the future trajectory, and _evaluate_ this future trajectory with the current reward, removing the incentive to corrupt the reward function. (This was later developed into <@current-RF optimization@>(@Designing agent incentives to avoid reward tampering@).)

Self-corruption awareness refers to whether or not the agent is aware that its policy can be modified. A self-corruption _unaware_ agent is one that behaves as though it's current policy function will never be changed, effectively ignoring the possibility of corruption. It is not clear which is more desirable: while a self-corruption unaware agent will be more corrigible (in the [MIRI sense](https://intelligence.org/files/Corrigibility.pdf)), it also will not preserve its utility function, as it believes that even if the utility function changes the policy will not change.

Action-observation grounding ensures that the agent only optimizes over policies that work on histories of observations and actions, preventing agents from constructing entirely new observation channels ("delusion boxes") which mislead the reward function into thinking everything is perfect.

The interactive setting in which a reward predictor is trained based on human feedback offers a new challenge: that the human data can be corrupted or manipulated. One technique to address this is to get _decoupled_ data: if your corruption is determined by the current state s, but you get feedback about some different state s', as long as s and s' aren't too correlated it is possible to mitigate potential corruptions.

Another leverage point is how we decide to use the reward predictor. We could consider the _stationary_ reward function, which evaluates simulated trajectories with the _current_ reward predictor, i.e. assuming that the reward predictor will never be updated again. If we combine this with self-corruption unawareness (so that the policy also never expects the policy to change), then the incentive to corrupt the reward predictor's data is removed. However, the resulting agent is _time-inconsistent_: it acts as though its reward never changes even though it in practice does, and so it can make a plan and start executing it, only to switch over to a new plan once the reward changes, over and over again.

The _dynamic_ reward function avoids this pitfall by evaluating the kth timestep of a simulated trajectory by also taking an expectation over future data that the reward predictor will get. This agent is no longer time-inconsistent, but it now incentivizes the agent to manipulate the data. This can be fixed by building a single integrated Bayesian agent, which maintains a single environment model that predicts both the reward function and the environment model. The resulting agent is time-consistent, utility-preserving, and has no direct incentive to manipulate the data. (This is akin to the setup in <@assistance games / CIRL@>(@Cooperative Inverse Reinforcement Learning@).)

One final approach is to use a _counterfactual_ reward function, in which the data is simulated in a counterfactual world where the agent executed some known safe default policy. This no longer depends on the current time, and is not subject to data corruption since the data comes from a hypothetical that is independent of the agent's actual policy. However, it requires a good default policy that does the necessary information-gathering actions, and requires the agent to have the ability to simulate human feedback in a counterfactual world.

This paper is a great organization and explanation of several older papers (that haven't been summarized in this newsletter because they were published before 2018 and I read them before starting this newsletter), and I wish I had read it sooner. It seems to me that the integrated Bayesian agent is the clear winner -- the only downside is the computational cost, which would be a bottleneck for any of the models considered here.

One worry I have with this sort of analysis is that the guarantees you get out of it depends quite a lot on how you model the situation. For example, let's suppose that after I sleep I wake up refreshed and more capable of intellectual work. Should I model this as "policy corruption", or as a fixed policy that takes as an input some information about how rested I am?


AI Research Considerations for Human Existential Safety

This research agenda out of CHAI directly attacks the problem longtermists care about: **how to prevent AI-related existential catastrophe**. This is distinctly different from the notion of being "provably beneficial": a key challenge for provable beneficence is defining what we even mean by "beneficial". In contrast, there are avenues for preventing AI-caused human extinction that do not require an understanding of "beneficial": most trivially, we could coordinate to never build AI systems that could cause human extinction.

Since the focus is on the _impact_ of the AI system, the authors need a new phrase for this kind of AI system. They define a **prepotent AI system** to be one that cannot be controlled by humanity **and** has the potential to transform the world in a way that is at least as impactful as humanity as a whole. Such an AI system need not be superintelligent, or even an AGI; it may have powerful capabilities in a narrow domain such as technological autonomy, replication speed, or social acumen that enable prepotence.

By definition, a prepotent AI system is capable of transforming the world drastically. However, there are a lot of conditions that are necessary for continued human existence, and most transformations of the world will not preserve these conditions. (For example, consider the temperature of the Earth or the composition of the atmosphere.) As a result, human extinction is the _default_ outcome from deploying a prepotent AI system, and can only be prevented if the system is designed to preserve human existence with very high precision relative to the significance of its actions. They define a misaligned prepotent AI system (MPAI) as one whose deployment leads to human extinction, and so the main objective is to avert the deployment of MPAI.

The authors break down the risk of deployment of MPAI into five subcategories, depending on the beliefs, actions and goals of the developers. The AI developers could fail to predict prepotence, fail to predict misalignment, fail to coordinate with other teams on deployment of systems that aggregate to form an MPAI, accidentally (unilaterally) deploy MPAI, or intentionally (unilaterally) deploy MPAI. There are also hazardous social conditions that could increase the likelihood of risks, such as unsafe development races, economic displacement of humans, human enfeeblement, and avoidance of talking about x-risk at all.

Moving from risks to solutions, the authors categorize their research directions along three axes based on the setting they are considering. First, is there one or multiple humans; second, is there one or multiple AI systems; and third, is it helping the human(s) comprehend, instruct, or control the AI system(s). So, multi/single instruction would involve multiple humans instructing a single AI system. While we will eventually need multi/multi, the preceding cases are easier problems from which we could gain insights that help solve the general multi/multi case. Similarly, comprehension can help with instruction, and both can help with control.

The authors then go on to list 29 different research directions, which I'm not going to summarize here.

I love the abstract and introduction, because of their directness at actually stating what we want and care about. I am also a big fan of the distinction between provably beneficial and reducing x-risk, and the single/multi analysis.

The human fragility argument, as applied to generally intelligent agents, is a bit tricky. One interpretation is that the "hardness" stems from the fact that you need a bunch of "bits" of knowledge / control in order to keep humans around. However, it seems like a generally intelligent AI should easily be able to keep humans around "if it wants", and so the bits already exist in the AI. (As an analogy: we make big changes to the environment, but we could easily preserve deer habitats if we wanted to.) Thus, it is really a question of what "distribution" you expect the AI system is sampled from: if you think we'll build AI systems that try to do what humanity wants, then we're probably fine, but if you think that there will be multiple AI systems that each do what their users want, but the users have conflicts, the overall system seems more "random" in its goals, and so more likely to fall into the "default" outcome of human extinction.

The research directions are very detailed, and while there are some suggestions that don't seem particularly useful to me, overall I am happy with the list. (And as the paper itself notes, what is and isn't useful depends on your models of AI development.)


Summary of the Technical Safety Workshop

David identifies two broad types of AI safety work: human in the loop approaches, and theory approaches. A notable subset of the former category is methods which improve our ability to give advanced systems meaningful feedback - this includes debate, IDA, and recursive reward modeling. CIRL and CAIS are also human-in-the-loop. Meanwhile the theory category includes MIRI's work on agent foundations; side effect metrics; and verified boxing.


Measurement, Optimization, and Take-off Speed

In this blogpost, the author argues that "trying to measure pretty much anything you can think of is a good mental move that is heavily underutilized in machine learning". He motivates the value of measurement and additional metrics by (i) citing evidence from the history of science, policy-making, and engineering (e.g. x-ray crystallography contributed to rapid progress in molecular biology), (ii) describing how, conceptually, "measurement has several valuable properties" (one of which is to act as interlocking constraints that help to error-check theories), and (iii) providing anecdotes from his own research endeavours where such approaches have been productive and useful (see, e.g. <@Rethinking Bias-Variance Trade-off@>(@Rethinking Bias-Variance Trade-off for Generalization of Neural Networks@)).

He demonstrates his proposal by applying it to the notion of _optimization power_ -- an important idea that has not been measured or even framed in terms of metrics. Two metrics are offered: (a) the change (typically deterioration) of performance when trained with a perturbed objective function with respect to the original objective function, named _Outer Optimization_, and (b) the change in performance of agents during their own lifetime (but without any further parameter updates), such as the log-loss on the next sentence for a language model after it sees X number of sequences at test time, or _Inner Adaptation_. Inspired by these, the article includes research questions and possible challenges.

He concludes with the insight that take-off would depend on these two continuous processes, Outer Optimization and Inner Adaptation, that work on very different time-scales, with the former being, at this time, much quicker than the latter. However, drawing an analogy from evolution, where it took billions of years of optimization to generate creatures like humans that were exceptional at rapid adaptation, we might yet see a fast take-off were Inner Adaptation turns out to be an exponential process that dominates capabilities progress. He advocates for early, sensitive measurement of this quantity as it might be an early warning sign of imminent risks.

Early on, this post reminded me of [Twenty Billion Questions](https://arxiv.org/pdf/1705.10720.pdf); even though they are concretely different, these two pieces share a conceptual thread. They both consider the measurement of multiple quantities essential for solving their problems: 20BQ for encouraging AIs to be low-impact, and this post for productive framings of ill-defined concepts and as a heads-up about potential catastrophes.

Measurement is important, and this article poignantly argues why and illustrates how. It volunteers potential ideas that can be worked on today by mainstream ML researchers, and offers up a powerful toolkit to improve one's own quality of analysis. It would be great to see more examples of this technique applied to other contentious, fuzzy concepts in ML and beyond. I'll quickly note that while there seems to be minimal interest in this from academia, measurement of optimization power has been discussed earlier in several ways, e.g. [Measuring Optimization Power](https://www.lesswrong.com/posts/Q4hLMDrFd8fbteeZ8/measuring-optimization-power), or <@the ground of optimization@>(@The ground of optimization@).

**Rohin's opinion:** I broadly agree with the perspective in this post. I feel especially optimistic about the prospects of measurement for (a) checking whether our theoretical arguments hold in practice and (b) convincing others of our positions (assuming that the arguments do hold in practice).


The Learning-Theoretic AI Alignment Research Agenda

This agenda aims to create a general abstract theory of intelligence (in a manner similar to [AIXI](https://en.wikipedia.org/wiki/AIXI), but with some deficiencies removed). In particular, once we use the framework of reinforcement learning, regret bounds are a particular way of provably quantifying an agent's intelligence (though there may be other ways as well). Once we have this theory, we can ground all other AI alignment problems within it. Specifically, alignment would be formalized as a value learning protocol that achieves some regret bound. With this formalization, we can solve hard metaphilosophy problems such as "What is imperfect rationality?" through the intuitions gained from looking at the problem through the lens of value learning protocols and universal reinforcement learning.

This agenda, like others, is motivated by the scenario where we need to get alignment right the first time, without empirical feedback loops, both because we might be facing one-shot success or failure, and because the stakes are so high that we should aim for high reliability subject to time constraints. I put low probability on the first reason (alignment being one-shot), and it seems much less tractable, so I mostly ignore those scenarios. I agree with the second reason, but aiming for this level of rigor seems like it will take much longer than the time we actually have. Given this high level disagreement, it's hard for me to evaluate the research agenda itself.


Conceptual issues in AI safety: the paradigmatic gap

Lots of current work on AI safety focuses on what we can call "mid-term safety" -- the safety of AI systems that are more powerful and more broadly deployed than the ones we have today, but work using relatively similar techniques as the ones we use today. However, it seems plausible that there will be a paradigm shift in how we build AI systems, and if so it's likely that we will have a new, completely different set of mid-term concerns, rendering the previous mid-term work useless. For example, at the end of the 19th century, horse excrement was a huge public health hazard, and "mid-term safety" would likely have been about how to remove the excrement. Instead, the automobile was developed and started replacing horses, leading to new set of mid-term concerns (eg. pollution, traffic accidents), and any previous work on removing horse excrement became near-useless.

I focus almost exclusively on mid-term safety (while thinking about long-term safety), not because I disagree with this argument, but in spite of it. I think there is a good chance that any work I do will be useless for aligning superintelligent AI because of a paradigm shift, but I do it anyway because it seems very important on short timelines, which are easier to affect; and I don't know of other approaches to take that would have a significantly higher probability of being useful for aligning superintelligent AI.

A possible stance for AI control research


The flaws that make today’s AI architecture unsafe and a new approach that could fix it

This podcast delves into many of the ideas in Stuart’s book <@Human Compatible@>(@Human Compatible: Artificial Intelligence and the Problem of Control@). Rob especially pushes on some aspects that are less talked about in the AI safety community, like the enfeeblement problem and whether we’d be locking in suboptimal values. They also discuss Stuart’s response to some counterarguments.

One of the counterarguments the podcast talks about is <@my position@>(@Conversation with Rohin Shah@) that we’ll probably learn from smaller catastrophes in order to avoid actual extinction. I just want to note that while it might sound like I disagree with Stuart on this point, I don’t think we actually do. I was arguing against the position that extinction is the default outcome (> 50% probability) while Stuart is arguing against the position that extinction is near-impossible (~0% probability). I ended up around 10%; I’d guess that if Stuart were forced to, he’d give a number similar to mine, for similar reasons as me.


Cooperation, Conflict, and Transformative Artificial Intelligence: A Research Agenda

This agenda by the [Effective Altruism Foundation](https://ea-foundation.org/) focuses on risks of astronomical suffering (s-risks) posed by <@Transformative AI@>(@Defining and Unpacking Transformative AI@) (TAI) and especially those related to conflicts between powerful AI agents. This is because there is a very clear path from extortion and executed threats against altruistic values to s-risks.  While especially important in the context of s-risks, cooperation between AI systems is also relevant from a range of different viewpoints. The agenda covers four clusters of topics: strategy, credibility and bargaining, current AI frameworks, as well as decision theory.

The extent of cooperation failures is likely influenced by how power is distributed after the transition to TAI. At first glance, it seems like widely distributed scenarios (as <@CAIS@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@)) are more problematic, but related literature from international relations paints a more complicated picture. The agenda seeks a better understanding of how the distribution of power affects catastrophic risk, as well as potential levers to influence this distribution. Other topics in the strategy/governance cluster include the identification and analysis of realistic scenarios for misalignment, as well as case studies on cooperation failures in humans and how they can be affected by policy.

TAI might enable unprecedented credibility, for example by being very transparent, which is crucial for both contracts and threats. The agenda aims at better models of the effects of credibility on cooperation failures. One approach to this is open-source game theory, where agents can see other agents' source codes. Promising approaches to prevent catastrophic cooperation failures include the identification of peaceful bargaining mechanisms, as well as surrogate goals. The idea of surrogate goals is for an agent to commit to act as if it had a different goal, whenever it is threatened, in order to protect its actual goal from threats.

As some aspects of contemporary AI architectures might still be present in TAI, it can be useful to study cooperation failure in current systems. One concrete approach to enabling cooperation in social dilemmas that could be tested with contemporary systems is based on bargaining over policies combined with punishments for deviations.  Relatedly, it is worth investigating whether or not multi-agent training leads to human-like bargaining by default. This has implications on the suitability of behavioural vs classical game theory to study TAI. The behavioural game theory of human-machine interactions might also be important, especially in human-in-the-loop scenarios of TAI.

The last cluster discusses the implications of bounded computation on decision theory as well as the decision theories (implicitly) used by current agent architectures. Another focus lies on acausal reasoning and in particular the possibility of [acausal trade](https://wiki.lesswrong.com/wiki/Acausal_trade), where different correlated AI systems cooperate without any causal links between them.

I am broadly sympathetic to the focus on preventing the worst outcomes and it seems plausible that extortion could play an important role in these, even though I worry more about distributional shift plus incorrigibility. Still, I am excited about the focus on cooperation, as this seems robustly useful for a wide range of scenarios and most value systems.

**Rohin's opinion:** Under a suffering-focused ethics under which s-risks far overwhelm x-risks, I think it makes sense to focus on this agenda. There don't seem to be many plausible paths to s-risks: by default, we shouldn't expect them, because it would be quite surprising for an amoral AI system to think it was particularly useful or good for humans to _suffer_, as opposed to not exist at all, and there doesn't seem to be much reason to expect an immoral AI system. Conflict and the possibility of carrying out threats are the most plausible ways by which I could see this happening, and the agenda here focuses on neglected problems in this space.

However, under other ethical systems (under which s-risks are worse than x-risks, but do not completely dwarf x-risks), I expect other technical safety research to be more impactful, because other approaches can more directly target the failure mode of an amoral AI system that doesn't care about you, which seems both more likely and more amenable to technical safety approaches (to me at least). I could imagine work on this agenda being quite important for _strategy_ research, though I am far from an expert here.


Just Imitate Humans?

This post asks whether it is safe to build AI systems that just imitate humans. The comments have a lot of interesting debate.


Four Ways An Impact Measure Could Help Alignment

Much <@recent@>(@Towards a New Impact Measure@) <@work@>(@Designing agent incentives to avoid side effects@) has focused on quantifying the effect an AI has on the world, aka measuring impact, though some are [skeptical](https://www.alignmentforum.org/posts/kCY9dYGLoThC3aG7w/best-reasons-for-pessimism-about-impact-of-impact-measures). This post presents four potential ways impact measures could help with AI alignment. _First_, impact could act as a **regularizer**: an untrained AI attempting to do value learning could have an impact penalty that prevents it from taking dangerous actions before it is confident it has learned the right utility function. _Second_, impact could act as a **safety protocol**: if our training process is dangerous, e.g. due to <@mesa optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@), we can penalize impact during training to safely test models that may be misaligned. _Third_, impact could act as an **influence-limiter**: impact measures could help us construct AIs with intentionally limited scope that won’t heavily optimize the world as a side effect. _Fourth_, impact could help us with **deconfusion**: even if impact measures themselves aren't used, conceptual clarity about impact could help us gain conceptual clarity about other important concepts such as corrigibility, mild optimization, etc.

I am most excited about impact as a **regularizer** and impact as a **safety protocol**. I feel like AIs that are impact-limited at runtime (the **influence-limiter** case) are unlikely to be competitive with other AIs that have no impact penalty (this is discussed in the post). I found the argument that impact could be particularly useful for **deconfusion** uncompelling.

**Rohin's opinion**: It seems to me like the safety protocol argument is for limited actions at training time, while the influence limiter argument is for limited actions at test time. I don't really get how the regularizer is supposed to be different from these two cases -- perhaps the idea is that it is a regularizer specifically on the distribution over utility functions that the AI is optimizing? This is still confusing, I would have expected the influence limiter case to also be a change to the utility function. Like Asya, I am worried about competitiveness: see the post about [reversible changes](https://www.alignmentforum.org/posts/zrunBA8B5bmm2XZ59/reversible-changes-consider-a-bucket-of-water) below.


Unsolved research problems vs. real-world threat models

Papers on adversarial examples often suggest that adversarial examples can lead to real world problems as their motivation. As we've <@seen@>(@Motivating the Rules of the Game for Adversarial Example Research@) <@previously@>(@Introducing the Unrestricted Adversarial Examples Challenge@), many adversarial example settings are not very realistic _threat models_ for any real world problem. For example, adversarial "stickers" that cause vision models to fail to recognize stop signs could cause an autonomous vehicle to crash... but an adversary could also just knock over the stop sign if that was their goal.

There are more compelling reasons that we might care about imperceptible perturbation adversarial examples. First, they are a proof of concept, demonstrating that our ML models are not robust and make "obvious" mistakes and so cannot be relied on. Second, they form an unsolved research problem, in which progress can be made more easily than in real settings, because it can be formalized straightforwardly (unlike realistic settings). As progress is made in this toy domain, it can be used to inform new paradigms that are closer to realistic settings. But it is _not_ meant to mimic real world settings -- in the real world, you need a threat model of what problems can arise from the outside world, which will likely suggest much more basic concerns than the "research problems", requiring solutions involving sweeping design changes rather than small fixes.

I strongly agree with the points made in this post. I don't know to what extent researchers themselves agree with this point -- it seems like there is _a lot_ of adversarial examples research that is looking at the imperceptible perturbation case and many papers that talk about new types of adversarial examples, without really explaining why they are doing this or giving a motivation that is about unsolved research problems rather than real world settings. It's possible that researchers do think of it as a research problem and not a real world problem, but present their papers differently because they think that's necessary in order to be accepted.

The distinction between research problems and real world threat models seem to parallel the distinction between theoretical or conceptual research and engineering in AI safety. The former typically asks questions of the form "how could we do this in principle, making simplifying assumptions X, Y and Z", even though X, Y and Z are known not to hold in the real world, for the sake of having greater conceptual clarity that can later be leveraged as a solution to a real world problem. Engineering work on the other hand is typically trying to scale an approach to a more complex environment (with the eventual goal of getting to a real world problem).


FLI Podcast: AI Breakthroughs and Challenges in 2018 with David Krueger and Roman Yampolskiy

Ariel Conn
David Krueger and Roman Yampolskiy

David and Roman review AI progress in 2018 and speculate about its implications. Roman identified a pattern where we see breakthroughs like [AlphaZero](https://deepmind.com/blog/alphazero-shedding-new-light-grand-games-chess-shogi-and-go/) ([AN #36](https://mailchi.mp/6751e45fbb48/alignment-newsletter-36)), [AlphaStar](https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/) ([AN #43](https://mailchi.mp/768a8130013f/alignment-newsletter-43)) and [AlphaFold](https://deepmind.com/blog/alphafold/) ([AN #36](https://mailchi.mp/6751e45fbb48/alignment-newsletter-36)) so frequently now that it no longer seems as impressive when a new one comes out. David on the other hand sounded less impressed by progress on Dota and StarCraft, since both AI systems were capable of executing actions that humans could never do (fast reaction times for Dota and high actions-per-minute for StarCraft). He also thought that these projects didn't result in any clear general algorithmic insights the way AlphaZero did.

On the deep RL + robotics side, David identified major progress in [Dactyl](https://blog.openai.com/learning-dexterity/) ([AN #18](https://mailchi.mp/51717855ea27/alignment-newsletter-18)) and [QT-Opt](https://ai.googleblog.com/2018/06/scalable-deep-reinforcement-learning.html) (which I remember reading and liking but apparently I failed to put in the newsletter). He also cited GANs as having improved significantly, and talked about feature-wise transformations in particular. Roman noted the improving performance of evolutionary algorithms.

David also noted how a lot of results were obtained by creating algorithms that could scale, and then using a huge amount of compute for them, quoting [AI and Compute](https://blog.openai.com/ai-and-compute/) ([AN #7](https://mailchi.mp/3e550712419a/alignment-newsletter-7)), [Interpreting AI Compute Trends](https://aiimpacts.org/interpreting-ai-compute-trends/) ([AN #15](https://mailchi.mp/4920e52dd61b/alignment-newsletter-15)) and [Reinterpreting AI and Compute](https://aiimpacts.org/reinterpreting-ai-and-compute/) ([AN #38](https://mailchi.mp/588354e4b91d/alignment-newsletter-38)).

On the policy side, they talked about deep fakes and the general trend that AI may be progressing to fast for us to keep up with its security implications. They do find it promising that researchers are beginning to accept that their research does have safety and security implications.

On the safety side, David noted that the main advance seemed to be with approaches using [superhuman feedback](https://www.lesswrong.com/posts/naccwaCQEEBXK7hiJ/my-use-of-the-phrase-super-human-feedback), including [debate](https://blog.openai.com/debate/) ([AN #5](https://mailchi.mp/0ae5d69de63b/alignment-newsletter-5)), [iterated amplification](https://blog.openai.com/amplifying-ai-training/) (discussed frequently in this newsletter, but that paper was in [AN #30](https://mailchi.mp/c1f376f3a12e/alignment-newsletter-30)) and [recursive reward modeling](https://medium.com/@deepmindsafetyresearch/scalable-agent-alignment-via-reward-modeling-bf4ab06dfd84) ([AN #34](https://mailchi.mp/f1947668b183/alignment-newsletter-34)). He also identified [unrestricted adversarial examples](https://ai.googleblog.com/2018/09/introducing-unrestricted-adversarial.html) ([AN #24](https://mailchi.mp/d7b5059d64ed/alignment-newsletter-24)) as an area to watch in the future.

I broadly agree with the areas of AI progress identified here, though I would probably also throw in NLP, e.g. [BERT](https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html). I disagree on the details -- for example, I think that [OpenAI Five](https://blog.openai.com/openai-five/) ([AN #13](https://mailchi.mp/8234356e4b7f/alignment-newsletter-13)) was much better than I would have expected at the time and the same would have been true of AlphaStar if I hadn't already seen OpenAI Five, and the fact that they did a few things that humans can't do barely diminishes the achievement at all. (My take is pretty similar to Alex Irpan's take in his [post on AlphaStar](https://www.alexirpan.com/2019/02/22/alphastar.html).)


Inverse Reinforcement Learning and Inferring Human Preference with Dylan Hadfield-Menell

A few weeks ago, Lucas Perry interviewed Dylan Hadfield-Menell on the FLI podcast about his research (which includes papers like [Cooperative Inverse Reinforcement Learning](https://arxiv.org/abs/1606.03137),  [The Off-Switch Game](https://arxiv.org/abs/1611.08219), and [Inverse Reward Design](https://arxiv.org/abs/1711.02827)). They discussed a variety of topics including the motivations behind Dylan's research, future directions, thoughts on hard problems such as corrigibility and preference aggregation, etc.

This is probably most useful for understanding the motivations behind many of Dylan's papers and how they all tie into each other, which can be hard to glean just from reading the papers. There were also a lot of framings of problems that felt useful to me that I haven't seen elsewhere.


Why I expect successful alignment

This post gives three arguments that we will likely solve the narrow alignment problem of having an AI system do what its operators intend it to do. First, advanced AI systems may be developed in such a way that the alignment problem doesn't even happen, at least as we currently conceive of it. For example, under the comprehensive AI services model, there are many different AI services that are superintelligent at particular tasks that can work together to accomplish complex goals, but there isn't a single unified agent to "align". Second, if it becomes obvious that alignment will be a serious problem, then we will devote a lot of resources to tackling the problem. We already see reward hacking in current systems, but it isn't sufficiently dangerous yet to merit the application of a lot of resources. Third, we have already come up with some decent approaches that seem like they could work.

I generally agree with these arguments and the general viewpoint that we will probably solve alignment in this narrow sense. The most compelling argument to me is the second one, that we will eventually devote significant resources to the problem. This does depend on the crux that we see examples of these problems and how they could be dangerous before it is too late.

I also agree that it's much less clear whether we will solve other related problems, such as how to deal with malicious uses of AI, issues that arise when multiple superintelligent AI systems aligned with different humans start to compete, and how to ensure that humans have "good" values. I don't know if this implies that _on the margin_ it is more useful to work on the related problems. It could be that these problems are so hard that there is not much that we can do. (I'm neglecting [importance of the problem](https://concepts.effectivealtruism.org/concepts/importance-neglectedness-tractability/) here.)


Mechanism design for AI

One cause of outcomes worse than extinction could be escalating conflicts between very capable AI systems (that could eg. threaten to simulate suffering beings). It is worth studying how we could have AI systems implement mechanism design in order to guide such systems into more cooperative behavior.

Adaptive Mechanism Design: Learning to Promote Cooperation


A Summary of Concrete Problems in AI Safety

A nice summary of [Concrete Problems in AI Safety](https://blog.openai.com/concrete-ai-safety-problems/) that's a lot quicker to read than the original paper.

I like it -- I think I will send this to newer researchers as a precursor to the full paper.


Mechanistic Transparency for Machine Learning

One useful thread of alignment research would be to figure out how to take a neural net, and distill parts or all of it into pseudocode or actual code that describes how the neural net actually works. This could then be read and analyzed by developers to make sure the neural net is doing the right thing. Key quote: "I'm excited about this agenda because I see it as giving the developers of AI systems tools to detect and correct properties of their AI systems that they see as undesirable, without having to deploy the system in a test environment that they must laboriously ensure is adequately sandboxed."

I would be really excited to see good work on this agenda, it would be a big step forward on how good our design process for neural nets is.


Alignment of Language Agents

Zachary Kenton
Tom Everitt
Laura Weidinger
Iason Gabriel
Vladimir Mikulik
Geoffrey Irving

This paper analyzes the various problems we consider in AI alignment from the perspective of language agents. Problems covered include <@specification gaming@>(@Specification gaming examples in AI@), <@whom and what to align to@>(@Artificial Intelligence, Values and Alignment@), <@intent alignment@>(@Clarifying "AI Alignment"@), <@removing tampering incentives@>(@Avoiding Tampering Incentives in Deep RL via Decoupled Approval@), and <@inner alignment@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@). These can be categorized as different kinds of misspecification, namely misspecification in the _training data_, the _training process_, and the _behavior under distributional shift_.

While the conceptual problems are similar to the ones already considered for embodied RL agents, the ways they manifest are different. In particular, the authors highlight the possibility that language agents will _deceive us_, _manipulate us_, or _produce harmful content_. The authors review some existing definitions of deception and manipulation that are purely behavioral (that is, the definitions do not require an _intent_ to deceive or manipulate). A signaller **deceives** a receiver if the signaller transmits (or suggestively doesn’t transmit) a signal that causes the receiver to believe some false claim that benefits the signaller. **Manipulation** is similar, except rather than causing the receiver to believe a false claim, it causes the receiver to take some action that benefits the signaller, that in some sense the receiver “shouldn’t” have taken. We could cash out “the receiver ‘shouldn’t’ have taken the action” just as "the action is harmful to the receiver", but from a safety / security mindset, the authors prefer a broader definition that aims to identify bad _means_ of influencing the receiver, instead of only focusing on whether the _ends_ were bad.

Some other miscellaneous points:
- Since the “action space” is just language, it seems like it should be easier (though still requires work) to prevent language agents from causing physical harm.
- It will hopefully be easier to train language agents to be explainable, since they have native fluency in natural language with which they can explain their behavior.


An introduction to worst-case AI safety

Argues that people with suffering-focused ethics should focus on "worst-case AI safety", which aims to find technical solutions to risks of AIs creating vast amounts of suffering (which would be much worse than extinction).

If you have strongly suffering-focused ethics (unlike me), this seems mostly right. The post claims that suffering-focused AI safety should be more tractable than AI alignment, because it focuses on a subset of risks and only tries to minimize them. However, it's not necessarily the case that focusing on a simpler problem makes it easier to solve. It feels easier to me to figure out how to align an AI system to humans, or how to enable human control of an AI system, than to figure out all the ways in which vast suffering could happen, and solve each one individually. You can make an analogy to mathematical proofs and algorithms -- often, you want to try to prove a _stronger_ statement than the one you are looking at, because when you use induction or recursion, you can rely on a stronger inductive hypothesis.


AI Paradigms and AI Safety: Mapping Artefacts and Techniques to Safety Issues

Jose Hernandez-Orallo
Fernando Martinez-Plumed
Shahar Avin
Jess Whittlestone
Seán Ó hÉigeartaigh

What should prioritization within the field of AI safety look like? Ideally, we would proactively look for potential issues that could arise with many potential AI technologies, making sure to cover the full space of possibilities rather than focusing on a single area. What does prioritization look like in practice? This paper investigates, and finds that it is pretty different from this ideal.

In particular, they define a set of 14 categories of AI _techniques_ (examples include neural nets, planning and scheduling, and combinatorial optimization), and a set of 10 kinds of AI _artefacts_ (examples include agents, providers, dialoguers, and swarms). They then analyze trends in the amount of attention paid to each technique or artefact, both for AI safety and AI in general. Note that they construe AI safety very broadly by including anything that addresses potential real-world problems with AI systems.

While there are a lot of interesting trends, the main conclusion is that there is an approximately 5-year delay between the emergence of an AI paradigm and safety research into that paradigm. In addition, safety research tends to neglect non-dominant paradigms.

One possible conclusion is that safety research should be more diversified across different paradigms and artefacts, in order to properly maximize expected safety. However, this isn’t obvious: it seems likely that if the dominant paradigm has 50% of the research, it will also have, say, 80% of future real-world deployments, and so it could make sense to have 80% of the safety research focused on it. Rather than try to predict which paradigm will become dominant (a very difficult task), it may be more efficient to simply observe which paradigm becomes dominant and then redirect resources at that time (even though that process takes 5 years to happen).


Multi-agent minds and AI alignment

This post argues against the model of humans as optimizing some particular utility function, instead favoring a model based on predictive processing. This leads to several issues with the way standard value learning approaches like inverse reinforcement learning work. There are a few suggested areas for future research. First, we could understand how hierarchical models of the world work (presumably for better value learning). Second, we could try to invert game theory to learn objectives in multiagent settings. Third, we could learn preferences in multiagent settings, which might allow us to better infer norms that humans follow. Fourth, we could see what happens if we take a system of agents, infer a utility function, and then optimize it -- perhaps one of the agents' utility functions dominates? Finally, we can see what happens when we take a system of agents and give it more computation, to see how different parts scale. On the non-technical side, we can try to figure out how to get humans to be more self-aligned (i.e. there aren't "different parts pulling in different directions").

I agree with the general point that figuring out a human utility function and then optimizing it is unlikely to work, but for different reasons (see the first chapter of the [Value Learning sequence](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc)). I also agree that humans are complex and you can’t get away with modeling them as Boltzmann rational and optimizing some fixed utility function. I wouldn’t try to make the model more accurate (eg. a model of a bunch of interacting subagents, each with their own utility function), I would try to make the model less precise (eg. a single giant neural net), because that reduces the chance of model misspecification. However, given the [impossibility result](https://arxiv.org/abs/1712.05812) saying that you must make assumptions to make this work, we probably have to give up on having some nice formally specified meaning of “values”. I think this is probably fine -- for example, iterated amplification doesn’t have any explicit formal value function.


ICML Uncertainty and Robustness Workshop Accepted Papers

The Uncertainty and Robustness Workshop accepted papers are available. Topics include out-of-distribution detection, generalization to stochastic corruptions, label corruption robustness, and so on.


Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data

Felipe Petroski Such
Aditya Rawal
Joel Lehman
Kenneth O. Stanley
Jeff Clune

The Generative Teaching Networks (GTN) paper breaks new ground by training generators that produce synthetic data that can enable learner neural networks to learn faster than when training on real data. The process is as follows: The generator produces synthetic training data by transforming some sampled noise vector and label; a newly-initialized learner is trained on this synthetic data and evaluated on real data; the error signal from this evaluation is backpropagated to the generator via meta-gradients, to enable it to produce synthetic samples that will train the learner networks better. They also demonstrate that their curriculum learning variant, where the input vectors and their order are learned along with generator parameters, is especially powerful at teaching learners with few samples and few steps of gradient descent.

They apply their system to neural architecture search, and show an empirical correlation between performance of a learner on synthetic data and its eventual performance when trained on real data. In this manner, they make the argument that data from a trained GTN can be used to cheaply assess the likelihood of a given network succeeding to learn on the real task, and hence GTN data can tremendously speed up architecture search.

I really like this paper; I think it shines a light in an interesting new direction, and I look forward to seeing future work that builds on this in theoretical, mechanistic, and applied manners. On the other hand, I felt they did gloss over how exactly they do curriculum learning, and their reinforcement learning experiment was a little unclear to me.

I think the implications of this work are enormous. In a future where we might be limited by the maturity of available simulation platforms or inundated by deluges of data with little marginal information, this approach can circumvent such problems for the selection and (pre)training of suitable student networks.


Unsupervised learning: the curious pupil

A high-level but well-written explanation of why many believe unsupervised learning will be key to achieving general intelligence, touching on the approaches of GANs and autoregressive models as examples.

This is a clean, clear summary, but one without any real technical depth or detail; this would be a good writeup to hand someone without any machine learning background who wanted to get an intuitive grasp for unsupervised learning as a field.


Evaluating the Unsupervised Learning of Disentangled Representations

This blog post and paper describe a Google-scale comparative study of different representation learning methods designed to learn "disentangled" representations, where the axes of the representation are aligned with the true underlying factors generating the data. The paper's claims are a sobering result for the field, both theoretically and empirically. Theoretically, they show that in an unsupervised context, it's not possible to find a disentangled representation without embedding some form of inductive bias into your model. Empirically, they present evidence suggesting that variation between random seeds for a given hyperparameter setting (in particular, regularization strength) matters as much or more than variation between that hyperparameter's values. Finally, they run experiments that call into question whether disentangled representations actually support transfer learning, or can be identified as in fact being disentangled without using a metric that relies on having ground truth factors of variation to begin with, making it difficult to evaluate on the many realistic contexts where these aren't available.

This strikes me as a really valuable injection of empirical realism, of the kind that tends to be good for research fields to have periodically, even if it can be a bit painful or frustrating. I appreciate in particular the effort and clarity that this paper puts into articulating the implicit assumptions of how disentanglement can be used or evaluated, and trying to test those assumptions under more real-world settings, such as the one where you don't have any ground truth factors of variation, since the real world doesn't tend to just hand out the Correct factorized model of itself.


Unsupervised Learning via Meta-Learning

This paper trains a meta-learner on tasks which were generated using unsupervised learning. This is done by first learning an (unsupervised) embedding for a dataset, then clustering in that embedding space using k-means. Clustering is done many times with random scaling on each dimension; each meta-learning task is then based on one set of clusters. The resulting meta-learner is then evaluated on the actual task for that dataset, performing better than approaches based just on embeddings, and sometimes getting fairly close to the supervised-learning equivalent.

This is a cool technique; I like the combination of two approaches (meta-learning and unsupervised learning) aimed at making deep learning applicable to many more real-world datasets. I can imagine promising follow-ups - e.g. randomly scaling embedding dimensions to get different clusters seems a bit hacky to me, so I wonder if there's a better approach (maybe learning many different embeddings?). It's interesting to note that their test-time performance is sometimes better than their training performance, presumably because some of the unsupervised training clusterings are "nonsensical", so there is room to improve here.


Understanding View Selection for Contrastive Learning

Yonglong Tian
Chen Sun
Ben Poole
Dilip Krishnan
Cordelia Schmid
Phillip Isola

<@Contrastive multiview learning@>(@Representation Learning with Contrastive Predictive Coding@) is a self-supervised approach to pretraining classifiers in which different views of data points are created and an encoder is trained to minimize the distance between encodings of views corresponding to data points with the same label while maximizing the distance between encodings of views with different labels.

The efficacy of this approach depends on the choice of views as well as the downstream task the neural network is going to be trained for. To find the most promising views, the authors propose the Infomin principle: all views should keep task-relevant information while the mutual information between views is minimized. The principle is supported by various observations: Firstly, earlier approaches to contrastive learning in the image domain that use data augmentation to preserve object identity while creating diverse views can be seen as an implicit application of the Infomin principle. Secondly, varying the mutual information between views (for example by changing the distance between two cropped views of the same image) creates an inverted U-curve for downstream performance corresponding to poor performance if there is too much or too little mutual information between the views. Lastly, the authors also find an inverted U-curve in performance for different colour spaces when using channels as views and the Lab colour space which was built to mimic human colour perception is close to the optimum, meaning that human colour perception might be near-optimal for self-supervised representation learning.

The authors then use the Infomin principle to select image augmentations for contrastive pretraining and improve the state of the art in linear readout on ImageNet from 69.3% to 73% for Top-1 accuracy and from 89% to 91.1% for Top-5 accuracy.

While the Infomin principle seems powerful and their results look impressive, I am not really convinced that the principle actually played an important role in finding the image augmentations they ended up using, as there is little description of how that happened and the augmentations rather look like the result of combining previously used approaches and doing some hyperparameter optimization.

What makes for good views for contrastive learning


The human side of interaction

The lens of [human-AI interaction](https://www.alignmentforum.org/posts/4783ufKpx8xvLMPc6/human-ai-interaction) ([AN #41](https://mailchi.mp/8c3f02cabccd/alignment-newsletter-41)) also suggests that we should focus on what the _human_ should do in AI alignment.

Any feedback that the AI system gets must be interpreted using some assumption. For example, when a human provides an AI system a reward function, it shouldn't be interpreted as a description of optimal behavior in every possible situation (which is what we currently do implicitly). [Inverse Reward Design](https://arxiv.org/abs/1711.02827) (IRD) suggests an alternative, more realistic assumption: the reward function is likely to the extent that it leads to high true utility in the training environment. Similarly, in inverse reinforcement learning (IRL) human demonstrations are often interpreted under the assumption of Boltzmann rationality.

Analogously, we may also want to train humans to give feedback to AI systems in the manner that they are expecting. With IRD, the reward designer should make sure to test the reward function extensively in the training environment. If we want our AI system to help us with long-term goals, we may want the overseers to be much more cautious and uncertain in their feedback (depending on how such feedback is interpreted). Techniques that learn to reason like humans, such as iterated amplification and debate, would by default learn to interpret feedback the way humans do. Nevertheless it will probably be useful to train humans to provide useful feedback: for example, in debate, we want humans to judge which side provided more true and useful information.


Reward uncertainty

Given that we need human feedback for the AI system to stay "on track" as the environment changes, we might design a system that keeps an estimate of the reward, chooses actions that optimize that reward, but also updates the reward over time based on feedback. This has a few issues: it typically assumes that the human Alice knows the true reward function, it makes a possibly-incorrect assumption about the meaning of Alice's feedback, and the AI system still looks like a long-term goal-directed agent where the goal is the current reward estimate.

This post takes the above AI system and considers what happens if you have a distribution over reward functions instead of a point estimate, and during action selection you take into account future updates to the distribution. (This is the setup of [Cooperative Inverse Reinforcement Learning](https://arxiv.org/abs/1606.03137).) While we still assume that Alice knows the true reward function, and we still require an assumption about the meaning of Alice's feedback, the resulting system looks less like a goal-directed agent.

In particular, the system no longer has an incentive to disable the system that learns values from feedback: while previously it changed the AI system's goal (a negative effect from the goal's perspective), now it provides more information about the goal (a positive effect). In addition, the system has more of an incentive to let itself be shut down. If a human is about to shut it down, it should update strongly that whatever it was doing was very bad, causing a drastic update on reward functions. It may still prevent us from shutting it down, but it will at least stop doing the bad thing. Eventually, after gathering enough information, it would converge on the true reward and do the right thing. Of course, this is assuming that the space of rewards is well-specified, which will probably not be true in practice.


Coherence arguments do not imply goal-directed behavior

In this post, Rohin argues against the claim that "simply knowing that an agent is intelligent lets us infer that it is goal-directed". He points out that all behaviour can be rationalized as expected utility maximisation over world-histories - but this may not meet our criteria for goal-directed behaviour, and slightly misspecifying such a utility function may well be perfectly safe. What's more interesting - and dangerous - is expected utility maximisation over world-states - but he claims that we shouldn't assume that advanced AI will have this sort of utility function, unless we have additional information (e.g. that it has a utility function simple enough to be explicitly represented). There are plenty of intelligent agents which aren't goal-directed - e.g. ones which are very good at inference but only take trivial actions.

I broadly agree with Rohin's points in this and the previous post, and am glad that he's making these arguments explicit. However, while goal-directedness is a tricky property to reason about, I think it's still useful to consider it a property of an agent rather than a property of our model of that agent. It's true that when we have a detailed explanation of how an agent works, we're able to think of cases in which its goal-directedness breaks down (e.g. adversarial examples). However, when these examples are very rare, they don't make much practical difference (e.g. knowing that AlphaGo has a blind spot in certain endgames might not be very helpful in beating it, because you can't get to those endgames).


Neurosymbolic Reinforcement Learning with Formally Verified Exploration

Greg Anderson
Abhinav Verma
Isil Dillig
Swarat Chaudhuri

A typical approach to formally verified safe exploration in RL is to compute a _shield_, which identifies a safe set of states and actions. After this shield is computed, it is “wrapped” around the environment to ensure that if a potentially unsafe action is about to be taken, it is replaced with a safe one. Then, a policy learning algorithm is applied as normal to learn a good policy.

The key insight of this paper is to compute shields for specific _policies_, rather than creating a one-time shield that must apply to the entire state space. Since any given policy will only visit a small fraction of the state space, the shields are easier to compute and can be more permissive.

They assume access to a _worst-case dynamics model_, which given a state and action outputs a _set_ of states that could be visited. Given a policy π, an _inductive safety invariant_ is a set of safe states that includes all possible initial states and is closed under worst-case transitions: if you start at a state in the set, for any action that π suggests and for any state from the worst-case transition dynamics, that new state will still be in the set. Our algorithm will ensure that any policy we execute will have a corresponding inductive safety invariant.

Formal verification techniques allow us to find inductive safety invariants for restricted classes of policies. This paper uses the space of deterministic, piecewise linear policies as its set of symbolic policies. But how do we apply this to neural nets? The key idea is to start with a safe symbolic policy, convert it to a neurosymbolic policy, take a neural net gradient step, convert back to a safe symbolic policy, and repeat until done. Let’s go over each of these steps.

First, let’s suppose we have a symbolic policy g with inductive safety invariant ø. Then for any neural net f, we construct the policy h = “f(s) if no matter what we stay within ø, otherwise g(s)”. It is easy to see that ø is also an inductive safety invariant for h. Which f should we use to create h? The authors train a neural net to imitate g, and use that as their f. (Note that imitating g only requires executing g in the environment, and we know that g is safe.)

Now that we have our neurosymbolic policy h, we need to take gradient steps on it. We collect data in the environment using h, but then for the gradient we ignore the symbolic part, and take a gradient step as though the data were collected using f. (It seems they used an on-policy algorithm for this, introducing bias; I am not sure why they didn’t simply use an off-policy algorithm.) This produces a new neurosymbolic policy h’ that is still safe (since g and ø are unchanged, and that’s what guarantees safety).

Finally, we need to convert h’ back into a symbolic policy g’. This is done by a version of imitation learning that works in the symbolic policy space, where a new inductive safety invariant for g’ is found using formal verification techniques.

To start off the whole process, we need an initial symbolic policy, which must be constructed by hand. The authors show using experiments in simple continuous control environments that this method can learn high-reward policies without ever having a safety violation.

I really like this as an example of combining the performance of neural networks with the robustness of symbolic approaches. I especially like the fact that the shield is specialized to the current policy and updated over time: I think ML scales so well partly because it only deals with a tiny portion of the input space and can completely ignore the vast majority of possible inputs, and so if you want to add anything on top of ML you need to ensure you preserve this property to ensure scalability. Previous approaches required a shield that is correct across all possible states, failing to preserve this property; in contrast, this approach only requires a shield that is correct for the sequence of learned policies (on whichever states they visit).

I should note that a large portion of why I like this paper is that it feels like it elegantly fits in _both_ the formal verification _and_ the ML fields. (I used to work in programming languages, of which formal verification is a subfield.) On the formal verification side, the guarantees are clean and simple, and the techniques used are canonical. On the ML side, I mentioned above why I like the fact that the shield is policy-specific and updated over time.

As I’ve said before, I think the real challenge in formal verification for AI alignment is how to handle fuzzy specifications. I think this paper shows a path forward: since the safety is established by an inductive invariant that can change over time, we could potentially use human feedback to establish these inductive invariants and update them over time, without requiring a human to fully specify at the outset exactly what is safe and what isn’t. You could think of it as an expanding whitelist of states which the policy is allowed to visit.


An Inductive Synthesis Framework for Verifiable Reinforcement Learning

He Zhu
Zikang Xiong
Stephen Magill
Suresh Jagannathan

This older paper has a pretty similar idea to the one in the highlighted paper. In order to compute a safety shield for a neural network RL agent, we first transform the neural network into a simpler more symbolic policy, prove safety of the symbolic policy, and then use the generated inductive safety invariant as a shield. This paper also uses deterministic piecewise linear policies as its space of symbolic policies. It only proves safety of the final learned RL policy, and so only guarantees safety at deployment, not at training time. (In other words, it does not guarantee safe exploration, and instead assumes that you are training in simulation so that safety is not a concern.)

Since this paper was published at PLDI, it is both longer and goes into a lot more of the details of how to actually perform each of these steps, as well as showing it with a running example on the inverted pendulum (where safety is defined as not going beyond a certain angle). I’m not going to summarize them here but anyone interested in these technical details should check out this paper before the highlighted one (which is constrained by ML page limits and can’t explain the techniques very well).

Just as a reminder that learning programs does not automatically confer interpretability, I present to you the symbolic policy learned by their method for the inverted pendulum:

&&IMAGE HERE&&


Ethical Mission Definition and Execution for Maritime Robots Under Human Supervision

Don Brutzman
Curtis L. Blais
Duane T. Davis
Robert B. McGhee

While underwater robots can perform missions that humans cannot, they cannot be held liable for their actions. Our society requires that someone be responsible for (and can be held liable for) the actions of any such robot, leading to a form of the specification problem: how do we program robots such that it is reasonable to hold their operators accountable for their actions?

This paper divides mission execution into three main parts: the execution level (hardware control), the tactical level (low-level behaviors), and the strategic level (what the robot should do). It proposes that, at the strategic level, we use formal methods to specify what the robot should do. The language should be expressive enough to be useful, while still keeping it sufficiently limited to allow exhaustive testing. They propose using state machines augmented with constraints. The constraints can be used to specify things like "the robot must stay at least 10m away from obstacles". The state machine decides which behaviors to execute, and each such behavior can have three results: success, failure, or exception (in the case that a constraint would have been violated had the behavior continued operating).

It's interesting to see other groups also aiming to have what are essentially robustness guarantees, but motivated instead from the perspective of responsibility and liability. The actual method seems reasonable for the impoverished systems we have today, where we must specify everything that we want the system to do.


Verification and Transparency

This post points out that verification and transparency have similar goals. Transparency produces an artefact that allows the user to answer questions about the system under investigation (e.g. "why did the neural net predict that this was a tennis ball?"). Verification on the other hand allows the user to pose a question, and then automatically answers that question (e.g. "is there an adversarial example for this image?").


Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability

Kai Y. Xiao
Vincent Tjeng
Nur Muhammad Shafiullah
Aleksander Madry

The idea behind verification is to consider all possible inputs at the same time, and show that no matter what the input is, a particular property is satisfied. In ML, this is typically applied to adversarial examples, where inputs are constrained to be within the L-infinity norm ball of dataset examples. Prior papers on verification (covered in [AN #19](https://mailchi.mp/4b19d2caa5a9/alignment-newsletter-19)) solve a computationally easier relaxation of the verification problem, that gives a lower bound on the performance of the classifier. This paper aims to use exact verification, since it can compute the exact adversarial performance of the classifier on the test set, and to figure out how to improve its performance.

One easy place to start is to encourage weights to be zero, since these can be pruned from the problem fed in to the constraint solver. (Or more likely, they feed it in anyway, but the constraint solver immediately gets rid of them -- constraint solvers are pretty smart.) This can be done using L1 regularization and pruning small weights. This already gives two orders of magnitude of speedup, making it able to verify that there is no adversarial attack with ϵ = 0.1 on a particular MNIST digit in 11 seconds on average.

Next, they note that verification with linear constraints and functions is easy -- the challenging aspect is the Relu units that force the verifier to branch into two cases. (Since relu(x) = max(x, 0), it is the identity function when x is positive, and the zero function otherwise.) So why not try to ensure that the Relu units are also linear? Obviously we can't just make all the Relu units linear -- the whole point of them is to introduce nonlinearity to make the neural net more expressive. But as a start, we can look at the behavior of the Relu units on the examples we have, and if they are almost always active (inputs are positive) or almost always inactive (inputs are negative), then we replace them with the corresponding linear function (identity and zero, respectively), which is easier to verify. This gets another ~2x speedup.

But what if we could also change the training procedure? Maybe we could augment the loss so that the Relu units are either decisively active or decisively inactive on any dataset example. They propose that _during training_ we consider the L-infinity norm ball around each example, use that to create intervals that each pixel must be in, and then make a forward pass through the neural net using interval arithmetic (which is fast but inexact). Then, we add a term to the loss that incentivizes the interval for the input to each Relu to exclude zero (so that the Relu is either always active or always inactive). They call this the Relu Stability loss, or RS loss.

This leads to a further 4-13x speedup with similar test set accuracy. They then also test on MNIST with ϵ = 0.2, 0.3 and CIFAR with ϵ = 2/255, 8/255. It leads to speedup in all cases, with similar test set accuracy on MNIST but reduced accuracy on CIFAR. The provable accuracy goes up, but this is probably because when there's no RS loss, more images time out in verification, not because the network becomes better at classification. Other verification methods do get better provable accuracies on CIFAR, even though in principle they could fail to detect that a safe example is safe. This could be because their method times out frequently, or because their method degrades the neural net classifier -- it's hard to tell since they don't report number of timeouts.

As with the previous papers on verification, I'm excited in the improvement in our capability to prove things about neural nets. I do think that the more important problem is how to even state properties that we care about in a way that we could begin to prove them. For example, [last week](https://mailchi.mp/d7b5059d64ed/alignment-newsletter-24) we saw the unrestricted adversarial examples challenge, where humans are the judge of what a legal example is -- how can we formalize that for a verification approach?

On this paper specifically, I wish they had included the number of timeouts that their method has -- it's hard to interpret the provable accuracy numbers without that. Based on the numbers in the paper, I'm guessing this method is still much more computationally expensive than other methods. If so, I'm not sure what benefit it gives over them -- presumably it's that we can compute the exact adversarial accuracy, but if we don't have enough compute, such that other methods can prove better lower bounds anyway, then it doesn't seem worth it.


Towards Mixed Optimization for Reinforcement Learning with Program Synthesis

Surya Bhupatiraju
Kumar Krishna Agrawal
Rishabh Singh

This paper proposes a framework in which policies are represented in two different ways -- as neural nets (the usual way) and as programs. To go from neural nets to programs, you use _program synthesis_ (as done by [VIPER](http://arxiv.org/abs/1805.08328) and [PIRL](https://arxiv.org/abs/1804.02477), both summarized in previous newsletters). To go from programs to neural nets, you use _distillation_ (basically use the program to train the neural net with supervised training). Given these transformations, you can then work with the policy in either space. For example, you could optimize the policy in both spaces, using standard gradient descent in neural-net-space, and _program repair_ in program-space. Having a program representation can be helpful in other ways too, as it makes the policy more interpretable, and more amenable to formal verification of safety properties.

It is pretty nice to have a program representation. This paper doesn't delve into specifics (besides a motivating example worked out by hand), but I'm excited to see an actual instantiation of this framework in the future!


Certified Adversarial Robustness for Deep Reinforcement Learning

Michael Everett*
Bjorn Lutjens*
Jonathan P. How

<@Certified adversarial robustness@>(@Certified Defenses against Adversarial Examples@) provides guarantees about the effects of small perturbations on a neural network’s outputs. This paper uses that approach to make reinforcement learning more robust by training a DQN and acting by choosing the action with the best worst-case Q-value under adversarial perturbations (called the robust-optimal action) estimated from the certificate bounds, instead of the action with the highest Q-value.

The approach is evaluated on Cartpole and a navigation task that requires avoiding collisions, with an adversary perturbing observations in both cases. For small perturbations, this technique actually increases performance, but as perturbations get large the agent’s conservatism can lead to a large degradation in performance.

While the approach is straightforward and will certainly increase robustness in many cases, it seems worth mentioning two serious issues. First, they assume that the initial DQN training learns the perfect Q function. Second, the provided certificates are about individual actions, not policy performance: the Q-values approximated in DQN assume optimal performance starting from the next action, which is not a given here. I am a bit concerned that these limitations were not really discussed, while the paper claims that “the resulting policy comes with a certificate of solution quality”.


Verifiable Reinforcement Learning via Policy Extraction

Osbert Bastani
Yewen Pu
Armando Solar-Lezama

Since it is hard to verify properties of neural nets, we can instead first train a decision tree policy to mimic the policy learned by deep RL, and then verify properties about that. The authors generalize [DAGGER](https://www.cs.cmu.edu/~sross1/publications/Ross-AIStats11-NoRegret.pdf) to take advantage of the Q-function and extract decision tree policies. They then prove a correctness guarantee for a toy version of Pong (where the dynamics are known), a robustness guarantee for Pong (with symbolic states, not pixels) (which can be done without known dynamics), and stability of cartpole.

Many people believe that ultimately we will need to prove theorems about the safety of our AIs. I don't understand yet what kind of theorems they have in mind, so I don't really want to speculate on how this relates to it. It does seem like the robustness guarantee is the most relevant one, since in general we won't have access to a perfect model of the dynamics.