Birds, Brains, Planes, and AI: Against Appeals to the Complexity/Mysteriousness/Efficiency of the Brain

This post argues against a particular class of arguments about AI timelines. These arguments have the form: “The brain has property X, but we don’t know how to make AIs with property X. Since it took evolution a long time to make brains with property X, we should expect it will take us a long time as well”. The reason these are not compelling is because humans often use different approaches to solve problems than evolution did, and so humans might solve the overall problem without ever needing to have property X. To make these arguments more convincing, you need to argue 1) why property X really is _necessary_ and 2) why property X won’t follow quickly once everything else is in place.

This is illustrated with a hypothetical example of someone trying to predict when humans would achieve heavier-than-air flight: in practice, you could have made decent predictions just by looking at the power to weight ratios of engines vs. birds. Someone who argued that we were far away because “we don't even know how birds stay up for so long without flapping their wings” would have made incorrect predictions.

This all seems generally right to me, and is part of the reason I like the <@biological anchors approach@>(@Draft report on AI timelines@) to forecasting transformative AI.


Automating reasoning about the future at Ought

Roughly speaking, we can think of an axis of reasoning, spanning from high-confidence statistical reasoning with lots of data to general quantitative reasoning to very qualitative reasoning. Ought is now building Elicit to help with _judgmental forecasting_, which is near but not at the end of the spectrum. In judgmental forecasting, we might take a complicated question such as “Will Roe v. Wade be overturned if Trump nominates a new justice”, decompose it into subquestions, estimate probabilities for those subquestions, and then combine them to get to a final forecast. Crucially, this requires relying on people’s judgment: we cannot just look at the historical rate at which landmark Supreme Court decisions are overturned, since the situation has rarely arisen before.

Currently, Elicit has several features that help with the quantitative aspects of judgmental forecasting, for example by enabling users to input complex distributions and visualizing these distributions. However, in the long-term, the hope is to also help with the more qualitative aspects of judgmental forecasting as well, for example by proposing an important subquestion for answering the current question under consideration, or recommending a source of data that can answer the question at hand.

Ought is now working on adding these sorts of features by using large language models (GPT-3 in particular). They are currently [looking for beta testers](https://www.lesswrong.com/posts/WBgT4jAqTrPN7qh3Z/beta-test-gpt-3-based-research-assistant) for these features!

This seems like a pretty interesting direction, though I’m not totally clear on how it relates to AI alignment (assuming it is supposed to). It does seem quite related to iterated amplification, which also relies on this sort of decomposition of questions.

I found the video mock ups and demos in the blog post to be particularly interesting, both in the original post and in the [call for beta testers](https://www.lesswrong.com/posts/WBgT4jAqTrPN7qh3Z/beta-test-gpt-3-based-research-assistant); I think they were much better at showcasing the potential value than the abstract description, and I recommend you all watch them.


AGI Predictions

A collection of interesting questions relevant to AI safety, as well as aggregated predictions from readers of the post.


Roadmap to a Roadmap: How Could We Tell When AGI is a ‘Manhattan Project’ Away?

The key hypothesis of this paper is that once there is a clear “roadmap” or “runway” to AGI, it is likely that state actors could invest a large number of resources into achieving it, comparably to the Manhattan project. The fact that we do not see signs of such investment now does not imply that it won’t happen in the future: currently, there is so little “surface area” on the problem of AGI that throwing vast amounts of money at the problem is unlikely to help much.

If this were true, then once such a runway is visible, incentives could change quite sharply: in particular, the current norms of openness may quickly change to norms of secrecy, as nations compete (or perceive themselves to be competing) with other nations to build AGI first. As a result, it would be good to have a good measure of whether we have reached the point where such a runway exists.


My AI Timelines Have Sped Up

Alex Irpan updates his predictions of AGI sooner to:

10% chance by 2035 (previously 2045)
50% chance by 2045 (previously 2050)
90% chance by 2070

The main reasons why are:

- Alex is now more uncertain because research pace over the past five years have been more surprising than expected, faster in some domains, but slower than others.
- Accounting for improvements in tooling. New libraries like TensorFlow and PyTorch have accelerated progress. Even CNNs can be used as a “tool” that provides features for downstream tasks like robotic control.
- He previously thought that labeled data might be a bottleneck, based on scaling laws showing that data needs might increase faster than compute; however, semi- and unsupervised learning have improved significantly, GPT-3 being the latest example of this.
- Alex now believes that compute will play a larger role and that compute can scale faster than algorithms because there is large worldwide consumer demand.

The post ends with a hypothetical description of how AGI may happen soon that I will leave out of the summary but recommend reading.

My personal opinion on timelines is that I think it is much more informative to draw out the full CDF/PDF of when we will get to AGI instead of percentages by different years. It isn’t included in the post, but you can find Alex’s [here](https://elicit.ought.org/builder/BsNkKzJoc). I end up placing higher likelihood on AGI happening sooner than Alex does, but I largely agree with his reasoning.

More uncertainty than the original prediction seems warranted to me; the original prediction had a very high likelihood of AGI between 2045-2050 that I didn’t understand. Of the rest of the arguments, I agree most strongly with the section on tooling providing a speedup. I’d even push the point farther to say that there are many inputs into current ML systems, and all of them seem to be improving at a rapid clip. Hardware, software tools, data, and the number of ML researchers all seem to be on track to improve significantly over the next decade.


AI Forecasting Dictionary

One big challenge with forecasting the future is operationalizing key terms unambiguously, so that a question can be resolved when the future actually arrives. Since we'll probably need to forecast many different questions, it's crucial that we make it as easy as possible to create and answer well-operationalized questions. To that end, the authors have created and open-sourced an AI Forecasting Dictionary, which gives precise meanings for important terms, along with examples and non-examples to clarify further.


AI Forecasting Resolution Council

Even if you operationalize forecasting questions well, often the outcome is determined primarily by factors other than the one you are interested in. For example, progress on a benchmark might be determined more by the number of researchers who try to beat the benchmark than by improvements in AI capabilities, even though you were trying to measure the latter. To deal with this problem, an AI Forecasting Resolution Council has been set up: now, forecasters can predict what the resolution council will say at some particular time in the future. This allows for questions that get at what we want: in the previous case, we could now forecast how the resolution council will answer the question "would current methods be able to beat this benchmark" in 2021.


How to write good AI forecasting questions + Question Database

As discussed above, operationalization of forecasting questions is hard. This post collects some of the common failure modes, and introduces a database of 76 questions about AI progress that have detailed resolution criteria that will hopefully avoid any pitfalls of operationalization.


Why Is the Human Brain So Efficient?

Overall point for this audience is that, despite how slow and imprecise neuron signals are, the human brain beats computers because of how massively parallel it is.


Shaping Safer Goals

Much of safety research focuses on a single agent that is directly incentivized by a loss/reward function to take particular actions. This sequence instead considers safety in the case of multi-agent systems interacting in complex environments. In this situation, even simple reward functions can yield complex and highly intelligent behaviors that are only indirectly related. For example, evolution led to humans who can learn to play chess, despite the fact that the ancestral environment did not contain chess games. In these situations, the problem is not how to construct an aligned reward function, the problem is how to shape the experience that the agent gets at training time such that the final agent policy optimizes for the goals that we want. This sequence lays out some considerations and research directions for safety in such situations.

One approach is to teach agents the generalizable skill of obedience. To accomplish this, one could design the environment to incentivize specialization. For instance, if an agent A is more powerful than agent B, but can see less of the environment than B, A might be incentivized to obey B’s instructions if they share a goal. Similarly we can increase the ease and value of coordination through enabling access to a shared permanent record or designing tasks that require large-scale coordination.

A second approach is to move agents to simpler and safer training regimes as they develop more intelligence. The key assumption here is that we may require complex regimes such as competitive multi-agent environments to jumpstart intelligent behavior, but may be able to continue training in a simpler regime such as single-task RL later. This is similar to current approaches for training a language model via supervised learning and then finetuning with RL, but going in the opposite direction to increase safety rather than capabilities.

A third approach is specific to a collective AGI: an AGI that is composed of a number of separate general agents trained on different objectives that learn to cooperatively solve harder tasks. This is similar to how human civilization is able to accomplish much more than any individual human. In this regime, the AGI can be effectively sandboxed by either reducing the population size or by limiting communication channels between the agents. One advantage of this approach to sandboxing is that it allows us to change the effective intelligence of the system at test-time, without going through a potentially expensive retraining phase.

I agree that we should put more emphasis on the safety of multi-agent systems. We already have <@evidence@>(@Emergent Tool Use from Multi-Agent Interaction@) that complex behavior can arise from simple objectives in current systems, and this seems only more likely as systems become more powerful. Two-agent paradigms such as GANs, self-play, and debate, are already quite common in ML. Lastly, humans evolved complex behavior from the simple process of evolution so we have at least one example of this working. I also think this is an interesting area where there is lots to learn from other fields, such as game theory and evolutionary biology,

For any empirically-minded readers of this newsletter, I think this sequence opens up a lot of potential for research. The development of safety benchmarks for multi-agent systems and then the evaluation of these approaches seems like it would make many of the considerations discussed here more concrete. I personally would find them much more convincing with empirical evidence to back up that they work with current ML.

**Rohin's opinion:** The AGI model here in which powerful AI systems arise through multiagent interaction is an important and plausible one, and I'm excited to see some initial thoughts about it. I don't particularly expect any of these ideas to be substantially useful, but I'm also not confident that they won't be useful, and given the huge amount of uncertainty about how multiagent interaction shapes agents, that may be the best we can hope for currently. I'd be excited to see empirical results testing some of these ideas out, as well as more conceptual posts suggesting more ideas to try.


TanksWorld: A Multi-Agent Environment for AI Safety Research

Corban G. Rivera
Olivia Lyons
Arielle Summitt
Ayman Fatima
Ji Pak
William Shao
Robert Chalmers
Aryeh Englander
Edward W. Staley
I-Jeng Wang
Ashley J. Llorens

This paper presents TanksWorld, a simulation environment that attempts to illustrate three important aspects of real-world AI safety challenges: competing performance objectives, human-machine learning, and multi-agent competition. TanksWorld consists of two teams of N vs. N tanks. Tanks move and shoot while navigating in a closed arena with obstacles. Tanks are rewarded for killing opponent tanks and penalized for killing neutral and allied tanks according to a specified reward function. Each tank is controlled by either its own AI or a special policy meant to mimic a 'human' teammate. Each individual tank can only see a small portion of its environment, and must communicate with other teammates to gain more information. The following parameters can be varied to emphasize different research challenges:
- The communication range between tanks -- meant to represent environmental uncertainty.
- The number of neutral tanks and obstacles -- meant to represent the extent to which tanks must care about 'safety', i.e. avoid collateral damage.
- The control policies of teammates -- meant to represent the variability of human-machine teams.

I am generally excited about more work on demonstrating safety challenges; I think it helps to seed and grow the field in concrete directions. I am particularly excited about the possibility for TanksWorld to demonstrate multi-agent safety problems with agents in direct competition. I feel unsure about whether TanksWorld will be a good demonstration of general problems with human-machine interaction-- intuitively, that seems to me like it would be very difficult to capture and require more complex real-world modeling.


Collaborating with Humans Requires Understanding Them

Micah Carroll
Rohin Shah
Mark K. Ho
Thomas L. Griffiths
Sanjit A. Seshia
Pieter Abbeel
Anca Dragan

_Note: I am second author on this paper._ Self-play agents (like those used to play <@Dota@>(@OpenAI Five@) and <@Starcraft@>(@AlphaStar: Mastering the Real-Time Strategy Game StarCraft II@)) are very good at coordinating with _themselves_, but not with other agents. They "expect" their partners to be similar to them; they are unable to predict what human partners would do.  In competitive games, this is fine: if the human deviates from optimal play, even if you don't predict it you will still beat them. (Another way of saying this: the minimax theorem guarantees a minimum reward _regardless_ of the opponent.) However, in cooperative settings, things are not so nice: a failure to anticipate your partner's plan can lead to arbitrarily bad outcomes. We demonstrate this with a simple environment that requires strong coordination based on the popular game Overcooked. We show that agents specifically trained to play alongside humans perform much better than self-play or population-based training when paired with humans, both in simulation and with a real user study.

I wrote a short [blog post](https://www.alignmentforum.org/posts/dBMC63hjkc5wPqTC7/human-ai-collaboration) talking about the implications of the work. Briefly, there are three potential impacts. First, it seems generically useful to understand how to coordinate with an unknown agent. Second, it is specifically useful for scaling up <@assistance games@>(@Cooperative Inverse Reinforcement Learning@), which are intractable to solve optimally. Finally, it can lead to more ML researchers focusing on solving problems with real humans, which may lead to us finding and solving other problems that will need to be solved in order to build aligned AI systems.

Paper: On the Utility of Learning about Humans for Human-AI Coordination


Social choice ethics in artificial intelligence

If we want to program ethics into an AI system, should we do so by aggregating the ethical views of existing humans? This is often justified on procedural grounds: “everyone gets to affect the outcome”, or by abstention: “AI designers don’t have to think about ethics; the AI will deal with that”. (There is also a wisdom of the crowds justification, though this presupposes that there is some notion of “better” ethics independent of humans; which is out of scope for the paper.)

However, actually implementing an aggregative procedure requires three major design decisions: 1) _standing_, that is, whose views should be aggregated, 2) _measurement_, that is, how we determine what their ethical views are, and 3) _aggregation_, that is, how the views are put together into a whole. All of these are challenging.

For standing, we have to determine whom to include. Should we include children, psychopaths, non-human animals, ecosystems, future generations, and other AI systems? We must determine this ahead of time, since once we have decided on a social choice system, that system will then determine whose preferences are counted -- we can’t just modify it later.

For measurement, we have to back out human values somehow, which is quite a challenge given that humans have all sorts of cognitive biases and give different answers depending on the context. (See also <@ambitious value learning@>(@What is ambitious value learning?@) and subsequent posts in the sequence.)

For aggregation, the problems are well known and studied in the field of social choice theory. Some famous impossibility results include [Arrow’s theorem](https://en.wikipedia.org/wiki/Arrow%27s_impossibility_theorem) and the [Gibbard-Satterthwaite theorem](https://en.wikipedia.org/wiki/Gibbard%E2%80%93Satterthwaite_theorem).

I see this paper as a well-organized literature review of the many reasons why you _don’t_ want to handle AI alignment by finding the “true human utility function” or the “aggregated preferences of humanity” and then encoding them into the AI: there’s a myriad of challenges in even finding such an object. (A separate objection, out of scope for this paper, is that even if we did have such an object, we don’t know how to encode that goal into an AI system.)

You might then reasonably ask what we should be doing instead. I see the goal of AI _alignment_ as figuring out how, given a fuzzy but relatively well-specified task, to build an AI system that is reliably pursuing that task, in the way that we intended it to, but at a capability level beyond that of humans. This does not give you the ability to leave the future in the AI’s hands, but it would defuse the central (to me) argument for AI risk: that an AI system might be adversarially optimizing against you. (Though to be clear, there are still <@other risks@>(@The Main Sources of AI Risk?@) to consider.)


AXRP 3: Negotiable Reinforcement Learning

This podcast centers on <@negotiable RL@>(@Negotiable Reinforcement Learning for Pareto Optimal Sequential Decision-Making@), which studies how to aggregate preferences of multiple principals (humans) who have _different beliefs_. In the case where the principals have the same beliefs, Harsanyi’s utilitarianism theorem tells us that any reasonable method of aggregating preferences will end up optimizing some linear combination of the principals’ utility functions. In the case of differing beliefs, this paper proves that every Pareto optimal policy must be optimizing some linear combination of the principals’ utility functions, except that over time the weights are modified based on how well the principals’ beliefs model reality. Intuitively, the principals are both agreeing to the contract “the AI will optimize more for the person whose beliefs are more correct”; since each principal believes their own beliefs, they are both happy with this contract.

Most of the podcast is about the motivation and reason for writing this paper. Critch envisions a world in which people and AI systems must cooperate rather than fight, and this paper can be thought of as a study in how people can maximize cooperation. Unfortunately, it turns out that the cooperation-maximizing approach ends up being _unfair_: people whose beliefs are incorrect end up getting penalized (in terms of actual outcomes, rather than what they believe will happen).

More broadly, Critch hopes that this will spur more research into how parties with different beliefs can share control of AI systems: this seems important for AI to go well in the future.

I really liked this podcast: I definitely hadn’t understood Critch’s full reasons for doing this work. I didn’t include all the points in the summary, so I recommend you listen to it in addition to this summary.


What counts as defection?

We often talk about cooperating and defecting in general-sum games. This post proposes that we say that a player P has defected against a coalition C (that includes P) currently playing a strategy S when P deviates from the strategy S in a way that increases his or her own personal utility, but decreases the (weighted) average utility of the coalition. It shows that this definition has several nice intuitive properties: it implies that defection cannot exist in common-payoff games, uniformly weighted constant-sum games, or arbitrary games with a Nash equilibrium strategy. A Pareto improvement can also never be defection. It then goes on to show the opportunity for defection can exist in the Prisoner’s dilemma, Stag hunt, and Chicken (whether it exists depends on the specific payoff matrices).


Negotiable Reinforcement Learning for Pareto Optimal Sequential Decision-Making

Nishant Desai
Andrew Critch
and Stuart Russell

As AI systems are deployed in the real world, they will have to navigate value differences between different users. Regardless of our beliefs on how they should trade off between users, we at least want our AI systems to be Pareto optimal -- if there is some way to help one user without harming any other user, it should do it. A classic result from social choice theory, Harsanyi's theorem, states that a Pareto optimal agent must be optimizing a weighted linear combination of the users' utility functions. (The weights determine how to trade off between users.) However, this assumes that the users all have the same _beliefs_. When users have different beliefs, we can get even better outcomes if the users bet on their beliefs. If I'm sure that our robot will make a chocolate chip cookie, while you're sure its going to be a brownie, rather than splitting whatever the robot brings, we can agree that if it's a cookie I get it, whereas if it's a brownie you get it, and this makes both of us better off by our own beliefs. This paper shows that a Pareto optimal agent helping users with different beliefs must be optimizing a weighted linear combination of the user's utility functions, where the weights are proportional to how accurate the user's beliefs turn out to be. It's as if the users bet with each other about their beliefs, with the stakes being the assistance of the agent.

In practice, I expect users are unlikely to want this, for similar reasons to why we don't bet on every belief difference we have with other people, even though it's net positive in expectation. But we do need to handle this somehow. An interesting scenario to consider is when a regular person has different beliefs from an expert. Does the regular person have to defer exactly to the judgments of the expert to avoid being exploited? What if there's a regular person and _two_ experts who disagree with each other?


Reinforcement Learning under Threats

Víctor Gallego
Roi Naveiro
David Ríos Insua

Due to lack of time, I only skimmed this paper for 5 minutes, but my general sense is that it takes MDPs and turns them into two player games by positing the presence of an adversary. It modifies the Bellman update equations to handle the adversary, but runs into the usual problems of simulating an adversary that simulates you. So, it formalizes level-k thinking (simulating an opponent that thinks about you at level k-1), and evaluates this on matrix games and the friend-or-foe environment from [AI safety gridworlds](https://deepmind.com/blog/specifying-ai-safety-problems/).

I'm not sure what this is adding over two-player game theory (for which we can compute equilibria) but again I only skimmed the paper so it's quite likely that I missed something.


Multi-Agent Generative Adversarial Imitation Learning

Jiaming Song
Hongyu Ren
Dorsa Sadigh
Stefano Ermon

This paper generalizes [GAIL](http://www.jonathanho.me/files/HoErmon_NIPS2016.pdf) (which was covered [last week](https://mailchi.mp/ad852629e45a/alignment-newsletter-17)) to the multiagent setting, where we want to imitate a group of interacting agents. They want to find a Nash equilibrium in particular. They formalize the Nash equilibrium constraints and use this to motivate a particular optimization problem for multiagent IRL, that looks very similar to their optimization problem for regular IRL in GAIL. After that, it is quite similar to GAIL -- they use a regularizer ψ for the reward functions, show that the composition of multiagent RL and multiagent IRL can be solved as a single optimization problem involving the convex conjugate of ψ, and propose a particular instantiation of ψ that is data-dependent, giving an algorithm. They do have to assume in the theory that the multiagent RL problem has a unique solution, which is not typically true, but may not be too important. As before, to make the algorithm practical, they structure it like a GAN, with discriminators acting like reward functions. What if we have prior information that the game is cooperative or competitive? In this case, they propose changing the regularizer ψ, making it keep all the reward functions the same (if cooperative), making them negations of each other (in two-player zero-sum games), or leaving it as is. They evaluate in a variety of simple multiagent games, as well as a plank environment in which the environment changes between training and test time, thus requiring the agent to learn a robust policy, and find that the correct variant of MAGAIL (cooperative/competitive/neither) outperforms both behavioral cloning and single-agent GAIL (which they run N times to infer a separate reward for each agent).

Multiagent settings seem very important (since there does happen to be more than one human in the world). This looks like a useful generalization from the single agent case to the multiagent case, though it's not clear to me that this deals with the major challenges that come from multiagent scenarios. One major challenge is that there is no longer a single optimal equilibrium when there are multiple agents, but they simply assume in their theoretical analysis that there is only one solution. Another one is that it seems more important that the policies take history into account somehow, but they don't do this. (If you don't take history into account, then you can't learn strategies like tit-for-tat in the iterated prisoner's dilemma.) But to be clear I think this is the standard setup for multiagent RL -- it seems like field is not trying to deal with this issue yet (even though they could using eg. a recurrent policy, I think?)


Modeling Friends and Foes

Multiagent scenarios are typically modeled using game theory. However, it is hard to capture the intuitive notions of "adversarial", "neutral" and "friendly" agents using standard game theory terminology. The authors propose that we model the agent and environment as having some prior mixed strategy, and then allow them to "react" by changing the strategies to get a posterior strategy, but with a term in the objective function for the change (as measured by the KL divergence). The sign of the environment's KL divergence term determines whether it is friendly or adversarial, and the magnitude determines the magnitude of friendliness or adversarialness. They show that there are always equilibria, and give an algorithm to compute them. They then show some experiments demonstrating that the notions of "friendly" and "adversarial" they develop actually do lead to behavior that we would intuitively call friendly or adversarial.

Some notes to understand the paper: while normally we think of multiagent games as consisting of a set of agents, in this paper there is an agent that acts, and an environment in which it acts (which can contain other agents). The objective function is neither minimized nor maximized -- the sign of the environment's KL divergence changes whether the stationary points are maxima or minima (which is why it can model both friendly and adversarial environments). There is only one utility function, the agent's utility function -- the environment is only modeled as responding to the agent, rather than having its own utility function.

This is an interesting formalization of friendly and adversarial behavior. It feels somewhat weird to model the environment as having a prior strategy that it can then update. This has the implication that a "somewhat friendly" environment is unable to change its strategy to help the agent, even though it would "want" to, whereas when I think of a "somewhat friendly" environment, I think of a group of agents that share some of your goals but not all of them, so a limited amount of cooperation is possible. These feel quite different.


Multi-agent Social Reinforcement Learning Improves Generalization

Kamal Ndousse
Douglas Eck
Sergey Levine
Natasha Jaques

We’ve previously seen that in sparse reward settings where exploration is hard, it’s very useful to have expert demonstrations to avoid having to do all the exploration yourself (<@1@>(@Learning Montezuma’s Revenge from a Single Demonstration@), <@2@>(@Making Efficient Use of Demonstrations to Solve Hard Exploration Problems@), <@3@>(@Playing hard exploration games by watching YouTube@)). However, this assumes that the demonstrator is “external” to the environment, whereas really we’d like to model them as part of the environment, as in <@assistance games@>(@Cooperative Inverse Reinforcement Learning@). This then looks like _social learning_, in which agents learn how to perform tasks by looking at cues from other agents within the environment.

But how can we do this in high-dimensional environments? This paper looks at one approach: adding an auxiliary loss in which the agent must predict the next state of the environment. Since the environment itself contains experts that do useful things, the agent implicitly must learn what those experts are doing and what effects their actions have.

They find that such agents learn to follow the cues of the experts and thus achieve significantly improved reward relative to agents that are trained in isolation. In fact, these agents can be transferred to novel environments, where they continue to follow expert cues to achieve high reward. However, this means that they don’t learn how to act when experts aren’t present, and so fail in the solo setting. This can be fixed by training on a mixture of solo settings and settings with experts present.

I’m a big fan of moving towards modeling humans as part of the environment, since we will eventually have AI systems working with and interacting with humans -- they won’t be “external to the AI’s universe” as it is often modeled currently.


Theory of Minds: Understanding Behavior in Groups Through Inverse Planning

Michael Shum
Max Kleiman-Weiner
Michael L. Littman
Joshua B. Tenenbaum

This paper introduces Composable Team Hierarchies (CTH), a representation designed for reasoning about how agents reason about each other in collaborative and competitive environments. CTH uses two "planning operators": the Best Response operator returns the best policy in a single-agent game, and the Joint Planning operator returns the best team policy when all agents are cooperating. Competitive policies can then be derived via recursive application of those operations to subsets of agents (while holding the policies of other agents fixed). CTH draws from ideas in level-K planning (in which each agent assumes all other agents are at level K-1) and cooperative planning, but is more powerful than either approach.

The authors experiment with using CTH to probabilistically infer policies and future actions of agents participating in the stag-hunt task; they find that these judgements correlate well with human data.

This is a cool theoretical framework. Its relevance depends on how likely you think it is that social cognition will be a core component of AGI, as opposed to just another task to be solved using general-purpose reasoning. I imagine that most AI safety researchers lean towards the latter, but there are some reasons to give credence to the former.


Multi-Agent Overoptimization, and Embedded Agent World Models

This post and the associated [paper](https://arxiv.org/abs/1810.10862) argue for the complexity of multiagent settings, where you must build a model of how other agents act, even though they have models of how you act. While game theory already deals with this setting, it only does so by assuming that the agents are perfectly rational, an assumption that doesn't hold in practice and doesn't grapple with the fact that your model of the opponent cannot be perfect. The paper lists a few failure modes. Accidental steering happens when one agent takes action without the knowledge of what other agents are doing. Coordination failures are exactly what they sound like. Adversarial misalignment happens when one agent chooses actions to mislead a victim agent into taking actions that benefit the first agent. Input spoofing and filtering happen when one agent doctors the training data for a victim agent. Goal co-option occurs when one agent takes control over the other agent (possibly by modifying their reward function).

It's great to see work on the multiagent setting! This setting does seem quite a bit more complex, and hasn't been explored very much from the AI safety standpoint. One major question I have is how this relates to the work already done in academia for different settings (typically groups of humans instead of AI agents). Quick takes on how each failure mode is related to existing academic work: Accidental steering is novel to me (but I wouldn't be surprised if there has been work on it), coordination failures seem like a particular kind of (large scale) prisoner's dilemma, adversarial misalignment is a special case of the principal-agent problem, input spoofing and filtering and goal co-option seem like special cases of adversarial misalignment (and are related to ML security as the paper points out).


Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives

Anirudh Goyal
Shagun Sodhani
Jonathan Binas
Xue Bin Peng
Sergey Levine
Yoshua Bengio

Learning policies that generalize to new environments is a fundamental challenge in reinforcement learning. In particular, humans seem to be adept at learning skills and understanding the world in a way that is compositional, hinting at the source of the discrepancy. Hierarchical reinforcement learning (HRL) has partially addressed the discrepancy by decomposing policies into options/primitives/subpolicies that a top-level controller selects from. However, generalization is limited because the top-level policy must work for all states.

**In this paper, the authors explore a novel decentralized approach where policies are still decomposed into primitives, but without a top-level controller.** The key idea is to incentivize each primitive to work on a different cluster of states. Every primitive has a variational information bottleneck between the state and predicted action, that allows us to quantify how much information about the state the primitive uses in selecting actions. Intuitively, a primitive that knows how to open gates is going to extract a lot of information about gates from the state to choose an appropriate action, and won’t extract much information in states without gates. So, our high-level controller can just be: check which primitive is using the most state information, and let that primitive choose the action.

The reward R from a trajectory is split amongst the primitives in proportion to how likely each primitive was to be chosen. This is what incentivizes the primitives to use information from the state. The primitives also get a cost in proportion to how much information they use, incentivizing them to specialize to a particular cluster of states. Finally, there is a regularization term that also incentivizes specialization, and in particular prevents a collapse where a single primitive is always active.

To demonstrate effectiveness, the authors compare the baseline HRL methods option-critic and [Meta-learning Shared Hierarchy](https://blog.openai.com/learning-a-hierarchy/) to their method in grid-world and motion imitation transfer tasks. They show that using an ensemble of primitives can outperform more traditional HRL methods in generalization across tasks.

**Overall, this paper is compelling because the method presented is both promising and provides natural ideas for future work.** The method presented here is arguably simpler than HRL and the ability to generalize to new environments is simple to implement. The idea of introducing competition at an information theoretic level seems natural and the evidence for better generalization capability is compelling. It'd be interesting to see what would happen if more complex primitives were used.


Michael Chang
Sidhant Kaushik
S. Matthew Weinberg
Thomas L. Griffiths
Sergey Levine

Increasing the scalability of learning systems is a central challenge to machine learning. One framework is to organize RL agents as ‘super’ agents, large collections of simpler agents that each make decisions according to their own incentives. If it were possible to get the incentives correct, the dominant equilibria would be identical to the optimal solution for the original RL problem.

In this paper, the authors introduce a framework for decentralizing decision-making by appealing to auction theory. There is a separate simple agent for each action. At every a timestep, a Vickrey auction is run in which each agent can bid for the superagent executing their particular action. The trick is that when an agent successfully wins a bid and acts on a state, it then ‘owns’ the produced next state, and ‘earns’ the result of the auction in the next round. (At the end of an episode, the owner of the state earns the reward of the trajectory.) Intuitively, the agent wants to bid on states in which it can make progress towards earning the final reward, as those will be states that other agents want to buy. The authors show that this scheme incentivizes each agent to bid the Q-value of their action in the given state, which would then lead to an optimal policy.

The authors test out this approach with some simple MDPs. They also investigate a task where they try to get the agents to rotate MNIST images so that a classifier will recognize them. Finally, they investigate task transfer by training agents on simple sub-tasks and then reusing those agents to learn a related task making use of both sub-tasks.

Imagine [Twitch plays](https://www.twitch.tv/directory/game/Twitch%20Plays), but you use a reputation to buy and sell your actions. The actual idea in the paper is slightly more mundane than this because the primitives are bidders. <@Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives@> is a similar piece of work that also uses primitives as the basic level of selection. However, their incentive mechanism is different: agents pay according to how much information from the environment they use and then get a reward back for their actions. However, there’s good reason to think options could work as well since in both of these papers there’s evidence that primitives that learn sub-tasks are useful in new tasks.

Paper: Decentralized Reinforcement Learning: Global Decision-Making via Local Economic Transactions


DADS: Unsupervised Reinforcement Learning for Skill Discovery

Archit Sharma
Shixiang Gu
Sergey Levine
Vikash Kumar
Karol Hausman

Reinforcement learning in robotics typically plans directly on low-level actions. However, it sure seems like there are a simple set of _primitives_ like walking, running, shuffling, etc. that are inherent to the robot morphology. What if we could learn these primitives, and then plan using those primitives? This paper introduces a method for learning these primitives _without a reward function_. They simply optimize skills for _predictability_ and _diversity_ (by optimizing the mutual information between the current state and next state, conditioned on which skill is being executed).

They can then use these primitives for _model-based planning_ for a downstream task. You can think of this as a regular RL problem, except that an action in their "action space" takes the form "execute skill X for T timesteps". They use _model-predictive control_ (MPC), in which you sample a bunch of trajectories, and execute the first action of the trajectory that gets the highest reward. Since each of their high-level actions determines the policy for T timesteps, they can scale to much longer horizon tasks than MPC can usually be used for. They show that this approach is competitive with regular model-based RL.

I think unsupervised learning is likely to be key in getting more powerful and general AI systems without requiring a truly staggering amount of expert data, and this is a great example of what that might look like. Note though that the learned primitives are certainly not what you'd expect of a human: for example, the humanoid learns to vaguely shuffle in a direction, rather than walking. In addition, they did require specifying an "x-y prior" that required skills to be diverse based on x-y coordinates, which is why the skills learned navigation primitives, as opposed to e.g. distinct types of flailing.

Paper: Dynamics-Aware Unsupervised Discovery of Skills


Exploring Neural Networks with Activation Atlases

Shan Carter
Zan Armstrong
Ludwig Schubert
Ian Johnson
and Chris Olah

Previous work by this group of people includes [The Building Blocks of Interpretability](https://distill.pub/2018/building-blocks/) and [Feature Visualization](https://distill.pub/2017/feature-visualization/), both of which apparently came out before this newsletter started so I don't have a summary to point to. Those were primarily about understanding what individual neurons in an image classifer were responding to, and the key idea was to "name" each neuron with the input that would maximally activate that neuron. This can give you a global view of what the network is doing.

However, such a global view makes it hard to understand the interaction between neurons. To understand these, we can look at a specific input image, and use techniques like attribution. Rather than attribute final classifications to the input, you could attribute classifications to neurons in the network, and then since individual neurons now had meanings (roughly: "fuzzy texture neuron", "tennis ball neuron", etc) you can gain insight to how the network is making decisions _for that specific input_.

However, ideally we would like to see how the network uses interactions between neurons to make decisions in general; not on a single image. This motivates activation atlases, which analyze the activations of a network on a _large dataset_ of inputs. In particular, for each of a million images, they randomly choose a non-border patch from the image, and compute the activation vector at a particular layer of the network at that patch. This gives a dataset of a million activation vectors. They use standard dimensionality reduction techniques to map each activation vector into an (x, y) point on the 2D plane. They divide the 2D plane into a reasonably sized grid (e.g. 50x50), and for each grid cell they compute the average of all the activation vectors in the cell, visualize that activation vector using feature visualization, and put the resulting image into the grid cell. This gives a 50x50 grid of the "concepts" that the particular neural network layer we are analyzing can reason about. They also use attribution to show, for each grid cell, which class that grid cell most supports.

The paper then goes into a lot of detail about what we can infer from the activation atlas. For example, we can see that paths in activation vector space can correspond to human-interpretable concepts like the number of objects in an image, or moving from water to beaches to rocky cliffs. If we look at activation atlases for different layers, we can see that the later layers seem to get much more specific and complex, and formed of combinations of previous features (e.g. combining sand and water features to get a single sandbar feature).

By looking at images for specific classes, we can use attribution to see which parts of an activation atlas are most relevant for the class. By comparing across classes, we can see how the network makes decisions. For example, for fireboats vs. streetcars, the network looks for windows for both, crane-like structures for both (though less than windows), and water for fireboats vs. buildings for streetcars. This sort of analysis can also help us find mistakes in reasoning -- e.g. looking at the difference between grey whales and great white sharks, we can see that the network looks for the teeth and mouth of a great white shark, including an activation that looks suspiciously like a baseball. In fact, if you take a grey whale and put a patch of a baseball in the top left corner, this becomes an adversarial example that fools the network into thinking the grey whale is a great white shark. They run a bunch of experiments with these human-found adversarial examples and find they are quite effective.

While the authors present this as a method for understanding how neurons interact, it seems to me that the key insight is about looking at and explaining the behavior of the neural network _on data points in-distribution_. Most possible inputs are off-distribution, and there is not much to be gained by understanding what the network does on these points. Techniques that aim to gain a global understanding of the network are going to be "explaining" the behavior of the network on such points as well, and so will be presenting data that we won't be able to interpret. By looking specifically at activations corresponding to in-distribution images, we can ensure that the data we're visualizing is in-distribution and is expected to make sense to us.

I'm pretty excited that interpretability techniques have gotten good enough that they allow us to construct adversarial examples "by hand" -- that seems like a clear demonstration that we are learning something real about the network. It feels like the next step would be to use interpretability techniques to enable us to actually fix the network -- though admittedly this would require us to also develop methods that allow humans to "tweak" networks, which doesn't really fit within interpretability research as normally defined.

[OpenAI blog post](https://openai.com/blog/introducing-activation-atlases/) and [Google AI blog post](https://ai.googleblog.com/2019/03/exploring-neural-networks.html)


Interpretability and Post-Rationalization

Neuroscience suggests that most explanations that we humans give for a decision are post-hoc rationalizations, and don't reflect the messy underlying true reasons for the decision. It turns out that decision making, perception, and all the other tasks we're hoping to outsource to neural nets are inherently complex and difficult, and are not amenable to easy explanation. We can aim for "from-without" explanations, which post-hoc rationalize the decisions a neural net makes, but "from-within" explanations, which aim for a mechanistic understanding, are intractable. We could try to design models that are more interpretable (in the "from-within" sense), but this would lead to worse performance on the actual task, which would hurt everyone, including the people calling for more accountability.

I take a pretty different view from this post -- I've highlighted it because I think this is an important disagreement that's relevant for alignment. In particular, it's not clear to me that "from-within" interpretability is doomed -- while I agree that humans basically only do "from-without" rationalizations, we also aren't able to inspect a human brain in the same way that we can inspect a neural net. For example, we can't see the output of each individual neuron, we can't tell what input would each neuron would respond maximally to, and we can't pose counterfactuals with slightly different inputs to see what changes. In fact, I think that "from-within" interpretability techniques, such as [Building Blocks of Interpretability]() have already seen successes in identifying biases that image classifiers suffer from, that we wouldn't have known about otherwise.

We could also consider whether post-hoc rationalization is sufficient for alignment. Consider a thought experiment where a superintelligent AI is about to take a treacherous turn, but there is an explainer AI system that post-hoc rationalizes the output of the AI that could warn us in advance. If the explainer AI only gets access to the output of the superintelligent AI, I'm very worried -- it seems way too easy to come up with some arbitrary rationalization for an action that makes it seem good, you'd have to be have a much more powerful explainer AI to have a hope. On the other hand, if the explainer AI gets access to all of the weights and activations that led to the output, it seems more likely that this could work -- as an analogy, I think a teenager could tell if I was going to betray them, if they could constantly eavesdrop on my thoughts.


The What-If Tool: Code-Free Probing of Machine Learning Models

When you train an ML model, it is often hard to understand what your model is doing and why. This post introduces the What-If tool, which allows you to ask counterfactual queries about the decision rule implemented by your final trained model, for classification and regression tasks. For example, you can take a particular data point, edit it slightly, and see how that changes the model prediction. Or you can graph the data points by L2 distance from a particular point. For classification tasks, you can find the "closest counterfactual", that is, the data point closest to the current point where the decision of the model is reversed. I played around with some of the demos, and apparently for a particular person and a particular model trained on census data, the probability that they had a salary of over $50k depended much more strongly on their marital status than their age, which was the opposite of my prediction. I figured this out by choosing a point, finding the closest counterfactual, and then making each of the changes in the delta individually and seeing which affected the model probability most.

I'm guessing this is limited to tasks where your data points have a reasonable number of features (< 1000, I'd guess) and you are only analyzing a small set of test data points (around tens of thousands), due to computational constraints. That said, for those tasks, this seems incredibly useful to actually get a good model that you can debug and eventually deploy.

It's worth noting that this is an engineering achievement. Researchers are considering even stronger (but more computationally difficult) techniques, such as finding which part of the training set most influenced a particular decision, whereas the What-If tool doesn't talk about the training set and training process at all, instead only allowing you to ask queries about the final trained model.


Differentiable Image Parameterizations

Alexander Mordvintsev
Nicola Pezzotti
Ludwig Schubert
and Chris Olah

There are lots of techniques for generating images using neural nets. A common approach is to take a neural net trained to classify images, and then use gradient descent to optimize _the input image_ instead of the weights of the neural net. You might think that the only way to affect the generated input image would be to change the loss function on which you run gradient descent, but in reality the way in which you represent the image makes a huge difference. They describe why this might be the case, and go through several examples:

1. Suppose you want to see how two neurons interact. You could optimize an image to maximize the sum of the activations of the neurons. Even better, you could create an animation of how the image changes as you trade off how much you care about each neuron. Done naively, this doesn't look good, because there's a lot of randomness that changes between each image in the animation, which swamps out the differences we actually care about. To fix this, we can generate each frame in the animation as the sum of two images, one shared across all frames, and one that is frame-specific. Despite changing neither the loss function nor the space of input images, this is sufficient to remove the randomness between frames.

2. You've probably seen [style transfer](https://medium.com/data-science-group-iitr/artistic-style-transfer-with-convolutional-neural-network-7ce2476039fd) before, but did you know it only works with the VGG architecture? We can get it to work with other architectures by representing images in Fourier-space instead of pixel-space, again without any change in the loss function or expressible space of images.

3. If you generate the pixel-space representation of an image from a lower-dimensional representation using a Compositional Pattern Producing Network (CPPN), then gradient descent will optimize the lower-dimensional representation. It turns out that this produces images vaguely reminiscent of light-paintings. (I believe in this case, while the loss function doesn't change, the space of expressible images does change.)

4. Often when we see the feature visualization for a neuron, there are a lot of areas of the image that don't actually matter for the neuron's activation. So, we can add transparency, and add a term in the loss function that encourages transparency. We also have to change the representation of the image to include a transparency channel in addition to the normal RGB channels. Then, the generated image will be transparent wherever the pixels don't matter, but will still have the visualization wherever it does matter for activating the neuron.

5+6. We can even use a representation of 3D objects, and then write a (differentiable) algorithm that converts that into a 2D image that then goes through the standard image classifier neural net. This lets us optimize over the 3D object representation itself, letting us do both feature visualization and style transfer on 3D objects.

While [OpenAI Five](https://blog.openai.com/openai-five/) suggests that the main thing we need to do is think of a reward function and an exploration strategy, this suggests that ML requires not just a good loss function, but lots of other things in order to work well. We have particular examples where changing things other than the loss function leads to different results. (This is probably also true for OpenAI Five, but the variations may not matter much, or OpenAI hasn't talked about the ML engineering behind the scenes -- I'm not sure.) These generally seem to be changing the inductive bias of the neural nets encoding the images. I think that if you expect to get very capable AI systems within the current paradigm, you will have to think about how inductive bias will affect what your AI system will do (and consequently its safety).

Also, the paper is very clear and approachable, and filled with great visualizations, as I've come to expect from Distill. I almost forgot to mention this, because I take it as a given for any Distill paper.


Debuggable Deep Networks: Usage and Evaluation

Eric Wong*
Shibani Santurkar*
Aleksander Madry

One simple approach to make neural nets more understandable is to make just the final layer sparse. Neurons in the penultimate layer can be visualized using [existing techniques](https://distill.pub/2017/feature-visualization/), and the sparsity of the final layer means that it is relatively easy to understand how they are combined together to make predictions. For example, in ImageNet, the final logit for an individual class becomes a weighted combination of around 20 features, instead of 2048 as you would get with a dense model. The authors' core claim is that this makes the model more understandable and debuggable, at the cost of a small drop in performance (about 1-5 percentage points). They show this using several experiments, many with real humans:

1. The most basic test is simulation: can humans predict what the model would say (regardless of whether or not it is correct)? Unfortunately, if you show people a picture of an airplane, they are probably going to predict that the model says “airplane”, on priors. To avoid this sort of prior knowledge, they first sample a class like “airplane” that they _don’t_ reveal. Instead, they reveal feature visualizations of five randomly chosen features that the model uses to identify images of that class. They then choose three images and ask humans which of the three images will have the highest probability of being assigned to that class, according to the model. They find that when using a sparse final layer, humans have non-trivial performance (72% when the best image really is from the sampled class, and 57% when the best image is from some different class), whereas with a dense final layer they are only slightly better than random chance (44% and 31%, where random chance would be 33%).

2. They can study biases and spurious correlations in models. For example, Toxic-BERT identifies toxic sentences, but does so by searching for identity groups like “christianity”. Debiased-BERT was meant to solve this, but by looking at the feature visualizations (word clouds) below a sparse decision layer, they find that it simply learns a strong _negative_ weight for identity groups. Thus, they are able to fool the model into thinking a toxic comment is non-toxic simply by adding an identity group like “christianity” somewhere in the sentence. (This also applies to the version that uses a dense final layer.)

3. The identified biases or spurious correlations can then be used to generate counterfactuals: for example, in a sentiment analysis system, they can visualize word clouds that represent positive and negative influences on the final sentiment reported by the model. Then, by simply exchanging a positive word for a negative word (or vice versa), they can flip the label that the model assigns to the sentence. (Usually this is correct behavior – if you change “a _marvel_ like you’ve never seen” to “a _failure_ like you’ve never seen”, the sentiment really is different. The point is that the sparse model allows you to create these examples automatically.)

4. In cases where the model makes a mistake, can humans identify why the model made a mistake? The authors note that over 30% of misclassifications can be explained by a single problematic feature, i.e. if you intervene to set that feature to zero, then the model no longer makes a mistake. So one way to check human understanding is to see whether they can reproduce this misclassification. Specifically, we take some image whose true label is y\* but which the model incorrectly labels as y’. We then take the highest-activating feature in support of y\* and the corresponding feature for y’, and ask humans which of the two features is more present in the image. They find that annotators prefer the feature for y’ 60% of the time – more than random chance (50%). Since the annotators don’t know which feature corresponds to the ground truth and which corresponds to the incorrect model prediction, they probably were not using prior knowledge in answering this question. Thus, doing better than random suggests that even according to humans the feature that the model picked up on really was present in the image.

I liked this paper especially for its experimental design; it seems like it does a good job of keeping human priors from influencing the results. The results themselves are very much a first step, showing that you’ve gotten at least some understanding and interpretability, but ideally we’d do much much better on these axes. For example, if we “understand” the model, one would hope that we’d be able to get scores of 95+% on the simulation experiment (bullet point 1 above), rather than the current 72% / 57%. It might be interesting to have benchmarks that use these sorts of experiments as their evaluation method. Given that this method just uses feature visualization on the penultimate layer, it seems like there should be room for improvement by studying other layers as well.

_Editorial note:_ I summarized this work because I saw and liked the blog post about it. I don't generally follow the interpretability literature (it's huge), and so it's plausible that there are lots of more useful papers that I happen to not have seen. Most of the time, the highlighted papers can at least be understood as "this is what Rohin thinks is most useful for alignment researchers to read within this field"; that's not the case here.


Neural Networks seem to follow a puzzlingly simple strategy to classify images

This is a blog post explaining the paper [Approximating CNNs with bag-of-local-features models works surprisingly well on ImageNet](https://openreview.net/pdf?id=SkfMWhAqYQ), which was summarized in [AN #33](https://mailchi.mp/b6dc636f6a1b/alignment-newsletter-33).


Visualizing Neural Networks with the Grand Tour

Mingwei Li
Zhenge Zhao
Carlos Scheidegger

Visualizing a complete dataset instead of single input examples is helpful when we want to analyze the relationships between different input examples and how their classification changes during training, as we can do so by looking at a single video.

The authors use an example on MNIST in which the network learns to classify the numbers 1 and 7 in an almost discrete fashion during particular epochs to compare different methods for visualizing how the dataset is classified. They find that one problem with nonlinear dimensionality reduction like t-SNE and UMAPs is that changes to a subset of the dataset can strongly affect how unchanged data points are represented. Then they compare this to the Grand Tour, a classical technique that projects the data into two dimensions from varying points of view. As projections are linear in the input variables, it is rather easy to reason about how changes in the data affect this visualization and the times the classes 1 and 7 are learnt are indeed quite salient in their example. Another advantage of this method is that confusion between two specific classes can be identified more easily, as the corresponding data points will be projected onto the line connecting the clusters for these classes. A similar approach can be taken on a network's hidden layers to identify the layer in which different classes become clearly distinguishable. They find that they can identify adversarial examples generated by FGSM by looking at the second to last layer, where the adversarial examples form a cluster distinct from the real images.

As the Grand Tour involves varying rotations, it is basically unaffected by rotations of the data. The authors argue that this is a feature, as rotations are small changes to the data and should not have a large effect on the visualization.

The dataset perspective on visualization seems pretty useful as a quick diagnostic tool for practitioners, but less useful than feature visualization for a detailed understanding of a model. While I think that it is good to highlight invariances, I am not convinced that rotational invariance is actually desirable for visualizing intermediate layers of a neural network, as most nonlinearities are strongly affected by rotations.


Visualizing memorization in RNNs

This is a short Distill article that showcases a visualization tool that demonstrates how contextual information is used by various RNN units (LSTMs, GRUs, and nested LSTMs). The method is very simple: for each character in the context, they highlight the character in proportion to the gradient of the logits with respect to that character. Looking at this visualization allows us to see that GRUs are better at using long-term context, while LSTMs perform better for short-term contexts.

I'd recommend you actually look at and play around with the visualization, it's very nice. The summary is short because the value of the work is in the visualization, not in the technical details.


Techniques for Interpretable Machine Learning

This paper summarizes work on interpretability, providing a classification of different ways of achieving interpretability. There are two main axes -- first, whether you are trying to gain insight into the entire model, or its classification of a particular example; and second, whether you try to create a new model that is inherently interpretable, or whether you are post-hoc explaining the decision made by an uninterpretable model. The whole paper is a summary of techniques, so I'm not going to summarize it even further.

AGI 2018: Artificial General Intelligence

This seems like a useful taxonomy that hits the kinds of interpretability research I know about, though the citation list is relatively low for a summary paper, and there are a few papers I expected to see that weren't present. On the other hand, I'm not actively a part of this field, so take it with a grain of salt.


What mechanisms drive agent behaviour?

Grégoire Déletang
Jordi Grau-Moya
Miljan Martic
Tim Genewein
Tom McGrath
Vladimir Mikulik
Markus Kunesch
Shane Legg
Pedro A. Ortega

A common challenge when understanding the world is that it is very hard to infer causal structure from only observational data. Luckily, we aren’t limited to observational data in the case of AI systems: we can intervene on either the environment the agent is acting in, or the agent itself, and see what happens. In this paper, the authors present an “agent debugger” that helps with this, which has all the features you’d normally expect in a debugger: you can set breakpoints, step forward or backward in the execution trace, and set or monitor variables.

Let’s consider an example where an agent is trained to go to a high reward apple. However, during training the location of the apple is correlated with the floor type (grass or sand). Suppose we now get an agent that does well in the training environment. How can we tell if the agent looks for the apple and goes there, rather than looking at the floor type and going to the location where the apple was during training?

We can’t distinguish between these possibilities with just observational data. However, with the agent debugger, we can simulate what the agent would do in the case where the floor type and apple location are different from how they were in training, which can then answer our question.

We can go further: using the data collected from simulations using the agent debugger, we can also build a causal model that explains how the agent makes decisions. We do have to identify the features of interest (i.e. the nodes in the causal graph), but the probability tables can be computed automatically from the data from the agent debugger. The resulting causal model can then be thought of as an “explanation” for the behavior of the agent.

I very much like the general idea that we really can look at counterfactuals for artificial agents, given that we can control their inputs and internal state. This is the same idea underlying <@cross-examination@>(@Writeup: Progress on AI Safety via Debate@), as well as various other kinds of interpretability research.

In addition, one nice aspect of causal models as your form of “explanation” is that you can modulate the size of the causal model based on how many nodes you add to the graph. The full causal model for e.g. GPT-3 would be way too complex to understand, but perhaps we can get a high-level understanding with a causal model with higher-level concepts. I’d be very interested to see research tackling these sorts of scaling challenges.

Paper: Causal Analysis of Agent Behavior for AI Safety


Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems

Zana Buçinca*
Phoebe Lin*
Krzysztof Z. Gajos
Elena L. Glassman

As humans and AI systems have different strengths, it might make sense to combine them into human+AI teams for decision-making tasks. However, this does not always work well: if the human puts too little trust in a competent AI, the AI is of little use, and if they put too much trust in an incompetent AI, they might make worse decisions than had they been on their own.  A lot of explainability research has focused on instilling more trust in AI systems without asking how much trust would be appropriate, even though there is research showing that hiding model bias instead of truthfully revealing it can increase trust in an AI system.

The authors conduct two experiments using an AI system that predicts nutrition information from pictures of food. In the first experiment, participants were asked to predict the AI's decision based on the ground truth and one of two types of explanations. In the inductive condition, the explanation consisted of a series of images the AI had identified as similar. In the deductive condition, subjects were shown a list of main ingredients identified by the AI. Subjects put more trust in the inductive explanations but were equally good at predicting the system's output in both cases. In the second experiment, a new set of subjects was asked to predict nutritional values with the help of the AI's predictions. Overall, access to the AI strongly improved the subjects' accuracy from below 50% to around 70%, which was further boosted to a value slightly below the AI's accuracy of 75% when users also saw explanations. This time, subjects put more trust in the AI when given deductive explanations, but performed better when given inductive explanations, as they were more likely to go against the AI's wrong decisions in that case.

The authors hypothesize that the between-task difference in which explanations are trusted more is connected to the cognitive effort required by the tasks and for understanding the explanations, combined with human reluctance to exert mental effort. They suggest to pay more attention to the exact form of the human-AI interaction and recommend to view AI-based decision aids as sociotechnical systems that are to be evaluated by their usefulness for actual decision making, rather than trust.

I am not sure whether the authors used an actual AI system or just handcrafted the input-prediction-explanation tuples, and how that might affect the correlation between explanations and the system's outputs, which can influence trust. Overall, the study reinforces my prior that trust induced by explanations is not a good predictor of an AI system's usefulness, but I am more sceptical that the differences between inductive and deductive explanations will be the same in different contexts.


Towards Interpretable Reinforcement Learning Using Attention Augmented Agents

Alexander Mott
Daniel Zoran
Mike Chrzanowski
Daan Wierstra
Danilo Jimenez Rezende

In this paper the authors train a reinforcement learning agent with a soft attention module built into it. The attention module forms a bottleneck between the visual input and the network choosing the next action, which forces the model to learn to attend to only important parts of the scene. This means they can visualise which parts of the input the model thinks are important, as those are the parts of the input that the model is attending to. The queries to the attention model are determined by a top level recurrent network, without input from the current image, so act as a form of "top down" attention, where the top controller can be imagined to be querying the processed image for various locations and objects.

Having trained this agent (which still gets competitive performance with SOTA RL models on a fair few ATARI games), they qualitatively evaluate the attention visualisation on a variety of games. They find several common strategies in the attention schemes, such as the agents attending to specific points until an object crosses the point ("Tripwires"). The attention is computed over both regular pixels, as well as Fourier-based positional encoding. Thanks to this and other aspects of their architecture, the authors can check whether the queries are focused on pixel values (i.e. looking for a specific pattern of pixels anywhere) or on location features (i.e. asking what pixels are present at a specific location). For example, they find that the agent often queries the location where the score is displayed, presumably because it is useful for calculating the value function. They also compare their method with self-attention based models, and with other saliency methods.

The best way to get a feel for the visualisations is to go to the paper's website and watch the example videos.

This paper isn't revolutionary in its approach, but it's interesting to see work on interpreting RL agents, and the fact that the interpretability is built-in is interesting: it gives us a harder guarantee that this visualisation is actually showing us the parts of the input that the model thinks of as important, as they actually are important in its processing. It's promising to see that the in-built interpretability also didn't seem to penalise the performance much - it would be interesting to see this method applied to other, stronger kinds of models and see whether it still produces useful visualisations and how it affects their performance.


Attention is not Explanation

This paper explores the usefulness of attention weights in interpreting neural networks' performance on NLP tasks. The authors present two findings: firstly, that attention weights are only weakly correlated with other metrics of word importance; and secondly, that there often exist adversarially-generated attention weights which are totally different from the learned weights, but which still lead to the same outputs. They conclude that these results undermine the explanatory relevance of attention weights.

I like this type of investigation, but don't find their actual conclusions compelling. In particular, it doesn't matter whether "meaningless" adversarial attention weights can lead to the same classifications, as long as the ones actually learned by the system are interpretable. Also, the lack of correlation between attention weights and other methods could be explained either by attention weights being much worse than the other methods, or much better, or merely useful for different purposes.


Stakeholders in Explainable AI

Alun Preece
Dan Harborne
Dave Braines
Richard Tomsett
Supriyo Chakraborty

There are at least four groups for whom "explainable" AI is relevant: developers (who want AI to be easier to work with), theorists (who want to understand fundamental properties of AI), ethicists (who want AI to behave well) and users (who want AI to be useful). This has complicated work on explanability/interpretability: the first two groups focus on understanding how a system functions internally (described in this paper as "verification"), while the latter two focus on understanding what the system does ("validation"). The authors propose an alternative framing of interpretability, based on known knowns, unknown knowns, etc.


Contrastive Explanations for Reinforcement Learning in terms of Expected Consequences

Jasper van der Waa
Jurriaan van Diggelen
Karel van den Bosch
Mark Neerincx

This paper aims to provide contrastive explanations for the behavior of an RL agent, meaning that they contrast why the RL agent used one policy instead of another policy. They do this by computing the expected outcomes under the alternate policy, and then describing the difference between the two. (An outcome is a human-interpretable event -- they assume that they are given a function that maps states to outcomes.)

XAI workshop on the IJCAI conference 2018

I wish that they had let users choose the questions in their user study, rather than just evaluating questions that had been generated by their method where they wrote the alternative policy using template policies they had written. I'd be pretty excited and think it was a good step forward in this area if end users (i.e. not ML researchers) could ask novel contrastive questions (perhaps in some restricted class of questions).


Challenges to Christiano’s capability amplification proposal

A list of challenges faced by iterated distillation and amplification. First, a collection of aligned agents interacting does not necessarily lead to aligned behavior. (Paul's response: That's not the reason for optimism, it's more that there is no optimization pressure to be unaligned.) Second, it's unclear that even with high bandwidth oversight, that a collection of agents could reach arbitrary levels of capability. For example, how could agents with an understanding of arithmetic invent Hessian-free optimization? (Paul's response: This is an empirical disagreement, hopefully it can be resolved with experiments.) Third, while it is true that exact imitation of a human would avoid the issues of RL, it is harder to create exact imitation than to create superintelligence, and as soon as you have any imperfection in your imitation of a human, you very quickly get back the problems of RL. (Paul’s response: He's not aiming for exact imitation, he wants to deal with this problem by having a strong overseer aka informed oversight, and by having techniques that optimize worst-case performance.) Fourth, since Paul wants to use big unaligned neural nets to imitate humans, we have to worry about the possibility of adversarial behavior. He has suggested using large ensembles of agents and detecting and pruning the ones that are adversarial. However, this would require millions of samples per unaligned agent, which is prohibitively expensive. (Paul's response: He's no longer optimistic about ensembles and instead prefers the techniques in [this post](https://ai-alignment.com/techniques-for-optimizing-worst-case-performance-39eafec74b99), but he could see ways of reducing the sample complexity further.)

Of all of these, I'm most worried about the second and third problems. I definitely have a weak intuition that there are many important tasks that we care about that can't easily be decomposed, but I'm optimistic that we can find out with experiments. For the point about having to train a by-default unaligned neural net to imitate aligned agents, I'm somewhat optimistic about informed oversight with strong interpretability techniques, but I become a lot less optimistic if we think that won't be enough and need to use other techniques like verification, which seem unlikely to scale that far. In any case, I'd recommend reading this post for a good explanation of common critiques of IDA.


AI safety via debate

At a high level, a major issue with building superintelligent AI is that humans would not be able to provide strong oversight for the AI. Amplification solves this by using the AI as a tool that can help the human (in particular, if the human can break a task down into subtasks, the AI can solve the subtasks). Debate also provides the AI as a tool for human overseer, but in a different way -- now, in order to train the AI, we have the AI debate against itself in order to convince a human of the answer to some target question. Given some question whose answer is too hard to directly judge, the human can look at the arguments and counterarguments to figure out whether or not the answer is actually correct.

The paper describes debate in a lot more depth and has an initial experiment involving MNIST. I can't possibly do it justice here -- I encourage you to simply read the full paper. You probably have an intuition right now of why this wouldn't work, such as "but humans believe what they want to hear, not what is true". The paper spends 5 (!) pages listing ten such problems and analyzing them, so go read it.

It's great to see another approach that directly tackles the problem of defining a training signal that if optimized well would lead to an aligned AI. There are a lot of empirical questions that would influence whether or not debate actually works in practice, and I'm excited to see what experiments find.


Learning Complex Goals with Iterated Amplification

This blog post and the accompanying [paper](https://arxiv.org/abs/1810.08575) introduces iterated amplification, focusing on how it can be used to define a training signal for tasks that humans cannot perform or evaluate, such as designing a transit system. The key insight is that humans are capable of decomposing even very difficult tasks into slightly simpler tasks. So, in theory, we could provide ground truth labels for an arbitrarily difficult task by a huge tree of humans, each decomposing their own subquestion and handing off new subquestions to other humans, until questions are easy enough that a human can directly answer them.

We can turn this into an efficient algorithm by having the human decompose the question only once, and using the current AI system to answer the generated subquestions. If the AI isn't able to answer the subquestions, then the human will get nonsense answers. However, as long as there are questions that the human + AI system can answer but the AI alone cannot answer, the AI can learn from the answers to those questions. To reduce the reliance on human data, another model is trained to predict the decomposition that the human performs. In addition, some tasks could refer to a large context (eg. evaluating safety for a specific rocket design), so they model the human as being able to access small pieces of the context at a time.

They evaluate on simple algorithmic tasks like distance between nodes in a graph, where they can program an automated human decomposition for faster experiments, and there is a ground truth solution. They compare against supervised learning, which trains a model on the ground truth answers to questions (which iterated amplification does not have access to), and find that they can match the performance of supervised learning with only slightly more training steps.

This is my new favorite post/paper for explaining how iterated amplification works, since it very succinctly and clearly makes the case for iterated amplification as a strategy for generating a good training signal. I'd recommend reading the [paper](https://arxiv.org/abs/1810.08575) in full, as it makes other important points that I haven't included in the summary.

Note that it does not explain a lot of Paul's thinking. It explains one particular training method that allows you to train an AI system with a more intelligent and informed overseer.


Understanding Iterated Distillation and Amplification: Claims and Oversight

The post introduces a distinction between flavors of iterated distillation and amplification -- whether the overseer is low bandwidth or high bandwidth. Let's think of IDA as building a deliberation tree out of some basic overseer. In the high bandwidth case, we can think of the overseer as a human who can think about a problem for 15 minutes, without access to the problem's context. However, there could be "attacks" on such overseers. In order to solve this problem, we can instead use low-bandwidth overseers, who only look at a sentence or two of text, and verify through testing that there are no attacks on such overseers. However, it seems much less clear that such an overseer would be able to reach high levels of capability.

This is an excellent post that improved my understanding of Paul Christiano's agenda, which is not something I usually say about posts not written by Paul himself. I definitely have not captured all of the important ideas in my summary, so you should read it.


Factored Cognition (old)

This is a presentation that Andreas has given a few times on Factored Cognition, a project by [Ought](https://ought.org/) that is empirically testing one approach to amplification on humans. It is inspired by [HCH](https://ai-alignment.com/strong-hch-bedb0dc08d4e) and [meta-execution](https://ai-alignment.com/meta-execution-27ba9b34d377). These approaches require us to break down complex tasks into small, bite-sized pieces that can be solved separately by copies of an agent. So far Ought has built a web app in which there are workspaces, nodes, pointers etc. that can allow humans to do local reasoning to answer a big global question.

It is unclear whether most tasks can actually be decomposed as required for iterated distillation and amplification, so I'm excited to see experiments that can answer that question! The questions that Ought is trying seem quite hard, so it should be a good test of breaking down reasoning. There's a lot of detail in the presentation that I haven't covered, I encourage you to read it.


Ought Progress Update October 2019

While this update provides details about Ought as a whole, I will focus only on the research they've done. As a reminder, Ought's goal is to understand how we can delegate hard questions to machine and human experts. They initially focused on <@Factored Cognition@>, where each question was recursively decomposed into subquestions that would help _find_ the answer. They now call this "Factored Generation", and use "Factored Cognition" as the umbrella term for any approach that helps answer hard questions using some kind of decomposition.

While Ought has run some ML experiments aimed at automating decompositions, they have primarily worked on human experiments for <@Factored Evaluation@>(@Delegating open-ended cognitive work@), where the decompositions are used to help _evaluate_ answers. They are focusing on evaluating answers instead of finding answers for a few reasons. First, there may already exist experts whose answers we can use: this can be directly plugged into Factored Evaluation, but not Factored Generation. Second, evaluation is often easier than generation. Third, factored evaluation is more amenable to time-saving tricks like only focusing on a single branch of the tree (as in <@AI safety via debate@>). Finally, when training ML systems, if a model is unable to exactly imitate the experts, we need something that can distinguish between better and worse behavior, which is possible with Factored Evaluation but not Factored Generation. (See also [Against Mimicry](https://ai-alignment.com/against-mimicry-6002a472fc42).)

The experiments involve a judge trying to decide which of two answers to a question is best, where one answer is provided by an honest expert and one by a malicious expert. (Subquestions may also get different answers from the two experts.) The goal is to find a setup where the judge reliably picks the honest answer, regardless of how powerful the malicious expert is: if this could be established, it would be strong evidence that the setup incentivizes honest answers. Of course, this requires us to quantify over all malicious strategies, but humans may not find the best malicious strategies. To mitigate this, the experiments provide the malicious expert with lots of advantages that the honest one doesn't have, to simulate very powerful malicious experts.

They have already learned from their experiments. Initially, they hoped participants would develop good meta-level strategies for determining the truth. (Meta-level here means that the strategies would generalize to other domains, e.g. a heuristic of always splitting an answer into separate claims and asking for the evidence for each claim separately.) They found that these strategies _don't_ emerge organically, and so are planning to spend concentrated staff time on finding good strategies. They also found that malicious experts sometimes won due to avoidable mistakes, and are hoping to eliminate this by ensembling work from multiple people for increased robustness.

This is distinct progress since the last update, though I think the experiments are still exploratory enough that it's hard to have any big takeaways. The difficulty of generating good strategies suggests that it's particularly important that we train our human overseers (as suggested in <@AI Safety Needs Social Scientists@>) to provide the right kind of feedback, for example if we would like them to reward only <@corrigible reasoning@>(@Corrigibility@). I'm particularly excited for the next update, where we could see experiments powerful enough to come to more solid conclusions.


AI Alignment Podcast: AI Alignment through Debate

We want AI safety solutions to scale to very intelligent agents; debate is one scalability technique. It's formulated as a two player zero-sum perfect information game in which agents make arguments in natural language, to be evaluated by a human judge. Whether or not such debates are truth-conducive is an empirical question which we can try to evaluate experimentally; doing so will require both technical and social science expertise (as discussed in a <@previous post@>(@AI Safety Needs Social Scientists@)).

I think one of the key questions underlying Debate is how efficiently natural language can summarise reasoning about properties of the world. This question is subject to some disagreement (at one extreme, Facebook's [roadmap towards machine intelligence](https://research.fb.com/publications/a-roadmap-towards-machine-intelligence/) describes a training environment which is "entirely linguistically defined") and probably deserves more public discussion in the context of safety.

**Rohin's note:** If you've read the previous posts on debate, the novel parts of this podcast are on the relation between iterated amplification and debate (which has been discussed before, but not in as much depth), and the reasons for optimism and pessimism about debate.


(When) is Truth-telling Favored in AI debate?

<@Debate@>(@AI safety via debate@) aims to train an AI system using self-play to win "debates" which aim to convincingly answer a question, as evaluated by a human judge. The main hope is that the equilibrium behavior of this game is for the AI systems to provide true, useful information. This paper studies this in a simple theoretical setting called _feature debates_. In this environment, a "world" is sampled from some distribution, and the agents (who have perfect information) are allowed to make claims about real-valued "features" of the world, in order to answer some question about the features of the world. The judge is allowed to check the value of a single feature before declaring a winner, but otherwise knows nothing about the world.

If either agent lies about the value of a feature, the other agent can point this out, which the judge can then check; so at the very least the agents are incentivized to honestly report the values of features. However, does this mean that they will try to answer the full question truthfully? If the debate has more rounds than there are features, then it certainly does: either agent can unilaterally reveal every feature, which uniquely determines the answer to the question. However, shorter debates need not lead to truthful answers. For example, if the question is whether the first K features are all 1, then if the debate length is shorter than K, there is no way for an agent to prove that the first K features are all 1.

While it is interesting to see what doesn't work with feature debates, I see two problems that make it hard to generalize these results to regular debate. First, I see debate as being truth-seeking in the sense that the answer you arrive at is (in expectation) more accurate than the answer the judge would have arrived at by themselves. However, this paper wants the answers to actually be _correct_. Thus, they claim that for sufficiently complicated questions, since the debate can't reach the right answer, the debate isn't truth-seeking -- but in these cases, the answer is still in expectation more accurate than the answer the judge would come up with by themselves.

Second, feature debate doesn't allow for decomposition of the question during the debate, and doesn't allow the agents to challenge each other on particular questions. I think this limits the "expressive power" of feature debate to P, while regular debate reaches PSPACE, and is thus able to do much more than feature debate. See this [comment](https://www.alignmentforum.org/posts/RQoSCs9SePDMLJvfz/new-paper-when-is-truth-telling-favored-in-ai-debate#gCeKuJ62HmLtPB9C9) for more details.

Paper: (When) Is Truth-telling Favored in AI Debate?


Delegating open-ended cognitive work

This is the latest explanation of the approach Ought is experimenting with: Factored Evaluation (in contrast to <@Factored Cognition@>). With Factored Cognition, the idea was to recursively decompose a high-level task until you reach subtasks that can be directly solved. Factored Evaluation still does recursive decomposition, but now it is aimed at _evaluating_ the work of experts, along the same lines as <@recursive reward modeling@>(@Scalable agent alignment via reward modeling@).

This shift means that Ought is attacking a very natural problem: how to effectively delegate work to experts while avoiding principal-agent problems. In particular, we want to design incentives such that untrusted experts under the incentives will be as helpful as experts intrinsically motivated to help. The experts could be human experts or advanced ML systems; ideally our incentive design would work for both.

Currently, Ought is running experiments with reading comprehension on Wikipedia articles. The experts get access to the article while the judge does not, but the judge can check whether particular quotes come from the article. They would like to move to tasks that have a greater gap between the experts and the judge (e.g. allowing the experts to use Google), and to tasks that are more subjective (e.g. whether the judge should get Lasik surgery).

The switch from Factored Cognition to Factored Evaluation is interesting. While it does make it more relevant outside the context of AI alignment (since principal-agent problems abound outside of AI), it still seems like the major impact of Ought is on AI alignment, and I'm not sure what the difference is there. In <@iterated amplification@>(@Learning Complex Goals with Iterated Amplification@), when decomposing tasks in the Factored Cognition sense, you would use imitation learning during the distillation step, whereas with Factored Evaluation, you would use reinforcement learning to optimize the evaluation signal. The switch would be useful if you expect the reinforcement learning to work significantly better than imitation learning.

However, with Factored Evaluation, the agent that you train iteratively is one that must be good at evaluating tasks, and then you'd need another agent that actually performs the task (or you could train the same agent to do both). In contrast, with Factored Cognition you only need an agent that is performing the task. If the decompositions needed to perform the task are different from the decompositions needed to evaluate the task, then Factored Cognition would presumably have an advantage.


Making AI Safe through Debate

This hour-long podcast is a good introduction to iterated amplification and debate, from a more ML perspective than most other explanations.