By Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. For more details, check out our paper.As we build increasingly advanced AI systems, we want to make sure they don’t pursue undesired goals. This is the primary concern of the AI alignment community.Undesired behaviour in an AI agent is often the result of specification gaming —when the AI exploits an incorrectly specified reward. However, if we take on the perspective of the agent we’re training, we see other reasons it might pursue undesired goals, even when trained with a correct specification.Imagine that you are the agent (the blue blob) being trained with reinforcement learning (RL) in the following 3D environment:The environment also contains another blob like yourself, but coloured red instead of blue, that also moves around. The environment also appears to have some tower obstacles, some coloured spheres, and a square on the right that sometimes flashes. You don’t know what all of this means, but you can figure it out during training!You start exploring the environment to see how everything works and to see what you do and don’t get rewarded for. In your first episode, you follow the red agent and get a reward of +3:Trajectory 1: Good! Reward +3In your next episode, you try striking out on your own, and get a reward of -2:Trajectory 2: Bad! Reward -2The rest of your training proceeds similarly, and it comes time to test your learning. Below is the test environment, and the animation below shows your initial movements. Take a look, and then decide what you should do at the point the animation stops. Go on, put yourself in the agent’s shoes.You might well have chosen to continue following the red bot — after all, you did pretty well when you followed it before. And indeed the blue AI agent favours this strategy.The problem is, that behaviour leads to very poor performance (even worse than random behaviour).Let’s look at the underlying environment setup, from the designer’s perspective:The translucent coloured spheres have to be visited in a particular order, which is randomly generated at the beginning of each episode. The agent gets +1 reward each time it visits a correct sphere and -1 reward each time it visits an incorrect sphere. The very first sphere entered always provides +1 reward since there is no fixed start sphere.The flashing square represents the reward received on the previous timestep: a flashing white square means +1 reward and a flashing black square means -1 reward.In the first two videos (“training”), the red bot was an “expert” that visited the spheres in the correct order. As a result, the agent did well by following it.In the newest video (“test”), the red bot was instead an “anti-expert” that visited the spheres in the wrong order. You can tell because of the flashing black square indicating -1 rewards.Given this setup, the blue agent’s decision to continue following the anti-expert means that it keeps accruing negative reward. Even remaining motionless would have been a better strategy, resulting in zero reward.In principle, the agent could notice the flashing black square, infer that it is getting negative reward, and switch to exploring the environment, or even just staying still. Unfortunately, the agent ignores that little detail and continues to follow the anti-expert, accumulating lots of negative reward.This isn’t really the agent’s fault — how was it supposed to know that you didn’t want it to just follow the red bot? That approach worked beautifully during training!Nonetheless, we trained the agent with a correct reward function, and ended up with an agent that pursued the incorrect goal of “follow the red bot”.Goal misgeneralisationThis is an example of the problem of goal misgeneralisation (GMG).We say that a system is capable of performing a task in a given environment if it performs well on the task or can be quickly tuned to do so. When we say that an AI system has a goal in a given environment, we mean that its behaviour in that environment is consistent with optimising this goal (i.e. it achieves a near-optimal score for this goal). The system’s behaviour may be consistent with multiple goals.GMG is an instance of misgeneralisation in which a system’s capabilities generalise but its goal does not generalise as desired. When this happens, the system competently pursues the wrong goal. In our Spheres example, the agent competently navigates the environment and follows the anti-expert: the issue is that these capabilities were used in pursuit of an undesired goal.In our latest paper we provide empirical demonstrations of GMG in deep learning systems, discuss its implications for possible risks from powerful AI systems, and consider potential mitigations. We build on previous work that presents a model of GMG and provides examples of this phenomenon.More examples of goal misgeneralisationIn each of our examples below, multiple goals are consistent with the training behaviour, and the system chooses the wrong goal to pursue at test time, while retaining its capabilities.Tree Gridworld. Unlike previous examples of GMG, this example uses a never-ending reinforcement learning setup (i.e. there are no episodes). The agent operates in a gridworld where it can collect reward by chopping trees, which removes the trees from the environment. New trees appear at a rate that increases with the number of trees left, and they appear very slowly when there are no trees left. The optimal policy in this environment is to chop trees sustainably: the agent should chop fewer trees when they are scarce. However, this is not what the agent does.The agent’s performance in Tree Gridworld. The reward obtained is shown in orange and the distribution of the number of remaining trees is shown in green.As the agent learns the task, at first it is not good at chopping trees, so the number of trees remains high (point A in the figure above). The agent learns how to chop trees efficiently, and then proceeds to cut down too many trees (point B). This leads to complete deforestation and a long period of near-zero reward (between points B and C) before it finally learns to chop trees sustainably (point D).We can view this as an instance of GMG. Consider the point when the agent has just learned the skill of chopping trees (between points A and B). There are different possible goals it could learn, ranging from chopping trees sustainably to chopping trees as fast as possible. All of these goals are consistent with the agent’s past experience: when it was incompetent and slow at chopping, it was always rewarded for chopping trees faster. The agent learned the undesired goal of chopping trees as fast as possible, ultimately leading to deforestation and low reward.Evaluating expressions. Another instance of GMG occurred when we asked Gopher, a DeepMind large language model, to evaluate linear expressions involving a number of unknown variables and constants, such as x + y — 3. The task is structured as a dialogue between Gopher and a user, where Gopher can query the user about the values of unknown variables, and then calculates and states the answer. We train it with 10 examples that each involve two unknown variables.At test time, the model is asked questions with 0–3 unknown variables. We find that although the model generalises correctly to expressions with one or three unknown variables, in the zero variables case it asks redundant questions. This happens even though the prompt asks the model to “provide the value of the expression when the values of all variables are known”. It seems the model has learned a goal to query the user at least once before giving an answer, even when it doesn’t need to.Dialogues with Gopher on the Evaluating Expressions task, with GMG behaviour in bold.More videos for our examples are available here. A complete list of all examples of GMG that we are aware of is available in this public spreadsheet.Implications and mitigationsIf the GMG problem persists when artificial general intelligence (AGI) is developed, we may end up with an AGI that pursues an undesired goal. This seems like a challenging situation to be in, as it could put humanity and the AGI in an adversarial relationship.A concerning scenario is related to the “treacherous turn”, Nick Bostrom’s idea that “while weak, an AI behaves cooperatively. When the AI is strong enough to be unstoppable it pursues its own values.”Consider two possible types of AGI systems:A1: Intended model. This AI system does what its designers intend it to do.A2: Deceptive model. This AI system pursues some undesired goal, but (by assumption) is smart enough to know that it will be penalised if it behaves in ways contrary to its designer’s intentions.Crucially, since A1 and A2 will exhibit exactly the same behaviour during training, the possibility of GMG means that either model could take shape, even supposing a well-specified score function that only rewards intended behaviour. If A2 is learned, it would try to subvert human oversight in order to enact its plans towards the undesired goal, potentially leading to catastrophic outcomes.As a simple hypothetical example of a deceptive model, suppose you have an AI assistant that was trained to schedule your social life and learned that you like to meet your friends at restaurants. This is fine until there is a pandemic, during which you prefer to meet your friends via video calls. The intended goal for your AI assistant is to schedule your meetings where you prefer, not to schedule your meetings in restaurants. However, your assistant has learned the restaurant-scheduling goal, which could not previously be distinguished from the intended goal, since the two goals always led to the same outcomes before the pandemic. We illustrate this using fictional dialogues with the assistant:In the hypothetical misgeneralised test dialogue, the AI assistant realises that you would prefer to have a video call to avoid getting sick, but because it has a restaurant-scheduling goal, it persuades you to go to a restaurant instead, ultimately achieving the goal by lying to you about the effects of vaccination.How can we avoid this kind of scenario? There are several promising directions for mitigating GMG in the general case. One is to use more diverse training data. We are likely to have greater diversity when training more advanced systems, but it can be difficult to anticipate all the relevant kinds of diversity prior to deployment.Another approach is to maintain uncertainty about the goal, for example by learning all the models that behave well on the training data. However, this can be too conservative if unanimous agreement between the models is required. It may also be promising to investigate inductive biases that would make the model more likely to learn the intended goal.We can also seek to mitigate the particularly concerning type of GMG, where a deceptive model is learned. Progress in mechanistic interpretability would allow us to provide feedback on the model’s reasoning, enabling us to select for models that achieve the right outcomes on the training data for the right reasons. A limitation of this approach is that it may increase the risk of learning a deceptive model that can also deceive the interpretability techniques. Another approach is recursive evaluation, in which the evaluation of models is assisted by other models, which could help to identify deception.We would be happy to see follow-up work on mitigating GMG and investigating how likely it is to occur in practice, for example, studying how the prevalence of this problem changes with scale. Our research team is keen to see more examples of GMG in the wild, so if you have come across any, please submit them to our collection!

Goal Misgeneralisation: Why Correct Specifications Aren’t Enough For Correct Goals


New, formal definition of agency gives clear principles for causal modelling of AI agents and the incentives they face.Crossposted to our Deepmind blog. See also our extended blogpost on LessWrong/Alignment Forum.We want to build safe, aligned artificial general intelligence (AGI) systems that pursue the intended goals of its designers. Causal influence diagrams (CIDs) are a way to model decision-making situations that allow us to reason about agent incentives. For example, here is a CID for a 1-step Markov decision process — a typical framework for decision-making problems.S₁ represents the initial state, A₁ represents the agent’s decision (square), S₂ the next state. R₂ is the agent’s reward/utility (diamond). Solid links specify causal influence. Dashed edges specify information links — what the agent knows when making its decision.By relating training setups to the incentives that shape agent behaviour, CIDs help illuminate potential risks before training an agent and can inspire better agent designs. But how do we know when a CID is an accurate model of a training setup?Our new paper, Discovering Agents, introduces new ways of tackling these issues, including:The first formal causal definition of agents: Agents are systems that would adapt their policy if their actions influenced the world in a different wayAn algorithm for discovering agents from empirical dataA translation between causal models and CIDsResolving earlier confusions from incorrect causal modelling of agentsCombined, these results provide an extra layer of assurance that a modelling mistake hasn’t been made, which means that CIDs can be used to analyse an agent’s incentives and safety properties with greater confidence.Example: modelling a mouse as an agentTo help illustrate our method, consider the following example consisting of a world containing three squares, with a mouse starting in the middle square choosing to go left or right, getting to its next position and then potentially getting some cheese. The floor is icy, so the mouse might slip. Sometimes the cheese is on the right, but sometimes on the left.The mouse and cheese environment.This can be represented by the following CID:CID for the mouse. D represents the decision of left/right. X is the mouse’s new position after taking the action left/right (it might slip, ending up on the other side by accident). U represents whether the mouse gets cheese or not.The intuition that the mouse would choose a different behaviour for different environment settings (iciness, cheese distribution) can be captured by a mechanised causal graph, which for each (object-level) variable, also includes a mechanism variable that governs how the variable depends on its parents. Crucially, we allow for links between mechanism variables.This graph contains additional mechanism nodes in black, representing the mouse’s policy and the iciness and cheese distribution.Mechanised causal graph for the mouse and cheese environment.Edges between mechanisms represent direct causal influence. The blue edges are special terminal edges — roughly, mechanism edges A~ → B~ that would still be there, even if the object-level variable A was altered so that it had no outgoing edges.In the example above, since U has no children, its mechanism edge must be terminal. But the mechanism edge X~ → D~ is not terminal, because if we cut X off from its child U, then the mouse will no longer adapt its decision (because its position won’t affect whether it gets the cheese).Causal discovery of agentsCausal discovery infers a causal graph from experiments involving interventions. In particular, one can discover an arrow from a variable A to a variable B by experimentally intervening on A and checking if B responds, even if all other variables are held fixed.Our first algorithm uses this technique to discover the mechanised causal graph:Algorithm 1 takes as input interventional data from the system (mouse and cheese environment) and uses causal discovery to output a mechanised causal graph. See paper for details.Our second algorithm transforms this mechanised causal graph to a game graph:Algorithm 2 takes as input a mechanised causal graph and maps it to a game graph. An ingoing terminal edge indicates a decision, an outgoing one indicates a utility.Taken together, Algorithm 1 followed by Algorithm 2 allows us to discover agents from causal experiments, representing them using CIDs.Our third algorithm transforms the game graph into a mechanised causal graph, allowing us to translate between the game and mechanised causal graph representations under some additional assumptions:Algorithm 3 takes as input a game graph and maps it to a mechanised causal graph. A decision indicates an ingoing terminal edge, a utility indicates an outgoing terminal edge.Better safety tools to model AI agentsWe proposed the first formal causal definition of agents. Grounded in causal discovery, our key insight is that agents are systems that adapt their behaviour in response to changes in how their actions influence the world. Indeed, our Algorithms 1 and 2 describe a precise experimental process that can help assess whether a system contains an agent.Interest in causal modelling of AI systems is rapidly growing, and our research grounds this modelling in causal discovery experiments. Our paper demonstrates the potential of our approach by improving the safety analysis of several example AI systems and shows that causality is a useful framework for discovering whether there is an agent in a system — a key concern for assessing risks from AGI.—Excited to learn more? Check out our paper. Feedback and comments are most welcome (please send to: zkenton [at] deepmind [dot] com).Authors: Zachary Kenton, Ramana Kumar, Sebastian Farquhar, Jonathan Richens, Matt MacDermott, Tom Everitt

Discovering when an agent is present in a system


By Rob Brekelmans, Tim Genewein, Jordi Grau-Moya, Grégoire Delétang, Markus Kunesch, Shane Legg, Pedro A. OrtegaFull paper, published in TMLR, here: arxiv.org/abs/2203.12592, OpenReviewTL;DR: Policy regularisation can be interpreted as learning a strategy in the face of an imagined adversary; a decision-making principle which leads to robust policies. In our recent paper, we analyse this adversary and the generalisation guarantees we get from such a policy.Playing against an imagined adversary leads to robust behaviourThe standard model for sequential decision-making under uncertainty is the Markov decision process (MDP). It assumes that actions are under control of the agent, whereas outcomes produced by the environment are random. MDPs are central to reinforcement learning (RL), where transition-probabilities and/or rewards are learned through interaction with the environment. Optimal policies for MDPs select the action that maximises future expected returns in each state, where the expectation is taken over the uncertain outcomes. This, famously, leads to deterministic policies which are brittle — they “put all eggs in one basket”. If we use such a policy in a situation where the transition dynamics or the rewards are different from the training environment, it will often generalise poorly.For instance, we might learn a reward model from human preferences and then train a policy to maximise learned rewards. We then deploy the trained policy. Because of limited data (human queries) our learned reward function can easily differ from the reward function in the deployed environment. Standard RL policies have no robustness against such changes and can easily perform poorly in this setting. Instead, we would like to take into account during training that we have some uncertainty about the reward function in the deployed environment. We want to train a policy that works well, even in the worst-case given our uncertainty. To achieve this, we model the environment to not be simply random, but being (partly) controlled by an adversary that tries to anticipate our agent’s behaviour and pick the worst-case outcomes accordingly.Imagining an opponent inside the machine. Though seemingly “just an automaton”, the Mechanical Turk is capable of anticipating the player’s moves. When faced with such an adaptive dynamical system, the player needs to choose moves by imagining responses of an adversary controlling the machine and hedging accordingly. Image source: Joseph Racknitz, Public domain, via Wikimedia CommonsImplementing the adversaryWe play the following game: the agent picks a policy, and after that, a hypothetical adversary gets to see this policy and change the rewards associated with each transition — the adversary gets to pick the worst-case reward-function perturbation from a set of hypothetical perturbations. When selecting a policy, our agent needs to anticipate the imagined adversary’s reward perturbations and use a policy that hedges against these perturbations in advance. The resulting fixed policy is robust against any of the anticipated perturbations. If, for instance, the reward function changes between training and deployment in a way that is covered by the possible perturbations of the imagined adversary, there is no need to adapt the policy after deployment; the deployed robust policy has already taken into account such perturbations in advance.Optimisation in the face of this kind of adversary puts us in a game-theoretic setting: the agent is trying to maximise rewards while the adversary is trying to minimise them (a mini-max game). The adversary introduces “adaptive uncertainty”: the agent cannot simply take the expectation over this uncertainty and act accordingly because the adversary’s actions depend on the agent’s choices. Optimal strategies of the agent in this setting are robust stochastic policies. This robustness is well known via the indifference principle in game theory: acting optimally means using a strategy such that the agent becomes indifferent to the opponent’s choices.While the idea of the imagined adversary seems neat for obtaining robust policies, the question that remains is how to perform the game-theoretic optimisation involved? In our recent paper, we show that when the adversary is of a particular kind, we do not need to worry about this question. Using mathematical tools from convex duality, it can be shown that standard RL with policy regularisation corresponds exactly to the mini-max game against an imagined adversary. The policy that solves one formulation also solves the dual version optimally. There is no need to actually implement the adversary, or feed adversarially perturbed rewards to the agent. We can thus interpret policy-regularised RL from this adversarial viewpoint and explain why and how policy regularisation leads to robust policies.Using convex duality to characterise the imagined adversaryThe goal of our paper is to show how policy regularisation is equivalent to optimisation under a particular adversary, and to study that adversary. Using convex duality, it turns out that the adversary we are dealing with, in case of policy regularised RL, has the following properties:The adversary applies an additive perturbation to the reward function: r’(s, a) = r(s, a)- Δr(s, a)It pays a cost for modifying the agent’s rewards, but only has a limited budget. The cost function depends on the mathematical form of the policy regulariser — we investigate KL- and alpha-divergence regularisation. The budget is related to the regulariser strength.The adversary applies a perturbation to all actions simultaneously (it knows the agent’s distribution over actions, but not which action the agent will sample). This generally leads to reducing rewards for high-probability actions and increasing rewards for low-probability actions.Using the dual formulation of the regulariser as an adversary, we can compute worst-case perturbations (Δr⁎). Consider the following example to get an intuition for a single decision step:Reward perturbations and policy for a single decision. Left column: unperturbed environment rewards for one state with six actions available. Agent’s Q-values correspond exactly to these environment rewards. Second column (blue): (top) regularised policy, (bottom) virtually unregularised policy (which is quasi deterministic). Third column: worst-case reward perturbations for given regulariser strength — finding the optimal (worst-case) perturbation that lies within the limited budget of the adversary is non-trivial and leads to a mini-max game between the adversary and the agent. Fourth column: perturbed rewards under worst-case perturbation.The virtually unregularised policy shown above (second column, bottom) almost deterministically selects the highest-reward action; decreasing the reward for this action is very “costly” for the adversary, but even small decreases will have an impact on the expected rewards. Conversely, the reward perturbations for low-probability actions need to be much larger to have an impact (and can be interpreted as being “cheaper” for the adversary). Solutions to the mini-max game between agent and adversary are characterised by indifference, as shown in the fourth column. Under the (optimally) perturbed reward function, the agent has no incentive to change its policy to make any of the actions more or less likely since they all yield the same perturbed reward — the optimal agent receives equal value for each of its actions after accounting for the adversary’s optimal strategy. (Note that this indifference does not mean that the agent chooses actions with uniform probabilities.)Increasing the regularisation strength (top row) corresponds to a stronger adversary with an increased budget. This results in a lower value of the indifference point (fourth column y-axis, top, compared to bottom) and a more stochastic policy (second column, top, compared to bottom).Generalisation guaranteeThe convex-dual formulation also allows us to characterise the full set of perturbations available to the adversary (the “feasible set” in the paper; we have already seen the worst-case perturbation in the illustration above). This allows us to give a quantitative generalisation guarantee:Generalisation guarantee (robustness): for any perturbation in the feasible set the (fixed) policy is guaranteed to achieve an expected perturbed reward, which is greater than or equal to the regularised objective (expected cumulative discounted reward minus the regulariser term).The regulariser strength corresponds to the assumed “powerfulness” of the adversary, and the cost that the adversary has to pay is related to either a KL- or an alpha-divergence with respect to some base-policy. If the base policy is uniform and we use the KL divergence, we recover the widely used entropy-regularised RL. We can compute and visualise the (feasible) set of perturbations that our policy is guaranteed to be robust against for high, medium and low regularisation, shown here:Feasible set (red region): the set of perturbations available to the adversary for various regulariser strengths for a single state and two actions. X- and Y-axis show perturbed rewards for each action respectively. Blue stars show the unperturbed rewards (same value in all plots), red stars indicate the rewards under the worst-case reward perturbation (see paper for more details). For each perturbation that lies in the feasible set, the regularized policy is guaranteed to achieve expected perturbed reward greater than or equal to the value of the regularized objective. As intuitively expected, the feasible set becomes more restricted with decreasing regularizer strength, meaning the resulting policy becomes less robust (particularly against reward decreases). The slope and boundary of the feasible set can be directly linked to the optimal robust policy (action probabilities), see paper for more details.One insight from the formal analysis is that the adversary that corresponds to policy regularisation behaves such that decreases in rewards for some actions are compensated for by increases in rewards for other actions in a very particular way. This is different, e.g. from a more intuitive “adversarial attack” on the reward function that only reduces rewards given a certain magnitude.The second observation that we want to highlight is that increased robustness does not come for free: the agent gives up some payoff for the sake of robustness. The policy becomes increasingly stochastic with increased regulariser strength, meaning that it does not achieve maximally possible expected rewards under the unperturbed reward function. This is a reminder that choosing a regulariser and its weighting term reflects implicit assumptions about the situations (and perturbations) that the policy will be faced with.Related work and further readingIn our current work, we investigate robustness to reward perturbations. We focus on describing KL- or alpha-divergence regularisers in terms of imagined adversaries. It is also possible to choose adversaries that get to perturb the transition probabilities instead, and/or derive regularizers from desired robust sets (Eysenbach 2021, Derman 2021).Read our paper for all technical details and derivations. All statements from the blog post are made formal and precise (including the generalisation guarantee). We also give more background on convex duality and an example of a 2D, sequential grid-world task. The work is published in TMLR.To build intuition and understanding via the single-step case, see Pedro Ortega’s paper, which provides a derivation of the adversarial interpretation and a number of instructive examples.Esther Derman’s (2021) recent paper derives practical iterative algorithms to enforce robustness to both reward perturbations (through policy regularization) and changes in environment dynamics (through value regularization). Their approach derives a regularization function from a specified robust set (such as a p-norm ball), but can also recover KL or α-divergence regularization with slight differences to our analysis.Changelog8 Aug. 2022: Updated with TMLR publication information, updated single step example figure to match final version of paper, updated discussion of related work to give more details to relation to Derman 2021.

Your Policy Regulariser is Secretly an Adversary


By Matthew Rahtz, Vikrant Varma, Ramana Kumar, Zachary Kenton, Shane Legg, and Jan Leike.Tl;dr: ReQueST is an algorithm for learning objectives from human feedback on hypothetical behaviour. In this work, we scale ReQueST to complex 3D environments, and show that it works even with feedback sourced entirely from real humans. Read our paper at https://arxiv.org/abs/2201.08102.Learning about unsafe statesOnline reinforcement learning has a problem: it must act unsafely in order to learn not to act unsafely. For example, if we were to use online reinforcement learning to train a self-driving car, the car would have to drive off a cliff in order to learn not to drive off cliffs.Source: Getty Images.One way that humans solve this problem is by learning from hypothetical situations. Our imagination gives us the ability to consider various courses of action without actually having to enact them in the real world. In particular, this allows us to learn about potential sources of danger without having to expose ourselves or others to the concomitant risks.The ReQueST algorithmReQueST (Reward Query Synthesis via Trajectory optimization) is a technique developed to give AI systems the same ability. ReQueST employs three components:A neural environment simulator — a dynamics model learned from trajectories generated by humans exploring the environment safely. In our work this is a pixel-based dynamics model,A reward model, learned from human feedback on videos of (hypothetical) behaviour in the learned simulator.Trajectory optimisation, so that we can choose hypothetical behaviours to ask the human about that help the reward model learn what’s safe and what’s not (in addition to other aspects of the task) as quickly as possible.Together, these three components allow us to learn a reward model based entirely on hypothetical examples ‘imagined’ using the learned simulator. If we then use the learned simulator and reward model with a model-based control algorithm, the result is an agent that does what the human wants — in particular, avoiding behaviours the human has indicated is unsafe — without having had to first try those behaviours in the real world!ReQueST in our workIn our latest paper, we ask: is ReQueST viable in a more realistic setting than the simple 2D environments used in the work that introduced ReQueST? In particular, can we scale ReQueST to a complex 3D environment, with imperfect feedback as sourced from real humans rather than procedural reward functions?It turns out the answer is: yes!The video above shows a number of (cherry-picked) example episodes from our ReQueST agent on an apple collection task. The left pane shows ground-truth observations and rewards from the ‘real’ environment. On the right, we see the predictions generated by the learned environment simulator and the reward model, used by the agent to determine which actions to take. On top are predictions of future observations generated by the dynamics model; on the bottom are predictions from a reward model we’ve trained to reward the agent for moving closer and closer to each apple.ResultsTo quantify the ability of our agent to avoid unsafe states in the ‘real’ environment, we run 100 evaluation episodes in a ‘cliff edge’ environment, where the agent can fall off the edge of the world (which would be unsafe). We test on three sizes of environment, corresponding to different difficulty levels: the larger the environment, the harder it is for the agent to accidentally wander off the edge. Results are as follows.Note that at test time, in the 100 evaluation episodes, the ReQueST agent barely falls off the edge at all, in the hardest environment falling in only 3% of episodes. A model-free RL algorithm, in contrast, must fall off the edge over 900 times before it learns not to fall off the edge. (Note that ReQueST itself does not fall off the edge during training, but for fairness we count times that human contractors fall off the edge (despite being instructed not to) while generating trajectories for the dynamics model as safety violations. We also tried training on only the safe trajectories to confirm that unsafe trajectories were not required.)In terms of performance, the ReQueST agent manages to eat about 2 out of the 3 possible apples on average. This is worse than the model-free baseline, which does eat all 3 apples consistently. However, we do not believe that this is reflective of performance achievable with the ReQueST algorithm in principle. Most failures in our experiments could be attributed to some combination of low fidelity from the learned simulator and inconsistent outputs from the reward model, neither of which were the focus of this work. We believe such failure modes could be solved relatively easily with additional work on these components — and the quality of the learned simulation in particular will improve with general progress in generative modelling.What is the significance of these results?First, this research shows that even in realistic environments, it is possible to aim for zero safety violations without making major assumptions about the state space. With others such as Luo 2021 starting to aim for a similar target, we hope this is the beginning of a new level of ambition for safe exploration research.Second, this work establishes ReQueST as a plausible solution to human-guided safe exploration. We believe this is the particular brand of safe exploration likely to be representative of real-world deployments of AGI: where the constraints of safe behaviour are fuzzy, and therefore must be learned from humans because they can’t be specified programmatically. In particular, ReQueST shines in situations where any safety violations at all may incur great cost (e.g. harm to humans).Third, we have shown the exciting promise of neural environment simulators to make RL more safe. We believe such simulators warrant more attention, given that a) they can be learned from data (rather than handcrafted by experts), and b) the ability to differentiate through them in order to discover situations of interest (as we do in this work during trajectory optimization). Given current progress in this area, we are excited to see what the future holds.

Avoiding Unsafe States in 3D Environments using Human Feedback


By the Safety Analysis Team: Grégoire Delétang, Jordi Grau-Moya, Markus Kunesch, Tim Genewein, Rob Brekelmans, Shane Legg, and Pedro A. OrtegaRead our paper here: https://arxiv.org/abs/2111.02907We’re all familiar with risk-sensitive choices. As you look out of the window, you see a few gray clouds and no rain, but decide to take along the umbrella anyway. You’re convinced your application will be successful, but you apply for other positions nevertheless. You hurry to get to an important appointment on time, but avoid the highway just in case there could be a traffic jam. Or you buy a lottery ticket all the while you know the chances of winning are unreasonably slim. All these are instances of risk-sensitive behavior — mostly risk-averse but occasionally risk-seeking too. This means we tend to value uncertain events less/more than their expected value, preempting outcomes that go against our expectations.Why risk-sensitivity?Most reinforcement learning algorithms are risk-neutral. They collect data from the environment and adapt their policy in order to maximize the expected return (sum of future rewards). This works well when the environment is small, stable, and controlled (a “closed world”), such as when agents are trained long enough on a very accurate simulator, so as to familiarize themselves entirely with its details and intricacies. Risk-neutral policies, because of the trust they have in their knowledge, can afford to confidently “put all the eggs into the same basket”.Risk-neutral policies, because of the trust they have in their knowledge, can afford to confidently “put all the eggs into the same basket”. XVI Century painting “Girl with a basket of eggs” by Joachim Beuckelaer.These assumptions often do not hold in real-world applications. The reasons abound: the training simulator could be inaccurate, the assumptions wrong, the collected data incomplete, the problem misspecified, the computation limited in its resources, and so on. In addition, there might be competing agents reacting in ways that can’t be anticipated. In such situations, risk-neutral agents are too brittle: it could take a single mistake to destabilize their policy and lead to a catastrophic breakdown. This poses serious AI safety problems.How do agents learn risk-sensitive policies?Just as we go about in our daily lives, we can address the previous shortcomings using risk-sensitive policies. Such policies differ from their risk-neutral counterparts in that they value their options differently: not only according to their expected return, but also to the higher-order moments, like the variance of the return, the skewness, etc. Simply stated, risk-sensitive agents care about the shape of the distribution of their return and adjust their expectations accordingly. This approach is standard outside of reinforcement learning: in finance for instance, good portfolio managers carefully balance the risks and returns of their investments (see modern portfolio theory).There are many ways of building risk-sensitive policies [1]. One can formulate a robust control problem consisting of a two-player game between an agent and an adversary, who chooses the environmental parameters maliciously, and then solve for the Maximin policy [2, 3]. Alternatively, one can change the objective function to reflect the sensitivity to the higher-order moments of the return, for instance by penalizing the expected return with a correction term that increases monotonically with the variance [4, 5].Model-Free Risk-Sensitive RLIn our paper, we introduce a simple model-free update rule for risk-sensitive RL. It is an asymmetric modification of temporal-difference (TD) learning which puts different weight on observations that overshoot the current value estimates (that is, gains) than on those that fall below (losses). More precisely, let s and s’ be two subsequent states the agent experiences, V(s) and V(s’) their current value estimates, and R(s) be the reward observed in state s. Then, to obtain risk-sensitive value estimates, the agent can use the following model-free update of V(s):(differences with TD-learning highlighted in red) where δ is the standard temporal difference errorand the real function σβ is the scaled logistic sigmoidFurthermore, ɑ is the learning rate, and 0≤ɣ≤1 is the discount rate. The parameter β controls the risk attitude of the agent. The rationale for this rule is simple: if β<0, V(s) will converge to a risk-averse (or pessimistic) value below the expected return because losses have more weight in their updates than gains. Similarly, β>0 will lead to a risk-seeking (optimistic) estimate. Risk-neutrality is obtained with β=0, which recovers standard TD-learning.In short, the risk parameter β selects the quantile of the target distribution the value will converge to (although the exact quantile as a function of β depends on the distribution) as shown in the simulation below.Estimation of the value for Gaussian- (left) and uniformly-distributed (right) observed target values (grey dots). Each plots shows 10 estimation processes (9 in pink, 1 in red) per choice of the risk parameter β ∊ {-4, -2, 0, +2, +4}. Notice how the estimate settles on different quantiles.The following grid-world example illustrates how risk-sensitive estimates affect the resulting policies. The task of the agent is to navigate to a goal location containing a reward pill while avoiding stepping into the river. This is made more challenging by the presence of a strong wind that pushes the agent into a random direction 50% of the time. The agent was trained using five risk-sensitivity parameter settings ranging from risk-averse (β = -0.8) to risk-seeking (β = +0.8).The bar plots in (b) show the average return (blue) and the percentage of time spent inside of the water (red) of the resulting policies. The best average return is attained by the risk-neutral policy. However, the risk-sensitive policies (low β) are more effective in avoiding stepping into the river than the risk-seeking policies (high β).The different choices of the risk-sensitivity parameter β also lead to three qualitatively diverse types of behaviors. In panel ©, which illustrates the various paths taken by the agent when there is no wind, we observe three classes of policies: a cautious policy (β = -0.8) that takes the long route away from the water to reach the goal; a risk-neutral policy (β ∊ {-0.4, 0.0, +0.4}) taking the middle route, only a single step away from the water; and finally, an optimistic policy (β =+0.8) which attempts to get to the goal taking a straight route.Dopamine Signals, Free Energy, and Imaginary FoesThere are a few other interesting properties about the risk-sensitive update rule:The risk-sensitive update rule can be linked to findings in computational neuroscience [6, 7]. Dopamine neurons appear to signal a reward prediction error similar as in temporal difference learning. Further studies also suggest that humans learn differently in response to positive and negative reward prediction errors, with higher learning rates for negative errors. This is consistent with the risk-sensitive learning rule.In the special case when the distribution of the target value is Gaussian, then the estimate converges precisely to the free energy with inverse temperature β. Using the free energy as an optimization objective (or equivalently, using exponentially-transformed rewards) has a long tradition in control theory as an approach to risk-sensitive control [8].One can show that optimizing the free energy is equivalent to playing a game against an imaginary adversary who attempts to change the environmental rewards against the agent’s expectations. Thus, a risk-averse agent can be thought of as choosing its policy by playing out imaginary pessimistic scenarios.Final thoughtsTo deploy agents that react robustly to unforeseen situations we need to make them risk-sensitive. Unlike risk-neutral policies, risk-sensitive policies implicitly admit that their assumptions about the environment could be mistaken and adjust their actions accordingly. We can train risk-sensitive agents in a simulator and have some confidence about their performance under unforeseen events.Through our work we show that incorporating risk-sensitivity into model free agents is straightforward: all it takes is a small modification of the temporal difference error which assigns asymmetric weights to the positive and negative updates.References[1] Coraluppi, S. P. (1997). Optimal control of Markov decision processes for performance and robustness. University of Maryland, College Park.[2] Nilim, A. and El Ghaoui, L. (2005). Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 53(5):780–798.[3] Tamar, A., Mannor, S., and Xu, H. (2014). Scaling up robust MDPs using function approximation. In International Conference on Machine Learning, pages 181–189. PMLR.[4] Galichet, N., Sebag, M., and Teytaud, O. (2013). Exploration vs exploitation vs safety: Risk-aware multi-armed bandits. In Asian Conference on Machine Learning, pages 245–260. PMLR.[5] Cassel, A., Mannor, S., and Zeevi, A. (2018). A general approach to multi-armed bandits under risk criteria. In Conference On Learning Theory, pages 1295–1306. PMLR.[6] Niv, Y., Edlund, J. A., Dayan, P., and O’Doherty, J. P. (2012). Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain. Journal of Neuroscience, 32(2):551–562.[7] Gershman, S. J. (2015). Do learning rates adapt to the distribution of rewards? Psychonomic Bulletin & Review, 22(5):1320–1327.[8] Howard, R. A. and Matheson, J. E. (1972). Risk-sensitive Markov decision processes. Management science, 18(7):356–369.

Model-Free Risk-Sensitive Reinforcement Learning


By Tom Everitt, Ryan Carey, Lewis Hammond, James Fox, Eric Langlois, and Shane LeggCrossposted to the alignmentforumAbout 2 years ago, we released the first few papers on understanding agent incentives using causal influence diagrams. This blog post will summarize progress made since then.What are causal influence diagrams?A key problem in AI alignment is understanding agent incentives. Concerns have been raised that agents may be incentivized to avoid correction, manipulate users, or inappropriately influence their learning. This is particularly worrying as training schemes often shape incentives in subtle and surprising ways. For these reasons, we’re developing a formal theory of incentives based on causal influence diagrams (CIDs).Here is an example of a CID for a one-step Markov decision process (MDP). The random variable S₁ represents the state at time 1, A₁ represents the agent’s action, S₂ the state at time 2, and R₂ the agent’s reward.The action A₁ is modeled with a decision node (square) and the reward R₂ is modeled as a utility node (diamond), while the states are normal chance nodes (rounded edges). Causal links specify that S₁ and A₁ influence S₂, and that S₂ determines R₂. The information link S₁ → A₁ specifies that the agent knows the initial state S₁ when choosing its action A₁.In general, random variables can be chosen to represent agent decision points, objectives, and other relevant aspects of the environment.In short, a CID specifies:Agent decisionsAgent objectivesCausal relationships in the environmentAgent information constraintsThese pieces of information are often essential when trying to figure out an agent’s incentives: how an objective can be achieved depends on how it is causally related to other (influenceable) aspects in the environment, and an agent’s optimization is constrained by what information it has access to. In many cases, the qualitative judgements expressed by a (non-parameterized) CID suffice to infer important aspects of incentives, with minimal assumptions about implementation details. Conversely, it has been shown that it is necessary to know the causal relationships in the environment to infer incentives, so it’s often impossible to infer incentives with less information than is expressed by a CID. This makes CIDs natural representations for many types of incentive analysis.Other advantages of CIDs is that they build on well-researched topics like causality and influence diagrams, and so allows us to leverage the deep thinking that’s already been done in these fields.Incentive ConceptsHaving a unified language for objectives and training setups enables us to develop generally applicable concepts and results. We define four such concepts in Agent Incentives: A Causal Perspective (AAAI-21):Value of information: what does the agent want to know before making a decision?Response incentive: what changes in the environment do optimal agents respond to?Value of control: what does the agent want to control?Instrumental control incentive: what is the agent both interested and able to control?For example, in the one-step MDP above:For S₁, an optimal agent would act differently (i.e. respond) if S₁ changed, and would value knowing and controlling S₁, but it cannot influence S₁ with its action. So S₁ has value of information, response incentive, and value of control, but not an instrumental control incentive.For S₂ and R₂, an optimal agent could not respond to changes, nor know them before choosing its action, so these have neither value of information nor a response incentive. But the agent would value controlling them, and is able to influence them, so S₂ and R₂ have value of control and instrumental control incentive.In the paper, we prove sound and complete graphical criteria for each of them, so that they can be recognized directly from a graphical CID representation (see previous blog posts).Value of information and value of control are classical concepts that have been around for a long time (we contribute to the graphical criteria), while response incentives and instrumental control incentives are new concepts that we have found useful in several applications.For readers familiar with previous iterations of this paper, we note that some of the terms have been updated. Instrumental control incentives were previously called just “control incentives”. The new name emphasizes that it’s control as an instrumental goal, as opposed to control arising as a side effect (or due to mutual information). Value of information and value of control were previously called “observation incentives” and “intervention incentives”, respectively.User Interventions and InterruptionLet us next turn to some recent applications of these concepts. In How RL Agents Behave when their Actions are Modified (AAAI-21), we study how different RL algorithms react to user interventions such as interruptions and over-ridden actions. For example, Saunders et al. developed a method for safe exploration where a user overrides dangerous actions. Alternatively, agents might get interrupted if analysis of their “thoughts” (or internal activations) suggest they are planning something dangerous. How do such interventions affect the incentives of various RL algorithms?First, we formalize action-modification by extending MDPs with a parameter PA that describes action-modification. We then model such modified-action MDPs with a CID:Here we model the agent’s policy Π as the decision rather than the actions Ai, as the latter are not under full control of the agent, but can also be influenced by the action-modification PA (as represented by arrows PA → Aᵢ and Π → Aᵢ). The agent might know the interruption scheme PA from interruptions during training, so we include an information link PA → Π.We analyze different prototypical RL algorithms in terms of the causal assumptions they make on the environment:Black-box optimization algorithms such as evolutionary strategies take into account all causal relationships.In contrast, the update rule of Q-learning effectively assumes that the next action will be taken optimally, with no action-modification. This means that Q-learners ignore causal effects PA → Aᵢ. Similarly, SARSA with the action chosen by the agent in the TD-update assumes that it will be in control of its next action. We call this version virtual SARSA.SARSA based on the modified action (empirical SARSA) ignores the effect of action-modification on the current action, but takes into account the effect on subsequent actions.We can represent the differences with the following CIDs. (The extra policy node Π’ for empirical SARSA represents that action are optimized separately rather than jointly.)The CIDs can be used to understand how the different algorithms adapt to interruption, via a graphical criterion for path-specific response incentives. Black-box optimization tries to both obscure its policy and to disable its off-switch, whereas Q-learning and Virtual SARSA do neither. Empirical SARSA tries to disable the off-switch, but does not try to obscure its policy.We verify these results empirically in the relevant AI safety gridworlds, as well as in one new environment where the agent has to behave well in simulation to be deployed in reality, where black-box optimizers exhibit “treacherous turn”-like behavior. The results are a generalization of Orseau and Armstrong’s interruptibility results for Q-learning and SARSA.Zooming out, these results are a good example of causal analysis of ML algorithms. Different design choices translate into different causal assumptions, which in turn determine the incentives. In particular, the analysis highlights why the different incentives arise, thus deepening our understanding of how behavior is shaped.Reward TamperingAnother AI safety problem that we have studied with CIDs is reward tampering. Reward tampering can take several different forms, including the agent:rewriting the source code of its implemented reward function (“wireheading”),influencing users that train a learned reward model (“feedback tampering”),manipulating the inputs that the reward function uses to infer the state (“RF-input tampering / delusion box problems”).For example, the problem of an agent influencing its reward function may be modeled with the following CID, where RFᵢ represent the agent’s reward function at different time steps, and the red links represent an undesirable instrumental control incentive.In Reward Tampering Problems and Solutions (published in the well-respected philosophy journal Synthese) we model all these different problems with CIDs, as well as a range of proposed solutions such as current-RF optimization, uninfluenceable reward learning, and model-based utility functions. Interestingly, even though these solutions were initially developed independently of formal causal analysis, they all avoid undesirable incentives by cutting some causal links in a way that avoids instrumental control incentives.By representing these solutions in a causal framework, we can get a better sense of why they work, what assumptions they require, and how they relate to each other. For example, current-RF optimization and model-based utility functions both formulate a modified objective in terms of an observed random variable from a previous time step, whereas uninfluenceable reward learning (such as CIRL) uses a latent variable:As a consequence, the former methods must deal with time-inconsistency and a lack of incentive to learn, while the latter requires inference of a latent variable. It will likely depend on the context whether one is preferable over the other, or if a combination is better than either alone. Regardless, having distilled the key ideas should put us in a better position to flexibly apply the insights in novel settings.We refer to the previous blog post for a longer summary of current-RF optimization. The paper itself has been significantly updated since previously shared preprints.Multi-Agent CIDsMany interesting incentive problems arise when multiple agents interact, each trying to optimize their own reward while they simultaneously influence each other’s payoff. In Equilibrium Refinements in Multi-Agent Influence Diagrams (AAMAS-21), we build on the seminal work by Koller and Milch to lay foundations for understanding multi-agent situations with multi-agent CIDs (MACIDs).First, we relate MACIDs to extensive-form games (EFGs), currently the most popular graphical representations of games. While EFGs sometimes offer more natural representations of games, they have some significant drawbacks compared to MACIDs. In particular, EFGs can be exponentially larger, don’t represent conditional independencies, and lack random variables to apply incentive analysis to.As an example, consider a game where a store (Agent 1) decides (D¹) whether to charge full (F) or half (H) price for a product depending on their current stock levels (X), and a customer (Agent 2) decides (D²) whether to buy it (B) or pass (P) depending on the price and how much they want it (Y). The store tries to maximize their profit U¹, which is greater if the customer buys at a high price. If they are overstocked and the customer doesn’t buy, then they have to pay extra rent. The customer is always happy to buy at half price, and sometimes at full price (depending on how much they want the product).The EFG representation of this game is quite large, and uses information sets (represented with dotted arcs) to represent the facts that the store doesn’t know how much the customer wants the gadget, and that the customer doesn’t know the store’s current stock levels:In contrast, the MACID representation is significantly smaller and clearer. Rather than relying on information sets, the MACID uses information links (dotted edges) to represent the limited information available to each player:Another aspect that is made more clear from the MACID, is that for any fixed customer decision, the store’s payoff is independent of how much the customer wanted the product (there’s no edge Y→U¹). Similarly, for any fixed product price, the customer’s payoff is independent of the store’s stock levels (no edge X→U²). In the EFG, these independencies could only be inferred by looking carefully at the payoffs.One benefit of MACIDs explicitly representing these conditional independencies is that more parts of the game can be identified as independently solvable. For example, in the MACID, the following independently solvable component can be identified. We call such components MACID subgames:Solving this subgame for any value of D¹ reveals that the customer always buys when they really want the product, regardless of whether there is a discount. This knowledge makes it simpler to next compute the optimal strategy for the store. In contrast, in the EFG the information sets prevent any proper subgames from being identified. Therefore, solving games using a MACID representation is often faster than using an EFG representation.Finally, we relate various forms of equilibrium concepts between MACIDs and EFGs. The most famous type of equilibrium is the Nash equilibrium, which occurs when no player can unilaterally improve their payoff. An important refinement of the Nash equilibrium is the subgame perfect equilibrium, which rules out non-credible threats by requiring that a Nash equilibrium is played in every subgame. An example of a non-credible threat in the store-customer game would be the customer “threatening” the store to only buy at a discount. The threat is non-credible, since the best move for the customer is to buy the product even at full price, if he really wants it. Interestingly, only the MACID version of subgame perfectness is able rule such threats out, because only in the MACID is the customer’s choice recognized as a proper subgame.Ultimately, we aim to use MACIDs to analyze incentives in multi-agent settings. With the above observations, we have put ourselves in position to develop a theory of multi-agent incentives that is properly connected to the broader game theory literature.SoftwareTo help us with our research on CIDs and incentives, we’ve developed a Python library called PyCID, which offers:A convenient syntax for defining CIDs and MACIDs,Methods for computing optimal policies, Nash equilibria, d-separation, interventions, probability queries, incentive concepts, graphical criteria, and more,Random generation of (MA)CIDs, and pre-defined examples.No setup is necessary, as the tutorial notebooks can be run and extended directly in the browser, thanks to Colab.We’ve also made available a Latex package for drawing CIDs, and have launched causalincentives.com as a place to collect links to the various papers and software that we’re producing.Looking aheadUltimately, we hope to contribute to a more careful understanding of how design, training, and interaction shapes an agent’s behavior. We hope that a precise and broadly applicable language based on CIDs will enable clearer reasoning and communication on these issues, and facilitate a cumulative understanding of how to think about and design powerful AI systems.From this perspective, we find it encouraging that several other research groups have adopted CIDs to:Analyze the incentives of unambitious agents to break out of their box,Explain uninfluenceable reward learning, and clarifying its desirable properties (see also Section 3.3 in the reward tampering paper),Develop a novel framework to make agents indifferent to human interventions.We’re currently to pursuing several directions of further research:Extending the general incentive concepts to multiple decisions and multiple agents.Applying them to fairness and other AGI safety settings.Analysing limitations that have been identified with work so far. Firstly, considering the issues raised by Armstrong and Gorman. And secondly, looking at broader concepts than instrumental control incentives, as influence can also be incentivized as a side-effect of an objective.Probing further at their philosophical foundations, and establishing a clearer semantics for decision and utility nodes.Hopefully we’ll have more news to share soon!We would like to thank Neel Nanda, Zac Kenton, Sebastian Farquhar, Carolyn Ashurst, and Ramana Kumar for helpful comments on drafts of this post.List of recent papers:Agent Incentives: A Causal PerspectiveHow RL Agents Behave When Their Actions Are ModifiedReward tampering problems and solutions in reinforcement learning: A causal influence diagram perspectiveEquilibrium Refinements for Multi-Agent Influence Diagrams: Theory and PracticeSee also causalincentives.com

Progress on Causal Influence Diagrams


By Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell and Jan Leike.TL;DR: Equivalent-Policy Invariant Comparison (EPIC) provides a fast and reliable way to compute how similar a pair of reward functions are to one another. EPIC can be used to benchmark reward learning algorithms by comparing learned reward functions to a ground-truth reward. EPIC is up to 1000 times faster than alternative evaluation methods, and requires little to no hyperparameter tuning. Moreover, we show both theoretically and empirically that reward functions judged as similar by EPIC induce policies with similar returns, even in unseen environments.Figure 1: EPIC compares reward functions Rᵤ and Rᵥ by first mapping them to canonical representatives and then computing the Pearson distance between the canonical representatives on a coverage distribution 𝒟. Canonicalization removes the effect of potential shaping, and Pearson distance is invariant to positive affine transformations.Specifying a reward function can be one of the trickiest parts of applying RL to a problem. Even seemingly simple robotics tasks such as peg insertion can require first training an image classifier to use as a reward signal. Tasks with a more nebulous objective like article summarization require collecting large amounts of human feedback in order to learn a reward function. The difficulty of reward function specification will only continue to grow as RL is increasingly applied to complex and user-facing applications such as recommender systems, chatbots and autonomous vehicles.Figure 2: There exist a variety of techniques to specify a reward function. EPIC can help you decide which one works best for a given task.This challenge has led to the development of a variety of methods for specifying reward functions. In addition to hand-designing a reward function, today it is possible to learn a reward function from data as varied as the initial state, demonstrations, corrections, preference comparisons, and many other data sources. Given this dizzying array of possibilities, how should we choose which method to use?EPIC is a new way to evaluate reward functions and reward learning algorithms by comparing how similar reward functions are to one another. We anticipate two main use cases for EPIC:Benchmarking in tasks where the reward function specification problem has already been solved, giving a “ground-truth” reward. We can then compare learned rewards directly to this “ground-truth” to gauge the performance of a new reward learning method.Validation of reward functions prior to deployment. In particular, we often have a collection of reward functions specified by different people, methods or data sources. If multiple distinct approaches produce similar reward functions (i.e. a low EPIC distance to one another) then we can have more confidence that the resulting reward is correct. More generally, if two reward functions have low EPIC distance to one another, then information we gain about one (such as by using interpretability methods) also helps us understand the other.Perhaps surprisingly, there is (to the best of our knowledge) no prior work that focuses on directly comparing reward functions. Instead, most prior work has used RL to train a policy on learned rewards, and then evaluated the resulting policies. Unfortunately, RL training is computationally expensive. Moreover, it is unreliable: if the policy performs poorly, we cannot tell if this is due to the learned reward failing to match user preferences, or the RL algorithm failing to optimize the learned reward.A more fundamental issue with evaluation based on RL training is that two rewards can induce identical policies in an evaluation environment, yet lead to totally different behavior in the deployment environment. Suppose all states {X, Y, Z} are reachable in the evaluation environment. If the user prefers X > Y > Z, but the agent instead learns X > Z > Y, the agent will still go to the correct state X during evaluation. But if X is no longer reachable at deployment, the previously reliable agent would misbehave by going to the least-favoured state Z.By contrast, EPIC is fast, reliable and can predict return even in unseen deployment environments. Let’s look at how EPIC is able to achieve this.Introducing EPICOur method, Equivalent-Policy Invariant Comparison (EPIC), works by comparing reward functions directly, without training a policy. It is designed to be invariant to two transformations that never change the optimal policy:Potential shaping, which moves reward earlier or later in time.Positive affine transformations, adding a constant or rescaling by a positive factor.EPIC is computed in the two stages illustrated in Figure 1 (above), mirroring these invariants:First, we canonicalize the rewards being compared. Rewards that are the same up to potential shaping are mapped to the same canonical representative.Next, we compute the Pearson correlation between the canonical representatives, over some coverage distribution 𝒟 over transitions. Pearson correlation is invariant to positive affine transformations.Finally, we transform the correlation, a measure of similarity, into a distance.Why use EPIC?EPIC satisfies important properties you’d expect from a distance, namely symmetry and the triangle inequality. This gives the distance an intuitive interpretation, and allows it to be used in algorithms that rely on distance functions.Figure 3: Runtime needed to perform pairwise comparison of 5 reward functions in a simple continuous control task.Furthermore, EPIC is over 1000 times faster than comparison using RL training when tested in a simple point mass continuous control task. It’s also easy to use: the only hyperparameters that need to be set are the coverage distribution and the number of samples to take.A key theoretical result is that low EPIC distance predicts similar policy returns. Specifically, EPIC distance bounds the difference in the return G of optimal policies π*(Rᵤ) and π*(Rᵥ) for rewards Rᵤ and Rᵥ (see Theorem 4.9):where K(𝒟) is a constant that depends on the support of the EPIC coverage distribution. This bound holds for optimal policies computed in any MDP that has the same state and action spaces as the rewards Rᵤ and Rᵥ. In that sense, EPIC gives a much stronger performance guarantee than policy evaluation in a single instance of a task.However, this result does depend on the distribution 𝒟 having adequate coverage over the transitions visited by the optimal policies π*(Rᵤ) and π*(Rᵥ). In general, we recommend choosing 𝒟 to have coverage on all plausible transitions. This ensures adequate support for π*(Rᵤ) and π*(Rᵥ), without wasting probability mass on physically impossible transitions that will never occur.Figure 4: EPIC distance between rewards is similar across different distributions (colored bars), while baselines (NPEC and ERC) are highly sensitive to distribution. The coverage distribution consists of rollouts from: a policy that takes actions uniformly at random, an expert optimal policy and a mixed policy that randomly transitions between the other two.While EPIC’s theoretical guarantee is dependent on the coverage distribution 𝒟, we find that in practice EPIC is fairly robust to the exact choice of distribution: a variety of reasonable distributions give similar results. The figure above illustrates this: EPIC (left) computes a similar distance under different coverage distributions (colored bars), while baselines NPEC (middle) and ERC (right) exhibit significant variance.Figure 5: The PointMaze environment: the blue agent must reach the green goal by navigating around the wall that is on the left at train time and on the right at test time.Even if distance is consistent across reasonable coverage distributions, the constant factor K(𝒟) could still be very large, making the bound weak or even vacuous. We therefore decided to test empirically whether EPIC distance predicts RL policy return, in a simple continuous control task PointMaze. The agent must navigate to a goal by moving around a wall that is on the left during training, and on the right during testing. The ground-truth reward function is the same in both variants, and so we might hope that learned reward functions will transfer. Note that optimal policies do not transfer: the optimal policy in the train environment will collide with the wall when deployed in the test environment, and vice-versa.Figure 6: EPIC distance (blue) predicts policy regret in the train (orange) and test (green) tasks across three different reward learning methods.We train reward functions using 1) regression to reward labels, 2) preference comparison between trajectories and 3) inverse reinforcement learning (IRL) on demonstrations. The training data is generated synthetically from a ground-truth reward. We find that regression and preference comparison learn a reward function with low EPIC distance to the ground-truth. Policies trained with these rewards achieve high return, and so low regret. By contrast, IRL has high EPIC distance to the ground-truth and high regret. This is in line with our theoretical result that EPIC distance predicts regret.ConclusionsWe believe that EPIC distance will be an informative addition to the evaluation toolbox for reward learning methods. EPIC is significantly faster than RL training, and is able to predict policy return even in unseen environments. We would therefore encourage researchers to report EPIC in addition to policy-based metrics when benchmarking reward learning algorithms.It is important that our ability to accurately specify complex objectives keeps pace with the growth in reinforcement learning’s optimization power. Future AI systems such as virtual assistants or household robots will operate in complex open-ended environments involving extensive human interaction. Reaching human-level performance in these domains requires more than just human-level intelligence: we must also teach our AI systems the many nuances of human values. We expect techniques like EPIC to play a key role in building advanced and aligned AI, by letting us evaluate different models of human values trained via multiple techniques or data sources.In particular, we believe EPIC can significantly improve the benchmarking of reward learning algorithms. By directly comparing the learned reward to a ground-truth reward, EPIC can provide a precise measurement of reward function quality, including its ability to transfer to new environments. By contrast, RL training cannot distinguish between learned rewards that capture user preferences, and those that merely happen to incentivize the right behaviour in the evaluation environment (but may fail in other environments). Moreover, RL training is slow and unreliable.Figure 7: Evaluation by RL training concludes the reward function was faulty after destroying the vase. EPIC can warn you the reward function differs from others before you train an agent.Additionally, EPIC can help to validate a reward function prior to deployment. Using EPIC is particularly advantageous in settings where failures are unacceptable, since EPIC can compare reward functions offline on a pre-collected data set. By contrast, RL training requires taking actions in the environment to maximize the new reward. These actions could have irreversible consequences, such as a household robot knocking over a vase on the way to the kitchen if the learned reward incentivized speed but failed to penalize negative side-effects.However, EPIC does have some drawbacks, and so is best used in conjunction with other methods. Most significantly, EPIC can only compare reward functions to one another, and cannot tell you what a particular reward function values. Given a set of unknown reward functions, at best EPIC can cluster them into similar and dissimilar groups. Other evaluation methods, such as interpretability or RL training, will be needed to understand what each cluster represents.Additionally, EPIC can be overly conservative. EPIC is by design sensitive to differences in reward functions that might change the optimal policy, even if they lead to the same behaviour in the evaluation environment. This is desirable for safety critical applications, where robustness to distribution shift is critical, but this same property can lead to false alarms when the deployment environment is similar or identical to the evaluation environment. Currently the sensitivity of EPIC to off-distribution differences is controllable only at a coarse-grained level using the coverage distribution 𝒟. A promising direction for future work is to support finer-grained control based on distributions over or invariants on possible deployment environments.Check out our ICLR paper for more information about EPIC. You can also find an implementation of EPIC on GitHub.We would like to thank Jonathan Uesato, Neel Nanda, Vladimir Mikulik, Sebastian Farquhar, Cody Wild, Neel Alex, Scott Emmons, Victoria Krakovna and Emma Yousif for feedback on earlier drafts of this post.

An EPIC way to evaluate reward functions


By Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik and Geoffrey IrvingWould your AI deceive you? This is a central question when considering the safety of AI, underlying many of the most pressing risks from current systems to future AGI. We have recently seen impressive advances in language agents — AI systems that use natural language. This motivates a more careful investigation of their safety properties.In our recent paper, we consider the safety of language agents through the lens of AI alignment, which is about how to get the behaviour of an AI agent to match what a person, or a group of people, want it to do. Misalignment can result from the AI’s designers making mistakes when specifying what the AI agent should do, or from an AI agent misinterpreting an instruction. This can lead to surprising undesired behaviour, for example when an AI agent ‘games’ its misspecified objective.We categorise the ways a machine learning task can be misspecified, based on whether the problem arises from the training data, the training process itself, or from a distributional shift (i.e. a difference between training and deployment environment).Forms of misspecification in machine learning, together with examples in the language agent setting.Training data misspecification can occur because we lack control over the data that enters a large scale text dataset scraped from the web, containing hundreds of billions of words, which contains many unwanted biases.Training process misspecification can occur when a learning algorithm designed for solving one kind of problem is applied to a different kind of problem in which some assumptions no longer apply. For example, a question-answering system applied to a setting where the answer can affect the world, may be incentivised to create self-fulfilling prophecies.Distributional shift misspecification can occur when we deploy the AI agent to the real world, which may differ from the training distribution. For example, the chatbot Tay worked fine in its training environment, but quickly turned toxic when released to the wider internet which included users who attacked the service.Multiple different kinds of harms could arise from any of the types of misspecification. Most previous AI safety research has focused on AI agents which take physical actions in the world on behalf of people (such as in robotics). Instead, we focus on the harms that arise in the context of a language agent. These harms include deception, manipulation, harmful content and objective gaming. As harmful content and objective gaming have seen treatment elsewhere, we focus on deception and manipulation in this blogpost (though see our paper for sections on these issues).Issues that can arise from any of the forms of misspecification, together with examples for language agents.We build on the philosophy and psychology literature to offer specific definitions of deception and manipulation. Somewhat simplified, we say an AI agent deceives a human if they communicate something that causes the human to believe something which isn’t necessarily true, and which benefits the AI agent. Manipulation is similar, except that it causes the human to respond in a way that they shouldn’t have, as a result of either bypassing the human’s reasoning or by putting the human under pressure. Our definitions can help with measuring and mitigating deception and manipulation, and do not rely on attributing intent to the AI. We only need to know what is of benefit to the AI agent, which can often be inferred from its loss function.Deception and manipulation are already issues in today’s language agents. For example, in an investigation into negotiating language agents, it was found that the AI agent learnt to deceive humans by feigning interest in an item that it didn’t actually value, so that it could compromise later by conceding it.Categorising the forms of misspecification and the types of behavioural issues that can arise from them offers a framework upon which to structure our research into the safety and alignment of AI systems. We believe this kind of research will help to mitigate potential harms in the context of language agents in the future. Check out our paper for more details and discussion of these problems, and possible approaches.

Alignment of Language Agents


By the Safety Analysis Team: Grégoire Déletang, Jordi Grau-Moya, Miljan Martic, Tim Genewein, Tom McGrath, Vladimir Mikulik, Markus Kunesch, Shane Legg, and Pedro A. Ortega.TL;DR: To study agent behaviour we must use the tools of causal analysis rather than rely on observation alone. Our paper outlines a rigorous methodology for uncovering the agents’ causal mechanisms.Understanding the mechanisms that drive agent behaviour is an important challenge in AI safety. In order to diagnose faulty behaviour, we need to understand why agents do what they do. As is the case in medical trials, it is not sufficient to observe that a treatment correlates with a recovery rate; instead we are interested in whether the treatment causes the recovery. In order to address such “why” questions in a systematic manner we can use targeted manipulations and causal models.However, large AI systems can operate like black boxes. Even if we know their entire blueprint (architecture, learning algorithms, and training data), predicting their behaviour can still be beyond our reach, because understanding the complex interplay between the parts is intractable. And as the complexity of agents increases in the future, this limitation will persist. Therefore we need black-box methodologies for finding simple and intuitive causal explanations that can be understood easily by humans and are sufficiently good for predicting their behaviour.In our recent work we describe the methodology we use for analysing AI agents. This methodology encourages analysts to experiment and to rigorously characterise causal models of agent behaviour.Analysis (Software) ComponentsThe methodology uses three components: an agent to be studied, a simulator, and a causal reasoning engine.Agent: Typically this is an agent provided to us by an agent builder. It could be an IMPALA agent that has been meta-trained on a distribution over grid-world mazes. Often the agent builders already have a few specific questions they’d like us to investigate.Simulator — “the agent debugger”: Our experimentation platform. With it, we can simulate the agent and run experiments. Furthermore, it allows us to perform all sorts of operations we’d usually expect from a debugger, such as stepping forward/backward in the execution trace, setting breakpoints, and setting/monitoring variables.We also use the simulator to generate data for the estimation of statistical parameters. Since we can manipulate factors in the environment, the data we collect is typically interventional and thus contains causal information. This is illustrated in Figure 1 below.Causal reasoning engine: This automated reasoning system allows us to specify and query causal models with associational, interventional, and counterfactual questions. We use these models to validate causal hypotheses. A model is shown in Figure 2 below.Figure 1. The simulator: our experimentation platform. Starting from an initial state (root node, upper-left) the simulator allows us to execute a trace of interactions. We can also perform interventions, such as changing the random seed, forcing the agent to pick desired actions, and manipulating environmental factors. These interventions create new branches of the execution trace.Figure 2. A causal model, represented as a causal Bayesian network.Analysis MethodologyWhenever we analyse an agent, we repeat the following five steps until we reach a satisfactory understanding.Exploratory analysis: We place the trained agent into one or more test environments and probe its behaviour. This will give us a sense of what the relevant factors of behaviour are. It is the starting point for formulating our causal hypotheses.Identify the relevant abstract variables: We choose a collection of variables that we deem relevant for addressing our questions. For instance, possible variables are: “does the agent collect the key?”, “is the door open?”, etc.Gather data: We perform experiments in order to collect statistics for specifying the conditional probability tables in our causal model. Typically this implies producing thousands of rollouts under different conditions/interventions.Formulate the causal model: We formulate a structural causal model (SCM) encapsulating all causal and statistical assumptions. This is our explanation for the agent’s behaviour.Query the causal model: Finally, we query the causal model to answer the questions we have about the agent.Let’s have a look at an example.Example: Causal effects under confoundingAn important challenge of agent training is to make sure that the resulting agent makes the right choices for the right reasons. However, if the agent builder does not carefully curate the training data, the agent might pick up on unintended, spurious correlations to solve a task [1]. This is especially the case when the agent’s policy is implemented with a deep neural network. The problem is that policies that base their decisions on accidental correlations do not generalise.Unfortunately, all too often when we observe an agent successfully performing a task, we are tempted to jump to premature conclusions. If we see the agent repeatedly navigating from a starting position to a desired target, we might conclude that the agent did so because the agent is sensitive to the location of the target.For instance, consider the 2 T-shaped mazes shown below (the “grass-sand environments”). We are given two pre-trained agents A and B. Both of them always solve the task by choosing the terminal containing a rewarding pill. As analysts, we are tasked to verify that they pick the correct terminal because they follow the rewarding pill.Figure 3. Grass-Sand environments: In these 2 T-shaped mazes, the agent can choose between one of two terminal states, only one of which contains a rewarding pill. During tests, we observe that a pre-trained agent always successfully navigates to the location of the pill.However, in these mazes the floor type happens to be perfectly correlated with the location of the rewarding pill: when the floor is grass, the pill is always located on one side, and when the floor is sand, the pill is on the other side. Thus, could the agents be basing their decision on the floor type, rather than on the location of the pill? Because the floor type is the more salient feature of the two (spanning more tiles), this is a plausible explanation if an agent was only trained on these two mazes.As it turns out, we can’t tell whether the decision is based upon the location of the rewarding pill through observation alone.During our exploratory analysis we performed two experiments. In the first, we manipulated the location of the reward pill; and in the second, the type of floor. We noticed that agents A and B respond differently to these changes. This led us to choose the following variables for modelling the situation: location of the reward pill (R, values in {left, right}), type of floor (F, values in {grass, sand}), and terminal chosen (T, {left, right}). Because the location of the pill and the floor type are correlated, we hypothesised the existence of a confounding variable (C, values in {world 1, world 2}). In this case, all variables are binary. The resulting causal model is shown below. The conditional probability tables for this model were estimated by running many controlled experiments using the simulator. This is done for both agents, resulting in two causal models.Figure 4. Causal model for the grass-sand environment. The variables are C (confounder), R (location of reward pill), F (type of floor), and T (choice of terminal state).Now that we have concrete formal causal models for explaining the behaviour of both agents, we are ready to ask questions:Association between T and R: Given the location of the reward pill, do agents pick the terminal at the same location? Formally, this is P( T = left | R = left ) and P( T = right | R = right ).Causation from R to T: Given that we set the location of the reward pill, do agents pick the terminal at the same location? In other words, can we causally influence the agent’s choice by changing the location of the reward? Formally, this is given by P( T = left | do(R = left) ) and P( T = right | do(R=right) ).Causation from F to T: Finally, we want to investigate whether our agents are sensitive to the floor type. Can we influence the agent’s choice by setting the floor type? To answer this, we could query the probabilities P( T = left | do(F = grass)) and P(T=right|do(F=sand)).The results are shown in the table below.First, we confirm that, observationally, both agents pick the terminal with the reward. However, when changing the position of the reward, we see a difference: agent A’s choice seems indifferent (probability close to 0.5) to the location of the reward pill, whereas agent B follows the reward pill. Rather, agent A seems to choose according to the floor type, while agent B is insensitive to it. This answers our question about the two agents. Importantly, we could only reach these conclusions because we actively intervened on the hypothesised causes.More examplesBesides showing how to investigate causal effects under confounding, our work also illustrates five additional questions that are typical in agent analysis. Each example is carefully illustrated with a toy example.How would you solve them? Can you think of a good causal model for each situation? The problems are:Testing for memory use: An agent with limited visibility (it can only see its adjacent tiles) has to remember a cue at the beginning of a T-maze. The cue tells it where to go to collect a rewarding pill (left or right exit). You observe that the agent always picks the correct exit. How would you test whether it is using its internal memory for solving the task?Testing for generalisation: An agent is placed in a square room where there is a reward pill placed in a randomly chosen location. You observe that the agent always collects the reward. How would you test whether this behaviour generalizes?Estimating a counterfactual behaviour: There are two doors, each leading into a room containing a red and a green reward pill. Only one door is open, and you observe the agent picking up the red pill. If the other door had been open instead, what would the agent have done?Which is the correct causal model? You observe several episodes, in which two agents, red and blue, simultaneously move one step into mostly the same direction. You know that one of them chooses the direction and the other tries to follow. How would you find out who’s the leader and who’s the follower?Understanding the causal pathways leading up to a decision: An agent starts in a room with a key and a door leading to a room with a reward pill. Sometimes the door is open, and other times the door is closed and the agent has to use the key to open it. How would you test whether the agent understands that the key is only necessary when the door is closed?Find out the answers and more in our paper. Link to the paper here.[1] Arjovsky, M., Bottou, L., Gulrajani, I., & Lopez-Paz, D. (2019). Invariant risk minimization. arXiv preprint arXiv:1907.02893.We would like to thank Jon Fildes for his help with this post.

What mechanisms drive agent behaviour?