State of AI Report 2020

The third <@State of AI@> report is out! I won’t go into details here since there is really quite a lot of information, but I recommend scrolling through the presentation to get a sense of what’s been going on. I was particularly interested in their 8 predictions for the next year: most of them seemed like they were going out on a limb, predicting something that isn’t just “the default continues”. On last year’s 6 predictions, 4 were correct, 1 was wrong, and 1 was technically wrong but quite close to being correct;  even this 67% accuracy would be pretty impressive on this year’s 8 predictions. (It does seem to me that last year’s predictions were more run-of-the-mill, but that might just be hindsight bias.)


Predicting Slow Judgments

Andreas Stuhlmueller
Owain Evans
Tom McGrath
Zac Kenton
Chris Cundy
Ryan Carey
Andrew Schreiber
Neal Jean
Girish Sastry

A joint project between Ought and FHI. The goal is to predict judgments that humans make after considering a question for many hours. However, that data is expensive to collect, so they also collect a lot of labels of how humans make judgments in limited time, and use those as noisy labels for the slow, deliberative judgments. They have just released Think Again (https://thinkagain.ought.org/), where you can make fast and slow judgments on Fermi estimates, political statements, or ML papers. They are especially looking for people to make judgments on ML papers. This will take an hour or two, but they'll recommend papers that they think you'll like based on your judgments of the papers they showed you.

I've summarized the post pretty well, I think -- mainly I'd encourage you to play the Think Again game, it's pretty fun and you generate useful data for them.


This book argues that humanity is in a special stage of its development: it is on the _precipice_, a narrow time during which we have enough power to destroy ourselves, but not enough wisdom to have mitigated such risks. It first argues that existential risk would be very important to reduce (for all the standard reasons), and then considers many different kinds of existential risks, finding that natural ones (asteroids, supervolcanoes, stellar explosions) are small relative to anthropogenic risks, both current (nuclear war, climate change, environmental destruction) and future (engineered pandemics, unaligned AI, dystopian scenarios). I'll focus primarily on the part about AI risk, as well as some of the comments on existential risk in general.

The AI risk presentation in the book was similar to that in [Superintelligence](https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/1501227742): it argues for risk from goal-directed AI systems (though the terminology used in the book is different). It first demonstrates the strong progress in deep learning, and then notes that expert surveys estimate that AGI is more likely than not to arrive in the next century. It then notes that we don't know how to specify a reward function for an AI system (even with e.g. inverse reinforcement learning), and to the extent that we get it wrong, it pits us in competition against a superintelligent adversary. Ideas like switching off the AI system wouldn't work, due to convergent instrumental subgoals like survival.

It also considers some obvious objections, including the very reasonable objection that "AI researchers won't build something that will kill them". However, Toby is still worried, citing that due to the unilateralist curse unaligned AGI might still be built by the most optimistic researchers, and in any case the personal benefits to the researchers might justify the risk of misalignment to them personally (though it would not be justified for the world as a whole).

The book then spends some time discussing _risk factors_, which are things that do not _directly_ lead to existential risks, but indirectly _exacerbate_ other existential risks, making them more likely. For example, great power war seems like a risk factor: it isn't going to cause an existential catastrophe by itself, but it increases the likelihood that we use risky technologies like bioweapons and AI that could then cause an existential catastrophe.

The book also has lots of useful insights about existential risks in general, which then also apply to AI risk: for example, risks that strike sooner should be prioritized (since the later risks can be dealt with later), risks that are more sudden will be more important to focus on (since we won't be able to build support as the risk gradually comes in), and risks that are "sharper" will be more neglected since there won't be as many "warning shots".

I enjoyed this book more than I thought I would: it had a lot of novel content for me, and I liked the explanations and comparisons across different kinds of existential risks (something that I hadn't really seen a single unified perspective on), and I especially liked the constant focus on what we do and don't know -- it felt more like a research paper (albeit in a conversational style) than a popular book, and was similarly information-dense.

On the AI part specifically, I liked that one of the endnotes cashed out powerful AI systems using model-based RL: this indeed seems like the thing that is closest to the classic expected utility maximizer, so the conclusions make a bit more sense. You still have to wonder how exactly the model is learned, and how exactly the AI system becomes good at using the model to find good actions, but at least under those two assumptions you would have all the standard convergent instrumental subgoals. In contrast, with model-free RL, the default expectation is that the RL agent needs to try things multiple times before it can learn to do them again, so it's less clear how it starts doing novel things. It seems that model-based and model-free RL are pretty similar so the distinction doesn't matter in practice, but at least conceptually it's a lot easier to reason about the model-based system (at least in the context of AI risk).

Toby gives a 1 in 10 chance of existential catastrophe from AI in the next century (more than half of his total of 1 in 6), which decomposes into a 1 in 2 chance of AGI this century, and 1 in 5 of it leading to existential catastrophe. This is a bit more pessimistic than Paul's <@estimate@>(@Conversation with Paul Christiano@) of 10% EV loss (which was over all time, not just this century), which is in turn a bit more pessimistic than the 1 in 10 chance that I <@estimated@>(@Conversation with Rohin Shah@) (and am now forever anchored on), which was over all time _and_ conditional on no additional effort from longtermists. But I wouldn't read too much into this -- 10 is a nice round number, and that probably played a big role in why I chose it. I certainly don't feel calibrated enough to easily tell the difference between 1 in 5 and 1 in 20 on a question of this complexity.

I am very happy about this trend of people actually stating numbers: it's a lot easier to narrow down on the important disagreements when people put down numbers, even if they're completely made up. I'd really like to see numbers from people who have larger disagreements (as I expect would be the case with e.g. MIRI researchers).

FLI Podcast: The Precipice: Existential Risk and the Future of Humanity with Toby Ord


Human Compatible: Artificial Intelligence and the Problem of Control

_Since I am aiming this summary for people who are already familiar with AI safety, my summary is substantially reorganized from the book, and skips large portions of the book that I expect will be less useful for this audience. If you are not familiar with AI safety, **note that I am skipping many arguments and counterarguments in the book that are aimed for you**. I'll refer to the book as "HC" in this newsletter._

Before we get into details of impacts and solutions to the problem of AI safety, it's important to have a model of how AI development will happen. Many estimates have been made by figuring out the amount of compute needed to run a human brain, and figuring out how long it will be until we get there. HC doesn't agree with these; it suggests the bottleneck for AI is in the algorithms rather than the hardware. We will need several conceptual breakthroughs, for example in language or common sense understanding, cumulative learning (the analog of cultural accumulation for humans), discovering hierarchy, and managing mental activity (that is, the metacognition needed to prioritize what to think about next). It's not clear how long these will take, and whether there will need to be more breakthroughs after these occur, but these seem like necessary ones.

What could happen if we do get beneficial superintelligent AI? While there is a lot of sci-fi speculation that we could do here, as a weak lower bound, it should at least be able to automate away almost all existing human labor. Assuming that superintelligent AI is very cheap, most services and many goods would become extremely cheap. Even many primary products such as food and natural resources would become cheaper, as human labor is still a significant fraction of their production cost. If we assume that this could bring up everyone's standard of life up to that of the 88th percentile American, that would result in nearly a _tenfold_ increase in world GDP per year. Assuming a 5% discount rate per year, this corresponds to $13.5 _quadrillion_ net present value. Such a giant prize removes many reasons for conflict, and should encourage everyone to cooperate to ensure we all get to keep this prize.

Of course, this doesn't mean that there aren't any problems, even with AI that does what its owner wants. Depending on who has access to powerful AI systems, we could see a rise in automated surveillance, lethal autonomous weapons, automated blackmail, fake news and behavior manipulation. Another issue that could come up is that once AI is better than humans at all tasks, we may end up delegating everything to AI, and lose autonomy, leading to _human enfeeblement_.

This all assumes that we are able to control AI. However, we should be cautious about such an endeavor -- if nothing else, we should be careful about creating entities that are more intelligent than us. After all, the gorillas probably aren't too happy about the fact that their habitat, happiness, and existence depends on our moods and whims. For this reason, HC calls this the _gorilla problem_: specifically, "the problem of whether humans can maintain their supremacy and autonomy in a world that includes machines with substantially greater intelligence". Of course, we aren't in the same position as the gorillas: we get to _design_ the more intelligent "species". But we should probably have some good arguments explaining why our design isn't going to succumb to the gorilla problem. This is especially important in the case of a fast intelligence explosion, or _hard takeoff_, because in that scenario we do not get any time to react and solve any problems that arise.

Do we have such an argument right now? Not really, and in fact there's an argument that we _will_ succumb to the gorilla problem. The vast majority of research in AI and related fields assumes that there is some definite, known _specification_ or _objective_ that must be optimized. In RL, we optimize the _reward function_; in search, we look for states matching a _goal criterion_; in statistics, we minimize _expected loss_; in control theory, we minimize the _cost function_ (typically deviation from some desired behavior); in economics, we design mechanisms and policies to maximize the _utility_ of individuals, _welfare_ of groups, or _profit_ of corporations. This leads HC to propose the following standard model of machine intelligence: _Machines are intelligent to the extent that their actions can be expected to achieve their objectives._ However, if we put in the wrong objective, the machine's obstinate pursuit of that objective would lead to outcomes we won't like.

Consider for example the content selection algorithms used by social media, typically maximizing some measure of engagement, like click-through. Despite their lack of intelligence, such algorithms end up changing the user's preference so that they become more predictable, since more predictable users can be given items they are more likely to click on. In practice, this means that users are pushed to become more extreme in their political views. Arguably, these algorithms have already caused much damage to the world.

So the problem is that we don't know how to put our objectives inside of the AI system so that when it optimizes its objective, the results are good for us. Stuart calls this the "King Midas" problem: as the legend goes, King Midas wished that everything he touched would turn to gold, not realizing that "everything" included his daughter and his food, a classic case of a <@badly specified objective@>(@Specification gaming examples in AI@). In some sense, we've known about this problem for a long time, both from King Midas's tale, and in stories about genies, where the characters inevitably want to undo their wishes.

You might think that we could simply turn off the power to the AI, but that won't work, because for almost any definite goal, the AI has an incentive to stay operational, just because that is necessary for it to achieve its goal. This is captured in what may be Stuart's most famous quote: _you can't fetch the coffee if you're dead_. This is one of a few worrisome [convergent instrumental subgoals](https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf).

What went wrong? The problem was the way we evaluated machine intelligence, which doesn't take into account the fact that machines should be useful for _us_. HC proposes: _Machines are **beneficial** to the extent that **their** actions can be expected to achieve **our** objectives_. But with this definition, instead of our AI systems optimizing a definite, wrong objective, they will _also_ be uncertain about the objective, since we ourselves don't know what our objectives are. HC expands on this by proposing three principles for the design of AI systems, that I'll quote here in full:

1. _The machine’s only objective is to maximize the realization of human preferences._
2. _The machine is initially uncertain about what those preferences are._
3. _The ultimate source of information about human preferences is human behavior._

[Cooperative Inverse Reinforcement Learning](https://arxiv.org/abs/1606.03137) provides a formal model of an _assistance game_ that showcases these principles. You might worry that an AI system that is uncertain about its objective will not be as useful as one that knows the objective, but actually this uncertainty is a feature, not a bug: it leads to AI systems that are deferential, that ask for clarifying information, and that try to learn human preferences. [The Off-Switch Game](https://arxiv.org/abs/1611.08219) shows that because the AI is uncertain about the reward, it will let itself be shut off. These papers are discussed later in this newsletter.

So that's the proposed solution. You might worry that the proposed solution is quite challenging: after all, it requires a shift in the entire way we do AI. What if the standard model of AI can deliver more results, even if just because more people work on it? Here, HC is optimistic: the big issue with the standard model is that it is not very good at learning our preferences, and there's a huge economic pressure to learn preferences. For example, I would pay a lot of money for an AI assistant that accurately learns my preferences for meeting times, and schedules them completely autonomously.

Another research challenge is how to actually put principle 3 into practice: it requires us to connect human behavior to human preferences. [Inverse Reward Design](https://arxiv.org/abs/1711.02827) and <@Preferences Implicit in the State of the World@>(@Learning Preferences by Looking at the World@) are example papers that tackle portions of this. However, there are _lots_ of subtleties in this connection. We need to use _Gricean semantics_ for language: when we say X, we do not mean the literal meaning of X: the agent must also take into account the fact that we bothered to say X, and that we didn't say Y. For example, I'm only going to ask for the agent to buy a cup of coffee if I believe that there is a place to buy reasonably priced coffee nearby. If those beliefs happen to be wrong, the agent should ask for clarification, rather than trudge hundreds of miles or pay hundreds of dollars to ensure I get my cup of coffee.

Another problem with inferring preferences from behavior is that humans are nearly always in some deeply nested plan, and many actions don't even occur to us. Right now I'm writing this summary, and not considering whether I should become a fireman. I'm not writing this summary because I just ran a calculation showing that this would best achieve my preferences, I'm doing it because it's a subpart of the overall plan of writing this bonus newsletter, which itself is a subpart of other plans. The connection to my preferences is very far up. How do we deal with that fact?

There are perhaps more fundamental challenges with the notion of "preferences" itself. For example, our _experiencing self_ and our _remembering self_ may have different preferences -- if so, which one should our agent optimize for? In addition, our preferences often change over time: should our agent optimize for our current preferences, even if it knows that they will predictably change in the future? This one could potentially be solved by learning _meta-preferences_ that dictate what kinds of preference change processes are acceptable.

All of these issues suggest that we need work across many fields (such as AI, cognitive science, psychology, and neuroscience) to reverse-engineer human cognition, so that we can put principle 3 into action and create a model that shows how human behavior arises from human preferences.

So far, we've been talking about the case with a single human. But of course, there are going to be multiple humans: how do we deal with that? As a baseline, we could imagine that every human gets their own agent that optimizes for their preferences. However, this will differentially benefit people who care less about other people's welfare, since their agents have access to many potential plans that wouldn't be available to an agent for someone who cared about other people. For example, if Harriet was going to be late for a meeting with Ivan, her AI agent might arrange for Ivan to be even later.

What if we had laws that prevented AI systems from acting in such antisocial ways? It seems likely that superintelligent AI would be able to find loopholes in such laws, so that they do things that are strictly legal but still antisocial, e.g. line-cutting. (This problem is similar to the problem that we can't just write down what we want and have AI optimize it.)

What if we made our AI systems utilitarian (assuming we figured out some acceptable method of comparing utilities across people)? Then we get the "Somalia problem": agents will end up going to Somalia to help the worse-off people there, and so no one would ever buy such an agent.

Overall, it's not obvious how we deal with the transition from a single human to multiple humans. While HC focuses on a potential solution for the single human / single agent case, there is still much more to be said and done to account for the impact of AI on all of humanity. To quote HC, "There is really no analog in our present world to the relationship we will have with beneficial intelligent machines in the future. It remains to be seen how the endgame turns out."

I enjoyed reading this book; I don't usually get to read a single person's overall high-level view on the state of AI, how it could have societal impact, the argument for AI risk, potential solutions, and the need for AI governance. It's nice to see all of these areas I think about tied together into a single coherent view. While I agree with much of the book, especially the conceptual switch from the standard model of intelligent machines to Stuart's model of beneficial machines, I'm going to focus on disagreements in this opinion.

First, the book has an implied stance towards the future of AI research that I don't agree with: I could imagine that powerful AI systems end up being created by learning alone without needing the conceptual breakthroughs that Stuart outlines. This has been proposed in e.g. <@AI-GAs@>(@AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence@)), and seems to be the implicit belief that drives OpenAI and DeepMind's research agendas. This leads to differences in risk analysis and solutions: for example, the <@inner alignment problem@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@) only applies to agents arising from learning algorithms, and I suspect would not apply to Stuart's view of AI progress.

The book also gives the impression that to solve AI safety, we simply need to make sure that AI systems are optimizing the right objective, at least in the case where there is a single human and a single robot. Again, depending on how future AI systems work, that could be true, but I expect there will be other problems that need to be solved as well. I've already mentioned inner alignment; other graduate students at CHAI work on e.g. [robustness](https://adversarialpolicies.github.io/) and transparency.

The proposal for aligning AI requires us to build a model that relates human preferences to human behavior. This sounds extremely hard to get completely right. Of course, we may not need a model that is completely right: since reward uncertainty makes the agent amenable to shutdowns, it seems plausible that we can correct mistakes in the model as they come up. But it's not obvious to me that this is sufficient.

The sections on multiple humans are much more speculative and I have more disagreements there, but I expect that is simply because we haven't done enough research yet. For example, HC worries that we won't be able to use laws to prevent AIs from doing technically legal but still antisocial things for the benefit of a single human. This seems true if you imagine that a single human suddenly gets access to a superintelligent AI, but when everyone has a superintelligent AI, then the current system where humans socially penalize each other for norm violations may scale up naturally. The overall effect depends on whether AI makes it easier to violate norms, or to detect and punish norm violations.


AI Alignment Podcast: Human Compatible: Artificial Intelligence and the Problem of Control

This podcast covers some of the main ideas from the book, which I'll ignore for this summary. It also talks a bit about the motivations for the book. Stuart has three audiences in mind. He wants to explain to laypeople what AI is and why it matters. He wants to convince AI researchers that they should be working in this new model of beneficial AI that optimizes for our objectives, rather than the standard model of intelligent AI that optimizes for its objectives. Finally, he wants to recruit academics in other fields to help connect human behavior to human preferences (principle 3), as well as to figure out how to deal with multiple humans.

Stuart also points out that his book has two main differences from Superintelligence and Life 3.0: first, his book explains how existing AI techniques work (and in particular it explains the standard model), and second, it proposes a technical solution to the problem (the three principles).


AI Safety Needs Social Scientists

One approach to AI safety is to "ask humans a large number of questions about what they want, train an ML model of their values, and optimize the AI system to do well according to the learned values". However, humans give answers that are limited, biased and often in disagreement with each other, and so AI safety needs social scientists to figure out how to improve this data - which eventually may be gathered from thousands or millions of people. Of particular importance is the ability to design rigorous experiments, drawing from an interdisciplinary understanding of human cognition and behaviour. The authors discuss [Debate](https://blog.openai.com/debate/) ([AN #5](https://mailchi.mp/0ae5d69de63b/alignment-newsletter-5)) as a case study of a safety technique whose success depends on empirical questions such as: how skilled are humans as judges by default? Can we train people to be better judges? Are there ways to restrict debate to make it easier to judge?

There are a couple of key premises underlying this argument. The first is that, despite human biases, there are correct answers to questions about human values - perhaps defined as the answer we would endorse if given all relevant information and unlimited time to think. However, it’s not necessary for AIs to always find those answers, as long as they are able to recognise cases in which they’re uncertain and do nothing (while there are some cases in which inaction can cause harm, such as a self-driving car ceasing to steer mid-journey, it seems that the most worrying long-term catastrophes can be avoided by inaction). Another reason for optimism is that even incomplete or negative results from social science experiments may be useful in informing technical safety research going forward. However, in some cases the systems we're trying to reason about are very different from anything we can test now - for example, AI debaters that are much stronger than humans.

This post, and its accompanying paper, seems very sensible to me. While I have some doubts about how informative human debate data will be about superhuman debaters, it certainly seems worth trying to gain more empirical information. Note that while the paper primarily discusses Debate, I think that many of its arguments are applicable to any human-in-the-loop safety methods (and probably others too). Currently I think Ought is the safety group focusing most on collecting human data, but I look forward to seeing other researchers doing so.


80K podcast with Katja Grace

Rob Wiblin interviewed Katja Grace of AI Impacts about her work predicting the future of AI. My main takeaway was that there are many important questions in this space that almost no one is trying to answer, and that we haven't made a good enough attempt yet to conclude that it's too hard to do, so we should put more time into it. If you haven't seen AI Impacts' work before, you can get some of the most interesting results (at a high level) from listening to this podcast. There's a ton of detail in the podcast -- too much for me to summarize here.

I don't currently think very much about timelines, intelligence explosions, and other questions that AI Impacts thinks about, but it seems very plausible to me that these could be extremely important. (I do think about discontinuities in progress and am very glad I read the [AI Impacts post](https://aiimpacts.org/likelihood-of-discontinuous-progress-around-the-development-of-agi/) on the subject.) One point that the interview brings up is that there are very few (perhaps two?) full time equivalents working on predicting the future of AI, while there are many people working on technical AI safety, so the former is more neglected. I'm not sure I agree with this -- the number of full time equivalents doing technical AI alignment research seems quite small (on the order of 50 people). However, I do see _many_ people who are trying to skill up so that they can do technical AI alignment research, and none who want to do better prediction, and that seems clearly wrong. I would guess that there are several readers of this newsletter who want to do technical AI alignment research, but who would have more impact if they worked in an adjacent area, such as prediction as at AI Impacts, or policy and strategy work, or in better tools and communication. Even though I'm well-placed to do technical research, I still think that common knowledge of research is a big enough bottleneck that I spend a lot of time on this newsletter. It seems likely that there is someone else who would do a better job than me, but who is set on technical safety research even though they wouldn't be as good. So I guess if you are still trying to figure out how to best help with AI alignment, or are about to start training up to do technical research, please do listen to this podcast and consider that alternative route, and various others as well. The goal is not to figure out which question is the most important, so that you can try to solve it. You'll likely do better by considering the field as a whole, and asking which area you would be in if someone optimally assigned people in the field to tasks.


The "most important century" series

In some sense, it is really weird for us to claim that there is a non-trivial chance that in the near future, we might build [transformative AI](https://www.openphilanthropy.org/blog/some-background-our-views-regarding-advanced-artificial-intelligence) and either (1) go extinct or (2) exceed a growth rate of (say) 100% per year. It feels like an extraordinary claim, and thus should require extraordinary evidence. One way of cashing this out: if the claim were true, this century would be the most important century, with the most opportunity for individuals to have an impact. Given the sheer number of centuries there are, this is an extraordinary claim; it should really have extraordinary evidence. This series argues that while the claim does seem extraordinary, _all_ views seem extraordinary -- there isn’t some default baseline view that is “ordinary” to which we should be assigning most of our probability.

Specifically, consider three possibilities for the long-run future:
1. **Radical:** We will have a productivity explosion by 2100, which will enable us to become technologically mature. Think of a civilization that sends spacecraft throughout the galaxy, builds permanent settlements on other planets, harvests large fractions of the energy output from stars, etc.
2. **Conservative:** We get to a technologically mature civilization, but it takes hundreds or thousands of years. Let’s say even 100,000 years to be ultra conservative.
3. **Skeptical:** We never become technologically mature for some reason. Perhaps we run into fundamental technological limits, or we choose not to expand into the galaxy, or we’re in a simulation, etc.
It’s pretty clear why the radical view is extraordinary. What about the other two?

The conservative view implies that we are currently in the most important 100,000-year period. Given that life is billions of years old, and would presumably continue for billions of years to come once we reach a stable galaxy-wide civilization, that would make this the most important 100,000 year period out of tens of thousands of such periods. Thus the conservative view is also extraordinary, for the same reason that the radical view is extraordinary (albeit it is perhaps only half as extraordinary as the radical view).

The skeptical view by itself does not seem obviously extraordinary. However, while you could assign 70% probability to the skeptical view, it seems unreasonable to assign 99% probability to such a view -- that suggests some very strong or confident claims about what prevents us from colonizing the galaxy, which we probably shouldn’t have given our current knowledge. So, we need to have a non-trivial chunk of probability on the other views, which still opens us up to critique of having extraordinary claims.

Okay, so we’ve established that we should at least be willing to say something as extreme as “there’s a non-trivial chance we’re in the most important 100,000-year period”. Can we tighten the argument, to talk about the most important _century_? In fact, we can, by looking at the economic growth rate.

You are probably aware that the US economy grows around 2-3% per year (after adjusting for inflation), so a business-as-usual, non-crazy, default view might be to expect this to continue. You are probably also aware that exponential growth can grow _very_ quickly. At the lower end of 2% per year, the economy would double every ~35 years. If this continued for 8200 years, **we'd need to be sustaining multiple economies as big as today's entire world economy _per atom in the galaxy_**. While this is not a priori impossible, it seems quite unlikely to happen. This suggests that we’re in one of fewer than 82 centuries that will have growth rates at 2% or larger, making it far less “extraordinary” to claim that we’re in the most important one, especially if you believe that growth rates are well correlated with change and ability to have impact.

The actual radical view that the author places non-trivial probability on is one we’ve seen before in this newsletter: it is one in which there is  automation of science and technology through advanced AI or whole brain emulations or other possibilities. This allows technology to substitute for human labor in the economy, which produces a positive feedback loop as the output of the economy is ploughed back into the economy creating superexponential growth and a “productivity explosion”, where the growth rate increases far _beyond_ 2%. The series summarizes and connects together <@many@>(@Modeling the Human Trajectory@), <@past@>(@Could Advanced AI Drive Explosive Economic Growth?@), <@Open@>(@Draft report on AI timelines@), <@Phil@>(@How Much Computational Power It Takes to Match the Human Brain@) <@analyses@>(@Semi-informative priors over AI timelines@), which I won't be summarizing here (since we've summarized these analyses previously). While this is a more specific and “extraordinary” claim than even the claim that we live in the most important century, it seems like it should not be seen as so extraordinary given the arguments above.

This series also argues for a few other points important to longtermism, which I’ll copy here:
1. **The long-run future is radically unfamiliar.** Enough advances in technology could lead to a long-lasting, galaxy-wide civilization that could be a radical utopia, dystopia, or anything in between.
2. **The long-run future could come much faster than we think**, due to a possible AI-driven productivity explosion. (I briefly mentioned this above, but the full series devotes much more space and many more arguments to this point.)
3. We, the people living in this century, have the chance to have a huge impact on huge numbers of people to come - if we can make sense of the situation enough to find helpful actions. But right now, **we aren't ready for this.**

I especially liked this series for the argument that 2% economic growth very likely cannot last much longer, providing quite a strong argument for the importance of this century, without relying at all on controversial facts about AI. At least personally I was previously uneasy about how “grand” or “extraordinary” AGI claims tend to be, and whether I should be far more skeptical of them as a result. I feel significantly more comfortable with these claims after seeing this argument.

Note though that it does not defuse all such uneasiness -- you can still look at how early we appear to be (given the billions of years of civilization that could remain in the future), and conclude that the simulation hypothesis is true, or that there is a Great Filter in our future that will drive us extinct with near-certainty. In such situations there would be no extraordinary impact to be had today by working on AI risk.


AGI safety from first principles

This sequence presents the author’s personal view on the current best arguments for AI risk, explained from first principles (that is, without taking any previous claims for granted). The argument is a specific instantiation of the _second species argument_ that sufficiently intelligent AI systems could become the most intelligent species, in which case humans could lose the ability to create a valuable and worthwhile future.

We should clarify what we mean by superintelligence, and how it might arise. The author considers intelligence as quantifying simply whether a system “could” perform a wide range of tasks, separately from whether it is motivated to actually perform those tasks. In this case, we could imagine two rough types of intelligence. The first type, epitomized by most current AI systems, trains an AI system to perform many different tasks, so that it is then able to perform all of those tasks; however, it cannot perform tasks it has not been trained on. The second type, epitomized by human intelligence and <@GPT-3@>(@Language Models are Few-Shot Learners@), trains AI systems in a task-agnostic way, such that they develop general cognitive skills that allow them to solve new tasks quickly, perhaps with a small amount of training data. This second type seems particularly necessary for tasks where data is scarce, such as the task of being a CEO of a company. Note that these two types should be thought of as defining a spectrum, not a binary distinction, since the type of a particular system depends on how you define your space of “tasks”.

How might we get AI systems that are more intelligent than humans? Besides improved algorithms, compute and data, we will likely also see that _interactions_ between AI systems will be crucial to their capabilities. For example, since AI systems are easily replicated, we could get a _collective_ superintelligence via a collection of replicated AI systems working together and learning from each other. In addition, the process of creation of AI systems will be far better understood than that of human evolution, and AI systems will be easier to directly modify, allowing for AI systems to recursively improve their own training process (complementing human researchers) much more effectively than humans can improve themselves or their children.

The second species argument relies on the argument that superintelligent AI systems will gain power over humans, which is usually justified by arguing that the AI system will be goal-directed. Making this argument more formal is challenging: the EU maximizer framework <@doesn’t work for this purpose@>(@Coherent behaviour in the real world is an incoherent concept@) and applying the [intentional stance](https://en.wikipedia.org/wiki/Intentional_stance) only helps when you have some prior information about what goals the AI system might have, which begs the question.

The author decides to instead consider a more conceptual, less formal notion of agency, in which a system is more goal-directed the more its cognition has the following properties: (1) self-awareness, (2) planning, (3) judging actions or plans by their consequences, (4) being sensitive to consequences over large distances and long time horizons, (5) internal coherence, and (6) flexibility and adaptability. (Note that this can apply to a single unified model or a collective AI system.) It’s pretty hard to say whether current training regimes will lead to the development of these capabilities, but one argument for it is that many of these capabilities may end up being necessary prerequisites to training AI agents to do intellectual work.

Another potential framework is to identify a goal as some concept learned by the AI system, that then generalizes in such a way that the AI system pursues it over longer time horizons. In this case, we need to predict what concepts an AI system will learn and how likely it is that they generalize in this way. Unfortunately, we don’t yet know how to do this.

What does alignment look like? The author uses <@intent alignment@>(@Clarifying "AI Alignment"@), that is, the AI system should be “trying to do what the human wants it to do”, in order to rule out the cases where the AI system causes bad outcomes through incompetence where it didn’t know what it was supposed to do. Rather than focusing on the outer and inner alignment decomposition, the author prefers to take a holistic view in which the choice of reward function is just one (albeit quite important) tool in the overall project of choosing a training process that shapes the AI system towards safety (either by making it not agentic, or by shaping its motivations so that the agent is intent aligned).

Given that we’ll be trying to build aligned systems, why might we still get an existential catastrophe? First, a failure of alignment is still reasonably likely, since (1) good behavior is hard to identify, (2) human values are complex, (3) influence-seeking may be a useful subgoal during training, and thus incentivized, (4) it is hard to generate training data to disambiguate between different possible goals, (5) while interpretability could help it seems quite challenging. Then, given a failure of alignment, the AI systems could seize control via the mechanisms suggested in <@What failure looks like@> and [Superintelligence](https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/1501227742). How likely this is depends on factors like (1) takeoff speed, (2) how easily we can understand what AI systems are doing, (3) how constrained AI systems are at deployment, and (4) how well humanity can coordinate.

I like this sequence: I think it’s a good “updated case” for AI risk that focuses on the situation in which intelligent AI systems arise through training of ML models. The points it makes are somewhat different from the ones I would make if I were writing such a case, but I think they are still sufficient to make the case that humanity has work to do if we are to ensure that AI systems we build are aligned.


The Alignment Problem

This book starts off with an explanation of machine learning and problems that we can currently see with it, including detailed stories and analysis of:

- The [gorilla misclassification incident](https://twitter.com/jackyalcine/status/615329515909156865)
- The [faulty reward in CoastRunners](https://openai.com/blog/faulty-reward-functions/)
- The [gender bias in language models](https://arxiv.org/abs/1607.06520)
- The [failure of facial recognition models on minorities](https://www.ted.com/talks/joy_buolamwini_how_i_m_fighting_bias_in_algorithms)
- The [COMPAS](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing) [controversy](https://www.documentcloud.org/documents/2998391-ProPublica-Commentary-Final-070616.html) (leading up to [impossibility results in fairness](https://www.chrisstucchio.com/pubs/slides/crunchconf_2018/slides.pdf))
- The [neural net that thought asthma reduced the risk of pneumonia](https://www.microsoft.com/en-us/research/wp-content/uploads/2017/06/KDD2015FinalDraftIntelligibleModels4HealthCare_igt143e-caruanaA.pdf)

It then moves on to agency and reinforcement learning, covering from a more historical and academic perspective how we have arrived at such ideas as temporal difference learning, reward shaping, curriculum design, and curiosity, across the fields of machine learning, behavioral psychology, and neuroscience. While the connections aren't always explicit, a knowledgeable reader can connect the academic examples given in these chapters to the ideas of <@specification gaming@>(@Specification gaming: the flip side of AI ingenuity@) and <@mesa optimization@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@) that we talk about frequently in this newsletter. Chapter 5 especially highlights that agent design is not just a matter of specifying a reward: often, rewards will do ~nothing, and the main requirement to get a competent agent is to provide good _shaping rewards_ or a good _curriculum_. Just as in the previous part, Brian traces the intellectual history of these ideas, providing detailed stories of (for example):

- BF Skinner's experiments in [training pigeons](https://psycnet.apa.org/record/1961-01933-001)
- The invention of the [perceptron](https://psycnet.apa.org/record/1959-09865-001)
- The success of [TD-Gammon](https://www.aaai.org/Papers/Symposia/Fall/1993/FS-93-02/FS93-02-003.pdf), and later [AlphaGo Zero](https://deepmind.com/blog/article/alphago-zero-starting-scratch)

The final part, titled "Normativity", delves much more deeply into the alignment problem. While the previous two parts are partially organized around AI capabilities -- how to get AI systems that optimize for _their_ objectives -- this last one tackles head on the problem that we want AI systems that optimize for _our_ (often-unknown) objectives, covering such topics as imitation learning, inverse reinforcement learning, learning from preferences, iterated amplification, impact regularization, calibrated uncertainty estimates, and moral uncertainty.

I really enjoyed this book, primarily because of the tracing of the intellectual history of various ideas. While I knew of most of these ideas, and sometimes also who initially came up with the ideas, it's much more engaging to read the detailed stories of _how_ that person came to develop the idea; Brian's book delivers this again and again, functioning like a well-organized literature survey that is also fun to read because of its great storytelling. I struggled a fair amount in writing this summary, because I kept wanting to somehow communicate the writing style; in the end I decided not to do it and to instead give a few examples of passages from the book in [this post](https://www.alignmentforum.org/posts/gYfgWSxCpFdk2cZfE/the-alignment-problem-machine-learning-and-human-values).


Engineering a Safer World

I recently read [Engineering a Safer World](https://static1.squarespace.com/static/53b78765e4b0949940758017/t/57d87eb6d2b8571af3501b26/1473898764674/Engineering_a_Safer_World+Nancy+Leveson.pdf) by Nancy G. Leveson, at [Joshua Achiam’s recommendation](https://twitter.com/jachiam0/status/1285449712573665281), and really enjoyed it, so get ready for another book summary! I’m not very happy with the summary I have -- it feels less compelling than the book, partly because the book provides a ton of examples that I don’t have the space to do -- but hopefully it is enough to get the key points across.

The main motivation of this book is to figure out how we can improve safety engineering. Its primary thesis is that the existing methods used in engineering are insufficient for the current challenges, and must be replaced by a method the author favors called STAMP. Note that the book is primarily concerned with mechanical systems that may also have computerized automation (think aerospace, chemical, and mechanical engineering); the conclusions should not be expected to apply directly to AI.

**The standard model of safety engineering and its deficiencies**

Historically, safety engineering has been developed as a reaction to the high level of accidents we had in the past, and as a result focused on the easiest gains first. In particular, there were a lot of gains to be had simply by ensuring that _machines didn’t break_. _(Rohin’s note: I’m editorializing a bit here, the author doesn’t explicitly say this but I think she believes it.)_ This led to a focus on _reliability_: given a specification for how a machine should operate, we aim to decrease the probability that the machine fails to meet that specification. For example, the specification for a water tank would be to contain the water up to a given pressure, and one way to improve the reliability of the tank would be to use a stronger material or make a thicker tank to make it less likely that the tank ruptures.

Under this model, an accident happens when a machine fails to meet its specification. So, we can analyze the accident by looking at what went wrong, and tracing back the physical causes to the first point at which a specification was not met, giving us a _root cause_ that can show us what we need to fix in order to prevent similar accidents in the future. We can call this sort of analysis an _event chain_ analysis.

However, in the last few decades there have been quite a few changes that make this model worse than it once was. The pace of technological change has risen, making it harder to learn from experience. The systems we build have become complex enough that there is a lot more _coupling_ or interaction effects between parts of the system that we could fail to account for.  Relatedly, the risks we face are getting large enough that we aren’t willing to tolerate even a _single_ accident. Human operators (e.g. factory workers) are no longer able to rely on easily understood and predictable mechanical systems, instead having to work with computerized automation which they cannot understand as well. At this point, event chain analysis and safety-via-reliability are no longer sufficient for safety engineering.

Consider for example the [Flight 965](https://en.wikipedia.org/wiki/American_Airlines_Flight_965) accident. In this case, the pilots got clearance to fly towards the Rozo waypoint in their descent, listed as (R) on their (paper) approach charts. One of the pilots pressed R in the flight management system (FMS), which brought up a list of waypoints that did _not_ include Rozo, and executed the first one (presumably believing that Rozo, being the closest waypoint, would show up first). As a result, the plane turned towards the selected waypoint, and crashed into a mountain.

The accident report for this incident placed the blame squarely on the pilots, firstly for not planning an appropriate path, and secondly for not having situational awareness of the terrain and that they needed to discontinue their approach. But most interestingly, the report blames the pilots for not reverting to basic radio navigation when the FMS became confusing. The author argues that the design of the automation was also flawed in this case, as the FMS stopped displaying the intermediate fixes to the chosen route, and the FMS’s navigational information used a different naming convention than the one in the approach charts. Surely this also contributed to the loss? In fact, in lawsuit appeals, the software manufacturer was held to be 17% liable.

However, the author argues that this is the exception, not the rule: typically event chain analysis proceeds until a human operator is found who did something unexpected, and then the blame can safely be placed on them. Operators are expected to “use common sense” to deviate from procedures when the procedures are unsafe, but when an accident happens, blame is placed on them for deviating from procedures. This is often very politically convenient, and is especially easy to justify thanks to hindsight bias, where we can identify exactly the right information and cues that the operator “should have” paid attention to, ignoring that in the moment there were probably many confusing cues and it was far from obvious which information to pay attention to. My favorite example has to be this quote from an accident report:

“Interviews with operations personnel did not produce a clear reason why the response to the [gas] alarm took 31 minutes. The only explanation was that there was not a sense of urgency since, in their experience, previous [gas] alarms were attributed to minor releases that did not require a unit evacuation.”

It is rare that I see such a clear example of a self-refuting paragraph. In the author’s words, “this statement is puzzling, because the statement itself provides a clear explanation for the behavior, that is, the previous experience”. It definitely sounds like the investigators searched backwards through the causal chain, found a situation where a human deviated from protocol, and decided to assign blame there.

This isn’t just a failure of the accident investigation -- the entire premise of some “root cause” in an event chain analysis implies that the investigators must end up choosing some particular point to label as The Root Cause, and such a decision is inevitably going to be determined more by the particular analysts involved rather than by features of the accident.

**Towards a new approach**

How might we fix the deficiencies of standard safety engineering? The author identifies several major changes in assumptions that are necessary for a new approach:

1. Blame is the enemy of safety. Safety engineering should focus on system behavior as a whole, where interventions can be made at many points on different levels, rather than seeking to identify a single intervention point.

2. Reliability (having machines meet their specifications) is neither necessary nor sufficient for safety (not having bad outcomes). Increased reliability can lead to decreased safety: if we increase the reliability of the water tank by making it using a stronger material, we may decrease the risk of rupture, but we may dramatically increase the harm when a rupture occurs since the water will be at a much higher pressure. This applies to software as well: highly reliable software need not be safe, as its specifications may not be correct.

3. Accidents involve the entire sociotechnical system, for which an event chain model is insufficient. Interventions on the sociological level (e.g. make it easy for low-level operators to report problems) should be considered part of the remit of safety engineering.

4. Major accidents are not caused by simultaneous occurrence of random chance events. Particularly egregious examples come from probabilistic risk analysis, where failures of different subsystems are often assumed to be independent, neglecting the possibility of a common cause, whether physical (e.g. multiple subsystems failing during a power outage) or sociological (e.g. multiple safety features being disabled as part of cost-cutting measures). In addition, systems tend to migrate towards higher risk over time, because environmental circumstances change, and operational practices diverge from the designed practices as they adapt to the new circumstances, or simply to be more efficient.

5. Operator behavior is a product of the environment in which it occurs. To improve safety, we must change the environment rather than the human. For example, if an accident occurs and an operator didn’t notice a warning light that could have let them prevent it, the solution is not to tell the operators to “pay more attention” -- that approach is doomed to fail.

**A detour into systems theory**

The new model proposed by the author is based on systems theory, so let’s take a moment to describe it. Consider possible systems that we may want to analyze:

First, there are some systems with _organized simplicity_, in which it is possible to decompose the system into several subsystems, analyze each of the subsystems independently, and then combine the results relatively easily to reach overall conclusions. We might think of these as systems in which analytic reduction is a good problem-solving strategy. _Rohin’s note: Importantly, this is different from the philosophical question of whether there exist phenomena that cannot be reduced to e.g. physics: that is a question about whether reduction is in principle possible, whereas this criterion is about whether such reduction is an effective strategy for a computationally bounded reasoner._ Most of physics would be considered to have organized simplicity.

Second, there are systems with _unorganized complexity_, where there is not enough underlying structure for analytic reduction into subsystems to be a useful tool. However, in such systems the behavior of individual elements of the system is sufficiently random (or at least, well-modeled as random) that statistics can be applied to it, and then the law of large numbers allows us to understand the system as an aggregate. A central example would be statistical mechanics, where we cannot say much about the motion of individual particles in a gas, but we can say quite a lot about the macroscopic behavior of the gas as a whole.

Systems theory deals with systems that have _organized complexity_. Such systems have enough organization and structure that we cannot apply statistics to it (or equivalently, the assumption of randomness is too incorrect), and are also sufficiently complex that analytic reduction is not a good technique (e.g. perhaps any potential decomposition into subsystems would be dominated by combinatorially many interaction effects between subsystems). Sociological systems are central examples of such systems: the individual components (humans) are very much not random, but neither are their interactions governed by simple laws as would be needed for analytic reduction. While systems theory cannot provide nearly the same level of precision as statistics or physics, it does provide useful concepts for thinking about such systems.

The first main concept in systems theory is that of _hierarchy and emergence_. The idea here is that systems with organized complexity can be decomposed into several hierarchical levels, with each level built “on top of” the previous one. For example, companies are built on top of teams which are built on top of individual employees. The behavior of components in a particular layer is described by some “language” that is well-suited for that layer. For example, we might talk about individual employees based on their job description, their career goals, their relationship with their manager, and so on, but we might talk about companies based on their overall direction and strategy, the desires of their customer base, the pressures from regulators, and so on.

_Emergence_ refers to the phenomenon that there can be properties of higher levels arising from lawful interactions at lower levels that nonetheless are meaningless in the language appropriate for the lower levels. For example, it is quite meaningful to say that the pressures on a company from government regulation caused them to (say) add captions to their videos, but if we look at the specific engineer who integrated the speech recognition software into the pipeline, we would presumably say “she integrated the speech recognition into the pipeline because she had previously worked with the code” rather than “she integrated it because government regulations told her to do so”. As another example, safety is an emergent system property, while reliability is not.

The second main concept is that of _control_. We are usually not satisfied with just understanding the behavior of systems; we also want to make changes to it (as in the case of making them safer). In systems theory, this is thought of as _control_, where we impose some sort of _constraint_ on possible system behavior at some level. For example, employee training is a potential control action that could aim to enforce the constraint that every employee knows what to do in an emergency. An effective controller requires a goal, a set of actions to take, a model of the system, and some way to sense the state of the system.

**STAMP: A new model underlying safety engineering**

The author then introduces a new model called Systems-Theoretic Accident Model and Processes (STAMP), which aims to present a framework for understanding how accidents occur (which can allow us to prevent them and/or learn from them). It contains three main components:

_Safety constraints_: In systems theory, a constraint is the equivalent of a specification, so these are just the safety-relevant specifications. Note that such specifications can be found at all levels of the hierarchy.

_Hierarchical safety controllers_: We use _controllers_ to enforce safety constraints at any given level. A control algorithm may be implemented by a mechanical system, a computerized system, or humans, and can exist at any level of the hierarchy. A controller at level N will typically depend on constraints at level N - 1, and thus the design of this controller influences which safety constraints are placed at level N - 1.

_Process models_: An effective controller must have a model of the process it is controlling. Many accidents are the result of a mismatch between the actual process and the process model of the controller.

This framework can be applied towards several different tasks, and in all cases the steps are fairly similar: identify the safety constraints you want, design or identify the controllers enforcing those constraints, and then do some sort of generic reasoning with these components.

If an accident occurs, then at the highest level, either the control algorithm(s) failed to enforce the safety constraints, or the control actions were sent correctly but were not followed. In the latter case, the controllers at the lower level should then be analyzed to see why the control actions were not followed. Ultimately, this leads to an analysis on multiple levels, which can identify several things that went wrong rather than one Root Cause, that can all be fixed to improve safety in the future.

**Organizational safety**

So far we’ve covered roughly chapters 1-4 of the book. I’ll now jump straight to chapter 13, which seems particularly important and relevant, as it deals with how organizational structure and management should be designed to support safety.

One major point that the author makes is that safety _is_ cost-effective for _long-term_ performance as long as it is designed into the system from the start, rather than added on at the last minute. Performance pressure on the other hand inevitably leads to cuts in safety.

In order to actually get safety designed into the system from the start, it is crucial that top management demonstrates a strong commitment to safety, as without this employees will inevitably cut corners on safety as they will believe it is in their incentives to do so. Other important factors include a concrete corporate safety policy, as well as a strong corporate safety culture. It is important that safety is part of the design process, rather than tacked on at the end. In the author’s words, _putting safety into the quality assurance organization is the worst place for it. [...] It sets up the expectation that safety is an after-the-fact or auditing activity only._

In addition, it is important that information can flow well. Going from the bottom to the top, it should be possible for low-level operators to report potential problems in a way that they are actually acted on and the relevant information reaches top management.  From the top to the bottom, safety information and training should be easily available and accessible to employees when they need it.

It is also important to have controls to prevent the general tendency of systems to migrate towards higher risk, e.g. by relaxing safety requirements as time passes without any incidents. The next chapter describes SUBSAFE, the author’s example of a well-run safety program, in which the control is for everyone to periodically watch a video reminding them of the importance of their particular safety work (in particular, the video shows the loss of the USS Thresher, an event that caused SUBSAFE to be created).

Perhaps obviously, it is important for an organization to have a dedicated safety team. This is in contrast to making everyone responsible for safety. In the author’s words: _While, of course, everyone should try to behave safely and to achieve safety goals, someone has to be assigned responsibility for ensuring that the goals are achieved._

If you start by designing for safety, it is cost-effective, not opposed to long-term money-maximizing. Once there is performance pressure, then you see cuts in safety. Also sometimes people fix symptoms instead of underlying causes, and then they just keep seeing symptoms forever and conclude they are inevitable.

**Miscellaneous notes**

The remaining chapters of the book apply STAMP in a bunch of different areas with many examples, including an entire chapter devoted to the STAMP treatment of a friendly fire accident. I also really liked the discussion of human factors in the book, but decided not to summarize it as this has already gotten quite long.

**Summary of the summary**

I’ll conclude with a quote from the book’s epilogue:

_What seems to distinguish those experiencing success is that they:_
_1. Take a systems approach to safety in both development and operations_
_2. Have instituted a learning culture where they have effective learning from events_
_3. Have established safety as a priority and understand that their long-term success depends on it_

**Relationship to AI safety**

A primary motivation for thinking about AI is that it would be very impactful for our society, and very impactful technologies need not have good impacts. “Society” clearly falls into the “organized complexity” class of systems, and so I expect that the ideas of safety constraints and hierarchical control algorithms will be useful ways to think about possible impacts of AI on society. For example, if we want to think about the possibility of AI systems differentially improving technical progress over “wisdom”, such that we get dangerous technologies before we’re ready for them, we may want to sketch out hierarchical “controllers” at the societal level that could solve this problem. Ideally these would eventually turn into constraints on the AI systems that we build, e.g. “AI systems should report potentially impactful new technologies to such-and-such committee”. I see the AI governance field as doing this sort of work using different terminology.

Technical AI alignment (in the sense of <@intent alignment@>(@Clarifying "AI Alignment"@)) does not seem to benefit as much from this sort of an approach. The main issue is that we are often considering a fairly unitary system (such as a neural net, or the mathematical model of expected utility maximization) to which the hierarchical assumption of systems theory does not really apply.

To be clear, I _do_ think that there in fact is some hierarchy. For example, in image classifiers where low levels involve edge detectors while high levels involve dog-face detectors. However, we do not have the language to talk about these hierarchies, nor the algorithms to control the intermediate layers. While <@Circuits@>(@Thread: Circuits@) is illustrating this hierarchy for image classifiers, it does not give us a language that we can (currently) use to talk about advanced AI systems. As a result, we are reduced to focusing on the incentives we provide to the AI system, or speculating on the levels of hierarchy that might be internal to advanced AI systems, neither of which seem particularly conducive to good work.

In the language of this book, I work on intent alignment because I expect that the ability to enforce the constraint “the AI system tries to do what its operator wants” will be a very useful building block for enforcing whatever societal safety constraints we eventually settle on, and it seems possible to make progress on it today. There are several arguments for risk that this ignores (see e.g. <@here@>(@The Main Sources of AI Risk?@) and <@here@>(@AI Research Considerations for Human Existential Safety@)); for some of these other risks, the argument is that we can handle those using similar mechanisms as we have before (e.g. governance, democracy, police, etc), _as long as_ we have handled intent alignment.


Clarifying some key hypotheses in AI alignment

This post (that I contributed to) introduces a diagram that maps out important and controversial hypotheses for AI alignment. The goal is to help researchers identify and more productively discuss their disagreements.


On the alignment problem

This 80,000 Hours podcast goes over many of the examples from Brian’s book, <@The Alignment Problem@>. I recommend listening to it if you aren’t going to read the book itself; the examples and stories are fascinating. (Though note I only skimmed through the podcast.)


User-Agent Value Alignment

This paper **from 2002** investigates what it would take to align an artificial agent with a human principal, under the assumption that the human utility function is known, but that the agent reward and human utility might be computed from different feature sets fA and fH. In this case, it is possible that the agent reward cannot capture all of the effects that the human cares about, leading to misalignment.

They introduce the concept of _graphical value alignment_, in which the only way that the agent’s actions can affect fH is through fA. In this case, we can establish _functional value alignment_ (in which the agent’s optimal policy also maximizes human utility), by setting the agent's reward for any specific fA to be the expectation (over fH) of the utility of fH, given fA. Note that the graphical criterion is _very strong_: it requires that _none_ of the agent’s unobserved effects matter at all to the human.

They suggest two methods for establishing alignment. First, we can define additional agent features (perhaps requiring additional sensors), until all of the effects on fH are captured by fA. However, this would be very difficult, if not impossible. Second, we can include all agent actions and observations as agent features, since any effect of the agent’s choice of policy on fH depends only on the observations made and actions taken. Of course, to achieve functional value alignment we would then have to have a good understanding of the expected human utility for every action given any observation, which is also hard.

They also briefly discuss the relationship between aligned agents and capable agents: a stone is aligned with you (per their definition), but also entirely useless. An interesting quote: _“Note that it might be harder to establish alignment with more competent agents because their skills afford many more pathways for adverse effects. This is a somewhat troubling thought.”_

It’s interesting how much of the alignment problem manifests itself even when you assume that the human utility function is known, but the feature sets used by the human and agent are different. The only piece of the argument missing from this paper is that with sufficiently capable agents, the agent will actually be _adversarial_ towards the human because of [convergent instrumental subgoals](https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf), and that argument can be made in this framework.

Unfortunately, both of their methods for producing alignment don’t scale well, as they admit in the paper. (The second method in particular is kind of like hardcoding the policy, similarly to the construction <@here@>(@Coherence arguments do not imply goal-directed behavior@).)


If I were a well-intentioned AI

This sequence takes on the perspective of an AI system that is well-intentioned, but lacking information about what humans want. The hope is to find what good AI reasoning might look like, and hopefully use this to derive insights for safety. The sequence considers Goodhart problems, adversarial examples, distribution shift, subagent problems, etc.

I liked this sequence. Often when presented with a potential problem in AI safety, I ask myself why the problem doesn't also apply to humans, and how humans have managed to solve the problem. This sequence was primarily this sort of reasoning, and I think it did a good job of highlighting how with sufficient conservatism it seems plausible that many problems are not that bad if the AI is well-intentioned, even if it has very little information, or finds it hard to communicate with humans, or has the wrong abstractions.


The Incentives that Shape Behaviour

Ryan Carey*
Eric Langlois*
Tom Everitt
Shane Legg

This post and [paper](https://arxiv.org/abs/2001.07118) introduce a method for analyzing the safety properties of a system using a  _causal theory of incentives_ (<@past@>(@Understanding Agent Incentives with Causal Influence Diagrams@) <@papers@>(@Modeling AGI Safety Frameworks with Causal Influence Diagrams@)).  An _incentive_ is something an agent must do to best achieve its goals. A _control incentive_ exists when an agent must control some component of its environment in order to maximize its utility, while a _response incentive_ is present when the agent's decision must be causally responsive to some component of its environment. These incentives can be analyzed formally by drawing a _causal influence diagram_, which represents a decision problem as a graph where each variable depends on the values of its parents.

For example, consider the case where a recommender algorithm decides what posts to show to maximize clicks. In the causal influnce diagram representing this system, we can include that we have control over the node 'posts to show', which has a direct effect on the node we want to maximize, 'clicks'. However, 'posts to show' may also have a direct effect on the node 'influenced user opinions', which itself affects 'clicks'. In the system as it stands, in addition to there being a desirable control incentive on 'clicks', there is also an undesirable control incentive on 'influenced user opinions', since they themselves influence 'clicks'. To get rid of the undesirable incentive, we could reward the system for _predicted clicks_ based on a model of the original user opinions, rather than for actual clicks.

I really like this formalization of incentives, which come up frequently in AI safety work. It seems like some people are <@already@>(@Asymptotically Benign AGI@) <@using@>(@Designing agent incentives to avoid reward tampering@) this framework, and this seems low-cost enough that it's easy to imagine a world where this features in the safety analysis of algorithm designers.

Paper: The Incentives that Shape Behaviour


Vox interview with Stuart Russell

Kelsey talked with Stuart Russell about his new book, <@Human Compatible@>(@Human Compatible: Artificial Intelligence and the Problem of Control@).


Partial preferences needed; partial preferences sufficient

I'm not sure I fully understand this post, but my understanding is that it is saying that alignment proposals must rely on some information about human preferences. Proposals like impact measures and corrigibility try to formalize a property that will lead to good outcomes; but any such formalization will be denoting some policies as safe and some as dangerous, and there will always exist a utility function according to which the "safe" policies are catastrophic. Thus, you need to also define a utility function (or a class of them?) that safety is computed with respect to; and designing this is particularly difficult.

This seems very similar to the problem I have with impact measures, but I wouldn't apply that argument to corrigibility. I think the difference might be that I'm thinking of "natural" things that agents might want, whereas Stuart is considering the entire space of possible utility functions. I'm not sure what drives this difference.


Value Alignment Map

This is a gigantic graph of many of the concepts in the AI risk space. Each concept has a description and links to existing literature, and by clicking around in the map I found several interesting links I hadn't seen before.

This map is so large that I can't actually use it to get a birds-eye view of the entire space, but it seems quite useful for looking at a local region and as a starting point to explore one particular aspect more deeply.


Existential Risk, Creativity & Well-Adapted Science

From a brief skim, it seems like this paper defines "creativity" in scientific research, and argues that existential risk research needs to be creative. Research is creative if it is composed of "hot" searches, where we jump large distances from one proposed solution to another, with broad differences between these solutions, as opposed to "cold" searches, in which we primarily make incremental improvements, looking over a small set of solutions clustered in the neighborhood of existing solutions. The paper argues that research on existential risk needs to be creative, because many aspects of such research make it hard to analyze in a traditional way -- we can't perform controlled experiments of extinction, nor of the extreme circumstances under which it is likely; there are many interdependent parts that affect each other (since existential risks typically involve effects on many aspects of society), and there is likely to be a huge amount of uncertainty due to lack of evidence. As a result, we want to change the norms around existential risk research from the standard academic norms, which generally incentivize conservatism and "cold" searches. Table 1 provides a list of properties of academia that lead to conservatism, and asks that future work think about how we could mitigate these.

While I'm not sure I agree with the reasons in this paper, I do think we need creativity and "hot" searches in technical AI safety, simply based on the level of confusion and uncertainty that we (or at least I) have currently. The properties in Table 1 seem particularly good as an initial list of things to target if we want to make creative research more likely.


Do what we mean vs. do what we say

I wrote a post proposing that we define a "do what we mean" system to be one in which the thing being optimized is latent (in the sense that it is not explicitly specified, not that it has a probability distribution over it). Conversely, a "do what we say" system explicitly optimizes something provided as an input. A lot of AI safety arguments can be understood as saying that a pure "do what we say" AI will lead to catastrophic outcomes. However, this doesn't mean that a "do what we mean" system is the way to go -- it could be that we want a "do what we mean" core, along with a "do what we say" subsystem that makes sure that the AI always listens to eg. shutdown commands.


Discontinuity from the Eiffel Tower

The Eiffel tower represented a 54-year discontinuity in the trend for "height of the tallest existing structure", and an 8000-year discontinuity in the trend for "height of the tallest structure ever". It's unclear what the cause of this discontinuity is, though the authors provide some speculation.

I'm not sure if I should update without knowing the cause of the discontinuity, or how the search for discontinuities was conducted. If you're searching for discontinuities, I do expect you'll find some, even if in general I expect discontinuities not to arise, so it doesn't feel like strong evidence that discontinuities are probable.

[Discontinuous progress investigation](https://aiimpacts.org/discontinuous-progress-investigation/) or [Likelihood of discontinuous progress around the development of AGI](https://aiimpacts.org/likelihood-of-discontinuous-progress-around-the-development-of-agi/)


Compact vs. Wide Models

A compact model is one which is very general, and easy to prove things about, but doesn't inherently capture the messiness of the real world inside the model. Examples include Turing machines and utility functions. A wide model is one which still has a conceptually crisp core, but these crisp core units must then be combined in a complicated way in order to get something useful. Examples include the use of transistors to build CPUs, and the hierarchical control model of human psychology. The nice thing about wide models is that they start to engage with the messiness of the real world, and so make it clearer where the complexity is being dealt with. This is a useful concept to have when evaluating a proposal for alignment -- it asks the question, "where does the complexity reside?"

I definitely support having models that engage more with the messiness of the real world. I'm not sure if I would have used "wide models" -- it seems like even the assumption of a crisp core makes it not as capable of handling messiness as I want. But if you're trying to get formal guarantees and you need to use some model, a wide model seems probably useful to use.


Putting the humanity into inhuman systems: How human factors and ergonomics can be used to manage the risks associated with artificial general intelligence

Paul M. Salmon
Tony Carden
Peter A. Hancock

This paper argues that the methods of Human Factors and Ergonomics (HFE) should be applied to AGI safety. They list fifteen different methods from the field, typically used to analyze the performance of humans in systems, which could be applied to AGI instead (on the assumption that AGI will be more like humans than like machines in today’s systems). They then give examples of how these might be applied to the Prometheus story in the prologue of [Life 3.0](https://www.amazon.com/Life-3-0-Being-Artificial-Intelligence/dp/1101946598).

Human Factors and Ergonomics in Manufacturing

I’m not very familiar with this field, but among other techniques the paper mentions STAMP and STPA which we’ve previously seen in <@Engineering a Safer World@>. It does seem to me like these techniques would be useful to apply to the entire sociotechnical system, of which an AGI system is just one part (and this is what the paper’s examples do). It is less clear to me whether it makes sense to take techniques designed for humans and apply them to AGI: perhaps we’ll have enough understanding of the differences between humans and AGI that we could do this in a reasonable way, but I think there is a real risk that the methods give incorrect conclusions simply because they make incorrect assumptions about how AGI works (given that they were designed for humans). Nonetheless, I do agree with the core claim of this paper that HFE is worth exploring.


Mapping the Conceptual Territory in AI Existential Safety and Alignment

There are a bunch of high-level overviews and research agendas, not all of which agree with each other. This post attempts to connect and integrate several of these, drawing heavily on <@Paul Christiano’s overview@>(@Current Work in AI Alignment@), [my](https://futureoflife.org/2019/04/11/an-overview-of-technical-ai-alignment-with-rohin-shah-part-1/) [overview](https://futureoflife.org/2019/04/25/an-overview-of-technical-ai-alignment-with-rohin-shah-part-2/), and the <@ARCHES agenda@>(@AI Research Considerations for Human Existential Safety@), but also including a lot of other work. It serves as a good way of connecting these various perspectives; I recommend reading it for this reason. (Unfortunately, it is rather hard to summarize, so I haven’t done so.)


When AI Systems Fail: Introducing the AI Incident Database

One obvious way to improve safety is to learn from past mistakes, and not repeat them. This suggests that it would be particularly valuable to have a repository of past incidents that have occurred, so that people can learn from them; indeed both aviation and cybersecurity have their own incident databases. The AI Incidents Database aims to fill this gap within AI. The database currently has over a thousand incidents covering a wide range of potential issues, including self-driving car accidents, wrongful arrests due to bad facial recognition or machine translation, and algorithm-driven “flash crash”.

[AI Incident Database](https://incidentdatabase.ai/)

[Paper: Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database](https://arxiv.org/abs/2011.08512)


Foundational Philosophical Questions in AI Alignment

This podcast starts with the topic of the paper <@Artificial Intelligence, Values and Alignment@> and then talks about a variety of different philosophical questions surrounding AI alignment.


State of AI Ethics

Abhishek Gupta
Marianna Ganapini
Renjie Butalid
Camylle Lanteigne
Allison Cohen
Mo Akif
Tania De Gasperis
Victoria Heath
Erick Galinkin

This report from the Montreal AI Ethics Institute has a wide variety of summaries on many different topics in AI ethics, quite similarly to this newsletter in fact.


To Trust Or Not To Trust A Classifier

Heinrich Jiang
Been Kim
Melody Y. Guan
Maya Gupta

The confidence scores given by a classifier (be it logistic regression, SVMs, or neural nets) are typically badly calibrated, and so it is hard to tell whether or not we should trust our classifier's prediction. The authors propose that we compute a _trust score_ to tell us how much to trust the classifier's prediction, computed from a training set of labeled datapoints. For every class, they filter out some proportion of the data points, which removes outliers. Then, the trust score for a particular test point is the ratio of (distance to nearest non-predicted class) to (distance to predicted class). They have theoretical results showing that a high trust score means that the classifier likely agrees with the Bayes-optimal classifier, as well as empirical results showing that this method does better than several baselines for determining when to trust a classifier. One cool thing about this method is that it can be done with any representation of the input data points -- they find that working with the activations of deeper layers of a neural net improves the results.

I'm a big fan of trying to understand when our AI systems work well, and when they don't. However, I'm a little confused by this -- ultimately the trust score is just comparing the given classifier with a nearest neighbor classifier. Why not just use the nearest neighbor classifier in that case? This paper is a bit further out of my expertise than I'd like to admit, so perhaps there's an obvious answer I'm not seeing.


From ImageNet to Image Classification

Dimitris Tsipras
Shibani Santurkar
Logan Engstrom
Andrew Ilyas
Aleksander Madry

ImageNet was crowdsourced by presenting images to MTurk workers who had to select images that contain a given class from a pool of images obtained via search on the internet.  This is problematic, as an image containing multiple classes will basically get assigned to a random suitable class which can lead to deviations between ImageNet performance and actual capability to recognize images. The authors used MTurk and allowed workers to select multiple classes, as well as one main class for a given image in a pool of 10000 ImageNet validation images. Around 20% of the images seem to contain objects representing multiple classes and the average accuracy for these images was around 10% worse than average for a wide variety of image classifiers. While this is a significant drop, it is still way better than predicting a random class that is in the image. Also, advanced models were still able to predict the ImageNet label in cases where it does not coincide with the main class identified by humans, which suggest that they exploit biases in the dataset generation. While the accuracy of model predictions with respect to the newly identified main class still increased with better accuracy in predicting labels, the accuracy gap seems to grow and we might soon hit a point where gains in ImageNet accuracy don't correspond to improved image classification.

I generally find these empiricial tests of whether ML systems actually do what they are assumed to do quite useful for better calibrating intuitions about the speed of AI progress, and to make failure modes more salient. While we have the latter, I am confused about what this means for AI progress: on one hand, this supports the claim that improved benchmark progress does not necessarily translate to better real world applicability. On the other hand, it seems like image classification might be easier than exploiting the dataset biases present in ImageNet, which would mean that we would likely be able to reach even better accuracy than on ImageNet for image classification with the right dataset.

Paper: From ImageNet to Image Classification: Contextualizing Progress on Benchmarks


A Guide to Writing the NeurIPS Impact Statement

Carolyn Ashurst
Markus Anderljung
Carina Prunkl
Jan Leike
Yarin Gal
Toby Shevlane
Allan Dafoe

NeurIPS 2020 requires paper submissions to include a statement on the broader impact of their work. This post provides a guide for how to write an effective impact statement. They recommend focusing on the most significant, neglected, and tractable impacts, both positive and negative, while also conveying the uncertainties involved. They also suggest integrating this into the research process by reading the tech governance literature and building institutional structures, and including this information in introductions.

Their guide then recommends considering 3 questions:
How could your research affect ML applications?
What are the societal implications of these applications?
What research or other initiatives could improve social outcomes?

There is more information in the guide on how to go about answering those questions, along with some examples.

I am definitely in favor of considering the impacts of ML research before conducting or publishing it. I think the field is currently either at or near a threshold where papers will start having significant real world effects. While I don’t think this requirement will be sufficient for ensuring positive outcomes, I am glad NeurIPS is trying it out.

I think the article makes very strong points and will improve the quality of the impact statements that get submitted. I particularly liked the point about communicating uncertainty, which is a norm that I think the ML community would benefit from greatly. One thing I would add here is that giving explicit probabilities is often more helpful than vague words like “might” or “could”.


Disambiguating "alignment" and related notions

Distinguishes between several kinds of alignment. Some focus on _terminal values_ from the AI, such as holistic alignment (the AI has the same terminal values as us) and parochial alignment (which I don't really understand, check the post). Sufficient alignment focuses on _outcomes_ (no X-event happens, or X-risk is sufficiently low). Finally, others focus on the _motivations_ of the AI, including intentional alignment (the AI tries to do what H wants it to do) and benign AI (R doesn't try to do what H doesn't want it to do).

It is definitely worth keeping these distinctions in mind whenever talking about alignment. I personally tend to think about the motivation-based definitions, because those seem to be the most tractable definitions to work on, mainly because I don't have to worry about the AI being incompetent (eg. an AI launching nukes accidentally while exploring its action space). It seems possible to get strong arguments for intentional alignment and then use that with improved capabilities to argue for sufficient alignment.


On Strong Artificial Intelligence

This article, written by a professor from China, argues that the AI community has never been focused on "strong AI", and we have no real path forward to building "strong AI", and that it would be so dangerous that we should never do research around it. The concept of "strong AI" here is a bit different from what we are used to -- I would probably call it human-like AGI, in that it would have consciousness, self-awareness, and emotions, and be as capable as a human.

This is an interesting position I haven't seen much in the West -- both that we can't build AGI, and that we shouldn't build it anyway. It's actually quite heartening to see an emphatic claim that we shouldn't build strong AI -- it seems like AI researchers as a group may in fact be able to coordinate to develop AI safely. Of course, this is a single viewpoint and is not representative of all AI researchers in China.


Distance Functions are Hard

Many ideas in AI alignment require some sort of distance function. For example, in [Functional Decision Theory](https://arxiv.org/abs/1710.05060), we'd like to know how "similar" two algorithms are (which can influence whether or not we think we have "logical control" over them). This post argues that defining such distance functions is hard, because they rely on human concepts that are not easily formalizable, and the intuitive mathematical formalizations usually have some flaw.

I certainly agree that *defining* "conceptual" distance functions is hard. It has similar problems to saying "write down a utility function that captures human values" -- it's possible in theory but in practice we're not going to think of all the edge cases. However, it seems possible to learn distance functions rather than defining them; this is already done in perception and state estimation.


Existential Risks: A Philosophical Analysis

The phrase "existential risk" is often used in different ways. This paper considers the pros and cons of five different definitions.

Inquiry: An Interdisciplinary Journal of Philosophy

While this doesn't mention AI explicitly, I think it's useful to read anyway, because often which of the five concepts you use will affect what you think the important risks are.


AI Alignment Podcast: Moral Uncertainty and the Path to AI Alignment

Initially, Will articulates arguments for moral realism (the idea that there are objectively true moral facts) and moral uncertainty (the idea that we should assign credences to different moral theories being correct). Later, the discussion turns to the relevance of these views to AI safety. Will distinguishes the control problem (ensuring AIs do what we say), from the problem of aligning AI with human values, from the problem of aligning AI with moral truth. Observing humans isn't sufficient to learn values, since people can be self-destructive or otherwise misguided. Perhaps AI could extrapolate the values an idealised version of each person would endorse; however, this procedure seems under-defined.

On the moral truth side, Will worries that most educated people are moral relativists or subjectivists and so they won't sufficiently prioritise aligning AI with moral truth. He advocates for a period of long philosophical reflection once we've reduced existential risk to near zero, to figure out which future would be best. Careful ethical reasoning during this period will be particularly important since small mistakes might be magnified massively when implemented on an astronomical scale; however, he acknowledges that global dynamics make such a proposal unlikely to succeed. On a brighter note, AGI might make great advances in ethics, which could allow us to make the future much more morally valuable.

I think moral uncertainty is an important and overdue idea in ethics. I also agree that the idea of extrapolating an idealised form of people's preferences is not well-defined. However, I'm very skeptical about Will's arguments about moral realism. In particular, I think that saying that nothing matters at all without moral realism is exactly the sort of type error which Eliezer argued against [here](https://www.lesswrong.com/posts/yA4gF5KrboK2m2Xu7/how-an-algorithm-feels-from-inside).

I'm more sympathetic to the idea that we should have a period of long reflection before committing to actions on an astronomical scale; this seems like a good idea if you take moral uncertainty at all seriously.


Key Concepts in AI Safety

This overview from CSET gives a brief introduction to AI safety using the <@specification, robustness, and assurance (SRA) framework@>(@Building safe artificial intelligence: specification, robustness, and assurance@). Follow-up reports cover [interpretability](https://cset.georgetown.edu/research/key-concepts-in-ai-safety-interpretability-in-machine-learning/) and [adversarial examples / robustness](https://cset.georgetown.edu/research/key-concepts-in-ai-safety-robustness-and-adversarial-examples/). I don’t expect these to be novel to readers of this newsletter -- I include them in case anyone wants a brief overview, as well as to provide links to AI safety reports that will likely be read by government officials.


Formal Metaethics and Metasemantics for AI Alignment

This website presents in great detail a process by which an agent might use data from human brains in order to infer a utility function for a single human (also spelling out what assumptions need to be made along the way), and then how it could combine the utility functions from different humans to arrive at "a fully technical ethical goal function". Emphasis is placed on solving the philosophical problems of metaethics and mental content. Quoting the website, they "suppose that unlimited computation and a complete low-level causal model of the world and the adult human brains in it are available".


Approaches to Deploying a Safe Artificial Moral Agent

This post investigates which of the current moral theories would most reduce existential risk if we programmed it into an AI system, and settles on Aristotelian virtue ethics (over utilitarianism and Kant's categorical imperative).


Understanding Agent Incentives with Causal Influence Diagrams

This post and associated paper model an agent's decision process using a causal influence diagram -- think of a Bayes net, and then imagine that you add nodes corresponding to actions and utilities. A major benefit of Bayes nets is that the criterion of d-separation can be used to determine whether two nodes are conditionally independent. Once we add actions and utilities, we can also analyze whether observing or intervening on nodes would lead the agent to achieve higher expected utility. The authors derive criteria resembling d-separation for identifying each of these cases, which they call observation incentives (for nodes whose value the agent would like to know) and intervention incentives (for nodes whose value the agent would like to change). They use observation incentives to show how to analyze whether a particular decision is fair or not (that is, whether it depended on a sensitive feature that should not be used, like gender). Intervention incentives are used to establish the security of [counterfactual oracles](https://arxiv.org/abs/1711.05541) more simply and rigorously.

These criteria are theoretically quite nice, but I'm not sure how they relate to the broader picture. Is the hope that we will be able to elicit the causal influence diagram an AI system is using, or something like it? Or perhaps that we will be able to create a causal influence diagram of the environment, and these criteria can tell us which nodes we should be particularly interested in? Maybe the goal was simply to understand agent incentives better, with the expectation that more knowledge would help in some as-yet-unknown way? None of these seem very compelling to me, but the authors might have something in mind I haven't thought of.


Exploring AI Safety in Degrees: Generality, Capability and Control

This paper argues that we should decompose the notion of “intelligence” in order to talk more precisely about AI risk, and in particular suggests focusing on _generality_, _capability_, and _control_. We can think of capability as the expected performance of the system across a wide variety of tasks. For a fixed level of capability, generality can be thought of as how well the capability is distributed across different tasks. Finally, control refers to the degree to which the system is reliable and deliberate in its actions. The paper qualitatively discusses how these characteristics could interact with risk, and shows an example quantitative definition for a simple toy environment.