80K podcast: How can policy keep up with AI advances?

Rob Wiblin
Jack Clark
Miles Brundage and Amanda Askell

OpenAI policy researchers Jack Clark, Amanda Askell and Miles Brundage cover a large variety of topics relevant to AI policy, giving an outside-view perspective on the field as a whole. A year or two ago, the consensus was that the field required [disentanglement research](https://forum.effectivealtruism.org/posts/RCvetzfDnBNFX7pLH/personal-thoughts-on-careers-in-ai-policy-and-strategy); now, while disentanglement research is still needed, there are more clearly defined important questions that can be tackled independently. People are now also taking action in addition to doing research, mainly by accurately conveying relevant concepts to policymakers. A common thread across policy is the framing of the problem as a large coordination problem, for which an important ingredient of the solution is to build _trust_ between actors.

Another thread was the high uncertainty over specific details of scenarios in the future, but the emergence of some structural properties that allow us to make progress anyway. This implies that the goal of AI policy should be aiming for _robustness_ rather than _optimality_. Some examples:
    - The [malicious use of AI report](https://maliciousaireport.com/) was broad and high level because each individual example is different and the correct solution depends on the details; a general rule will not work. In fact, Miles thinks that they probably overemphasized how much they could learn from other fields in that report, since the different context means that you quickly hit diminishing returns on what you can learn.
    - None of them were willing to predict specific capabilities over more than a 3-year period, especially due to the steep growth rate of compute, which means that things will change rapidly. Nonetheless, there are structural properties that we can be confident will be important: for example, a trained AI system will be easy to scale via copying (which you can't do with humans).
    - OpenAI's strategy is to unify the fields of capabilities, safety and policy, since ultimately these are all facets of the overarching goal of developing beneficial AI. They aim to either be the main actor developing beneficial AGI, or to help the main actor, in order to be robust to many different scenarios.
    - Due to uncertainty, OpenAI tries to have policy institutions that make sense over many different time horizons. They are building towards a world with formal processes for coordinating between different AI labs, but use informal relationships and networking for now.

AI policy is often considered a field where it is easy to cause harm. They identify two (of many) ways this could happen: first, you could cause other actors to start racing (which you may not even realize, if it manifests as a substantial increase in some classified budget), and second, you could build coordination mechanisms that aren't the ones people want and that work fine for small problems but break once they are put under a lot of stress. Another common one people think about is information hazards. While they consider info hazards all the time, they also think that (within the AI safety community) these worries are overblown. Typically people overestimate how important or controversial their opinion is. Another common reason for not publishing is not being sure whether the work meets high intellectual standards, but in this case the conversation will be dominated by people with lower standards.

Miscellaneous other stuff:
    - Many aspects of races can make them much more collaborative, and it is not clear that AI corresponds to an adversarial race. In particular, large shared benefits make races much more collaborative.
    - Another common framing is to treat the military as an adversary, and try to prevent them from gaining access to AI. Jack thinks this is mistaken, since then the military will probably end up developing AI systems anyway, and you wouldn't have been able to help them make it safe.
    - There's also a lot of content at the end about career trajectories and working at OpenAI or the US government, which I won't get into here.

It does seem like building trust between actors is a pretty key part of AI policy. That said, there are two kinds of trust that you can have: first, trust that the statements made by other actors are true, and second, trust that other actors are aligned enough with you in their goals that their success is also your success. The former can be improved by mechanisms lie monitoring, software verification, etc. while the latter cannot. The former is often maintained using processes that impose a lot of overhead, while the latter usually does not require much overhead once established. The former can scale to large groups comprising thousands or millions of people, while the latter is much harder to scale. I think it's an open question in AI policy to what extent we need each of these kinds of trust to exist between actors. This podcast seems to focus particularly on the latter kind.

Other miscellaneous thoughts:
    - I think a lot of these views are conditioned on a gradual view of AI development, where there isn't a discontinuous jump in capabilities, and there are many different actors all deploying powerful AI systems.
    - Conditional on the military eventually developing AI systems, it seems worth it to work with them to make their AI systems safer. However, it's not inconceivable that AI researchers could globally coordinate to prevent military AI applications. This wouldn't prevent it from happening eventually, but could drastically slow it down, and let defense scale faster than offense. In that case, working with the military can also be seen as a defection in a giant coordination game with other AI researchers.
    - One of my favorite lines: "I would recommend everyone who has calibrated intuitions about AI timelines spend some time doing stuff with real robots and it will probably … how should I put this? … further calibrate your intuitions in quite a humbling way." (Not that I've worked with real robots, but many of my peers have.)


Thinking About Risks From AI: Accidents, Misuse and Structure

The authors argue that in addition to risk from misuse of AI and "accidents", we should pay attention to the structural perspective: how AI changes the broader environment and incentives of various actors. Possible examples include creating winner-take-all competition or creating overlap between offensive and defensive actions. In the face of these effects, even competent and well-intentioned decision-makers might be pressured into making risky choices. To ameliorate this problem, more people should focus on AI policy, particularly social scientists and historians; and we should think hard about creating collective norms and institutions for AI.

This post makes an important point in a clear and concise way. My only concern is that "structural problems" is such a broad heading that practically anything can be included, making it more difficult to specifically direct attention towards existential threats (the same is true for the term "accidents", which to me doesn't properly reflect the threat of adversarial behaviour from AI). I don't know how to best handle this tradeoff, but think it's a point worth raising.

*Rohin's opinion*: I just wanted to add a note on why we've highlighted this piece. While many of the particular concrete examples have been explained before, the underlying system for thinking about AI is new and useful. I particularly liked the distinction made between focusing on _agency_ in AI (which leads you to think about accidents and misuse) vs. thinking about incentives and structure (which leads you to think about the entire causal chain leading up to the moment where an agent causes something bad to happen).


OpenAI Charter

In their words, this is "a charter that describes the principles we use to execute on OpenAI’s mission".

I'm very excited by this charter, it's a good sign suggesting that we can get the important actors to cooperate in building aligned AI, and in particular to avoid a competitive race. Key quote: "if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project".


‘Solving for X?’ Towards a problem-finding framework to ground long-term governance strategies for artificial intelligence

The typical workflow in governance research might go something like this: first, choose an existing problem to work on; second, list out possible governance mechanisms that could be applied to the problem; third, figure out which of these is best. We might call this the _problem-solving_ approach. However, such an approach has several downsides:

1. Such an approach will tend to use existing analogies and metaphors used for that problem, even when they are no longer appropriate.
2. If there are problems which aren’t obvious given current frameworks for governance, this approach won’t address them.
3. Usually, solutions under this approach build on earlier, allegedly similar problems and their solutions, leading to path-dependencies in what kind of solutions are being sought. This makes it harder to identify and/or pursue new classes of solutions
4. It is hard to differentiate between problems that are symptoms vs. problems that are root causes in such a framework, since not much thought is put into comparisons across problems
5. Framing our job as solving an existing set of problems lulls us into a false sense of security, as it makes us think we understand the situation better than we actually do (“if only we solved these problems, we’d be done; nothing else would come up”).

The core claim of this paper is that we should also invest in a _problem-finding_ approach, in which we do not assume that we even know what the problem is, and are trying to figure it out in advance before it arises. This distinction between problem-solving and problem-finding is analogous to the distinction between normal science and paradigm-changing science, between exploitation and exploration, and between “addressing problems” and “pursuing mysteries”. Including a problem-finding approach in our portfolio of research techniques helps mitigate the five disadvantages listed above. One particularly nice advantage is that it can help avoid the [Collingridge dilemma](https://en.wikipedia.org/wiki/Collingridge_dilemma): by searching for problems in advance, we can control them before they get entrenched in society (when they would be harder to control).

The authors then propose a classification of governance research, where levels 0 and 1 correspond to problem-solving and levels 2 and 3 correspond to problem-finding:

- **Business as usual** (level 0): There is no need to change the existing governance structures; they will naturally handle any problems that arise.
- **Puzzle-solving** (level 1): Aims to solve the problem at hand (something like deepfakes), possibly by changing the existing governance structures.
- **Disruptor-finding** (level 2): Searches for properties of AI systems that would be hard to accommodate with the existing governance tools, so that we can prepare in advance.
- **Charting macrostrategic trajectories** (level 3): Looks for crucial considerations about how AI could affect the trajectory of the world.

These are not just meant to apply to AGI. For example, autonomous weapons may make it easier to predict and preempt conflict, in which case rather than very visible drone strikes we may instead have “invisible” high-tech wars. This may lessen the reputational penalties of war, and so we may need to increase scrutiny of, and accountability for, this sort of “hidden violence”. This is a central example of a level 2 consideration.

The authors note that we could extend the framework even further to cases where governance research fails: at level -1, governance stays fixed and unchanging in its current form, either because reality is itself not changing, or because the governance got locked in for some reason. Conversely, at level 4, we are unable to respond to governance challenges, either because we cannot see the problems at all, or because we cannot comprehend them, or because we cannot control them despite understanding them.

One technique I like a lot is backchaining: starting from the goal you are trying to accomplish, and figuring out what actions or intermediate subgoals would most help accomplish that goal. I’ve spent a lot of time doing this sort of thing with AI alignment. This paper feels like it is advocating the same for AI governance, but also gives a bunch of concrete examples of what this sort of work might look like. I’m hoping that it inspires a lot more governance work of the problem-finding variety; this does seem quite neglected to me right now.

One important caveat to all of this is that I am not a governance researcher and don’t have experience actually trying to do such research, so it’s not unlikely that even though I think this sounds like good meta-research advice, it is actually missing the mark in a way I failed to see.

While I do recommend reading through the paper, I should warn you that it is rather dense and filled with jargon, at least from my perspective as an outsider.


AGI Strategy - List of Resources


Technology Roulette: Managing Loss of Control as Many Militaries Pursue Technological Superiority

US policy so far has been to pursue technological superiority in order to stay ahead of its adversaries and to prevent conflict through deterrence. This paper argues that policymakers should shift some attention to preparing for other risks, such as accidents, emergent effects, sabotage and proliferation (where other actors get and use the technology, without the same safety standards as the US). There were several interesting sections, but the one I was particularly interested in was the section arguing that keeping a human in the loop would not be sufficient. In military situations, decisions must often be made in time-sensitive, high-stress situations, and in such scenarios humans are not very good at making decisions. For example, if an AI system detects an incoming missile, it must autonomously aim and fire to prevent the missile from hitting its target -- there is not enough time for a human to be in the loop. The biggest issue though is that while a human may be part of the decision-making process, they are reliant on various machine readings and calculations in order to reach their decision, and so a human in the loop doesn't provide an independent check on the answer, and so is of limited utility. And as AI systems get better, humans will become less useful for checking the AI's decisions, making this a temporary solution at best.

I found the paper to be quite compelling, especially the comments on the human-in-the-loop solution. This feels relevant to problems in technical AI alignment, though I'm not exactly sure how. One question that it suggests -- how can we learn human preferences, when the human answers may themselves depend on the AI's actions? Stuart Armstrong has [pointed out](https://agentfoundations.org/item?id=1678) this problem as well.


AI Governance in 2019 - A Year in Review: Observations from 50 Global Experts

This report contains short essays from 50 experts reviewing progress in AI governance. I’ll describe a few themes here rather than try to summarize each essay.

The first is a strong emphasis on issues of bias, privacy, deception, and safety. Bias can occur both due to biases of programmers designing algorithms as well as bias that exists in the data. Deception includes deepfakes as well as online accounts that impersonate humans, a subset of which were made illegal in California this year.

The benefit of international collaborations and conferences and getting broad agreement from many stakeholders both in government and companies was frequently highlighted throughout. One example is the [OECD Principles on AI](https://www.oecd.org/going-digital/ai/principles/), which were later adopted by the G20 including both the US and China, but there were many working groups and committees organized as well, both within industry and governments.

The other shift in 2019 was moving from broad principles towards more specific sets of requirements and policy decisions. The principles agreed to have been quite similar, but the specific implementations vary significantly by country. There were individual essays describing the regional challenges in Europe, the UK, Japan, Singapore, India, and East Asia. Many essays also highlighted the debate around <@publication norms@>(@GPT-2: 1.5B Release@), which garnered a lot of attention in 2019 following OpenAI’s staged release of GPT-2.

I am very impressed by the number and diversity of experts that contributed to this report. I think it is quite valuable to get people with such different backgrounds and areas of expertise to collaborate on how we should be using AI ahead of time. I was also pleasantly surprised to hear that there was broad international agreement on principles so far, particularly given an overall political trend against global institutions that has occurred recently. I’m definitely interested to know what the key factors were in managing that and how we can make sure these things continue.

Another piece that jumped out at me is the overlap between longer-term issues of safety and shorter-term issues of bias and privacy. For technical safety work, I think the problems are largely distinct and it is important for safety researchers to remain focused on solving problems with major long-term consequences. However, in the governance context, the problems seem to have much more in common and require many similar institutions / processes to address. So I hope that these communities continue to work together and learn from each other.


The Windfall Clause: Distributing the Benefits of AI

Cullen O’Keefe
Peter Cihon
Ben Garfinkel
Carrick Flynn
Jade Leung
Allan Dafoe

The Windfall Clause is a proposed policy lever for improving outcomes from transformative AI. Corporations can voluntarily agree to be bound by the clause, in which case they must donate some proportion of _windfall profits_ (profits in excess of e.g. 1% of world GDP) for the benefit of humanity. Since such a scenario is exceedingly unlikely, it can be in corporations' interests to be bound by this clause, in order to reap the benefits of improved public relations. If the scenario actually occurs, we can then use the donations to solve many societal problems that would likely arise, e.g. job loss, inequality, etc.

While there are certainly major benefits to the Windfall Clause in the case of an actual windfall, it seems to me like there are benefits even when windfalls do not occur (a point mentioned but not emphasized in the full report). For example, in a world in which everyone has agreed to the Windfall Clause, the incentives to "win an economic race" decrease: even if it is possible for e.g. one company to "win" via a monopoly on AI, at least a portion of their "winnings" must be distributed to everyone else, plausibly decreasing incentives to race, and increasing the likelihood that companies pay attention to safety. (This of course assumes that the clause remains binding even after "winning", which is not obviously true.)


How does the offense-defense balance scale?

The offense-defense balance that characterises how easy it is to successfully attack others can affect what kinds of conflicts break out and how often that happens. This paper analyses how growing capabilities on both sides affect that balance. For example, consider an idealized model of cyber defense with a fixed set of vulnerabilities that are discovered independently by attackers and defenders. The attacker will initially be able to use almost all of the vulnerabilities they found. This is because, with only a small percentage of vulnerabilities discovered by both sides, the defender is unlikely to have found the same ones as the attacker. Marginal increases of the defender's capabilities are unlikely to uncover vulnerabilities used by the attacker in this regime, such that attacks become easier as both sides invest resources. Once most vulnerabilities have been found by both sides, this effect reverses as marginal investments by the attacker become unlikely to uncover vulnerabilities the defender has not fixed yet.

This pattern, where increasingly growing capabilities first favour offense but lead to defensive stability in the long run, dubbed **OD-scaling** seems to be common and can be expected to be found whenever there are **multiple attack vectors**, the attacker only needs to break through on some of them and the defender enjoys **local defense superiority**, meaning that with sufficient coverage by the defender for a given attack vector, it is almost impossible for the attacker to break through.

Because the use of digital and AI systems can be scaled up quickly, scale-dependent shifts of the offense-defense balance are going to increase in importance as these systems become ubiquitous.

I found it quite surprising that the paper mentions a lack of academic consensus about whether or not offensive advantage is destabilizing. Assuming that it is, OD-scaling might provide a silver lining concerning cybersecurity, provided things can be scaled up sufficiently. These kinds of dynamics also seem to put a natural ceiling on arms races: above a certain threshold, gains in capabilities provide advantage to both sides such that resources are better invested elsewhere.


Why Responsible AI Development Needs Cooperation on Safety

Amanda Askell
Miles Brundage
Jack Clark
Gillian Hadfield

AI systems are increasingly being developed by companies, and as such it is important to understand how competition will affect the safety and robustness of these systems. This paper models companies as agents engaging in a cooperate-defect game, where cooperation represents responsible development, and defection represents a failure to develop responsibly. This model yields five factors that increase the likelihood of companies cooperating on safety. Ideally, companies will have **high trust** that others cooperate on safety, large benefits from mutual cooperation (**shared upside**), large costs from mutual defection (**shared downside**), not much incentive to defect when others cooperate (**low advantage**), and not be harmed too much if others defect when they cooperate (**low exposure**).

They then suggest four concrete strategies that can help improve norms today. First, companies should help promote accurate beliefs about the benefits of safety. Second, companies should collaborate on research and engineering. Third, companies should be transparent and allow for proper oversight and feedback. Fourth, the community should incentivize adhering to high safety standards by rewarding safety work and penalizing unsafe behavior.

Given that much of current AI progress is being driven by increases in computation power, it seems likely to me that companies will soon become more significant players in the AI space. As a result, I appreciate that this paper tries to determine what we can do now to make sure that the competitive landscape is conducive to taking proper safety precautions. I do, however, believe that the single step cooperate-defect game which they use to come up with their factors seems like a very simple model for what will be a very complex system of interactions. For example, AI development will take place over time, and it is likely that the same companies will continue to interact with one another. Iterated games have very different dynamics, and I hope that future work will explore how this would affect their current recommendations, and whether it would yield new approaches to incentivizing cooperation.

The Role of Cooperation in Responsible AI Development


AI Alignment Podcast: China’s AI Superpower Dream

See also <@these@>(@Rationally Speaking #231 - Helen Toner on "Misconceptions about China and artificial intelligence"@) <@three@>(@The new 30-person research group in DC investigating how emerging technologies could affect national security@) <@podcasts@>(@FLI Podcast: Beyond the Arms Race Narrative: AI and China@).


AGI will drastically increase economies of scale

Economies of scale would normally mean that companies would keep growing larger and larger. With human employees, the coordination costs grow superlinearly, which ends up limiting the size to which a company can grow. However, with the advent of AGI, many of these coordination costs will be removed. If we can align AGIs to particular humans, then a corporation run by AGIs aligned to a single human would at least avoid principal-agent costs. As a result, the economies of scale would dominate, and companies would grow much larger, leading to more centralization.

This argument is quite compelling to me under the assumption of human-level AGI systems that can be intent-aligned. Note though that while the development of AGI systems removes principal-agent problems, it doesn't remove issues that arise due to different agents having different (non-value-related) information.

The argument probably doesn't hold with <@CAIS@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@), where each AI service is optimized for a particular task, since there would be principal-agent problems between services.

It seems like the argument should mainly make us more worried about stable authoritarian regimes: the main effect based on this argument is a centralization of power in the hands of the AGI's overseers. This is less likely to happen with companies, because we have institutions that prevent companies from gaining too much power, though perhaps competition between countries could weaken such institutions. It could happen with government, but if long-term governmental power still rests with the people via democracy, that seems okay. So the risky situation seems to be when the government gains power, and the people no longer have effective control over government. (This would include scenarios with e.g. a government that has sufficiently good AI-fueled propaganda that they always win elections, regardless of whether their governing is actually good.)


Beijing AI Principles

These principles are a collaboration between Chinese academia and industry, and hit upon many of the problems surrounding AI discussed today, including fairness, accountability, transparency, diversity, job automation, responsibility, ethics, etc. Notably for long-termists, it specifically mentions control risks, AGI, superintelligence, and AI races, and calls for international collaboration in AI governance.

Beijing publishes AI ethical standards, calls for int'l cooperation


How Sure are we about this AI Stuff?

Ben outlines four broad arguments for prioritising work on superintelligent AGI: that AI will have a big influence over the long-term future, and more specifically that it might cause instability, lock-in or large-scale "accidents". He notes the drawbacks of each line of argument. In particular, the "AI is a big deal" argument doesn't show that we have useful leverage over outcomes (compare a Victorian trying to improve the long-term effects of the industrial revolution). He claims that the next two arguments have simply not been researched thoroughly enough to draw any conclusions. And while the argument from accidents has been made by Bostrom and Yudkowsky, there hasn't been sufficient elaboration or criticism of it, especially in light of the recent rise of deep learning, which reframes many ideas in AI.

I find this talk to be eminently reasonable throughout. It highlights a concerning lack of public high-quality engagement with the fundamental ideas in AI safety over the last few years, relative to the growth of the field as a whole (although note that in the past few months this has been changing, with three excellent sequences released on the Alignment Forum, plus Drexler's technical report). This is something which motivates me to spend a fair amount of time writing about and discussing such ideas.

One nitpick: I dislike the use of "accidents" as an umbrella term for AIs behaving in harmful ways unintended by their creators, since it's misleading to describe deliberately adversarial behaviour as an "accident" (although note that this is not specific to Ben's talk, since the terminology has been in use at least since the Concrete problems paper).


AI Index 2018 Report

Yoav Shoham
Raymond Perrault
Erik Brynjolfsson
Jack Clark
James Manyika
Juan Carlos Niebles
Terah Lyons
John Etchemendy
Barbara Grosz

Lots of data about AI. The report highlights how AI is global, the particular improvement in natural language understanding over the last year, and the limited gender diversity in the classroom. We also see the expected trend of huge growth in AI, both in terms of interest in the field as well as in performance metrics.


AI Now 2018 Report

Meredith Whittaker
Kate Crawford
Roel Dobbe
Genevieve Fried
Elizabeth Kaziunas
Varoon Mathur
Sarah Myers West
Rashida Richardson
Jason Schultz
Oscar Schwartz


AI development incentive gradients are not uniformly terrible

This post considers a model of AI development somewhat similar to the one in [Racing to the precipice](https://www.fhi.ox.ac.uk/wp-content/uploads/Racing-to-the-precipice-a-model-of-artificial-intelligence-development.pdf) paper. It notes that under this model, assuming perfect information, the utility curves for each player are _discontinuous_. Specifically, the models predict deterministically that the player that spent the most on something (typically AI capabilities) is the one that "wins" the race (i.e. builds AGI), and so there is a discontinuity at the point where the players are spending equal amounts of money. This results in players fighting as hard as possible to be on the right side of the discontinuity, which suggests that they will skimp on safety. However, in practice, there will be some uncertainty about which player wins, even if you know exactly how much each is spending, and this removes the discontinuity. The resulting model predicts more investment in safety, since buying expected utility through safety now looks better than increasing the probability of winning the race (whereas before, it was compared against changing from definitely losing the race to definitely winning the race).

The model in [Racing to the precipice](https://www.fhi.ox.ac.uk/wp-content/uploads/Racing-to-the-precipice-a-model-of-artificial-intelligence-development.pdf) had the unintuitive conclusion that if teams have _more_ information (i.e. they know their own or other’s capabilities), then we become _less_ safe, which puzzled me for a while. Their explanation is that with maximal information, the top team takes as much risk as necessary in order to guarantee that they beat the second team, which can be quite a lot of risk if the two teams are close. While this is true, the explanation from this post is more satisfying -- since the model has a discontinuity that rewards taking on risk, anything that removes the discontinuity and makes it more continuous will likely improve the prospects for safety, such as not having full information. I claim that in reality these discontinuities mostly don't exist, since (1) we're uncertain about who will win and (2) we will probably have a multipolar scenario where even if you aren't first-to-market you can still capture a lot of value. This suggests that it likely isn't a problem for teams to have more information about each other on the margin.

That said, these models are still very simplistic, and I mainly try to derive qualitative conclusions from them that my intuition agrees with in hindsight.

Racing to the precipice: a model of artificial intelligence development


The Future of Surveillance

While we often think of there being a privacy-security tradeoff and an accountability-security tradeoff with surveillance, advances in AI and cryptography can make advances on the Pareto frontier. For example, automated systems could surveil many people but only report a few suspicious cases to humans, or they could be used to redact sensitive information (eg. by blurring faces), both of which improve privacy and security significantly compared to the status quo. Similarly, automated ML systems can be applied consistently to every person, can enable collection of good statistics (eg. false positive rates), and are more interpretable than a human making a judgment call, all of which improve accountability.


Debunking the AI Arms Race Theory

This article, published recently in the Texas National Security Review, argues that various national trends of military spending on AI do not meet the traditional definition of an 'arms race'. However, the current situation can be termed a _security dilemma_, a "more generalized competitive dynamic between states." The article identifies two ways in which race-style dynamics in AI competition towards the aims of national security might create new risks: (i) a need for increasingly rapid decision-making might leave humans with diminished control or 'out of the loop'; and (ii) the pressure to quickly improve military AI capabilities could result in sacrificing supplementary goals like robustness and reliability, leading to unsafe systems being deployed.

The article offers the following strategies as panaceas to such dynamics. Competing nations should institute strong internal processes to ensure their systems are robust and secure, and that human control can be maintained. Further, nations should encourage other countries to take similar steps to mitigate these risks within their own militaries. Finally, nations should cooperate in regulating the conduct of war to avoid mutual harm. It concludes after citing several sources that advocate for the US to adopt these strategies.

I think the headline was chosen by the editor and not the author: the AI arms race 'debunking' is less than a fourth of the whole article, and it's not even an important beat of the piece; instead, the article is about how use of technology/AI/deep learning for military applications in multipolar geopolitics *can actually result* in arms-race-style dynamics and tangible risks.

Even so, I'm not convinced that the traditional definition of 'arms race' isn't met. The author invokes percentage _growth_ in military spending of more than 10% over the previous year as a qualifying criterion for an arms race, but then compares this with the actual spending of 0.7% of the US military budget on AI in 2020 to make their case that there is no arms race. These two are not comparable; at the very least, we would need to know the actual spending on AI by the military across two years to see at what rate this spending changed, and whether or not it then qualifies to be an arms race.


NSCAI Final Report

Eric Schmidt
Robert Work
Safra Catz
Eric Horvitz
Steve Chien
Andrew Jassy
Mignon Clyburn
Gilman Louie
Chris Darby
William Mark
Kenneth Ford
Jason Matheny
José-Marie Griffiths
Katharina McFarland
Andrew Moore

In the US, the National Security Commission on AI released their report to Congress. The [full pdf](https://www.nscai.gov/wp-content/uploads/2021/03/Full-Report-Digital-1.pdf) is over 750 pages long, so I have not read it myself, and instead I’m adding in some commentary from others. In their [newsletter](https://cset.georgetown.edu/newsletters/), CSET says that highlights include:

- A warning that the U.S. military could be at a competitive disadvantage within the next decade if it does not accelerate its AI adoption. The report recommends laying the foundation for widespread AI integration by 2025, comprising a DOD-wide digital ecosystem, a technically literate workforce, and more efficient business practices aided by AI.
- A recommendation that the White House establish a new “Technology Competitiveness Council,” led by the vice president, to develop a comprehensive technology strategy and oversee its implementation.
- A recommendation that the U.S. military explore using autonomous weapons systems, provided their use is authorized by human operators.
- A proposal to establish a new Digital Service Academy and a civilian National Reserve to cultivate domestic AI talent.
- A call to provide $35 billion in federal investment and incentives for domestic semiconductor manufacturing.
- A recommendation to double non-defense AI R&D funding annually until it reaches $32 billion per year, and to triple the number of National AI Research Institutes.
- A call for reformed export controls, coordinated with allies, on key technologies such as high-end semiconductor manufacturing equipment.
- A recommendation that Congress pass a second National Defense Education Act and reform the U.S. immigration system to attract and retain AI students and workers from abroad.

While none of the report’s recommendations are legally binding, it has [reportedly been well-received by key members of both parties](https://apnews.com/article/ai-panel-urges-us-boost-tech-skills-95b210543d4a42bd6cd5347a46cb74d6).

Matthew van der Merwe also summarizes the recommendations in [Import AI](https://jack-clark.net/2021/03/08/import-ai-239-china-trains-a-massive-10b-model-vicarious-does-pickplace-the-gchq-publishes-some-of-its-thoughts-on-ai/); this has a lot of overlap with the CSET summary so I won't copy it here.

Jeff Ding adds in [ChinAI #134](https://chinai.substack.com/p/chinai-134-weaponized-interdependence):
[I]f you make it past the bluster in the beginning — or take it for what it is: obligatory marketing to cater to a DC audience hooked on a narrow vision of national security — there’s some smart moderate policy ideas in the report (e.g. chapter 7 on establishing justified confidence in AI systems).

In email correspondence, Jon Rodriguez adds some commentary on the safety implications:

1. The report acknowledges the potential danger of AGI, and specifically calls for value alignment research to take place (pg. 36). To my knowledge, this is one of the first times a leading world government has called for value alignment.
2. The report makes a clear statement that the US prohibits AI from authorizing the launch of nuclear weapons (pg. 98).
3. The report calls for dialogues with China and Russia to ensure that military decisions made by military AI at "machine speed" does not lead to out-of-control conflict escalation which humans would not want (pg. 97).

National Security Commission on Artificial Intelligence


Why those who care about catastrophic and existential risk should care about autonomous weapons

This post argues for a focus on autonomous weapons systems (AWs) for three main reasons:

**AWs Provide a Trial Run for AGI governance**. Governance of AWs shares many properties with AGI safety. Preventing an AW arms race would require international cooperation that would provide a chance to understand and improve AI governance institutions. As with any AI system, AWs have the potential to be *effective* without necessarily being aligned with human values, and accidents could quickly lead to deadly consequences. Public opinion and the vast majority of AI researchers oppose AW arms races, so there is an opportunity for global coordination on this issue.

**Some AWs can directly cause catastrophic risk**. Cheap drones could potentially be created at scale that are easy to transport and hard to detect. This could enable an individual to kill many people without the need to convince many others that it is justified. They can discriminate targets better than other WMDs and cause less environmental damage. This has the potential to make war less harmful, but also makes it easier to justify.

**AWs increase the likelihood and severity of conflict** by providing better tools for terrorists and assassins, lowering the threshold for violence between and within states, upsetting the relative power balance of current militaries, and increasing the likelihood of accidental escalation. In particular, AWs that are being used to counter other AWs might intentionally be made hard to understand and predict, and AWs may react to each other at timescales that are too quick for humans to intervene or de-escalate.

An international agreement governing autonomous weapons could help to alleviate the above concerns. In particular, some classes of weapons could be banned, and others could be tracked and subjected to regulations. This would hopefully lead us to an equilibrium where offensive AWs are prohibited, but defended against in a stable way.

I agree completely with the first two points. Much of technical safety work has been based around solving currently existing analogs of the alignment problem. Governance does seem to have less of these, so autonomous weapon governance could provide a great opportunity to test and build credibility for AI governance structures. The ability for autonomous weapons to cause catastrophic risk seems hard to argue against. With powerful enough AI, even accidents can pose catastrophic risk, but I would expect military use to only increase those.

For the third point, I agree with the reasons provided, but I think there are also ways in which AWs may reduce the likelihood and severity of war. For instance, currently soldiers bear most of the risk in wars, whereas decision-makers are often protected. Targeted AW attacks may increase the relative risk for those making decisions and thus disincentivize them from declaring war. An equilibrium of AW mutually assured destruction might also be attained if we can find reliable ways to attribute AW attacks and selectively retaliate. I’d be interested to see a more extensive analysis of how these and other factors trade off as I am unsure of the net effect.

The piece that gives me the most doubt that this is an area for the x-risk community to focus on is tractability. An international agreement runs the risk of weakening the states that sign on without slowing the rate of AW development in countries that don’t. Getting all actors to sign on seems intractable to me. As an analogy, nuclear weapons proliferation has been a challenge and nuclear weapons development is much more complex and visible than development of AWs.

**Rohin's opinion:** I particularly liked this piece because it actually made the case for work on autonomous weapons -- I do not see such work as obviously good (see for example [this post](https://forum.effectivealtruism.org/posts/vdqBn65Qaw77MpqXz/on-ai-weapons) that I liked, for the perspective against banning autonomous weapons). I still feel pretty uncertain overall, but I think this post meaningfully moved the debate forward.


Society-in-the-loop: programming the algorithmic social contract

Earlier in this newsletter we saw arguments that we should not build AI systems that are maximizing “humanity’s aggregated preferences”. Then how else are we supposed to build AI systems that work well for _society as a whole_, rather than an individual human? When the goal of the system is uncontested (e.g. “don’t crash”), we can use human-in-the-loop (HITL) algorithms where the human provides oversight; this paper proposes that for contested goals (e.g. “be fair”) we should put society in the loop (SITL), through _algorithmic social contracts_.

What is a social contract? A group of stakeholders with competing interests have a (non-algorithmic) social contract when they “agree” to allow use of force or social pressure to enforce some norm that guards people’s rights and punishes violators. For example, we have a social contract against murder, which legitimates the use of force by the government in order to punish violators.

In an algorithmic social contract, the norms by which the AI system operates, and the goals which it pursues, are determined through typical social contracts amongst the group of stakeholders that care about the AI system’s impacts. Notably, these goals and norms can change over time, as the stakeholders see what the AI system does. Of course, this all happens on relatively long timescales; more immediate oversight and control of the AI system would have to be done by specific humans who are acting as _delegates_ of the group of stakeholders.

The paper then goes into many open challenges for creating such algorithmic social contracts: How does society figure out what goals the AI system should pursue? How do we deal with externalities and tradeoffs? How can these fuzzy values be translated into constraints on the AI system? It provides an overview of some approaches to these problems.

I really like the notion of an algorithmic social contract: it much better captures my expectation of how AI systems will be integrated into society. With this vocabulary, I would put technical AI alignment research squarely in the last category, of how we translate fuzzy values that society agrees on into constraints on the AI system’s behavior.


AI Benefits

This sequence of posts investigates _AI Benefits_: how a benefactor can leverage advanced AI systems to benefit humanity. It focuses on what can be done by a single benefactor, outside of what we might think of as the “norm” -- in particular, the sequence ignores benefits that would be provided by default market incentives. This is relevant to OpenAI (where the author works) given their focus on ensuring AI is beneficial to humanity.

Note that AI Benefits is distinct from AI alignment. Sometimes AI alignment is defined broadly enough to encompass AI Benefits, but often it is not, e.g. if the notion of being “aligned” depends on an AI system being aligned with some principal, that would not be AI Benefits, since AI Benefits are meant to accrue to all of humanity. While it is about maximizing well-being by default, it should also have secondary goals of equality, autonomy, democratization, and epistemic modesty.

The obvious approach to AI Benefits is the _direct_ approach: figuring out how to apply advanced AI to directly generate benefits for humanity, e.g. by producing electricity more efficiently to mitigate climate change. However, it is important to also consider the indirect approach of making money using AI, and then donating the surplus to a different organization that can better produce benefits.

Given the massive number of potential ways to benefit humanity and our uncertainty about their efficacy, it is important to have a portfolio approach to AI Benefits, rather than scaling up a single intervention. In addition, since any given intervention will probably primarily benefit some subset of humanity, a portfolio approach should help lead to more equal distribution of benefits.

There are many outstanding questions on how AI Benefits should be done in practice. Should the benefactor pursue a direct or indirect approach? To what extent should they explore potential approaches for generating benefits, relative to exploiting approaches that we know work? Should they generate benefits now, or invest in the ability to generate benefits later? Should they focus on global (supranational) approaches, or allocate resources to each nation that can be used in a manner specialized to their citizens?

There are many questions on the governance side as well. We will presumably want some Benefits Group involving external experts to help distribute benefits optimally. When should such a group get democratic input? How do we evaluate such a group to ensure they are actually benefiting humanity optimally? To what extent will we also need internal governance within the group and benefactor, and how can this be done?

AI Benefits is effectively asking how we can answer the question of how to do the most good in the future, and as such many of the considerations also come up in effective altruism, especially at the current high level of abstraction. Nonetheless, there are differences in the situation, which will matter: for example, the effective altruism community does not currently need to plan for the situation where they control a majority of the world’s resources; a sufficiently ambitious and optimistic AI company may need to. Such a situation vastly increases the importance of e.g. democratic input, portfolio approaches, and information value. I’m glad that these questions are being tackled now and look forward to seeing more details in the future.


Machine Learning and Artificial Intelligence: how do we make sure technology serves the open society?

A report from a workshop held in the UK in December 2017, where a diverse group of people ("omputer scientists, entrepreneurs, business leaders, politicians, journalists, researchers and a pastor") talked about the "impact of AI on societies, governments and the relations between states, and between states and companies". I'm glad to see more thought being put into AI policy, and especially what the effects of AI may be on world stability.

As we might expect, the report raises more questions than answers. If you think a lot about this space, it is worth reading -- it proposes a classification of problems that's different from ones I've seen before -- but I really only expect it to be useful to people actively doing research in this space.


Commoditisation of AI, digital forgery and the end of trust: how we can fix it

Giorgio Patrini
Simone Lini
Hamish Ivey-Law and Morten Dahl.

This blog post talks about the many issues that could arise as digital forgery of audio and video gets better, and also talks about potential solutions and their pitfalls, such as digital signatures, and training ML models that can distinguish between real and fake media. He even has a simple but well-done experiment that looks at the ability of an ML model to see whether or not a face swap has happened.

If you haven't thought about digital forgery and its implications before, I strongly recommend it; otherwise I'd recommend you only read the section titled "A weekend experiment."

The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation


‘Skynet’ Revisited: The Dangerous Allure of Nuclear Command Automation

While I won't summarize this article in full here, I found it useful to see how some academics are thinking about the risks of automation in the military, as well as to get a picture of what current automation efforts actually look like. One quote I found particularly interesting:

“You will find no stronger proponent of integration of AI capabilities writ large into the Department of Defense,” said Lieutenant General Jack Shanahan, director of the Joint Artificial Intelligence Center (JAIC), at a September 2019 conference at Georgetown University, “but there is one area where I pause, and it has to do with nuclear command and control.” Referring to [an] article’s assertion that an automated U.S. nuclear launch ability is needed, he said, “I read that. And my immediate answer is, ‘No. We do not.’”


Activism by the AI Community: Analysing Recent Achievements and Future Prospects

The AI community has been surprisingly effective at activism: it has led to discussions of a ban on lethal autonomous weapons systems (LAWS), created several initiatives on safety and ethics, and has won several victories through organizing (e.g. Project Maven). What explains this success, and should we expect it to continue in the future? This paper looks at this through two lenses.

First, the AI community can be considered an _epistemic community_: a network of knowledge-based experts with coherent beliefs and values on a relevant topic. This seems particularly relevant for LAWS: the AI community clearly has relevant expertise to contribute, and policymakers are looking for good technical input. From this perspective, the main threats to future success are that the issues (such as LAWS) become less novel, that the area may become politicized, and that the community beliefs may become less cohesive.

Second, the AI community can be modeled as organized labor (akin to unions): since there is high demand for AI researchers, and their output is particularly important for company products, and the companies are more vulnerable to public pressure, AI researchers wield a lot of soft power when they are united. The main threat to this success is the growing pool of talent that will soon be available (given the emphasis on training experts in AI today), which will reduce the supply-demand imbalance, and may reduce how commited the AI community as a whole is to collective action.

Overall, it seems that the AI community has had good success at activism so far, but it is unclear whether it will continue in the future.

I think the ability of the AI community to cause things to happen via activism is quite important: it seems much more likely that if AI x-risk concerns are serious, we will be able to convince the AI community of them, rather than say the government, or company executives. This mechanism of action seems much more like the "epistemic community" model used in this paper: we would be using our position as experts on AI to convince decision makers to take appropriate precautions with sufficiently powerful AI systems. Applying the discussion from the paper to this case, we get the perhaps unsurprising conclusion that it is primarily important that we build consensus amongst AI researchers about how risky any particular system is.


Beyond Near- and Long-Term: Towards a Clearer Account of Research Priorities in AI Ethics and Society

This paper argues that the existing near-term / long-term distinction conflates four different axes on which research could differ: the capability level of AI systems (current pattern-matching systems vs. future intelligent systems), the impacts of AI systems (impacts that are being felt now like fairness vs. ones that will be felt in the future like x-risks), certainty (things that will definitely be problems vs. risks that are more speculative) and extremity (whether to prioritize particularly extreme risks). While there are certainly correlations across these axes, they are not the same thing, and discourse would be significantly improved by disambiguating the axes. For example, both authors of the paper see their work as considering the medium-to-long-term impacts of near-to-medium-term AI capabilities.

I definitely agree that near-term and long-term often seem to mean many different things, and I certainly support efforts to be more precise in our language.

While we're talking about near-term and long-term, I'll add in my own gripe: "long-term" implies that the effects will be felt only in the far future, even though many people focused on such effects are doing so because there's a significant probability of such effects being felt in only a few decades.


How a Pentagon Contract Became an Identity Crisis for Google

Scott Shane
Cade Metz and Daisuke Wakabayashi

After Google accepted a share of the contract for the Maven program run by the Defense Department, Google has been internally fractured, with many employees strongly opposing the use of AI for military applications.

Stories like this make me optimistic that we can actually coordinate AI researchers to take appropriate safety precautions when developing advanced AI systems, even if the economic incentives point in the other direction (and I'm not sure they do).


AI Alignment Podcast: On the Long-term Importance of Current AI Policy

Lucas Perry
Nicolas Moës and Jared Brown

While this podcast focuses both on the details of current policy as well as the long-term impact of engaging in policy today, I'm mostly interested in the latter, and so will simply quote Lucas's summary of points for that part:

1) Experience gained on short-term AI policy issues is important to be considered a relevant advisor on long-term AI policy issues coming up in the future.
2) There are very few people that care about AGI safety currently in government, politics or in policy communities.
3) There are opportunities to influence current AI policy decisions in order to provide a fertile ground for future policy decisions or, better but rarer, to be directly shaping AGI safety policy today through evergreen texts. Future policy that is implemented is path dependent on current policy that we implement today. What we do now is precedent setting.
4) There are opportunities today to develop a skillset useful for other policy issues and causes.
5) Little resource is being spent on this avenue for impact, so the current return on investment is quite good.

I think quite a lot about points 1 and 3, which I think also apply to technical safety research, not just policy. For our research to have an impact, it is necessary that either the research or its authors have enough credibility to actually influence decision-makers. In addition, the problems we will face in the future could depend on technical work done today: for example, if we were convinced that (say) AIs trained via evolution are too risky, we could push for AI to be developed in other ways now.


FLI Podcast: Distributing the Benefits of AI via the Windfall Clause

[Last week](https://mailchi.mp/9d279b575b1a/an-88-how-the-principal-agent-literature-relates-to-ai-risk), we had a brief summary of the [Windfall Clause](https://www.fhi.ox.ac.uk/windfallclause/) paper. This podcast goes into more depth about the potential benefits and objections to this clause: it's in some sense a more accessible and conversational elaboration of many of the points made in the paper.


Who owns artificial intelligence? A preliminary analysis of corporate intellectual property strategies and why they matter

This paper analyzes intellectual property (IP) considerations as they relate to AI. They identify two main incentives for companies: first, to publish AI research openly in order to attract top talent, and second, holding enough patents that they can credibly threaten to sue other companies for patent infringement. This second criterion lets companies stay in a mutually-assured-destruction (MAD) scenario, where if any one company litigates for patent infringement, they will quickly be met with a countersuit, and so the (fragile) equilibrium is to avoid litigation. They also identify two incentives for governments: first, to provide patents as a financial incentive for innovation in order to incentivize research, and second, to allow their own national security apparatus to use state of the art research while keeping it secret from perceived rivals.

Based on this analysis, they propose three scenarios that could unfold in the future. First, the status quo continues, in which companies keep acquiring patents in order to maintain the MAD equilibrium. Second, the equilibrium breaks, with one company litigating that then causes all the other companies to also litigate. This could result in most research becoming secret, in order to ensure that other companies can't "steal" the work and get a patent first. Similarly, contributions to open-source research might decrease, as it would be particularly easy to use such contributions as evidence of patent infringement. Third, more "patent pools" get created, in which multiple companies pool their patents together, to reduce the risk of litigation. Such patent pools could also be used to enforce other principles: with a sufficiently large patent pool, it could be the case that in order to remain competitive actors must license from the patent pool, and such licensing agreements could enforce specific ethical principles (although it would have to be careful to avoid violating antitrust law).

I enjoyed this paper; it seems good to have a better picture of the potential future of openness in AI research, for the reasons given in [Strategic Implications of Openness in AI Development](https://www.nickbostrom.com/papers/openness.pdf). You could also imagine patent pools as a vehicle for safety, as they are one possible way by which companies can cooperate to ensure a shared commitment to safety (along the lines of <@OpenAI's charter@>(@OpenAI Charter@)): they could tie competitiveness (which requires use of the research protected by the patent pool) to safety (the conditions involved in licensing the research in the patent pool).


France, China, and the EU All Have an AI Strategy. Shouldn’t the US?


2018 White House Summit on Artificial Intelligence for American Industry


AI Alignment Podcast: Machine Ethics and AI Governance

Machine ethics has aimed to figure out how to embed ethical reasoning in automated systems of today. In contrast, AI alignment starts from an assumption of intelligence, and then asks how to make the system behave well. Wendell expects that we will have to go through stages of development where we figure out how to embed moral reasoning in less intelligent systems before we can solve AI alignment.

Generally in governance, there's a problem that technologies are easy to regulate early on, but that's when we don't know what regulations would be good. Governance has become harder now, because it has become very crowded: there are more than 53 lists of principles for artificial intelligence and lots of proposed regulations and laws. One potential mitigation would be **governance coordinating committees**: a sort of issues manager that keeps track of a field, maps the issues and gaps, and figures out how they could be addressed.

In the intermediate term, the worry is that AI systems are giving increasing power to those who want to manipulate human behavior. In addition, job loss is a real issue. One possibility is that we could tax corporations relative to how many workers they laid off and how many jobs they created.

Thinking about AGI, governments should probably not be involved now (besides perhaps funding some of the research), since we have so little clarity on what the problem is and what needs to be done. We do need people monitoring risks, but there’s a pretty robust existing community doing this, so government doesn't need to be involved.

I disagree with Wendell that current machine ethics will be necessary for AI alignment -- that might be the case, but it seems like things change significantly once our AI systems are smart enough to actually understand our moral systems, so that we no longer need to design special procedures to embed ethical reasoning in the AI system.

It does seem useful to have coordination on governance, along the lines of governance coordinating committees; it seems a lot better if there's only one or two groups that we need to convince of the importance of an issue, rather than 53 (!!).


GPT-2: 1.5B Release

Irene Solaiman
Jack Clark
Miles Brundage

Along with the release of the last and biggest GPT-2 model, OpenAI explains their findings with their research in the time period that the staged release bought them. While GPT-2 can produce reasonably convincing outputs that are hard to detect and can be finetuned for e.g. generation of synthetic propaganda, so far they have not seen any evidence of actual misuse.

While it is consistent to believe that OpenAI was just generating hype since GPT-2 was predictably not going to have major misuse applications, and this has now been borne out, I'm primarily glad that we started thinking about publication norms _before_ we had dangerous models, and it seems plausible to me that OpenAI was also thinking along these lines.


Policy Researcher

There is a job opportunity at OpenAI as a policy researcher, which does not seem to have any formal requirements.

It seems like a lot of the best policy work is happening at OpenAI (see for example the [OpenAI charter](https://blog.openai.com/openai-charter/)), I strongly encourage people to apply!


Grover: A State-of-the-Art Defense against Neural Fake News

Rowan Zellers
Ari Holtzman
Hannah Rashkin
Yonatan Bisk
Ali Farhadi
Franziska Roesner
Yejin Choi

Could we use ML to detect fake news generated by other ML models? This paper suggests that models that are used to generate fake news will also be able to be used to _detect_ that same fake news. In particular, they train a GAN-like language model on news articles, that they dub GROVER, and show that the generated articles are _better_ propaganda than those generated by humans, but they can at least be detected by GROVER itself.

Notably, they do plan to release their models, so that other researchers can also work on the problem of detecting fake news. They are following a similar release strategy as with <@GPT-2@>(@Better Language Models and Their Implications@): they are making the 117M and 345M parameter models public, and releasing their 1.5B parameter model to researchers who sign a release form.

It's interesting to see that this group went with a very similar release strategy, and I wish they had written more about why they chose to do what they did. I do like that they are on the face of it "cooperating" with OpenAI, but eventually we need norms for _how_ to make publication decisions, rather than always following the precedent set by someone prior. Though I suppose there could be a bit more risk with their models -- while they are the same size as the released GPT-2 models, they are better tuned for generating propaganda than GPT-2 is.


Google’s brand-new AI ethics board is already falling apart

Google [announced](https://www.blog.google/technology/ai/external-advisory-council-help-advance-responsible-development-ai/) an ethical advisory council, that quickly became controversial, and was then [cancelled](https://www.vox.com/future-perfect/2019/4/4/18295933/google-cancels-ai-ethics-board). The author makes the point that the council was not well-placed to actually advise on ethics -- it would only meet four times a year, and could only give recommendations. This committee, and others at Facebook and Microsoft, seem to be more about PR and less about AI ethics. Instead, an AI ethics council should include both insiders and outsiders, should be able to make formal, specific, detailed recommendations, and would publicly announce whether the recommendations were followed. **Key quote:** "The brouhaha has convinced me that Google needs an AI ethics board quite badly — but not the kind it seems to want to try to build."

In a [tweetstorm](https://twitter.com/KelseyTuoc/status/1113544870625308673), the author holds OpenAI up as a large organization that is at least *trying* to engage deeply with AI ethics, as evidenced by their safety and policy team, their <@charter@>(@OpenAI Charter@), and <@GPT-2@>(@Better Language Models and Their Implications@). They make public, contentful statements that are weird, controversial and seem bad from a PR perspective. The arguments they make and hear about AI ethics and policy lead to real decisions with consequences.

I broadly agree with this article -- I can't imagine how a council that meets four times a year could properly provide advice on Google's AI projects. I'm not sure if the solution is more powerful and intensive ethics councils whose primary power is public accountability. I expect that making good decisions about AI ethics requires either a technical background, or a long, detailed conversation with a person with that background, neither of which are possible with the public. This could mean that an ethics board could struggle to raise a legitimate issue, or that they could cause outrage about an issue that is upon closer examination not an issue at all. I would feel better about a board with some more formal power, such as the ability to create investigations that could lead to fines, the ability to sue Google, specific whistleblowing affordances, etc. (I have no idea how feasible any of those suggestions are, even assuming Google was okay with them.)

On the tweetstorm about OpenAI, I'm not sure if I've said it before in this newsletter, but I generally trust OpenAI to be trying to do the right thing, and this is one of the reasons for that. Of course, I also know and trust many people who work there.


Rationally Speaking #231 - Helen Toner on "Misconceptions about China and artificial intelligence"

In this podcast Helen talks about AI policy, China, and the Center for Security and Emerging Technology, where she is the director of strategy. Some of her opinions that stood out to me:

    - While Baidu is a huge tech company and is the main search engine, it's a bit misleading to call it the Google of China, since it doesn't have the same diversity of products that Google does.
    - While the social credit score story seems overblown, the reporting on the Uighur situation seems to be basically accurate.
    - Based on a very small sample of AI researchers in China, it seems like Chinese researchers are less interested in thinking about the real-world effects of the technology they're building, relative to Western researchers.
    - Since people in government have so little time to think about so many issues, they have simple versions of important ideas. For example, it's easy to conclude that China must have an intrinsic advantage at data since they have more people and fewer privacy controls. However, there's a lot of nuance: for example, the entire Internet is in English, which seems like a big advantage for the US.
    - The incentives in China can be quite different: in at least one case, a professor's salary depended on the number of papers published.
    - A particularly interesting question: "how does it help the US geopolitically if an American company is developing powerful AI?"


A Survey of the EU's AI Ecosystem

This report analyzes the European AI ecosystem. The key advantage that Europe has is a strong focus on ethical AI, as opposed to the US and China that are more focused on capabilities research. However, Europe does face a significant challenge in staying competitive with AI, as it lacks both startup/VC funding as well as talented researchers (who are often going to other countries). While there are initiatives meant to help with this problem, it is too early to tell whether they will have an impact. The report also recommends having large multinational projects, along the lines of CERN and the Human Brain Project. See also [Import AI](https://jack-clark.net/2019/03/25/making-better-healthcare-ai-systems-via-audio-de-identification-teaching-drones-to-help-humans-fight-fires-and-why-language-models-could-be-smarter-than-you-think/).


Toward AI Security: Global Aspirations for a More Resilient Future

This report analyzes various AI security risks (including both near-term and long-term concerns) and categorizes them, and then analyzes how different national strategies and policies have engaged with these risks. Most interestingly (to me) it comes to the conclusion that most national AI strategies are focused on very different areas and often ignore (in the sense of not mentioning) risks that other countries have highlighted, though there are still some areas for cooperation, such as improving the transparency and accountability of AI systems.

It's pretty strange to me that different governments would take such different approaches to AI - this suggests that either academics, think tanks, policy analysts etc. do not agree on the risks, or that there isn't enough political pressure for some of the risks to make it into the strategies. It seems like the AI community would have a significant opportunity to shape policy in the latter case -- I'd imagine for example that an open letter signed by thousands of researchers could be quite helpful in creating political will. (Of course, creating a comprehensive open letter that most researchers will approve of might be quite hard to do.)


Risk factors for s-risks

This post discusses four risk factors for creating extreme disvalue in the universe (s-risks): advanced technology, lack of effort to avoid those outcomes, inadequate security and law enforcement, and polarisation and divergence of values. Tobias notes that he's most worried about cases where most of these factors occur, because the absence of any of them mitigates the threat posed by the others.


FLI Podcast- Artificial Intelligence: American Attitudes and Trends

This is a podcast about [The American Public’s Attitudes Concerning Artificial Intelligence](https://www.fhi.ox.ac.uk/aipublic2019/) ([AN #41](https://mailchi.mp/8c3f02cabccd/alignment-newsletter-41)), you can see my very brief summary of that.


The American Public’s Attitudes Concerning Artificial Intelligence

This presents results from a survey of Americans about their attitudes towards AI. There's not a compelling objective story I can tell, so you might as well look at the executive summary, which presents a few interesting highlights. One interesting fact: the median person thinks that we're more likely than not to have "high-level machine intelligence" within 10 years! You could also read [Vox's take](https://www.vox.com/future-perfect/2019/1/9/18174081/fhi-govai-ai-safety-american-public-worried-ai-catastrophe), which emphasizes that the public is concerned about long-term AI risk.


Countering Superintelligence Misinformation

Two ways to have better discussions about superintelligence are correcting misconceptions, and preventing misinformation from being spread in the first place. The latter might be achieved by educating prominent voices, creating reputational costs to misinformers (both individuals and companies), focusing media attention, etc. Research suggests the former is very difficult; strategies include addressing pre-existing motivations for believing misinformation and using advance warnings to 'inoculate' people against false claims.

I'm glad to see this systematic exploration of an issue that the AI safety community has consistently had to grapple with. I would have liked to see a more nuanced definition of misinformation than "information that is already clearly false", since it's not always obvious what qualifies as clearly false, and since there are many varieties of misinformation.

Superintelligence Skepticism as a Political Tool


Podcast: Artificial Intelligence – Global Governance, National Policy, and Public Trust with Allan Dafoe and Jessica Cussins

Allan Dafoe
Jessica Cussins
and Ariel Conn

Topics discussed include the difference between AI governance and AI policy, externalities and solving them through regulation, whether governments and bureaucracies can keep up with AI research, the extent to which the US' policy of not regulating AI may cause citizens to lose trust, labor displacement and inequality, and AI races.


The Advent of Huang's Law


China Now Has the Most Valuable AI Startup in the World

It's a short, interesting piece, and it's got some actual numbers and quotes from Xu Li (one of the co-founders of the startup, SenseTime), so you should read it.


The Deep Roots and Long Branches of Chinese Technonationalism


Collective Action on Artificial Intelligence: A Primer and Review

This paper reviews much of the work in AI governance (specifically, work on AI races and other collective action problems).


AI Certification: Advancing Ethical Practice by Reducing Information Asymmetries

Peter Cihon
Moritz J. Kleinaltenkamp
Jonas Schuett
Seth D. Baum

_Certification_ is a method of reducing information asymmetries: it presents credible information about a product to an audience that they couldn’t have easily gotten otherwise. With AI systems, certification could be used to credibly share information between AI actors, which could promote trust amongst competitors, or to share safety measures to prevent a race to the bottom on safety, caused by worrying that “the other guys would be even more unsafe”. Certification is at its best when there is _demand_ from an audience to see such certificates; public education about the need for credible information can help generate such demand.

However, certification often runs into problems. _Symbol-substance decoupling_ happens when certificates are issued to systems that don’t meet the standards for certification. For example, in “ethics washing”, companies advertise a self-certificate in which their products are approved by ethics boards, but those ethics boards have no real power. _Means-ends decoupling_ happens when the standards for certification don’t advance the goals for which the certificate was designed. For example, a certificate might focus on whether a system was tested, rather than on what test was conducted, leading applicants to use easy-to-pass tests that don’t actually provide a check on whether the method is safe.

Effective certification for future AI systems needs to be responsive to changes in AI technology. This can be achieved in a few ways: first, we can try to test the underlying goals which are more likely to remain stable; for example, we could certify ethical principles that will likely remain the same in the future. Second, we can match the certification to the types of people and institutions, that is, our certifications talk about the executives, citizens, or corporations (rather than e.g. specific algorithms that may be replaced in the future). Third, the certification system can build in mechanisms for updating the certification criteria periodically.

The paper then analyzes seven existing certification systems for AI systems; you’ll have to read the paper for details.


Shaping economic incentives for collaborative AGI

This post considers how to encourage a culture of cooperation among AI researchers. Then, when researchers try to create AGI, this culture of cooperation may make it more likely that AGI is developed collaboratively, instead of with race dynamics, making it more likely to be safe. It specifically poses the question of what external economic or policy incentives could encourage such cooperation.

I am optimistic about developing AGI collaboratively, especially through AI researchers cooperating. I'm not sure whether external incentives from government are the right way to achieve this -- it seems likely that such regulation would be aimed at the wrong problems if it originated from government and not from AI researchers themselves. I'm more optimistic about some AI researchers developing guidelines and incentive structures themselves, that researchers buy into voluntarily, that maybe later get codified into law by governments, or adopted by companies for their AI research.


An Overview of National AI Strategies

A short reference on the AI policies released by various countries.

Reading through this, it seems that countries are taking quite different approaches towards AI. I don't know what to make of this -- are they acting close to optimally given their geopolitical situation (which must then vary a lot by country), or does no one know what's going on and as a result all of the strategies are somewhat randomly chosen? (Here by "randomly chosen" I mean that the strategies that one group of analysts would select with is only weakly correlated with the strategies another group would select.) It could also be that the approaches are not actually that different.


Bridging the Gap: The Case for an ‘Incompletely Theorized Agreement’ on AI Policy.

Like <@several@>(@Beyond Near- and Long-Term: Towards a Clearer Account of Research Priorities in AI Ethics and Society@) <@past@>(@Bridging near- and long-term concerns about AI@) <@papers@>(@Medium-Term Artificial Intelligence and Society@), this paper argues that the differences between the “near-term” and “long-term” communities are probably exaggerated. Collaboration between these communities would be particularly beneficial, since it could prevent the field of AI policy from becoming fragmented and ineffectual, which is especially important now while the field is nascent and there is political will for AI policy progress.

The authors propose the notion of an “incompletely theorized agreement” in order to foster this sort of collaboration. In an incompletely theorized agreement, the parties agree to suspend disagreement on some thorny theoretical question, in order to coordinate action towards a shared pragmatic purpose. Such agreements could be used to set aside relatively unimportant disagreements between the two communities, in favor of pursuing goals that both communities care about. For example, we could imagine that such an agreement would allow both communities to push for more and better reflection by AI researchers on the impacts of the systems that they build, or to enable action that ensures we preserve the integrity of public discourse and informed decision-making (e.g. by regulating AI-enabled disinformation).

I’m certainly on board with the goal of working together towards shared goals. That being said, I don't fully understand what's being proposed here: how exactly is an incompletely theorized agreement supposed to be made? Is this more of a “shared ethos” that gets spread by word of mouth, or is there a document that people sign on to? If there is a document, what goes into it, who would agree to it, and how binding is it? I’d be excited to see more work fleshing out these concrete details, or even better, actually causing such an agreement to exist in practice.


Fragmentation and the Future: Investigating Architectures for International AI Governance

Should AI governance be done centrally, through an international body, or in a fragmented, decentralized fashion? This paper identifies various considerations pointing in different directions:

1. Centralized institutions can have more political power when designed well: their regulations can have more “teeth”.
2. Centralized institutions can be more efficient from the participant’s perspectives: if there is only one set of regulations, it is much easier for each participant to adhere to those regulations.
3. A centralized institution will typically be slower to act, as there are many more parties with a larger stake in the outcome. This can make it brittle, especially when the pace of technological change outpaces that of regulatory change.
4. Centralized institutions face a breadth vs. depth dilemma: if the regulations are too stringent, then some actors (i.e. nations, companies, etc) won’t participate (there is depth but not breadth), and similarly, to get everyone to participate the regulations must often be quite weak (breadth but not depth). In contrast, with decentralized approaches, the depth of the regulations can be customized to each participant.
5. With more fragmented approaches, actors can “forum shop” for the regulations which they think are best. It is unclear whether this is helpful or harmful for AI governance.
6. It is unclear which approach leads to more coordination. While a centralized approach ensures that everyone has the same policies, leading to policy _coherence_, it does not necessarily mean that those policies are good. A decentralized approach could lead to faster adaptation leading to better policies that are then copied by others, leading to more effective coordination overall.


Future Indices: How Crowd Forecasting Can Inform the Big Picture

Michael Page
Catherine Aiken
Dewey Murdick

This paper explains the methodology behind CSET’s recent forecasting project, Foretell. We would like to know which of several potential geopolitical scenarios might happen in the next 3-7 years. We can get some insight into this by asking relevant experts for their opinions, but often many experts will disagree, making it hard to know what to conclude.

We’d like to mitigate this by leveraging the wisdom of the crowds. Unfortunately, this would require us to have a clear and precise operationalization of our scenarios; the scenarios we’re interested in are rarely amenable to such operationalization. Instead, we can find a number of _predictors_ that would argue for a specific scenario, and identify one or more _metrics_ which are themselves clear and precise and give us information about some predictor. We can get forecasts for these metrics using the wisdom of the crowds. We can then compute the deviations between crowd forecasts and simple trend extrapolations of historical data, and use the observed trend directions as arguments for or against particular scenarios.

The paper illustrates this in the case of potential scenarios involving the US, China, and AI. An example of an important predictor is “US-China tensions”. Associated metrics include the amount of US-China trade, the number of Chinese O visas, etc. In this case, the crowd predictions suggested trend deviations in the metrics that argued for increasing US-China tensions.


AI Nationalism

As AI becomes more important in the coming years, there will be an increasing amount of "AI nationalism". AI policy will be extremely important and governments will compete on keeping AI talent. For example, they are likely to start blocking company takeovers and acquisitions that cross national borders -- for example, the UK could have been in a much stronger position had they blocked the acquisition of DeepMind (which is UK-based) by Google (which is US-based).


Overcoming Barriers to Cross-cultural Cooperation in AI Ethics and Governance

Seán S. ÓhÉigeartaigh
Jess Whittlestone
Yang Liu
Yi Zeng
Zhe Liu

This paper argues that it is important that AI ethics and governance is cross-cultural, and provides a few recommendations towards this goal:
1. Develop AI ethics and governance research agendas requiring cross-cultural cooperation
2. Translate key papers and reports
3. Alternate continents for major AI research conferences and ethics and governance conferences
4. Establish joint and/or exchange programmes for PhD students and postdocs


Antitrust-Compliant AI Industry Self-Regulation

One way to reduce the risk of unsafe AI systems is to have agreements between corporations that promote risk reduction measures. However, such agreements may run afoul of antitrust laws. This paper suggests that this sort of self-regulation could be done under the “Rule of Reason”, in which a learned profession (such as “AI engineering”) may self-regulate in order to correct a market failure, as long as the effects of such a regulation promote rather than harm competition.

In the case of AI, if AI engineers self-regulate, this could be argued as correcting the information asymmetry between the AI engineers (who know about risks) and the users of the AI system (who don’t). In addition, since AI engineers arguably do not have a monetary incentive, the self-regulation need not be anticompetitive. Thus, this seems like a plausible method by which AI self-regulation could occur without running afoul of antitrust law, and so is worthy of more investigation.


AI at Google: our principles

Following the outcry over the Maven program, Google has written a blog post detailing the principles they will follow for AI.

I found this line particularly interesting: "We will design our AI systems to be appropriately cautious, and seek to develop them in accordance with best practices in AI safety research." It sounds like the time is ripe for someone to write a "best practices" paper!


Institutionalizing ethics in AI through broader impact requirements

Carina E. A. Prunkl
Carolyn Ashurst
Markus Anderljung
Helena Webb
Jan Leike
Allan Dafoe

This short perspective analyzes the policy implemented by NeurIPS last year in which paper submissions were required to have a section discussing the broader impacts of the research. Potential benefits include _anticipating_ potential impacts of research, _acting_ to improve these impacts, _reflecting_ on what research to do given the potential impacts, and improving _coordination_ across the community. However, the policy may also lead to _trivialization_ of ethics and governance (thinking that all the relevant thinking about impacts can be done in this single statement), _negative attitudes_ towards the burden of writing such statements or responsible research in general, a _false sense of security_ that the ethics are being handled, and a _perception_ of ethics as something to be done as an afterthought.

The main challenges that can cause these sorts of negative effects are:
1. Analyzing broader impacts can be difficult and complex,
2. There are not yet any best practices or guidance,
3. There isn’t a clear explanation of the purpose of the statements, or transparency into how they will be evaluated,
4. It’s tempting to focus on the research that determines whether or not your paper is published, rather than the broader impacts statement which mostly does not affect decisions,
5. Researchers may have incentives to emphasize the beneficial impacts of their work and downplay the negative impacts.
6. Biases like motivated reasoning may affect the quality and comprehensiveness of impact statements.

To mitigate these challenges, the authors recommend improving _transparency_, setting _expectations_, providing _guidance_ on how to write statements, improving _incentives_ for creating good impact statements, and learning from experience through _community deliberation_. To improve incentives in particular, broader impact statements could be made an explicit part of peer review which can affect acceptance decisions. These reviews could be improved by involving experts in ethics and governance. Prizes could also be given for outstanding impact statements, similarly to best paper awards.

I’ve been pretty skeptical of the requirement to write a broader impacts statement. My experience of it was primarily one of frustration, for a few reasons:
1. Forecasting the future is hard. I don’t expect a shallow effort to forecast to be all that correlated with the truth. There were lots of simple things I could say that “sound” right but that I don’t particularly expect to be true, such as “improving cooperation in multiagent RL will help build cooperative, helpful personal assistants”. It’s a lot harder to say things that are actually true; a real attempt to do this would typically be a paper in itself.
2. To the extent that the statement does affect reviews, I expect that reviewers want to hear the simple things that sound right; and if I _don’t_ write them, it would probably be a strike against the paper.
3. Even if I did write a good statement, I don’t expect anyone to read it or care about it.

From a birds-eye view, I was also worried that if such statements do become popular, they’ll tend to ossify and build consensus around fairly shallow views that people come up with after just a bit of thought.

I do think many of the proposals in this paper would help quite a bit, and there probably is a version of these statements that I would like and endorse.


Exploring AI Futures Through Role Play

Shahar Avin
Ross Gruetzemacher
James Fox

This paper argues that role playing (akin to the "wargames" used in the military) is a good way to explore possible AI futures, especially to discover unusual edge cases, in a 10-30 year time horizon. Each player is assigned a role (e.g. director of AI at Tencent, or president of the US) and asked to play out their role faithfully. Each game turn covers 2 simulated years, in which players can negotiate and take public and private actions. The game facilitator determines what happens in the simulated world based on these actions. While early games were unstructured, recent games have had an AI "tech tree", that determines what AI applications can be developed.

From the games played so far, the authors have found a few patterns:
- Cooperation between actors on AI safety and (some) restriction on destabilizing uses of AI seem to both be robustly beneficial.
- Even when earlier advances are risky, or when current advances are of unclear value, players tend to pursue AI R&D quite strongly.
- Many possible kinds of coalitions are possible, e.g. between governments, between corporations, between governments and corporations, and between sub-roles within a corporation.

It makes sense that role playing can help find extreme, edge case scenarios. I'm not sure how likely I should find such scenarios -- are they plausible but unlikely (because forecasting is hard but not impossible), or are they implausible (because it would be very hard to model an _entire government_, and no one person is going to do it justice)? Note that according to the paper, the prior literature on role playing is quite positive (though of course it's talking about role playing in other contexts, e.g. business and military contexts). Still, this seems like quite an important question that strongly impacts how seriously I take the results of these role playing scenarios.


An Interview with Ben Garfinkel

Joshua Monrad
Mojmír Stehlík and Ben Garfinkel

AI seems poised to be a very big deal, possibly through the development of AGI, and it's very hard to forecast what would happen next. However, looking at history, we can see a few very large trajectory shifts, such as the Agricultural Revolution and Industrial Revolution, where everything changed radically. We shouldn't assume that such change must be for the better. Even though it's hard to predict what will happen, we can still do work that seems robustly good regardless of the specific long-term risk. For example, Ben is optimistic about research into avoiding adversarial dynamics between different groups invested in AI, research into how groups can make credible commitments, and better forecasting. However, credible commitments are probably less tractable for AI than with nukes or biological weapons because AI systems don't leave a large physical footprint, can easily proliferate, and are not a clear category that can be easily defined.


The Hacker Learns to Trust

An independent researcher attempted to replicate <@GPT-2@>(@Better Language Models and Their Implications@) and was planning to release the model. However, he has now decided not to release, because releasing would set a bad precedent. Regardless of whether or not GPT-2 is dangerous, at some point in the future, we will develop AI systems that really are dangerous, and we need to have adequate norms then that allow researchers to take their time and evaluate the potential issues and then make an informed decision about what to do. **Key quote:** "sending a message that it is ok, even celebrated, for a lone individual to unilaterally go against reasonable safety concerns of other researchers is not a good message to send".

I quite strongly agree that the most important impact of the GPT-2 decision was that it has started a discussion about what appropriate safety norms should be, whereas before there were no such norms at all. I don't know whether or not GPT-2 is dangerous, but I am glad that AI researchers have started thinking about whether and how publication norms should change.


When Is It Appropriate to Publish High-Stakes AI Research?

Claire Leibowicz
Steven Adler
and Peter Eckersley

Following the <@GPT-2 controversy@>(@Better Language Models and Their Implications@), the Partnership on AI held a dinner with OpenAI and other members of the AI community to discuss the tension between the norm of openness and the desire to mitigate potential unintended consequences and misuse risks of AI research. The post discusses some of the relevant considerations, and highlights a key conclusion: while there is not yet a consensus on on review norms for AI research, there *is* a consensus that **whatever the review norms are, they should be standardized across the AI community**.

I definitely agree that having everyone follow the same review norms is important: it doesn't do much good to hold back from publishing something problematic if a different group will publish all of the details a few weeks later. However, getting everyone to agree on a change to the existing norms seems incredibly hard to do, though it might be feasible if it was limited to only the largest actors who can engage deeply in the debate of what these norms should be.


Global AI Talent Report 2019

This report has a lot of statistics on the growth of the field of AI over the last year.


Building an AI World

This report summarises the AI strategies released by 18 different countries and regions. In particular, it rates how much emphasis they put on each of 8 areas. Broadly speaking, countries were most focused on Industrial strategy, Research, and AI talent (in that order), moderately focused on Ethics and Data, and least focused on Future of work, AI in governments, and Inclusion.

Since this report discusses neither its methodology nor the implications of its findings, it's difficult to know what conclusions to draw from it. The overall priorities seem to be roughly what I would have expected, except that I'm positively surprised by how much emphasis was placed on ethics.


Computational Power and the Social Impact of Artificial Intelligence

This paper contains a lot of introductory material about hardware in ML, plus details on where it's being made. It discusses the competition between China and the US to dominate hardware production. Hwang notes that the trend towards more specialised hardware may decrease the price of implementing ML but also decrease flexibility after deployment. He points out that simulation learning, self-play and meta-learning reduce the need for data at the expense of increased compute, which may increase hardware's importance going forward.

This may be useful for AI policy researchers, since it explores which hardware is being made by which companies in which locations, and some of the geopolitical implications. While it's a long paper, AI researchers could probably skip the first half without missing much.


Medium-Term Artificial Intelligence and Society

Like a <@previously summarized paper@>(@Beyond Near- and Long-Term: Towards a Clearer Account of Research Priorities in AI Ethics and Society@), this paper aims to find common ground between near-term and long-term priorities in medium-term concerns. This can be defined along several dimensions of an AI system: when it chronologically appears, how feasible it is to build it, how certain it is that we can build it, how capable the system is, how impactful the system is, and how urgent it is to work on it.

The paper formulates and evaluates the plausibility of the _medium term AI hypothesis_: that there is an intermediate time period in which AI technology and accompanying societal issues are important from both presentist and futurist perspectives. However, it does not come to a strong opinion on whether the hypothesis is true or not.


France's AI strategy


A framework for thinking about wireheading

Humans don't wirehead (take heroin, which gives huge positive reward) because it does not further their current goals. Maybe analogously we could design an AI that realizes that wireheading would not help it achieve its current goals and so wouldn't wirehead.

I think this is anthropomorphizing the AI too much. To the extent that a (current) reinforcement learning system can be said to "have goals", the goal is to maximize reward, so wireheading actually is furthering its current goal. It might be that in the future the systems we design are more analogous to humans and then such an approach might be useful.