At Uber, we apply neural networks to fundamentally improve how we understand the movement of people and things in cities. Among other use cases, we employ them to enable [faster customer service response](https://eng.uber.com/cota/) with natural language models and lower wait times via spatiotemporal prediction of demand across cities, and in the process have developed infrastructure to [scale up training](https://eng.uber.com/horovod-pyspark-apache-mxnet-support/) and support faster [model](https://eng.uber.com/introducing-ludwig/) [development](https://eng.uber.com/michelangelo-pyml/).

Though neural networks are powerful, widely used tools, many of their subtle properties are still poorly understood. As scientists around the world make strides towards [illuminating](https://openai.com/blog/introducing-activation-atlases/) [fundamental](https://arxiv.org/abs/1704.05796) [network](https://arxiv.org/abs/1805.12177) [properties](https://arxiv.org/abs/1802.08760), much of our research at [Uber AI](https://www.uber.com/us/en/uberai/) aligns in this direction as well, including our work [measuring intrinsic network complexity](https://eng.uber.com/intrinsic-dimension/), [finding more natural input spaces](https://eng.uber.com/neural-networks-jpeg/), and [uncovering hidden flaws in popular models](https://eng.uber.com/coordconv/).

In our most recent paper aimed at demystifying neural networks, [Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask](https://eng.uber.com/research/deconstructing-lottery-tickets-zeros-signs-and-the-supermask/), we build upon the fascinating [Lottery Ticket Hypothesis](https://arxiv.org/abs/1803.03635) developed by Frankle and Carbin. Their work surprised many researchers by showing that a very simple algorithm—delete small weights and retrain—can find sparse trainable subnetworks, or “lottery tickets”, within larger networks that perform as well as the full network. Although they clearly demonstrated lottery tickets to be effective, their work (as often occurs with great research) raised as many questions as it answered, and many of the underlying mechanics were not yet well understood. Our paper proposes explanations behind these mechanisms, uncovers curious quirks of these subnetworks, introduces competitive variants of the lottery ticket algorithm, and derives a surprising by-product: the Supermask.

### The Lottery Ticket Hypothesis

We begin by briefly summarizing Frankle and Carbin’s paper, [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://arxiv.org/abs/1803.03635), which we abbreviated as “LT”. In this paper, the authors proposed a simple approach for producing sparse, performant networks: after training a network, set all weights smaller than some threshold to zero (prune them), rewind the rest of the weights to their initial configuration, and then retrain the network from this starting configuration keeping the pruned weights weights frozen (not trained). Using this approach, they obtained two intriguing results.

First, they showed that the pruned networks performed well. Aggressively pruned networks (with 85 percent to 95 percent of weights pruned) showed no drop in performance compared to the much larger, unpruned network. Moreover, networks only moderately pruned (with 50 percent to 90 percent of weights pruned) often *outperformed* their unpruned counterparts.

Second, as compelling as these results were, the characteristics of the remaining network structure and weights were just as interesting. Normally, if you take a trained network, re-initialize it with random weights, and then re-train it, its performance will be about the same as before. But with the skeletal Lottery Ticket (LT) networks, this property does not hold. The network trains well only if it is rewound to its initial state, including the specific initial weights that were used. Reinitializing it with new weights causes it to train poorly. As pointed out in Frankle and Carbin’s study, it would appear that the specific combination of pruning mask (a per-weight binary value indicating whether or not to delete the weight) and weights underlying the mask form a lucky sub-network found within the larger network, or, as named by the original study, a winning “Lottery Ticket.”

We found this demonstration intriguing because of all left untold. What about LT networks causes them to show better performance? Why are the pruning mask and the initial set of weight so tightly coupled, such that re-initializing the network makes it less trainable? Why does simply selecting large weights constitute an effective criterion for choosing a mask? Would other criteria for creating a mask work, too?

### Curiously effective masks

We start our investigation with the observation of a curious phenomenon that demands explanation. While training LT networks, we observed that many of the rewound, masked networks had *accuracy significantly better than chance at initialization*. That is, an untrained network with a particular mask applied to it results in a partially working network.

This might come as a surprise, because if you use a randomly initialized and untrained network to, say, classify images of handwritten digits from the [MNIST dataset](https://en.wikipedia.org/wiki/MNIST_database), you would expect accuracy to be no better than chance (about 10%). But now imagine you multiply the network weights by a mask containing only zeros and ones. In this instance, weights are either unchanged or deleted entirely, but the resulting network now achieves nearly 40 percent accuracy at the task! This is strange, but it is exactly what happens when applying masks created using the procedure in the LT paper that selects weights with large final values (which we will call the “large final” mask criterion):

[![](https://blogapi.uber.com/wp-content/uploads/2022/08/image3-2-4.png)](https://blogapi.uber.com/wp-content/uploads/2022/08/image3-2-4.png)Figure 1. Untrained networks perform at chance (10 percent accuracy, for example, on the MNIST dataset as depicted), if they are randomly initialized, or randomly initialized and randomly masked. However, applying the Lottery Ticket mask improves the network accuracy beyond the chance level.

We call masks with the property that they immediately produce partially working networks without training of the underlying weights *Supermasks*.

As depicted in Figure 1, in randomly-initialized networks and randomly-initialized networks with random masks, neither weights nor the mask contain any information about the labels, so accuracy cannot reliably be better than chance. In randomly-initialized networks with LT “large final” masks, it is not entirely implausible to have better-than-chance performance since the masks are indeed derived from the training process. But it was unexpected since the only transmission of information from the training back to the initial network is via a zero-one mask, and the criterion for masking simply selects weights with large final magnitudes.

### Masking is training, or why zeros matter

So why do we see a large improvement in test accuracy from simply applying an LT mask?

The masking procedure as implemented in the LT paper performs two actions: it sets weights to zero, and it freezes them. By figuring out which of these two components leads to increased performance in *trained* networks, it turns out we’ll also uncover the principles underlying the peculiar performance of the untrained networks.

To separate the above two factors, we run a simple experiment: reproduce the LT iterative pruning experiments in which network weights are masked out in alternating train/mask/rewind cycles, but try an additional treatment: freeze zero-masked weights at their initial values instead of at zero. If zero isn’t special, both treatments should perform similarly. We follow Frankle and Carbin (2019) and train three convolutional neural networks (CNNs), Conv2, Conv4, and Conv6 (small CNNs with 2/4/6 convolutional layers, same as used in the LT paper), on [CIFAR-10](https://en.wikipedia.org/wiki/CIFAR-10).

Results are shown in Figure 2, below, with pruning (or more correctly, “freezing at some value”) progressing from unpruned on the left to very pruned networks on the right. The horizontal black lines represent the performance of the original, unpruned networks, averaged over five runs. The uncertainty bands here and in other figures represent minimum and maximum values over five runs. Solid blue lines represent networks trained using the LT algorithm, which sets pruned weights to zero and freezes them. Dotted blue lines represent networks trained using the LT algorithm except that pruned weights are frozen at their initial values:

[![](https://blogapi.uber.com/wp-content/uploads/2022/08/image4-4-1.png)](https://blogapi.uber.com/wp-content/uploads/2022/08/image4-4-1.png)Figure 2. When testing three CNNs on CIFAR-10, we find that the accuracy of networks with pruned weights frozen at their initial values degrades significantly more than those with pruned weights set to zero.

We see that networks perform better when weights are frozen specifically at zero rather than at random initial values. For these networks masked via the LT “large final” criterion, zero would seem to be a particularly good value to set weights to when they had small final values.

So why is zero an ideal value? One hypothesis is that the mask criterion we use *tends to mask to zero those weights that were headed toward zero anyway*. To test out this hypothesis, let’s consider a new approach to freezing. We run another experiment interpolated between the previous two: for any weight to be frozen, we freeze it to zero if it moved *toward* zero over the course of training, and we freeze it at its random initial value if it moved *away* from zero. Results are shown in Figure 3, below:

[![](https://blogapi.uber.com/wp-content/uploads/2022/08/image5-3-2.png)](https://blogapi.uber.com/wp-content/uploads/2022/08/image5-3-2.png)Figure 3: Selectively freezing weights to their initial value or zero depending on the direction they move during training produces better performance than freezing all weights at zero or init.

We see that this treatment performs just as well as the original LT networks, even though we did not freeze all the pruned weights to zero. In fact, if we apply this treatment to all weights, including weights we keep (that is, for all weights, initialize them at zero if they decreased in magnitude and keep their original initial values otherwise, then freeze pruned weights at their new initialization values), we get networks that perform even better than the LT networks!

This supports our hypothesis that the benefit derived from freezing values to zero comes from the fact that those values were moving toward zero anyway. For a deeper discussion of why the “large final” mask criterion biases toward selecting those weights heading toward zero, [see our paper](https://arxiv.org/abs/1905.01067).

Thus we find for certain mask criteria, like “large final”, that *masking is training*: the masking operation tends to move weights in the direction they would have moved during training.

This simultaneously explains why Supermasks exist and hints that other mask criteria may produce better Supermasks if they preferentially mask to zero weights that training drives toward zero.

### Alternate mask criteria

Now that we’ve explored why the original LT mask criterion, “large final,” works as well as it does, we can ask what other masking criteria would also perform well. The “large final” criterion keeps weights with large final magnitudes and sets the rest to zero. We can think of this pruning criterion and many others as a division of the 2D (wi = initial weight, wf = final weight) space into regions corresponding to weights that should be kept (mask-1) vs. pruned (mask-0), as shown in Figure 5, below:

[![](https://blogapi.uber.com/wp-content/uploads/2022/08/image5-1-7.png)](https://blogapi.uber.com/wp-content/uploads/2022/08/image5-1-7.png)Figure 5. Different mask criteria can be thought of as segmenting the (wi, wf) space into regions corresponding to mask values of one vs. zero. The ellipse represents in cartoon form the area occupied by the positively correlated initial and final weights from a given layer. The mask shown corresponds to a “large final” criterion, which was used in the LT paper: weights with large final magnitude are kept, and weights with final values near zero are pruned. Note that this criterion ignores the initial magnitude of the weight.

In the previous section, we showed some supporting evidence for the hypothesis that networks work well when those weights already moving toward zero are set to zero. This hypothesis suggests that other criteria may also work if they respect this basic rule. One such mask criterion is to preferentially keep those weights that move most away from zero, which we can write as the scoring function |wf| – |wi|. We call this criterion “magnitude increase” and depict it along with other criteria run as control cases in Figure 6, below:

[![](https://blogapi.uber.com/wp-content/uploads/2022/08/image1-1-10.png)](https://blogapi.uber.com/wp-content/uploads/2022/08/image1-1-10.png)Figure 6. The eight mask criteria considered in this study are shown, starting with the “large final” criterion that starred in the LT paper. Names we use to refer to the various methods are given along with the formula that projects each (wi, wf) pair to a score. Weights with the largest scores (colored regions) are kept, and weights with the smallest scores (gray regions) are pruned.

This “magnitude increase” criterion turns out to work just as well as the “large final” criterion, and in some cases significantly better. Results of all criteria are shown in Figure 7, below, for the fully connected (FC) and Conv4 networks; see [our paper](https://arxiv.org/abs/1905.01067) for performance results on other networks. As a baseline, we also show results on a random pruning criterion that simply chooses a random mask with the desired pruning percentage. Note that the first six criteria out of the eight form three opposing pairs; in each case, we see when one member of the pair performs better than the random baseline, the opposing member performs worse than it.

[![](https://blogapi.uber.com/wp-content/uploads/2022/08/image10-1-2.png)](https://blogapi.uber.com/wp-content/uploads/2022/08/image10-1-2.png)Figure 7. Measurements of the accuracy vs. pruning percentage for two networks, FC on MNIST (left) and Conv4 on CIFAR-10 (right), show that multiple mask criteria—large final, magnitude increase, and two others—reliably outperform the black random pruning baseline. In the Conv4 network, the performance boost of “magnitude increase” is larger than that of other mask criteria; asterisks mark where the difference between “large final” and “magnitude increase” is statistically significant at the p=0.05 level.

In general, we observe that those methods that bias towards keeping weights with large final magnitude are able to uncover performant subnetworks.

### Show me a sign

We have explored various ways of choosing which weights to prune and what values to set pruned weights to. We will now consider what values to set kept weights to. In particular, we want to explore an interesting observation in Frankle and Carbin (2019) which showed that the pruned, skeletal LT networks train well when you rewind to its original initialization, but degrades in performance when you randomly reinitialize the network.

Why does reinitialization cause LT networks to train poorly? Which components of the initialization are important?

We evaluate a number of variants of reinitialization to find out the answer.

* “Reinit” experiments: reinitialize kept weights based on the original initialization distribution
* “Reshuffle” experiments: reinitializing while respecting the original distribution of remaining weights in that layer, which is achieved by reshuffling the kept weights’ initial values
* “Constant” experiments: reinitializing by setting remaining weights values to a positive or negative constant, with the constant set to be the standard deviation of each layer’s original initialization

All of the reinitialization experiments are based on the same original networks and use the “large final” mask criterion with iterative pruning. We include the original LT network (rewind, large final) and the randomly pruned network (random) as baselines for comparison.

We find that none of these three variants alone are able to train as well as the original LT network, as shown in dashed lines in Figure 8 below:

[![](https://blogapi.uber.com/wp-content/uploads/2022/08/image7-1-4.png)](https://blogapi.uber.com/wp-content/uploads/2022/08/image7-1-4.png)Figure 8: We show test accuracy vs. pruning percentage of two networks, FC (left) and Conv4 (right), while using different reinitialization methods. A clear distinction of performances between those that respect the consistency of signs and those that do not suggests that the specific initial values of kept weights do not matter as much as their signs.

However, all three variants work better when we control the consistency of sign by ensuring that the reassigned values of the kept weights are of the same sign as their original initial values. These are shown as solid color lines in Figure 8. Clearly, the common factor in all the variants that perform better than chance, including the original “rewind”, is the sign. This suggests that reinitialization is not the deal breaker as long as you keep the sign. In fact, as long as we respect the original sign, even as simple as setting all kept weights to a constant value consistently performs well!

### Better Supermasks

At the beginning of the article we introduced the idea of Supermasks, which are binary masks that when applied to a randomly initialized network, produce better-than-chance accuracy without additional training. We now turn our attention to finding methods that would produce the best Supermasks.

We can evaluate the same pruning methods and pruning percentages seen in Figure 7 for their potential as Supermasks. For simplicity, we evaluate Supermasks based on one-shot pruning rather than iterative pruning. We can also consider additional mask criteria optimized for generating Supermasks. Based on the insight about the importance of the initial sign of LT weights and the idea of having weights close to their final values, we introduce a new mask criterion that selects for weights with large final magnitudes that also maintained the same sign at the end of training. This method is referred to as “large final, same sign”, and we depict it in Figure 9, below. We also add “large final, diff sign” as a control case, which looks for weights that changed sign at the end of training.

[![](https://blogapi.uber.com/wp-content/uploads/2022/08/image2-1-12.png)](https://blogapi.uber.com/wp-content/uploads/2022/08/image2-1-12.png)Figure 9. The “large final, same sign” mask criterion produces the highest performing Supermasks in this study. In contrast to the “large final” mask in Figure 5, note this criterion masks out the quadrants where the sign of wi and wf differ.

By using a simple mask criterion of “large final same sign”, we can create networks that obtain a remarkable 80 percent test accuracy on MNIST and 24 percent on CIFAR-10 without training. Another curious observation is that if we apply the mask to a signed constant (as described in the previous section) rather than the actual initial weights, we can produce even higher test accuracy of up to 86 percent on MNIST and 41 percent on CIFAR-10.

[![](https://blogapi.uber.com/wp-content/uploads/2022/08/image1-3-6.png)](https://blogapi.uber.com/wp-content/uploads/2022/08/image1-3-6.png)Figure 10: We evaluate accuracy at initialization (with no training) of a single FC network on MNIST subject to the application of various masks. The x-axis depicts the percent of weights remaining in the network; all other weights are set to zero. The “large final same sign” mask creates the highest performing Supermask by a wide margin. Note that aside from the five independent runs performed to generate uncertainty bands, every data point on this plot is the same underlying network, just with different masks applied.

We find it fascinating that these Supermasks exist and can be found via such simple criteria. Besides being a scientific curiosity, they could have implications for transfer learning and meta-learning — networks to approximately solve, say, any permutation of MNIST input pixels and permutation of output classes are all in there, just with different masks. They also present us with a method for network compression, since we only need to save a binary mask and a single random seed to reconstruct the full weights of the network.

If you’re curious how far we can push the performance of these Supermasks, check out [our paper](https://eng.uber.com/research/deconstructing-lottery-tickets-zeros-signs-and-the-supermask/) where we try training for them directly. If you’d like to run experiments similar to this paper, check out our [code](https://github.com/uber-research/deconstructing-lottery-tickets) and let us know what you find!

*If working with neural networks interests you, consider applying for a machine learning [role at Uber](https://www.uber.com/us/en/careers/list/?query=machine%20learning).*

*The authors would like to acknowledge Jonathan Frankle, Joel Lehman, and Sam Greydanus for combinations of helpful discussion and comments on early drafts of this work.*

![](https://blog.uber-cdn.com/cdn-cgi/image/width=2160,quality=80,onerror=redirect,format=auto/wp-content/uploads/2022/08/wd_bc32cbfe-e1b8-4620-905c-bbd5b9f973b4-N_BTe9rnrG.jpg)![](https://blog.uber-cdn.com/cdn-cgi/image/width=2160,quality=80,onerror=redirect,format=auto/wp-content/uploads/2022/08/bg_10480343-172f-48cc-8cd8-4d127a3b73b8-WHGtisBV9M.jpeg)![](https://blog.uber-cdn.com/cdn-cgi/image/width=2160,quality=80,onerror=redirect,format=auto/wp-content/uploads/2022/08/image3-8-1024x1024.jpg)![](https://blog.uber-cdn.com/cdn-cgi/image/width=2160,quality=80,onerror=redirect,format=auto/wp-content/uploads/2022/08/jason-yosinski.jpg)Posted by Hattie Zhou, Janice Lan, Rosanne Liu, Jason Yosinski

Category: [Engineering](/en-PL/blog/engineering/), [AI](/en-PL/blog/ai/)


post
 6 minute read
ICLR Safe ML Workshop Report
============================

![](https://futureoflife.org/wp-content/uploads/2019/06/cropped-pagodas.jpg)PublishedJune 18, 2019AuthorViktoriya Krakovna

This year the ICLR conference hosted topic-based workshops for the first time (as opposed to a single track for workshop papers), and I co-organized the [Safe ML workshop](https://sites.google.com/corp/view/safeml-iclr2019/home?authuser=0). One of the main goals was to bring together near and long term safety research communities.

![](https://futureoflife.org/wp-content/uploads/2019/06/near-long-term.png)The workshop was structured according to a taxonomy that incorporates both near and long term safety research into three areas — specification, robustness, and assurance.

**Specification:** define the purpose of the system

* Reward hacking
* Side effects
* Preference learning
* Fairness

**Robustness:** design system to withstand perturbations

* Adaptation
* Verification
* Worst-case robustness
* Safe exploration

**Assurance:** monitor and control system activity

* Interpretability
* Monitoring
* Privacy
* Interruptibility

We had an invited talk and a contributed talk in each of the three areas.

Talks
-----

In the **specification** area, Dylan Hadfield-Menell spoke about [formalizing the value alignment problem](https://slideslive.com/38915783/formalizing-the-value-alignment-problem-in-ai) in the Inverse RL framework.

![](https://futureoflife.org/wp-content/uploads/2019/06/formalizing-value-alignment.png)

David Krueger [presented](https://slideslive.com/38915784/misleading-metaobjectives-and-hidden-incentives-for-distributional-shift) a [paper](https://drive.google.com/uc?export=download&id=1k93292JCoIHU0h6xVO3qmeRwLyOSlS4o) on hidden incentives for the agent to shift its task distribution in the meta-learning setting.

![](https://futureoflife.org/wp-content/uploads/2019/06/distributional-shift.png)

In the **robustness** area, Ian Goodfellow argued for [dynamic defenses against adversarial examples](https://slideslive.com/38915790/the-case-for-dynamic-defenses-against-adversarial-examples) and encouraged the research community to consider threat models beyond small perturbations within a norm ball of the original data point.

![](https://futureoflife.org/wp-content/uploads/2019/06/threat-models.png)

Avraham Ruderman [presented](https://slideslive.com/38915789/uncovering-surprising-behaviors-in-reinforcement-learning-via-worstcase-analysis) a [paper](https://drive.google.com/uc?export=download&id=1z_d1EjvKWlh2L39xxGXYDWP1wQwnlNLQ) on worst-case analysis for discovering surprising behaviors (e.g. failing to find the goal in simple mazes).

![](https://futureoflife.org/wp-content/uploads/2019/06/mazes.png)

In the **assurance** area, Cynthia Rudin argued that interpretability doesn’t have to trade off with accuracy (especially in applications), and that it is helpful for solving research problems in all areas of safety.

![](https://futureoflife.org/wp-content/uploads/2019/06/interpretability-taxonomy.png)

Beomsu Kim [presented](https://slideslive.com/38915785/bridging-adversarial-robustness-and-gradient-interpretability) a [paper](https://drive.google.com/uc?export=download&id=1FUlKR07jf1VQb6M-4VVTxxu3oDkK0x-8) explaining why adversarial training improves the interpretability of gradients for deep neural networks.

![](https://futureoflife.org/wp-content/uploads/2019/06/gradient-interpretability.png)

Panels
------

The workshop panels discussed possible overlaps between different research areas in safety and research priorities going forward.

In terms of **overlaps**, the main takeaway was that advancing interpretability is useful for all safety problems. Also, adversarial robustness can contribute to value alignment – e.g. reward gaming behaviors can be viewed as a system finding adversarial examples for its reward function. However, there was a cautionary point that while near- and long-term problems are often similar, solutions might not transfer well between these areas (e.g. some solutions to near-term problems might not be sufficiently general to help with value alignment).

![](https://futureoflife.org/wp-content/uploads/2019/06/panel.jpg)

The **research priorities** panel recommended more work on adversarial examples with realistic threat models (as mentioned above), complex environments for testing value alignment (e.g. creating new structures in Minecraft without touching existing ones), fairness formalizations with more input from social scientists, and improving cybersecurity.

Papers
------

Out of the 35 accepted papers, 5 were on long-term safety / value alignment, and the rest were on near-term safety. Half of the near-term paper submissions were on adversarial examples, so the resulting pool of accepted papers was skewed as well: 14 on adversarial examples, 5 on interpretability, 3 on safe RL, 3 on other robustness, 2 on fairness, 2 on verification, and 1 on privacy. Here is a summary of the value alignment papers:

[Misleading meta-objectives and hidden incentives for distributional shift](https://drive.google.com/uc?export=download&id=1k93292JCoIHU0h6xVO3qmeRwLyOSlS4o) by Krueger et al shows that RL agents in a meta-learning context have an incentive to shift their task distribution instead of solving the intended task. For example, a household robot whose task is to predict whether its owner will want coffee could wake up its owner early in the morning to make this prediction task easier. This is called a ‘self-induced distributional shift’ (SIDS), and the incentive to do so is a ‘hidden incentive for distributional shift’ (HIDS). The paper demonstrates this behavior experimentally and shows how to avoid it.

![](https://futureoflife.org/wp-content/uploads/2019/06/coffee-robot.png)

[How useful is quantilization for mitigating specification-gaming?](https://drive.google.com/uc?export=download&id=13qAfOm8McRvXS33MCNH0ia4ApMIClZP9) by Ryan Carey introduces variants of several classic environments (Mountain Car, Hopper and Video Pinball) where the observed reward differs from the true reward, creating an opportunity for the agent to game the specification of the observed reward. The paper shows that a quantilizing agent avoids specification gaming and performs better in terms of true reward than both imitation learning and a regular RL agent on all the environments.

![](https://futureoflife.org/wp-content/uploads/2019/06/spec-gaming.png)

[Delegative Reinforcement Learning: learning to avoid traps with a little help](https://drive.google.com/uc?export=download&id=1xa7UpGGODl6mszNWkA4XQGPyeopsNuWu) by Vanessa Kosoy introduces an RL algorithm that avoids traps in the environment (states where regret is linear) by delegating some actions to an external advisor, and achieves sublinear regret in a continual learning setting. (Summarized in [Alignment Newsletter #57](http://eepurl.com/gtZ8DD))

![](https://futureoflife.org/wp-content/uploads/2019/06/trap.gif)

[Generalizing from a few environments in safety-critical reinforcement learning](https://drive.google.com/uc?export=download&id=1Q_lGKZzJwc7h2f8oYgZDBAmRttMBkgCk) by Kenton et al investigates how well RL agents avoid catastrophes in new gridworld environments depending on the number of training environments. They find that both model ensembling and learning a catastrophe classifier (used to block actions) are helpful for avoiding catastrophes, with different safety-performance tradeoffs on new environments.

![](https://futureoflife.org/wp-content/uploads/2019/06/dist-shift-catastrophes.png)

[Regulatory markets for AI safety](https://drive.google.com/uc?export=download&id=1bFPiwLrZc7SQTMg2_bW4gt0PaS5NyqOH) by Clark and Hadfield proposes a new model for regulating AI development where regulation targets are required to choose regulatory services from a private market that is overseen by the government. This allows regulation to efficiently operate on a global scale and keep up with the pace of technological development and better ensure safe deployment of AI systems. (Summarized in [Alignment Newsletter #55](http://eepurl.com/gp5MFP))

![](https://futureoflife.org/wp-content/uploads/2019/06/regulatory-markets.png)

The workshop got a pretty good turnout (around 100 people). Thanks everyone for participating, and thanks to our reviewers, sponsors, and my fellow organizers for making it happen!

![](https://futureoflife.org/wp-content/uploads/2019/06/audience-1250x630.jpg)

*(Cross-posted from the [Deep Safety blog](https://vkrakovna.wordpress.com/2019/06/18/iclr-safe-ml-workshop-report/).)*

#### Contents

Our contentRelated posts
-------------

If you enjoyed this, you also might like:[Our content](https://futureoflife.org/our-content/)[![](https://futureoflife.org/wp-content/uploads/2023/06/Podcast-thumbnails-45-1-1024x576.jpg)](https://futureoflife.org/podcast/dan-hendrycks-on-why-evolution-favors-ais-over-humans/)podcast#### [Dan Hendrycks on Why Evolution Favors AIs over Humans](https://futureoflife.org/podcast/dan-hendrycks-on-why-evolution-favors-ais-over-humans/)

June 8, 2023[![](https://futureoflife.org/wp-content/uploads/2023/05/Podcast-thumbnails-1-1-1024x576.jpg)](https://futureoflife.org/podcast/nathan-labenz-on-how-ai-will-transform-the-economy/)podcast#### [Nathan Labenz on How AI Will Transform the Economy](https://futureoflife.org/podcast/nathan-labenz-on-how-ai-will-transform-the-economy/)

May 12, 2023[![](https://futureoflife.org/wp-content/uploads/2023/03/FLi_Banner-05-1024x576.jpg)](https://futureoflife.org/ai/faqs-about-flis-open-letter-calling-for-a-pause-on-giant-ai-experiments/)post#### [FAQs about FLI’s Open Letter Calling for a Pause on Giant AI Experiments](https://futureoflife.org/ai/faqs-about-flis-open-letter-calling-for-a-pause-on-giant-ai-experiments/)

Disclaimer: Please note that these FAQ’s have been prepared by FLI and do not necessarily reflect the views of the […]March 31, 2023

ICLR Safe ML Workshop Report


[All Open Letters](https://futureoflife.org/?page_id=39741)Research Priorities for Robust and Beneficial Artificial Intelligence: An Open Letter
=====================================================================================

There is now a broad consensus that AI research is progressing steadily, and that its impact on society is likely to increase. The potential benefits are huge, since everything that civilization has to offer is a product of human intelligence. Because of the great potential of AI, it is important to research how to reap its benefits while avoiding potential pitfalls.Signatures11251PublishedOctober 28, 2015

Click here to see this page in other languages: **[Chinese](https://futureoflife.org/ai-open-letter-chinese/)  [![](https://futureoflife.org/wp-content/uploads/2016/06/china_flag-e1464798319604.png)](https://futureoflife.org/ai-open-letter-chinese/) [German](https://futureoflife.org/ai-open-letter-german/)[![](https://futureoflife.org/wp-content/uploads/2016/06/Germany_flag.jpg?x57718)](https://futureoflife.org/ai-open-letter-german/) [Japanese](https://futureoflife.org/ai-open-letter-japanese/) [![](https://futureoflife.org/wp-content/uploads/2016/02/red_circle-1.jpg)](https://futureoflife.org/ai-open-letter-japanese/) [Russian](https://futureoflife.org/ai-open-letter-russian/) [![](https://futureoflife.org/wp-content/uploads/2016/02/Russian_Flag.jpg?x56934)](https://futureoflife.org/ai-open-letter-russian/)**

Artificial intelligence (AI) research has explored a variety of problems and approaches since its inception, but for the last 20 years or so has been focused on the problems surrounding the construction of intelligent agents - systems that perceive and act in some environment. In this context, "intelligence" is related to statistical and economic notions of rationality - colloquially, the ability to make good decisions, plans, or inferences. The adoption of probabilistic and decision-theoretic representations and statistical learning methods has led to a large degree of integration and cross-fertilization among AI, machine learning, statistics, control theory, neuroscience, and other fields. The establishment of shared theoretical frameworks, combined with the availability of data and processing power, has yielded remarkable successes in various component tasks such as speech recognition, image classification, autonomous vehicles, machine translation, legged locomotion, and question-answering systems.

As capabilities in these areas and others cross the threshold from laboratory research to economically valuable technologies, a virtuous cycle takes hold whereby even small improvements in performance are worth large sums of money, prompting greater investments in research. There is now a broad consensus that AI research is progressing steadily, and that its impact on society is likely to increase. The potential benefits are huge, since everything that civilization has to offer is a product of human intelligence; we cannot predict what we might achieve when this intelligence is magnified by the tools AI may provide, but the eradication of disease and poverty are not unfathomable. Because of the great potential of AI, it is important to research how to reap its benefits while avoiding potential pitfalls.

The progress in AI research makes it timely to focus research not only on making AI more capable, but also on maximizing the societal benefit of AI. Such considerations motivated the AAAI 2008-09 Presidential Panel on Long-Term AI Futures and other projects on AI impacts, and constitute a significant expansion of the field of AI itself, which up to now has focused largely on techniques that are neutral with respect to purpose. We recommend expanded research aimed at ensuring that increasingly capable AI systems are robust and beneficial: our AI systems must do what we want them to do. The attached [research priorities document](https://futureoflife.org/static/data/documents/research_priorities.pdf) gives many examples of such research directions that can help maximize the societal benefit of AI. This research is by necessity interdisciplinary, because it involves both society and AI. It ranges from economics, law and philosophy to computer security, formal methods and, of course, various branches of AI itself.

In summary, we believe that research on how to make AI systems robust and beneficial is both important and timely, and that there are concrete research directions that can be pursued today.

*If you have questions about this letter, please contact [Max Tegmark](mailto:max@futureoflife.org).*

Signatories
-----------

[Click here](https://futureoflife.org/open-letter/ai-open-letter-signatories/) to view the full list of signatories.

To date, the open letter has been signed by over 8,000 people. The list of signatories includes:

### Prominent Signatories

**Stuart Russell**, Berkeley, Professor of Computer Science, director of the Center for Intelligent Systems, and co-author of the standard textbook Artificial Intelligence: a Modern Approach.

**Tom Dietterich**, Oregon State, President of AAAI, Professor and Director of Intelligent Systems

**Eric Horvitz**, Microsoft research director, ex AAAI president, co-chair of the AAAI presidential panel on long-term AI futures

**Bart Selman**, Cornell, Professor of Computer Science, co-chair of the AAAI presidential panel on long-term AI futures

**Francesca Rossi**, Padova & Harvard, Professor of Computer Science, IJCAI President and Co-chair of AAAI committee on impact of AI and Ethical Issues

**Demis Hassabis**, co-founder of DeepMind

**Shane Legg**, co-founder of DeepMind

**Mustafa Suleyman**, co-founder of DeepMind

**Dileep George**, co-founder of Vicarious

**Scott Phoenix**, co-founder of Vicarious

**Yann LeCun**, head of Facebook’s Artificial Intelligence Laboratory

**Geoffrey Hinton**, University of Toronto and Google Inc.

**Yoshua Bengio**, Université de Montréal

**Peter Norvig**, Director of research at Google and co-author of the standard textbook Artificial Intelligence: a Modern Approach

**Oren Etzioni**, CEO of Allen Inst. for AI

**Guruduth Banavar**, VP, Cognitive Computing, IBM Research

**Michael Wooldridge**, Oxford, Head of Dept. of Computer Science, Chair of European Coordinating Committee for Artificial Intelligence

**Leslie Pack Kaelbling**, MIT, Professor of Computer Science and Engineering, founder of the Journal of Machine Learning Research

**Tom Mitchell**, CMU, former President of AAAI, chair of Machine Learning Department

**Toby Walsh**, Univ. of New South Wales & NICTA, Professor of AI and President of the AI Access Foundation

**Murray Shanahan**, Imperial College, Professor of Cognitive Robotics

**Michael Osborne**, Oxford, Associate Professor of Machine Learning

**David Parkes**, Harvard, Professor of Computer Science

**Laurent Orseau**, Google DeepMind

**Ilya Sutskever**, Google, AI researcher

**Blaise Aguera y Arcas**, Google, AI researcher

**Joscha Bach**, MIT, AI researcher

**Bill Hibbard**, Madison, AI researcher

**Steve Omohundro**, AI researcher

**Ben Goertzel**, OpenCog Foundation

**Richard Mallah**, Cambridge Semantics, Director of Advanced Analytics, AI researcher

**Alexander Wissner-Gross**, Harvard, Fellow at the Institute for Applied Computational Science

**Adrian Weller**, Cambridge, AI researcher

**Jacob Steinhardt**, Stanford, AI Ph.D. student

**Nick Hay**, Berkeley, AI Ph.D. student

**Jaan Tallinn**, co-founder of Skype, CSER and FLI

**Elon Musk**, SpaceX, Tesla Motors

**Steve Wozniak**, co-founder of Apple

**Luke Nosek**, Founders Fund

**Aaron VanDevender**, Founders Fund

**Erik Brynjolfsson**, MIT, Professor at and director of MIT Initiative on the Digital Economy

**Margaret Boden**, U. Sussex, Professor of Cognitive Science

**Martin Rees**, Cambridge, Professor Emeritus of Cosmology and Astrophysics, Gruber & Crafoord laureate

**Huw Price**, Cambridge, Bertrand Russell Professor of Philosophy

**Nick Bostrom**, Oxford, Professor of Philosophy, Director of Future of Humanity Institute (Oxford Martin School)

**Stephen Hawking**, Director of research at the Department of Applied Mathematics and Theoretical Physics at Cambridge, 2012 Fundamental Physics Prize laureate for his work on quantum gravity

**Luke Muehlhauser**, Executive Director of Machine Intelligence Research Institute (MIRI)

**Eliezer Yudkowsky**, MIRI researcher, co-founder of MIRI (then known as SIAI)

**Katja Grace**, MIRI researcher

**Benja Fallenstein**, MIRI researcher

**Nate Soares**, MIRI researcher

**Paul Christiano**, Berkeley, Computer Science graduate student

**Anders Sandberg**, Oxford, Future of Humanity Institute researcher (Oxford Martin School)

**Daniel Dewey**, Oxford, Future of Humanity Institute researcher (Oxford Martin School)

**Stuart Armstrong**, Oxford, Future of Humanity Institute researcher (Oxford Martin School)

**Toby Ord**, Oxford, Future of Humanity Institute researcher (Oxford Martin School), Founder of Giving What We Can

**Neil Jacobstein**, Singularity University

**Dominik Grewe**, Google DeepMind

**Roman V. Yampolskiy**, University of Louisville

**Vincent C. Müller**, ACT/Anatolia College

**Amnon H Eden**, University Essex

**Henry Kautz**, University of Rochester

**Boris Debic**, Google, Chief History Officer

**Kevin Leyton-Brown**, University of British Columbia, Professor of Computer Science

**Trevor Back**, Google DeepMind

**Moshe Vardi**, Rice University, editor-in-chief of Communications of the ACM

**Peter Sincak**, prof. TU Kosice, Slovakia

**Tom Schaul**, Google DeepMind

**Grady Booch**, IBM Fellow

**Alan Mackworth**, Professor of Computer Science, University of British Columbia. Ex AAAI President

**Andrew Davison**, Professor of Robot Vision, Director of the Dyson Robotics Lab at Imperial College London

**Daniel Weld**, WRF / TJ Cable Professor of Computer Science & Engineering, University of Washington

**Michael Witbrock**, Cycorp Inc & AI4Good.org

**Stephen L. Reed**, ai-coin.com

**Thomas Stone**, Co-founder of PredictionIO

**Dan Roth**, University of Illinois, Editor in Chief of The Journal of AI Research (JAIR)

**Babak Hodjat**, Sentient Technologies

**Vincent Vanhoucke**, Google, AI researcher

**Itamar Arel**, Stanford University, Prof. of Computer Science

**Ramon Lopez de Mantaras**, Director of the Artificial Intelligence Research Institute, Spanish National Research Council

**Antoine Blondeau**, Sentient Technologies

**George Dvorsky**, Contributing Editor, io9; Chair of the Board, Institute for Ethics and Emerging Technologies

**George Church**, Harvard & MIT

**Klaus-Dieter Althoff**, University of Hildesheim, Professor of Artificial Intelligence; Head of Competence Center Case-Based Reasoning, German Research Center for Artificial Intelligence, Kaiserslautern; Editor-in-Chief German Journal on Artificial Intelligence

**Christopher Bishop**, Distinguished Scientist, Microsoft Research

**Jen-Hsun Huang**, NVIDIA CEO

[Close](http://)### How does verification work?

Verified signatures are those which we have taken one or more extra steps to confirm as legitimate:
• **Direct contact** - We have been in direct contact with this person to verify that they have signed the letter.
• **Declaration URL** - This person has made a public declaration of signing the open letter which can be viewed online.
All published signatures, ‘verified’ or otherwise, are subject to several forms of verification: email verification, spam and duplicate filters, and a review by a member of our data vetting team.

OPEN LETTERSRelated posts
-------------

If you enjoyed this, you also might like:[Our Open Letters](https://futureoflife.org/fli-open-letters/)Signatories31810#### [Pause Giant AI Experiments: An Open Letter](https://futureoflife.org/open-letter/pause-giant-ai-experiments/)

We call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4.Taylor JonesMarch 22, 2023Signatories998#### [Open Letter Against Reckless Nuclear Escalation and Use](https://futureoflife.org/open-letter/open-letter-against-reckless-nuclear-escalation-and-use/)

The abhorrent Ukraine war has the potential to escalate into an all-out NATO-Russia nuclear conflict that would be the greatest catastrophe in human history. More must be done to prevent such escalation.Taylor JonesOctober 18, 2022SignatoriesClosed#### [Foresight in AI Regulation Open Letter](https://futureoflife.org/open-letter/foresight-in-ai-regulation-open-letter/)

The emergence of artificial intelligence (AI) promises dramatic changes in our economic and social structures as well as everyday life […]Anna YelizarovaJune 14, 2020Signatories276#### [Autonomous Weapons Open Letter: Global Health Community](https://futureoflife.org/open-letter/medical-lethal-autonomous-weapons-open-letter/)

Given our commitment to do no harm, the global health community has a long history of successful advocacy against inhumane weapons, and the World and American Medical Associations have called for bans on nuclear, chemical and biological weapons. Now, recent advances in artificial intelligence have brought us to the brink of a new arms race in lethal autonomous weapons.adminMarch 13, 2019

Research priorities for robust and beneficial artificial intelligence: an open letter


![](http://gcrinstitute.org/wp-content/uploads/2021/01/AGI-2020-map-1024x508.png)
[View the paper “2020 Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy”](https://gcrinstitute.org/papers/055_agi-2020.pdf)

In 2017, GCRI published [the first-ever survey](https://gcrinstitute.org/a-survey-of-artificial-general-intelligence-projects-for-ethics-risk-and-policy/) of artificial general intelligence (AGI) research and development (R&D) projects for ethics, risk, and policy. This paper updates the 2017 survey. The 2020 survey features improved methodology, enabling it to find more projects than the 2017 survey and characterize them more precisely. The 2020 survey also evaluates how the landscape of AGI R&D projects has changed from 2017 to 2020.

AGI is AI that can reason across a wide range of domains. Most current AI R&D is narrow, but as the 2017 and 2020 surveys both document, there is a significant amount of dedicated AGI R&D. AGI is important because, if built, it could have major consequences. Depending on how it is designed and built, it may be able to help solve many of the world’s problems, including problems involving global catastrophic risk, or it could itself cause global catastrophe. Therefore, it is important to monitor AGI R&D and identify opportunities to orient it in better directions.

The 2017 and 2020 surveys characterize AGI R&D projects in terms of seven attributes:

 • The type of institution the project is based in

 • Whether the project publishes open-source code

 • Whether the project has military connections

 • The nation(s) that the project is based in

 • The project’s goals for its AGI

 • The extent of the project’s engagement with AGI safety issues

 • The overall size of the project

To accomplish this, the surveys use openly published information as found in scholarly publications, project websites, popular media articles, and other websites. The 2020 survey uses information from the 2017 survey as well as the past three years of the *Journal of Artificial General Intelligence*, the past three years of AGI conference proceedings (the 2017 survey covered prior content from the *Journal of Artificial General Intelligence* and the AGI conference proceedings), keyword searches in Google web search, Google Scholar, Crunchbase, GitHub, the authors’ prior knowledge, suggestions from readers of the 2017 survey, and additional literature and webpages identified via all of the above. The use of Crunchbase, GitHub, and reader suggestions is new to the 2020 survey.

Whereas the 2017 survey identified 45 AGI R&D projects spread across 30 countries, the 2020 survey finds that, in 2020, there are 72 AGI R&D projects spread across 37 countries. The 2020 further finds that in 2017, there were 70 AGI R&D projects spread across 36 countries. 57 of the projects active in 2017 remain active in 2020, with an additional 15 projects new to 2020. The projects vary widely in size, with the largest being over 100 times larger than the smallest as measured in terms of the number of project personnel.

Relative to the 2017 survey, the AGI R&D projects presented in the 2020 survey tend to be smaller, more geographically diverse, less open-source, less focused on intellectual goals, more focused on humanitarian goals, and more concentrated in private corporations.

The 2020 survey also finds that, from 2017 to 2020, there has been a decrease in academic projects, an increase in private corporation projects, an increase in projects stating humanitarian goals, a decrease in projects with military connections, and a decrease in projects based in the United States (though the US remains the dominant country in AGI R&D); all of these changes are relatively small compared to the differences between the 2017 and 2020 surveys.

The projects active in 2020 are diverse, with three major clusters: (1) corporate projects that are active on AGI safety and state that their goals are to benefit humanity, (2) academic projects that are not active on AGI safety and state that their goals are to advance the forefront of knowledge, and (3) small private corporations that are not active on AGI safety and state a range of different goals. Governments and nonprofits play relatively minor roles in AGI R&D. The 2020 survey continues to observe an absence of large government AGI R&D projects, including military projects. The small handful of projects with military connections mostly involve basic research. The data show no indication of militaries or other government divisions pursuing AGI R&D for major strategic purposes.

The data suggest the following conclusions:

**Regarding ethics,** the two most common goals are to benefit humanity and to advance knowledge. This is the same as in the 2017 survey, but in the 2020 survey, the order is reversed, with there now being more projects seeking to benefit humanity. The 2020 survey also finds a large increase in the number of corporate projects. These projects seldom state a goal of pursuing profit, but they may nonetheless have profit as a motivation.

**Regarding risk,** the proliferation of corporate projects relative to the 2017 survey heightens the concern that these projects could put profit ahead of safety and the public interest. Additionally, academic projects remain relatively inattentive to safety. On the other hand, many projects are active on safety. Additionally, the partial consensus on ethics, the concentration of projects in the US and its allies, and the various interconnections between different projects all suggest potential for cooperation on safety issues; these matters are unchanged from the 2017 survey.

**Regarding policy,** the proliferation of corporate projects suggests an important role for corporate governance and attention to the political economy of AGI R&D. The modest decline of academic projects suggests a smaller but still significant role for academic research policy. Additionally, as in the 2017 survey, international policy is facilitated by the concentration of projects in the US and its allies, though the preponderance of projects with open-source code complicates the political geography of AGI R&D. Finally, the absence of large government AGI R&D projects suggests that governments may be involved in AGI R&D primarily as regulators of private-sector R&D instead of as drivers of the R&D.

As with the 2017 survey, the 2020 survey has some limitations, meaning that the actual state of AGI R&D may differ from what is presented in the surveys. This is due to the fact that the surveys are based exclusively on openly published information. It is possible that some AGI R&D projects were missed by the surveys. Indeed, the 2020 survey documents many projects that were missed by the 2017 survey. Therefore, the number of projects identified in the 2020 survey should be taken as a lower bound. Furthermore, it is possible that projects’ actual attributes differ from those found in openly published information. For example, most corporate projects did not state the goal of profit, even though many presumably seek profit. Therefore, this study’s results should not be assumed to necessarily reflect the actual current state of AGI R&D. That said, the study nonetheless provides the most thorough description yet of AGI R&D in terms of ethics, risk, and policy.

This paper was recently covered in the media article [What Is AGI?](https://www.fierceelectronics.com/sensors/what-agi).

Academic citation:
Fitzgerald, McKenna, Aaron Boddy, and Seth D. Baum, 2020. [2020 survey of artificial general intelligence projects for ethics, risk, and policy](https://gcrinstitute.org/papers/055_agi-2020.pdf). Global Catastrophic Risk Institute Technical Report 20-1.

 Tagged with [artificial general intelligence](https://gcrinstitute.org/tag/artificial-general-intelligence/)

2020 Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy | Global Catastrophic Risk Institute


![](http://gcrinstitute.org/wp-content/uploads/2021/06/ComputerLeaf_Buckyball_2021-06-04.jpg)
[View the paper “Artificial Intelligence Needs Environmental Ethics”](https://gcrinstitute.org/papers/059_ai-environmental-ethics.pdf)

Artificial intelligence is an interdisciplinary topic. As such, it benefits from contributions from a wide range of disciplines. This short paper calls for greater contributions from the discipline of environmental ethics and presents several types of contributions that environmental ethicists can make.

First, environmental ethicists can raise the profile of the environmental dimensions of AI. For example, discussions of the ethics of autonomous vehicles have thus far focused mainly on “trolley problem” scenarios in which the vehicle must decide whom to harm in hypothetical crash scenarios. Less attention has been paid to the environmental impacts of autonomous vehicles, even though the environmental impacts are arguably much more important. Environmental ethicists could make the moral case for attention to this and to other environmental issues involving AI.

Second, environmental ethicists can help analyze novel ethical situations involving AI. These situations specifically involve artificial versions of phenomena that have long been studied in environmental ethics research. For example, AI and related technology could result in the creation of artificial life and artificial ecosystems. Environmental ethicists have often argued that “natural” life and ecosystems have important moral value. Perhaps the same reasoning would apply to artificial life and ecosystems, or perhaps it would not due to their artificiality. Environmental ethicists can help evaluate these sorts of novel ethical issues. Such work is especially important because existing work on AI ethics has focused more narrowly on issues centered on humans; environmental ethicists can help make the case for a broader scope.

Third, environmental ethicists can provide valuable perspectives on the future-orientation of certain AI issues. Within the communities of people working on AI issues, there is a divide between people focused on near-term AI issues and people focused on long-term AI issues. Global catastrophic risks associated with AI are often linked to the long-term AI issues. Similar debates exist on environmental issues, due to the long-term nature of major environmental issues such as global warming, natural resource depletion, and biodiversity loss. Environmental ethicists have made considerable progress on the ethics of the future, which can be applied to debates about AI.

The paper builds on prior GCRI research and experience as environmental ethicists working on AI. [Moral consideration of nonhumans in the ethics of artificial intelligence](https://gcrinstitute.org/moral-consideration-of-nonhumans-in-the-ethics-of-artificial-intelligence) documents the tendency for work on AI ethics to focus on humans and calls for more robust attention to nonhumans. [Reconciliation between factions focused on near-term and long-term artificial intelligence](https://gcrinstitute.org/reconciliation-between-factions-focused-on-near-term-and-long-term-artificial-intelligence) describes the debate between those favoring attention to near-term AI issues and those favoring attention to long-term AI issues. [Artificial intelligence, systemic risks, and sustainability](https://gcrinstitute.org/artificial-intelligence-systemic-risks-and-sustainability) analyzes risks associated with near-term applications of AI in sectors related to environmental sustainability such as agriculture and forestry.

Academic citation:
Baum, Seth D. and Andrea Owe, forthcoming. [Artificial intelligence needs environmental ethics](https://gcrinstitute.org/papers/059_ai-environmental-ethics.pdf). *Ethics, Policy, & Environment*, [DOI 10.1080/21550085.2022.2076538](https://doi.org/10.1080/21550085.2022.2076538).

 Tagged with [artificial intelligence](https://gcrinstitute.org/tag/artificial-intelligence/), [ethics](https://gcrinstitute.org/tag/ethics/)

Artificial Intelligence Needs Environmental Ethics | Global Catastrophic Risk Institute


![](http://gcrinstitute.org/wp-content/uploads/2021/07/Motion-earth3d-1_Scr-4-1024x576.jpg)
[View the paper “Collective Action on Artificial Intelligence: A Primer and Review”](https://gcrinstitute.org/papers/058_collective-action.pdf)

The development of safe and socially beneficial artificial intelligence (AI) will require collective action: outcomes will depend on the actions that many different people take. In recent years, a sizable but disparate literature has looked at the challenges posed by collective action on AI, but this literature is generally not well grounded in the broader social science literature on collective action. This paper advances the study of collective action on AI by providing a primer on the topic and a review of existing literature. It is intended to get an interdisciplinary readership up to speed on the topic, including social scientists, computer scientists, policy analysts, government officials, and other interested people.

The primer describes the theory of
collective action and relates it to different types of AI collective action
situations. A primary distinction is between situations in which individual and
collective interests diverge, as in the [prisoner’s
dilemma](https://en.wikipedia.org/wiki/Prisoner's_dilemma) or adversarial AI competition, and in
which they converge, as in [coordination
problems](https://en.wikipedia.org/wiki/Coordination_game) such as establishing common
platforms for AI. In general, collective action is easier to achieve when
interests converge, because individual actors pursuit of their own
self-interest can lead to outcomes that are worse for the group as a whole. The
primer also explains how AI collective action situations depend both on whether
the goods involved are [excludable](https://en.wikipedia.org/wiki/Excludability) or [rivalrous](https://en.wikipedia.org/wiki/Rivalry_(economics)) and
whether they hinge on the action of a single actor or on some combination of
actors.

One major focus of the AI collective action literature identified in this paper are potentially dangerous AI race scenarios. AI races are not necessarily dangerous and might even hasten the arrival of socially beneficial forms of AI, but could be dangerous if individual actors’ interest in developing AI quickly diverges from the collective interest in ensuring that AI is safe and socially beneficial. The paper looks at both near-term and long-term AI races. The literature identified in this paper looks in particular at near-term races to develop military applications and at long-term AI races to develop advanced forms of AI such as artificial general intelligence and superintelligence. The two types of races are potentially related, since near-term races could affect the long-term development of AI.

Finally, the paper evaluates different
types of potential solutions to collective action problems. The collective
action literature identifies three major types of solution: government
regulation, private markets, and community self-organizing. All three types of
solution can advance AI collective action, but no single type is likely to
address the entire range of AI collective action problems. Instead of looking
for narrow, silver-bullet solutions it may be to pursue a mix of solutions that
AI collective action issues in different ways and at different scales.
Governance regimes should also account for other factors that could affect
collective action, such as the extent to which AI developers are transparent
about their technology.

AI collective action issues are
increasingly pressing. Collective action will be necessary to ensure that AI
serves the public interest rather than just the narrow private interests of
those who develop it. Collective action will also be necessary to ensure that
AI is developed with adequate safety measures and risk management protocols. Further
work could provide more detailed analysis and support practical progress on AI
collective action issues.

This paper has also been [summarized](https://montrealethics.ai/collective-action-on-artificial-intelligence-a-primer-and-review/) in the [AI Ethics Brief #71](https://brief.montrealethics.ai/p/beginners-guide-collective-action-accountability?token=eyJ1c2VyX2lkIjo3OTA2OTI2LCJwb3N0X2lkIjo0MDc1ODcxNCwiXyI6IkVDTlJPIiwiaWF0IjoxNjMwNTI4MjIyLCJleHAiOjE2MzA1MzE4MjIsImlzcyI6InB1Yi0yOTk5OSIsInN1YiI6InBvc3QtcmVhY3Rpb24ifQ.6tyOCFcObM3E0BJQMQKRmDnFcdqCTZcVPMphh_pPlj8) of the Montreal AI Ethics Institute.

This paper extends GCRI’s interdisciplinary [research on AI](https://gcrinstitute.org/ai/). It builds on GCRI’s prior work on the governance of AI, particularly the papers [On the promotion of safe and socially beneficial artificial intelligence](https://gcrinstitute.org/on-the-promotion-of-safe-and-socially-beneficial-artificial-intelligence/) and [Lessons for artificial intelligence from other global risks](https://gcrinstitute.org/lessons-for-artificial-intelligence-from-other-global-risks/).

Academic citation:
de Neufville, Robert and Seth D. Baum, 2021. [Collective action on artificial intelligence: A primer and review](https://gcrinstitute.org/papers/058_collective-action.pdf). *Technology in Society*, vol. 66, (August), article 101649, [DOI 10.1016/j.techsoc.2021.101649.](https://doi.org/10.1016/j.techsoc.2021.101649)

*Image credit: [Volodymyr Goinyk](https://creativemarket.com/Goinyk)*

 Tagged with [artificial intelligence](https://gcrinstitute.org/tag/artificial-intelligence/)

Collective Action on Artificial Intelligence: A Primer and Review | Global Catastrophic Risk Institute


![](http://gcrinstitute.org/wp-content/uploads/2020/07/UN-Patrick-Gruban.jpg)
[View the paper “Minimizing Global Catastrophic and Existential Risks from Emerging Technologies Through International Law”](https://gcrinstitute.org/papers/006_international-law.pdf)

Mankind is rapidly developing “emerging technologies” in the fields of bioengineering, nanotechnology, and artificial intelligence that have the potential to solve humanity’s biggest problems, such as curing all disease, extending human life, or mitigating massive environmental problems like climate change. However, if these emerging technologies are misused or have an unintended negative effect, the consequences could be enormous, potentially resulting in serious, global damage to humans (known as “global catastrophic harm”) or severe, permanent damage to the Earth—including, possibly, human extinction (known as “existential harm”). The chances of a global catastrophic risk or existential risk actually materializing are relatively low, but mankind should be careful when a losing gamble means massive human death and irreversible harm to the planet. While international law has become an important source of global regulation for other global risks like climate change and biodiversity loss, emerging technologies do not fall neatly within existing international regimes, and thus any country is more or less free to develop these potentially dangerous technologies without practical safeguards that would curtail the risk of a catastrophic event. In light of these problems, this paper serves to discuss the risks associated with bioengineering, nanotechnology, and artificial intelligence; review the potential of existing international law to regulate these emerging technologies; and propose an international regulatory regime that would put the international world in charge of ensuring that low-probability, high-risk disasters never materialize.

Academic citation:
Wilson, Grant S., 2013. [Minimizing global catastrophic and existential risks from emerging technologies through international law.](https://gcrinstitute.org/papers/006_international-law.pdf) *Virginia Environmental Law Journal*, vol. 31, no. 2, pages 307-364.

*Image credit: [Patrick Gruban](https://www.flickr.com/photos/19473388@N00/336920038)*

---

*This blog post was published on 28 July 2020 as part of a website overhaul and backdated to reflect the time of the publication of the work referenced here.*

 Tagged with [emerging technologies](https://gcrinstitute.org/tag/emerging-technologies/), [international law](https://gcrinstitute.org/tag/international-law/)


![](http://gcrinstitute.org/wp-content/uploads/2021/06/ComputerLeaf_Buckyball_2021-06-04.jpg)
[View the paper “Moral Consideration of Nonhumans in the Ethics of Artificial Intelligence”](https://link.springer.com/article/10.1007/s43681-021-00065-0)

In the ethics of artificial intelligence, a major theme is the challenge of aligning AI to human values. This raises the question of the role of nonhumans. Indeed, AI can profoundly affect the nonhuman world, including nonhuman animals, the natural environment, and the AI itself. Given that large parts of the nonhuman world are already under immense threats from human affairs, there is reason to fear potentially catastrophic consequences should AI R&D fail to account for nonhumans, for example with AI systems for industrial and commercial infrastructure, or future artificial general intelligence (AGI). This paper documents the state of attention to nonhumans within the field of AI ethics and presents an argument for giving nonhumans adequate attention.

The paper specifically examines
the extent to which nonhumans are given moral consideration in AI ethics, and
the extent to which they should be. Moral consideration of nonhumans means
actively valuing nonhumans for their own sake—in philosophy terms,
intrinsically valuing them. Unfortunately, the paper finds that most work in AI
ethics ignore nonhumans, or value nonhumans only for the effects they have on
humans. This leaves apt opportunity for the development and use of AI that
adversely impacts nonhumans.

The paper documents moral consideration of nonhumans in academic AI ethics research, statements of AI ethics principles, [AGI R&D projects](https://gcrinstitute.org/2020-survey-of-artificial-general-intelligence-projects-for-ethics-risk-and-policy/), and select initiatives to design, build, apply, and govern AI. Aside from a line of research on the moral status of AI, the field of AI ethics generally fails to give moral consideration to nonhumans: The paper finds no attention to nonhumans in 76 of 84 sets of AI ethics principles surveyed by [Jobin et al.](https://www.nature.com/articles/s42256-019-0088-2), 40 of 45 AGI R&D projects surveyed by [Baum](https://gcrinstitute.org/a-survey-of-artificial-general-intelligence-projects-for-ethics-risk-and-policy/), 38 of 44 chapters in the [*Oxford Handbook of Ethics of AI*](https://www.oxfordhandbooks.com/view/10.1093/oxfordhb/9780190067397.001.0001/oxfordhb-9780190067397), and 13 of 17 chapters in the anthology [*Ethics of Artificial Intelligence*](https://global.oup.com/academic/product/ethics-of-artificial-intelligence-9780190905033). In the latter two, any dedicated attention is on the moral status of AI. No other types of nonhumans are given dedicated attention.

More could be done. The [Microsoft
AI for Earth](https://www.microsoft.com/en-us/ai/ai-for-earth) program is a good example of AI used in ways that benefit
nonhumans. It supports several programs for environmental protection and
biodiversity conservation that give explicit moral consideration to nonhumans,
including [Wild
Me](https://news.microsoft.com/2018/06/14/wild-me-joins-ai-for-earth/),
[eMammal](https://www.microsoft.com/en-us/ai/ai-for-earth-emammal), [NatureServe](https://www.microsoft.com/en-us/ai/ai-for-earth-natureserve), and [Zamba
Cloud](https://www.microsoft.com/en-us/ai/ai-for-earth-zamba-cloud).
Other AI groups could run similar programs. Within AI ethics research, the
paper outlines ideas for nonhuman algorithmic bias, such as by applying [ecolinguistics](https://en.wikipedia.org/wiki/Ecolinguistics) to biases in
natural language processing. While algorithmic bias is a major topic in AI
ethics, other literature has focused on social biases, but research in
ecolinguistics show that English—the primary language for AI system design—contains
biases in favor of humans over nonhumans.

Given the limited moral
consideration of nonhumans in the current field of AI ethics, the paper argues
for more consistent and extensive moral consideration of nonhumans. The
argument draws on concepts of ontological and ethical anthropocentrism as
developed in environmental ethics. Humans are members of the animal kingdom and
part of nature, and there is no sound basis for restricting moral consideration
exclusively to humans. There are important questions of how much moral
consideration to give to nonhumans relative to humans. The paper sets these
questions aside to call for a more basic improvement in moral consideration to
nonhumans across AI ethics.

This paper extends GCRI’s interdisciplinary [research on AI](https://gcrinstitute.org/ai/). It builds on prior GCRI research on AI ethics, especially the paper [Social Choice Ethics in Artificial Intelligence](https://gcrinstitute.org/social-choice-ethics-in-artificial-intelligence/). It uses ethics data compiled in our [2017](https://gcrinstitute.org/a-survey-of-artificial-general-intelligence-projects-for-ethics-risk-and-policy/) and [2020](https://gcrinstitute.org/2020-survey-of-artificial-general-intelligence-projects-for-ethics-risk-and-policy/) surveys of AGI R&D projects, especially project goals. It also continues our tradition of applying the rich body of environmental research to new AI issues, as previously done in our papers [On the Promotion of Safe and Socially Beneficial Artificial Intelligence](https://gcrinstitute.org/on-the-promotion-of-safe-and-socially-beneficial-artificial-intelligence/) and [Lessons for artificial intelligence from other global risks](https://gcrinstitute.org/lessons-for-artificial-intelligence-from-other-global-risks/).

This paper has also been [summarized](https://montrealethics.ai/moral-consideration-of-nonhumans-in-the-ethics-of-artificial-intelligence/) in the [AI Ethics Brief #73](https://brief.montrealethics.ai/p/deepfake-voices-embedding-values-bravery?token=eyJ1c2VyX2lkIjo3OTA2OTI2LCJwb3N0X2lkIjo0MTI2MTU2MSwiXyI6ImdSclZwIiwiaWF0IjoxNjMxNjUwMjU5LCJleHAiOjE2MzE2NTM4NTksImlzcyI6InB1Yi0yOTk5OSIsInN1YiI6InBvc3QtcmVhY3Rpb24ifQ.jJUM6lOMgU4U9eRB8CDDAqW27zZG85YoDUmYh6rSwNs) of the Montreal AI Ethics Institute, and in the [blog](https://mahb.stanford.edu/blog/moral-consideration-of-nonhumans-in-the-ethics-of-artificial-intelligence/) of [Stanford MAHB](https://mahb.stanford.edu/). It is also included in the 2022 [The AI Ethics Report](https://montrealethics.ai/volume6/) and is discussed in the MEDIUM article [“Is 2022 the Year that AI Ethics Takes Sustainability Seriously?”](https://josh-gellers.medium.com/is-2022-the-year-that-ai-ethics-takes-sustainability-seriously-8a10953105e9).

Academic citation:
Owe, Andrea and Seth D. Baum, 2021. [Moral consideration of nonhumans in the ethics of artificial intelligence](https://link.springer.com/article/10.1007%2Fs43681-021-00065-0). *AI & Ethics,* vol. 1, no. 4 (November), pages 517-528, [DOI 10.1007/s43681-021-00065-0](https://dx.doi.org/10.1007/s43681-021-00065-0). [ReadCube](https://rdcu.be/cl2Zv).

*Image credit: Buckyball
Design*

 Tagged with [artificial intelligence](https://gcrinstitute.org/tag/artificial-intelligence/), [ethics](https://gcrinstitute.org/tag/ethics/)

Moral Consideration of Nonhumans in the Ethics of Artificial Intelligence | Global Catastrophic Risk Institute


Created: 2018-11-08 | Updated: 2019-11-02 | Suggestions: please make suggestions directly in this Doc | List maintainer: Mati Roy ([contact@matiroy.com](mailto:contact@matiroy.com))

AI Safety Open Problems

Technical AGI safety research outside AI: [https://forum.effectivealtruism.org/posts/2e9NDGiXt8PjjbTMC/technical-agi-safety-research-outside-ai](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/2e9NDGiXt8PjjbTMC/technical-agi-safety-research-outside-ai&sa=D&source=editors&ust=1688249706070795&usg=AOvVaw23JRJ44mJrwWTghIfR4UaR)

Concrete problems in AI safety: [https://arxiv.org/abs/1606.06565](https://www.google.com/url?q=https://arxiv.org/abs/1606.06565&sa=D&source=editors&ust=1688249706071215&usg=AOvVaw1DglOrGmsnSxUbmNG_hPFU)

MIRI: Agent Foundations for Aligning Superintelligence with Human Interests: [https://intelligence.org/files/TechnicalAgenda.pdf](https://www.google.com/url?q=https://intelligence.org/files/TechnicalAgenda.pdf&sa=D&source=editors&ust=1688249706071526&usg=AOvVaw3gK-jp7Ha-fPK6NKp9_Mxb)

MIRI: Alignment for Advanced Machine Learning Systems: [https://intelligence.org/files/AlignmentMachineLearning.pdf](https://www.google.com/url?q=https://intelligence.org/files/AlignmentMachineLearning.pdf&sa=D&source=editors&ust=1688249706071809&usg=AOvVaw1lWLsTgWxtlL855S70Oa8Y)

Research Priorities for Robust and Beneficial Artificial Intelligence: [ttps://arxiv.org/pdf/1602.03506.pdf](https://www.google.com/url?q=https://arxiv.org/pdf/1602.03506.pdf&sa=D&source=editors&ust=1688249706072091&usg=AOvVaw2_p0Axl6srzM97AbHWxT_9)

Luke Muehlhauser: How to study superintelligence strategy: [http://lukemuehlhauser.com/some-studies-which-could-improve-our-strategic-picture-of-superintelligence/](https://www.google.com/url?q=http://lukemuehlhauser.com/some-studies-which-could-improve-our-strategic-picture-of-superintelligence/&sa=D&source=editors&ust=1688249706072405&usg=AOvVaw2WBwCPRNkNTgLINfKZN3Mr)

Andrew Critch: Abstract open problems in AI alignment: [http://acritch.com/abstract-open-problems/](https://www.google.com/url?q=http://acritch.com/abstract-open-problems/&sa=D&source=editors&ust=1688249706072674&usg=AOvVaw3UvOg4HK0YGTLpgvfJhtfY)

Foundational Research Institute: Open Research Questions: [https://foundational-research.org/open-research-questions/](https://www.google.com/url?q=https://foundational-research.org/open-research-questions/&sa=D&source=editors&ust=1688249706072985&usg=AOvVaw2V6AcTPqkyECKOJtn0VnBv)

AI Impacts: List of multipolar research projects: [https://aiimpacts.org/multipolar-research-projects/](https://www.google.com/url?q=https://aiimpacts.org/multipolar-research-projects/&sa=D&source=editors&ust=1688249706073276&usg=AOvVaw1s0mrv1Vk8D_RZL0zvAOwZ)

AI Impacts: Promising research projects: [https://aiimpacts.org/promising-research-projects/](https://www.google.com/url?q=https://aiimpacts.org/promising-research-projects/&sa=D&source=editors&ust=1688249706073552&usg=AOvVaw3dK2YkckGL3JxHiplPjKXR)

AI Impacts: Research Problems: [https://aiimpacts.org/category/research-problems/](https://www.google.com/url?q=https://aiimpacts.org/category/research-problems/&sa=D&source=editors&ust=1688249706073829&usg=AOvVaw1eiNSYOKZkQCan2ZMYXYGa)

Effective thesis in computer science: [http://effectivethesis.com/theses/?discipline=computer+science](https://www.google.com/url?q=http://effectivethesis.com/theses/?discipline%3Dcomputer%2Bscience&sa=D&source=editors&ust=1688249706074120&usg=AOvVaw0kAUPFIlGgMg9-4bMPR_8K)

Other ideas in the comment section here: [http://effective-altruism.com/ea/18p/concrete\_project\_lists/](https://www.google.com/url?q=http://effective-altruism.com/ea/18p/concrete_project_lists/&sa=D&source=editors&ust=1688249706074420&usg=AOvVaw1xmjhXtuAslQtPL33m92aH)

Other ideas about Ryan Carey again: Improving long-run civilisational robustness: [http://effective-altruism.com/ea/xg/improving\_longrun\_civilisational\_robustness/](https://www.google.com/url?q=http://effective-altruism.com/ea/xg/improving_longrun_civilisational_robustness/&sa=D&source=editors&ust=1688249706074753&usg=AOvVaw1uf-Ak_YXZ_rNjwD_tSOOQ)

Problems in AI Alignment that philosophers could potentially contribute to: [https://www.lesswrong.com/posts/rASeoR7iZ9Fokzh7L/problems-in-ai-alignment-that-philosophers-could-potentially](https://www.google.com/url?q=https://www.lesswrong.com/posts/rASeoR7iZ9Fokzh7L/problems-in-ai-alignment-that-philosophers-could-potentially&sa=D&source=editors&ust=1688249706075092&usg=AOvVaw1sGhx4Qv-idPA8rKYhrX95)

Cognitive Science/Psychology As a Neglected Approach to AI Safety: [https://forum.effectivealtruism.org/posts/WdMnmmqqiP5zCtSfv/cognitive-science-psychology-as-a-neglected-approach-to-ai](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/WdMnmmqqiP5zCtSfv/cognitive-science-psychology-as-a-neglected-approach-to-ai&sa=D&source=editors&ust=1688249706075430&usg=AOvVaw0b4VbQ0WP8e34yjuBFtn_M) (related: [https://ought.org/](https://www.google.com/url?q=https://ought.org/&sa=D&source=editors&ust=1688249706075582&usg=AOvVaw1ygbxiNNcd8fNKW8mJAMxf))

Machine Learning Projects on IDA: [https://www.alignmentforum.org/posts/Y9xD78kufNsF7wL6f/machine-learning-projects-on-ida](https://www.google.com/url?q=https://www.alignmentforum.org/posts/Y9xD78kufNsF7wL6f/machine-learning-projects-on-ida&sa=D&source=editors&ust=1688249706075896&usg=AOvVaw049J5n1nZxGMwFzakMsN4j)

Addendum

Landscape of current work on potential risks from advanced AI: [https://docs.google.com/document/d/16Te6HnZN2OEviYFA-42Tf9Pal\_Idovtgr5Y1RGEPW\_g/](https://www.google.com/url?q=https://docs.google.com/document/d/16Te6HnZN2OEviYFA-42Tf9Pal_Idovtgr5Y1RGEPW_g/&sa=D&source=editors&ust=1688249706076356&usg=AOvVaw0kP7_DM3Nq6_6AEFwtqyaR)

AI Safety Open Problems