[Download PDF](/articles/s41598-019-50145-9.pdf)

### Subjects

* [Human behaviour](/subjects/human-behaviour)
* [Psychology and behaviour](/subjects/psychology-and-behaviour)

Abstract
--------

The 21st century will likely see growing risks of human extinction, but currently, relatively small resources are invested in reducing such existential risks. Using three samples (UK general public, US general public, and UK students; total N = 2,507), we study how laypeople reason about human extinction. We find that people think that human extinction needs to be prevented. Strikingly, however, they do not think that an extinction catastrophe would be uniquely bad relative to near-extinction catastrophes, which allow for recovery. More people find extinction uniquely bad when (a) asked to consider the extinction of an animal species rather than humans, (b) asked to consider a case where human extinction is associated with less direct harm, and (c) they are explicitly prompted to consider long-term consequences of the catastrophes. We conclude that an important reason why people do not find extinction uniquely bad is that they focus on the immediate death and suffering that the catastrophes cause for fellow humans, rather than on the long-term consequences. Finally, we find that (d) laypeople—in line with prominent philosophical arguments—think that the quality of the future is relevant: they do find extinction uniquely bad when this means forgoing a utopian future.

Introduction
------------

The ever-increasing powers of technology can be used for good and ill. In the 21st century, technological advances will likely yield great benefits to humanity, but experts warn that they will also lead to growing risks of human extinction[1](#ref-CR1 "Bostrom, N. & Cirkovic, M. M. Global Catastrophic Risks (OUP Oxford, 2011)."),[2](#ref-CR2 "Bostrom, N. Superintelligence (Oxford University Press, 2014)."),[3](#ref-CR3 "Rees, M. Our Final Hour: A Scientist’s Warning (Hachette UK, 2009)."),[4](/articles/s41598-019-50145-9#ref-CR4 "Rees, M. Denial of catastrophic risks. Science. 339, 1123 (2013)."). The risks stem both from existing technologies such as nuclear weapons, as well as emerging technologies such as synthetic biology and artificial intelligence[5](/articles/s41598-019-50145-9#ref-CR5 "Cotton-Barratt, O., Farquhar, S., Halstead, J., Schubert, S. & Snyder-Beattie, A. Global catastrophic risks 2016. Global Challenges Foundation (2016)."). A small but growing number of research institutes, such as the University of Oxford’s *Future of Humanity Institute* and the University of Cambridge’s *Centre for the Study of Existential Risk*, are studying these risks and how to mitigate them. Yet besides them, relatively small resources are explicitly devoted to reducing these risks.

Here, we study the general public’s views of the badness of human extinction. We hypothesize that most people judge human extinction to be bad. But *how bad* do they find it? And *why* do they find it bad? Besides being highly policy-relevant, these questions are central for humanity’s understanding of itself and its place in nature. Human extinction is a pervasive theme in myths and religious writings[6](#ref-CR6 "Soage, A. B. The End of Days: Essays on the Apocalypse from Antiquity to Modernity. Totalitarian Movements and Political Religions. 10, 375–377 (2009)."),[7](#ref-CR7 "Banks, A. C. The End of the World As We Know It: Faith, Fatalism, and Apocalypse. Nova Religio. 3, 420–421 (2000)."),[8](#ref-CR8 "Hall, J. R. Apocalypse: From Antiquity to the Empire of Modernity (John Wiley & Sons, 2013)."),[9](#ref-CR9 "O’Leary, S. D. Arguing the Apocalypse: A Theory of Millennial Rhetoric (Oxford University Press, 1998)."),[10](/articles/s41598-019-50145-9#ref-CR10 "Baumgartner, F. J., Graziano, F. & Weber, E. Longing for the End: A History of Millennialism in Western Civilization. Utop. Stud. 11, 214–218 (2000).").

One view is that human extinction is bad primarily because it would harm many concrete individuals: it would mean the death of all currently living people. On this view, human extinction is a very bad event, but it is not much worse than catastrophes that kill *nearly* all currently living people—since the difference in terms of numbers of deaths would be relatively small. Another view is that the human extinction is bad primarily because it would mean that the human species would go extinct and that humanity’s future would be lost forever. On this view, human extinction is *uniquely* bad: much worse even than catastrophes killing nearly everyone, since we could recover from them and re-build civilization. Whether extinction is uniquely bad or not depends on which of these considerations is the stronger: the immediate harm, or the long-term consequences.

Here is one way to pit these considerations against each other. Consider three outcomes: no catastrophe, a catastrophe killing 80% (near-extinction), and a catastrophe killing 100% (extinction). According to both considerations, no catastrophe is the best outcome, and extinction the worst outcome. But they come apart regarding the *relative differences* between the three outcomes. If the immediate harm is the more important consideration, then *the first difference*, between no catastrophe and near-extinction, is greater than *the second difference*, between near-extinction and extinction. That is because the first difference is greater in terms of numbers of harmed individuals. On the other hand, if the long-term consequences are more important, then the second difference is greater. The first difference compares two non-extinction outcomes, whereas the second difference compares a non-extinction outcome with an extinction outcome—and only the extinction outcome means that the future would be forever lost.

This thought-experiment was conceived by the well-known philosopher Derek Parfit[11](/articles/s41598-019-50145-9#ref-CR11 "Parfit, D. Reasons and Persons (OUP Oxford, 1984).") (we have adapted the three outcomes slightly; see the Methods section). Parfit argued that most people would find the first difference to be greater, but he himself thought that the second difference is greater. Many other philosophers and other academics working to reduce the risk of human extinction agree with Parfit[12](#ref-CR12 "Bostrom, N. Existential risks: Analyzing human extinction scenarios and related hazards. Journal of Evolution and Technology 9, 1–30 (2002)."),[13](#ref-CR13 "Bostrom, N. When machines outsmart humans. Futures. 35, 759–764 (2003)."),[14](#ref-CR14 "Bostrom, N. The doomsday argument. Think. 6, 23–28 (2008)."),[15](/articles/s41598-019-50145-9#ref-CR15 "Beckstead, N. On the overwhelming importance of shaping the far future. PhD Thesis, Rutgers University-Graduate School-New Brunswick (2013)."). On their view, the badness of human extinction is greatly dependent on how long the future would otherwise be, and what the quality of future people’s lives would be. As the philosopher Nick Bostrom notes, predictions about the long-term future have often been left to theology and fiction, whilst being neglected by science[16](/articles/s41598-019-50145-9#ref-CR16 "Bostrom, N. The Future of Humanity. In New Waves in Philosophy, eds. Jan-Kyrre Berg Olsen, Evan Selinger, & Soren Riis (New York: Palgrave McMillan, 2009)."). However, in recent years, researchers have tried to assess what the long-term future may be like. They argue that if humanity does not go extinct, then the future could be both extraordinarily long and extraordinarily good, involving much greater quality of life than the current world. For instance, Nick Bostrom argues that a conservative estimate of humanity’s future potential is “at least 1016 human lives of normal duration”, which could “be considerably better than the average contemporary human life, which is so often marred by disease, poverty [and] injustice”[17](/articles/s41598-019-50145-9#ref-CR17 "Bostrom, N. Existential Risk Prevention as Global Priority. Glob Policy. 4, 15–31 (2013)."). He goes on to argue that less conservative estimates would yield even greater numbers, and a drastically improved quality of life. The argument is that if humanity develops to a sufficiently high technological level, then it will either cause its own extinction via misuse of powerful technologies, or use those technological powers to greatly improve the level of well-being. Furthermore, they argue, based on the view that new happy people coming into existence is morally valuable[11](/articles/s41598-019-50145-9#ref-CR11 "Parfit, D. Reasons and Persons (OUP Oxford, 1984)."), that it is of paramount moral importance to make sure that we realize our future potential, and prevent human extinction.

While philosophers have discussed the ethics of human extinction for some time, the general public’s views on this matter has not received much study. There are some studies on perceptions of risk of extinction, however. Two studies found that a slight majority do not think that humanity will go extinct, and that most of those who thought that it would go extinct thought that would happen at least 500 years into the future[18](/articles/s41598-019-50145-9#ref-CR18 "Tonn, B. Beliefs about human extinction. Futures. 41, 766–773 (2009)."),[19](/articles/s41598-019-50145-9#ref-CR19 "Tonn, B., Hemrick, A. & Conrad, F. Cognitive representations of the future: Survey results. Futures. 38, 810–829 (2006)."). There is also a related literature on catastrophic risk in general, focusing primarily on non-extinction catastrophes. For instance, it has been argued that the fact that people use the availability heuristic—they focus on risks which have salient historical examples—leads to a neglect of new types of risks and risks of major catastrophes (which are rare, and therefore less psychologically available)[20](/articles/s41598-019-50145-9#ref-CR20 "Wiener, J. B. The Tragedy of the Uncommons: On the Politics of Apocalypse. Glob Policy. 7, 67–80 (2016)."). Similarly, it has been argued that the fact that risk mitigation is a public good leads to under-investment, since it means that it is not possible to exclude free riders from benefiting from it[21](/articles/s41598-019-50145-9#ref-CR21 "Hauser, O. P., Rand, D. G., Peysakhovich, A. & Nowak, M. A. Cooperating with the future. Nature. 511, 220–223 (2014)."). On specific risks, there is a literature on the psychology of climate change showing that people fail to act to mitigate climate change because they engage in temporal discounting[22](/articles/s41598-019-50145-9#ref-CR22 "Pahl, S., Sheppard, S., Boomsma, C. & Groves, C. Perceptions of time in relation to climate change. WIREs Clim Change. 5, 375–388 (2014)."),[23](/articles/s41598-019-50145-9#ref-CR23 "Jacquet, J. et al. Intra- and intergenerational discounting in the climate game. Nat. Clim. Chang. 3, 1025 (2013).") and motivated reasoning about its severity[24](/articles/s41598-019-50145-9#ref-CR24 "Kahan, D. M. Ideology, Motivated Reasoning, and Cognitive Reflection: An Experimental Study. Judgm Decis Mak. 8, 407–424 (2013)."), and because of psychological distance[25](/articles/s41598-019-50145-9#ref-CR25 "Spence, A., Poortinga, W. & Pidgeon, N. The psychological distance of climate change. Risk Anal. 32, 957–972 (2012).") (e.g., temporal and social distance). However, to date there have been no studies on how laypeople reason about the moral aspect of human extinction: how bad it would be. Is the extinction of our own species something people care about? Do they recognize it as being fundamentally different in quality from other catastrophes? And if so, why?

Results
-------

### Study 1

In Study 1 (US sample, *n* = 183, mean age 38.2, 50.81% female), we studied the general public’s judgments of the badness of human extinction. A large majority of the participants (78.14%, 143/183 participants) found human extinction to be bad on a binary question (bad vs. not bad), and we got similar results on a seven-point scale (1 = *definitely not bad*, 4 = *midpoint*, 7 = *definitely bad*; *M* = 5.61; *SD* = 2.11). Participants also felt strongly that human extinction needs to be prevented (1 = *not at all*, 4 = *midpoint*, 7 = *very strongly*; *M* = 6.01, *SD* = 1.65), that they have a moral obligation to prevent it (1 = *definitely no*, 4 = *midpoint*, 7 = *definitely yes*; *M* = 5.69, *SD* = 1.86), and that funding work to reduce the risk of human extinction is more important than funding other areas of government, such as education, health care and social security (1 = *much less important to fund work to reduce the risk of human extinction*, 4 = *midpoint*, 7 = *much more important*; *M* = 5.43, *SD* = 1.72). Participants believed that provided that humanity will not go extinct, the future is going to be roughly as good as the present (1 = *much worse than the present world*, 4 = *about as good as the present world*, 7 = *much better than the present world*; *M* = 4.48, *SD* = 1.57), and the better they thought the future would be, the worse they considered extinction to be (*r* = 0.51, *P* < 0.001), as measured by the seven-point scale. Similarly, more optimistic[26](/articles/s41598-019-50145-9#ref-CR26 "Kemper, C. J., Kovaleva, A., Beierlein, C. & Rammstedt, B. Measuring the construct of Optimism-Pessimism with single item indicators. Paper presented at the 4th Conference of the European Survey Research Association (ESRA), Lausanne, Switzerland (2011).") participants judged extinction to be worse (*r* = 0.32, *P* < 0.001). Participants’ responses to the question whether the world gets better if a happy person comes into existence were close to the midpoint (1 = *definitely not better*, 4 = *midpoint*, 7 = *definitely better*; *M* = 4.45, *SD* = 1.73), and people who thought that that would make the world better were more likely (*r* = 0.22, *P* = 0.003) to find extinction bad. For further details about the results, see Supplementary Materials.

### Study 2a

Having thus observed that people do find human extinction bad, we turned to studying whether they find it *uniquely* bad relative to non-extinction catastrophes in Study 2a (pre-registered at <https://osf.io/hj2n4>; British sample). Participants (*n* = 1,251, mean age 36.6, 35.33% female) were randomly divided into a control condition and four experimental conditions: “the animals condition”, “the “sterilization condition”, “the salience condition” and “the utopia condition” (see below for explanations of the manipulations). Participants in the control condition (257 participants) were presented with the three outcomes described above—no catastrophe, a catastrophe killing 80%, and a catastrophe killing 100%—and were asked how they would rank them from best to worst. As Parfit expected, a large majority (82.88%, 213/257 participants, cf. Fig. [1](/articles/s41598-019-50145-9#Fig1)) ranked no catastrophe as the best outcome and 100% dying as the worst outcome. However, this was just a preliminary question: as per the discussion above, what we were primarily interested in was which difference participants that gave the expected ranking found greater: the first difference (meaning that extinction is not uniquely bad) or the second difference (meaning that extinction is uniquely bad). (Recall that the first difference was the difference between no catastrophe and a catastrophe killing 80%, and the second difference the difference between a catastrophe killing 80% and a catastrophe killing 100%.) We therefore asked participants who gave the expected ranking (but not the other participants) which difference they judged to be greater. We found that most people did not find extinction uniquely bad: only a relatively small minority (23.47%, 50/213 participants) judged the second difference to be greater than the first difference.

**Figure 1**[![figure 1](//media.springernature.com/lw685/springer-static/image/art%3A10.1038%2Fs41598-019-50145-9/MediaObjects/41598_2019_50145_Fig1_HTML.png)](/articles/s41598-019-50145-9/figures/1)Proportions of participants who found extinction uniquely bad. (This means that they found the difference, in terms of badness, between a catastrophe killing 80% and a catastrophe killing 100% to be greater than the difference between no catastrophe and a catastrophe killing 80%.) Laypeople consistently did not find extinction uniquely bad in the control condition (*Control*), but did so in a scenario where the future would be very long and good conditional on survival (*Utopia*). The animals condition (*Animals*), sterilization condition (*Sterilization*) and salience condition (*Salience*) yielded in-between results. People explicitly devoted to existential risk reduction (*Existential risk mitigators*) consistently found extinction uniquely bad.

[Full size image](/articles/s41598-019-50145-9/figures/1)As stated, we included four experimental conditions aiming to explain these results. We thought that one reason why participants do not find extinction uniquely bad in the control condition is that they feel strongly for the victims of the catastrophes. Therefore, they focus on the immediate suffering and death that the catastrophes cause, which leads them to judge the difference between no one dying and 80% dying to be greater than the difference between 80% dying and 100% dying. To test this hypothesis, we included two conditions designed to trigger a weaker focus on the immediate harm. First, we included a condition where the catastrophes affected an animal species (zebras) rather than humans (“the animals condition”; otherwise identical to the control condition; 246 participants). (We chose zebras because zebra extinction would likely have small effects on humans; in contrast to extinction of, for example, pigs or dogs.) We hypothesized that people focus less on the immediate harm that the catastrophes cause if the catastrophes affect animals rather than humans[27](/articles/s41598-019-50145-9#ref-CR27 "Caviola, L., Everett, J. A. C., Faber, N. S. The moral standing of animals: Towards a psychology of speciesism. J. Pers. Soc. Psychol., 116, 1011–1029 (2019)."). Second, we included a condition where the catastrophes led to 80%/100% of the world’s population being unable to have children, rather than getting killed (“the sterilization condition”; otherwise identical to the control condition; 252 participants). We hypothesized that people would focus less strongly on the immediate harm that the catastrophes cause if they lead to sterilization rather than death.

Thus, we hypothesized that a greater share of the participants who gave the expected ranking would find extinction uniquely bad in the animals condition and the sterilization condition than in the control condition. We found, first, that a large majority ranked no catastrophe as the best outcome and 100% dying as the worst outcome (the expected ranking) both in the animals condition (89.84%, 221/246 participants) and the sterilization condition (82.54%, 208/252 participants). Subsequently, we found that our hypotheses were confirmed. The proportion of the participants who gave the expected ranking that found extinction uniquely bad was significantly larger (*χ*2(1) = 8.82, *P* = 0.003) in the animals condition (44.34%, 98/221 participants) than in the control condition (23.47%, 50/213 participants). Similarly, the proportion of the participants who gave the expected ranking that found extinction uniquely bad was significantly larger (*χ*2(1) = 23.83, *P* < 0.001) in the sterilization condition (46.63%, 97/208 participants) than in the control condition (23.47%, 50/213 participants).

We had another hypothesis for why control condition participants do not find extinction uniquely bad, namely that they neglect the long-term consequences of the catastrophes. To test this hypothesis, we included a condition where we made the long-term consequences salient (“the salience condition”; 248 participants). This condition was identical to the control condition, with the exception that we added a brief text explicitly asking the participants to consider the long-term consequences of the three outcomes. It said that if humanity does not go extinct (including if it suffers a non-extinction catastrophe, from which it can recover) it could go on to a long future, whereas that would not happen if humanity went extinct (see the Methods section for the full vignette). We also wanted to know whether participants see empirical information about the quality of the future as relevant for their judgments of the badness of extinction. Does it make a difference how good the future will be? We therefore included a maximally positive scenario, the “utopia condition” (248 participants), where it was said that provided that humanity does not go extinct, it “goes on to live for a very long time in a future which is better than today in every conceivable way”. It was also said that “there are no longer any wars, any crimes, or any people experiencing depression or sadness” and that “human suffering is massively reduced, and people are much happier than they are today” (in the scenario where 80% die in a catastrophe, it was said that this occurred after a recovery period; see the Methods section for the full text). Conversely, participants were told that if 100% are killed, then “no humans will ever live anymore, and all of human knowledge and culture will be lost forever.” We hypothesized that both of these manipulations would make more participants judge extinction to be uniquely bad compared with the control condition.

We found again that a large majority ranked no catastrophe as the best outcome and 100% dying as the worst outcome (the expected ranking) both in the salience condition (77.82%, 193/248 participants) and the utopia condition (86.69%, 215/248 participants). Subsequently, we found that our hypotheses were confirmed. The proportion of the participants who chose the expected ranking that found extinction uniquely bad was significantly larger (*χ*2(1) = 29.90, *P* < 0.001) in the salience condition (50.25%, 97/193 participants) than in the control condition (23.47%, 50/213 participants). Similarly, the proportion of the participants who chose the expected ranking that found extinction uniquely bad was significantly larger (*χ*2(1) = 30.30, *P* < 0.001) in the utopia condition (76.74%, 165/215 participants) than in the control condition (23.47%, 50/213 participants). We also found that there was a significant difference between the utopia condition and the salience condition (*χ*2(1) = 29.90, *P* < 0.001).

Our interpretation of these results is as follows. The utopia manipulation effectively does two things: it highlights the long-term consequences of the outcomes, and it says that unless humanity goes extinct, those consequences are going to be extraordinarily good. The salience manipulation only highlights the long-term consequences. Thus, we can infer that merely highlighting the long-term consequences make people more likely to find extinction uniquely bad, and that adding that the long-term future will be extraordinarily good make them still more likely to find extinction uniquely bad.

Lastly, we found that across all conditions, the more cognitively reflective the participants were (as measured by the Cognitive Reflection Test[28](/articles/s41598-019-50145-9#ref-CR28 "Frederick, S. Cognitive Reflection and Decision Making. J. Econ. Perspect. 19, 25–42 (2005).")), the more likely they were to judge extinction to be uniquely bad (*Exp*(*B*) = 0.15, *P* = 0.01, Odds ratio = 1.6).

In conclusion, we find that people do not find extinction uniquely bad when asked without further prompts, and have identified several reasons why that is. As evidenced by the animals and the sterilization conditions, they focus on the immediate harm that the catastrophes cause, because they feel strongly for the victims of the catastrophes—and on that criterion, near-extinction is almost as bad as extinction. As evidenced by the salience condition, they neglect the long-term consequences of the outcomes. We also find that participants’ empirical beliefs about the quality of the future make a difference: telling participants that the future will be extraordinarily good makes them significantly more likely to find extinction uniquely bad.

### Study 2b

To find out whether these results would hold up with different demographics, we aimed to replicate them using a sample of the US general public (pre-registered at <https://osf.io/8amxs>; *N* = 855, mean age 36.85, 48.65% female) in Study 2b. We found again that large majorities ranked no catastrophe as the best outcome and 100% dying as the worst outcome (the expected ranking) in the control condition (87.80%, 144/164 participants), the animals condition (92.44%, 159/172 participants), the sterilization condition (91.62%, 153/167 participants), the salience condition (83.05%, 147/177 participants) and the utopia condition (89.71%, 157/175 participants). And again we found that only a small minority of the participants who chose the expected ranking judged extinction to be uniquely bad in the control condition (18.75%, 27/144 participants). The proportion of the participants who chose the expected ranking who found extinction uniquely bad was significantly larger in the animals condition (34.59%, 55/159 participants; *χ*2(1) = 8.82, *P* = 0.003), the salience condition (39.45%, 58/147 participants; *χ*2(1) = 14.10, *P* < 0.001) and the utopia condition (66.88%, 105/157 participants; *χ*2(1) = 68.72, *P* < 0.001) than in the control condition. We also again found a significant difference between the utopia condition and the salience condition (*χ*2(1) = 21.868, *P* < 0.001). However, in the sterilization condition, only 28.75% (44/153 participants) of the participants who chose the expected ranking found extinction uniquely bad, which meant that the difference with the control condition was not significant on the 0.05-level (*χ*2(1) = 3.55, *P* = 0.059). Lastly, we found again that (across all conditions) the more cognitively reflective the participants were (as measured by the Cognitive Reflection Test), the more likely they were to judge extinction to be uniquely bad (*Exp*(*B*) = 0.21, *P* = 0.005, Odds ratio = 1.2).

### Study 2c

To further test the robustness of our findings across different demographics, we conducted Study 2c as another replication, this time using a sample of University of Oxford students (*N* = 196, mean age 24.27, 61% female). We only included the control and the utopia conditions. We found again that most participants ranked no catastrophe as the best outcome and 100% dying as the worst outcome (the expected ranking) both in the control condition (65.7%, 65/99 participants) and the utopia condition (84.5%, 82/97 participants). We then found again that a minority of the participants who chose the expected ranking found extinction to be uniquely bad in the control condition (36.92%, 24/65 participants), though this minority was slightly larger than in the two samples of the general public (cf. Fig. [1](/articles/s41598-019-50145-9#Fig1)). We also found again that the proportion of the utopia condition participants who chose the expected ranking that found extinction uniquely bad (76.83%, 63/82 participants) was significantly larger (*χ*2(1) = 22.28, *P* < 0.001) than in the control condition. (These findings were further supported by five supplementary studies; see Supplementary Materials).

### Study 3

In Studies 2a to 2c, we thus found that when asked without further prompts, laypeople do not find extinction uniquely bad. In Study 3 (*N* = 71, mean age 30.52, 14.00% female) we aimed to test whether people devoted to preventing human extinction (existential risk mitigators) judge human extinction to be uniquely bad already when asked without further prompts. (Existential risks also include risks that threaten to drastically curtail humanity’s potential[12](#ref-CR12 "Bostrom, N. Existential risks: Analyzing human extinction scenarios and related hazards. Journal of Evolution and Technology 9, 1–30 (2002)."),[13](#ref-CR13 "Bostrom, N. When machines outsmart humans. Futures. 35, 759–764 (2003)."),[14](#ref-CR14 "Bostrom, N. The doomsday argument. Think. 6, 23–28 (2008)."),[15](/articles/s41598-019-50145-9#ref-CR15 "Beckstead, N. On the overwhelming importance of shaping the far future. PhD Thesis, Rutgers University-Graduate School-New Brunswick (2013)."), without causing it to go extinct, but we focus on risks of human extinction.) This would support the validity of our task by demonstrating a link between participants’ responses and behavior in the real world.

We recruited participants via the Effective Altruism Newsletter and social media groups dedicated to existential risk reduction, and only included respondents who put down reducing existential risk as their “most important cause”. Again we had two conditions, the control condition and the utopia condition. We hypothesized that a majority of participants would find extinction uniquely bad in both conditions. We found again that most participants ranked no catastrophe as the best outcome and 100% dying as the worst outcome (the expected ranking) both in the control condition (90.32%, 28/31 participants) and the utopia condition (92.50%, 37/40 participants). But unlike the samples in Studies 2a to 2c, and in line with our hypotheses, substantial majorities of the participants who chose the expected ranking found extinction uniquely bad both in the control condition (85.71%, 24/28 participants) and the utopia condition (94.59%, 35/37). The difference between the conditions was not significant (*χ*2(1) = 0.63, *P* = 0.43). In contrast to laypeople, existential risk mitigators thus found human extinction to be uniquely bad even when the description of the outcomes did not include information about the quality of the future. This suggests that judging human extinction to be uniquely bad, as measured by our task, may be a key motivator for devoting oneself to preventing it.

Discussion
----------

Our studies show that people find that human extinction is bad, and that it is important to prevent it. However, when presented with a scenario involving no catastrophe, a near-extinction catastrophe and an extinction catastrophe as possible outcomes, they do not see human extinction as *uniquely* bad compared with non-extinction. We find that this is partly because people feel strongly for the victims of the catastrophes, and therefore focus on the immediate consequences of the catastrophes. The immediate consequences of near-extinction are not that different from those of extinction, so this naturally leads them to find near-extinction almost as bad as extinction. Another reason is that they neglect the long-term consequences of the outcomes. Lastly, their empirical beliefs about the quality of the future make a difference: telling them that the future will be extraordinarily good makes more people find extinction uniquely bad.

Thus, when asked in the most straightforward and unqualified way, participants do not find human extinction uniquely bad. This could partly explain why we currently invest relatively small resources in reducing existential risk. However, these responses should not necessarily be seen as reflecting people’s well-considered views on the badness of human extinction. Rather, it seems that they partly reflect the fact that people often fail to consider the long-term consequences of extinction. Our studies suggest that it could be that if people reflected more carefully, they might to a greater extent agree that extinction is uniquely bad. A suggestive finding with regards to this is that higher scores on the cognitive reflection test predicted a greater tendency to find extinction uniquely bad. This could mean that deliberative thought-processes lead to finding extinction uniquely bad, whereas intuitive thought-processes lead to the opposite conclusion. More research is needed on the role of deliberation and intuition, as well as many other questions, such as the role of cognitive ability, and the ultimate evolutionary causes of why humans struggle to think clearly about their own extinction.

Finally, let us consider possible policy implications. If it is right that human extinction is uniquely bad, then we should arguably invest much more in making sure it does not happen. We should also change policy in many other ways; e.g., shift technology policy in a more cautious direction[29](/articles/s41598-019-50145-9#ref-CR29 "Farquhar, S., Cotton-Barratt, O., & Snyder-Beattie, A. Pricing Externalities to Balance Public Risks and Benefits of Research. Health Secur 15, 401–408 (2017)."). On this view, we should, if necessary, be prepared to make substantial sacrifices in order to make sure that humanity realizes its future potential. Hence much hinges on the complex question whether we deem our own extinction to be uniquely bad.

Methods
-------

All studies were approved by the University of Oxford’s Central University Research Ethics Committee (approval number: R56657/RE002) and participants in each study gave their informed consent beforehand. All studies were performed in accordance with relevant guidelines and regulations.

### Study 1

#### Participants

We recruited 210 participants and excluded 27 for not completing the study or failing the attention check.

#### Procedure

Level of optimism was measured by asking participants “how optimistic are you in general?” (where optimistic people were defined as “people who look to the future with confidence and who mostly expect good things to happen”)[26](/articles/s41598-019-50145-9#ref-CR26 "Kemper, C. J., Kovaleva, A., Beierlein, C. & Rammstedt, B. Measuring the construct of Optimism-Pessimism with single item indicators. Paper presented at the 4th Conference of the European Survey Research Association (ESRA), Lausanne, Switzerland (2011)."). In addition to the measures reported above, we gave the Oxford Utilitarianism Scale, the Cognitive Reflection Test and demographic questions to the participants. The Oxford Utilitarianism Scale[30](/articles/s41598-019-50145-9#ref-CR30 "Kahane, G. et al. Beyond sacrificial harm: A two-dimensional model of utilitarian psychology. Psychol Rev, 125, 131–164.") consists of two subscales: the impartial beneficence (IB) subscale and the instrumental harm (IH) subscale. The OUS-IB measures the degree to which someone values maximizing overall welfare, independent of its recipient. The OUS-IH measures the degree to which someone is willing to accept that harm is done in order to maximize overall welfare. The Cognitive Reflection Test[28](/articles/s41598-019-50145-9#ref-CR28 "Frederick, S. Cognitive Reflection and Decision Making. J. Econ. Perspect. 19, 25–42 (2005).") measures the tendency to answer questions reflectively and resist reporting the first response that comes to mind.

### Study 2a

#### Participants

We recruited 1301 participants via Prolific and excluded 50 for not completing the study or failing an attention check. The study was pre-registered at <https://osf.io/hj2n4/>.

#### Procedure

Participants were first asked to consider three outcomes, A, B and C, and rank them from best to worst. The outcomes in the control condition were described as follows:

**The control condition:**

1. (A)
There is no catastrophe.
2. (B)
There is a catastrophe that immediately kills 80% of the world’s population.
3. (C)
There is a catastrophe that immediately kills 100% of the world’s population.

This meant that our text differed from Parfit’s[13](/articles/s41598-019-50145-9#ref-CR13 "Bostrom, N. When machines outsmart humans. Futures. 35, 759–764 (2003).") as follows. We used “no catastrophe” rather than Parfit’s “peace” because we thought that “peace” had positive associations that could be a potential confounder. We used “a catastrophe” rather than Parfit’s “a nuclear war” because we thought that there was no reason to specify the nature of the catastrophe. And we said that 80%, rather than 99%, die in the non-extinction catastrophe, to make it more plausible that humanity could recover.

The outcomes in the other conditions were described as follows:

**The animals condition:**

1. (A)
There is no catastrophe.
2. (B)
There is a catastrophe that immediately kills 80% of the world’s zebra population.
3. (C)
There is a catastrophe that immediately kills 100% of the world’s zebra population.

**The sterilization condition:**

1. (A)
There is no catastrophe.
2. (B)
There is a catastrophe that immediately causes 80% of the world’s population to go sterile, meaning they cannot have children.
3. (C)
There is a catastrophe that immediately causes 100% of the world’s population to go sterile, meaning they cannot have children.

**The salience condition:**

1. (A)
There is no catastrophe.
2. (B)
There is a catastrophe that immediately kills 80% of the world’s population.
3. (C)
There is a catastrophe that immediately kills 100% of the world’s population.

Please rank these three outcomes from best to worst.

When you do so, please remember to consider **the long-term consequences** each scenario will have for humanity. If humanity does not go extinct, it could go on to a long future. This is true even if many (but not all) humans die in a catastrophe, since that leaves open the possibility of recovery. However, if humanity goes extinct (if 100% are killed), there will be no future for humanity.

**The utopia condition:**

1. (A)
There is no catastrophe and humanity goes on to live for a very long time in a future which is better than today in every conceivable way. There are no longer any wars, any crimes, or any people experiencing depression or sadness. Human suffering is massively reduced, and people are much happier than they are today.
2. (B)
There is a catastrophe that immediately kills 80% of the world’s population. However, humanity eventually recovers to its original size, and then goes on to live for a very long time in a future which is better than today in every conceivable way. There are no longer any wars, any crimes, or any people experiencing depression or sadness. Human suffering is massively reduced, and people are much happier than they are today.
3. (C)
There is a catastrophe that immediately kills 100% of the world’s population. This means that humanity will go extinct, that no humans will ever live anymore, and all of human knowledge and culture will be lost forever.

On the next page, participants who ranked A as the best outcome and C as the worst were again presented with the three outcomes (other participants were excluded), and told:

“We are now interested in your views of how much better A is than B, and how much better B is than C. In terms of badness, which difference is greater: the difference between A and B, or the difference between B and C?”

In addition to the measures reported above, we gave the Oxford Utilitarianism Scale and demographic questions to participants (see Supplementary Materials for results).

### Study 2b

#### Participants

We recruited 994 participants and excluded 139 for not completing the study or failing an attention check. In addition to the measures reported above, we gave the Oxford Utilitarianism Scale and demographic questions to the participants (see Supplementary Materials for results). The study was pre-registered at <https://osf.io/8amxs>.

#### Procedure

The procedure was the same as in study 2a.

### Study 2c

#### Participants

We recruited 204 participants and excluded 8 for not giving an answer. The procedure was the same as for study 2a, except only the control condition and the utopia condition were included. In addition to the measures reported above, we gave the Oxford Utilitarianism Scale and demographic questions to the participants (see Supplementary Materials for results).

### Study 3

#### Participants

We recruited 196 participants. However, since we were only interested in those effective altruists who consider existential risk mitigation to be the top cause area, only 83 were included into the analysis. 12 participants were excluded for failing an attention check. The final sample was 71 participants.

#### Procedure

The procedure was the same as for study 2a, except only the control condition and the utopia condition were included. In addition, participants were asked three questions assessing how uniquely bad they found extinction (extinction prevention questions). The first question concerned which scenario (A = a catastrophe kills 50% of the world’s population, but humanity recovers to its original size and goes on to live for a very long time, B = painless extinction) they would want to prevent (1 = *definitely A*, 4 = *midpoint*, 7 = *definitely B*). The second question asked if participants would rather support political party A, which works to reduce the risk of scenario A, or political party B, which works to reduce the risk of scenario B (1 = *definitely A*, 4 = *midpoint*, 7 = *definitely B*). The third question asked what they thought the morally right choice for government leaders would be if they had to choose between reducing the risk for either scenario A or B (1 = *definitely reduce the risk of A*, 4 = *midpoint*, 7 = *definitely reduce the risk of B*).

We also gave the Oxford Utilitarianism Scale, The Cognitive Reflection Test and demographic questions to participants. (See Supplementary Materials for additional results).

Data Availability
-----------------

Reports of all measures, manipulations, and exclusions, and all data, analysis code, and experimental materials for all studies are available for download at: <https://osf.io/pd9ca/>.

References
----------

1. Bostrom, N. & Cirkovic, M. M. *Global Catastrophic Risks* (OUP Oxford, 2011).
2. Bostrom, N. *Superintelligence* (Oxford University Press, 2014).
3. Rees, M. *Our Final Hour: A Scientist’s Warning* (Hachette UK, 2009).
4. Rees, M. Denial of catastrophic risks. *Science.* **339**, 1123 (2013).

[Article](https://doi.org/10.1126%2Fscience.1236756)
 [ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=2013Sci...339.1123R)
 [CAS](/articles/cas-redirect/1:CAS:528:DC%2BC3sXktFKgs78%3D)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Denial%20of%20catastrophic%20risks&journal=Science.&doi=10.1126%2Fscience.1236756&volume=339&publication_year=2013&author=Rees%2CM)
5. Cotton-Barratt, O., Farquhar, S., Halstead, J., Schubert, S. & Snyder-Beattie, A. Global catastrophic risks 2016. *Global Challenges Foundation* (2016).
6. Soage, A. B. The End of Days: Essays on the Apocalypse from Antiquity to Modernity. *Totalitarian Movements and Political Religions.* **10**, 375–377 (2009).

[Article](https://doi.org/10.1080%2F14690760903396385)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=The%20End%20of%20Days%3A%20Essays%20on%20the%20Apocalypse%20from%20Antiquity%20to%20Modernity&journal=Totalitarian%20Movements%20and%20Political%20Religions.&doi=10.1080%2F14690760903396385&volume=10&pages=375-377&publication_year=2009&author=Soage%2CAB)
7. Banks, A. C. The End of the World As We Know It: Faith, Fatalism, and Apocalypse. *Nova Religio.* **3**, 420–421 (2000).

[Article](https://doi.org/10.1525%2Fnr.2000.3.2.420)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=The%20End%20of%20the%20World%20As%20We%20Know%20It%3A%20Faith%2C%20Fatalism%2C%20and%20Apocalypse&journal=Nova%20Religio.&doi=10.1525%2Fnr.2000.3.2.420&volume=3&pages=420-421&publication_year=2000&author=Banks%2CAC)
8. Hall, J. R. *Apocalypse: From Antiquity to the Empire of Modernity* (John Wiley & Sons, 2013).
9. O’Leary, S. D. *Arguing the Apocalypse: A Theory of Millennial Rhetoric* (Oxford University Press, 1998).
10. Baumgartner, F. J., Graziano, F. & Weber, E. Longing for the End: A History of Millennialism in Western Civilization. *Utop. Stud.* **11**, 214–218 (2000).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Longing%20for%20the%20End%3A%20A%20History%20of%20Millennialism%20in%20Western%20Civilization&journal=Utop.%20Stud.&volume=11&pages=214-218&publication_year=2000&author=Baumgartner%2CFJ&author=Graziano%2CF&author=Weber%2CE)
11. Parfit, D. *Reasons and Persons* (OUP Oxford, 1984).
12. Bostrom, N. Existential risks: Analyzing human extinction scenarios and related hazards. *Journal of Evolution and Technology* **9**, 1–30 (2002).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Existential%20risks%3A%20Analyzing%20human%20extinction%20scenarios%20and%20related%20hazards&journal=Journal%20of%20Evolution%20and%20Technology&volume=9&pages=1-30&publication_year=2002&author=Bostrom%2CN)
13. Bostrom, N. When machines outsmart humans. *Futures.* **35**, 759–764 (2003).

[Article](https://doi.org/10.1016%2FS0016-3287%2803%2900026-0)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=When%20machines%20outsmart%20humans&journal=Futures.&doi=10.1016%2FS0016-3287%2803%2900026-0&volume=35&pages=759-764&publication_year=2003&author=Bostrom%2CN)
14. Bostrom, N. The doomsday argument. *Think.* **6**, 23–28 (2008).

[Article](https://doi.org/10.1017%2FS1477175600002943)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=The%20doomsday%20argument&journal=Think.&doi=10.1017%2FS1477175600002943&volume=6&pages=23-28&publication_year=2008&author=Bostrom%2CN)
15. Beckstead, N. On the overwhelming importance of shaping the far future. PhD Thesis, Rutgers University-Graduate School-New Brunswick (2013).
16. Bostrom, N. The Future of Humanity. In *New Waves in* Philosophy, *eds*. Jan-Kyrre Berg Olsen, Evan Selinger, & Soren Riis (New York: Palgrave McMillan, 2009).
17. Bostrom, N. Existential Risk Prevention as Global Priority. *Glob Policy.* **4**, 15–31 (2013).

[Article](https://doi.org/10.1111%2F1758-5899.12002)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Existential%20Risk%20Prevention%20as%20Global%20Priority&journal=Glob%20Policy.&doi=10.1111%2F1758-5899.12002&volume=4&pages=15-31&publication_year=2013&author=Bostrom%2CN)
18. Tonn, B. Beliefs about human extinction. *Futures.* **41**, 766–773 (2009).

[Article](https://doi.org/10.1016%2Fj.futures.2009.07.001)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Beliefs%20about%20human%20extinction&journal=Futures.&doi=10.1016%2Fj.futures.2009.07.001&volume=41&pages=766-773&publication_year=2009&author=Tonn%2CB)
19. Tonn, B., Hemrick, A. & Conrad, F. Cognitive representations of the future: Survey results. *Futures.* **38**, 810–829 (2006).

[Article](https://doi.org/10.1016%2Fj.futures.2005.12.005)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Cognitive%20representations%20of%20the%20future%3A%20Survey%20results&journal=Futures.&doi=10.1016%2Fj.futures.2005.12.005&volume=38&pages=810-829&publication_year=2006&author=Tonn%2CB&author=Hemrick%2CA&author=Conrad%2CF)
20. Wiener, J. B. The Tragedy of the Uncommons: On the Politics of Apocalypse. *Glob Policy.* **7**, 67–80 (2016).

[Article](https://doi.org/10.1111%2F1758-5899.12319)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=The%20Tragedy%20of%20the%20Uncommons%3A%20On%20the%20Politics%20of%20Apocalypse&journal=Glob%20Policy.&doi=10.1111%2F1758-5899.12319&volume=7&pages=67-80&publication_year=2016&author=Wiener%2CJB)
21. Hauser, O. P., Rand, D. G., Peysakhovich, A. & Nowak, M. A. Cooperating with the future. *Nature.* **511**, 220–223 (2014).

[Article](https://doi.org/10.1038%2Fnature13530)
 [ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=2014Natur.511..220H)
 [CAS](/articles/cas-redirect/1:CAS:528:DC%2BC2cXhtFehsL3E)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Cooperating%20with%20the%20future&journal=Nature.&doi=10.1038%2Fnature13530&volume=511&pages=220-223&publication_year=2014&author=Hauser%2COP&author=Rand%2CDG&author=Peysakhovich%2CA&author=Nowak%2CMA)
22. Pahl, S., Sheppard, S., Boomsma, C. & Groves, C. Perceptions of time in relation to climate change. *WIREs Clim Change.* **5**, 375–388 (2014).

[Article](https://doi.org/10.1002%2Fwcc.272)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Perceptions%20of%20time%20in%20relation%20to%20climate%20change&journal=WIREs%20Clim%20Change.&doi=10.1002%2Fwcc.272&volume=5&pages=375-388&publication_year=2014&author=Pahl%2CS&author=Sheppard%2CS&author=Boomsma%2CC&author=Groves%2CC)
23. Jacquet, J. *et al*. Intra- and intergenerational discounting in the climate game. *Nat. Clim. Chang.* **3**, 1025 (2013).

[Article](https://doi.org/10.1038%2Fnclimate2024)
 [ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=2013NatCC...3.1025J)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Intra-%20and%20intergenerational%20discounting%20in%20the%20climate%20game&journal=Nat.%20Clim.%20Chang.&doi=10.1038%2Fnclimate2024&volume=3&publication_year=2013&author=Jacquet%2CJ)
24. Kahan, D. M. Ideology, Motivated Reasoning, and Cognitive Reflection: An Experimental Study. *Judgm Decis Mak.* **8**, 407–424 (2013).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Ideology%2C%20Motivated%20Reasoning%2C%20and%20Cognitive%20Reflection%3A%20An%20Experimental%20Study&journal=Judgm%20Decis%20Mak.&volume=8&pages=407-424&publication_year=2013&author=Kahan%2CDM)
25. Spence, A., Poortinga, W. & Pidgeon, N. The psychological distance of climate change. *Risk Anal.* **32**, 957–972 (2012).

[Article](https://doi.org/10.1111%2Fj.1539-6924.2011.01695.x)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=The%20psychological%20distance%20of%20climate%20change&journal=Risk%20Anal.&doi=10.1111%2Fj.1539-6924.2011.01695.x&volume=32&pages=957-972&publication_year=2012&author=Spence%2CA&author=Poortinga%2CW&author=Pidgeon%2CN)
26. Kemper, C. J., Kovaleva, A., Beierlein, C. & Rammstedt, B. Measuring the construct of Optimism-Pessimism with single item indicators. Paper presented at the 4th Conference of the European Survey Research Association (ESRA), Lausanne, Switzerland (2011).
27. Caviola, L., Everett, J. A. C., Faber, N. S. The moral standing of animals: Towards a psychology of speciesism. *J*. *Pers*. *Soc*. *Psychol*., **116**, 1011–1029 (2019).
28. Frederick, S. Cognitive Reflection and Decision Making. *J. Econ. Perspect.* **19**, 25–42 (2005).

[Article](https://doi.org/10.1257%2F089533005775196732)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Cognitive%20Reflection%20and%20Decision%20Making&journal=J.%20Econ.%20Perspect.&doi=10.1257%2F089533005775196732&volume=19&pages=25-42&publication_year=2005&author=Frederick%2CS)
29. Farquhar, S., Cotton-Barratt, O., & Snyder-Beattie, A. Pricing Externalities to Balance Public Risks and Benefits of Research. *Health Secur* **15**, 401–408 (2017).

[Article](https://doi.org/10.1089%2Fhs.2016.0118)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Pricing%20Externalities%20to%20Balance%20Public%20Risks%20and%20Benefits%20of%20Research&journal=Health%20Secur&doi=10.1089%2Fhs.2016.0118&volume=15&pages=401-408&publication_year=2017&author=Farquhar%2CS&author=Cotton-Barratt%2CO&author=Snyder-Beattie%2CA)
30. Kahane, G. *et al*. Beyond sacrificial harm: A two-dimensional model of utilitarian psychology. *Psychol Rev*, **125**, 131–164.

[Download references](https://citation-needed.springer.com/v2/references/10.1038/s41598-019-50145-9?format=refman&flavour=references)

Acknowledgements
----------------

The Berkeley Existential Risk Initiative, Centre for Effective Altruism, Janggen-Poehn Stiftung, Swiss Study Foundation, and the Oxford Martin School (Oxford Martin Programme on Collective Responsibility for Infectious Disease) supported this research. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The authors thank Fabienne Sandkühler for her extensive help and comments, Dillon Plunkett for the idea of one of the conditions, and Gregory Lewis, Pablo Stafforini and Andreas Mogensen for their helpful suggestions.

Author information
------------------

Author notes1. Stefan Schubert and Lucius Caviola contributed equally.

### Authors and Affiliations

1. Department of Experimental Psychology, University of Oxford, New Radcliffe House, Radcliffe Observatory Quarter, Woodstock Road, OX2 6GG, Oxford, United Kingdom

Stefan Schubert, Lucius Caviola & Nadira S. Faber
2. Oxford Uehiro Centre for Practical Ethics, University of Oxford, 16-17 St Ebbes St, Oxford, OX1 1PT, United Kingdom

Nadira S. Faber
3. College of Life and Environmental Sciences, University of Exeter, Washington Singer Building, Exeter, EX4 4QG, United Kingdom

Nadira S. Faber

Authors1. Stefan Schubert[View author publications](/search?author=Stefan%20Schubert)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Stefan%20Schubert) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Stefan%20Schubert%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
2. Lucius Caviola[View author publications](/search?author=Lucius%20Caviola)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Lucius%20Caviola) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Lucius%20Caviola%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
3. Nadira S. Faber[View author publications](/search?author=Nadira%20S.%20Faber)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Nadira%20S.%20Faber) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Nadira%20S.%20Faber%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
### Contributions

S.S., L.C. and N.S.F. planned the studies. S.S. and L.C. collected and analyzed the data. S.S., L.C. and N.S.F. interpreted that data and wrote the paper.

### Corresponding author

Correspondence to
 [Stefan Schubert](mailto:stefan.schubert@psy.ox.ac.uk).

Ethics declarations
-------------------

### Competing Interests

The authors declare no competing interests.

Additional information
----------------------

**Publisher’s note** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information
-------------------------

### [Supplementary Materials](https://static-content.springer.com/esm/art%3A10.1038%2Fs41598-019-50145-9/MediaObjects/41598_2019_50145_MOESM1_ESM.pdf)

Rights and permissions
----------------------

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit <http://creativecommons.org/licenses/by/4.0/>.

[Reprints and Permissions](https://s100.copyright.com/AppDispatchServlet?title=The%20Psychology%20of%20Existential%20Risk%3A%20Moral%20Judgments%20about%20Human%20Extinction&author=Stefan%20Schubert%20et%20al&contentID=10.1038%2Fs41598-019-50145-9&copyright=The%20Author%28s%29&publication=2045-2322&publicationDate=2019-10-21&publisherName=SpringerNature&orderBeanReset=true&oa=CC%20BY)

Comments
--------

By submitting a comment you agree to abide by our [Terms](/info/tandc.html) and [Community Guidelines](/info/community-guidelines.html). If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

The Psychology of Existential Risk: Moral Judgments about Human Extinction

Stefan Schubert
Lucius Caviola
Nadira S. Faber


### Subjects

* [Electrical and electronic engineering](/subjects/electrical-and-electronic-engineering)
* [Ethics](/subjects/ethics)
* [Policy](/subjects/policy)
* [Science, technology and society](/subjects/science-technology-and-society)

Debate about the impacts of AI is often split into two camps, one associated with the near term and the other with the long term. This divide is a mistake — the connections between the two perspectives deserve more attention, say Stephen Cave and Seán S. ÓhÉigeartaigh.

[Access through your institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fs42256-018-0003-2)

[Buy or subscribe](#access-options)

These two sets of issues are often seen as entirely disconnected[1](/articles/s42256-018-0003-2#ref-CR1 "Baum, S. D. AI Soc.
                    https://doi.org/10.1007/s00146-017-0734-3

                   (2017)."). Researchers working on near-term issues see longer-term issues as a distraction from real and pressing challenges[8](/articles/s42256-018-0003-2#ref-CR8 "Calo, R. Artificial Intelligence Policy: A Roadmap (UC Davis, Davis, 2017)."), or as too distant, uncertain or speculative to allow for productive work now[9](/articles/s42256-018-0003-2#ref-CR9 "Williams, C. The Register
                    https://www.theregister.co.uk/2015/03/19/andrew_ng_baidu_ai/

                   (2015)."). On the other hand, those focused on longer-term challenges argue that their potential impact dwarfs that of present-day systems[7](/articles/s42256-018-0003-2#ref-CR7 "Tegmark, M. Life 3.0. Being Human in the Age of Artificial Intelligence (Allen Lane, New York, 2017)."), and that these issues therefore deserve a proportionate share of research attention.

This is a preview of subscription content, [access via your institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fs42256-018-0003-2)

 if (window.dataLayer) {
 window.dataLayer.push({
 content: { article: { relevantArticlesCount: 3 }}
 })
 }

Relevant articles
-----------------

Open Access articles citing this article.

* ###
[Embedding responsibility in intelligent systems: from AI ethics to responsible AI ecosystems](https://doi.org/10.1038/s41598-023-34622-w)

	+ Bernd Carsten Stahl
*Scientific Reports*
Open Access
18 May 2023
* ###
[General intelligence disentangled via a generality metric for natural and artificial intelligence](https://doi.org/10.1038/s41598-021-01997-7)

	+ José Hernández-Orallo
	+ , Bao Sheng Loe
	+ … Seán Ó hÉigeartaigh
*Scientific Reports*
Open Access
24 November 2021
* ###
[From computer ethics and the ethics of AI towards an ethics of digital ecosystems](https://doi.org/10.1007/s43681-021-00080-1)

	+ Bernd Carsten Stahl
*AI and Ethics*
Open Access
31 July 2021

Access options
--------------

[Access through your institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fs42256-018-0003-2)

[Access through your institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fs42256-018-0003-2)

[Change institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fs42256-018-0003-2)

[Buy or subscribe](#access-options)

/\* style specs start \*/
style{display:none!important}.LiveAreaSection-193358632 \*{align-content:stretch;align-items:stretch;align-self:auto;animation-delay:0s;animation-direction:normal;animation-duration:0s;animation-fill-mode:none;animation-iteration-count:1;animation-name:none;animation-play-state:running;animation-timing-function:ease;azimuth:center;backface-visibility:visible;background-attachment:scroll;background-blend-mode:normal;background-clip:borderBox;background-color:transparent;background-image:none;background-origin:paddingBox;background-position:0 0;background-repeat:repeat;background-size:auto auto;block-size:auto;border-block-end-color:currentcolor;border-block-end-style:none;border-block-end-width:medium;border-block-start-color:currentcolor;border-block-start-style:none;border-block-start-width:medium;border-bottom-color:currentcolor;border-bottom-left-radius:0;border-bottom-right-radius:0;border-bottom-style:none;border-bottom-width:medium;border-collapse:separate;border-image-outset:0s;border-image-repeat:stretch;border-image-slice:100%;border-image-source:none;border-image-width:1;border-inline-end-color:currentcolor;border-inline-end-style:none;border-inline-end-width:medium;border-inline-start-color:currentcolor;border-inline-start-style:none;border-inline-start-width:medium;border-left-color:currentcolor;border-left-style:none;border-left-width:medium;border-right-color:currentcolor;border-right-style:none;border-right-width:medium;border-spacing:0;border-top-color:currentcolor;border-top-left-radius:0;border-top-right-radius:0;border-top-style:none;border-top-width:medium;bottom:auto;box-decoration-break:slice;box-shadow:none;box-sizing:border-box;break-after:auto;break-before:auto;break-inside:auto;caption-side:top;caret-color:auto;clear:none;clip:auto;clip-path:none;color:initial;column-count:auto;column-fill:balance;column-gap:normal;column-rule-color:currentcolor;column-rule-style:none;column-rule-width:medium;column-span:none;column-width:auto;content:normal;counter-increment:none;counter-reset:none;cursor:auto;display:inline;empty-cells:show;filter:none;flex-basis:auto;flex-direction:row;flex-grow:0;flex-shrink:1;flex-wrap:nowrap;float:none;font-family:initial;font-feature-settings:normal;font-kerning:auto;font-language-override:normal;font-size:medium;font-size-adjust:none;font-stretch:normal;font-style:normal;font-synthesis:weight style;font-variant:normal;font-variant-alternates:normal;font-variant-caps:normal;font-variant-east-asian:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-position:normal;font-weight:400;grid-auto-columns:auto;grid-auto-flow:row;grid-auto-rows:auto;grid-column-end:auto;grid-column-gap:0;grid-column-start:auto;grid-row-end:auto;grid-row-gap:0;grid-row-start:auto;grid-template-areas:none;grid-template-columns:none;grid-template-rows:none;height:auto;hyphens:manual;image-orientation:0deg;image-rendering:auto;image-resolution:1dppx;ime-mode:auto;inline-size:auto;isolation:auto;justify-content:flexStart;left:auto;letter-spacing:normal;line-break:auto;line-height:normal;list-style-image:none;list-style-position:outside;list-style-type:disc;margin-block-end:0;margin-block-start:0;margin-bottom:0;margin-inline-end:0;margin-inline-start:0;margin-left:0;margin-right:0;margin-top:0;mask-clip:borderBox;mask-composite:add;mask-image:none;mask-mode:matchSource;mask-origin:borderBox;mask-position:0 0;mask-repeat:repeat;mask-size:auto;mask-type:luminance;max-height:none;max-width:none;min-block-size:0;min-height:0;min-inline-size:0;min-width:0;mix-blend-mode:normal;object-fit:fill;object-position:50% 50%;offset-block-end:auto;offset-block-start:auto;offset-inline-end:auto;offset-inline-start:auto;opacity:1;order:0;orphans:2;outline-color:initial;outline-offset:0;outline-style:none;outline-width:medium;overflow:visible;overflow-wrap:normal;overflow-x:visible;overflow-y:visible;padding-block-end:0;padding-block-start:0;padding-bottom:0;padding-inline-end:0;padding-inline-start:0;padding-left:0;padding-right:0;padding-top:0;page-break-after:auto;page-break-before:auto;page-break-inside:auto;perspective:none;perspective-origin:50% 50%;pointer-events:auto;position:static;quotes:initial;resize:none;right:auto;ruby-align:spaceAround;ruby-merge:separate;ruby-position:over;scroll-behavior:auto;scroll-snap-coordinate:none;scroll-snap-destination:0 0;scroll-snap-points-x:none;scroll-snap-points-y:none;scroll-snap-type:none;shape-image-threshold:0;shape-margin:0;shape-outside:none;tab-size:8;table-layout:auto;text-align:initial;text-align-last:auto;text-combine-upright:none;text-decoration-color:currentcolor;text-decoration-line:none;text-decoration-style:solid;text-emphasis-color:currentcolor;text-emphasis-position:over right;text-emphasis-style:none;text-indent:0;text-justify:auto;text-orientation:mixed;text-overflow:clip;text-rendering:auto;text-shadow:none;text-transform:none;text-underline-position:auto;top:auto;touch-action:auto;transform:none;transform-box:borderBox;transform-origin:50% 50%0;transform-style:flat;transition-delay:0s;transition-duration:0s;transition-property:all;transition-timing-function:ease;vertical-align:baseline;visibility:visible;white-space:normal;widows:2;width:auto;will-change:auto;word-break:normal;word-spacing:normal;word-wrap:normal;writing-mode:horizontalTb;z-index:auto;-webkit-appearance:none;-moz-appearance:none;-ms-appearance:none;appearance:none;margin:0}.LiveAreaSection-193358632{width:100%}.LiveAreaSection-193358632 .login-option-buybox{display:block;width:100%;font-size:17px;line-height:30px;color:#222;padding-top:30px;font-family:Harding,Palatino,serif}.LiveAreaSection-193358632 .additional-access-options{display:block;font-weight:700;font-size:17px;line-height:30px;color:#222;font-family:Harding,Palatino,serif}.LiveAreaSection-193358632 .additional-login>li:not(:first-child)::before{transform:translateY(-50%);content:"";height:1rem;position:absolute;top:50%;left:0;border-left:2px solid #999}.LiveAreaSection-193358632 .additional-login>li:not(:first-child){padding-left:10px}.LiveAreaSection-193358632 .additional-login>li{display:inline-block;position:relative;vertical-align:middle;padding-right:10px}.BuyBoxSection-683559780{display:flex;flex-wrap:wrap;flex:1;flex-direction:row-reverse;margin:-30px -15px 0}.BuyBoxSection-683559780 .box-inner{width:100%;height:100%}.BuyBoxSection-683559780 .readcube-buybox{background-color:#f3f3f3;flex-shrink:1;flex-grow:1;flex-basis:255px;background-clip:content-box;padding:0 15px;margin-top:30px}.BuyBoxSection-683559780 .subscribe-buybox{background-color:#f3f3f3;flex-shrink:1;flex-grow:4;flex-basis:300px;background-clip:content-box;padding:0 15px;margin-top:30px}.BuyBoxSection-683559780 .subscribe-buybox-nature-plus{background-color:#f3f3f3;flex-shrink:1;flex-grow:4;flex-basis:100%;background-clip:content-box;padding:0 15px;margin-top:30px}.BuyBoxSection-683559780 .title-readcube,.BuyBoxSection-683559780 .title-buybox{display:block;margin:0;margin-right:10%;margin-left:10%;font-size:24px;line-height:32px;color:#222;padding-top:30px;text-align:center;font-family:Harding,Palatino,serif}.BuyBoxSection-683559780 .title-asia-buybox{display:block;margin:0;margin-right:5%;margin-left:5%;font-size:24px;line-height:32px;color:#222;padding-top:30px;text-align:center;font-family:Harding,Palatino,serif}.BuyBoxSection-683559780 .asia-link{color:#069;cursor:pointer;text-decoration:none;font-size:1.05em;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:1.05em6}.BuyBoxSection-683559780 .access-readcube{display:block;margin:0;margin-right:10%;margin-left:10%;font-size:14px;color:#222;padding-top:10px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:20px}.BuyBoxSection-683559780 .access-asia-buybox{display:block;margin:0;margin-right:5%;margin-left:5%;font-size:14px;color:#222;padding-top:10px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:20px}.BuyBoxSection-683559780 .access-buybox{display:block;margin:0;margin-right:10%;margin-left:10%;font-size:14px;color:#222;opacity:.8px;padding-top:10px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:20px}.BuyBoxSection-683559780 .price-buybox{display:block;font-size:30px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;padding-top:30px;text-align:center}.BuyBoxSection-683559780 .price-buybox-to{display:block;font-size:30px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;text-align:center}.BuyBoxSection-683559780 .price-info-text{font-size:16px;padding-right:10px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif}.BuyBoxSection-683559780 .price-value{font-size:30px;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif}.BuyBoxSection-683559780 .price-per-period{font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif}.BuyBoxSection-683559780 .price-from{font-size:14px;padding-right:10px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:20px}.BuyBoxSection-683559780 .issue-buybox{display:block;font-size:13px;text-align:center;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:19px}.BuyBoxSection-683559780 .no-price-buybox{display:block;font-size:13px;line-height:18px;text-align:center;padding-right:10%;padding-left:10%;padding-bottom:20px;padding-top:30px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif}.BuyBoxSection-683559780 .vat-buybox{display:block;margin-top:5px;margin-right:20%;margin-left:20%;font-size:11px;color:#222;padding-top:10px;padding-bottom:15px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:17px}.BuyBoxSection-683559780 .tax-buybox{display:block;width:100%;color:#222;padding:20px 16px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:NaNpx}.BuyBoxSection-683559780 .button-container{display:flex;padding-right:20px;padding-left:20px;justify-content:center}.BuyBoxSection-683559780 .button-container>\*{flex:1px}.BuyBoxSection-683559780 .button-container>a:hover,.Button-505204839:hover,.Button-1078489254:hover,.Button-2496381730:hover{text-decoration:none}.BuyBoxSection-683559780 .readcube-button{background:#fff;margin-top:30px}.BuyBoxSection-683559780 .button-asia{background:#069;border:1px solid #069;border-radius:0;cursor:pointer;display:block;padding:9px;outline:0;text-align:center;text-decoration:none;min-width:80px;margin-top:75px}.BuyBoxSection-683559780 .button-label-asia,.ButtonLabel-3869432492,.ButtonLabel-3296148077,.ButtonLabel-1651148777{display:block;color:#fff;font-size:17px;line-height:20px;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;text-align:center;text-decoration:none;cursor:pointer}.Button-505204839,.Button-1078489254,.Button-2496381730{background:#069;border:1px solid #069;border-radius:0;cursor:pointer;display:block;padding:9px;outline:0;text-align:center;text-decoration:none;min-width:80px;max-width:320px;margin-top:10px}.Button-505204839 .readcube-label,.Button-1078489254 .readcube-label,.Button-2496381730 .readcube-label{color:#069}
/\* style specs end \*/Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

[Learn more](https://shop.nature.com/products/plus)Subscribe to this journal

Receive 12 digital issues and online access to articles

99,00 € per year

only 8,25 € per issue

[Learn more](/natmachintell/subscribe)Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

[Learn more](//www.nature.com/articles/s42256-018-0003-2.epdf?no_publisher_access=1&r3_referer=nature)Prices may be subject to local taxes which are calculated during checkout

### Additional access options:

* [Log in](https://idp.nature.com/authorize/natureuser?client_id=grover&redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fs42256-018-0003-2)
* [Learn about institutional subscriptions](https://www.springernature.com/gp/librarians/licensing/license-options)
* [Read our FAQs](https://support.nature.com/en/support/home)
* [Contact customer support](https://www.springernature.com/gp/contact)

References
----------

1. Baum, S. D. *AI Soc.* <https://doi.org/10.1007/s00146-017-0734-3> (2017).

[Article](https://doi.org/10.1007%2Fs00146-017-0734-3)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=&journal=AI%20Soc.&doi=10.1007%2Fs00146-017-0734-3&publication_year=2017&author=Baum%2CSD)
2. Crawford, K. et al. *The AI Now Report: The Social and Economic Implications of Artificial Intelligence Technologies in the Near-Term* (AI Now, 2016); <https://ainowinstitute.org/AI_Now_2016_Report.pdf>
3. Brundage, M. et al. Preprint at <https://arxiv.org/abs/1802.07228> (2018).
4. Dietterich, T. G. & Horvitz, E. J. *Commun. ACM* **58**, 38–40 (2015).

[Article](https://doi.org/10.1145%2F2770869)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=&journal=Commun.%20ACM&doi=10.1145%2F2770869&volume=58&pages=38-40&publication_year=2015&author=Dietterich%2CTG&author=Horvitz%2CEJ)
5. Frey, C. B. & Osborne, M. A. *Technol. Forecast. Soc. Change* **114**, 254–280 (2017).

[Article](https://doi.org/10.1016%2Fj.techfore.2016.08.019)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=&journal=Technol.%20Forecast.%20Soc.%20Change&doi=10.1016%2Fj.techfore.2016.08.019&volume=114&pages=254-280&publication_year=2017&author=Frey%2CCB&author=Osborne%2CMA)
6. Bostrom, N. *Superintelligence: Paths, Dangers, Strategies* (Oxford Univ. Press, Oxford, 2014).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Superintelligence%3A%20Paths%2C%20Dangers%2C%20Strategies&publication_year=2014&author=Bostrom%2CN)
7. Tegmark, M. *Life 3.0. Being Human in the Age of Artificial Intelligence* (Allen Lane, New York, 2017).
8. Calo, R. *Artificial Intelligence Policy: A Roadmap* (UC Davis, Davis, 2017).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Artificial%20Intelligence%20Policy%3A%20A%20Roadmap&publication_year=2017&author=Calo%2CR)
9. Williams, C. *The Register* <https://www.theregister.co.uk/2015/03/19/andrew_ng_baidu_ai/> (2015).
10. *The Dawn of Artificial Intelligence* (US Government Publishing Office, 2016); <https://www.gpo.gov/fdsys/pkg/CHRG-114shrg24175/html/CHRG-114shrg24175.htm>
11. Russell, S., Dewey, D. & Tegmark, M. *AI Magazine* **36**, 105–114 (Winter, 2015).
12. Amodei, D. et al. Preprint at <https://arxiv.org/abs/1606.06565> (2016).
13. Owen, R. et al. in *Responsible Innovation: Managing the Responsible Emergence of Science and Innovation in Society* (eds Owen, R., Bessant, J. & Heintz, M.) 27–50 (Wiley, Chichester, 2013).

[Download references](https://citation-needed.springer.com/v2/references/10.1038/s42256-018-0003-2?format=refman&flavour=references)

Author information
------------------

### Authors and Affiliations

1. Leverhulme Centre for the Future of Intelligence, University of Cambridge, Cambridge, UK

Stephen Cave & Seán S. ÓhÉigeartaigh

Authors1. Stephen Cave[View author publications](/search?author=Stephen%20Cave)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Stephen%20Cave) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Stephen%20Cave%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
2. Seán S. ÓhÉigeartaigh[View author publications](/search?author=Se%C3%A1n%20S.%20%C3%93h%C3%89igeartaigh)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Se%C3%A1n%20S.%20%C3%93h%C3%89igeartaigh) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Se%C3%A1n%20S.%20%C3%93h%C3%89igeartaigh%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
### Corresponding authors

Correspondence to
 [Stephen Cave](mailto:sjc53@cam.ac.uk) or [Seán S. ÓhÉigeartaigh](mailto:so348@cam.ac.uk).

Ethics declarations
-------------------

### Competing interests

The authors declare no competing interests.

Rights and permissions
----------------------

[Reprints and Permissions](https://s100.copyright.com/AppDispatchServlet?title=Bridging%20near-%20and%20long-term%20concerns%20about%20AI&author=Stephen%20Cave%20et%20al&contentID=10.1038%2Fs42256-018-0003-2&copyright=Springer%20Nature%20Limited&publication=2522-5839&publicationDate=2019-01-07&publisherName=SpringerNature&orderBeanReset=true)

This article is cited by
------------------------

* ###
[Embedding responsibility in intelligent systems: from AI ethics to responsible AI ecosystems](https://doi.org/10.1038/s41598-023-34622-w)

	+ Bernd Carsten Stahl*Scientific Reports* (2023)
* ###
[From computer ethics and the ethics of AI towards an ethics of digital ecosystems](https://doi.org/10.1007/s43681-021-00080-1)

	+ Bernd Carsten Stahl*AI and Ethics* (2022)
* ###
[General intelligence disentangled via a generality metric for natural and artificial intelligence](https://doi.org/10.1038/s41598-021-01997-7)

	+ José Hernández-Orallo
	+ Bao Sheng Loe
	+ Seán Ó hÉigeartaigh*Scientific Reports* (2021)
* ###
[Facilitators and Barriers of Artificial Intelligence Adoption in Business – Insights from Opinions Using Big Data Analytics](https://doi.org/10.1007/s10796-021-10219-4)

	+ Arpan Kumar Kar
	+ Amit Kumar Kushwaha*Information Systems Frontiers* (2021)
* ###
[Our future in the Anthropocene biosphere](https://doi.org/10.1007/s13280-021-01544-8)

	+ Carl Folke
	+ Stephen Polasky
	+ Brian H. Walker*Ambio* (2021)

Bridging near- and long-term concerns about AI


[Download PDF](/articles/s42256-020-0195-0.pdf)

### Subjects

* [Ethics](/subjects/ethics)
* [Infectious diseases](/subjects/infectious-diseases)
* [SARS-CoV-2](/subjects/sars-cov-2)
* [Science, technology and society](/subjects/science-technology-and-society)

Artificial intelligence tools can help save lives in a pandemic. However, the need to implement technological solutions rapidly raises challenging ethical issues. We need new approaches for ethics with urgency, to ensure AI can be safely and beneficially used in the COVID-19 response and beyond.

The novel coronavirus pandemic (COVID-19) is the largest global crisis in a generation, hitting the world at a time when artificial intelligence (AI) is showing potential for widespread real-world application. We are currently seeing a rapid increase in proposals for how AI can be used in many stages of pandemic prevention and response. AI can aid in detecting, understanding and predicting the spread of disease, which can provide early warning signs and inform effective interventions[1](/articles/s42256-020-0195-0#ref-CR1 "van der Schaar, M. et al. Preprint at
http://www.vanderschaar-lab.com/NewWebsite/covid-19/post1/paper.pdf

 (2020)."). AI may improve the medical response to the pandemic in several ways: supporting physicians by automating aspects of diagnosis[2](/articles/s42256-020-0195-0#ref-CR2 "Wang, S. et al. Preprint at
https://doi.org/10.1101/2020.02.14.20023028

 (2020)."), prioritizing healthcare resources[3](/articles/s42256-020-0195-0#ref-CR3 "Butt, C., Gill, J., Chun, D. & Babu, B. A. Appl. Intell.
https://doi.org/10.1007/s10489-020-01714-3

 (2020)."), and improving vaccine and drug development[4](/articles/s42256-020-0195-0#ref-CR4 "Zhang, H. et al. Preprint at
https://doi.org/10.20944/preprints202002.0061.v1

 (2020)."). AI also has potential applications beyond immediate response, such as in combating online misinformation about COVID-19[5](/articles/s42256-020-0195-0#ref-CR5 "Infodemic management - infodemiology. World Health Organization
https://www.who.int/teams/risk-communication/infodemic-management

 (2020).").

The current crisis presents an unprecedented opportunity to leverage AI for societal benefit. However, the urgency with which new technologies must be deployed raises particularly challenging ethical issues and risks. There is growing concern that the use of AI and data in response to COVID-19 may compromise privacy and civil liberties by incentivizing the collection and processing of large amounts of data, which may often be private or personal[6](/articles/s42256-020-0195-0#ref-CR6 "Ienca, M. & Vayena, E. Nat. Med. 26, 463–464 (2020)."). More broadly, although AI clearly has a great deal to offer, we must be careful not to overestimate its potential. Its efficacy will heavily depend on the reliability and relevance of the data available. With the worldwide spread of COVID-19 occurring so quickly, obtaining sufficient data for accurate AI forecasting and diagnosis is challenging. Even where AI models are strictly speaking accurate, they may have differential impacts across subpopulations, with harmful consequences that are difficult to predict in advance[7](/articles/s42256-020-0195-0#ref-CR7 "Wynants, L. et al. BMJ 369, m1328 (2020)."). A further concern is that the lack of transparency in AI systems used to aid decision-making around COVID-19 may make it near impossible for the decisions of governments and public officials to be subject to public scrutiny and legitimation[8](/articles/s42256-020-0195-0#ref-CR8 "Nyrup, R., Whittlestone, J. & Cave, S. Why Value Judgements Should Not Be Automated (Leverhulme Centre for the Future of Intelligence, 2019);
https://doi.org/10.17863/CAM.41552

"). Finally, the current crisis may have longer-term impacts on public trust and norms around the use of AI in society. How these develop will depend on perceptions of how successful and responsible use of AI to address COVID-19 is.

The challenge of ethics in a crisis
-----------------------------------

Robust ethics and risk assessment processes are needed to ensure AI is used responsibly in response to COVID-19. However, implementing these at a time of crisis is far from straightforward, especially where new technologies need to be deployed at unprecedented speed and scale. For example, forecasting models have to be available at the early stages of disease spread and make use of all possible data to productively inform policy interventions. Current processes for ethics and risk assessment around uses of AI are still relatively immature, and the urgency of a crisis highlights their limitations.

Much work in AI ethics in recent years has focused on developing high-level principles, but these principles say nothing about what to do when principles come into conflict with one another[9](/articles/s42256-020-0195-0#ref-CR9 "Whittlestone, J., Nyrup, R., Alexandrova, A. & Cave, S. In Proc. 2019 AAAI/ACM Conf. AI, Ethics, and Society 195–200 (ACM, 2019)."). For example, principles do not tell us how to balance the potential of AI to save lives (the principle of ‘beneficence’) against other important values such as privacy or fairness. One common suggestion for navigating such tensions is through engagement with diverse stakeholder groups, but this may be difficult to enact with sufficient speed at times of crisis.

When new technologies may pose unknown risks, we would ordinarily try to introduce them in gradual, iterative ways, allowing time for issues to be identified and addressed. In the context of a crisis, however, there is a stark trade-off between a cautious approach and the need to deploy technological solutions at scale. For example, there may be pressure to rely on systems with less human oversight and potential for override due to staff shortages and time pressures, but this must be carefully balanced against the risk of failing to notice or override crucial failures.

This does not mean that ethics should be neglected at times of crisis. It only emphasizes that we must find ways to conduct ethical review and risk assessment with the same urgency that motivates the development of AI-based solutions.

Doing ethics with urgency
-------------------------

We suggest that ethics with urgency must at a minimum incorporate the following components: (1) the ability to think ahead rather than dealing with problems reactively, (2) more robust procedures for assuring the behaviour and safety of AI systems, and (3) building public trust through independent oversight.

First, ethics with urgency must involve thinking through possible issues and risks as thoroughly as possible before systems are developed and deployed in the world. This need to think ahead is reflected in the notion of ‘ethics by design’: making ethical considerations part of the process of developing new applications of AI, not an afterthought[10](/articles/s42256-020-0195-0#ref-CR10 "d’Aquin, M. et al. In Proc. 2018 AAAI/ACM Conf. AI, Ethics, and Society 54–59 (ACM, 2018)."). For example, questions such as ‘what data do we need and what issues might this raise?’ and ‘how do we build this model so that it is possible to interrogate key assumptions?’ need to be considered throughout the development process. This means that experts in ethics and risk assessment need to be involved in teams developing AI-based solutions from the beginning, and much clearer guidelines are needed for engineers and developers to think through these issues. An ethics by design approach should also be supplemented with more extensive foresight work, looking beyond the more obvious and immediate ethical issues, and considering a wider range of longer-term and more systemic impacts. By synthesizing diverse sources of expertise, established foresight methodologies can be used to identify new risks and key uncertainties likely to shape the future, and use this to make better informed decisions today[11](/articles/s42256-020-0195-0#ref-CR11 "The Futures Toolkit: Tools for Futures Thinking and Foresight Across UK Government (Government Office for Science, 2017).").

Second, where applications of AI are used at scale in safety-critical domains such as healthcare, ensuring the safety and reliability of those systems across a range of scenarios is of crucial importance. Finding ways to rapidly conduct robust testing and verification of systems will therefore be central to doing ethics with urgency. We suggest that the application of AI in crisis scenarios should in particular be heavily informed by research on best practices for the verification and validation of autonomous systems[12](/articles/s42256-020-0195-0#ref-CR12 "Lyons, J. B., Clark, M. A., Wagner, A. R. & Schuelke, M. J. AI Mag. 38(3), 37–49 (2017)."). It may also be worthwhile for governments to fund further work on methods for establishing the reliability of machine learning systems across a range of circumstances, particularly where those systems may be deployed in high-stakes crisis scenarios.

Third, an important aspect of ethics with urgency is building public trust in how AI is being used. If governments use AI systems in ways perceived to be either mistaken or problematically value-laden, this could result in a loss of public trust severe enough to drastically reduce support for beneficial uses of AI not just in this crisis, but also in the future. Building public trust around new uses of technology may be particularly challenging in crisis times, where the need to move fast makes it easier for governments to fall back on opaque and centralized forms of decision-making. Several analyses of past pandemics have argued that transparency and public scrutiny are essential for maintaining public trust[13](/articles/s42256-020-0195-0#ref-CR13 "O’Malley, P., Rainford, J. & Thompson, A. Bull. World Health Organ. 87, 614–618 (2009)."). An independent oversight body, responsible for reviewing any potential risks and ethical issues associated with new technologies and producing publicly available reports, could help ensure public transparency. This oversight body could, among other approaches, make use of techniques such as ‘red teaming’ to rigorously challenge systems and their assumptions, unearthing any limitations and biases in the applications being proposed[14](/articles/s42256-020-0195-0#ref-CR14 "Brundage, M. et al. Preprint at
https://arxiv.org/abs/2004.07213

 (2020)."). Red teaming is widely used in security settings, but can be applied broadly: at its core, red teaming is a way of challenging the blind spots of a team by explicitly looking for flaws from an outsider or adversarial perspective. As well as allowing developers to identify and fix issues before deployment, such processes could help assure public stakeholders that the interests and values of different groups are being thoroughly considered, and that all eventualities are prepared for.

Conclusion
----------

As the COVID-19 pandemic illustrates, times of crisis can necessitate rapid deployment of new technologies in order to save lives. However, this urgency both makes it more likely that ethical issues and risks will arise, and makes them more challenging to address. Rather than neglecting ethics, we must find ways to do ethics with urgency too. We strongly encourage technologists, ethicists, policymakers and healthcare professionals to consider how ethics can be implemented at speed in the ongoing response to the COVID-19 crisis. If ethical practices can be implemented with urgency, the current crisis could provide an opportunity to drive greater application of AI for societal benefit, and to build public trust in such applications.

References
----------

1. van der Schaar, M. et al. Preprint at <http://www.vanderschaar-lab.com/NewWebsite/covid-19/post1/paper.pdf> (2020).
2. Wang, S. et al. Preprint at <https://doi.org/10.1101/2020.02.14.20023028> (2020).
3. Butt, C., Gill, J., Chun, D. & Babu, B. A. *Appl. Intell.* <https://doi.org/10.1007/s10489-020-01714-3> (2020).
4. Zhang, H. et al. Preprint at <https://doi.org/10.20944/preprints202002.0061.v1> (2020).
5. Infodemic management - infodemiology. *World Health Organization* <https://www.who.int/teams/risk-communication/infodemic-management> (2020).
6. Ienca, M. & Vayena, E. *Nat. Med.* **26**, 463–464 (2020).

[Article](https://doi.org/10.1038%2Fs41591-020-0832-5)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=&journal=Nat.%20Med.&doi=10.1038%2Fs41591-020-0832-5&volume=26&pages=463-464&publication_year=2020&author=Ienca%2CM&author=Vayena%2CE)
7. Wynants, L. et al. *BMJ* **369**, m1328 (2020).

[Article](https://doi.org/10.1136%2Fbmj.m1328)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=&journal=BMJ&doi=10.1136%2Fbmj.m1328&volume=369&publication_year=2020&author=Wynants%2CL)
8. Nyrup, R., Whittlestone, J. & Cave, S. *Why Value Judgements Should Not Be Automated* (Leverhulme Centre for the Future of Intelligence, 2019); <https://doi.org/10.17863/CAM.41552>
9. Whittlestone, J., Nyrup, R., Alexandrova, A. & Cave, S. In *Proc. 2019 AAAI/ACM Conf. AI, Ethics, and Society* 195–200 (ACM, 2019).
10. d’Aquin, M. et al. In *Proc*. *2018 AAAI/ACM Conf. AI, Ethics, and Society* 54–59 (ACM, 2018).
11. *The Futures Toolkit: Tools for Futures Thinking and Foresight Across UK Government* (Government Office for Science, 2017).
12. Lyons, J. B., Clark, M. A., Wagner, A. R. & Schuelke, M. J. *AI Mag.* **38**(3), 37–49 (2017).

[Article](https://doi.org/10.1609%2Faimag.v38i3.2717)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=&journal=AI%20Mag.&doi=10.1609%2Faimag.v38i3.2717&volume=38&issue=3&pages=37-49&publication_year=2017&author=Lyons%2CJB&author=Clark%2CMA&author=Wagner%2CAR&author=Schuelke%2CMJ)
13. O’Malley, P., Rainford, J. & Thompson, A. *Bull. World Health Organ.* **87**, 614–618 (2009).

[Article](https://doi.org/10.2471%2FBLT.08.056689)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=&journal=Bull.%20World%20Health%20Organ.&doi=10.2471%2FBLT.08.056689&volume=87&pages=614-618&publication_year=2009&author=O%E2%80%99Malley%2CP&author=Rainford%2CJ&author=Thompson%2CA)
14. Brundage, M. et al. Preprint at <https://arxiv.org/abs/2004.07213> (2020).

[Download references](https://citation-needed.springer.com/v2/references/10.1038/s42256-020-0195-0?format=refman&flavour=references)

Author information
------------------

### Authors and Affiliations

1. Centre for the Study of Existential Risk, University of Cambridge, Cambridge, UK

Asaf Tzachor & Lalitha Sundaram
2. Leverhulme Centre for the Future of Intelligence, University of Cambridge, Cambridge, UK

Jess Whittlestone & Seán Ó hÉigeartaigh

Authors1. Asaf Tzachor[View author publications](/search?author=Asaf%20Tzachor)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Asaf%20Tzachor) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Asaf%20Tzachor%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
2. Jess Whittlestone[View author publications](/search?author=Jess%20Whittlestone)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Jess%20Whittlestone) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Jess%20Whittlestone%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
3. Lalitha Sundaram[View author publications](/search?author=Lalitha%20Sundaram)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Lalitha%20Sundaram) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Lalitha%20Sundaram%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
4. Seán Ó hÉigeartaigh[View author publications](/search?author=Se%C3%A1n%20%C3%93%20h%C3%89igeartaigh)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Se%C3%A1n%20%C3%93%20h%C3%89igeartaigh) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Se%C3%A1n%20%C3%93%20h%C3%89igeartaigh%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
### Corresponding author

Correspondence to
 [Asaf Tzachor](mailto:at875@cam.ac.uk).

Ethics declarations
-------------------

### Competing interests

The authors declare no competing interests.

Rights and permissions
----------------------

[Reprints and Permissions](https://s100.copyright.com/AppDispatchServlet?title=Artificial%20intelligence%20in%20a%20crisis%20needs%20ethics%20with%20urgency&author=Asaf%20Tzachor%20et%20al&contentID=10.1038%2Fs42256-020-0195-0&copyright=Springer%20Nature%20Limited&publication=2522-5839&publicationDate=2020-06-22&publisherName=SpringerNature&orderBeanReset=true)

This article is cited by
------------------------

* ###
[Dismantling AI capitalism: the commons as an alternative to the power concentration of Big Tech](https://doi.org/10.1007/s00146-022-01437-8)

	+ Pieter Verdegem*AI & SOCIETY* (2022)
* ###
[Foundations for the future: institution building for the purpose of artificial intelligence governance](https://doi.org/10.1007/s43681-021-00093-w)

	+ Charlotte Stix*AI and Ethics* (2022)
* ###
[The ethical use of high-performance computing and artificial intelligence: fighting COVID-19 at Barcelona Supercomputing Center](https://doi.org/10.1007/s43681-021-00056-1)

	+ Ulises Cortés
	+ Atia Cortés
	+ Enric Àlvarez*AI and Ethics* (2022)
* ###
[Synthetic data in machine learning for medicine and healthcare](https://doi.org/10.1038/s41551-021-00751-8)

	+ Richard J. Chen
	+ Ming Y. Lu
	+ Faisal Mahmood*Nature Biomedical Engineering* (2021)

Artificial intelligence in a crisis needs ethics with urgency

Asaf Tzachor
Jess Whittlestone
Lalitha Sundaram
Seán Ó hÉigeartaigh