How Would Catastrophic Risks Affect Prospects for Compromise?
=============================================================

29 August 2015
by [Brian Tomasik](https://longtermrisk.org/author/brian-tomasik/ "Posts by Brian Tomasik")

First written: 24 Feb. 2013; major updates: 13 Nov. 2013; last update: 4 Dec. 2017

 Summary
-------

Catastrophic risks -- such as engineered pathogens, nanotech weapons, nuclear war, or financial collapse -- would cause major damage in the short run, but their effects on the long-run direction that humanity takes are also significant. In particular, to the extent these disasters increase risks of war, they may contribute to faster races between nations to build artificial general intelligence (AGI), less opportunity for compromise, and hence less of what everyone wants in expectation, including less suffering reduction. In this way, even pure negative utilitarians may oppose catastrophic risks, though this question is quite unsettled. While far from ideal, today's political environment is more democratic and peaceful than what we've seen historically and what could have been the case, and disrupting this trajectory might have more downside than upside. I discuss further considerations about how catastrophes could have negative and positive consequences. Even if averting catastrophic risks is net good to do, I see it as less useful than directly promoting compromise scenarios for AGI and setting the stage for such compromise via cooperative political, social, and cultural institutions.

*Note, 20 Jul. 2015*: Relative to when I first wrote this piece, I'm now less hopeful that catastrophic-risk reduction is plausibly good for pure negative utilitarians. The main reason is that some catastrophic risks, such as from malicious biotech, do seem to pose nontrivial risk of causing complete extinction relative to their probability of merely causing mayhem and conflict. So I now don't support efforts to reduce non-AGI "existential risks". (Reducing AGI extinction risks is a very different matter, since most AGIs would colonize space and spread suffering into the galaxy, just like most human-controlled future civilizations would.) Regardless, negative utilitarians should just focus their sights on more clearly beneficial suffering-reduction projects, like promoting suffering-focused ethical viewpoints and researching more how best to reduce wild-animal and far-future suffering.

Contents

* [Introduction](#Introduction)
* [Most catastrophic risks would not cause extinction](#Most_catastrophic_risks_would_not_cause_extinction)
* [Degree of compromise as a key metric](#Degree_of_compromise_as_a_key_metric)
* [How contingent is the future?](#How_contingent_is_the_future)
* [War as a key risk](#War_as_a_key_risk)
* [Dislocation makes conflict more likely](#Dislocation_makes_conflict_more_likely)
* [If it ain't broke, don't fix it](#If_it_aint_broke_dont_fix_it)
* [How robust is technological civilization?](#How_robust_is_technological_civilization)
* [Might humans be replaced by other species?](#Might_humans_be_replaced_by_other_species)
* [Other costs to catastrophes](#Other_costs_to_catastrophes)
	+ [Greater desperation](#Greater_desperation)
	+ [Darwinian futures?](#Darwinian_futures)
* [Silver linings to catastrophes](#Silver_linings_to_catastrophes)
	+ [Greater concern for suffering?](#Greater_concern_for_suffering)
	+ [More time for reflection?](#More_time_for_reflection)
	+ [Resource curse?](#Resource_curse)
	+ [Greater impetus for cooperation?](#Greater_impetus_for_cooperation)
	+ [Minority views in defense of alternate political systems](#Minority_views_in_defense_of_alternate_political_systems)
* [What if the conclusions flipped?](#What_if_the_conclusions_flipped)
* [Is work on catastrophic risks optimal?](#Is_work_on_catastrophic_risks_optimal)
* [Recovery measures are not supported by this argument](#Recovery_measures_are_not_supported_by_this_argument)
* [Appendix: Inoculation in general](#Appendix_Inoculation_in_general)
	+ [Inoculation](#Inoculation)
	+ [Slippery slopes](#Slippery_slopes)
* [Footnotes](#Footnotes)

Introduction
------------

Some in the effective-altruist community consider [global catastrophic risks](http://gcrinstitute.org/) to be a pressing issue. Catastrophic risks include possibilities of world financial collapse, major pandemics, bioweapons, nanoweapons, environmental catastrophes like runaway global warming, and nuclear war.

Typically discussions of these risks center on massive harm to humanity in the short term and/or remote risks that they would lead to human extinction, affecting the long term. In this piece, I'll explore another consideration that might trump both short-term harm and extinction considerations: the flow-through effects of catastrophic risks on the degree of compromise in future politics and safety of future technology.

Most catastrophic risks would not cause extinction
--------------------------------------------------

I think the only known technological development that is highly likely to cause all-out human extinction is AGI.[1](#link_ajs-fn-id_1-267) Carl Shulman [has defended](http://lesswrong.com/lw/bnc/against_ai_risk/6axy) this view, although he notes that while he doesn't consider nanotech a big extinction risk, "Others disagree (Michael Vassar has worked with the [Center for Responsible Nanotechnology], and Eliezer [Yudkowsky] often names molecular nanotechnology as the [extinction ]risk he would move to focus on if he knew that AI was impossible)." Reinforcing this assessment was the "[Global Catastrophic Risks Survey](https://web.archive.org/web/20161020051016/http://www.philosophy.ox.ac.uk/__data/assets/pdf_file/0020/3854/global-catastrophic-risks-report.pdf)" of 2008, in which the cumulative risk of extinction was estimated as at most 19% (median), the highest two subcomponents being AI risk and nanotech risk at median 5% each. Nuclear war was median 1%, consistent with [general expert sentiment](http://www.overcomingbias.com/2012/11/nuclear-winter-and-human-extinction-qa-with-luke-oman.html).

Of course, there's model uncertainty at play. Many ecologists, for instance, feel the risk of human extinction due to environmental issues is far higher than what those in the techno-libertarian circles cited in the previous paragraph believe. Others fear peak oil or impending economic doom. Still others may hold religious or philosophical views that incline them to find extinction likely via alternate means. In any event, whether catastrophic risks are likely to cause extinction is not relevant to the remainder of this piece, which will merely examine what implications -- both negative and positive -- catastrophic risks might have for the [trajectory](http://lesswrong.com/lw/hjb/a_proposed_adjustment_to_the_astronomical_waste/) of social evolution conditional on human survival.

Degree of compromise as a key metric
------------------------------------

Ignoring extinction considerations, how else are catastrophic risks likely to matter for the future? Of course, they would obviously cause massive human damage in the short run. But they would also have implications for the long-term future of humanity to the extent that they affected the ways in which society developed: How much international cooperation is there? How humane are people's moral views? How competitive is the race to develop technology the fastest?

![](https://longtermrisk.org/files/Ford_signing_accord_with_Brehznev_November_24_1974-350x238.jpg "'President Gerald Ford and Soviet General Secretary Leonid Brezhnev sign a Joint Communiqué following talks on the limitation of strategic offensive arms. The document was signed in the conference hall of the Okeansky Sanitarium, Vladivostok, USSR.' David Hume Kennerly [Public domain], via Wikimedia Commons: https://commons.wikimedia.org/wiki/File:Ford_signing_accord_with_Brehznev,_November_24,_1974.jpg")

How contingent is the future?
-----------------------------

Much of the trajectory of the future may be inexorable. Especially as people become smarter, we might expect that if compromise is a Pareto-improving outcome, our descendants should converge on it. Likewise, even if catastrophe sets humanity back to a much more primitive state, it may be that a relatively humane, non-violent culture will emerge once more as civilization matures. [For example](https://web.archive.org/web/20170829110145/http://levine.sscnet.ucla.edu/general/aandrreview.pdf "'A Review of Acemoglu and Robinson’s Why Nations Fail'"):

> Acemoglu and Robinson’s [Why Nations Fail](https://en.wikipedia.org/wiki/Why_Nations_Fail "'Why Nations Fail'") [2012] is a grand history in the style of Diamond [1997] or McNeil [1963]. [...] Acemoglu and Robinson theorize that political institutions can be divided into two kinds - “extractive” institutions in which a “small” group of individuals do their best to exploit - in the sense of Marx - the rest of the population, and “inclusive” institutions in which “many” people are included in the process of governing hence the exploitation process is either attenuated or absent.
>
>
> [...] inclusive institutions enable innovative energies to emerge and lead to continuing growth as exemplified by the Industrial Revolution. Extractive institutions can also deliver growth but only when the economy is distant from the technological frontier.

If this theory is right, it would suggest that more inclusive societies will in general tend to control humanity's long-run future.

Still, humanity's trajectory is not completely inevitable, and it can be sensitive to initial conditions.

* A society that has better institutions for cooperation may be able to achieve compromise solutions that would simply be unavailable to one in which those institutions did not exist, no matter how smart the people involved were.
* It may be that social values matter -- e.g., if people care more intrinsically about compromise, this can change the outcomes that result from pure game-theoretic calculations because the payoffs are different.
* Historical, cultural, and political conditions can influence [Schelling points](https://en.wikipedia.org/wiki/Focal_point_(game_theory)). Sometimes there are multiple possible Nash equilibria, and [expectations determine which one is reached](http://oyc.yale.edu/economics/econ-159/lecture-5 "'Lecture 5 - Nash Equilibrium: Bad Fashion and Bank Runs', 'ECON 159: GAME THEORY'").

There's a long literature on the extent to which history is inevitable or contingent, but it seems that it's at least a little bit of both. Even in cases of modern states engaged in strategic calculations, there have been a great number of contingent factors. For example:

* As an analyst for US military policy, Thomas Schelling suggested strategies and safeguards that would not have been thought of without him. Roger Myerson [said](http://youtu.be/oG-dErfUunw?t=7m48s): "You know, I think there's some chance that Tom Schelling may have saved the world." At least as dramatic is [Stanislav Petrov](http://lesswrong.com/lw/jq/) helping to avert accidental nuclear war.
* *[Confronting the Bomb: A Short History of the World Nuclear Disarmament Movement](http://www.amazon.com/dp/0804756325/)* argues the case that anti-nuclear activism made some key differences in how governments conducted nuclear policy, including Ronald Reagan's flip in stance from very hawkish to doveish, [in part due](http://www.alternet.org/story/149821/how_reagan_brought_the_world_to_the_brink_of_nuclear_destruction?page=0%2C1) to the largest political demonstration in American history on 12 June 1982 against increasing nuclear arsenals. This is one example to demonstrate that political action can make a non-inevitable difference to even highly strategic matters of world dominance by major powers.
* In general, US presidents have [sometimes](http://www.theatlantic.com/magazine/archive/2013/06/do-presidents-matter/309307/) had significant contingent effects on the nation's direction.
* There are countless other historical examples that could be cited.

Even if we think that greater intelligence by future people will mean less contingency in how events play out, we clearly won't eliminate contingency any time soon, and the decisions we make in the coming decades may matter a lot to the final outcome.

In general, if you think there's only an X% chance that the success or failure of compromise to avoid an AGI arms race is contingent on what we do, you can multiply the expected costs/benefits of our actions by X%. But probably X should not be too small. I would put it definitely above 33%, and a more realistic estimate should be higher.

War as a key risk
-----------------

What factors are most likely to lead to an AGI arms race in which groups compete to build whatever crude AGI works rather than cautiously constructing an AGI that better encapsulates many value systems, including that of suffering reduction? If AGI is built by corporations, then fierce market competition could be risky. That said, it seems most plausible to me that AGI would be built by, or at least under the control of, governments, because unless the AGI project took off really quickly, it seems the military would not allow a national (indeed, world) security threat to proceed without restraint.

In this case, the natural scenario that could lead to a reckless race for AGI would be international competition -- say, between the US and China. AGI would be in many ways like nuclear weapons, because whoever builds it first can literally take over the world (though Carl Shulman points out [some differences](http://intelligence.org/files/ArmsControl.pdf) between AGI and nuclear weapons as well).

If we think historically about what has led to [nuclear development](http://www.icanw.org/the-facts/the-nuclear-age/), it has always been international conflict, usually precipitated by wars:

* World War II
	+ The US [Manhattan Project](https://en.wikipedia.org/wiki/Manhattan_Project), which required over 130,000 people and cost ~$26 billion dollars as measured in 2013 currency.
	+ The uncompleted [Nazi nuclear project](http://www.pbs.org/wgbh/nova/military/nazis-and-the-bomb.html).
* Cold War
	+ Soviet Union, China, UK, France, etc.
* India-China conflict
	+ [Led to](https://en.wikipedia.org/wiki/India_and_weapons_of_mass_destruction#Nuclear_weapons) India's development of nuclear weapons.
* India-Pakistan conflict
	+ Harsh treaty conditions following the Indo-Pakistani War of 1971 [spurred](https://en.wikipedia.org/wiki/Pakistan_and_weapons_of_mass_destruction#Development_of_nuclear_weapons) Pakistan's nuclear-weapons program.
* Israeli conflicts with Middle Eastern neighbors
	+ Led to Israel's nuclear-weapons program.

In general, war tends to cause

* Fast technological development, often enhanced by major public investments
* Willingness to take risks in order to develop the technology first
* International hostility that makes cooperation difficult.

Thus, war seems to be a major risk factor for fast AGI development that results in less good and greater expected suffering than more careful, cooperative scenarios. To this extent, anything else that makes war more likely entails some expected harm through this pathway.

Dislocation makes conflict more likely
--------------------------------------

While catastrophic risks are unlikely to cause extinction, they are fairly likely to cause damage on a mass scale. For example, from the "[Global Catastrophic Risks Survey](https://web.archive.org/web/20161020051016/http://www.philosophy.ox.ac.uk/__data/assets/pdf_file/0020/3854/global-catastrophic-risks-report.pdf)":

|  |  |  |  |
| --- | --- | --- | --- |
| *Catastrophe* | *Median probability of >1 million dead* | *Median probability of >1 billion dead* | *Median probability of extinction* |
| nanotech weapons | 25% | 10% | 5% |
| all wars | 98% | 30% | 4% |
| biggest engineered pandemic | 30% | 10% | 2% |
| nuclear wars | 30% | 10% | 1% |
| natural pandemic | 60% | 5% | 0.05% |

...and so on. Depending on the risk, these disasters may or may not contribute appreciably to risks of AGI arms race, and it would be worth exploring in more detail which risks are most likely to lead to a breakdown of compromise. Still, in general, all of these risks seem likely to increase the chance of warfare, and by that route alone, they imply nonzero risks for increasing suffering in the far future.

If it ain't broke, don't fix it
-------------------------------

For all its shortcomings, contemporary society is remarkably humane by historical standards and relative to what other possibilities one can imagine. This trend is unmistakable to anyone who reads history books and witnesses how often societies in times past were controlled by violent takeover, fear, and oppression. Steven Pinker's *[The Better Angels of Our Nature](https://en.wikipedia.org/wiki/The_Better_Angels_of_Our_Nature)* is a quantitative defense of this thesis. Pinker cites "six major trends" toward greater peace and cooperation that have taken place in the past few millennia:

1. *The Pacification Process*: The beginnings of cities and governments
2. *The Civilizing Process*: Smaller territories uniting into larger kingdoms
3. *The Humanitarian Revolution*: Reductions in slavery, torture, corporal punishment, etc.
4. *The Long Peace*: After World War II, the great powers [have not directly fought each other](https://en.wikipedia.org/wiki/The_long_peace)
5. *The New Peace*: A short-term (and hence "more tenuous") decline in conflicts since the end of the Cold War
6. *The Rights Revolutions*: Greater moral consideration for women, minorities, homosexuals, animals, etc.

Catastrophic risks -- especially those like nuclear winter that could set civilization back to a much less developed state -- are essentially a "roll of the dice" on the initial conditions for how society develops, and while things could certainly be better, they could also be a lot worse.

To a large extent, it may be that the relatively peaceful conditions of the present day are a required condition for technological progress, such that any advanced civilization will necessarily have those attributes. Insofar as this is true, it reduces the expected cost of catastrophic risks. But there's some chance that these peaceful conditions are not inevitable. One could imagine, for instance, a technologically advanced dictatorship taking control instead, and insofar as its policies would be less determined by those of its population, the degree of compromise with many value systems would be reduced in expectation. Consider ancient Sparta, monarchies, totalitarian states, the mafia, gangs, and many other effective forms of government where ruthlessness by leaders is more prevalent than compassion.

Imagine what would have happened if [Hitler's atomic-bomb project](https://en.wikipedia.org/wiki/German_nuclear_weapon_project) had succeeded before the Manhattan Project. Or [if the US South had won](https://en.wikipedia.org/wiki/American_Civil_War_alternate_histories) the American Civil War. Or various other scenarios. Plausibly the modern world would have turned out somewhat similar to the present (for instance, I doubt slavery would have lasted forever even if the US South had won the Civil War), but conditions probably would have been somewhat worse than they are now. (Of course, various events in history also could have turned out better than they actually did.)

![AGI suffering vs. social organization](https://longtermrisk.org/wp-content/uploads/2015/09/agi-suffering-vs-social-organization.png)

A Cold War scenario seems likely to accelerate AGI relative to its current pace. Compare with the [explosion of STEM education](http://www.nytimes.com/2007/09/25/science/space/25educ.html?pagewanted=all&_r=0 "\"When Science Suddenly Mattered, in Space and in Class\"") as a result of the Space Race.

How robust is technological civilization?
-----------------------------------------

Civilization in general seems to me fairly robust. The world witnessed civilizations emerge independently all over the globe -- from Egypt to the Fertile Crescent to China to the Americas. The Mayan civilization was completely isolated from happenings in Africa and Asia and yet shared many of the same achievements. It's true that civilizations often collapse, but they just as often rebuild themselves. The history of ancient Egypt, China, and many other regions of the world is a history of an empire followed by its collapse followed by the emergence of another empire.

It's less clear whether *industrial* civilization is robust. One reason is that we haven't seen completely independent industrial revolutions in history, since trade was well developed by the time industrialization could take place. Still, for example, China and Europe were both on the verge of industrial revolutions in the early 1800s[2](#link_ajs-fn-id_2-267), and the two civilizations were pretty independent, despite [some trade](http://www.historyorb.com/asia/china_trade.php). China and Europe invented the printing press independently.[3](#link_ajs-fn-id_3-267) And so on.

Given the written knowledge we've accumulated, it's not plausible that post-disaster peoples would not relearn how to build industry. But it's not clear whether they would have the resources and/or social organization requisite to do so. Consider how long it takes for developing nations to industrialize even with relative global stability and trade. On the other hand, military conflicts if nothing else would probably force post-disaster human societies to improve their technological capacities at some point. Technology seems more inevitable than democracy, because technology is compelled by conflict dynamics. Present-day China is an example of a successful, technologically advanced non-democracy. (Of course, China certainly exhibits some degree of deference to popular pressure, and conversely, Western "democracies" also give excessive influence to wealthy elites.)

Some suggest that rebuilding industrial civilization might be impossible the second time around because abundant surface minerals and easy-to-drill fossil fuels would have been used up. Others contend that human ingenuity would find alternate ways to get civilization off the ground, especially given the plenitude of scientific records that would remain. I incline toward the latter of these positions, but I maintain modesty on this question. It reflects a [more general divide](https://en.wikipedia.org/wiki/Simon%E2%80%93Ehrlich_wager) between scarcity doomsayers vs. techno-optimists. (The "optimist" in "techno-optimist" is relative to the goal of human economic growth, not necessarily reducing suffering.)

Robin Hanson [takes](http://hanson.gmu.edu/collapse.pdf "\"Catastrophe, Social Collapse, and Human Extinction\", 2007") the techno-optimist view:

> Once [post-collapse humans] could communicate to share innovations and grow at the rate that our farming ancestors grew, humanity should return to our population and productivity level within twenty thousand years. (The fact that we have used up some natural resources this time around would probably matter little, as growth rates do not seem to depend much on natural resource availability.)
>
>

But even if historical growth rates didn't depend much on resources, might there be some minimum resource threshold below which resources do become essential? Indeed, in the limit of *zero* resources, growth is not possible.

A [blog post](http://reflectivedisequilibrium.blogspot.com/2013/12/current-thoughts-on-nuclear-war-as.html) by Carl Shulman includes a section titled "Could a vastly reduced population eventually recover from nuclear war?". It reviews reasons why rebuilding civilization would be harder and reasons it would be easier the second time around. Shulman concludes: "I would currently guess that the risk of permanent drastic curtailment of human potential from failure to recover, conditional on nuclear war causing the deaths of the overwhelming majority of humanity, is on the lower end." Shulman also seems to agree with the (tentative and uncertain) main thrust of my current article: "Trajectory change" effects of civilizational setback, possibly including diminution of liberal values, "could have a comparable or greater role in long-run impacts" of nuclear war (and other catastrophic risks) than outright extinction.

Stuart Armstrong [also agrees](https://www.youtube.com/watch?v=i4LjoJGpqIY&t=11m29s "\"Stuart Armstrong: The future is going to be wonderful if we don't get whacked\"") that rebuilding following nuclear war seems likely. He points out that formation of governments is common in history, and social chaos is rare. There would be many smart, technically competent survivors of a nuclear disaster, e.g., in submarines.

Might humans be replaced by other species?
------------------------------------------

As noted above, full-out human extinction from catastrophic risks seems relatively unlikely compared with just social destabilization. If human extinction did occur from causes other than AI, presumably parts of the biosphere would still remain. In many scenarios, at least some other animals would survive. What's the probability that those animals would then replace humans and colonize space? My guess is it's small but maybe not negligibly so. Robin Hanson [seems to agree](http://hanson.gmu.edu/collapse.pdf "\"Catastrophe, Social Collapse, and Human Extinction\", 2007"): "it is also possible that without humans within a few million years some other mammal species on Earth would evolve to produce" a technological civilization.

In the [history of life on Earth](http://www.pbs.org/wgbh/nova/origins/life-nf.html), boney fish and insects emerged around 400 million years ago (mya). Dinosaurs emerged around 250 mya. Mammals blossomed less than 100 mya. [Earth's future](http://en.wikipedia.org/wiki/Future_of_the_Earth) allows for about [1000 million years](https://web.archive.org/web/20160330230725/http://news.sky.com/story/1110337/life-on-earth-to-die-out-in-one-billion-years) of life to come. So even if, as the cliche goes, the most complex life remaining after nuclear winter was cockroaches, there would still be 1000 million years in which human-like intelligence might re-evolve, and it took just 400 million years the first time around starting from insect-level intelligence. Of course, it's unclear how improbable the development of human-like intelligence was. For instance, if the dinosaurs hadn't been killed by an asteroid, plausibly they would still rule the Earth, without any advanced civilization.[4](#link_ajs-fn-id_4-267) The Fermi paradox also has something to say about how likely we should assess the evolution of advanced intelligence from ordinary animal life to be.

Some extinction scenarios would involve killing all humans but leaving higher animals. Perhaps a bio-engineered pathogen or nanotech weapon could do this. In that case, re-emergence of intelligence would be even more likely. For example, cetaceans [made large strides](http://archive.seti.org/news/features/intelligence-gathering.php) in intelligence 35 mya, jumping from an encephalization quotient (EQ) of 0.5 to 2.1. Some went on to develop EQs of 4-5, which is close to the human EQ of 7. As quoted in "[Intelligence Gathering: The Study of How the Brain Evolves Offers Insight Into the Mind](http://archive.seti.org/news/features/intelligence-gathering.php)," Lori Marino explains:

> Cetaceans and primates are not closely related at all, but both have similar behavior capacities and large brains -- the largest on the planet. Cognitive convergence seems to be the bottom line.
>
>

One hypothesis for why humans have such large brains despite metabolic cost is that big brains resulted from an [arms race](https://web.archive.org/web/20141201202010/http://www.jstor.org/discover/10.2307/4602561?uid=2&uid=4&sid=21105339709343 "'The Mental Arms Race Amplifier' by Michael R. Rose") of social competition. Similar conditions could obtain for cetaceans or other social mammals. Of course, many of the [other features](http://www.as.utexas.edu/astronomy/education/fall08/scalo/secure/309l_nov20_evocomplexintel.pdf) of primates that may have given rise to civilization are not present in most other mammals. In particular, it seems hard to imagine developing written records underwater.

If another species took over and built a space-faring civilization, would it be better or worse than our own? There's some chance it could be more compassionate, such as if bonobos took our place. But it might also be much less compassionate, such as if chimpanzees had won the evolutionary race, not to mention killer whales. On balance it's plausible our hypothetical replacements would be less compassionate, because compassion is something humans value a lot, while a random other species probably values something else more. The reason I'm asking this question in the first place is because humans are outliers in their degree of compassion. Still, in social animals, various norms of fair play are likely to emerge regardless of how intrinsically caring the species is. Simon Knutsson pointed out to me that if human survivors do recover from a near-extinction-level catastrophe, or if humans go extinct and another species with potential to colonize space evolves, they'll likely need to be able to cooperate rather than fighting endlessly if they are to succeed in colonizing space. This suggests that if they colonize space, they will be more moral or peaceful than we were. My reply is that while this is possible, a rebuilding civilization or new species might curb infighting via authoritarian power structures or strong ingroup loyalty that doesn't extend to outgroups, which might imply less compassion than present-day humans have.

My naive guess is that it's relatively unlikely another species would colonize space if humans went extinct -- maybe a ~10% chance? I suspect that most of the [Great Filter](https://en.wikipedia.org/wiki/Great_Filter "'Great Filter'") is behind us, and some of those filter steps would have to be crossed again for a new non-human civilization to emerge. As long as that new civilization wouldn't be more than several times worse in expectation than our current civilization, then this scenario is unlikely to dominate our calculations.

Other costs to catastrophes
---------------------------

### Greater desperation

In general, people in more hardscrabble or fearful conditions have less energy and emotional resources to concern themselves with the suffering of others, especially with powerless computations that might be run by a future spacefaring civilization. Fewer catastrophes means more people who can focus on averting suffering by other sentients.

### Darwinian futures?

Current long-term political trends suggest that a world government may develop at some point, as is hinted by the increasing degrees of unity among rich countries (European Union, international trade agreements, etc.). A world government would offer greater possibilities for enforcing [mutually beneficial cooperation](http://utilitarian-essays.com/compromise.html) and thereby fulfilling more of what all value systems want in expectation, relative to unleashing a [Darwinian future](http://www.utilitarian-essays.com/future-of-darwinism.html).

Silver linings to catastrophes
------------------------------

In this section I suggest some possible upsides of catastrophic risks. I think it's important not to shy from these ideas merely because they don't comport with our intuitive reactions. Arguments should not be [soldiers](http://wiki.lesswrong.com/wiki/Arguments_as_soldiers). At the same time, it's also essential to constrain our speculations in this area by [common sense](http://lesswrong.com/lw/iao/common_sense_as_a_prior/).

Also note that even in the unlikely event that we concluded catastrophic risks were net positive for the far future, we should still not support them, to avoid stepping on the toes of so many other people who care deeply about preventing short-term harm. Rather, in this hypothetical scenario, we should find other, win-win ways to improve the future that don't encroach on what so many other people value.

### Greater concern for suffering?

Is it possible that some amount of disruption in the near term could heighten concern about potential future sources of suffering, whereas if things go along smoothly, people will give less thought to futures full of suffering? This question lies in analogy with the concern that reducing hardship and depression might make people less attuned to the pain of others. Many of the people I know who care most about reducing suffering [have gone through](http://reducing-suffering.org/how-important-is-experiencing-suffering-for-caring-about-suffering/) severe personal trauma or depression at one point. When things are going well, you can [forget](https://en.wikipedia.org/wiki/Empathy_gap) how [horrifying](http://www.utilitarian-essays.com/horror-of-suffering.html) suffering can be.

It's often said that World War I transformed art and cultural attitudes more generally. [Johnson (2012)](http://articles.latimes.com/2012/jul/21/entertainment/la-et-cm-world-war-art-20120722 "'Art forever changed by World War I - latimes'"): "During and after World War I, flowery Victorian language was blown apart and replaced by more sinewy and R-rated prose styles. [...] 'World War I definitely gives a push forward to the idea of dystopia rather than utopia, to the idea that the world is going to get worse rather than better,' Braudy said."

### More time for reflection?

Severe catastrophes might depress economic output and technological development, with the possibility of allowing more time for reflection on the risks that such technology would bring. That said, this cuts both ways: Faster technology also allows for faster wisdom, better ability to monitor tech developments, and greater prosperity that allows more people to even think about these questions, as well as reduced social animosity and greater positive-sum thinking. The net sign of all of this is very unclear.

### Resource curse?

There are suggestions (hotly debated) in the political-science literature of a "resource curse" in which greater reserves of oil and other natural resources may contribute to authoritarianism and repression. The [Wikipedia article](https://en.wikipedia.org/wiki/Resource_curse) cites a number of mechanisms by which the curse may operate. A related trend is the observation that cooler climates sometimes have a greater degree of compassion and cooperation -- perhaps because to survive cold winters you have to work together, while in warm climates, success is determined by being the best at forcibly stealing the resources that already exist?

To the extent these trends are valid, does this suggest that if humanity were to rebuild after a significant catastrophe, it might be more democratic owing to having less oil, metals, and other resources?

The "Criticisms" section of the Wikipedia article explains that some studies attribute causation in the other direction: Greater authoritarianism leads countries to exploit their resources faster. Indeed, some studies even find a "resource blessing."

### Greater impetus for cooperation?

Fighting factions are often brought together when they face a common enemy. For instance, in the 1954 [Robbers Cave study](https://web.archive.org/web/20160901100552/http://faculty.haas.berkeley.edu:80/kurkoski/BA105/BA%20105%20materials/READINGS/Design/sherif_robbers_cave_experiment.html), the two hostile factions of campers were brought together by "superordinate goals" that required them to unite to solve a problem they all faced. Catastrophic risks are a common enemy of humanity, so could efforts to prevent them build cooperative institutions? For instance, cooperation to solve climate change could be seen as an easy test bed for the much harder challenges that will confront humanity in cooperating on AGI. Could greater climate danger provide greater impetus for building better cooperative institutions early on? Wolf Bullmann [compared](https://www.facebook.com/groups/effective.altruists/permalink/585286124861082/?comment_id=587499161306445&offset=0&total_comments=31) this to vaccination.

Of course, this is not an argument against building international-cooperation efforts against climate change -- those are the very things we want to happen. But it would be a slight counter-consideration against, say, personally trying to reduce greenhouse-gas emissions. I hasten to explain that this "silver lining" point is *extremely speculative*, and on balance, it seems most plausible that personally reducing greenhouse-gas emissions is net good overall, in terms of reducing risks of wars that could degrade into bad outcomes. The "inoculation" idea (see the [Appendix](#inoculation)) should be explored further, though it needs to be cast in a way that's not amenable to being quoted out of context.

### Minority views in defense of alternate political systems

Reshuffling world political conditions *could* produce a better outcome, and indeed, there are minority [views](https://web.archive.org/web/20150610055901/http://www.moreright.net/about/ "'About | More Right'") that greater authoritarianism could actually improve future prospects by reducing coordination problems and enforcing safeguards. We should continue to explore a broad range of viewpoints on these topics, while at the same time not wandering too easily from mainstream consensus.

What if the conclusions flipped?
--------------------------------

Like any empirical question, the net impact of catastrophic risks on the degree of compromise in the future isn't certain, and a hypothetical scenario in which we concluded that catastrophic risks would actually improve compromise is not impossible. Not being concerned about catastrophic risks is more likely for pure negative utilitarians who also fear the [risks of astronomical suffering](http://www.utilitarian-essays.com/astronomical-suffering.html) that space colonization would entail. I find it plausible that the detrimental effects of catastrophic risks on compromise outweigh effects on probability of colonization, but this conclusion is contingent and not inevitable. What if the calculation flipped around?

Even if so, we should still probably oppose catastrophic risks when it's very cheap to do so, and we should never support them. Why? Because many other people care a lot about preventing short-term disasters, and stepping on so many toes so dramatically would not be an [efficient](https://en.wikipedia.org/wiki/Kaldor%E2%80%93Hicks_efficiency) course of action, much less a wise move by any reasonable heuristics about how to get along in society. Rather, we should find other, win-win approaches to improving compromise prospects that everyone can get behind. In any event, even ignoring the importance of cooperating with other people, it seems unlikely that focusing on catastrophic risks would be the best leverage point for accomplishing one's goals.

Is work on catastrophic risks optimal?
--------------------------------------

My guess is that there are better projects for altruists to pursue, because

* Working *directly* on improving cooperation scenarios for AGI seems to target the problem more head-on, and at the same time, this field has *less* funding because it's more fringe and has fewer immediately visible repercussions and less historical precedent that tend to motivate mainstream philanthropists, and
* Working *directly* on improving worldwide cooperation, global governance, etc. in general also seem more promising insofar as these efforts are, in my mind, more clearly positive with fewer question marks and shorter causal chains to the ultimate goal. If catastrophic risks were clearly being neglected relative to general compromise work, then the calculation might change, but as things stand, I would prefer to push on compromise directly.

Recovery measures are not supported by this argument
----------------------------------------------------

The argument in this essay applies only to preventing risks before they happen, so as to reduce societal dislocation. It doesn't endorse measures to ensure human recovery *after* catastrophic risks have already happened, such as [disaster shelters](http://www.effective-altruism.com/ea/5r/improving_disaster_shelters_to_increase_the/ "'Improving disaster shelters to increase the chances of recovery from a global catastrophe'") or [space colonies](https://en.wikipedia.org/wiki/Alliance_to_Rescue_Civilization). These post-disaster measures don't avert the increased anarchy and confusion that would result from catastrophes but do help humans stick around to potentially cause [cosmic harm](http://www.utilitarian-essays.com/astronomical-suffering.html) down the road. Moreover, disaster-recovery solutions [might even](https://en.wikipedia.org/wiki/Feeding_Everyone_No_Matter_What#Criticisms "'Feeding Everyone No Matter What': 'Criticisms'") increase the chance that catastrophic risks occur because of moral hazard. I probably don't endorse post-disaster recovery efforts except maybe in rare cases when they also substantially help to maintain social stability in scenarios that cause less-than-extinction-level damage.

Appendix: Inoculation in general
--------------------------------

### Inoculation

The idea of inoculation -- accepting some short-term harm in order to improve long-term outcomes -- is a general concept.

Even with warfare, there's some argument about an inoculation effect. For example, the United Nations was formed after World War II in an effort to prevent similar conflicts from happening again. And there's a widespread debate about inoculation in activism. Sometimes the "radicals" fear that if the "moderates" compromise too soon, the partial concessions will quell discontent and prevent a more revolutionary change. For example, some animal advocates say that if we improve the welfare of farm animals, people will have less incentive to completely "end animal exploitation" by going vegan. In this case, the radicals claim that greater short-term suffering is the inoculation necessary to prevent long-term "exploitation."

### Slippery slopes

The flip side to inoculation is the slippery slope: A little bit of something in the short term tends to imply *more* of it in the long term. In general, I think slippery-slope arguments are stronger than inoculation arguments, with some exceptions like in organisms' immune systems.

Usually wars cause more wars. World War II would not have happened absent World War I. Conflicts breed animosity and perpetuate a cycle of violence as tit-for-tat retributions continue indefinitely, with each side claiming the other side was the first aggressor. We see this in terrorism vs. counter-terrorism response and in many other domains.

Likewise, a few animal-welfare reforms now can enhance a culture of caring about animals that eventually leads to greater empathy for them. The Humane Society of the United States (HSUS) is often condemned by more purist animal-rights advocates as being in the hands of Big Ag, but in fact, Big Ag has [a whole website](http://humanewatch.org/) devoted to trying to discredit HSUS. This is hardly behavior that one would expect if HSUS is actually helping ensure Big Ag's long-term future. [↩](#inoculation-back)

Footnotes
---------

1. Note that even if AGI causes human extinction, it would likely still undertake space colonization to advance whatever value it wanted to maximize.  [(back)](#back_ajs-fn-id_1-267)
2. See *[When China Rules the World](https://en.wikipedia.org/wiki/When_China_Rules_the_World)*, Ch. 2.  [(back)](#back_ajs-fn-id_2-267)
3. "[Gutenberg and the history of the printing press](http://didyouknow.org/gutenberg/)": "Gutenberg was unaware of the Chinese and Korean printing methods."  [(back)](#back_ajs-fn-id_3-267)
4. John Maxwell [disputes](http://effective-altruism.com/ea/14y/saving_expected_lives_at_10_apiece/9k7 "'John_Maxwell_IV comments on Saving expected lives at $10 apiece? - Effective Altruism Forum'") this claim. My reasoning is that dinosaurs lasted for at least [135 million years](https://en.wikipedia.org/wiki/Mesozoic "'Mesozoic'") but only went extinct [66 mya](https://en.wikipedia.org/wiki/Dinosaur "'Dinosaur': "). It's easy to imagine that dinosaurs might have lasted, say, twice as long as they did, in which case they would still rule the Earth today.  [(back)](#back_ajs-fn-id_4-267)

Note that even if AGI causes human extinction, it would likely still undertake space colonization to advance whatever value it wanted to maximize.See *[When China Rules the World](https://en.wikipedia.org/wiki/When_China_Rules_the_World)*, Ch. 2."[Gutenberg and the history of the printing press](http://didyouknow.org/gutenberg/)": "Gutenberg was unaware of the Chinese and Korean printing methods."John Maxwell [disputes](http://effective-altruism.com/ea/14y/saving_expected_lives_at_10_apiece/9k7 "'John_Maxwell_IV comments on Saving expected lives at $10 apiece? - Effective Altruism Forum'") this claim. My reasoning is that dinosaurs lasted for at least [135 million years](https://en.wikipedia.org/wiki/Mesozoic "'Mesozoic'") but only went extinct [66 mya](https://en.wikipedia.org/wiki/Dinosaur "'Dinosaur': "). It's easy to imagine that dinosaurs might have lasted, say, twice as long as they did, in which case they would still rule the Earth today.


International Cooperation vs. AI Arms Race
==========================================

8 April 2015
by [Brian Tomasik](https://longtermrisk.org/author/brian-tomasik/ "Posts by Brian Tomasik")

First written: 5 Dec. 2013; last update: 29 Feb. 2016

 Summary
-------

There's a decent chance that governments will be the first to build artificial general intelligence (AI). International hostility, especially an [AI arms race](http://wiki.lesswrong.com/wiki/AI_arms_race), could exacerbate risk-taking, hostile motivations, and errors of judgment when creating AI. If so, then international cooperation could be an important factor to consider when evaluating the [flow-through effects](http://blog.givewell.org/2013/05/15/flow-through-effects/) of charities. That said, we may not want to popularize the arms-race consideration too openly lest we accelerate the race.

### Other versions

[![](/files/pdf-icon.png)](https://longtermrisk.org/files/international-cooperation-ai-arms-race.pdf)

Contents

+ [Other versions](#Other_versions)

* [Will governments build AI first?](#Will_governments_build_AI_first)
* [AI arms races](#AI_arms_races)
* [Ways to avoid an arms race](#Ways_to_avoid_an_arms_race)
* [Are these efforts cost-effective?](#Are_these_efforts_cost-effective)
* [Should we publicize AI arms races?](#Should_we_publicize_AI_arms_races)
* [How do our prospects look?](#How_do_our_prospects_look)
* [Robot arms races](#Robot_arms_races)
* [Nanotech arms races](#Nanotech_arms_races)
* [Feedback](#Feedback)

Will governments build AI first?
--------------------------------

AI poses a national-security threat, and unless the militaries of powerful countries are very naive, it seems to me unlikely they'd allow AI research to proceed in private indefinitely. At some point the US military would confiscate the project from Google or Facebook, if the US military isn't already ahead of them in secret by that point.

While the US government as a whole is fairly slow and incompetent when it comes to computer technology, specific branches of the government are on the cutting edge, including the NSA and DARPA (which already funds a lot of public AI research). When we consider historical examples as well, like the Manhattan Project, the Space Race, and ARPANET, it seems that the US government has a strong track record of making technical breakthroughs when it really tries.

Sam Altman [agrees](http://blog.samaltman.com/machine-intelligence-part-2 "'Machine intelligence, part 2: THE NEED FOR REGULATION'") that in the long run governments will probably dominate AI development: "when governments gets serious about [superhuman machine intelligence] SMI they are likely to out-resource any private company".

There are *some* scenarios in which private AI research wouldn't be nationalized:

* An unexpected AI foom before anyone realizes what was coming.
* The private developers stay underground for long enough not to be caught. This becomes less likely the more government surveillance improves (see "[Arms Control and Intelligence Explosions](http://intelligence.org/files/ArmsControl.pdf)").
* AI developers move to a "safe haven" country where they can't be taken over. (It seems like the international community might prevent this, however, in the same way it now seeks to suppress terrorism in other countries.)

Each of these scenarios could happen, but it seems reasonably likely to me that governments would ultimately control AI development, or at least partner closely with Google.

AI arms races
-------------

Government AI development could go wrong in several ways. Plausibly governments would botch the process by not realizing the risks at hand. It's also possible that governments would use the AI and robots for totalitarian purposes.

It seems that both of these bad scenarios would be exacerbated by international conflict. Greater hostility means countries are more inclined to use AI as a weapon. Indeed, whoever builds the first AI can take over the world, which makes building AI the ultimate arms race. A [USA-China race](http://lesswrong.com/lw/hoz/do_earths_with_slower_economic_growth_have_a/9590) is one reasonable possibility.

Arms races encourage risk-taking -- being willing to skimp on safety measures to improve your odds of winning ("[Racing to the Precipice](http://intelligence.org/2013/11/27/new-paper-racing-to-the-precipice/)"). In addition, the weaponization of AI could lead to worse expected outcomes in general. [CEV](https://arbital.com/p/cev/ "'Coherent extrapolated volition (alignment target)'") seems to have less hope of success in a Cold War scenario. ("What? You want to include the evil *Chinese* in your CEV??") With a pure CEV, presumably it would eventually count Chinese values even if it started with just Americans, because people would become more enlightened during the process. However, when we imagine more crude democratic decision outcomes, this becomes less likely.

In *Superintelligence: Paths, Dangers, Strategies* (Ch. 14), Nick Bostrom proposes that another reason AI arms races would crimp AI safety is that competing teams wouldn't be able to share insights about AI control. What Bostrom doesn't mention is that competing teams also wouldn't share insights about AI *capability*. So even if less inter-team information sharing reduces safety, it also reduces speed, and the net effect isn't clear to me.

Of course, there are situations where arms-race dynamics can be desirable. In the original prisoner's dilemma, the *police* benefit if the prisoners defect. Defection on a tragedy of the commons by companies is the heart of [perfect competition](https://en.wikipedia.org/wiki/Perfect_competition)'s efficiency. It also underlies competition among countries to improve quality of life for citizens. Arms races generally speed up innovation, which can be good if the innovation being produced is both salutary and not risky. This is not the case for general AI. Nor is it the case for other "races to the bottom".

Ways to avoid an arms race
--------------------------

Averting an AI arms race seems to be an important topic for research. It could be partly informed by the Cold War and other nuclear arms races,![](https://longtermrisk.org/files/Reagan_and_Gorbachev_signing-350x233.jpg "'President Reagan and General Secretary Gorbachev signing the INF Treaty in the East Room of the White House.' By White House Photographic Office [Public domain], via Wikimedia Commons: https://commons.wikimedia.org/wiki/File:Reagan_and_Gorbachev_signing.jpg") as well as by [other efforts](http://cns.miis.edu/) at nonproliferation of chemical and biological weapons. Forthcoming robotic and [nanotech weapons](http://crnano.typepad.com/crnblog/2004/02/nanotech_weapon.html) might be even better analogues of AI arms races than nuclear weapons because these newer technologies can be built more secretly and used in a more targeted fashion.

Apart from more robust arms control, other factors might help:

* Improved international institutions like the UN, allowing for better enforcement against defection by one state.
* In the long run, a scenario of [global governance](https://en.wikipedia.org/wiki/Global_governance) would likely be ideal for strengthening international cooperation, just like nation states [reduce intra-state violence](https://en.wikipedia.org/wiki/The_Better_Angels_of_Our_Nature).
* Better construction and enforcement of nonproliferation treaties.
* Improved game theory and international-relations scholarship on the causes of arms races and how to avert them. (For instance, arms races have sometimes been modeled as iterated prisoner's dilemmas with imperfect information.)
* How to improve verification, which has historically been a weak point for nuclear arms control. (The concern is that if you haven't verified well enough, the other side might be arming while you're not.)
* Moral tolerance and multicultural perspective, aiming to reduce people's sense of nationalism. (In the limit where neither Americans nor Chinese care which government wins the race, there would be no point in having the race.)
* Improved trade, democracy, and other forces that historically have reduced the likelihood of war.

Are these efforts cost-effective?
---------------------------------

World peace is hardly a goal unique to effective altruists (EAs), so we shouldn't necessarily expect low-hanging fruit. On the other hand, projects like nuclear nonproliferation seem relatively underfunded even compared with anti-poverty charities.

I suspect more direct [MIRI](https://en.wikipedia.org/wiki/Machine_Intelligence_Research_Institute)-type research has higher expected value, but among EAs who don't want to fund MIRI specifically, encouraging donations toward international cooperation could be valuable, since it's certainly a more mainstream cause. I wonder if GiveWell would consider studying global cooperation specifically beyond its [indirect relationship](http://utilitarian-essays.com/catastrophic-risks-and-compromise.html) with catastrophic risks.

Should we publicize AI arms races?
----------------------------------

When I mentioned this topic to a friend, he pointed out that we might not want the idea of AI arms races too widely known, because then governments might take the concern more seriously and therefore start the race earlier -- giving us less time to prepare and less time to work on FAI in the meanwhile. From David Chalmers, "[The Singularity: A Philosophical Analysis](http://consc.net/papers/singularity.pdf)" (footnote 14):

> When I discussed these issues with cadets and staff at the West Point Military Academy, the question arose as to whether the US military or other branches of the government might attempt to prevent the creation of AI or AI+, due to the risks of an intelligence explosion. The consensus was that they would not, as such prevention would only increase the chances that AI or AI+ would first be created by a foreign power. One might even expect an AI arms race at some point, once the potential consequences of an intelligence explosion are registered. According to this reasoning, although AI+ would have risks from the standpoint of the US government, the risks of Chinese AI+ (say) would be far greater.
>
>

We should take this information-hazard concern seriously and remember the [unilateralist's curse](http://www.nickbostrom.com/papers/unilateralist.pdf). If it proves to be fatal for explicitly discussing AI arms races, we might instead encourage international cooperation without explaining *why*. Fortunately, it wouldn't be hard to encourage international cooperation on grounds other than AI arms races if we wanted to do so.

Also note that a government-level arms race could easily be *preferable* to a Wild West race among a dozen private AI developers where coordination and compromise would be not just difficult but potentially impossible. Of course, if we did decide it was best for governments to take AI arms races seriously, this would also encourage private developers to step on the gas pedal. That said, once governments do recognize the problem, they may be able to impose moratoria on private development.

How concerned should we be about accidentally accelerating arms races by talking about them? My gut feeling is it's not too risky, because

* It's hard to contain the basic idea. Super-powerful AI is already well known not just by governments but even in popular movies.
* Developing verification measures, technology restrictions, and so on require governments knowing what technology they're dealing with.
* If governments can think about these issues ahead of time (decades before strong AI becomes feasible), they're more likely to go for cooperation and less likely to panic and build up their own defenses, because they see that there's time for negotiations to potentially work before losing that much ground. Right now most AI research appears to be done in public, so there's not a huge cost for a given country in delaying at this point.
* Most risk analysts don't express concerns like these too much when talking about military arms races. Of course, there's selection bias; maybe most of the military does think it's dangerous to talk about these issues in public, and we only hear form the minority that defects from this view. But I've never heard criticism against people who talk too much about arms races in public, except this one comment from my friend.
* Talking about arms-race scenarios specifically makes it much more clear *why* we need global governance and improved cooperation. It's more persuasive than just saying, "Wouldn't it be great if the world could sing Kumbaya?"

That said, I remain open to being persuaded otherwise, and it seems important to think more carefully about how careful to be here. The good news is that the information hazards are unlikely to be disastrous, because all of this material is already publicly available somewhere. In other words, the upsides and downsides of making a bad judgment seem roughly on the same order of magnitude.

How do our prospects look?
--------------------------

In *Technological change and nuclear arms control* (1986), Ted Greenwood suggests that arms control has historically had little counterfactual impact:

> In no case has an agreement inhibited technological change that the United States both actually wanted to pursue at the time of agreement and was capable of pursuing during the intended duration of the agreement. Only in one area of technological innovation (i.e., SALT II constraints on the number of multiple independently-targetable reentry vehicles, or MIRVs, on existing missiles) is it possible that such agreements actually inhibited Soviet programs, although in another (test of new light ICBMs [intercontinental ballistic missiles]) their program is claimed by the United States to violate the SALT II Treaty that the Soviets have stated they will not undercut.
>
>

In "Why Military Technology Is Difficult to Restrain" (1987), Greenwood adds that the [INF Treaty](https://en.wikipedia.org/wiki/Intermediate-Range_Nuclear_Forces_Treaty) was arguably more significant, but it still didn't stop technological development, just a particular application of known technology.

John O. McGinnis [argues against](http://scholarlycommons.law.northwestern.edu/cgi/viewcontent.cgi?article=1193&context=nulr_online "'Accelerating AI', pp. 374-75") the feasibility of achieving global cooperation on AI:

> the only realistic alternative to unilateral relinquishment would be a global agreement for relinquishment or regulation of AI-driven weaponry. But such an agreement would face the same insuperable obstacles nuclear disarmament has faced. [...] Not only are these weapons a source of geopolitical strength and prestige for such nations, but verifying any prohibition on the preparation and production of these weapons is a task beyond the capability of international institutions.
>
>

In other domains we also see competition prevail over cooperation, such as in most markets, where usually there are at least several companies vying for customers. Of course, this is partly by social design, because we have anti-trust laws. Competition in business makes companies worse off while making consumers better off. Likewise, competition to build a quick, hacky AI makes human nations worse off while perhaps making the unsafe AIs better off. If we care some about the unsafe AIs for their own sakes as intelligent [preference-satisfying agents](http://www.utilitarian-essays.com/hedonistic-vs-preference.html), then this is less of a loss than it at first appears, but it still seems like there's room to expand the pie, and reduce suffering, if everyone takes things more slowly.

Maybe the best hope comes from the possibility of global unification. There is just one US government, with a monopoly on military development. If instead we had just one world government with a similar monopoly, arms races would not be necessary. Nationalism has been a potent force for gluing countries together and if channeled into internationalism, perhaps it could help to bind together a unified globe. Of course, we shouldn't place all our hopes on a world government and need to prepare for arms-control mechanisms that can also work with the present-day nation-state paradigm.

Robot arms races
----------------

Robots require AI that contains clear goal systems and an ability to act effectively in the world. Thus, they seem like a reasonable candidate for where artificial general intelligence will first emerge. Facebook's image-classification algorithms and Google's search algorithms don't need *general* intelligence, with many human-like cognitive faculties, as much as a smart robot does.

Military robotics seems like one of the most likely reasons that a robot arms race might develop. Indeed, to some degree there's already an arms race to build drones and autonomous weapons systems. [Mark Gubrud](http://gubrud.net/?p=35):

> Killer robots are not the only element of the global technological arms race, but they are currently the most salient, rapidly-advancing and fateful. If we continue to allow global security policies to be driven by advancing technology, then the arms race will continue, and it may even reheat to Cold War levels, with multiple players this time. Robotic armed forces controlled by AI systems too complex for anyone to understand will be set in confrontation with each other, and sooner or later, our luck will run out.
>
>

Nanotech arms races
-------------------

Nanotechnology admits the [prospect of severe arms races](http://www.crnano.org/dangers.htm#arms) as well. "[Can an MM Arms Race Avoid Disaster?](http://crnano.typepad.com/crnblog/2004/06/can_an_mm_arms_.html)" lists many reasons why a nanotech race should be less stable than the nuclear race was. In "[War, Interdependence, and Nanotechnology](http://www.futurebrief.com/miketrederwar002.asp)," Mike Treder suggests that because nanotech would allow countries to produce their own goods and energy with less international trade, there would be less incentive to refrain from preemptive aggression. Personally, I suspect that countries would still be very desperate to trade *knowledge* about nanotech itself to avoid falling behind in the race, but perhaps if a country was the world's leader in nanoweapons, it would have incentive to attack everyone else before the tables turned.

Mark Gubrud's "[Nanotechnology and International Security](http://www.foresight.org/Conferences/MNT05/Papers/Gubrud/)" presents an excellent overview of issues with both AI and nanotech races. He suggests:

> Nations must learn to trust one another enough to live without massive arsenals, by surrendering some of the prerogatives of sovereignty so as to permit intrusive verification of arms control agreements, and by engaging in cooperative military arrangements. Ultimately, the only way to avoid nanotechnic confrontation and the next world war is by evolving an integrated international security system, in effect a single global regime. World government that could become a global tyranny may be undesirable, but nations can evolve a system of international laws and norms by mutual agreement, while retaining the right to determine their own local laws and customs within their territorial jurisdictions.
>
>

According to Jürgen Altmann's talk, "[Military Uses of Nanotechnology and Nanoethics](http://www.youtube.com/watch?v=MANPyybo-dA)," 1/4 to 1/3 of US federal funding in the National NT Initiative is for defense -- $460 million out of $1.554 billion in 2008 (video time: 18:00). The US currently spends 4-10 times the rest of the world in military nanotech R&D, compared with "only" 2 times the rest of the world in overall military R&D (video time: 22:28). Some claim the US should press ahead with this trend in order to maintain a monopoly and prevent conflicts from breaking out, but it's dubious that nanotech can be contained in this way, and Altmann instead proposes active arms-control arrangements with anytime, anywhere inspections and in the long run, progress toward global governance to allay security dilemmas. We have seen many successful bans on classes of technology (bioweapons, chemical weapons, blinding lasers, etc.), so nano agreements are not out of the question, though they will take effort because many of the applications are so inherently dual-use. Sometimes commentators scoff at enforcement of norms against use of chemical weapons when just as many people can be killed by conventional forces, but these agreements are actually really important, as precedents for setting examples that can extend to more and more domains.

Like AI, nanotech may involve the prospect of the technology leader taking over the world. It's not clear which technology will arrive first. Nanotech contributes to the continuation of Moore's law and therefore makes brute-force evolved AI easier to build. Meanwhile, AI would vastly accelerate nanotech. Speeding up either leaves less time to prepare for both.

Feedback
--------

To read comments on this piece, see the [original LessWrong discussion](http://lesswrong.com/lw/j9u/international_cooperation_vs_ai_arms_race/).


Reasons to Be Nice to Other Value Systems
=========================================

29 August 2015
by [Brian Tomasik](https://longtermrisk.org/author/brian-tomasik/ "Posts by Brian Tomasik")

First written: 16 Jan. 2014; last update: 17 Oct. 2017

 I suggest several arguments in support of the heuristic that we should help groups holding different value systems from our own when doing so is cheap, unless those groups prove uncooperative to our values. This is true even if we don't directly care at all about other groups' value systems. Exactly how nice to be depends on the particulars of the situation, but there are some cases where helping others' moral views is clearly beneficial for us.

Contents

* [Introduction](#Introduction)
* [Example: Altruism tabling](#Example_Altruism_tabling)
* [Reasons to be nice](#Reasons_to_be_nice)
	+ [Iterated prisoner's dilemmas](#Iterated_prisoners_dilemmas)
	+ [Evolved emotions](#Evolved_emotions)
	+ [Reputation](#Reputation)
	+ [Common sense](#Common_sense)
	+ [Norms and universal rules](#Norms_and_universal_rules)
	+ [Encouraging global cooperation](#Encouraging_global_cooperation)
	+ [Utilitarianism](#Utilitarianism)
	+ [Moral uncertainty](#Moral_uncertainty)
	+ [Superrationality](#Superrationality)
* [Is it ok to cheat in secret?](#Is_it_ok_to_cheat_in_secret)
* [Risks to being nice](#Risks_to_being_nice)
* [How nice should you be?](#How_nice_should_you_be)
* [Applications to space colonization](#Applications_to_space_colonization)

Introduction
------------

A basic premise of economic policy, business strategy, and effective altruism is to choose the option with highest value per dollar. Ordinarily this simple rule suffices because we're engaged in one-player games against the environment. For instance, if Program #1 to distribute bed nets saves twice as many lives per dollar as Program #2, we choose Program #1. If Website B has 25% longer dwell time than Website A, we choose Website B. These are essentially engineering problems where one option is better for us, and no other agent else feels differently.

However, this mindset can run into trouble in social situations involving more than one player. I'll illustrate with a toy example that avoids naming specific groups, but the general structure transfers to many real-world cases.

Example: Altruism tabling
-------------------------

Suppose there's an Effective Altruism Fair at your local university, and altruists from various ideological stripes will be hosting the event and presenting their individual work. You really care about promoting Emacs, the [one true text editor](https://en.wikipedia.org/wiki/Editor_war). However, the Fair will also host a booth for the advocates of the Vi editor, which you consider not just inferior but actively harmful to the world.

The Fair requires some general organizing help -- to publicize, set up tables, and provide refreshments. Beyond that, it's up to the individual groups to showcase their own work to the visitors. Your Emacs club is deciding: How much effort should we put into helping out with general organizing, and how much should we devote to making our individual booth really awesome? You might evaluate this on the metric of how many email signups you'd get per hour of preparation work. And while you appreciate some things the Vi crowd does, you think they cause net harm on balance, so you might subtract off from your utility 1/2 times the number of email signups your effort allows them to get per hour.

If you help out with the general logistics of the Fair, it would produce a lot of new visitors, but only some fraction of them will be interested in Emacs. Say that every hour you put in provides 10 new Emacs signups, as well as 10 new Vi signups (plus maybe signups to other groups that are irrelevant to you). The overall value of this to you is only 10 - (1/2)\*10 = 5. In contrast, if you optimize your own booth, you can snatch an extra 15 visitors to yourself, with no extra Vi visitors in the process. Since 15 > 5, cost-effectiveness analysis says you should optimize only your booth. After all, this is the more efficient allocation of resources, right?

Suppose the Vi team faces the same cost-benefit tradeoffs. Then depending on which decisions each team makes, the following are the possible numbers of signups that each side will get, written in the format (# of Emacs signups), (# of Vi signups).

Total numbers of email signups

|  |  |  |
| --- | --- | --- |
|  | Vi help on logistics | Vi focus on own booth |
| Emacs help on logistics | 10+10 = 20,    10+10 = 20 | 10+0 = 10,    10+15 = 25 |
| Emacs focus on own booth | 15+10 = 25,    0+10 = 10 | 15+0 = 15,    0+15 = 15 |

Now remember that Emacs supporters consider Vi harmful, so that Emacs utility = (number of Emacs signups) - (1/2)\*(number of Vi signups). Suppose the Vi side feels exactly the same way in reverse. Then the actual utility values for each side, computed based on the above table, will be

Utility values

|  |  |  |
| --- | --- | --- |
|  | Vi help on logistics | Vi focus on own booth |
| Emacs help on logistics | 10,    10 | -2.5,    20 |
| Emacs focus on own booth | 20,    -2.5 | 7.5,    7.5 |

Just as we saw in the naive cost-effectiveness calculation, there's an advantage of 20 - 10 = 7.5 - (-2.5) = 10 to focusing on your own booth, regardless of what the other team does.

The game that this table represents is a [prisoner's dilemma](https://en.wikipedia.org/wiki/Prisoner%27s_dilemma) (PD) -- arguably the most famous in game theory. The dominant strategy in a one-shot PD is to defect, and this is what our naive calculation was capturing. In fact, both of the above tables are PDs, so the PD structure would have applied even absent enmity between the text-editor camps. PDs show up in very many real-world situations.

There's debate on whether defection in a one-shot PD is rational, but what is clear is that most of the world does not consist in one-shot PDs. For instance, what if the EA Fair is held again next year? How will the Vi team react then if you defect this year?

In addition, it may be in all of our interests to structure society in ways that prevent games from turning into one-shot PDs, because the outcome is worse for both sides than cooperation would have been, if only it could have been arranged.

Reasons to be nice
------------------

![](https://longtermrisk.org/files/Handshake_-_Pergamonmuseum-350x234.jpg "Ancient Roman sculpture at Pergamon Museum. By User: Sinbad. Licenses: (1) Creative Commons Attribution-ShareAlike 3.0 or (2) GNU Free Documentation License (GFDL), Version 1.2 or any later version published by the Free Software Foundation with no invariant sections. See https://fa.wikipedia.org/wiki/%D9%BE%D8%B1%D9%88%D9%86%D8%AF%D9%87:Handshake_-_Pergamonmuseum.JPG")

> If you have an opportunity to significantly help other value systems at small cost to yourself, you should do so.
>
>
> Likewise, if you have opportunity to avoid causing significant harm to other value systems by foregoing small benefit to yourself, you should do so. This is more true the more powerful is the value system you're helping. That said, if groups championing the other value system are defecting against you, then stop helping it.
>
>

### Iterated prisoner's dilemmas

Most of life has multiple rounds. Other groups of people generally don't go away after we've stepped on their toes, and if we defect now, they can defect on us in future interactions. There's extensive literature on the iterated prisoner's dilemma (IPD), but the general finding is that it tends to yield cooperation, especially over long time horizons without a definite end point. *[The Evolution of Cooperation](https://en.wikipedia.org/wiki/The_Evolution_of_Cooperation)* is an important book on this subject.

### Evolved emotions

One can debate whether a given situation adequately fits the properties of being a pure IPD. The translation from real-world situations to theoretical games is always messy. Regardless, the fact remains that empirically, humans feel reciprocal gratitude and indebtedness to those who helped them.

What's more, these feelings often persist *even when* there's no obvious further benefit from doing so. Emotions are humans' ways of making credible commitments, and the fact that humans feel loyalty and duty means that they can generally be trusted to reciprocate.

Of course, if you interact with people who are conniving and tend to backstab, then don't help them. Being nice does not mean being a sucker, and indeed, continuing to assist those who just take for themselves only encourages predation. (Of course, evolution has produced emotional exceptions to this, like in the case of altruism towards children and family members who share DNA with you, even if they never reciprocate.)

### Reputation

Reciprocal altruism typically occurs between individuals or groups, but there are also broader ways in which society transmits information about how generous someone is toward other values. When others discuss your work, wouldn't you rather have them say that you're a fair-minded and charitable individual who helps many different value systems, even those she doesn't agree with?

### Common sense

The heuristic of helping others when it's cheap to do so strikes most people as [common sense](http://lesswrong.com/lw/iao/common_sense_as_a_prior/). These values are taught in kindergarten and children's books.

### Norms and universal rules

Mahatma Gandhi [said](http://www.nytimes.com/2011/08/30/opinion/falser-words-were-never-spoken.html): "If we could change ourselves, the tendencies in the world would also change." We can see this idea expressed in other forms, such as the [categorical imperative](https://en.wikipedia.org/wiki/Categorical_imperative) or the dictums of a rule utilitarian. Society would be better -- even according to your own particular values -- if everyone followed the rule of helping other value systems when doing so had low cost.

When we follow and believe in these principles, it rubs off on others. Collectively it helps reinforce a norm of cooperation with those who feel differently from ourselves. Norms have significant social power, both for individuals and even for nations. [For instance](http://www.amazon.com/Norm-Dynamics-Multilateral-Arms-Control/dp/0820344230):

> Internationally, a cooperative security norm, if close to universality, can become the defining standard for how a good international citizen should behave. It is striking how in the 1980s and 1990s scores of formerly reluctant states were flocking to [Nonproliferation Treaty] NPT membership, notably after change in the national system of rule and particularly in the course of democratization processes: turning unequivocally nonnuclear or confirming nonnuclear status became the "right thing to do" (Rublee 2009; Müller and Schmidt 2010). [p. 4]
>
>

When we defect in any particular situation, we weaken cooperative norms for everyone for many future situations to come.

### Encouraging global cooperation

Norms of mutual assistance and tolerance among different groups are important not just for our own projects but also for international peace on a larger scale. To be sure, the contribution of our individual actions to this goal are miniscule, but the stakes are also high. A globally cooperative future could contain significantly less suffering and more of what other people value in expectation.

### Utilitarianism

Utilitarians care about the well-being or preference satisfaction of others. Thus, if many people feel that something is wrong, even if you don't, there's a utilitarian cost to it. This argument is stronger for preference utilitarians who value people's preferences about the external world even when they aren't consciously aware of violations of those preferences. Of course, this alone is probably not enough to encourage nice behavior, because present-day humans are vastly outweighed in direct value by non-human animals and future generations.

### Moral uncertainty

If you had grown up with different genes and environmental circumstances, you would have held the moral values that others espouse. In addition, you yourself might *actually*, not just hypothetically, later come to share those views -- due to new arguments, updated information, future life experiences, accretion of wisdom, or social influence. Or you might have come to hold those views if only you had heard arguments or learned things that you will not actually discover. What others believe provides [some evidence](http://blog.givewell.org/2013/05/02/broad-market-efficiency/comment-page-1/#comment-542986) for what an idealized version of you would believe. If so, then you might be mistaken that others' moral values are worthless in your estimation.

I should clarify that the value of cooperation does not rely on moral uncertainty; the other arguments are strong enough on their own. Moral uncertainty just provides some additional oomph, depending on how strongly it motivates you. (And you may want to apply some meta-level uncertainty on how much you care about moral uncertainty, if you care about meta-level uncertainty.)

### Superrationality

*This section was written by Caspar Oesterheld.*

Some decision theorists have argued that cooperation in a one-shot PD is justified if we face an opponent that uses a similar decision-making procedure as we do. After all, if we cooperate in such a PD, then our opponent is likely to do the same. [Hofstadter (1983)](https://www.gwern.net/docs/xrisks/1985-hofstadter#dilemmas-for-superrational-thinkers-leading-up-to-a-luring-lottery "'DILEMMAS FOR SUPERRATIONAL THINKERS, LEADING UP TO A LURING LOTTERY', 'Metamagical Themas: Sanity and Survival - Gwern.net'") calls this idea [superrationality](https://en.wikipedia.org/wiki/Superrationality "'Superrationality - Wikipedia'").

Some have used superrationality to argue that it is in our self-interest to be nice to other humans ([Leslie 1991](https://sl4librarian.files.wordpress.com/2016/12/two-bird-deaths-one-throw-leslie.pdf "'Ensuring Two Bird Deaths With One Throw'"), sec. 8; [Drescher 2006](https://smile.amazon.com/Good-Real-Demystifying-Paradoxes-Physics/dp/0262042339/ "'Good and Real: Demystifying Paradoxes from Physics to Ethics'"), ch. 7). For example, if I save a stranger from drowning, this makes it more likely that others will make a similar decision when I need help. However, in practice it seems that most people are not sufficiently similar to each other for this reasoning to apply in most situations. In fact, you may already know what other people think about when they decide whether to pull someone out of the water and that this is uncorrelated with your thoughts on superrationality. Thus, it is unclear whether superrationality has strong implications for how one should deal with other humans ([Oesterheld 2017](https://longtermrisk.org/files/Multiverse-wide-Cooperation-via-Correlated-Decision-Making.pdf "'Multiverse-wide Cooperation via Correlated Decision Making'"), sec. 6.6; [Almond 2010a](https://casparoesterheld.files.wordpress.com/2016/12/almond_edt_1.pdf "'On Causation and Correlation Part 1: Evidential decision theory is correct.'"), sec. 4.6; [Almond 2010b](https://web.archive.org/web/20120310010225/http://www.paul-almond.com/Correlation2.pdf "'On Causation and Correlation Part 2: Implications of Evidential Decision Theory'"), sec. 1; [Ahmed 2014](https://doi.org/10.1017/CBO9781139107990 "'Evidence, Decision and Causality'"), ch. 4).

However, even if Earth doesn’t harbor agents that are sufficiently similar to me, the multiverse as a whole probably does. In particular, it may contain a large set of agents who think about decision theory exactly like I do but have different values. Some of these will also care about what happens on Earth. If this is true and I also care about these other parts of the multiverse, then superrationality gives me a reason to be nice to these value systems. If I am nice toward them, then this makes it more likely that similar agents will also take my values into account when they make decisions in their parts of the multiverse (Oesterheld 2017).

Is it ok to cheat in secret?
----------------------------

Many of the reasons listed, especially the stronger ones, only have consequences when your cooperation or defection is visible: IPDs, evolved emotions, reputation, norms and universal rules, and encouraging global cooperation. Assuming the other, remaining reasons are weak enough, doesn't this license us to trash other value systems in our private decisions, so long as no one will find out?

No. There's too much risk of it backfiring in your face. One slip-up could damage your reputation, and your deception might show through in ways you don't realize. I think it's best to *actually be* someone who wants to help other value systems, regardless of whether others find out. This may sound suboptimal, and maybe there is a little bit of faith to it, but consider that almost everyone in the world recognizes this idea at least to some extent, such as in the law of karma or the Golden Rule. If it were an "irrational" policy for social success, why would we see it so widespread? [Eliezer Yudkowsky](http://lesswrong.com/lw/nc/newcombs_problem_and_regret_of_rationality/): "Be careful of [...] any time you find yourself defining the 'winner' as someone other than the agent who is currently smiling from on top of a giant heap of utility."

Not hiding your defection against others is a special case of the [general argument for honesty](http://www.utilitarian-essays.com/honesty.html). This isn't to say you always have to be cooperative, but if you're not, don't go out of your way to hide it.

I regard not trashing other value systems as a weak ethical injunction for guiding my decisions. I recommend reading Eliezer Yudkowsky's [sequence](http://wiki.lesswrong.com/wiki/Ethical_injunction#Sequence) for greater elaboration of why ethical injunctions can win better than naive act-utilitarianism. The injunction not to step on others' toes is not as strong as the injunction against lying, stealing, and so on; indeed, it's *never* possible to not step on some people's toes. But in cases where it's relatively easy to avoid causing major harm to what a significant number of others care about, you should try to avoid causing that harm. Of course, if others are substantially and unremorsefully stepping on your toes, then this advice no longer applies, and you should stop being nice to them, until they start being cooperative again.

Risks to being nice
-------------------

Being nice is not guaranteed to yield the best outcomes. There are reasons we evolved selfish motives as well as altruistic ones, and the "nice guys finish last" slogan is sometimes accurate. The other side might cheat you and get away with it. Maybe the IPD structure isn't sufficient to guarantee cooperation. Maybe it's a tragedy of the commons (multi-player prisoner's dilemma) where it's much harder to change defection to cooperation, and your efforts fail to make their intended impact.

It's important to assess these risks and be conscious of when your efforts at cooperation fail. But remember: Being nice means defecting on the other side if it defects on you. Niceness doesn't mean being exploited permanently. It's better to *try* a gesture of cooperation first rather than assume it won't work; predicting defection may become a self-fulfilling prophecy. In addition, I think niceness is increasingly rewarded in our more interconnected and transparent world, facilitated by governments and media. Our ancestral selfish tendencies probably overfire relative to the strategic optimum.

However, there are many real-world cases where niceness fails. One striking demonstration of this was the attempts by US president Barack Obama to compromise with opposing Republicans, which repeatedly resulted in Obama and the Democrats [making concessions for](https://www.youtube.com/watch?v=r-_RPbcuGEA "'Obama Concedes To GOP On Cuts Again - Why?'") nothing in return. This is not how to play an iterated prisoner's dilemma. If niceness repeatedly fails to achieve cooperation, then one has to go on the offensive instead.

If you hold a popular position, then I think it's [often successful](https://www.youtube.com/watch?v=ukVhZkexq-c&t=4m47s "'Obama: Increase Social Security Benefits', Jun 4, 2016") to firmly stand your ground rather than making concessions in response to [squeaky-wheel](https://en.wikipedia.org/wiki/The_squeaky_wheel_gets_the_grease "'The squeaky wheel gets the grease'") opponents. [Cenk Uygur](https://www.youtube.com/watch?v=M1qoQkL_QDU&t=7m36s "'If Cenk Were Obama: Supreme Court Nominee Edition'"): "Do you know what works in politics? Strength."

Cooperation can also entail overhead costs in terms of negotiating and verifying commitments, as well as assessing whether an apparent concession is actually a concession or just something the other side was already going to do. For small interactions, these overhead costs may outweigh the benefits of trade. Verifying cooperation is often easy if a business partner does a favor for you, because you can see what the favor is, and it's unlikely the partner would have done the favor without expecting anything in return. Verifying cooperation is often harder for big organizations or governments, because (1) the impacts of a change in policy can be diffuse and costly to measure and (2) it's difficult to know how much the change in policy is due to cooperation versus how much it's something the organization was going to do anyway.

How nice should you be?
-----------------------

I hope it's clear that at least in some cases, being nice pays. The harder question is how nice to be, i.e., above what threshold of cost to yourself do you stop providing benefits to others?

If bargains could be transacted in an airtight fashion, and if [utility was completely transferable](http://www.ssc.wisc.edu/~dquint/econ522%20fall%202012/522section1.pdf), then the answer would be simple: Maximize total social "pie," because if you can provide someone a benefit B that's bigger than its cost C to yourself, the other person could pay you back in the amount C, and then the surplus B-C could be divided between the two of you, making you both better off. Alas, most situations in life aren't airtight, so intuitively, in many cases it would not be in your interest to purely maximize pie. There might be some noise or cheating between your incurring the cost and someone else paying back a higher benefit. Not everything you do is fully recognized and rewarded by others, especially when they assume that you're helping them because you intrinsically value their cause rather than just to be nice despite not caring or even slightly disvaluing it.

How nice to be depends on the details of the social situation, expectations, norms, and enforcement mechanisms involved. There's some balance to strike between purely pushing your own agenda without regard to what anyone else cares about versus purely helping all value systems without any preference for your personal concerns. One could construct various game-theoretic models, but the world is complicated, and interactions are not *just*, say, a series of two-player IPDs. It could also help to look at real examples in society for where to strike this balance.

Applications to space colonization
----------------------------------

Being nice suggests that people whose primary concern is reducing suffering should accept others' ambitions to colonize space, so long as colonizers work harder to reduce the [suffering that space colonization entails](http://www.utilitarian-essays.com/astronomical-suffering.html). On the flip side, being nice also means that those who do want to colonize space should focus more on making space colonization better (more humane and better governed to stay in line with our values) rather than making it more likely to happen.


Reducing long-term risks from malevolent actors
===============================================

7 July 2020
by [David Althaus](https://longtermrisk.org/author/david-althaus/ "Posts by David Althaus") and [Tobias Baumann](https://longtermrisk.org/author/tobias-baumann/ "Posts by Tobias Baumann")

### Summary

* Dictators who exhibited highly narcissistic, psychopathic, or sadistic traits were involved in some of the greatest catastrophes in human history.
* Malevolent individuals in positions of power could negatively affect humanity’s long-term trajectory by, for example, exacerbating international conflict or other broad risk factors.
* Malevolent humans with access to advanced technology—such as whole brain emulation or other forms of transformative AI—could cause serious existential risks and suffering risks.
* We therefore consider interventions to reduce the expected influence of malevolent humans on the long-term future.
* The development of manipulation-proof measures of malevolence seems valuable, since they could be used to screen for malevolent humans in high-impact settings, such as heads of government or CEOs.
* We also explore possible future technologies that may offer unprecedented leverage to mitigate against malevolent traits.
* Selecting against psychopathic and sadistic tendencies in genetically enhanced, highly intelligent humans might be particularly important. However, risks of unintended negative consequences must be handled with extreme caution.
* We argue that further work on reducing malevolence would be valuable from many moral perspectives and constitutes a promising focus area for longtermist EAs.

**Full article**

* [PDF](https://longtermrisk.org/files/Reducing_long_term_risks_from_malevolent_actors.pdf)
* [EA Forum post](https://forum.effectivealtruism.org/posts/LpkXtFXdsRd4rG8Kb/reducing-long-term-risks-from-malevolent-actors)