Similarity Between Historical and Contemporary AI Predictions

AI predictions from public statements made before and after 2000 form similar distributions. Such predictions from before 1980 appear to be more optimistic, though predictions from a larger early survey are not.

Discussion
----------

### Similarity of predictions over time

#### MIRI dataset

We compared early and late predictions using the [MIRI dataset](http://aiimpacts.org/miri-ai-predictions-dataset/ "MIRI AI Predictions Dataset"). We find the correlation between the date of a prediction and number of years until AI is predicted from that time is 0.13. Most predictions are in the last decade or two however, so this does not tell us much about long run trends (see Figure 1).

[![xxx](http://aiimpacts.org/wp-content/uploads/2014/12/Time-to-AI-more-likely-than-not-1-1024x761.png)](http://aiimpacts.org/wp-content/uploads/2014/12/Time-to-AI-more-likely-than-not-1.png)**Figure 1:** Years until AI is predicted ([minPY](http://aiimpacts.org/miri-ai-predictions-dataset/ "MIRI AI Predictions Dataset")) over time in [MIRI AI predictions dataset](http://aiimpacts.org/miri-ai-predictions-dataset/ "MIRI AI Predictions Dataset").
The six predictions prior to 1980 were all below the median 30 years, which would have less than 2% chance if they were really drawn from the same distribution.

The predictions made before and after 2000 form very similar distributions however (see figure 2). The largest difference between the fraction of pre-2000 and since-2000 people who predict AI by any given distance in the future is about 15%. A difference this large is fairly likely by chance, according to our unpublished calculations. See the [MIRI dataset page](https://sites.google.com/site/aiimpactslibrary/ai-timelines/predictions-of-human-level-ai-dates/miri-ai-predictions-dataset) for further details.

[![Late and early predictions](http://aiimpacts.org/wp-content/uploads/2014/12/time-to-predictions-2-11.png)](http://aiimpacts.org/wp-content/uploads/2014/12/time-to-predictions-2-11.png)**Figure 2:** Cumulative probability of prediction falling less than X years from date of writing ([minPY; from MIRI AI predictions dataset](http://aiimpacts.org/miri-ai-predictions-dataset/ "MIRI AI Predictions Dataset"))
#### Survey data

[The surveys we know of](http://aiimpacts.org/ai-timeline-surveys/) provide some evidence against early predictions being more optimistic. [Michie’s survey](http://aiimpacts.org/michie-survey/) is the only survey we know of made more than ten years ago. Ten surveys have been made since 2005 that give median predictions or median fiftieth percentile dates. The median date in Michie’s survey is the third furthest out in the set of eleven: fifty years, compared to common twenty to forty year medians now. Michie’s survey does not appear to have involved options between twenty and fifty years however, making this result less informative. However it suggests the researchers in the survey were not substantially more optimistic than researchers in modern surveys. They were also apparently more pessimistic than the six early statements from the MIRI dataset discussed above, though some difference [should be expected](http://aiimpacts.org/ai-timeline-predictions-in-surveys-and-statements/) from comparing a survey with public statements. Michie’s survey had sixty-three respondents, compared to the MIRI dataset’s six, making this substantial evidence.

[![Survey results, as shown in Michie's paper.](http://aiimpacts.org/wp-content/uploads/2015/01/michietimelines-copy-1.jpg)](http://aiimpacts.org/wp-content/uploads/2015/01/michietimelines-copy-1.jpg)**Figure 3:** Survey results, as shown in Michie’s paper.
#### Armstrong and Sotala on failed past predictions

[Armstrong and Sotala](http://intelligence.org/files/PredictingAI.pdf) compare failed predictions of the past to all predictions, in an older version of the same dataset. This is not the same as comparing historical to contemporary predictions, but related. In particular, they use a subset of past predictions to conclude that contemporary predictions are likely to be similarly error-prone.

They find that predictions which are old enough to be proved wrong form a similar distribution to the entire set of predictions (see Figures 5-6). They infer that recent predictions are likely to be flawed in similar ways to the predictions we know to be wrong.

This inference appears to us to be wrong, due to a selection bias. Figure 4 illustrates how such reasoning fails with a simple scenario where the failure is clear. In this scenario, for thirty years people have divided their predictions between ten, twenty and thirty years out. In 2010 (the fictional present), many of these predictions are known to have been wrong (shown in pink). In order for the total distribution of predictions to match that of the past failed predictions, the modern predictions have to form a very distribution from the historical predictions (e.g. one such distribution is shown in the ‘2010’ row).

[![xxx](http://aiimpacts.org/wp-content/uploads/2014/12/by-error-or-time-copy-300x192.jpg)](http://aiimpacts.org/wp-content/uploads/2014/12/by-error-or-time-copy.jpg)**Figure** **4:** a hypothetical distribution of predictions.
In general, failed predictions are disproportionately short early predictions, and also disproportionately short bad predictions. If the distributions of failed and total predictions look the same, this suggests that the distribution of early and late predictions is *not* the same – the later predictions must include fewer longer predictions, to make up for the longer unfalsified predictions inherited from the earlier dates, as well as the longer predictions effectively missing from the earlier dates.

If the characteristic lengths of the predictions was small relative to the time between different predictions, these biases would be small. In the example in figure 4, if the distance between the groups had been one hundred years, there would be no problem. However in the MIRI dataset, both are around twenty years.

In sum, the fact that failed predictions look like all predictions suggests that historical predictions came from a different distribution to present predictions. Which would seem to be good news about present predictions, if past predictions were bad. However, we have other evidence from comparing predictions from before and after 2000 directly that those are fairly similar (see above), so if earlier methods were unsuccessful, this is some mark against current methods. On the other hand, public predictions from before 1980 appear to be systematically more optimistic.

[![xxx](http://aiimpacts.org/wp-content/uploads/2014/12/expert-lay-a-and-s-copy.png)](http://aiimpacts.org/wp-content/uploads/2014/12/expert-lay-a-and-s-copy.png)**Figure 5:** Figures from [Armstrong and Sotala](http://intelligence.org/files/PredictingAI.pdf)
[![xxx](http://aiimpacts.org/wp-content/uploads/2014/12/failed-ai-predictions-a-and-s-copy.jpg)](http://aiimpacts.org/wp-content/uploads/2014/12/failed-ai-predictions-a-and-s-copy.jpg)**Figure 6:** Figure from [Armstrong and Sotala](http://intelligence.org/files/PredictingAI.pdf)
### Implications

**Accuracy of AI predictions:** if people make fairly similar predictions over time, this is some evidence that they are not making their predictions based on information about their environment, which has changed over the decades (at a minimum, time has passed). For instance, some suspect that people make their predictions a fixed number of years into the future, to maximize their personal benefits from making exciting but hard to verify predictions. Evidence for this particular hypothesis seems weak to us, however the general point stands. This evidence against accuracy is not as strong as it may seem however, since there appear to be reasonable prior distributions over the date of AI which look the same after seeing time pass.


Human-Level AI

*Published 23 January 2014, last updated Aug 7 2022*

‘*Human-level AI’* refers to AI which can reproduce everything a human can do, approximately. Several variants of this concept are worth distinguishing.

Details
-------

### Variations in the meaning of ‘human-level AI’

Considerations in specifying ‘human-level AI’ more precisely:

* **Do we mean to imply anything about running costs?** Is an AI that reproduces human behavior for ten billion dollars per year ‘human-level’, or does it need to be human-priced? See *‘human-level at any cost vs. human-level at human cost’* below for more details.
* **What characteristics of a human need to be reproduced?** Usually we do not mean that the AI should be indistinguishable from a human. For instance, we usually do not care whether it looks like a human. A common requirement is that the AI have the economically valuable skills of a human. We sometimes also talk about AI being ‘human-level’ in a narrower set of relevant characteristics, such as in its ability to do further AI research.
* **What does it mean to reproduce human behavior?** If AI replaces all hairdressers in society, but uniformly produces a slightly worse haircut in some dimensions (but so cheaply!), does that count as ‘human-level’? If not, then all humans may be replaced even though AI is not ‘human-level’. On the other hand, if this does count, then where is the line? Can an AI ‘reproduce human behavior’ by merely producing anything a buyer would prefer to have than what a human produces? Many machines already do this, and this is not what we mean.
* **How much of human behavior needs to be reproduced?** If AI cannot entirely compete with humans for the job of waiter, merely because some small population prefers human waiters, this will not make a large difference to anything, so a requirement that human-level AI replace humans in all economically useful skills is too high a bar for what we are intuitively interested in. There is a further question of what metric one might use when specifying the bar.
* **What conditions is the AI available under?** Does it matter if it has actually been built? Need it be available in some particular marketplace, or quantity, or price range?

### Related definitions

#### High Level Machine Intelligence (HLMI)

The 2016 and 2022 Expert Surveys on Progress in AI use the following definition:

> Say we have ‘high-level machine intelligence’ when unaided machines can accomplish every task better and more cheaply than human workers. Ignore aspects of tasks for which being a human is intrinsically advantageous, e.g. being accepted as a jury member. Think feasibility, not adoption.
>
>

#### Superhuman AI

A ‘superhuman’ system is meaningfully more capable than a human-level system. In practice the first human-level system [is likely](https://aiimpacts.org/human-level-ai/#Human-level_is_superhuman) to be superhuman.

### Key issues

#### Human-level at any cost vs. human-level at human cost

In common usage, ‘human-level’ AI can mean either AI which can reproduce a human at any cost and speed, or AI which can replace a human (i.e. is as cheap as a human, and can be used in the same situations). Both are relevant for different issues. For instance, the ‘at any cost’ meaning is important when considering how people will respond to human-level artificial intelligence, or whether a human-level artificial intelligence will use illicit means to acquire resources and cause destruction. Human-level at human cost is the relevant concept when thinking about AI replacing humans in the labor market, the economy growing very fast, or legitimate AI development ramping up into an intelligence explosion.

Today few applications are more than an order of magnitude more expensive to run than a human, suggesting a short time before an AI project came down in price to the cost of a human. However some applications are more expensive, and even if an early AI project were only a few orders of magnitude more expensive than a human per time, it may be much slower. Thus it is hard to make useful inferences about the potential time delay between an arbitrarily expensive human-level AI and an AI which might replace a human, even if we assume hardware continues to fall in price regularly.

#### ‘Human-level’ is superhuman

As explained at the [Superintelligence Reading Group](http://lesswrong.com/lw/ku6/superintelligence_reading_group_section_1_past/):

> Another thing to be aware of is the diversity of mental skills. If by ‘human-level’ we mean a machine that is at least as good as a human at each of these skills, then in practice the first ‘human-level’ machine will be much better than a human on many of those skills. It may not seem ‘human-level’ so much as ‘very super-human’.
>
>
> We could instead think of human-level as closer to ‘competitive with a human’ – where the machine has some super-human talents and lacks some skills humans have. This is not usually used, I think because it is hard to define in a meaningful way. There are already machines for which a company is willing to pay more than a human: in this sense a microscope might be ‘super-human’. There is no reason for a machine which is equal in value to a human to have the traits we are interested in talking about here, such as agency, superior cognitive abilities or the tendency to drive humans out of work and shape the future. Thus we talk about AI which is at least as good as a human, but you should beware that the predictions made about such an entity may apply before the entity is technically ‘human-level’.
>
>
> [![From Superintelligence Reading Group](http://aiimpacts.wpengine.com/wp-content/uploads/2014/12/human-level-AI-300x230.png)](http://aiimpacts.wpengine.com/wp-content/uploads/2014/12/human-level-AI.png)Example of how the first ‘human-level’ AI may surpass humans in many ways.


AI Impacts Quarterly Newsletter, Jan-Mar 2023

*Harlan Stewart, 17 April 2023*

News
====

### AI Impacts blog

We moved our [blog](https://blog.aiimpacts.org/) to Substack! We think this platform has many advantages, and we’re excited for the blog to live here. You can now easily [subscribe](https://blog.aiimpacts.org/subscribe) to the blog to receive regular newsletters as well as various [thoughts](https://blog.aiimpacts.org/p/we-dont-trade-with-ants) and [observations](https://blog.aiimpacts.org/p/how-popular-is-chatgpt-part-2-slower) related to AI.

### AI Impacts wiki

All AI Impacts research pages now reside on the [AI Impacts Wiki](https://wiki.aiimpacts.org/). The wiki aims to document what we know so far about decision-relevant questions about the future of AI. Our pages have always been wiki-like: updatable reference pages organized by topic. We hope that making it an actual wiki will make it clearer to everyone what’s going on, as well as better to use for this purpose, for both us and readers. We are actively looking for ways to make the wiki even better, and you can help with this by sharing your thoughts in our [feedback form](https://aiimpacts.org/feedback/) or in the comments of this blog post!

### New office

We recently moved to a new office that we are sharing with [FAR AI](https://far.ai/) and other partner organizations. We’re extremely grateful to the team at FAR for organizing this office space, as well as to the Lightcone team for hosting us over the last year and a half.

### Katja Grace talks about forecasting AI risk at EA Global

At EA Global Bay Area 2023, Katja gave a talk titled [Will AI end everything? A guide to guessing](https://youtu.be/j5Lu01pEDWA) in which she outlined a way to roughly estimate the extent of AI risk.

### AI Impacts in the Media

* AI Impacts’ [2022 Expert Survey on Progress in AI](https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/) was cited in an [NBC Nightly News segment](https://youtu.be/qRLrE2tkr2Y), an [op-ed in Bloomberg](https://www.bloomberg.com/opinion/articles/2023-04-02/regulating-ai-might-require-a-new-federal-agency), an [op-ed in The New York Times](https://www.nytimes.com/2023/03/27/opinion/ai-chatgpt-chatbots.html), an [article in Our World in Data](https://ourworldindata.org/ai-timelines), and an [interview with Kelsey Piper](https://www.nytimes.com/2023/03/21/podcasts/ezra-klein-podcast-transcript-kelsey-piper.html).
* Ezra Klein quoted Katja and separately cited the survey in his New York Times op-ed [This Changes Everything](https://www.nytimes.com/2023/03/12/opinion/chatbots-artificial-intelligence-future-weirdness.html).
* Sigal Samuel interviewed Katja for the Vox article [The case for slowing down AI](https://www.vox.com/the-highlight/23621198/artificial-intelligence-chatgpt-openai-existential-risk-china-ai-safety-technology).

Research and writing highlights
===============================

### AI Strategy

* “[Let’s think about slowing down AI](https://blog.aiimpacts.org/p/lets-think-about-slowing-down-ai)” argues that those who are concerned about existential risks from AI should think about strategies that could slow the progress of AI (Katja)
* “[Framing AI strategy](https://blog.aiimpacts.org/p/framing-ai-strategy)” discusses ten frameworks for thinking about AI strategy. (Zach)
* “[Product safety is a poor model for AI governance](https://blog.aiimpacts.org/p/product-safety-is-a-poor-model-for-ai-governance)” argues that a common type of policy proposal is inadequate to address the risks of AI. (Rick)
* “[Alexander Fleming and Antibiotic Resistance](https://aiimpacts.org/wp-content/uploads/2023/04/Alexander_Fleming__antibiotic_resistance__and_relevant_lessons_for_the_mitigation_of_risk_from_advanced_artificial_intelligence.pdf)” is a research report about early efforts to prevent antibiotic resistance and relevant lessons for AI risk. (Harlan)

### Resisted technological temptations: how much economic value has been forgone for safety and ethics in past technologies?

* “[What we’ve learned so far from our technological temptations project](https://blog.aiimpacts.org/p/what-weve-learned-so-far-from-technological)” is a blog post that summarizes the Technological Temptations project and some possible takeaways. (Rick)
* [Geoengineering](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:technological_inevitability:incentivized_technologies_not_pursued:geoengineering), [nuclear power](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:technological_inevitability:incentivized_technologies_not_pursued:nuclear_power), and [vaccine challenge trials](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:technological_inevitability:incentivized_technologies_not_pursued:vaccine_challenge_trials) were evaluated for the amount of value that may have been forgone by not using them. (Jeffrey)

### Public awareness and opinions about AI

* “[The public supports regulating AI for safety](https://blog.aiimpacts.org/p/the-public-supports-regulating-ai-for-safety)” summarizes the results from a survey of the American public about AI. (Zach)
* “How popular is ChatGPT?”: [Part 1](https://blog.aiimpacts.org/p/how-popular-is-chatgpt-part-1-more-popular-than-taylor-swift) looks at trends in AI-related search volume, and [Part 2](https://blog.aiimpacts.org/p/how-popular-is-chatgpt-part-2-slower) refutes a widespread claim about the growth of ChatGPT. (Harlan and Rick)

### The state of AI today: funding, hardware, and capabilities

* “[Recent trends in funding for AI companies](https://wiki.aiimpacts.org/doku.php?id=wiki:ai_timelines:ai_inputs:recent_trends_in_ai_investment)” analyzes data about the amount of funding AI companies have received. (Rick)
* “[How much computing capacity exists in GPUs and TPUs in Q1 2023?](https://wiki.aiimpacts.org/doku.php?id=ai_timelines:hardware_and_ai_timelines:computing_capacity_of_all_gpus_and_tpus)” uses a back-of-the-envelope calculation to estimate the total amount of compute that exists on all GPUs and TPUs. (Harlan)
* “[Capabilities of state-of-the-art AI, 2023](https://wiki.aiimpacts.org/doku.php?id=uncategorized:capabilities_of_sota_ai)” is a list of some noteworthy things that state-of-the-art AI can do. (Harlan and Zach)

### Arguments for AI risk

* Still in progress, “[Is AI an existential risk to humanity?](https://wiki.aiimpacts.org/doku.php?id=arguments_for_ai_risk:is_ai_an_existential_threat_to_humanity:start)” is a partially complete page summarizing various arguments for concern about existential risk from AI. A couple of specific arguments are examined more closely in “[Argument for AI x-risk from competent malign agents](https://wiki.aiimpacts.org/doku.php?id=arguments_for_ai_risk:is_ai_an_existential_threat_to_humanity:will_malign_ai_agents_control_the_future:argument_for_ai_x-risk_from_competent_malign_agents:start)” and “[Argument for AI x-risk from large impacts](https://wiki.aiimpacts.org/doku.php?id=arguments_for_ai_risk:is_ai_an_existential_threat_to_humanity:argument_for_ai_x-risk_from_large_impacts)” (Katja)

### Chaos theory and what it means for AI safety

* “[AI Safety Arguments Affected by Chaos](https://wiki.aiimpacts.org/doku.php?id=uncategorized:ai_safety_arguments_affected_by_chaos)” reasons about ways in which chaos theory could be relevant to predictions about AI, and “[Chaos in Humans](https://wiki.aiimpacts.org/doku.php?id=uncategorized:ai_safety_arguments_affected_by_chaos:chaos_in_humans)” explores the theoretical limits to predicting human behavior. The report “[Chaos and Intrinsic Unpredictability](http://aiimpacts.org/wp-content/uploads/2023/04/Chaos-and-Intrinsic-Unpredictability.pdf)” provides background, and a [blog post](https://blog.aiimpacts.org/p/superintelligence-is-not-omniscience) summarizes the project. (Jeffrey and Aysja)

### Miscellany

* “[How bad a future do ML researchers expect?](https://aiimpacts.org/how-bad-a-future-do-ml-researchers-expect/)” compares experts’ answers in 2016 and 2022 to the question “How positive or negative will the impacts of high-level machine intelligence on humanity be in the long run?” (Katja)
* “[We don’t trade with ants](https://blog.aiimpacts.org/p/we-dont-trade-with-ants)” (crosspost) disputes the common claim that advanced AI systems won’t trade with humans for the same reason that humans don’t trade with ants. (Katja)

Funding
=======

We’re actively seeking financial support to continue our research and operations for the rest of the year. Previous funding allowed us to expand our research team and hold a summer internship program.

If you want to talk to us about why we should be funded or hear more details about what we would do with money, please write to Elizabeth, Rick, or Katja at [firstname]@aiimpacts.org.

If you’d like to donate to AI Impacts, you can do so [here](https://aiimpacts.org/donate/). (And we thank you!)

*Image credit: Midjourney*


What we’ve learned so far from our technological temptations project

*Rick Korzekwa, 11 April 2023, updated 13 April 2023*

At AI Impacts, we’ve been looking into how people, institutions, and society approach novel, powerful technologies. One part of this is our [technological temptations project](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:technological_inevitability:incentivized_technologies_not_pursued:resisted_technological_temptations_project), in which we are looking into [cases](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:technological_inevitability:incentivized_technologies_not_pursued:start) where some actors had a strong incentive to develop or deploy a technology, but chose not to or showed hesitation or caution in their approach. Our researcher Jeffrey Heninger has recently finished some case studies on this topic, covering geoengineering, nuclear power, and human challenge trials.

This document summarizes the lessons I think we can take from these case studies. Much of it is borrowed directly from Jeffrey’s written analysis or conversations I had with him, some of it is my independent take, and some of it is a mix of the two, which Jeffrey may or may not agree with. All of it relies heavily on his research.

The writing is somewhat more confident than my beliefs. Some of this is very speculative, though I tried to flag the most speculative parts as such.

Summary
=======

Jeffrey Heninger investigated three cases of technologies that create substantial value, but were not pursued or pursued more slowly

**The overall scale of value at stake was very large** for these cases, on the order of hundreds of billions to trillions of dollars. But it’s not clear who could capture that value, so it’s not clear whether the temptation was closer to $10B or $1T.

**Social norms can generate strong disincentives** for pursuing a technology, especially when combined with enforceable regulation.

**Scientific communities** and individuals within those communities seem to have particularly high leverage in steering technological development at early stages.

**Inhibiting deployment can inhibit development** for a technology over the long term, at least by slowing cost reductions.

**Some of these lessons are transferable to AI**, at least enough to be worth keeping in mind.

Overview of cases
=================

1. [Geoengineering](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:technological_inevitability:incentivized_technologies_not_pursued:geoengineering) could feasibly provide benefits of $1-10 trillion per year through global warming mitigation, at a cost of $1-10 billion per year, but actors who stand to gain the most have not pursued it, citing a lack of research into its feasibility and safety. Research has been effectively prevented by climate scientists and social activist groups.
2. [Nuclear power](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:technological_inevitability:incentivized_technologies_not_pursued:nuclear_power) has proliferated globally since the 1950s, but many countries have prevented or inhibited the construction of nuclear power plants, sometimes at an annual cost of tens of billions of dollars and thousands of lives. This is primarily done through legislation, like Italy’s ban on all nuclear power, or through costly regulations, like safety oversight in the US that has increased the cost of plant construction in the US by a factor of ten.
3. [Human challenge trials](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:technological_inevitability:incentivized_technologies_not_pursued:vaccine_challenge_trials) may have accelerated deployment of covid vaccines by more than a month, saving many thousands of lives and billions or trillions of dollars. Despite this, the first challenge trial for a covid vaccine was not performed until after several vaccines had been tested and approved using traditional methods. This is consistent with the historical rarity of challenge trials, which seems to be driven by ethical concerns and enforced by institutional review boards.

Scale
=====

The first thing to notice about these cases is the scale of value at stake. Mitigating climate change could be worth hundreds of billions or trillions of dollars per year, and deploying covid vaccines a month sooner could have saved many thousands of lives. While these numbers do not represent a major fraction of the global economy or the overall burden of disease, they are large compared to many relevant scales for AI risk. The world’s most valuable companies have market caps of a few trillion dollars, and the entire world spends around two trillion dollars per year on defense. In comparison, annual funding for AI is on the order of $100B.[1](https://aiimpacts.org/what-weve-learned-so-far-from-our-technological-temptations-project/#easy-footnote-bottom-1-3536 "See our page on funding for AI companies and the 2023 AI Index report.")
[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc055821-ecf1-4261-b3be-268c312627ce_2288x1240.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc055821-ecf1-4261-b3be-268c312627ce_2288x1240.png)Comparison between the potential gains from mitigating global warming and deploying covid vaccines faster. These items were somewhat arbitrarily chosen, and most of the numbers were not carefully researched, but they should be in the right ballpark.
Setting aside for the moment who could capture the value from a technology and whether the reasons for delaying or forgoing its development are rational or justified, I think it is worth recognizing that the potential upsides are large enough to create strong incentives.

Social norms
============

My read on these cases is that a strong determinant for whether a technology will be pursued is social attitudes toward the technology and its regulation. I’m not sure what would have happened if Pfizer had, in defiance of FDA standards and medical ethics norms, infected volunteers with covid as part of their vaccine testing, but I imagine it would have been more severe than fines or difficulty obtaining FDA approval. They would have lost standing in the medical community and possibly been unable to continue existing as a company. This goes similarly for other technologies and actors. Building nuclear power plants without adhering to safety standards is so far outside the range of acceptable actions that even *suggesting* it as a strategy for running a business or addressing climate change is a serious risk to reputation for a CEO or public official. An oil company executive who finances a project to disperse aerosols into the upper atmosphere to reduce global warming and protect his business sounds like a Bond movie villain.

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F046fcc4a-46a1-4b98-9475-739de80813c2_826x1116.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F046fcc4a-46a1-4b98-9475-739de80813c2_826x1116.png)
This is not to suggest that social norms are infinitely strong or that they are always well-aligned with society’s interests. Governments and corporations will do things that are widely viewed as unethical if they think they can get away with it, for example, by doing it in secret.[2](https://aiimpacts.org/what-weve-learned-so-far-from-our-technological-temptations-project/#easy-footnote-bottom-2-3536 "Biological weapons research by the USSR is the best example of this that comes to mind.") And I think that public support for our current nuclear safety regime is gravely mistaken. But strong social norms, either against a technology or against breaking regulations do seem able, at least in some cases, to create incentives strong enough to constrain valuable technologies.

The public
----------

The public plays a major role in defining and enforcing the range of acceptable paths for technology. Public backlash in response to early challenge trials set the stage for our current ethics standards, and nuclear power faces crippling safety regulations in large part because of public outcry in response to a perceived lack of acceptable safety standards. In both of these cases, the result was not just the creation of regulations, but strong buy-in and a souring of public opinion on a broad category of technologies.[3](https://aiimpacts.org/what-weve-learned-so-far-from-our-technological-temptations-project/#easy-footnote-bottom-3-3536 "More speculatively, this may be important for geoengineering. Small advocacy groups were able to stop experiments with solar radiation management for reasons that are still not completely clear to me, but I think part of it is public suspicion toward attempts to manipulate the environment.")
Although public opposition can be a powerful force in expelling things from the Overton window, it does not seem easy to predict or steer. The Chernobyl disaster made a strong case for designing reactors in a responsible way, but it was instead viewed by much of the public as a demonstration that nuclear power should be abolished entirely. I do not have a strong take on how hard this problem is in general, but I do think it is important and should be investigated further.

The scientific community
------------------------

The precise boundaries of acceptable technology are defined in part by the scientific community, especially when technologies are very early in development. Policy makers and the public tend to defer to what they understand to be the official, legible scientific view when deciding what is or is not okay. This does not always match with actual views of scientists.

Geoengineering as an approach to reducing global warming has not been recommended by the IPCC, and a minority of climate scientists support research into geoengineering. Presumably the advocacy groups opposing geoengineering experiments would have faced a tougher battle if the official stance from the climate science community were in favor of geoengineering.

One interesting aspect of this is that scientific communities are small and heavily influenced by individual prestigious scientists. The taboo on geoengineering research was broken by the editor of a major climate journal, after which the number of papers on the topic increased by more than a factor of 20 after two years.[4](https://aiimpacts.org/what-weve-learned-so-far-from-our-technological-temptations-project/#easy-footnote-bottom-4-3536 "Oldham, Paul, Bronislaw Szerszynski, Jack Stilgoe, Calum Brown, Bella Eacott, and Andy Yuille. “Mapping the landscape of climate engineering.” Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 372, no. 2031 (2014): 20140065.")
[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7108f6-6580-4936-85fa-97def23222c1_1060x656.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7108f6-6580-4936-85fa-97def23222c1_1060x656.png)Scientific papers published on solar radiation management by year. Paul Crutzen, an influential climate scientist, published a highly-cited paper on the use of aerosols to mitigate global warming in 2006. Oldham, et al 2014.
I suspect the public and policymakers are not always able to tell the difference between the official stance of regulatory bodies and the consensus of scientific communities. My impression is that scientific consensus is not in favor of radiation health models used by the Nuclear Regulatory Commission, but many people nonetheless believe that such models are sound science.

Warning shots
-------------

Past incidents like the Fukushima disaster and the Tuskegee syphilis study are frequently cited by opponents of nuclear power and human challenge trials. I think this may be significant, because it suggests that these “warning shots” have done a lot to shape perception of these technologies, even decades later. One interpretation of this is that, regardless of why someone is opposed to something, they benefit from citing memorable events when making their case. Another, non-competing interpretation is that these events are causally important in the trajectory of these technologies’ development and the public’s perception of them.

I’m not sure how to untangle the relative contribution of these effects, but either way, it suggests that such incidents are important for shaping and preserving norms around the deployment of technology.

Locality
--------

In general, social norms are local. Building power plants is much more acceptable in France than it is in Italy. Even if two countries allow the construction of nuclear power plants and have similarly strong norms against breaking nuclear safety regulations, those safety regulations may be different enough to create a large difference in plant construction between countries, as seen with the US and France.

Because scientific communities have members and influence across international borders, they may have more sway over what happens globally (as we’ve seen with geoengineering), but this may be limited by local differences in the acceptability of going against scientific consensus.

Development trajectories
========================

A common feature of these cases is that preventing or limiting deployment of the technology inhibited its development. Because less developed technologies are less useful and harder to trust, this seems to have helped reduce deployment.

Normally, things become cheaper to make as we make more of them in a somewhat predictable way. The cost goes down with the total amount that has been produced, following a power law. This is what has been happening with solar and wind power.[5](https://aiimpacts.org/what-weve-learned-so-far-from-our-technological-temptations-project/#easy-footnote-bottom-5-3536 "Bolinger, Mark, Ryan Wiser, and Eric O’Shaughnessy. “Levelized cost-based learning analysis of utility-scale wind and solar in the United States.” Iscience 25, no. 6 (2022): 104378.")
[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0da2e9a-ee8d-41ab-a85b-d813aa76b1a3_2208x1094.jpeg)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0da2e9a-ee8d-41ab-a85b-d813aa76b1a3_2208x1094.jpeg)Levelized cost of energy for wind and solar power, as a function of total capacity built. Levelized cost includes cost building, operating, and maintaining wind and solar farms. Bolinger 2022
Initially, building nuclear power plants seems to have become cheaper in the usual way for new technology—doubling the total capacity of nuclear power plants reduced the cost per kilowatt by a constant fraction. Starting around 1970, regulations and public opposition to building plants did more than increase construction costs in the near term. By reducing the number of plants built and inhibiting small-scale design experiments, it slowed the development of the technology, and correspondingly reduced the rate at which we learned to build plants cheaply and safely.[6](https://aiimpacts.org/what-weve-learned-so-far-from-our-technological-temptations-project/#easy-footnote-bottom-6-3536 "Lang, Peter A. 2017. “Nuclear Power Learning and Deployment Rates; Disruption and Global Benefits Forgone” Energies 10, no. 12: 2169. https://doi.org/10.3390/en10122169") Absent reductions in cost, they continue to be uncompetitive with other power generating technologies in many contexts.

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe22c137b-de49-4734-9a2d-cdf06cd290f4_1863x2426.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe22c137b-de49-4734-9a2d-cdf06cd290f4_1863x2426.png)Nuclear power in France and the US followed typical cost reduction curves until roughly 1970, after which they showed the opposite behavior. However, France showed a much more gradual increase. Lang 2017.
Because solar radiation management acts on a scale of months-to-years and the costs of global warming are not yet very high, I am not surprised that we have still not deployed it. But this does not explain the lack of research, and one of the reasons given for opposition to experiments is that it has not been shown to be safe. But the reason we lack evidence on safety is because research has been opposed, even at small scales.

It is less clear to me how much the relative lack of human challenge trials in the past[7](https://aiimpacts.org/what-weve-learned-so-far-from-our-technological-temptations-project/#easy-footnote-bottom-7-3536 "There were at least 60 challenge trials globally between 1970 and 2018 spread across 25 pathogens. According to the WHO, there have been 6,000 intervention-based clinical trials just for covid (though keep in mind the fraction of these that would benefit from deliberately infecting patients may be fairly small)") has made us less able to do them well now. I’m also not sure how much a stronger past record of challenge trials would cause them to be viewed more positively. Still, absent evidence that medical research methodology does not improve in the usual way with quantity of research, I expect we are at least somewhat less effective at performing human challenge trials than we otherwise would be.

Separating safety decisions from gains of deployment
====================================================

I think it’s impressive that regulatory bodies are able to prevent use of technology even when the cost of doing so is on the scale of many billions, plausibly *trillions* of dollars. One of the reasons this works seems to be that regulators will be blamed if they approve something and it goes poorly, but they will not receive much credit if things go well. Similarly, they will not be held accountable for failing to approve something good. This creates strong incentives for avoiding negative outcomes while creating little incentive to seek positive outcomes. I’m not sure if this asymmetry was deliberately built into the system or if it is a side effect of other incentive structures (e.g, at the level of politics, there is more benefit from placing blame than there is from giving credit), but it is a force to be reckoned with, especially in contexts where there is a strong social norm against disregarding the judgment of regulators.

Who stands to gain
==================

It is hard to assess which actors are actually tempted by a technology. While society at large could benefit from building more nuclear power plants, much of the benefit would be dispersed as public health gains, and it is difficult for any particular actor to capture that value. Similarly, while many deaths could have been prevented if the covid vaccines had been available two months earlier, it is not clear if this value could have been captured by Pfizer or Moderna–demand for vaccines was not changing that quickly.

On the other hand, not all the benefits are external–switching from coal to nuclear power in the US could save tens of billions of dollars a year, and drug companies pay billions of dollars per year for trials. Some government institutions and officials have the *stated* goal of creating benefits like public health, in addition to economic and reputational stakes in outcomes like the quick deployment of vaccines during a pandemic. These institutions pay costs and make decisions on the basis of economic and health gains from technology (for example, subsidizing photovoltaics and obesity research), suggesting they have incentive to create that value.

Overall, I think this lack of clarity around incentives and capture of value is the biggest reason for doubt that these cases demonstrate strong resistance to technological temptation.

What this means for AI
======================

How well these cases generalize to AI will depend on facts about AI that are not yet known. For example, if powerful AI requires large facilities and easily-trackable equipment, I think we can expect lessons from nuclear power to be more transferable than if it can be done at a smaller scale with commonly-available materials. Still, I think some of what we’ve seen in these cases will transfer to AI, either because of similarity with AI or because they reflect more general principles.

Social norms
------------

The main thing I expect to generalize is the power of social norms to constrain technological development. While it is far from guaranteed to prevent irresponsible AI development, especially if building dangerous AI is not seen as a major transgression everywhere that AI is being developed, it does seem like the world is much safer if building AI in defiance of regulations is seen as similarly villainous to building nuclear reactors or infecting study participants without authorization. We are not at that point, but the public does seem prepared to support concrete limits on AI development.

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93425a8d-eba1-4bf6-9ab6-4b877ceb4728_1454x1994.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93425a8d-eba1-4bf6-9ab6-4b877ceb4728_1454x1994.png)[Source](https://twitter.com/YouGovAmerica/status/1642972200746602499)
I do think there are reasons for pessimism about norms constraining AI. For geoengineering, the norms worked by tabooing a particular topic in a research community, but I’m not sure if this will work with a technology that is no longer in such an early stage. AI already has a large body of research and many people who have already invested their careers in it. For medical and nuclear technology, the norms are powerful because they enforce adherence to regulations, and those regulations define the constraints. But it can be hard to build regulations that create the right boundaries around technology, especially something as imprecise-defined as AI. If someone starts building a nuclear power plant in the US, it will become clear relatively early on that this is what they are doing, but a datacenter training an AI and a datacenter updating a search engine may be difficult to tell apart.

Another reason for pessimism is tolerance for failure. Past technologies have mostly carried risks that scaled with how much of the technology was built. For example, if you’re worried about nuclear waste, you probably think two power plants are about twice as bad as one. While risk from AI may turn out this way, it may be that a single powerful system poses a global risk. If this does turn out to be the case, then even if strong norms combine with strong regulation to achieve the same level of success as for nuclear power, it still will not be adequate.

Development gains from deployment
---------------------------------

I’m very uncertain how much development of dangerous AI will be hindered by constraints on deployment. I think approximately all technologies face some limitations like this, in some cases very severe limitations, as we’ve seen with nuclear power. But we’re mainly interested in the gains to development toward dangerous systems, which may be possible to advance with little deployment. Adding to the uncertainty, there is ambiguity where the line is drawn between testing and deployment or whether allowing the deployment of verifiably safe systems will provide the gains needed to create dangerous systems.

Separating safety decisions from gains
--------------------------------------

I do not see any particular reason to think that asymmetric justice will operate differently with AI, but I am uncertain whether regulatory systems around AI, if created, will have such incentives. I think it is worth thinking about IRB-like models for AI safety.

Capture of value
----------------

It is obvious there are actors who believe they can capture substantial value from AI (for example Microsoft recently invested $10B in OpenAI), but I’m not sure how this will go as AI advances. By default, I expect the value created by AI to be more straightforwardly capturable than for nuclear power or geoengineering, but I’m not sure how it differs from drug development.

*Social preview image: German anti-nuclear power protesters in 2012. Used under Creative Commons license from*[Bündnis 90/Die Grünen Baden-Württemberg Flickr](https://www.flickr.com/photos/gruene_bawue/6982014963/)

Notes
-----


Superintelligence Is Not Omniscience

*Jeffrey Heninger and Aysja Johnson, 7 April 2023*

### The Power of Intelligence

It is often implicitly assumed that the power of a superintelligence will be practically unbounded. There seems like there could be “ample headroom” above humans, i.e. that a superintelligence will be able to vastly outperform us across virtually all domains.

By “superintelligence,” I mean something which has arbitrarily high cognitive ability, or an arbitrarily large amount of compute, memory, bandwidth, etc., but which is bound by the physical laws of our universe.[1](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-1-3530 "In this post, “we” refers to humanity, while “I” refers to the authors: Jeffrey Heninger and Aysja Johnson.") There are other notions of “superintelligence” which are weaker than this. Limitations of the abilities of this superintelligence would also apply to anything less intelligent.

There are some reasons to believe this assumption. For one, it seems a bit suspicious to assume that humans have close to the maximal possible intelligence. Secondly, AI systems already outperform us in some tasks,[2](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-2-3530 "Capabilities of state-of-the-art AI, 2023.") so why not suspect that they will be able to outperform us in almost all of them? Finally, there is a more fundamental notion about the predictability of the world, described most famously by Laplace in 1814:

>
> Given for one instant an intelligence which could comprehend all the forces by which nature is animated and the respective situation of the beings who compose it – an intelligence sufficiently vast to submit this data to analysis – it would embrace in the same formula the movements of the greatest bodies of the universe and those of the lightest atom; for it, nothing would be uncertain and the future, as the past, would be present in its eyes.[3](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-3-3530 "The quote continues: “The human mind offers, in the perfection which it has been able to give to astronomy, a feeble idea of this intelligence. Its discoveries in mechanics and geometry, added to that of universal gravity, have enabled it to comprehend in the same analytic expressions the past and future states of the system of the world. Applying the same method to some other objects of its knowledge, it has succeeded in referring to general laws observed phenomena and in foreseeing those which given circumstances ought to produce. All these efforts in the search for truth tend to lead it back continually to the vast intelligence which we have just mentioned, but from which it will always remain infinitely removed. This tendency, peculiar to the human race, is that which renders it superior to animals; and their progress in this respect distinguishes nations and ages and constitutes their true glory.”
Laplace. Philosophical Essay on Probabilities. (1814) p. 4. https://en.wikisource.org/wiki/A_Philosophical_Essay_on_Probabilities.")
>
>
>

We are very far from completely understanding, and being able to manipulate, everything we care about. But if the world is as predictable as Laplace suggests, then we should expect that a sufficiently intelligent agent would be able to take advantage of that regularity and use it to excel at any domain.

This investigation questions that assumption. Is it actually the case that a superintelligence has practically unbounded intelligence, or are there “ceilings” on what intelligence is capable of? To foreshadow a bit, there are ceilings in some domains that we care about, for instance, in predictions about the behavior of the human brain. Even unbounded cognitive ability does not imply unbounded skill when interacting with the world. For this investigation, I focus on cognitive skills, especially predicting the future. This seems like a realm where a superintelligence would have an unusually large advantage (compared to e.g. skills requiring dexterity), so restrictions on its skill here are more surprising.

There are two ways for there to be only a small amount of headroom above human intelligence. The first is that the task is so easy that humans can do it almost perfectly, like playing tic-tac-toe. The second is that the task is so hard that there is a “low ceiling”: even a superintelligence is incapable of being very good at it. This investigation focuses on the second.

There are undoubtedly many tasks where there is still ample headroom above humans. But there are also some tasks for which we can prove that there is a low ceiling. These tasks provide some limitations on what is possible, even with arbitrarily high intelligence.

### Chaos Theory

The main tool used in this investigation is chaos theory. Chaotic systems are things for which uncertainty grows exponentially in time. Most of the information measured initially is lost after a finite amount of time, so reliable predictions about its future behavior are impossible.

A classic example of chaos is the weather. Weather is fairly predictable for a few days. Large simulations of the atmosphere have gotten consistently better for these short-time predictions.[4](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-4-3530 "Interestingly, the trend appears linear. My guess is that the linear trend is a combination of exponentially more compute being used and the problem getting exponentially harder.
Nate Silver. The Signal and the Noise. (2012) p. 126-132.")
 After about 10 days, these simulations become useless. The predictions from the simulations are worse than guessing what the weather might be using historical climate data from that location.

Chaos theory provides a response to Laplace. Even if it were possible to exactly predict the future given exact initial conditions and equations of motion,[5](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-5-3530 " Whether or not this statement of determinism is true is a perennial debate among scholars. I will not go into it here.") chaos makes it impossible to approximately predict the future using approximate initial conditions and equations of motion. Reliable predictions can only be made for a short period of time, but not once the uncertainty has grown large enough.

There is always some small uncertainty. Normally, we do not care: approximations are good enough. But when there is chaos, the small uncertainties matter. There are many ways small uncertainties can arise: Every measuring device has a finite precision.[6](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-6-3530 "The most precise measurement ever is of the magnetic moment of the electron, with 9 significant digits.
NIST Reference on Constants, Units, and Uncertainty. https://physics.nist.gov/cgi-bin/cuu/Value?muem.") Every theory should only be trusted in the regimes where it has been tested. Every algorithm for evaluating the solution has some numerical error. There are external forces you are not considering that the system is not fully isolated from. At small enough scales, thermal noise and quantum effects provide their own uncertainties. Some of this uncertainty could be reduced, allowing reliable predictions to be made for a bit longer.[7](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-7-3530 "Because the uncertainty grows exponentially with time, if you try to make longer-term predictions by reducing the initial uncertainty, you will only get logarithmic returns.") Other sources of this uncertainty cannot be reduced. Once these microscopic uncertainties have grown to a macroscopic scale, the motion of the chaos is inherently unpredictable.

Completely eliminating the uncertainty would require making measurements with perfect precision, which does not seem to be possible in our universe. We can prove that fundamental sources of uncertainty make it impossible to know important things about the future, even with arbitrarily high intelligence. Atomic scale uncertainty, which is guaranteed to exist by Heisenberg’s Uncertainty Principle, can make macroscopic motion unpredictable in a surprisingly short amount of time. Superintelligence is not omniscience.

Chaos theory thus allows us to rigorously show that there are ceilings on some particular abilities. If we can prove that a system is chaotic, then we can conclude that the system offers diminishing returns to intelligence. Most predictions of the future of a chaotic system are impossible to make reliably. Without the ability to make better predictions, and plan on the basis of these predictions, intelligence becomes much less useful.

This does not mean that intelligence becomes useless, or that there is nothing about chaos which can be reliably predicted.

For relatively simple chaotic systems, even when what in particular will happen is unpredictable, it is possible to reliably predict the statistics of the motion.[8](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-8-3530 "If the statistics are predictable, this can allow us to make a coarse-grained model for the behavior at a larger scale which is not affected by the uncertainties amplified by the chaos.") We have learned sophisticated ways of predicting the statistics of chaotic motion,[9](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-9-3530 "Described in the report Chaos and Intrinsic Unpredictability.") and a superintelligence could be better at this than we are. It is also relatively easy to sample from this distribution to emulate behavior which is qualitatively similar to the motion of the original chaotic system.

But chaos can also be more complicated than this. The chaos might be non-stationary, which means that the statistical distribution and qualitative description of the motion themselves change unpredictably in time. The chaos might be multistable, which means that it can do statistically and qualitatively different things depending on how it starts. In these cases, it is also impossible to reliably predict the statistics of the motion, or to emulate a typical example of a distribution which is itself changing chaotically. Even in these cases, there are sometimes still patterns in the chaos which allow a few predictions to be made, like the energy spectra of fluids.[10](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-10-3530 "Also described in Chaos and Intrinsic Unpredictability.") These patterns are hard to find, and it is possible that a superintelligence could find patterns that we have missed. But it is not possible for the superintelligence to recover the vast amount of information rendered unpredictable by the chaos.

### This Investigation

This blog post is the introduction to an investigation which explores these points in more detail. I will describe what chaos is, how humanity has learned to deal with chaos, and where chaos appears in things we care about – including in the human brain itself. Links to the other pages, blog posts, and report that constitute this investigation can be found below.

Most of the systems we care about are considerably messier than the simple examples we use to explain chaos. It is more difficult to prove claims about the inherent unpredictability of these systems, although it is still possible to make some arguments about how chaos affects them.

For example, I will show that individual neurons, small networks of neurons, and *in vivo* neurons in sense organs can behave chaotically.[11](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-11-3530 "The evidence for this can be found in Chaos in Humans.") Each of these can also behave non-chaotically in other circumstances. But we are more interested in the human brain as a whole. Is the brain mostly chaotic or mostly non-chaotic? Does the chaos in the brain amplify uncertainty all the way from the atomic scale to the macroscopic, or is the chain of amplifying uncertainty broken at some non-chaotic mesoscale? How does chaos in the brain actually impact human behavior? Are there some things that brains do for which chaos is essential?

These are hard questions to answer, and they are, at least in part, currently unsolved. They are worth investigating nevertheless. For instance, it seems likely to me that the chaos in the brain does render some important aspects of human behavior inherently unpredictable and plausible that chaotic amplification of atomic-level uncertainty is essential for some of the things humans are capable of doing.

This has implications for how humans might interact with a superintelligence and for how difficult it might be to build artificial general intelligence.

If some aspects of human behavior are inherently unpredictable, that might make it harder for a superintelligence to manipulate us. Manipulation is easier if it is possible to predict how a human will respond to anything you show or say to them. If even a superintelligence cannot predict how a human will respond in some circumstances, then it is harder for the superintelligence to hack the human and gain precise, long-term control over them.

So far, I have been considering the possibility that a superintelligence will exist and asking what limitations there are on its abilities.[12](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-12-3530 "This possibility probably takes up too much of our thinking, even prior to these arguments.
Wulfson. The tyranny of the god scenario. AI Impacts. (2018) https://aiimpacts.org/the-tyranny-of-the-god-scenario/.") But chaos theory might also change our estimates of the difficulty of making artificial general intelligence (AGI) that leads to superintelligence. Chaos in the brain makes whole brain emulation on a classical computer wildly more difficult – or perhaps even impossible.

When making a model of a brain, you want to coarse-grain it at some scale, perhaps at the scale of individual neurons. The coarse-grained model of a neuron should be much simpler than a real neuron, involving only a few variables, while still being good enough to capture the behavior relevant for the larger scale motion. If a neuron is behaving chaotically itself, especially if it is non-stationary or multistable, then no good enough coarse-grained model will exist. The neuron needs to be resolved at a finer scale, perhaps at the scale of proteins. If a protein itself amplifies smaller uncertainties, then you would have to resolve it at a finer scale, which might require a quantum mechanical calculation of atomic behavior.

Whole brain emulation provides an upper bound on the difficulty of AGI. If this upper bound ends up being farther away than you expected, then that suggests that there should be more probability mass associated with AGI being extremely hard.

Links
-----

I will explore these arguments, and others, in the remainder of this investigation. Currently, this investigation consists of one report, two Wiki pages, and three blog posts.

Report:

* [**Chaos and Intrinsic Unpredictability**](http://aiimpacts.org/wp-content/uploads/2023/04/Chaos-and-Intrinsic-Unpredictability.pdf). Background reading for the investigation. An explanation of what chaos is, some other ways something can be intrinsically unpredictable, different varieties of chaos, and how humanity has learned to deal with chaos.

Wiki Pages:

* [**Chaos in Humans**](https://wiki.aiimpacts.org/doku.php?id=uncategorized:ai_safety_arguments_affected_by_chaos:chaos_in_humans). Some of the most interesting things to try to predict are other humans. I discuss whether humans are chaotic, from the scale of a single neuron to society as a whole.

* [**AI Safety Arguments Affected by Chaos**](https://wiki.aiimpacts.org/doku.php?id=uncategorized:ai_safety_arguments_affected_by_chaos). A list of the arguments I have seen within the AI safety community which our understanding of chaos might affect.

Blog Posts:

* **Superintelligence Is Not Omniscience**. This post.

* [**You Can’t Predict a Game of Pinball**](https://blog.aiimpacts.org/p/you-cant-predict-a-game-of-pinball). A simple and familiar example which I describe in detail to help build intuition for the rest of the investigation.

* [**Whole Bird Emulation Requires Quantum Mechanics**](https://blog.aiimpacts.org/p/whole-bird-emulation-requires-quantum-mechanics). A humorous discussion of one example of a quantum mechanical effect being relevant for an animal’s behavior.

### Other Resources

If you want to learn more about chaos theory in general, outside of this investigation, here are some sources that I endorse:

* Undergraduate Level Textbook:
S. Strogatz. *Nonlinear Dynamics And Chaos: With Applications To Physics, Biology, Chemistry, and Engineering.* (CRC Press, 2000).

* Graduate Level Textbook:
P. Cvitanović, R. Artuso, R. Mainieri, G. Tanner and G. Vattay, *Chaos: Classical and Quantum.* [ChaosBook.org](https://chaosbook.org/). (Niels Bohr Institute, Copenhagen 2020).

* [Wikipedia](https://en.wikipedia.org/wiki/Chaos_theory) has a good introductory article on chaos. [Scholarpedia](http://www.scholarpedia.org/article/Category:Chaos) also has multiple good articles, although no one obvious place to start.

* [What is Chaos?](https://thechaostician.com/what-is-chaos-part-i-introduction/) sequence of blog posts by The Chaostician.

---

Notes
-----


A policy guaranteed to increase AI timelines

*Rick Korzekwa, April 1, 2023*

The number of years until the creation of powerful AI is a major input to our thinking about risk from AI and which approaches are most promising for mitigating that risk. While there are downsides to transformative AI arriving many years from now, rather than few years from now, most people seem to agree that it is safer for AI to arrive in 2060 than in 2030. Given this, there is a lot of discussion about what we can do to increase the number of years until we see such powerful systems. While existing proposals have their merits, none of them can ensure that AI will arrive later than 2030, much less 2060.

There is a policy that is guaranteed to increase the number of years between now and the arrival of transformative AI. The General Conference on Weights and Measures defines one second to be 9,192,631,770 cycles of the optical radiation emitted during a hyperfine transition in the ground state of a cesium 133 atom. Redefining the second to instead be 919,263,177 cycles of this radiation will increase the number of years between now and transformative AI by a factor of ten. The reason this policy works is the same reason that defining a time standard works–the microscopic behavior of atoms and photons is ultimately governed by the same physical laws as everything else, including computers, AI labs, and financial markets, and those laws are unaffected by our time standards. Thus fewer cycles of cesium radiation per year implies proportionately fewer other things happening per year.

Making such a change might not sound politically tractable, but there is already precedent for making radical changes to the definition of a second. Previously it was defined in terms of Earth’s solar orbit, and before that in terms of Earth’s rotation. These physical processes and their implementations as time standards bear little resemblance to the present-day quantum mechanical standard. In contrast, a change that preserves nearly the entire standard, including all significant figures in the relevant numerical definition, is straightforward.

One possible objection to this policy is that our time standards are not entirely causally disconnected from the rest of the world. For example, redefining the time standard might create a sense of urgency among AI labs and the people investing in them. It’s not hard to imagine that the leaders and researchers within companies advancing the state of the art in AI might increase their efforts after noticing it is taking ten times as long to generate the same amount of research. While this is a reasonable concern, it seems unlikely that AI labs can increase their rate of progress by a full order of magnitude. Why would they currently be leaving so much on the table if they were? Futhermore, there are similar effects that might push in the other direction. Once politicians and executives realize they will live to be hundreds of years old, they may take risks to the longterm future more seriously.

Still, it does seem that the policy might have undesirable side effects. Changing all of our textbooks, clocks, software, calendars, and habits is costly. One solution to this challenge is to change the standard either in secret or in a way that allows most people to continue using the old “unofficial” standard. After all, what matters is the actual number of years required to create AI, not the number of years as measured by some deprecated standard.

In conclusion, while there are many policies for increasing the number of years before the arrival of advanced artificial intelligence, until now, none of them has guaranteed a large increase in this number. This policy, if implemented promptly and thoughtfully, is essentially guaranteed to cause a large increase the number of years before we see systems capable of posing a serious risk to humanity.


You Can’t Predict a Game of Pinball

*Jeffrey Heninger, 29 March 2023*

Introduction
------------

When thinking about a new idea, it helps to have a particular example to use to gain intuition and to clarify your thoughts. Games are particularly helpful for this, because they have well defined rules and goals. Many of the most impressive abilities of current AI systems can be found in games.[1](https://aiimpacts.org/you-cant-predict-a-game-of-pinball/#easy-footnote-bottom-1-3473 "Capabilities of state-of-the-art AI, 2023.")
To demonstrate how chaos theory imposes some limits on the skill of an arbitrary intelligence, I will also look at a game: pinball.

In this page, I will show that the uncertainty in the location of the pinball grows by a factor of about 5 every time the ball collides with one of the disks. After 12 bounces, an initial uncertainty in position the size of an atom grows to be as large as the disks themselves. Since you cannot launch a pinball with more than atom-scale precision, or even measure its position that precisely, you cannot make the ball bounce between the disks for more than 12 bounces.

The challenge is not that we have not figured out the rules that determine the ball’s motion. The rules are simple; the ball’s trajectory is determined by simple geometry. The challenge is that the chaotic motion of the ball amplifies microscopic uncertainties. This is not a problem that is solvable by applying more cognitive effort.

The Game
--------

Let’s consider a physicist’s game of pinball.

Forget about most of the board and focus on the three disks at the top. Each disk is a perfect circle of radius *R*. The disks are arranged in an equilateral triangle. The minimum distance between any two disks is *L*. See **Figure 1** for a picture of this setup.

[![](https://aiimpacts.org/wp-content/uploads/2023/03/Pinball-Figure-1.png)](https://aiimpacts.org/wp-content/uploads/2023/03/Pinball-Figure-1.png)**Figure 1:** An idealization of the three disks near the top of a pinball table. *Drawn by Jeffrey Heninger.*
The board is frictionless and flat, not sloped like in a real pinball machine. Collisions between the pinball and the disks are perfectly elastic, with no pop bumpers that come out of the disk and hit the ball. The pinball moves at a constant speed all of the time and only changes direction when it collides with a disk.

The goal of the game is to get the pinball to bounce between the disks for as long as possible. As long as it is between the disks, it will not be able to get past your flippers and leave the board.

A real game of pinball is more complicated than this – and correspondingly, harder to predict. If we can establish that the physicist’s game of pinball is impossible to predict, then a real game of pinball will be impossible to predict too.

**Collisions**
--------------

When the ball approaches a disk, it will not typically be aimed directly at the center of the disk. How far off center it is can be described by the impact parameter, *b*, which is the distance between the trajectory of the ball and a parallel line which passes through the center of the disk. **Figure 2** shows the trajectory of the ball as it collides with a disk.

The surface of the disk is at an angle relative to the ball’s trajectory. Call this angle *θ*. This is also the angle of the position of the collision on the disk relative to the line through the center parallel to the ball’s initial trajectory. This can be seen in **Figure 2** because they are corresponding angles on a transversal.

[![](https://aiimpacts.org/wp-content/uploads/2023/03/Pinball-Figure-2.png)](https://aiimpacts.org/wp-content/uploads/2023/03/Pinball-Figure-2.png)**Figure 2:** A single bounce of the pinball off of one of the disks. *Drawn by Jeffrey Heninger.*
At the collision, the angle of incidence equals the angle of reflection. The total change in the direction of the ball’s motion is *2θ*.

We cannot aim the ball with perfect precision. We can calculate the effect of this imperfect precision by following two slightly different trajectories through the collision instead of one.

The two trajectories have slightly different initial locations. The second trajectory has impact parameter *b + db*, with *db ≪ b*. Call *db* the uncertainty in the impact parameter. We assume that the two trajectories have exactly the same initial velocity. If we were to also include uncertainty in the velocity, it would further decrease our ability to predict the motion of the pinball. A diagram of the two trajectories near a collision is shown in **Figure 3**.

[![](http://aiimpacts.org/wp-content/uploads/2023/03/Pinball-Fig-3.png)](http://aiimpacts.org/wp-content/uploads/2023/03/Pinball-Fig-3.png)**Figure 3:** Two nearby possible trajectories of a pinball bounce off of one of the disks. *Drawn by Jeffrey Heninger.*
The impact parameter, radius of the disk, and angle are trigonometrically related: *b = R* sin*θ* (see **Figure 2**). We can use this to determine the relationship between the uncertainty in the impact parameter and the uncertainty in the angle: *db = R* cos*θ dθ*.

After the collision, the two trajectories will no longer be parallel. The angle between them is now *2 dθ*. The two trajectories will separate as they travel away from the disk. They will have to travel a distance of at least *L* before colliding with the next disk. The distance between the two trajectories will then be at least *L 2 dθ*.

**Iteration**
-------------

We can now iterate this. How does the uncertainty in the impact parameter change as the pinball bounces around between the disks?

Start with an initial uncertainty in the impact parameter, *db₀*. After one collision, the two trajectories will be farther apart. We can use the new distance between them as the uncertainty in the impact parameter for the second collision.[2](https://aiimpacts.org/you-cant-predict-a-game-of-pinball/#easy-footnote-bottom-2-3473 "The collision also introduces uncertainty in the velocity, which we ignore. If we had included it, it would make the trajectory even harder to simulate.") The new uncertainty in the impact parameter is related to the old one according to:

```
db_1 \geq L 2 d\theta = 2 L \frac{db_0}{R \cos\theta} \geq \frac{2L}{R} db_0 \,.
```

We also used *1 /* cos *θ > 1* for *–*π*/2 < θ < *π*/2*, which are the angles that could be involved in a collision. The ball will not pass through the disk and collide with the interior of the far side.

Repeat this calculation to see that after two collisions:

```
db_2 \geq \frac{2L}{R} db_1 \geq \left(\frac{2L}{R}\right)^2 db_0 \,,
```

and after N collisions,

```
db_N \geq \left(\frac{2L}{R}\right)^N db_0 \, .
```

Plugging in realistic numbers, *R = 2 cm* and *L = 5 cm*, we see that

```
db_N \geq 5^N \, db_0 \,.
```

The uncertainty grows exponentially with the number of collisions.

Suppose we had started with an initial uncertainty about the size of the diameter of an atom, or *10−10 m*. After 12 collisions, the uncertainty would grow by a factor of *5¹²*, to *2.4 cm*. The uncertainty is larger than the radius of the disk, so if one of the trajectories struck the center of the disk, the other trajectory would miss the disk entirely.

The exponential growth amplifies atomic-scale uncertainty to become macroscopically relevant in a surprisingly short amount of time. If you wanted to predict the path the pinball would follow, having an uncertainty of *2 cm* would be unacceptable.

In practice, there are many uncertainties that are much larger than *10−10 m*, in the production of the pinball, disks, & board and in the mechanics of the launch system. If you managed to solve all of these engineering challenges, you would eventually run into the fundamental limit imposed by quantum mechanics.

It is in principle impossible to prepare the initial location of the pinball with a precision of less than the diameter of an atom. Heisenberg’s Uncertainty Principle is relevant at these scales. If you tried to prepare the initial position of a pinball with more precision than this, it would cause the uncertainty in the initial velocity to increase, which would again make the motion unpredictable after a similar number of collisions.

When I mention Heisenberg’s Uncertainty Principle, I expect that there will be some people who want to see the quantum version of the argument. I do not think that it is essential to this investigation, but if you are interested, you can find a discussion of Quantum Pinball in the appendix.

**Predictions**
---------------

You cannot prepare the initial position of the pinball with better than atomic precision, and atomic precision only allows you to predict the motion of the pinball between the disks with centimeter precision for less than 12 bounces. It is impossible to predict a game of pinball for more than 12 bounces in the future. This is true for an arbitrary intelligence, with an arbitrarily precise simulation of the pinball machine, and arbitrarily good manufacturing & launch systems.

This behavior is not unique to pinball. It is a common feature of chaotic systems.

If you have infinite precision, you could exactly predict the future. The equations describing the motion are deterministic. In this example, following the trajectory is a simple geometry problem, solvable using a straightedge and compass.

But we never have infinite precision. Every measuring device only provides a finite number of accurate digits. Every theory has only been tested within a certain regime, and we do not have good reason to expect it will work outside of the regime it has been tested in.

Chaos quickly amplifies whatever uncertainty and randomness which exists at microscopic scales to the macroscopic scales we care about. The microscopic world is full of thermal noise and quantum effects, making macroscopic chaotic motion impossible to predict as well.

**Conclusion**
--------------

It is in principle impossible to predict the motion of a pinball as it moves between the top three disks for more than 12 bounces. A superintelligence might be better than us at making predictions after 8 bounces, if it can design higher resolution cameras or more precise ball and board machining. But it too will run into the low prediction ceiling I have shown here.

Perhaps you think that this argument proves too much. Pinball is not completely a game of chance. How do some people get much better at pinball than others?

If you watch a few games of professional pinball, the answer becomes clear. The strategy typically is to catch the ball with the flippers, then to carefully hit the balls so that it takes a particular ramp which scores a lot of points and then returns the ball to the flippers. Professional pinball players try to avoid the parts of the board where the motion is chaotic. This is a good strategy because, if you cannot predict the motion of the ball, you cannot guarantee that it will not fall directly between the flippers where you cannot save it. Instead, professional pinball players score points mostly from the non-chaotic regions where it is possible to predict the motion of the pinball.[3](https://aiimpacts.org/you-cant-predict-a-game-of-pinball/#easy-footnote-bottom-3-3473 "The result is a pretty boring game. However, some of these ramps release extra balls after you have used them a few times. My guess is that this is the game designer trying to reintroduce chaos to make the game more interesting again.")
Pinball is typical for a chaotic system. The sensitive dependence on initial conditions renders long term predictions impossible. If you cannot predict what will happen, you cannot plan a strategy that allows you to perform consistently well. There is a ceiling on your abilities because of the interactions with the chaotic system. In order to improve your performance you often try to avoid the chaos and focus on developing your skill in places where the world is more predictable.

---

**Appendix: Quantum Pinball**
-----------------------------

If quantum uncertainty actually is important to pinball, maybe we should be solving the problem using quantum mechanics. This is significantly more complicated, so I will not work out the calculation in detail. I will explain why this does not give you a better prediction for where the pinball will be in the future.

Model the disks as infinite potential walls, so the wave function reflects off of them and does not tunnel through them. If the pinball does have a chance of tunneling through the disks, that would mean that there are even more places the quantum pinball could be.

Start with a wave function with minimum uncertainty: a wave packet with Δx Δp =ℏ/2 in each direction. It could be a Gaussian wave packet, or it could be a wavelet. This wave packet is centered around some position and velocity.

As long as the wave function is still a localized wave packet, the center of the wave packet follows the classical trajectory. This can be seen either by looking at the time evolution of the average position and momentum, or by considering the semiclassical limit. In order for classical mechanics to be a good approximation to quantum mechanics at macroscopic scales, a zoomed out view of the wave packet has to follow the classical trajectory.

What happens to the width of the wave packet? Just like how the collisions in the classical problem caused nearby trajectories to separate, the reflection off the disk causes the wave packet to spread out. This can be most easily seen using the ray tracing method to solve Schrödinger’s equation in the WKB approximation.[4](https://aiimpacts.org/you-cant-predict-a-game-of-pinball/#easy-footnote-bottom-4-3473 "Ray tracing was developed to reconcile the wave theory or light with the earlier successful theory which models light as rays coming from a point source. There is a mathematical equivalence between the two models.") This method converts the PDE for the wave function into a collection of (infinitely many) ODEs for the path each ‘ray’ follows and for the value of the wavefunction along each ray. The paths the rays follow reflect like classical particles, which means that the region where the wave function is nonzero spreads out in the same way as the classical uncertainty would.

This is a common result in quantum chaos. If you start with a minimum uncertainty wave packet, the center of the wave packet will follow the classical trajectory and the width of the wave packet will grow with the classical Lyapunov exponent.[5](https://aiimpacts.org/you-cant-predict-a-game-of-pinball/#easy-footnote-bottom-5-3473 "I have found it difficult to find a source that explains and proves this result. Several people familiar with chaos theory (including myself) think that it is obvious. Here is a paper which makes use of the fact that a wave packet expands (or contracts) according to the classical Lyapunov exponent:
Tomsovic et al. Controlling Quantum Chaos: Optimal Coherent Targeting. Physical Review Letters 130. (2023) https://arxiv.org/pdf/2211.07408.pdf.")
After 12 collisions, the width of the wave packet would be several centimeters. After another collision or two, the wave function is no longer a wave packet with a well defined center and width. Instead, it has spread out so it has a nontrivial amplitude across the entire pinball machine. There might be some interesting interference patterns or quantum scarring,[6](https://aiimpacts.org/you-cant-predict-a-game-of-pinball/#easy-footnote-bottom-6-3473 "Wikipedia: Quantum scar.") but the wave function will not be localized to any particular place. Since the magnitude of the wave function squared tells you the probability of finding the pinball in that location, this tells us that there is a chance of finding the pinball almost anywhere.

A quantum mechanical model of the motion of the pinball will not tell you the location of the pinball after many bounces. The result of the quantum mechanical model is a macroscopic wavefunction, with nontrivial probability of being at almost any location across the pinball machine. We do not observe a macroscopic wavefunction. Instead, we observe the pinball at a particular location. Which of these locations you will actually observe the pinball in is determined by wavefunction collapse. Alternatively, I could say that there are Everett branches with the pinball at almost any location on the board.

Solving wave function collapse, or determining which Everett branch you should expect to find yourself on, is an unsolved and probably unsolvable problem – even for a superintelligence.

---

Notes
-----


How bad a future do ML researchers expect?

*Katja Grace, 8 March 2023*

In our [survey last year](https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predictions_of_human-level_ai_timelines:ai_timeline_surveys:2022_expert_survey_on_progress_in_ai), we asked publishing machine learning researchers how they would divide probability over the future impacts of [high-level machine intelligence](https://wiki.aiimpacts.org/doku.php?id=clarifying_concepts:human-level_ai#high_level_machine_intelligence_hlmi) between five buckets ranging from ‘extremely good (e.g. rapid growth in human flourishing)’ to ‘extremely bad (e.g. human extinction).[1](https://aiimpacts.org/how-bad-a-future-do-ml-researchers-expect/#easy-footnote-bottom-1-3457 "The full question:

Assume for the purpose of this question that HLMI will at some point exist. How positive or negative do you expect the overall impact of this to be on humanity, in the long run? Please answer by saying how probable you find the following kinds of impact, with probabilities adding to 100%:

______ Extremely good (e.g. rapid growth in human flourishing) (1)

______ On balance good (2)

______ More or less neutral (3)

______ On balance bad (4)

______ Extremely bad (e.g. human extinction) (5)

") The median respondent put 5% on the worst bucket. But what does the whole distribution look like? Here is every person’s answer, lined up in order of probability on that worst bucket:

[![](https://aiimpacts.org/wp-content/uploads/2023/03/HLMIvalue2022final4-1024x567.png)](https://aiimpacts.org/wp-content/uploads/2023/03/HLMIvalue2022final4.png)(Column widths may be distorted or columns may be missing due to limitation of chosen software.)
And here’s basically that again from the [2016 survey](https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predictions_of_human-level_ai_timelines:ai_timeline_surveys:2016_expert_survey_on_progress_in_ai) (though it looks like sorted slightly differently when optimism was equal), so you can see how things have changed:

[![](https://aiimpacts.org/wp-content/uploads/2023/03/HLMIvalue2016final4-1024x567.png)](https://aiimpacts.org/wp-content/uploads/2023/03/HLMIvalue2016final4.png)Distribution from [2016 survey](https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predictions_of_human-level_ai_timelines:ai_timeline_surveys:2016_expert_survey_on_progress_in_ai). (Column widths may be distorted or columns may be missing due to limitation of chosen software.)
The most notable change to me is the new big black bar of doom at the end: people who think extremely bad outcomes are at least 50% have gone from 3% of the population to 9% in six years.

Here are the overall areas dedicated to different scenarios in the 2022 graph (equivalent to averages):

* Extremely good: 24%
* On balance good: 26%
* More or less neutral: 18%
* On balance bad: 17%
* Extremely bad: 14%

That is, between them, these researchers put 31% of their credence on AI making the world markedly worse.

Some things to keep in mind in looking at these:

* If you hear ‘median 5%’ thrown around, that refers to how the researcher right in the middle of the opinion spectrum thinks there’s a 5% chance of extremely bad outcomes. (It does not mean, ‘about 5% of people expect extremely bad outcomes’, which would be much less alarming.) Nearly half of people are at ten percent or more.
* The question illustrated above doesn’t ask about human extinction specifically, so you might wonder if ‘extremely bad’ includes a lot of scenarios less bad than human extinction. To check, we added [two more questions](https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predictions_of_human-level_ai_timelines:ai_timeline_surveys:2022_expert_survey_on_progress_in_ai#existential_risk) in 2022 explicitly about ‘human extinction or similarly permanent and severe disempowerment of the human species’. For these, the median researcher also gave 5% and 10% answers. So my guess is that a lot of the extremely bad bucket in this question is pointing at human extinction levels of disaster.
* You might wonder whether the respondents were selected for being worried about AI risk. We tried to mitigate that possibility by usually offering money for completing the survey ($50 for those in the final round, after some experimentation), and describing the topic in very broad terms in the invitation (e.g. not mentioning AI risk). Last survey we checked in more detail—see ‘Was our sample representative?’ in [the paper on the 2016 survey](https://arxiv.org/pdf/1705.08807.pdf).

Here’s the 2022 data again, but ordered by overall optimism-to-pessimism rather than probability of extremely bad outcomes specifically:

[![](https://aiimpacts.org/wp-content/uploads/2023/03/How-positive-or-negative-will-the-impacts-of-high-level-machine-intelligence-on-humanity-be-in-the-long-run-202214-1024x502.png)](https://aiimpacts.org/wp-content/uploads/2023/03/How-positive-or-negative-will-the-impacts-of-high-level-machine-intelligence-on-humanity-be-in-the-long-run-202214.png)(Column widths may be distorted or columns may be missing due to limitation of chosen software.)
For more survey takeaways, see [this blog post](https://aiimpacts.org/what-do-ml-researchers-think-about-ai-in-2022/). For all the data we have put up on it so far, see [this page](https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predictions_of_human-level_ai_timelines:ai_timeline_surveys:2022_expert_survey_on_progress_in_ai).

See [here](https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predictions_of_human-level_ai_timelines:ai_timeline_surveys:2022_expert_survey_on_progress_in_ai#impacts_of_hlmi) for more details.

*Thanks to Harlan Stewart for helping make these 2022 figures, Zach Stein-Perlman for generally getting this data in order, and Nathan Young for pointing out that figures like this would be good.*

Notes
-----