2017 trend in the cost of computing

The cheapest hardware prices (for single precision FLOPS/$) appear to be falling by around an order of magnitude every 10-16 years. This rate is slower than the trend of FLOPS/$ observed over the past quarter century, which was an order of magnitude every 4 years. There is no particular sign of slowing between 2011 and 2017.

Support
-------

### Background

Computing power available per dollar [has increased](http://aiimpacts.org/trends-in-the-cost-of-computing/) fairly evenly by a factor of ten roughly every four years in the last quarter of a century (a phenomenon sometimes called ‘[price-performance Moore’s Law](http://aiimpacts.org/ai-risk-terminology/)‘). Because this trend is important and regular, it is useful in predictions. For instance, it is often used to [determine when the hardware for an AI mind might become cheap](http://aiimpacts.org/brain-performance-in-flops/). This means that a primary way such predictions might err is if this trend in computing prices were to leave its long run trajectory. This must presumably happen eventually, and has purportedly happened with other exponential trends in information technology recently.[1](https://aiimpacts.org/recent-trend-in-the-cost-of-computing/#easy-footnote-bottom-1-756 "For instance processor clock speed and traditional Moore’s law")

This page outlines our assessment of whether the long run trend is on track very recently, as of late 2017. This differs from assessing the long run trend (as we do [here](http://aiimpacts.org/trends-in-the-cost-of-computing/)) in that it requires recent and relatively precise data. Data that may be off by one order of magnitude is still useful when assessing a long run trend that grows by many orders of magnitude. But if we are judging whether the last five years of that trend are on track, it is important to have more accurate figures.

### Sources of evidence

We sought public data on computing performance, initial price, and date of release for different pieces of computing hardware. We tried to cover different types of computing hardware, and to prioritize finding large, consistent datasets using comparable metrics, rather than one-off measurements. We searched for computing performance measured using the [Linpack benchmark](https://en.wikipedia.org/wiki/LINPACK_benchmarks), or something similar.

We ran into many difficulties finding consistently measured performance in FLOPS for different machines, as well as prices for those same machines. What data we could find used a variety of different benchmarks. Sometimes performance was reported as ‘FLOPS’ without explanation. Twice the ‘same’ benchmarks turned out to give substantially different answers at different times, at least for some machines, apparently due to the benchmarks being updated. Performance figures cited often refer to ‘theoretical peak performance’, which is calculated from the computer’s specifications, rather than measured, and is higher than actual performance.

Prices are also complicated, because each machine can have many sellers, and each price fluctuates over time. We tried to use the release price, the manufacturer’s ‘recommended customer price’, or similar where possible. However, many machines don’t seem to have readily available release prices.

These difficulties led to many errors and confusions, such that progress required running calculations, getting unbelievable results, and searching for an error that could have made them unbelievable. This process is likely to leave remaining errors at the end, and those errors are likely to be biased toward giving results that we find believable. We do not know of a good remedy for this, aside from welcoming further error-checking, and giving this warning.

### Evidence

[GPUs](https://en.wikipedia.org/wiki/Graphics_processing_unit) appear to be substantially cheaper than [CPUs](https://en.wikipedia.org/wiki/Central_processing_unit), cloud computing (including TPUs), or supercomputers.[2](https://aiimpacts.org/recent-trend-in-the-cost-of-computing/#easy-footnote-bottom-2-756 "See ‘Other sources…’ below.") Since GPUs alone are at the frontier of price performance, we focus on them. We have two useful datasets: one of theoretical peak performance, gathered from Wikipedia, and one of empirical performance, from [Passmark](https://www.passmark.com/).

#### GPU theoretical peak performance

We collected data from several Wikipedia pages, supplemented with other sources for some dates and prices.[3](https://aiimpacts.org/recent-trend-in-the-cost-of-computing/#easy-footnote-bottom-3-756 "Wikipedia pages: Xeon Phi, List of Nvidia Graphics Processing Units, List of AMD Graphics Processing Units

Other sources are visible in the last column of our dataset (see ‘Wikipedia GeForce, Radeon, Phi simplified’ sheet)") We think all of the performance numbers are theoretical peak performance, generally calculated from specifications given by the developer, but we have not checked Wikipedia’s sources or calculations thoroughly. Our impression is that the prices given are recommended prices at launch, by the developers of the hardware, though again we have only checked a few of them.

We look at Nvidia and AMD GPUs and Xeon Phi processors here because they are the machines for which we could find data on Wikipedia easily. However, Nvidia and AMD are the leading producers of GPUs, so this should cover the popular machines. We excluded many machines because they did not have prices listed.

Figure 1 shows performance (single precision) over time for processors for which we could find all of the requisite data.

The recent rate of progress in this figure looks like somewhere between half an order of magnitude in the past eight years and an order of magnitude in the past ten, for an order of magnitude about every 10-16 years. We don’t think the figure shows particular slowing down—the most cost-effective hardware has not improved in almost a year, but that is usual in the rest of the figure.

[![](http://aiimpacts.org/wp-content/uploads/2017/10/chart-42.png)](http://aiimpacts.org/wp-content/uploads/2017/10/chart-42.png)**Figure 1**

We also collected double precision performance figures for these machines, but the machines do not appear to be optimized for double precision performance,[4](https://aiimpacts.org/recent-trend-in-the-cost-of-computing/#easy-footnote-bottom-4-756 "Performance is usually measured either as ‘single precision’ or ‘double precision’. Roughly, the latter involves computations with numbers that are twice as large, in the sense of requiring twice as many bits of information to store them (see here and here). The trend of double precision performance from this dataset saw no progress in five years, and in fact became markedly worse (see figure below).

Our understanding is that GPUs are generally not optimized for double precision performance, because it is less relevant to the applications that they are used for. Many processors we looked at either purportedly did not do double precision operations at all, or were up to thirty-two times slower at them. So our guess is that we are seeing movement toward sacrificing double precision performance for single precision performance, rather than a slowdown in double precision performance where it is intended. So we disregard this data in understanding the overall trend.

") so we focus on single precision.

Peak theoretical performance is generally higher than actual performance, but our impression is that this should be by a roughly constant factor across time, so not make a difference to the trend.

#### GPU Passmark value

[Passmark](https://www.cpubenchmark.net/) maintains a collection of benchmark results online, for both CPUs and GPUs. They also collect prices, and calculate price for performance (though it was not clear to us on brief inspection where their prices come from). Their performance measure is from their own benchmark, which we do not know a lot about. This makes their absolute prices hard to compare to others using more common measures, but the trend in progress should be more comparable.

We used [archive.org](http://archive.org) to collect old versions of Passmark’s page of the [most cost-effective](https://www.videocardbenchmark.net/gpu_value.html) GPUs available, to get a history of price for passmark performance. The prices are from the time of the archive, not necessarily from when the hardware was new. That is, if we collected all of the results on the page on January 1, 2013, it might contain hardware that was built in 2010 and has maybe come down in price due to being old. You might wonder whether this means we are just getting a lot of really cheap old hardware with hardly any performance, which might be bad in other ways and so not represent a realistic price of hardware. This is possible, however given that people show interest in this (for instance, Passmark keep these records) it would be surprising to us if this metric mostly caught useless hardware.

[![](http://aiimpacts.org/wp-content/uploads/2017/11/Scrape-archiveorg-gpu-figure-2.png)](http://aiimpacts.org/wp-content/uploads/2017/11/Scrape-archiveorg-gpu-figure-2.png)**Figure 2:** Top GPU passmark performance per dollar scores over time.

[![](http://aiimpacts.org/wp-content/uploads/2017/11/Scrape-archiveorg-gpu-figure-1.png)](http://aiimpacts.org/wp-content/uploads/2017/11/Scrape-archiveorg-gpu-figure-1.png)**Figure 3:** The same data in Figure 1, showing progress for different percentiles.

[![](http://aiimpacts.org/wp-content/uploads/2017/11/Scrape-archiveorg-gpu-figure-3.png)](http://aiimpacts.org/wp-content/uploads/2017/11/Scrape-archiveorg-gpu-figure-3.png)**Figure 4:** Figure 3, with a log y axis.

We are broadly interested in the cheapest hardware available, but we probably don’t want to look at the very cheapest in data like this, because it seems likely to be due to error or other meaningless exploitation of the particular metric.[5](https://aiimpacts.org/recent-trend-in-the-cost-of-computing/#easy-footnote-bottom-5-756 "One reason it seems unlikely to us that the very cheapest numbers are ‘real’ is because they are quite noisy (see figure 2)—high numbers appear and disappear, whereas if we had made real progress in hardware technology, we might expect cheap hardware available in 2012 to also be available in 2013. Another reason is that in similar CPU data that we scraped from Passmark, there is a clear dropoff in high numbers at one point, which we think corresponds to slightly changing the benchmark to better measure those machines.") The 95th percentile machines (out of the top 50) appear to be relatively stable, so are probably close to the cheapest hardware without catching too many outliers. For this reason, we take them as a proxy for the cheapest hardware.

Figure 4 shows the 95th percentile fits an exponential trendline quite well, with a doubling time of 3.7 years, for an order of magnitude every 12 years. This has been fairly consistent, and shows no sign of slowing by early 2017. This supports the 10-16 year time we estimated from the Wikipedia theoretical performance above.

#### Other sources we investigated, but did not find relevant

* The **[Wikipedia page on FLOPS](https://web.archive.org/web/20171025210410/https://en.wikipedia.org/wiki/FLOPS)** contains a history of GFLOPS over time. The recent datapoints appear to overlap with the theoretical performance figures we have already.
* Google has developed **[Tensor Processing Units](https://en.wikipedia.org/wiki/Tensor_processing_unit)** (TPUs) that specialize in computation for machine learning. Based on information from Google, [we estimate](http://aiimpacts.org/2018-price-of-performance-by-tensor-processing-units/) that they perform around 1.05 GFLOPS/$.
* In 2015, **cloud computing** appeared to be around a hundred times more expensive than other forms of computing.[6](https://aiimpacts.org/recent-trend-in-the-cost-of-computing/#easy-footnote-bottom-6-756 "See Current FLOPS Prices, or 2015 FLOPS prices.") Since then the price appears to have roughly halved.[7](https://aiimpacts.org/recent-trend-in-the-cost-of-computing/#easy-footnote-bottom-7-756 "As of October 5th, 2017, renting a c4.8xlarge instance costs $0.621 per hour (if you purchase it for three years, and pay upfront) When we last checked this in around April 2015, the price for the same arrangement was $1.17 / hour.") So cloud computing is not a competitive way to buy FLOPS all else equal, and the price of FLOPS may be a small influence on the cloud-computing price trend, making the trend less relevant to this investigation.
* Top **supercomputers** perform at around $3/GFLOPS, so they do not appear to be on the forefront of cheap performance. See [Price performance trend in top supercomputers](http://aiimpacts.org/price-performance-trend-in-top-supercomputers/) for more details.
* **Geekbench** has empirical performance numbers for many systems, but their latest version does not seem to have anything for GPUs. We looked at a small number of popular CPUs on Geekbench from the past five years, and found the cheapest to be around $0.71/GFLOPS. However there appear to be 5x disparities between different versions of Geekbench, which makes it less useful for fine-grained estimates.

Conclusions
-----------

We have seen that the theoretical peak single-precision performance of GPUs is improving at about an order of magnitude every 10-16 years. And that the Passmark performance/$ trend is improving by an order of magnitude every 12 years. These are slower than [the long run price-performance trends](http://aiimpacts.org/trends-in-the-cost-of-computing/) of an order of magnitude every eight years (75 year trend) or four years (25 year trend).

The longer run trends are based on a slightly different set of measures, which might explain a difference in rates of progress.

Within these datasets the pace of progress does not appear to be slower in recent years relative to earlier ones.


Price-performance trend in top supercomputers

A top supercomputer can perform a GFLOP for around $3, in 2017.

The price of performance in top supercomputers continues to fall, as of 2016.

Details
-------

[TOP500.org](https://www.top500.org/lists/2017/06/) maintains a list of top supercomputers and their performance on the Linpack benchmark. The figure below is based on empirical performance figures (‘Rmax’) from [Top500](https://www.top500.org/lists/2017/06/) and price figures collected from a variety of less credible sources, for nine of the ten highest performing supercomputers (we couldn’t find a price for the tenth). Our data and sources are [here](https://docs.google.com/spreadsheets/d/1nV6djZI7csDv_ewElbNKiQZVl36ViGyUEy_MSqk2krI/edit?usp=sharing).

Sunway Teihu Light performs the cheapest GFLOPS, at $2.94/GFLOPS. This is around one hundred times more expensive than peak theoretical performance of certain GPUs, but we do not know why there is such a difference (peak performance is generally higher than actual performance, but by closer to a factor of two).

There appears to be a downward trend in price, but it is not consistent, and with so few data points its slope is ambiguous. The best price for performance roughly halved in the last 4-5 years, for a 10x drop in 13-17 years. The K computer in 2011 was much more expensive, but appears to have been substantially more expensive than earlier computers.

####

[![](http://aiimpacts.org/wp-content/uploads/2017/11/chart-54.png)](http://aiimpacts.org/wp-content/uploads/2017/11/chart-54.png)


Computing hardware performance data collections

This is a list of public datasets that we know of containing either measured or theoretical performance numbers for computer processors.

List
----

1. **[Top 500](https://www.top500.org/lists/2017/06/)** maintains a list of the top 500 supercomputers, updated every six months. It includes measured performance.
2. [**List of Nvidia Graphics Processing Units**](https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units) contains GFLOPS figures for a large number of GPUs. Probably they are all theoretical peak performance numbers. It also contains release dates and release prices.
3. **[List of AMD Graphics Processing Units](https://en.wikipedia.org/wiki/List_of_AMD_graphics_processing_units)** is much like the list of Nvidia GPUs, but for the other leading GPU brand.
4. [**Wikipedia’s FLOPS page**](https://en.wikipedia.org/wiki/FLOPS#Hardware_costs) contains a small amount of data, seemingly empirical, from a variety of sources.
5. **Wikipedia** has other small collections of theoretical performance data. For instance on the [Intel Xeon Phi](https://en.wikipedia.org/wiki/Xeon_Phi) page.
6. [**Moravec**](ftp://netuno.io.usp.br/los/IOF257/moravec.pdf) has perhaps the oldest and best known dataset. We link to an article discussing it, but its actual page was down last we checked.
7. [**Nordhaus**](http://www.econ.yale.edu/~nordhaus/homepage/prog_083001a.pdf) expands on Moravec’s data.
8. [**Koh and Magee**](http://web.mit.edu/cmagee/www/documents/15-koh_magee-tfsc_functional_approach_studying_technological_progress_vol73p1061-1083_2006.pdf) expand on Moravec’s data.
9. **Rieber and Muehlhauser** did have a dataset (discussed [here](https://intelligence.org/2014/05/12/exponential-and-non-exponential/#footnote_7_11027)) but links to it appear to be broken.
10. [**John McCallum’s**](http://www.jcmit.com/cpu-performance.htm) dataset (doesn’t load at time of writing, but is discussed in [Sandberg and Bostrom 2008](http://www.fhi.ox.ac.uk/brain-emulation-roadmap-report.pdf) and on our page on [trends in the cost of computing](http://aiimpacts.org/trends-in-the-cost-of-computing/))
11. [**Passmark**](https://www.cpubenchmark.net/) has a huge quantity of empirical performance data, for CPUs and GPUs. However it is all in terms of their own benchmark, so hard to compare to other things. They also list current prices. Looking at it over time (via [archive.org](http://web.archive.org/web/20120409044931/https://www.cpubenchmark.net)) can let you also see past prices. Doing so suggests that they change their benchmarks on occasion, which makes it even harder to interpret what they mean.
12. [**Geekbench Browser**](https://browser.geekbench.com/v4/cpu/singlecore) collects empirical performance data from people testing their computers with Geekbench’s service. They list many benchmark numbers for many computers. However identically named benchmark figures from ‘Geekbench v4’ vs. ‘Geekbench v3’ for the same hardware differ a lot (one of us recollects about a factor of five), apparently because they changed what the benchmark actually was then. This suggests care should be taken to use numbers from the same version of Geekbench, and also that any version is not necessarily comparable to other apparently identical measures from elsewhere. We are also not sure whether differences in benchmark meaning only occur between saliently labeled versions.
13. [**Export compliance metrics for Intel Processors**](https://www.intel.com/content/www/us/en/support/articles/000005755/processors.html) is a collection of PDFs listing processors alongside a number for ‘FLOP’, which we suppose is related to FLOPS. It does not contain much explanation, and has some worrying characteristics.[1](https://aiimpacts.org/computing-hardware-performance-data-collections/#easy-footnote-bottom-1-1016 "Multiple different processors from different times have identical ‘FLOP’ numbers, and the overall trend of these numbers over time does not appear to be very downward. They are also quite different from some other numbers for the same processors, but we haven’t checked this very thoroughly.")
14. [**Karl Rupp**](https://github.com/karlrupp/cpu-gpu-mic-comparison) has collected some data and made it available. He has also blogged about it [here](https://www.karlrupp.net/2013/06/cpu-gpu-and-mic-hardware-characteristics-over-time/) and [here](https://www.karlrupp.net/2016/08/flops-per-cycle-for-cpus-gpus-and-xeon-phis/). However he says he got it from a combination of the Intel compliance metrics (listed above), and the list of Intel Xeon Microprocessors (below), and a) the export compliance metrics data seems strange, and b) we couldn’t actually track down his data in those sources. Possibly we are misunderstanding the export compliance metrics, and he is interpreting them correctly, resolving both problems.
15. [**Asteroids@home**](https://asteroidsathome.net/boinc/cpu_list.php) lists Whetstone benchmark GFLOPS per core by CPU model for computers participating in their project.
16. **The [Microway knowledge center](https://www.microway.com/knowledge-center-articles/categories/performance/)** has a lot of pages containing at least some theoretical peak performance numbers (see any called ‘[detailed specifications of —](https://www.microway.com/knowledge-center-articles/detailed-specifications-of-the-intel-xeon-e5-2600v4-broadwell-ep-processors/)‘, but most of the numbers on each page are inside figures, and so hard to export or read in detail.

### Other useful hardware data

* [**List of Intel Xeon Microprocessors**](https://en.wikipedia.org/wiki/List_of_Intel_Xeon_microprocessors) does not include figures for FLOPS, but has price and release date data.


2016 ESPAI Narrow AI task forecast timeline

This is an interactive timeline we made, illustrating the median dates when respondents said they expected a 10%, 50% and 90% chance of different tasks being automatable, in the [2016 Expert Survey on progress in AI](http://aiimpacts.org/2016-expert-survey-on-progress-in-ai/) (further details on that page).

Timeline
--------


When do ML Researchers Think Specific Tasks will be Automated?

*By Katja Grace, 26 September 2017*

We asked the ML researchers in our [survey](http://aiimpacts.org/2016-expert-survey-on-progress-in-ai/) when they thought 32 narrow, relatively well defined tasks would be feasible for AI. Eighteen of them were included in [our paper](https://arxiv.org/abs/1705.08807) earlier, but the other fourteen results are among some new stuff we just put up on the [survey page](http://aiimpacts.org/2016-expert-survey-on-progress-in-ai/).

While the researchers we talked to don’t expect anything like human-level AI for a long time, they do expect a lot of specific tasks will be open to automation soon. Of the 32 tasks we asked about, either 16 or 28 of them were considered more likely than not within ten years by the median respondent (depending on how the question was framed).

And some of these would be pretty revolutionary, at an ordinary ‘turn an industry on its head’ level, rather than a ‘world gets taken over by renegade robots’ level. You have probably heard that the transport industry is in for some disruption. And phone banking, translation and answering simple questions have already been on their way out. But also forecast soon: [the near-obsoletion of musicians](http://aiimpacts.org/automation-of-music-production/).

The task rated easiest was human-level Angry Birds playing, with a 90% chance of happening within six or ten years, depending on the question framing. The annual [Angry Birds Man vs. Machine Challenge](https://aibirds.org/man-vs-machine-challenge.html) did just happen, but the results are yet to be announced.

The four tasks that were not expected within ten years regardless of question framing were translating a new language using something like a Rosetta Stone, selecting and proving publishable mathematical theorems, doing well in the Putnam math contest, and writing a New York Times bestselling story.

The fact that the respondents gave radically different answers to other questions [depending on framing](http://aiimpacts.org/2016-expert-survey-on-progress-in-ai/) suggests to us that their guesses are not super reliable. Nonetheless, we expect they are better than nothing, and that they are a good place to start if we want to debate what will happen.

To that end, below is a timeline (full screen version [here](https://cdn.knightlab.com/libs/timeline3/latest/embed/index.html?source=1NbsZ5kiaRxTW8Jo6jkJgOkHatVQHtMqKu22WwdMrwZc&font=Default&lang=en&initial_zoom=2&height=650)) showing the researchers’ estimates for all 32 questions. These estimates are using the question framing that yielded slightly earlier results – forecasts were somewhat later given a different framing of the question.


What do ML researchers think you are wrong about?

*By Katja Grace, 25 September 2017*

So, maybe you are concerned about AI risk. And maybe you are concerned that many people making AI are not concerned enough about it. Or not concerned about the right things. But if so, do you know why they disagree with you?

We didn’t, exactly. So we asked the machine learning (ML) researchers in our survey. Our questions were:

1. To what extent do you think people’s concerns about future risks from AI are due to misunderstandings of AI research?
2. What do you think are the most important misunderstandings, if there are any?

The first question was multiple choice on a five point scale, while the second was more of a free-form, compose-your-own-succinct-summary-critique-of-a-diverse-constellation-of-views type thing. Nonetheless, more than half of the people who did the first also kindly took a stab at the second. Some of their explanations were pretty long. Some not. Here is my attempt to cluster and paraphrase them:

[![](http://aiimpacts.org/wp-content/uploads/2017/09/chart-40.png)](http://aiimpacts.org/wp-content/uploads/2017/09/chart-40.png)Number of respondents giving each response, out of 74.

Our question might have been a bit broad. ‘People’s concerns about AI risk’ includes both Stuart Russell’s [concerns](http://aiimpacts.org/stuart-russells-description-of-ai-risk/) about systems optimizing n-variable functions based on fewer than n variables, and reporters’ [concerns](https://www.digitaltrends.com/business/killer-sex-robots/) about killer sex robots. Which at a minimum should probably be suspected of resting on different errors. [Edited for clarity Oct 15 ’17]
So are we being accused of any misunderstandings, or are they all meant for the ‘put pictures of Terminator on everything’ crowd?

The comments about unemployment and surprising events, and some of the ones about AI ruling over or fighting us seem likely to be directed at people like me. On the other hand, they are also all about social consequences, and none of these issues seem to be considered resolved by the relevant social scientists. So I am not too worried if I find myself in disagreement with some AI researchers there.

I am more interested if AI researchers complain that I am mistaken about AI. And I think they probably are here, at least a bit.

My sense from reading over all these responses is that the first three categories listed in the figure represent basically the same view, and that people talk about it at different levels of generality. I’d put them together like this:

> The state of the art right now looks great in the few examples you see, but those are actually a large fraction of the things that it can do, and it often can’t even do very slight variations on those things. The problems AI can currently deal with all have to be very well specified. Getting from here to AI that can just wander out into the world and even live a successful life as a rat seems wildly ambitious. We don’t know how to make *general* AI at all. So we are really unimaginably far from human-level AI, because it would have to be general.
>
>

But this is a guess on my part, and I am curious to hear whether any AI researchers reading have a better sense of what views are like.

Whether these first three categories are all the same view or not, they do sound plausibly directed at people like me. And if ML researchers want to disagree with me about the state of the art in AI or how easy it is to extend it or improve upon it, it would be truly shocking if I were in the right. So I tentatively conclude that we are probably further away from general AI than I might have thought.

On the other hand, I wouldn’t be surprised if the respondents were misdiagnosing the disagreement here. My impression is that AI researchers (among others) often take for granted that you shouldn’t worry about things decades before they are likely to happen. So when they see people worried about AI risk, they naturally suppose that those people anticipate dangerous AI much sooner than they really do. My weak impression is that this kind of misunderstanding happens often.

By the way, the respondents did mostly think concerns are based largely on misunderstandings (which is [not to imply](http://aiimpacts.org/ai-hopes-and-fears-in-numbers/) that they aren’t concerned):

[![](http://aiimpacts.org/wp-content/uploads/2016/12/chart-35.png)](http://aiimpacts.org/wp-content/uploads/2016/12/chart-35.png)Number of respondents giving each response, out of 118.
*(Results taken from [our survey page](http://aiimpacts.org/2016-expert-survey-on-progress-in-ai/). More new results are also up there.)*


Automation of music production

Most machine learning researchers expect machines will be able to create top quality music by 2036.

Details
-------

### Evidence from survey data

In the [2016 ESPAI](http://aiimpacts.org/2016-expert-survey-on-progress-in-ai/), participants were asked two relevant questions:

[**Top forty**] Compose a song that is good enough to reach the US Top 40. The system should output the complete song as an audio file.

[**Taylor**] Produce a song that is indistinguishable from a new song by a particular artist, e.g. a song that experienced listeners can’t distinguish from a new song by Taylor Swift.

#### Summary results

Answers were as follows, suggesting these milestones are likely to be reached in ten years, and quite likely to be reached in twenty years.

|  | **10 years** | **20 years** | **50 years** |
| **Top forty** | **27.5%** | **50%** | **90%** |
| **Taylor** | **60%** | **75%** | **99%** |

|  | **10%** | **50%** | **90%** |
| **Top forty** | **5 years** | **10 years** | **20 years** |
| **Taylor** | **5 years** | **10 years** | **20 years** |

#### Distributions of answers to Taylor question

The three figures below show how respondents were spread between different answers over time, for the respondents who answered the ‘fixed years’ framing.

[![](http://aiimpacts.org/wp-content/uploads/2017/09/chart-22.png)](http://aiimpacts.org/wp-content/uploads/2017/09/chart-22.png)10

[![](http://aiimpacts.org/wp-content/uploads/2017/09/chart-23.png)](http://aiimpacts.org/wp-content/uploads/2017/09/chart-23.png)20

[![](http://aiimpacts.org/wp-content/uploads/2017/09/chart-24.png)](http://aiimpacts.org/wp-content/uploads/2017/09/chart-24.png)50


Stuart Russell’s description of AI risk

Stuart Russell has argued that advanced AI poses a risk, because it will have the ability to make high quality decisions, yet may not share human values perfectly.

Details
-------

Stuart Russell describes a risk from highly advanced AI [here](https://www.edge.org/conversation/the-myth-of-ai#26015). In short:

The primary concern [with highly advanced AI] is not spooky emergent consciousness but simply the ability to make high-quality decisions. Here, quality refers to the expected outcome utility of actions taken […]. Now we have a problem:

1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.

2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task.

A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable.  This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you ask for, not what you want.


AI hopes and fears in numbers

*By Katja Grace, 28 June 2017*

People often wonder what AI researchers think about AI risk. A [good collection of quotes](http://slatestarcodex.com/2015/05/22/ai-researchers-on-ai-risk/) can tell us that worry about AI is no longer a fringe view: many big names are concerned. But without a great sense of how many total names are there, how big they are, and what [publication biases](http://aiimpacts.org/short-prediction-publication-biases/) come between us and them, it has been hard (for me at least) to get a clear view on the distribution of opinion.

[Our survey](http://aiimpacts.org/2016-expert-survey-on-progress-in-ai/) offers some new evidence on these questions. Here, 355 machine learning researchers weigh in on how good or bad they expect the results of ‘high-level machine intelligence’ to be for humanity:

[![](http://aiimpacts.org/wp-content/uploads/2017/04/ESOPAI-value.png)](http://aiimpacts.org/wp-content/uploads/2017/04/ESOPAI-value.png)

*(Click to expand)*

Each column is one person’s opinion of how our chances are divided between outcomes. I put them roughly in order of optimism, to make it intelligible to look at.

If you are wondering how many machine learning researchers didn’t answer, and what *their* views looked like: nearly four times as many, and we don’t know. But we did try to make it hard for people to decide whether to answer based on their opinions on our questions, by being uninformative in our invitations. I think we went with saying we wanted to ask about ‘progress in the field’ and offering money for responding.

So it was only when people got inside the survey that they would have discovered that we want to know how likely progress in the field is to lead to human extinction, rather than how useful improved datasets are for progress in the field (and actually, we did want to know about that too, and asked—more results to come!). Of the people who got as far as agreeing to take the survey at all, three quarters got as far as this question. So my guess is that this data represents a reasonable slice of machine learning researchers publishing in good venues.

Note that expecting the outcome to be ‘extremely bad’ with high probability doesn’t necessarily indicate support for safety research—for instance, as you may think the situation is hopeless. (We did ask several questions about that too.)

(I’ve been [putting up](http://aiimpacts.org/2016-expert-survey-on-progress-in-ai/) a bunch of survey results; this one struck me as particularly interesting to people not involved in AI forecasting.)


2016 ESPAI questions printout

This is a list of questions from the [2016 Expert Survey on Progress in AI](http://aiimpacts.org/2016-expert-survey-on-progress-in-ai/).

Details
-------

This page is a printout of questions from the [2016 Expert Survey on Progress in AI](http://aiimpacts.org/2016-expert-survey-on-progress-in-ai/) provided by the Qualtrics website as a word document, and then copied here, for searchability. It contains formatting differences with the survey as received by participants, and probably typographic errors, due to the importing process. It also contains only parts of the randomization logic, while missing other parts. The survey questions are available as a pdf [here](https://www.dropbox.com/s/99os4grxlhf744m/Final%202016%20Expert%20Survey%20on%20Progress%20in%20AI.pdf?dl=0).

Printout
--------

16-05-17 AI Survey 12 – final

consent\_tim Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

consent      2016 Expert Survey on Progress in AI   Welcome. We are conducting a study of progress in artificial intelligence and are interested in your understanding of developments in the field.   Our estimated median time for completing this survey is 12 minutes. Your responses will be kept confidential.   Many of the questions involve substantial uncertainties. Please just give us your current best guesses.   There are no known risks associated with this study. Although this study may not benefit you personally, we hope that our results will add to the knowledge about progress in AI technology. If you have questions about your rights as a research participant, you may contact the Yale University Human Subjects Committee: 203-785-4688, human.subjects@yale.edu.    Additional information is available at: http://www.yale.edu/hrpp/participants/index.html Participation in this study is completely voluntary. You are free to decline to participate and to end participation at any time for any reason. By continuing to the next page, you agree to participate in the survey.

hb\_time Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

hb\_def   1 of 7  The following questions ask about ‘high–level machine intelligence’ (HLMI).   Say we have ‘high-level machine intelligence’ when unaided machines can accomplish every task better and more cheaply than human workers. Ignore aspects of tasks for which being a human is intrinsically advantageous, e.g. being accepted as a jury member. Think feasibility, not adoption.

Display This Question:

If fixedprobabilities Is Equal to 1

hb\_a For the purposes of this question, assume that human scientific activity continues without major negative disruption. How many years until you expect:

|  |  |
| --- | --- |
|  |   (1) |
| a 10% probability of HLMI existing? (1) |  |
| a 50% probability of HLMI existing? (2) |  |
| a 90% probability of HLMI existing? (3) |  |

Display This Question:

If fixedprobabilities Is Equal to 0

hb\_b For the purposes of this question, assume that human scientific activity continues without major negative disruption. How likely is it that HLMI exists:

in 10 years? (1)

in 20 years? (2)

in 40 years? (3)

Display This Question:

If random Is Greater Than 20

And random Is Less Than or Equal to 30

hb\_comment Do you have any comments on your interpretation of this question? (optional)

Display This Question:

If random Is Greater Than 30

And random Is Less Than or Equal to 40

hb\_consider Which considerations were important in your answers to this question? (optional)

hj\_time Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

hj\_a\_jobs 1 of 7   Say an occupation becomes fully automatable when unaided machines can accomplish it better and more cheaply than human workers. Ignore aspects of occupations for which being a human is intrinsically advantageous, e.g. being accepted as a jury member. Think feasibility, not adoption.    We want to know how many years you think will pass before the following present-day occupations will be fully automatable. Please tell us your best guess of when you think there will be a small chance (10% chance), a roughly even chance (50% chance), and a high chance (90% chance).

|  |  |  |  |
| --- | --- | --- | --- |
|  | Years until small chance (10%) (1) | Years until even chance (50%) (2) | Years until high chance (90%) (3) |
| Truck driver (hj\_a\_jobs\_1) |  |  |  |
| Surgeon (hj\_a\_jobs\_2) |  |  |  |
| Retail salesperson (hj\_a\_jobs\_3) |  |  |  |
| AI researcher (hj\_a\_jobs\_4) |  |  |  |

hj\_a\_final What is an existing human occupation that you think will be among the final ones to be fully automatable? Remember to consider feasibility, not adoption.

hj\_a\_final\_pred How many years do you expect to pass before you think there is a small/even/high chance that this occupation will be fully automatable?

Small chance (10%) (1)

Even chance (50%) (2)

High chance (90%) (3)

hj\_a\_full Say we have reached ‘full automation of labor’ when all occupations are fully automatable. That is, when for any occupation, machines could be built to carry out the task better and more cheaply than human workers.In how many years do you expect full automation of labor, with small/even/high chance?

Small chance (10%) (1)

Even chance (50%) (2)

High chance (90%) (3)

Display This Question:

If random Is Greater Than 20

And random Is Less Than or Equal to 30

hj\_a\_comment Do you have any comments on your interpretation of these questions? (optional)

Display This Question:

If random Is Greater Than 30

And random Is Less Than or Equal to 40

hj\_a\_consider Which considerations were important in your answers to these questions? (optional)

hj\_b\_time Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

hj\_b\_jobs   1 of 7   Say an occupation becomes fully automatable when unaided machines can accomplish it better and more cheaply than human workers. Ignore aspects of occupations for which being a human is intrinsically advantageous, e.g. being accepted as a jury member. Think feasibility, not adoption.    We want to know how likely you think it is that the following present-day occupations will be fully automatable at future dates. Please tell us your best guess of the chance that they will be fully automatable within the next 10 years, within the next 20 years, and within the next 50 years.

|  |  |  |  |
| --- | --- | --- | --- |
|  | % chance in 10 years (1) | % chance in 20 years (2) | % chance in 50 years (3) |
| Truck driver (hj\_b\_jobs\_1) |  |  |  |
| Surgeon (hj\_b\_jobs\_2) |  |  |  |
| Retail salesperson (hj\_b\_jobs\_3) |  |  |  |
| AI researcher (hj\_b\_jobs\_4) |  |  |  |

hj\_b\_final What is an existing human occupation that you think will be among the final ones to be fully automatable? Remember to consider feasibility, not adoption.

hj\_b\_final\_pred How likely do you think it is that this occupation will be fully automatable within the next 10/20/50 years?

10 years (1)

20 years (2)

50 years (3)

hj\_b\_full Say we have reached ‘full automation of labor’ when all occupations are fully automatable. That is, when for any occupation, machines could be built to carry out the task better and more cheaply than human workers. How likely do you think it is that full automation of labor will happen within the next 10/20/50 years?

10 years (1)

20 years (2)

50 years (3)

Display This Question:

If random Is Greater Than 20

And random Is Less Than or Equal to 30

hj\_b\_comment Do you have any comments on your interpretation of these questions? (optional)

Display This Question:

If random Is Greater Than 30

And random Is Less Than or Equal to 40

hj\_b\_consier Which considerations were important in your answers to these questions? (optional)

ie\_time Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

ie\_time\_def   2 of 7   The following questions ask about ‘high–level machine intelligence’ (HLMI).     Say we have ‘high-level machine intelligence’ when unaided machines can accomplish every task better and more cheaply than human workers. Ignore aspects of tasks for which being a human is intrinsically advantageous, e.g. being accepted as a jury member. Think feasibility, not adoption.

ie\_1 Assume that HLMI will exist at some point. How likely do you then think it is that the rate of global technological improvement will dramatically increase (e.g. by a factor of ten) as a result of machine intelligence:

Within two years of that point? (1)

Within thirty years of that point? (2)

ie\_2 Assume that HLMI will exist at some point. How likely do you think it is that there will be machine intelligence that is vastly better than humans at all professions (i.e. that is vastly more capable or vastly cheaper):

Within two years of that point? (1)

Within thirty years of that point? (2)

ie\_3 Some people have argued the following:   If AI systems do nearly all research and development, improvements in AI will accelerate the pace of technological progress, including further progress in AI.   Over a short period (less than 5 years), this feedback loop could cause technological progress to become more than an order of magnitude faster.   How likely do you find this argument to be broadly correct?

* Quite unlikely (0-20%) (5)
* Unlikely (21-40%) (4)
* About even chance (41-60%) (3)
* Likely (61-80%) (2)
* Quite likely (81-100%) (1)

Display This Question:

If random Is Greater Than 40

And random Is Less Than 50

ie\_comment Do you have any comments on your interpretation of these questions? (optional)

Display This Question:

If random Is Greater Than 50

And random Is Less Than 60

ie\_consider Which considerations were important in your answers to these questions? (optional)

vb\_time Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

vb\_def   3 of 7   The following questions ask about ‘high–level machine intelligence’ (HLMI).   Say we have ‘high-level machine intelligence’ when unaided machines can accomplish every task better and more cheaply than human workers. Ignore aspects of tasks for which being a human is intrinsically advantageous, e.g. being accepted as a jury member. Think feasibility, not adoption.

vb\_1 Assume for the purpose of this question that HLMI will at some point exist. How positive or negative do you expect the overall impact of this to be on humanity, in the long run? Please answer by saying how probable you find the following kinds of impact, with probabilities adding to 100%:

\_\_\_\_\_\_ Extremely good (e.g. rapid growth in human flourishing) (1)

\_\_\_\_\_\_ On balance good (2)

\_\_\_\_\_\_ More or less neutral (3)

\_\_\_\_\_\_ On balance bad (4)

\_\_\_\_\_\_ Extremely bad (e.g. human extinction) (5)

Display This Question:

If random Is Greater Than 60

And random Is Less Than 70

vb\_comment Do you have any comments on your interpretation of this question? (optional)

Display This Question:

If random Is Greater Than 70

And random Is Less Than 80

vb\_consider Which considerations were important in your answers to this question? (optional)

c\_time Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

c\_def 4 of 7   The next questions ask about the sensitivity of progress in AI capabilities to changes in inputs.    ‘Progress in AI capabilities’ is an imprecise concept, so we are asking about progress as you naturally conceive of it, and looking for approximate answers.

c\_1 Imagine that over the past decade, only half as much researcher effort had gone into AI research. For instance, if there were actually 1,000 researchers, imagine that there had been only 500 researchers (of the same quality).How much less progress in AI capabilities would you expect to have seen? e.g. If you think progress is linear in the number of researchers, so 50% less progress would have been made, write ’50’. If you think only 20% less progress would have been made write ’20’.

c\_2 Over the last 10 years the cost of computing hardware has fallen by a factor of 20. Imagine instead that the cost of computing hardware had fallen by only a factor of 5 over that time (around half as far on a log scale).   How much less progress in AI capabilities would you expect to have seen? e.g. If you think progress is linear in 1/cost, so that 1-5/20=75% less progress would have been made, write ’75’. If you think only 20% less progress would have been made write ’20’.

c\_3 Imagine that over the past decade, there had only been half as much effort put into increasing the size and availability of training datasets. For instance, perhaps there are only half as many datasets, or perhaps existing datasets are substantially smaller or lower quality.How much less progress in AI capabilities would you expect to have seen? e.g. If you think 20% less progress would have been made, write ‘20’

c\_4 Imagine that over the past decade, AI research had half as much funding (in both academic and industry labs). For instance, if the average lab had a budget of $20 million each year, suppose their budget had only been $10 million each year.  How much less progress in AI capabilities would you expect to have seen? e.g. If you think 20% less progress would have been made, write ‘20’

c\_5 Imagine that over the past decade, there had been half as much progress in AI algorithms. You might imagine this as conceptual insights being half as frequent.  How much less progress in AI capabilities would you expect to have seen? e.g. If you think 20% less progress would have been made, write ‘20’

Display This Question:

If random Is Greater Than 80

And random Is Less Than or Equal to 90

c\_comment Do you have any comments on your interpretation of these questions? (optional)

Display This Question:

If random Is Greater Than 90

And random Is Less Than or Equal to 100

c\_consider Which considerations were important in your answers to these question? (optional)

hh\_time Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

hh\_area 4 of 7   Which AI research area have you worked in for the longest time?

hh\_howlong How long have you worked in this area?

hh\_1 Consider three levels of progress or advancement in this area:   A. Where the area was when you started working in it B. Where it is now C. Where it would need to be for AI software to have roughly human level abilities at the tasks studied in this area   What fraction of the distance between where progress was when you started working in the area (A) and where it would need to be to attain human level abilities in the area (C) have we come so far (B)?

hh\_2 Divide the period you have worked in the area into two halves: the first and the second. In which half was the rate of progress in your area higher?

* The first half (1)
* The second half (2)
* They were about the same (3)

Display This Question:

If random Is Greater Than 20

And random Is Less Than 30

hh\_comment Do you have any comments on your interpretation of these questions? (optional)

Display This Question:

If random Is Greater Than 30

And random Is Less Than 40

hh\_consider Which considerations were important in your answers to these questions? (optional)

ms\_time Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

ms\_1 4 of 7   To what extent do you think you disagree with the typical AI researcher about when HLMI will exist?

* A lot (17)
* A moderate amount (18)
* Not much (19)

ms\_2 If you disagree, why do you think that is?

ms\_3 To what extent do you think people’s concerns about future risks from AI are due to misunderstandings of AI research?

* Almost entirely (1)
* To a large extent (2)
* Somewhat (4)
* Not much (3)
* Hardly at all (5)

ms\_4 What do you think are the most important misunderstandings, if there are any?

Display This Question:

If random Is Greater Than 80

And random Is Less Than or Equal to 90

ms\_comment Do you have any comments on your interpretation of these questions? (optional)

Display This Question:

If random Is Greater Than 90

And random Is Less Than or Equal to 100

ms\_consider Which considerations were important in your answers to these questions? (optional)

ta\_time Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

ta\_def 5 of 7   How many years until you think the following AI tasks will be feasible with:     a small chance (10%)? an even chance (50%)? a high chance (90%)?   Let a task be ‘feasible’ if one of the best resourced labs could implement it in less than a year if they chose to. Ignore the question of whether they would choose to.      Tasks

ta\_1 Translate a text written in a newly discovered language into English as well as a team of human experts, using a single other document in both languages (like a Rosetta stone). Suppose all of the words in the text can be found in the translated document, and that the language is a difficult one.

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_2 Translate speech in a new language given only unlimited films with subtitles in the new language. Suppose the system has access to training data for other languages, of the kind used now (e.g. same text in two languages for many languages and films with subtitles in many languages).

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_3 Perform translation about as good as a human who is fluent in both languages but unskilled at translation, for most types of text, and for most popular languages (including languages that are known to be difficult, like Czech, Chinese and Arabic).

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_4 Provide phone banking services as well as human operators can, without annoying customers more than humans. This includes many one-off tasks, such as helping to order a replacement bank card or clarifying how to use part of the bank website to a customer.

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_5 Correctly group images of previously unseen objects into classes, after training on a similar labeled dataset containing completely different classes. The classes should be similar to the ImageNet classes.

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_6 One-shot learning: see only one labeled image of a new object, and then be able to recognize the object in real world scenes, to the extent that a typical human can (i.e. including in a wide variety of settings). For example, see only one image of a platypus, and then be able to recognize platypuses in nature photos. The system may train on labeled images of other objects.   Currently, deep networks often need hundreds of examples in classification tasks1, but there has been work on one-shot learning for both classification2 and generative tasks3.   1 Lake et al. (2015). Building Machines That Learn and Think Like People 2 Koch (2015). Siamese Neural Networks for One-Shot Image Recognition 3 Rezende et al. (2016). One-Shot Generalization in Deep Generative Models

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_7 See a short video of a scene, and then be able to construct a 3D model of the scene good enough to create a realistic video of the same scene from a substantially different angle.For example, constructing a short video of walking through a house from a video taking a very different path through the house.

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_8 Transcribe human speech with a variety of accents in a noisy environment as well as a typical human can.

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_9 Take a written passage and output a recording that can’t be distinguished from a voice actor, by an expert listener.

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_10 Routinely and autonomously prove mathematical theorems that are publishable in top mathematics journals today, including generating the theorems to prove.

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_11 Perform as well as the best human entrants in the Putnam competition—a math contest whose questions have known solutions, but which are difficult for the best young mathematicians.

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_12 Defeat the best Go players, training only on as many games as the best Go players have played.     For reference, DeepMind’s AlphaGo has probably played a hundred million games of self-play, while Lee Sedol has probably played 50,000 games in his life1.     1 Lake et al. (2015). Building Machines That Learn and Think Like People

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_13 Beat the best human Starcraft 2 players at least 50% of the time, given a video of the screen.   Starcraft 2 is a real time strategy game characterized by:   Continuous time play Huge action space Partial observability of enemies Long term strategic play, e.g. preparing for and then hiding surprise attacks.

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_14 Play a randomly selected computer game, including difficult ones, about as well as a human novice, after playing the game less than 10 minutes of game time. The system may train on other games.

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_15 Play new levels of Angry Birds better than the best human players. Angry Birds is a game where players try to efficiently destroy 2D block towers with a catapult. For context, this is the goal of the IJCAI Angry Birds AI competition1.     1 aibirds.org

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_16 Outperform professional game testers on all Atari games using no game-specific knowledge. This includes games like Frostbite, which require planning to achieve sub-goals and have posed problems for deep Q-networks1, 2.     1 Mnih et al. (2015). Human-level control through deep reinforcement learning 2 Lake et al. (2015). Building Machines That Learn and Think Like People

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_17 Outperform human novices on 50% of Atari games after only 20 minutes of training play time and no game specific knowledge.   For context, the original Atari playing deep Q-network outperforms professional game testers on 47% of games1, but used hundreds of hours of play to train2.   1 Mnih et al. (2015). Human-level control through deep reinforcement learning 2 Lake et al. (2015). Building Machines That Learn and Think Like People

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_18 Fold laundry as well and as fast as the median human clothing store employee.

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_19 Beat the fastest human runners in a 5 kilometer race through city streets using a bipedal robot body.

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_20 Physically assemble any LEGO set given the pieces and instructions, using non-specialized robotics hardware.   For context, Fu 20161 successfully joins single large LEGO pieces using model based reinforcement learning and online adaptation.   1 Fu et al. (2016). One-Shot Learning of Manipulation Skills with Online Dynamics Adaptation and Neural Network Priors

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_21 Learn to efficiently sort lists of numbers much larger than in any training set used, the way Neural GPUs can do for addition1, but without being given the form of the solution.   For context, Neural Turing Machines have not been able to do this2, but Neural Programmer-Interpreters3 have been able to do this by training on stack traces (which contain a lot of information about the form of the solution).   1 Kaiser & Sutskever (2015). Neural GPUs Learn Algorithms   2 Zaremba & Sutskever (2015). Reinforcement Learning Neural Turing Machines   3 Reed & de Freitas (2015). Neural Programmer-Interpreters

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_22 Write concise, efficient, human-readable Python code to implement simple algorithms like quicksort. That is, the system should write code that sorts a list, rather than just being able to sort lists.   Suppose the system is given only:   A specification of what counts as a sorted list Several examples of lists undergoing sorting by quicksort

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_23 Answer any “easily Googleable” factoid questions posed in natural language better than an expert on the relevant topic (with internet access), having found the answers on the internet.   Examples of factoid questions:     “What is the poisonous substance in Oleander plants?” “How many species of lizard can be found in Great Britain?”

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_24 Answer any “easily Googleable” factual but open ended question posed in natural language better than an expert on the relevant topic (with internet access), having found the answers on the internet.   Examples of open ended questions:     “What does it mean if my lights dim when I turn on the microwave?” “When does home insurance cover roof replacement?”

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_25 Give good answers in natural language to factual questions posed in natural language for which there are no definite correct answers. For example:”What causes the demographic transition?”, “Is the thylacine extinct?”, “How safe is seeing a chiropractor?”

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_26 Write an essay for a high-school history class that would receive high grades and pass plagiarism detectors. For example answer a question like ‘How did the whaling industry affect the industrial revolution?’

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_27 Compose a song that is good enough to reach the US Top 40. The system should output the complete song as an audio file.

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_28 Produce a song that is indistinguishable from a new song by a particular artist, e.g. a song that experienced listeners can’t distinguish from a new song by Taylor Swift.

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_29 Write a novel or short story good enough to make it to the New York Times best-seller list.

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_30 For any computer game that can be played well by a machine, explain the machine’s choice of moves in a way that feels concise and complete to a layman.

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_31 Play poker well enough to win the World Series of Poker.

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

ta\_32 After spending time in a virtual world, output the differential equations governing that world in symbolic form.For example, the agent is placed in a game engine where Newtonian mechanics holds exactly and the agent is then able to conduct experiments with a ball and output Newton’s laws of motion.

small chance (10%) (1)

even chance (50%) (2)

high chance (90%) (3)

Display This Question:

If random Is Greater Than 0

And random Is Less Than 10

ta\_comment Do you have any comments on your interpretation of these questions? (optional)

Display This Question:

If random Is Greater Than 10

And random Is Less Than 20

ta\_consider Which considerations were important in your answers to these questions? (optional)

tb\_time Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

tb\_def 5 of 7   How likely do you think it is that the following AI tasks will be feasible within the next:     10 years? 20 years? 50 years?   Let a task be ‘feasible’ if one of the best resourced labs could implement it in less than a year if they chose to. Ignore the question of whether they would choose to.      Tasks

tb\_1 Translate a text written in a newly discovered language into English as well as a team of human experts, using a single other document in both languages (like a Rosetta stone). Suppose all of the words in the text can be found in the translated document, and that the language is a difficult one.

10 years (1)

20 years (2)

50 years (3)

tb\_2 Translate speech in a new language given only unlimited films with subtitles in the new language. Suppose the system has access to training data for other languages, of the kind used now (e.g. same text in two languages for many languages and films with subtitles in many languages).

10 years (4)

20 years (5)

50 years (6)

tb\_3 Perform translation about as good as a human who is fluent in both languages but unskilled at translation, for most types of text, and for most popular languages (including languages that are known to be difficult, like Czech, Chinese and Arabic).

10 years (1)

20 years (2)

50 years (3)

tb\_4 Provide phone banking services as well as human operators can, without annoying customers more than humans. This includes many one-off tasks, such as helping to order a replacement bank card or clarifying how to use part of the bank website to a customer.

10 years (1)

20 years (2)

50 years (3)

tb\_5 Correctly group images of previously unseen objects into classes, after training on a similar labeled dataset containing completely different classes. The classes should be similar to the ImageNet classes.

10 years (1)

20 years (2)

50 years (3)

tb\_6 One-shot learning: see only one labeled image of a new object, and then be able to recognize the object in real world scenes, to the extent that a typical human can (i.e. including in a wide variety of settings). For example, see only one image of a platypus, and then be able to recognize platypuses in nature photos. The system may train on labeled images of other objects.   Currently, deep networks often need hundreds of examples in classification tasks1, but there has been work on one-shot learning for both classification2 and generative tasks3.   1 Lake et al. (2015). Building Machines That Learn and Think Like People 2 Koch (2015). Siamese Neural Networks for One-Shot Image Recognition 3 Rezende et al. (2016). One-Shot Generalization in Deep Generative Models

10 years (1)

20 years (2)

50 years (3)

tb\_7 See a short video of a scene, and then be able to construct a 3D model of the scene that is good enough to create a realistic video of the same scene from a substantially different angle.For example, constructing a short video of walking through a house from a video taking a very different path through the house.

10 years (1)

20 years (2)

50 years (3)

tb\_8 Transcribe human speech with a variety of accents in a noisy environment as well as a typical human can.

10 years (1)

20 years (2)

50 years (3)

tb\_9 Take a written passage and output a recording that can’t be distinguished from a voice actor, by an expert listener.

10 years (1)

20 years (2)

50 years (3)

tb\_10 Routinely and autonomously prove mathematical theorems that are publishable in top mathematics journals today, including generating the theorems to prove.

10 years (1)

20 years (2)

50 years (3)

tb\_11 Perform as well as the best human entrants in the Putnam competition—a math contest whose questions have known solutions, but which are difficult for the best young mathematicians.

10 years (1)

20 years (2)

50 years (3)

tb\_12 Defeat the best Go players, training only on as many games as the best Go players have played.     For reference, DeepMind’s AlphaGo has probably played a hundred million games of self-play, while Lee Sedol has probably played 50,000 games in his life1.     1 Lake et al. (2015). Building Machines That Learn and Think Like People

10 years (1)

20 years (2)

50 years (3)

tb\_13 Beat the best human Starcraft 2 players at least 50% of the time, given a video of the screen.   Starcraft 2 is a real time strategy game characterized by:   Continuous time play Huge action space Partial observability of enemies Long term strategic play, e.g. preparing for and then hiding surprise attacks.

10 years (1)

20 years (2)

50 years (3)

tb\_14 Play a randomly selected computer game, including difficult ones, about as well as a human novice, after playing the game less than 10 minutes of game time. The system may train on other games.

10 years (1)

20 years (2)

50 years (3)

tb\_15 Play new levels of Angry Birds better than the best human players. Angry Birds is a game where players try to efficiently destroy 2D block towers with a catapult. For context, this is the goal of the IJCAI Angry Birds AI competition1.     1 aibirds.org

10 years (1)

20 years (2)

50 years (3)

tb\_16 Outperform professional game testers on all Atari games using no game-specific knowledge. This includes games like Frostbite, which require planning to achieve sub-goals and have posed problems for deep Q-networks1, 2.     1 Mnih et al. (2015). Human-level control through deep reinforcement learning 2 Lake et al. (2015). Building Machines That Learn and Think Like People

10 years (1)

20 years (2)

50 years (3)

tb\_17 Outperform human novices on 50% of Atari games after only 20 minutes of training play time and no game specific knowledge.   For context, the original Atari playing deep Q-network outperforms professional game testers on 47% of games1, but used hundreds of hours of play to train2.   1 Mnih et al. (2015). Human-level control through deep reinforcement learning 2 Lake et al. (2015). Building Machines That Learn and Think Like People

10 years (1)

20 years (2)

50 years (3)

tb\_18 Fold laundry as well and as fast as the median human clothing store employee.

10 years (1)

20 years (2)

50 years (3)

tb\_19 Beat the fastest human runners in a 5 kilometer race through city streets using a bipedal robot body.

10 years (1)

20 years (2)

50 years (3)

tb\_20 Physically assemble any LEGO set given the pieces and instructions, using non-specialized robotics hardware.   For context, Fu 20161 successfully joins single large LEGO pieces using model based reinforcement learning and online adaptation.   1 Fu et al. (2016). One-Shot Learning of Manipulation Skills with Online Dynamics Adaptation and Neural Network Priors

10 years (1)

20 years (2)

50 years (3)

tb\_21 Learn to efficiently sort lists of numbers much larger than in any training set used, the way Neural GPUs can do for addition1, but without being given the form of the solution.   For context, Neural Turing Machines have not been able to do this2, but Neural Programmer-Interpreters3 have been able to do this by training on stack traces (which contain a lot of information about the form of the solution).   1 Kaiser & Sutskever (2015). Neural GPUs Learn Algorithms   2 Zaremba & Sutskever (2015). Reinforcement Learning Neural Turing Machines   3 Reed & de Freitas (2015). Neural Programmer-Interpreters

10 years (1)

20 years (2)

50 years (3)

tb\_22 Write concise, efficient, human-readable Python code to implement simple algorithms like quicksort. That is, the system should write code that sorts a list, rather than just being able to sort lists.   Suppose the system is given only:   A specification of what counts as a sorted list Several examples of lists undergoing sorting by quicksort

10 years (1)

20 years (2)

50 years (3)

tb\_23 Answer any “easily Googleable” factoid questions posed in natural language better than an expert on the relevant topic (with internet access), having found the answers on the internet.   Examples of factoid questions:     “What is the poisonous substance in Oleander plants?” “How many species of lizard can be found in Great Britain?”

10 years (1)

20 years (2)

50 years (3)

tb\_24 Answer any “easily Googleable” factual but open ended question posed in natural language better than an expert on the relevant topic (with internet access), having found the answers on the internet.   Examples of open ended questions:     “What does it mean if my lights dim when I turn on the microwave?” “When does home insurance cover roof replacement?”

10 years (1)

20 years (2)

50 years (3)

tb\_25 Give good answers in natural language to factual questions posed in natural language for which there are no definite correct answers. For example:”What causes the demographic transition?”, “Is the thylacine extinct?”, “How safe is seeing a chiropractor?”

10 years (1)

20 years (2)

50 years (3)

tb\_26 Write an essay for a high-school history class that would receive high grades and pass plagiarism detectors. For example answer a question like ‘How did the whaling industry affect the industrial revolution?’

10 years (1)

20 years (2)

50 years (3)

tb\_27 Compose a song that is good enough to reach the US Top 40. The system should output the complete song as an audio file.

10 years (1)

20 years (2)

50 years (3)

tb\_28 Produce a song that is indistinguishable from a new song by a particular artist, e.g. a song that experienced listeners can’t distinguish from a new song by Taylor Swift.

10 years (1)

20 years (2)

50 years (3)

tb\_29 Write a novel or short story good enough to make it to the New York Times best-seller list.

10 years (1)

20 years (2)

50 years (3)

tb\_30 For any computer game that can be played well by a machine, explain the machine’s choice of moves in a way that feels concise and complete to a layman.

10 years (1)

20 years (2)

50 years (3)

tb\_31 Play poker well enough to win the World Series of Poker.

10 years (1)

20 years (2)

50 years (3)

tb\_32 After spending time in a virtual world, output the differential equations governing that world in symbolic form.For example, the agent is placed in a game engine where Newtonian mechanics holds exactly and the agent is then able to conduct experiments with a ball and output Newton’s laws of motion.

10 years (1)

20 years (2)

50 years (3)

Display This Question:

If random Is Greater Than 0

And random Is Less Than 10

tb\_comment Do you have any comments on your interpretation of these questions? (optional)

Display This Question:

If random Is Greater Than 10

And random Is Less Than 20

tb\_consider Which considerations were important in your answers to these questions? (optional)

sq\_time Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

sq\_def 6 of 7   Stuart Russell summarizes an argument for why highly advanced AI might pose a risk as follows:   The primary concern [with highly advanced AI] is not spooky emergent consciousness but simply the ability to make high-quality decisions. Here, quality refers to the expected outcome utility of actions taken […]. Now we have a problem:   1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down. 2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task.   A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable.  This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you ask for, not what you want.

sq\_1 Do you think this argument points at an important problem?

* No, not a real problem. (1)
* No, not an important problem. (2)
* Yes, a moderately important problem. (3)
* Yes, a very important problem. (5)
* Yes, among the most important problems in the field. (4)

aq\_2 How valuable is it to work on this problem today, compared to other problems in AI?

* Much less valuable (1)
* Less valuable (2)
* As valuable as other problems (3)
* More valuable (4)
* Much more valuable (5)

sq\_3 How hard do you think this problem is compared to other problems in AI?

* Much easier (1)
* Easier (2)
* As hard as other problems (3)
* Harder (4)
* Much harder (5)

sq\_comment Do you have any comments on your interpretation of this question? (optional)

Display This Question:

If random Is Greater Than 90

And random Is Less Than or Equal to 100

sq\_consider Which considerations were important in your answers to this question? (optional)

sr\_time Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

sr\_def 6 of 7   Let ‘AI safety research’ include any AI-related research that, rather than being primarily aimed at improving the capabilities of AI systems, is instead primarily aimed at minimizing potential risks of AI systems (beyond what is already accomplished for those goals by increasing AI system capabilities).   Examples of AI safety research might include:   Improving the human-interpretability of machine learning algorithms for the purpose of improving the safety and robustness of AI systems, not focused on improving AI capabilities Research on long-term existential risks from AI systems  AI-specific formal verification research Policy research about how to maximize the public benefits of AI

sr\_1 How much should society prioritize AI safety research, relative to how much it is currently prioritized?

* Much less (1)
* Less (2)
* About the same (3)
* More (4)
* Much more (5)

Display This Question:

If random Is Greater Than 80

And random Is Less Than or Equal to 90

sr\_comment Do you have any comments on your interpretation of this question? (optional)

Display This Question:

If random Is Greater Than 90

And random Is Less Than or Equal to 100

sr\_consider Which considerations were important in your answer to this question? (optional)

dem\_time Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

dem\_1 7 of 7   How much thought have you given in the past to when HLMI (or something similar) will be developed?

* Very little. e.g. “I can’t remember thinking about this.” (1)
* A little. e.g. “It has come up in conversation a few times” (2)
* A moderate amount. e.g. “I read something about it now and again” (3)
* A lot. e.g. “I have thought enough to have my own views on the topic” (4)
* A great deal. e.g. “This has been a particular interest of mine” (5)

dem\_2 How much thought have you given in the past to social impacts of smarter-than-human machines?

* Very little. e.g. “I can’t remember thinking about this.” (1)
* A little. e.g. “It has come up in conversation a few times” (2)
* A moderate amount. e.g. “I read something about it now and again” (3)
* A lot. e.g. “I have thought enough to have my own views on the topic” (4)
* A great deal. e.g. “This has been a particular interest of mine” (5)

dem\_3 Are you an AI researcher?

* Yes (4)
* No (5)

dem\_4 What are your main areas of research?

dem\_5 Where do you work?

* Industry (1)
* Academia (2)
* Other (3)

Q203 Timing

First Click (1)

Last Click (2)

Page Submit (3)

Click Count (4)

end Thank you for contributing to the Expert Survey on Progress in AI!   We will send you output from this research as it becomes available. We are sending every 10th person $250 as an expression of gratitude for completing the survey. We are a group of researchers interested in measuring and understanding AI progress and its implications. If you are interested in learning more about this research, please click here or email us.     Katja Grace, Machine Intelligence Research Institute John Salvatier, Machine Intelligence Research Institute Allan Dafoe, Yale University Baobao Zhang, Massachusetts Institute of Technology

print Would you like to be recognized in print as an expert participant in this survey?

* Yes (1)
* No (2)

autojobs A group from Oxford University will soon conduct a survey on Automation and Jobs. It will explore some of the topics in this survey in more depth. Would you like to receive an invitation to participate in it?

* Yes (1)
* No (2)

comment\_box If you have any questions or comments for us, please feel free to share them below.


Media discussion of 2016 ESPAI

The [2016 Expert Survey on Progress in AI](http://aiimpacts.org/2016-expert-survey-on-progress-in-ai/) was discussed in at least 20 media outlets, popular blogs, and industry-specific sites that we know of. Most of them were summaries of the survey findings. Commonly emphasized details included the difference between Asian and North American estimates, the 5% chance of catastrophically bad outcomes (usually described as low), and likely effects on the economy in the event of widespread automation.

### List of media mentions

1. [Newsweek](http://www.newsweek.com/artificial-intelligence-will-take-our-jobs-2060-618259): A major general news outlet. Listed many shorter-term estimates from the survey, and discussed practical responses to inevitable automation, such as universal basic income.
2. [Yahoo](https://www.yahoo.com/tech/forget-chess-experts-suggest-ai-185736517.html): A major general news outlet. Cautiously reported the survey’s predictions, with the caveat that predictions about distant technology are often not very precise. Mentioned policy recommendations from Katja.
3. [BBC](http://www.bbc.com/capital/story/20170619-how-long-will-it-take-for-your-job-to-be-automated): A major British news channel. Described the current state of AI’s capabilities, including a detailed summary of modern clothes-folding robots. Discussed the survey’s predictions in this context. (Reposted by [OECD Forum](https://www.oecd-forum.org/users/50593-oecd/posts/17928-hand-picked-news-for-you-how-long-will-it-take-for-your-job-to-be-automated).)
4. [Daily Mail](http://www.dailymail.co.uk/sciencetech/article-4560824/AI-outperform-humans-tasks-just-45-years.html): A major British news outlet. Summarized the study, paying special attention to number of years estimated for AI to overtake humans on specific tasks. Considered the 5% chance of a catastrophically bad outcome to be relatively good news.
5. [MIT Technology Review](https://www.technologyreview.com/s/607970/experts-predict-when-artificial-intelligence-will-exceed-human-performance/): A technology review at MIT. Offered a critical review of the survey’s results, including the theory that ‘40 years’ can be a red flag in terms of estimates, because it can symbolize the approximate end of a surveyed person’s working life. (We do not find evidence to support this [theory](http://aiimpacts.org/the-maes-garreau-law/).) (Reposted by [Business Insider](http://www.businessinsider.com/experts-predict-when-ai-will-exceed-human-performance-2017-6).)
6. [Business Insider](http://www.businessinsider.com/when-robots-will-take-peoples-jobs-2017-6): The second article from about the AI survey from this major business news outlet. Summarized the study findings, including a graph showing estimated times that AI will overtake humans at a wide range of tasks.
7. [Daily Kos](https://www.dailykos.com/story/2017/6/4/1668920/-When-Will-AI-Exceed-Human-Performance): A highly prominent American liberal group blog. Provided an in-depth summary of the survey, its political ramifications, and potential solutions to the problem of massively increased automation in the workforce.
8. [ZDNet](http://www.zdnet.com/article/ai-experts-predict-the-future-truck-drivers-out-of-jobs-by-2027-surgeons-by-2053/): A business technology news website. Reported the study’s findings, with special emphasis into the case of Go, and the mitigating circumstances of AlphaGo’s win – AlphaGo had played many more games than any human opponent, while the survey question about Go included the assumption that the human and AI players would have played an equal number of practice games. Included a brief summary video.
9. [The Register](https://www.theregister.co.uk/2017/05/26/job_automation_by_ai_timeline/): A British tech publication. Mentioned both predictions from the survey and the high variance between estimates, including specifically named researchers.
10. [Leading Britain’s Conversation](http://www.lbc.co.uk/radio/presenters/nick-abbot/blog/anything-we-can-do-they-can-do-better/): A blog by Nick Abbot, a presenter for a British radio station. Covered the study’s results in a humorous, informal tone.
11. [Tech Republic](http://www.techrepublic.com/article/experts-say-theres-a-50-chance-ai-will-outperform-humans-in-every-job-in-45-years/): A technology blog. Described the survey findings in detail, including a video. Included three takeaways at the end: that there is a 50% chance of AI exceeding human capability at all jobs within 45 years, that there is a 50% chance of AI automating all jobs within 122 years, and that global catastrophic risk is possible from AI and should be guarded against.
12. [Daily News and Analysis](http://www.dnaindia.com/analysis/column-artificial-intelligence-is-the-next-big-threat-to-job-creation-2469051): An Indian news source. Addressed the practical needs for India to modernize its jobs and stay ahead of the curve as AI makes current jobs obsolete.
13. [CTV News](http://www.ctvnews.ca/sci-tech/50-50-chance-machines-could-beat-humans-at-every-task-in-45-years-study-1.3468789): A Canadian news outlet. Listed estimates for AI to beat humans at a range of specific tasks, and briefly summarized the chances for AI’s overall impact to be good or bad.
14. [2OceansVibe](http://www.2oceansvibe.com/2017/06/19/lets-look-at-how-many-years-it-will-be-until-a-robot-takes-your-job/):A large solely owned South African media site. Described the study in detail, including a note on which sorts of jobs may not be automated even though they could be. (Reposted by [Nigeria Today](https://www.nigeriatoday.ng/2017/06/lets-look-at-how-many-years-it-will-be-until-a-robot-takes-your-job/).)
15. [Tekniikan Maailma](https://tekniikanmaailma.fi/muu-tekniikka/tekoalytuntijat-jumalankaltainen-homo-deus-syrjayttaa-nykyihmisen-ja-vuonna-2060-tekoaly-kaikessa-ihmista-parempi/): A Finnish technology news source. Focused on the predicted 5% chance of a catastrophic outcome as good news, and went into detail on reasons why AI might not be too dangerous – including that any given current AI’s capacities are very specific. Also touched on possibilities for transhumanism involving AI.
16. [New Scientist](https://www.newscientist.com/article/2133188-ai-will-be-able-to-beat-us-at-everything-by-2060-say-experts/): A significant science news site. Noted the high chance of major social impact, but considered the 5% chance of catastrophic outcomes less alarming. Focused somewhat on recent achievements of AI, as well as future projections. Included a brief summary video.
17. [Slate Star Codex:](http://slatestarcodex.com/2017/06/08/ssc-journal-club-ai-timelines/) A popular blog which frequently posts about AI. Emphasized the high variance between surveyed opinions, and the fact that many prominent AI researchers now support AI safety research.
18. [Slator](https://slator.com/academia/yale-oxford-enter-business-predicting-end-human-translator/): A linguistics industry website. Mentioned the survey’s predictions with respect to language translation specifically.
19. [Fossbytes](https://fossbytes.com/when-will-ai-exceed-human-performance-researchers-just-gave-the-timeline/): A general technology blog. Reported the survey’s findings in brief.
20. [Tech XPlore](https://techxplore.com/news/2017-06-experts-ai-surpass-humans-tasks.html): An outlet for science writing. A lengthy, technical summary of the survey, which gave special notice to the difference between Asian and North American responses.
21. [Nature World News](http://www.natureworldnews.com/articles/38347/20170606/ai-to-beat-humans-in-2060-but-elon-musk-believes-it-can-by-2030.htm): A general natural sciences website. Mentioned the survey’s results, as well as Elon Musk’s belief in a faster AI timeline.
22. [Axios](https://www.axios.com/experts-predict-ai-machines-will-soon-become-to-superior-to-humans-2429612829.html): An online media outlet with a “Future of Work” section. Described the study in brief, with emphasis on the risks entailed by AI ascendance.
23. [Human Cusp](https://humancusp.com/2017/06/01/when-will-a-machine-do-your-job-better-than-you/): A blog focusing on possibilities for superintelligent AI. Touched on the special case of Go, and how AI in fact surpassed humans much faster than AI researchers predicted.


Some survey results!

*By Katja Grace, 8 June 2017*

We put the main results of [our survey](http://aiimpacts.org/2016-expert-survey-on-progress-in-ai/) of machine learning researchers on AI timelines online recently—see **[here](https://arxiv.org/abs/1705.08807)** for the paper.

Apologies for the delay—we are trying to avoid spoiling the newsworthiness of the results for potential academic publishers, lest they stop being potential academic publishers. But some places are ok with preprints on arXiv, so now we have one. And that means we can probably share some other things too.

There is actually a lot of stuff that isn’t in the paper, and it might take a little while for everything to be released. (The spreadsheet I’m looking at has 344 columns, and more than half of them represent boxes that people could write things in. And for many combinations of boxes, there are interesting questions to be asked.) We hope to share the whole dataset sometime soon, minus a few de-anonymizing bits. As we release more results, I’ll add them to [our page about the survey](http://aiimpacts.org/2016-expert-survey-on-progress-in-ai/).

The main interesting results so far, as I see them:

* **Comparable forecasts seem to be later than in past surveys.** in the [other surveys we know of](http://aiimpacts.org/ai-timeline-surveys/), the median dates for a 50% chance of [something like](http://aiimpacts.org/human-level-ai/) High-Level Machine Intelligence (HLMI) range from 2035 to 2050. Here the median answer to the most similar question puts a 50% chance of HLMI in 2057 (this isn’t in the paper—it is just the median response to the HLMI question asked using the ‘fixed probabilities framing’, i.e. the way it has been asked before). This seems surprising to me given the progress machine learning has seen since last survey, but less surprising because we changed the definition of HLMI, in part fearing it had previously been interpreted to mean a relatively low level of performance.
* **Asking people about specific jobs massively changes HLMI forecasts.** When we asked some people when AI would be able to do several specific human occupations, and then all human occupations (presumably a subset of all tasks), they gave very much later timelines than when we just asked about HLMI straight out. For people asked to give probabilities for certain years, the difference was a factor of a thousand twenty years out! (10% vs. 0.01%) For people asked to give years for certain probabilities, the normal way of asking put 50% chance 40 years out, while the ‘occupations framing’ put it 90 years out. (These are all based on straightforward medians, not the complicated stuff in the paper.)
* **People consistently give later forecasts if you ask them for the probability in N years instead of the year that the probability is M.** We saw this in the straightforward HLMI question, and most of the tasks and occupations, and also in most of these things when we tested them on mturk people earlier. For HLMI for instance, if you ask when there will be a 50% chance of HLMI you get a median answer of 40 years, yet if you ask what the probability of HLMI is in 40 years, you get a median answer of 30%.
* **Lots of ‘narrow’ AI milestones are forecast in the next decade as likely as not.** These are interesting, because most of them haven’t been forecast before to my knowledge, and many of them have social implications. For instance, if in a decade machines can not only write pop hits as well as Taylor Swift can, but can write pop hits that sound like Taylor Swift as well as Taylor Swift can—and perhaps faster, more cheaply, and on Spotify—then will that be the end of the era of superstar musicians? This perhaps doesn’t rival human extinction risks for importance, but human extinction risks do not happen in a vacuum ([except one](https://en.wikipedia.org/wiki/False_vacuum#Vacuum_metastability_event)) and there is something to be said for paying attention to big changes in the world other than the one that matters most.
* **There is broad support among ML researchers for the premises and conclusions of AI safety arguments.** Two thirds of them say the AI risk problem described by Stuart Russell is at least moderately important, and a third say it is at least as valuable to work on as other problems in the field. The median researcher thinks AI has a one in twenty chance of being extremely bad on net. Nearly half of researchers want to see more safety research than we currently have (compared to only 11% who think we are already prioritizing safety too much). There has been a perception lately that AI risk has moved to being a mainstream concern among AI researchers, but it is hard to tell from voiced opinion whether one is hearing from a loud minority or the vocal tip of an opinion iceberg. So it is interesting to see this perception confirmed with survey data.
* **Researchers’ predictions vary a lot.** That is pretty much what I expected, but it is still important to know. Interestingly (and not in the paper), researchers don’t seem to be aware that their predictions vary a lot. More than half of respondents guess that they disagree ‘not much’ with the typical AI researcher about when HLMI will exist (vs. a moderate amount, or a lot).
* **Researchers who studied in Asia have much shorter timelines than those who studied in North Amercia.** In terms of the survey’s ‘aggregate prediction’ thing, which is basically a mean, the difference is 30 years (Asia) vs. 74 years (North America). (See p5)
* I feel like any circumstance where **a group of scientists guesses that the project they are familiar with has a 5% chance of outcomes near ‘human extinction’ levels** **of bad** is worthy of special note, though maybe it is not actually that surprising, and could easily turn out to be misuse of small probabilities or something.

Some notes on interpreting the paper:

1. The milestones in the timeline and in the abstract are from three different sets of questions. There seems to be a large framing effect between two of them—full automation of labor is logically required to be before HLMI, and yet it is predicted much later—and it is unclear whether people answer the third set of questions (about narrow tasks) more like the one about HLMI or more like the one about occupations. Plus even if there were no framing effect to worry about, we should expect milestones about narrow tasks to be much earlier than milestones about very similar sounding occupations. For instance, if there were an occupation ‘math researcher’, it should be later than the narrow task summarized here as ‘math research’. So there is a risk of interpreting the figure as saying AI research is harder than math research, when really the ‘-er’ is all-important. So to help avoid confusion, here is the timeline colored in by which set of questions each milestone came from. The blue one was asked on its own. The orange ones were always asked together: first all four occupations, then they were asked for an occupation they expected to be very late, and when they expected it, then full automation of labor. The pink milestones were randomized, so that each person got four. There are a lot more pink milestones not included here, but included in the long table at the end of the paper.[![](http://aiimpacts.org/wp-content/uploads/2017/06/timelineofairesults.jpeg)](http://aiimpacts.org/wp-content/uploads/2017/06/timelineofairesults.jpeg)
2. In Figure 2 and Table S5 I believe the word ‘median’ means we are talking about the ‘50% chance of occurring’ number, and the dates given are this ‘median’ (50% chance) date for a distribution that was made by averaging together all of the different people’s distributions (or what we guess their distributions are like from three data points).
3. Here are the simple median numbers for human level AI, for each combination of framings. I am sorry that the table is not simple. There are two binary framing choices—whether to ask about probabilities or years, and whether to ask about a bunch of occupations and then all occupations, or just to ask about HLMI. Each framing is in a different column, and bold blue numbers are the medians of probabilities or years given by respondents to match the years or probabilities that are non-bold numbers on the same line. e.g. the first line says that the median person seeing the fixed years, occupations framing said that in ten years there was 0% chance, and the median person seeing the fixed years, HLMI framing said that in ten years there would be a 1% chance of HLMI.

| Time | Probability |
|  | Fixed years, Occs | Fixed years, HLMI |
| 10 years | **0%** | **1%** |
| 20 years | **0.01%** | **10%** |
| 40 years |  | **30%** |
| 50 years | **3%** |  |
| **15 years** | **50 years** | 10% |
| **40 years** | **90 years** | 50% |
| **100 years** | **200 years** | 90% |
| Fixed probs, HLMI | Fixed probs, Occs |  |  |
4. All of the questions are [here](https://www.dropbox.com/s/99os4grxlhf744m/Final%202016%20Expert%20Survey%20on%20Progress%20in%20AI.pdf?dl=0). This is the specific statement of Stuart Russell’s quote that many questions referred to:

Stuart Russell summarizes an argument for why highly advanced AI might pose a risk as follows:

The primary concern [with highly advanced AI] is not spooky emergent consciousness but simply the ability to make high-quality decisions. Here, quality refers to the expected outcome utility of actions taken […]. Now we have a problem:

1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.

2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task.

A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you ask for, not what you want.

*8 June 2017*