Concrete AI tasks bleg

*By Katja Grace, 30 March 2016*

We’re making a [survey](http://aiimpacts.org/ai-timeline-surveys/). I hope to write soon about our general methods and plans, so anyone kind enough to criticize them has the chance. Before that though, we have a different request: we want a list of concrete tasks that AI can’t do yet, but may achieve sometime between now and surpassing humans at everything. For instance, ‘beat a top human Go player in a five game match’ would have been a good example until recently. We are going to ask AI researchers to predict a subset of these tasks, to better chart the murky path ahead.

We hope to:

1. **Include tasks from across the range of AI subfields**
2. **Include tasks from across the range of time** (i.e. some things we can nearly do, some things that are really hard)
3. **Have the tasks relate relatively closely to narrowish AI projects**, to make them easier to think about (e.g. winning a 5k bipedal race is fairly close to existing projects, whereas winning an interpretive dance-off would require a broader mixture of skills, so is less good for our purposes)
4. **Have the tasks relate to specific hard technical problems** (e.g. one-shot learning or hierarchical planning)
5. **Have the tasks relate to large changes in the world** (e.g. replacing all drivers would viscerally change things)

Here are some that we have:

* Win a 5km race over rough terrain against the best human 5k runner.
* Physically assemble any LEGO set given the pieces and instructions.
* Be capable of winning an International Mathematics Olympiad Gold Medal (ignoring entry requirements). That is, solve mathematics problems with known solutions that are hard for the best high school students in the world, better than those students can solve them.
* Watch a human play any computer game a small number of times (say 5), then perform as well as human novices at the game without training more on the game. (The system can train on other games).
* Beat the best human players at Starcraft, with a human-like limit on moves per second.
* Translate a new language using unlimited films with subtitles in the new language, but the kind of training data we have now for other languages (e.g. same text in two languages for many languages and films with subtitles in many languages).
* Be about as good as unskilled human translation for most popular languages (including difficult languages like Czech, Chinese and Arabic).
* Answer tech support questions as well as humans can.
* Train to do image classification on half a dataset (say, ImageNet) then take the other half of the images, containing previously unseen objects, and separate them into the correct groupings (without the correct labels of course).
* See a small number of examples of a new object (say 10), then be able to recognize it in novel scenes as well as humans can.
* Reconstruct a 3d scene from a 2d image as reliably as a human can.
* Transcribe human speech with a variety of accents in a quiet environment as well as humans can.
* Routinely and autonomously prove mathematical theorems that are publishable in mathematics journals today.

Can you think of any interesting ones?


Mysteries of global hardware

*By Katja Grace, 7 March 2016*

*This blog post summarizes recent research on our [Global Computing Capacity page](http://aiimpacts.org/global-computing-capacity/). See that page for full citations and detailed reasoning.*

We recently investigated this intriguing puzzle:

>
> **FLOPS (then) apparently performed by all of the world’s computing hardware:** 3 x 1022 – 3 x 1024
>
>
>

(Support: Vipul Naik [estimated](https://intelligence.org/wp-content/uploads/2014/02/Naik-Distribution-of-Computation.pdf) 1022 – 1024 IPS by February 2014, and reports a long term growth rate of 85% per year for application-specific computing hardware, which made up 97% of hardware by 2007, suggesting total hardware should have tripled by now. FLOPS are roughly equivalent to IPS).

>
> **Price of FLOPS:** $3 x 10-9
>
>
>

[(Support: See our page on it)](http://aiimpacts.org/current-flops-prices/)

>
> **Implied value of global hardware:** $1014-1016
>
>
>

(Support: 3 x 1022 to 3 x 1024 \* $3 x 10-9= $1014-1016)

>
> **Estimated total global wealth:** $2.5 \* 1014
>
>
>

(Support: see for instance [Credit Suisse](https://www.credit-suisse.com/us/en/about-us/responsibility/news-stories/articles/news-and-expertise/2015/10/en/global-wealth-in-2015-underlying-trends-remain-positive.html))

>
> **Implication:** 40%-4,000% of global wealth is in the form of computing hardware.
>
>
>

***Question:*** *What went wrong?*

---

**Clues**

Could most hardware be in large-scale, unusually cheap, projects? Probably not – [our hardware price figures](http://aiimpacts.org/current-flops-prices/) include supercomputing prices. Also, [Titan](https://en.wikipedia.org/wiki/Titan_(supercomputer)) is a supercomputer made from GPUs and CPUs, and [doesn’t seem to be cheaper per computation](http://aiimpacts.org/current-flops-prices/) than the component GPUs and CPUs.

Could the global wealth figure be off? We get roughly the same anomaly when comparing global GDP figures and the value of computation used annually.

**Our solution**

We think the estimate of global hardware is the source of the anomaly. We think this because the amount that people apparently spend on hardware each year doesn’t seem like it would buy nearly this much hardware.

Annual hardware revenue seems to be around $300bn-$1,500bn recently.[1](https://aiimpacts.org/mysteries-of-global-hardware/#easy-footnote-bottom-1-649 "“In 2012, the worldwide computing hardware spending is expected at 418 billion U.S. dollars.” – Statista

Statista’s figure of ‘Forecast hardware spendings worldwide from 2013 to 2019 (in billion U.S. dollars)’ reports a 2013 figure of $987bn, increasing to $1075bn in 2015. It is unclear why these spending forecasts differ so much from Statista’s reported 2012 spending.

Statista also reports a prediction of 2016 hardware revenue at $409bn Euro, which is around $447bn USD. It looks like the prediction was made in 2012. Note that revenue is not identical to spending, but is probably a reasonable proxy.

For 2009, Reuters reports a substantially lower revenue figure than Statista, suggesting Statista figures may be systematically high, e.g. by being relatively inclusive:

“The global computer hardware market had total revenue of $193.2 billion in 2009, representing a compound annual growth rate (CAGR) of 5.4% for the period spanning 2005-2009.” – Research and Markets press release, Reuters,

Statista‘s figure indicates revenue of 296 billion Euros, or around $320 billion USD in 2009 (this is the same figure as for 2007, which may be the only number you can see without a subscription—so while it may look like we made an error here, we do have the figure for the correct year). This is around 50% more than the Research and Markets press release.

From these figures we estimate that spending on hardware in 2015 was $300bn-$1,500bn") Based on the [prices of FLOPS](http://aiimpacts.org/current-flops-prices/), (and making some assumptions, e.g. about how long hardware lasts) this suggests the total global stock of hardware can perform around 7.5 x 1019– 1.5 x 1021 FLOPS/year.[2](https://aiimpacts.org/mysteries-of-global-hardware/#easy-footnote-bottom-2-649 "See our page on this topic for all the citations and calculations") However the lower end of this range is below a relatively detailed estimate of global hardware made in 2007. It seems unlikely that the hardware base actually shrunk in recent years, so we push our estimate up to 2 x 1020 – 1.5 x 1021 FLOPS/year.

This is about 0.3%-1.9% of global GDP—a more plausible number, we think—so resolves the original problem. But a big reason Naik gave such high estimates for global hardware was that the last time someone measured it—between 1986 and 2007—computing hardware was growing very fast. General purpose computing was growing at 61% per year, and the application specific computers studied (such as GPUs) were growing at 86% per year. Application specific computers made up the vast majority too, so we might expect growth to progress at close to 86% per year.

However if global hardware is as low as we estimate, the growth rate of total computing hardware since 2007 has been 25% or less, much lower than in the previous 21 years. Which would present us with another puzzle: **what happened?**

We aren’t sure, but this is still our best guess for the solution to the original puzzle. Hopefully we will have time to look into this puzzle too, but for now I’ll leave interested readers to speculate.

---

**Added March 11 2016:** Assuming the 2007 hardware figures are right, how much of the world’s wealth was in hardware in 2007? Back then, GWP [was probably](https://web.archive.org/web/20080401214624/https://www.cia.gov/library/publications/the-world-factbook/fields/2001.html) about $66T (in 2007 dollars). According to [Hilbert & Lopez](http://ijoc.org/index.php/ijoc/article/view/1562/742), the world could then perform 2 x 1020 IPS, which is  2 x 1014 MIPS. According to [Muehlhauser & Rieber](https://intelligence.org/2014/05/12/exponential-and-non-exponential/#footnote_7_11027), hardware cost roughly $5 x 10-3/MIPS in 2007. Thus the total value of hardware would have been around $5 x 10-3/MIPS x 2 x 1014 MIPS = $1012 (a trillion dollars), or 1.5% of GWP.

---

[![By An employee of the Oak Ridge National Laboratory. - http://www.olcf.ornl.gov/titan/, Public Domain, https://commons.wikimedia.org/w/index.php?curid=22575721](http://aiimpacts.org/wp-content/uploads/2016/03/Titan_supercomputer_at_the_Oak_Ridge_National_Laboratory-1024x364.jpg)](http://aiimpacts.org/wp-content/uploads/2016/03/Titan_supercomputer_at_the_Oak_Ridge_National_Laboratory.jpg)**Titan Supercomputer.** *[By](https://commons.wikimedia.org/w/index.php?curid=22575721) an employee of the Oak Ridge National Laboratory.*

---


Global computing capacity

*[This page is out of date and its contents may have been inaccurate in 2015, in light of new information that we are yet to integrate. See [Computing capacity of all GPUs and TPUs](https://wiki.aiimpacts.org/doku.php?id=ai_timelines:hardware_and_ai_timelines:computing_capacity_of_all_gpus_and_tpus "ai_timelines:hardware_and_ai_timelines:computing_capacity_of_all_gpus_and_tpus") for a related and more up-to-date analysis.]*

Computing capacity worldwide was probably around 2 x 1020– 1.5 x 1021 FLOPS, at around the end of 2015.

Support
-------

We are not aware of recent, plausible estimates for hardware capacity.

Vipul Naik estimated global hardware capacity in February 2014, based on Hilbert & Lopez’s estimates for 1986-2007. He calculated that if all computers ran at full capacity, they would perform 10-1000 zettaFLOPS, i.e. 1022– 1024 FLOPS.[1](https://aiimpacts.org/global-computing-capacity/#easy-footnote-bottom-1-644 "“My current guesstimate for total computation being done would be 0.1-10 zettaFLOPS and current estimate for how much computation can be done if all computers ran at full capacity would be 10-1000 zettaFLOPS (but this would entail prohibitive energy costs and not be sustainable).”

– Vipul Naik, research notes") We think these are substantial overestimates, because producing so much computing hardware would cost more than 10% of gross world product (GWP), which is implausibly high. The most cost-efficient computing hardware we are aware of today are GPUs, which still cost about [$3/GFLOPS](http://aiimpacts.org/current-flops-prices/), or $1/GFLOPSyear if we assume hardware is used for around three years. This means maintaining hardware capable of 1022– 1024 FLOPS would cost at least $1013– $1015 per year. Yet [gross world product](https://en.wikipedia.org/wiki/Gross_world_product) (GWP) is only around $8 x 1013, so this would imply hardware spending constitutes more than 13% – 1300% of GWP. Even the lower end of this range seems implausible.[2](https://aiimpacts.org/global-computing-capacity/#easy-footnote-bottom-2-644 "One might wonder if the total stock of hardware seems high because hardware lasts for much longer than three years. This probably does not account for it, because the growth rate is high enough that even if substantially older hardware persisted, it would make up a small share of total hardware. For instance in 2007 the world’s computing capacity (for the general purpose and application specific computers Hilbert & Lopez estimated) was only 2 x 1020 IPS (similar to FLOPS), roughly 0.02%-2% of Naik’s estimate.

")

One way to estimate global hardware capacity ourselves is based on annual hardware spending. This is slightly complicated because hardware lasts for several years. So to calculate how much hardware exists in 2016, we would ideally like to know how much was bought in every preceding year, and also how much of each annual hardware purchase has already been discarded. To simplify matters, we will instead assume that hardware lasts for around three years.

It appears that very roughly $300bn-$1,500bn was spent on hardware in 2015.[3](https://aiimpacts.org/global-computing-capacity/#easy-footnote-bottom-3-644 "“In 2012, the worldwide computing hardware spending is expected at 418 billion U.S. dollars.” – Statista

Statista’s figure of ‘Forecast hardware spendings worldwide from 2013 to 2019 (in billion U.S. dollars)’ reports a 2013 figure of $987bn, increasing to $1075bn in 2015. It is unclear why these spending forecasts differ so much from Statista’s reported 2012 spending.

Statista also reports a prediction of 2016 hardware revenue at $409bn Euro, which is around $447bn USD. It looks like the prediction was made in 2012. Note that revenue is not identical to spending, but is probably a reasonable proxy.

For 2009, Reuters reports a substantially lower revenue figure than Statista, suggesting Statista figures may be systematically high, e.g. by being relatively inclusive:

“The global computer hardware market had total revenue of $193.2 billion in 2009, representing a compound annual growth rate (CAGR) of 5.4% for the period spanning 2005-2009.” – Research and Markets press release, Reuters,

Statista‘s figure indicates revenue of 296 billion Euros, or around $320 billion USD in 2009 (this is the same figure as for 2007, which may be the only number you can see without a subscription—so while it may look like we made an error here, we do have the figure for the correct year). This is around 50% more than the Research and Markets press release.

From these figures we estimate that spending on hardware in 2015 was $300bn-$1,500bn") We [previously estimated](http://aiimpacts.org/current-flops-prices/) that the cheapest available hardware (in April 2015) was around $3/GFLOPS. So if humanity spent $300bn-$1,500bn on hardware in 2015, and it was mostly the cheapest hardware, then the hardware we bought should perform around 1020 – 5 x 1020 FLOPS. If we multiply this by three to account for the previous two years’ hardware purchases still being around, we have about  3 x 1020– 1.5 x 1021 FLOPS.

This estimate is rough, and could be improved in several ways. Most likely, more hardware is being bought each year than the previous year. So approximating last years’ hardware purchase to this years’ will yield too much hardware. In particular, the faster global hardware is growing, the closer the total is to whatever humanity bought this year (that is, counterintuitively, if you think hardware is growing faster, you should suppose that there is *less of it* by this particular method of estimation). Furthermore, perhaps a lot of hardware is not the cheapest for various reasons. This too suggests there is less hardware than we estimated.

On the other hand, hardware may often last for more than three years (we don’t have a strong basis for our assumption there). And our prices are from early 2015, so hardware is likely somewhat cheaper now (in early 2016). Our guess is that overall these considerations mean our estimate should be lower, but probably by less than a factor of four in total. This suggests 7.5 x 1019 – 1.5 x 1021 FLOPS of hardware.

However [Hilbert & Lopez](http://ijoc.org/index.php/ijoc/article/view/1562/742) (2012) estimated that in 2007 the world’s computing capacity was around 2 x 1020IPS (similar to FLOPS) already, after constructing a detailed inventory of technologies.[4](https://aiimpacts.org/global-computing-capacity/#easy-footnote-bottom-4-644 "“It is written from the perspective of the results of our recent inventory of 60 technological categories between 1986 and 2007 (measured in bits and MIPS [million-instructions-per-second])…”

“…The combined capacity of both groups of computers has grown from 730 tera-IPS in 1986 (73010^12 instructions per second), over 22.5 peta-IPS in 1993 (22.510^15 instructions per second), to 1.8 exa-IPS in 2000 (1.810^18 IPS), and 196 exa-IPS in 2007 (or roughly 210^20 instructions per second).”

–  Hilbert & Lopez (2012)

") Their estimate does not appear to conflict with data about the global economy at the time.[5](https://aiimpacts.org/global-computing-capacity/#easy-footnote-bottom-5-644 "In 2007, GWP was probably about $66T (in 2007 dollars). According to Hilbert & Lopez, the world could then perform 2 x 1020 IPS, which is  2 x 1014 MIPS. According to Muehlhauser & Rieber, hardware cost roughly $5 x 10-3/MIPS in 2007. Thus the total value of hardware would have been around $5 x 10-3/MIPS x 2 x 1014 MIPS = $1012 (a trillion dollars), or 1.5% of GWP.") Growth is unlikely to have been negative since 2007, though Hilbert & Lopez may have overestimated. So we revise our estimate to 2 x 1020 – 1.5 x 1021 FLOPS for the end of 2015.

This still suggests that in the last nine years, the world’s hardware has grown by a factor of 1-7.5, implying a growth rate of 0%-25%. Even 25% would be quite low compared to growth rates between 1986 and 2007 according to [Hilbert & Lopez](http://ijoc.org/index.php/ijoc/article/view/1562/742) (2012), which were 61% for general purpose computing and 86% for the set of ASICs they studied (which in 2007 accounted for 32 times as much computing as general purpose computers).[6](https://aiimpacts.org/global-computing-capacity/#easy-footnote-bottom-6-644 "“The respective compound annual growth rates between 1986–2007 were 61% for general-purpose computations and 86% for application-specific computations, which is 10 and 14 times faster than global GDP during that period, respectively.”

Hilbert & Lopez (2012)

") However if we are to distrust estimates which imply hardware is a large fraction of GWP, then we must expect hardware growth has slowed substantially in recent years. For comparison, our estimates are around 2-15% of Naik’s lower bound, and suggest that hardware constitutes around 0.3%-1.9% of GWP.

Such large changes in the long run growth rate are surprising to us, and—if they are real—we are unsure what produced them. One possibility is that hardware prices have stopped falling so fast (i.e. Moore’s Law is ending for the price of computation). Another is that spending on hardware decreased for some reason, for instance because people stopped enjoying large returns from additional hardware. We think this question deserves further research.

Implications
------------

### Global computing capacity in terms of human brains

According to [different estimates](http://aiimpacts.org/brain-performance-in-flops/), the human brain performs the equivalent of between 3 x 1013 and 1025 FLOPS. The median estimate we know of is 1018 FLOPS. According to that median estimate and our estimate of global computing hardware, if the world’s entire computing capacity could be directed at running minds around as efficient as those of humans, we would have the equivalent of 200-1500 extra human minds.[7](https://aiimpacts.org/global-computing-capacity/#easy-footnote-bottom-7-644 "2 x 1020 /1018  = 2 x 102

1.5 x 1021/1018=1.5 x 103") That is, turning all of the world’s hardware into human-efficiency minds at present would increase the world’s population of minds by at most about 0.00002%. If we select the most favorable set of estimates for producing large numbers, turning all of the world’s computing hardware into minds as efficient as humans’ would produce around 50 million extra minds, increasing the world’s effective population by about 1%.[8](https://aiimpacts.org/global-computing-capacity/#easy-footnote-bottom-8-644 "1.5 x 1021 FLOPS of hardware divided by 3 x 1013 FLOPS/brain gives us 5 x 107 minds.")

![Figure: Projected number of human brains equivalent ](http://aiimpacts.org/wp-content/uploads/2016/02/Global-hardware-in-FLOPS-equivalent-brains.png)Figure: Projected number of human brains equivalent to global hardware under various assumptions. For brains, ‘small’ = 3 x 10^ 13, ‘median’ = 10^18, ‘large’ = 10^25. For ‘world hardware’, ‘high’ =2 x 10^20, ‘low’ = 1.5 x 10^21. ‘Growth’ is growth in computing hardware, the unlabeled default used in most projections is 25% per annum (our estimate above), ‘high’ = 86% per annum (the apparent growth rate in ASIC hardware in around 2007).

---


Coordinated human action as example of superhuman intelligence

Collections of humans organized into groups and institutions provide many historical examples of the creation and attempted control of intelligences that routinely outperform individual humans. A preliminary look at the available evidence suggests that individuals are often cognitively outperformed in head-to-head competition with groups of similar average intelligence. This article surveys considerations relevant to the topic and lays out what a plausible research agenda in this area might look like.

Background
----------

Humans are often organized into groups in order to perform tasks beyond the abilities of any single human in the group. Many such groups perform cognitive tasks. The history of forming such groups is long and varied, and provides some evidence about what new forms of superhuman intelligence might be like.

Some examples of humans cooperating on a cognitive task that no one member could perform include:

* Ten therapists can see ten times as many patients as one therapist can.
* A hospital can perform many more kinds of medical procedure and treat many more kinds of illness than any one person in the hospital.
* A team of friends on trivia night might be able to answer more questions than any one of them individually might.

How such institutions are formed, and the sensitivity of their behavior to starting conditions, may help us predict the behavior of similarly constituted AIs or systems of AIs. This information would be especially useful if control or value alignment problems have been solved in some cases, or to the extent that existing human institutions resemble superintelligences or constitute an intelligence explosion.

There are several reasons these kinds of groups may present only a limited analogy to digital artificial intelligence. For instance, humans have no software-hardware distinction, so physical measures such as fences that can control the spread of humans are not likely to be as reliable at controlling the spread of digital intelligences. An individual human cannot easily be separated into different cognitive modules, which limits the design flexibility of intelligences constructed from humans. More generally, AIs may be programmed in ways very different from the heuristics and algorithms executed by the human brain, so while human organizations may be a kind of superhuman intelligence, they are not necessarily representative of the broader space of possible superintelligences.

### Questions for further investigation:

* Do any human organizations have the characteristics of superintelligences that some AI researchers and futurists expect to cause an intelligence explosion with catastrophic consequences? If so, do we expect catastrophe from human organizations? If not, what distinguishes them from other, potential artificial intelligences?
* How similar is the problem of controlling institutional behavior to the value alignment problem with respect to powerful digital AIs? Are the expected consequences similar?
* Do control mechanisms require limiting the cognitive performance of groups, or are there control mechanisms that do not appear to degrade in effectiveness as the intelligence of the group increases?
* How relevant are the differences between human collective intelligence and digital AI?

Group vs individual performance
-------------------------------

Institutions are mainly relevant as an example of constructed intelligence if their intelligence is higher than that of humans, in some sense. This section examines reasons to believe this might be the case.

### Mechanisms for cognitive superiority of groups

We can think of several mechanisms by which a group might outperform individual humans on cognitive tasks, although this list is not comprehensive:

* **Aggregation** – A large number of people can often perform cognitive tasks at a higher rate than a single person performing the same tasks. For example, a large accounting firm ought to be able to perform more audits, or prepare more tax returns, than a single accountant. In practice, there are often impediments to work scaling linearly with the number of people involved, as noted in observations such as [Parkinson’s Law](https://en.wikipedia.org/wiki/Parkinson%27s_law).
* **Cognitive economies of scale**
	+ It is often less costly to teach someone how to perform a task than for them to figure it out on their own. Knowledge transfer between members of a group may therefore accelerate the learning process.
	+ Individuals with different skills can cooperate to produce things or quantities of things that no one person could have produced, through specialization and gains from trade. For example, [I, Pencil](http://www.econlib.org/library/Essays/rdPncl1.html) describes the large number of processes, each requiring a very different set of skills and procedures that it would take a long time to learn, to produce a single pencil.

* ****Model combination and adjustment****
	+ In groups solving problems, people can make different suggestions and identify one another’s incorrect suggestions, which may help the group avoid wasting time on blind alleys or adopting premature, incorrect solutions.
	+ The average of the individual estimates from a group of people is typically more reliably accurate than the estimate of any individual in the group, because random errors tend to cancel each other out. This is often called the “wisdom of crowds”.
	+ Groups of people can also coordinate by comparing predictions and accepting the claim the group finds most credible. Trivia teams typically use this strategy. Groups of people have also been pitted against individuals in chess games.
	+ Markets can be used to combine information from many individuals.

Further investigation on this topic could include:

* Generating a more comprehensive list of potential mechanisms by which institutions and groups may have a cognitive advantage, by examining the historical record, arguments, and experimental and case studies of individual vs group performance.
* Assessing which mechanisms can be shown to work, and how much group intelligence can exceed individual intelligence, by evaluating historical examples, case studies, and experimental studies.
* Assessing in which aspects of intelligence, if any, groups have not outperformed individuals.

### Evidence of cognitive superiority of groups

An incomplete survey of literature on collective intelligence found several measures where group performance, distinct from individual performance, has been explicitly evaluated:

* [Wooley et al. 2010](http://www.sebbm.es/archivos_tinymce/woolley2010.pdf) examined the performance of groups on tasks such as solving visual puzzles, brainstorming, making collective moral judgments, negotiating over limited resources, and playing checkers against a standardized computer opponent. The study found correlation between performance on different tasks, related more to the ability of members to coordinate than to the average or maximum intelligence of group members.
* [Shaw 1932](http://www.jstor.org/stable/1415351?seq=1#page_scan_tab_contents) compared the timed performance of individuals and four-person groups on simple spatial and logical reasoning problems, and verbal tasks (arranging a set of words to form the end of some text). The study found that on problems where anyone was able to solve them, groups substantially outperformed individuals, mostly by succeeding much more often than individuals did. No one was able to solve the last two problems, but the study did find that on those problems, suggestions rejected during the process of group problem-solving were predominantly incorrect suggestions rejected by someone other than the person who proposed them, which shows error-correction to be potentially an important part of the advantage of group cognition.
* [Thorndike 1938](http://psycnet.apa.org/journals/abn/33/3/409/) compared group and individual performance on vocabulary completion, limerick completion, and solving and making cross-word puzzle tests. Groups outperformed individuals on everything except making crossword puzzles.
* [Taylor and Faust 1952](http://psycnet.apa.org/journals/xge/44/5/360/) tested the ability of individuals, groups of two, and groups of four, to solve “twenty questions” style problems. Groups outperformed individuals, but larger groups did not outperform smaller groups.
* [Gurnee 1936](http://www.tandfonline.com/doi/abs/10.1080/00223980.1937.9917512?journalCode=vjrl20&#.VpwKGFMrKAw) compared individual and group performance at maze learning. Groups completed mazes faster and with fewer false moves.
* [Gordon 1924](http://psycnet.apa.org/journals/xge/7/5/398/) compared individual estimates of an object’s weight with the average of members of a group. The study found that group averages outperformed individual estimates, and that larger groups performed better than smaller groups.
* [McHaney et al. 2015](http://crx.sagepub.com/content/early/2015/09/29/0093650215607627.abstract) compared the performance of individuals, ad hoc groups, and groups with a prior history of working together, at detecting deception. The study found that groups with a prior history of working together outperform ad hoc groups, and refers to earlier literature that found no difference between the performance of individuals and that of ad hoc groups.

Mostly these studies appear to show groups outperforming individuals. We also found review articles referencing tens of other studies. We may follow up with a more comprehensive review of the evidence in this area in the future.

### Questions for further investigation:

* Which of the possible mechanisms for cognitive superiority of groups do human institutions demonstrate in practice? Do they have important advantages other than the ones enumerated?
* In what contexts has the difference between group and individual performance been measured? Are there measures on which large organizations do much better than a single human? On what kinds of tasks does group performance most exceed that of individuals? How are these groups constituted?
* Are there measures on which large organizations cannot be arbitrarily better than a single human? (These might still be things that an AI could do much better, and so where organizations are not a good analogue.) Are there measures for which large organizations have not yet even reached human level intelligence? (It is deprecatory to say something was “written by a committee.”)


Recently at AI Impacts

*By Katja Grace, 24 November 2015*

We’ve been working on a few longer term projects lately, so here’s an update in the absence of regular page additions.

### New researchers

Stephanie Zolayvar and John Salvatier have recently joined us, to try out research here.

Stephanie recently moved to Berkeley from Seattle, where she was a software engineer at Google. She is making sense of a recent spate of interviews with AI researchers (more below), and investigating purported instances of discontinuous progress. She also just made this [glossary of AI risk terminology](http://aiimpacts.org/ai-risk-terminology/).

John also recently moved to Berkeley from Seattle, where he was a software engineer at Amazon. He has been interviewing AI researchers with me, helping to design a new survey on AI progress, and evaluating different research avenues.

I’ve also been working on several smaller scale collaborative projects with other researchers.

### AI progress survey

We are making a survey, to help us ask AI researchers about AI progress and timelines. We hope to get answers that are less ambiguous and more current than [past timelines surveys](http://aiimpacts.org/ai-timeline-surveys/). We also hope to learn about the landscape of progress in more detail than we have, to help guide our research.

### AI researcher interviews

We have been having in-depth conversations with AI researchers about AI progress and predictions of the future. This is partly to inform the survey, but mostly because there are lots of questions where we want elaborate answers from at least one person, instead of hearing everybody’s one word answers to potentially misunderstood questions. We plan to put up notes on these conversations soon.

### Bounty submissions

Ten people have submitted many more entries to [our bounty experiment](http://aiimpacts.org/ai-impacts-research-bounties/). We are investigating these, but are yet to verify that any of them deserve a bounty. Our request was for examples of discontinuous progress, or very early action on a risk. So far the more lucrative former question has been substantially more popular.

### Glossary

We just put up a [glossary of AI safety terms](http://aiimpacts.org/ai-risk-terminology/). Having words for things often helps in thinking about them, so we hope to help in the establishment of words for things. If you notice important words without entries, or concepts without words, please send them our way.


Glossary of AI Risk Terminology and common AI terms

Terms
-----

### A

#### **AI timeline**

An expectation about how much time will lapse before important AI events, especially the advent of *[human-level AI](http://aiimpacts.org/human-level-ai/)* or a similar milestone. The term can also refer to the actual periods of time (which are not yet known), rather than an expectation about them.

#### **Artificial General Intelligence** (also, ***AGI***)

Skill at performing intellectual tasks across at least the range of variety that a human being is capable of. As opposed to skill at certain specific tasks (‘narrow’ AI). That is, synonymous with the more ambiguous *Human-Level AI* for some meanings of the latter*.*

#### **Artificial Intelligence** (also, ***AI***)

Behavior characteristic of human minds exhibited by man-made machines, and also the area of research focused on developing machines with such behavior. Sometimes used informally to refer to *human-level AI* or another strong form of AI not yet developed.

#### **Associative value accretion**

A hypothesized approach to value learning in which the AI acquires values using some machinery for synthesizing appropriate new values as it interacts with its environment, inspired by the way humans appear to acquire values (Bostrom 2014, p189-190)[1](https://aiimpacts.org/ai-risk-terminology/#easy-footnote-bottom-1-358 "Bostrom, Nick. Superintelligence: Paths, Dangers, Strategies. 1st edition. Oxford: Oxford University Press, 2014.").

#### **Anthropic capture**

A hypothesized control method in which the AI thinks it might be in a simulation, and so tries to behave in ways that will be rewarded by its simulators (Bostrom 2014 p134).

#### **Anthropic reasoning**

Reaching beliefs (posterior probabilities) over states of the world and your location in it, from priors over possible physical worlds (without your location specified) and evidence about your own situation. For an example where this is controversial, see [The Sleeping Beauty Problem](https://en.wikipedia.org/wiki/Sleeping_Beauty_problem). For more on the topic and its relation to AI, see [here](https://meteuphoric.wordpress.com/anthropic-principles/).

#### **Augmentation**

An approach to obtaining a superintelligence with desirable motives that consists of beginning with a creature with desirable motives (eg, a human), then making it smarter, instead of designing good motives from scratch (Bostrom 2014, p142).

### B

#### **Backpropagation**

A fast method of computing the derivative of cost with respect to different parameters in a network, allowing for training neural nets through gradient descent. See [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/chap2.html)[2](https://aiimpacts.org/ai-risk-terminology/#easy-footnote-bottom-2-358 "Nielsen, Michael A. “Neural Networks and Deep Learning,” 2015. http://neuralnetworksanddeeplearning.com.") for a full explanation.

#### **Boxing**

A control method that consists of constructing the AI’s environment so as to minimize interaction between the AI and the outside world. (Bostrom 2014, p129).

### C

#### **Capability control methods**

Strategies for avoiding undesirable outcomes by limiting what an AI can do (Bostrom 2014, p129).

#### **Cognitive enhancement**

Improvements to an agent’s mental abilities.

#### **Collective superintelligence**

“A system composed of a large number of smaller intellects such that the system’s overall performance across many very general domains vastly outstrips that of any current cognitive system” (Bostrom 2014, p54).

#### **Computation**

A sequence of mechanical operations intended to shed light on something other than this mechanical process itself, through an established relationship between the process and the object of interest.

#### **The common good principle**

“Superintelligence should be developed only for the benefit of all of humanity and in the service of widely shared ethical ideals” (Bostrom 2014, p254).

#### **Crucial consideration**

An idea with the potential to change our views substantially, such as by reversing the sign of the desirability of important interventions.

### D

#### **Decisive strategic advantage**

Strategic superiority (by technology or other means) sufficient to enable an agent to unilaterally control most of the resources of the universe.

#### **Direct specification**

An approach to the control problem in which the programmers figure out what humans value, and code it into the AI (Bostrom 2014, p139-40).

#### **Domesticity**

An approach to the control problem in which the AI is given goals that limit the range of things it wants to interfere with (Bostrom 2014, p140-1).

### E

#### **Emulation modulation**

Starting with brain emulations with approximately normal human motivations (see ‘Augmentation’), and modifying their motivations using drugs or digital drug analogs.

#### **Evolutionary selection approach to value learning**

A hypothesized approach to the value learning problem which obtains an AI with desirable values by iterative selection, the same way evolutionary selection produced humans  (Bostrom 2014, p187-8).

#### **Existential risk**

Risk of an adverse outcome that would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential [(Bostrom 2002)](http://www.nickbostrom.com/existential/risks.html)

### F

#### **Feature**

A dimension in the vector space of activations in a single layer of a neural network (i.e. a neuron activation or linear combination of activations of different neurons)

#### **First principal-agent problem**

The well-known problem faced by a sponsor wanting an employee to fulfill their wishes (usually called ‘the principal agent problem’).

### G

#### **Genie**

An AI that carries out a high level command, then waits for another (Bostrom 2014, p148).

### H

#### **[Hardware overhang](http://aiimpacts.org/hardware-overhang/)**

A situation where large amounts of hardware being used for other purposes become available for AI, usually posited to occur when AI reaches human-level capabilities.

#### **Human-level AI**

An AI that matches human capabilities in virtually every domain of interest.  Note that this term is used ambiguously; see [our page on human-level AI](http://aiimpacts.org/human-level-ai/).

#### **Human-level hardware**

Hardware that matches the information-processing ability of the human brain.

#### **Human-level software**

Software that matches the algorithmic efficiency of the human brain, for doing the tasks the human brain does.

### I

#### **Impersonal perspective**

The view that one should act in the best interests of everyone, including those who may be brought into existence by one’s choices (see Person-affecting perspective).

#### **Incentive methods**

Strategies for controlling an AI that consist of setting up the AI’s environment such that it is in the AI’s interest to cooperate. e.g. a social environment with punishment or social repercussions often achieves this for contemporary agents (Bostrom 2014, p131).

#### **Incentive wrapping**

Provisions in the goals given to an AI that allocate extra rewards to those who helped bring the AI about  (Bostrom 2014, p222-3).

#### **Indirect normativity**

An approach to the control problem in which we specify a way to specify what we value, instead of specifying what we value directly (Bostrom, p141-2).

#### **Instrumental convergence thesis**

We can identify ‘convergent instrumental values’. That is, subgoals that are useful for a wide range of more fundamental goals, and in a wide range of situations (Bostrom 2014, p109).

#### **Intelligence explosion**

A hypothesized event in which an AI rapidly improves from ‘relatively modest’ to superhuman level (usually imagined to be as a result of recursive self-improvement).

### M

#### **Macrostructural development accelerator**

An imagined lever used in thought experiments which slows the large scale features of history (e.g. technological change, geopolitical dynamics) while leaving the small scale features the same.

#### **Mind crime**

The mistreatment of morally relevant computations.

#### **Moore’s Law**

Any of several different consistent, many-decade patterns of exponential improvement that have been observed in digital technologies. The classic version concerns the number of transistors in a dense integrated circuit, which was observed to be doubling around every year when the ‘law’ was formulated in [1965](https://en.wikipedia.org/wiki/Moore%27s_law). [Price-Performance Moore’s Law](https://aiimpacts.org/ai-risk-terminology/#Price-Performance_Moores_Law) is often relevant to AI forecasting.

#### **Moral rightness (MR) AI**

An AI which seeks to do what is morally right.

#### **Motivational scaffolding**

A hypothesized approach to value learning in which the seed AI is given simple goals, and these goals are replaced with more complex ones once it has developed sufficiently sophisticated representational structure (Bostrom 2014, p191-192).

#### **Multipolar outcome**

A situation after the arrival of superintelligence in which no single agent controls most of the resources.

### O

#### **Optimization power**

The strength of a process’s ability to improve systems.

#### **Oracle**

An AI that only answers questions (Bostrom 2014, p145).

#### **Orthogonality thesis**

Intelligence and final goals are orthogonal: more or less any level of intelligence could in principle be combined with more or less any final goal.

### P

#### **Person-affecting perspective**

The view that one should act in the best interests of everyone who already exists, or who will exist independent of one’s choices (see Impersonal perspective).

#### **Perverse instantiation**

A solution to a posed goal (eg, make humans smile) that is destructive in unforeseen ways (eg, paralyzing face muscles in the smiling position).

#### **Price-Performance Moore’s Law**

The [observed pattern](http://aiimpacts.org/trends-in-the-cost-of-computing/) of relatively consistent, long term, exponential price decline for computation.

#### **Principle of differential technological development**

“Retard the development of dangerous and harmful technologies, especially the ones that raise the level of existential risk; and accelerate the development of beneficial technologies, especially those that reduce the existential risk posed by nature or by other technologies” (Bostrom 2014, p230).

#### **Principle of epistemic deference**

“A future superintelligence occupies an epistemically superior vantage point: it’s beliefs are (probably, on most topics) more likely than our to be true.  We should therefore defer to the superintelligence’s position whenever feasible” (Bostrom 2014, p226).

### Q

#### **Quality superintelligence**

“A system that is at least as fast as a human mind and vastly qualitatively smarter” (Bostrom 2014, p56).

### R

#### **Recalcitrance**

How difficult a system is to improve.

#### **Recursive self-improvement**

The envisaged process of AI (perhaps a seed AI) iteratively improving itself.

#### **Reinforcement learning approach to value learning**

A hypothesized approach to value learning in which the AI is rewarded for behaviors that more closely approximate human values (Bostrom 2014, p188-9).

### S

#### **Second principal-agent problem**

The emerging problem of a developer wanting their AI to fulfill their wishes.

#### **Seed AI**

A modest AI which can bootstrap into an impressive AI by improving its own architecture.

#### **Singleton**

An agent that is internally coordinated and has no opponents.

#### **Sovereign**

An AI that acts autonomously in the world, in pursuit of potentially long range objectives (Bostrom 2014, p148).

#### **Speed superintelligence**

“A system that can do all that a human intellect can do, but much faster” (Bostrom 2014, p53).

#### **State risk**

A risk that comes from being in a certain state, such that the amount of risk is a function of the time spent there. For example, the state of not having the technology to defend from asteroid impacts carries risk proportional to the time we spend in it.

#### **Step risk**

A risk that comes from making a transition. Here the amount of risk is not a simple function of how long the transition takes.  For example, traversing a minefield is not safer if done more quickly.

#### **Stunting**

A control method that consists of limiting the AI’s capabilities, for instance as by limiting the AI’s access to information (Bostrom 2014, p135).

#### **Superintelligence**

Any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest (Bostrom 2014, p22).

### T

#### **Takeoff**

The event of the emergence of a superintelligence, often characterized by its speed: ‘slow takeoff’ takes decades or centuries, ‘moderate takeoff’ takes months or years and ‘fast takeoff’ takes minutes to days.

#### **Technological completion conjecture**

If scientific and technological development efforts do not cease, then all important basic capabilities that could be obtained through some possible technology will be obtained (Bostrom 2014, p127).

#### **Technology coupling**

A predictable timing relationship between two technologies, such that hastening of the first technology will hasten the second, either because the second is a precursor or because it is a natural consequence (Bostrom 2014, p236-8) e.g. brain emulation is plausibly coupled to ‘neuromorphic’ AI, because the understanding required to emulate a brain might allow one to more quickly create an AI on similar principles.

#### **Tool AI**

An AI that is not ‘like an agent’, but like a more flexible and capable version of contemporary software. Most notably perhaps, it is not goal-directed (Bostrom 2014, p151).

### U

#### Utility function

A mapping from states of the world to real numbers (‘utilities’), describing an entity’s degree of preference for different states of the world. Given the choice between two lotteries, the entity prefers the lottery with the highest ‘expected utility’, which is to say, sum of utilities of possible states weighted by the probability of those states occurring.

### V

#### **Value learning**

An approach to the value loading problem in which the AI learns the values that humans want it to pursue (Bostrom 2014, p207).

#### **Value loading problem**

The problem of causing the AI to pursue human values (Bostrom 2014, p185).

### W

#### **Wise-Singleton Sustainability Threshold**

A capability set exceeds the wise-singleton threshold if and only if a patient and existential risk-savvy system with that capability set would, if it face no intelligent opposition or competition, be able to colonize and re-engineer a large part of the accessible universe (Bostrom 2014, p100).

#### **Whole-brain emulation**

Machine intelligence created by copying the computational structure of the human brain.

#### **Word embedding**

A mapping of words to high-dimensional vectors that has been trained to be useful in a word task such that the arrangement of words in the vector space is meaningful. For instance, words near one other in the vector-space are related, and similar relationships between different pairs of words correspond to similar vectors between them, so that e.g. if E(x) is the vector for the word ‘x’, then E(king) – E(queen) ≈ E(woman) – E(man). Word embeddings are explained in more detail [here](https://colah.github.io/posts/2014-07-NLP-RNNs-Representations/).

Notes
-----


AI timelines and strategies

*AI Impacts sometimes invites guest posts from fellow thinkers on the future of AI. These are not intended to relate closely to our current research, nor to necessarily reflect our views. However we think they are worthy contributions to the discussion of AI forecasting and strategy.*

*This is a guest post by Sarah Constantin, 20 August 2015*

One frame of looking at AI risk is the “geopolitical” stance. Who are the major players who might create risky strong AI? How could they be influenced or prevented from producing existential risks? How could safety-minded institutions gain power or influence over the future of AI? What is the correct strategy for reducing AI risk?

The correct strategy depends sharply on the timeline for when strong AI is likely to be developed. Will it be in 10 years, 50 years, 100 years or more? This has implications for AI safety research. If a basic research program on AI safety takes 10-20 years to complete and strong AI is coming in 10 years, then research is relatively pointless. If basic research takes 10-20 years and strong AI is coming more than 100 years from now (if at all), then research can wait. If basic research takes 10-20 years and strong AI is coming in around 50 years, then research is a good idea.

Another relevant issue for AI timelines and strategies is the boom-and-bust cycle in AI. Funding for AI research and progress on AI has historically fluctuated since the 1960s, with roughly 15 years between “booms.” The timeline between booms may change in the future, but fluctuation in investment, research funding, and popular attention seems to be a constant in scientific/technical fields.

Each AI boom has typically focused on a handful of techniques (GOFAI in the 1970’s, neural nets and expert systems in the 1980’s) which promised to deliver strong AI but eventually ran into limits and faced a collapse of funding and investment. The current AI boom is primarily focused on massively parallel processing and machine learning, particularly deep neural nets.

This is relevant because institutional and human capital is lost between booms. While leading universities can survive for centuries, innovative companies are usually only at their peak for a decade or so. It is unlikely that the tech companies doing the most innovation in AI during one boom will be the ones leading subsequent booms. (We don’t usually look to 1980’s expert systems companies for guidance on AI today.) If there were to be a Pax Googleiana lasting 50 years, it might make sense for people concerned with AI safety to just do research and development within Google. But the history of the tech industry suggests that’s not likely. Which means that any attempt to influence long-term AI risk will need to survive the collapse of current companies and the end of the current wave of popularity of AI.

The “extremely short-term AI risk scenario” (of strong AI arising within a decade) is not a popular view among experts; most contemporary surveys of AI researchers predict that strong AI will arise sometime in the mid-to-late 21st century. If we take the view that strong AI in the 2020’s is vanishingly unlikely (which is more “conservative” than the results of most AI surveys, but may be more representative of the mainstream computer science view), then this has various implications for AI risk strategy that seem to be rarely considered explicitly.

In the “long-term AI risk scenario”, there will be at least one “AI winter” before strong AI is developed. We can expect a period (or multiple periods) in the future where AI will be poorly funded and popularly discredited. We can expect that there are one or more jumps in innovation that will need to occur before human-level AI will be possible. And, given the typical life cycle of corporations, we can expect that if strong AI is developed, it will probably be developed by an institution that does not exist yet.

In the “long-term AI risk scenario”, there will probably be time to develop at least some theory of AI safety and the behavior of superintelligent agents. Basic research in computer science (and perhaps neuroscience) may well be beneficial in general from an AI risk perspective. If research on safety can progress during “AI winters” while progress on AI in general halts, then winters are particularly good news for safety. In this long-term scenario, there is no short-term imperative to cease progress on “narrow AI”, because contemporary narrow AI is almost certainly not risky.

In the “long-term AI risk scenario”, another important goal besides basic research is to send a message to the future. Today’s leading tech CEOs will not be facing decisions about strong AI; the critical decisionmakers may be people who haven’t been born yet, or people who are currently young and just starting their careers. Institutional cultures are rarely built to last decades. What can we do today to ensure that AI safety will be a priority decades from now, long after the current wave of interest in AI has come to seem faddish and misguided?

The mid- or late 21st century may be a significantly different place than the early 21st century. Economic and political situations fluctuate. The US may no longer be the world’s largest economy. Corporations and universities may look very different. Imagine someone speculating about artificial intelligence in 1965 and trying to influence the world of 2015. Trying to pass laws or influence policy at leading corporations in 1965 might not have had a lasting effect (this would be a useful historical topic to investigate in more detail.)

And what if the next fifty years looks more like the cataclysmic first half of the 20th century than the comparatively stable second half of the 20th century? How could a speculative thinker of 1895 hope to influence the world of 1945?

Educational and cultural goals, broadly speaking, seem relevant in this scenario. It will be important to have a lasting influence on the intellectual culture of future generations.

For instance: if fields of theoretical computer science relevant for AI risk are developed and included in mainstream textbooks, then the CS majors of 2050 who might grow up to build strong AI will know about the concerns being raised today as more than a forgotten historical curiosity. Of course, they might not be CS majors, and perhaps they won’t even be college students. We have to think about robust transmission of information.

In the “long-term AI risk scenario”, the important task is preparing future generations of AI researchers and developers to avoid dangerous strong AI. This means performing and disseminating and teaching basic research in new theoretical fields necessary for understanding the behavior of superintelligent agents.

A “geopolitical” approach is extremely difficult if we don’t know who the players will be. We’d like the future institutions that will eventually develop strong AI to be run and staffed by people who will incorporate AI safety into their plans. This means that a theory of AI safety needs to be developed and disseminated widely.

Ultimately, long-term AI strategy bifurcates, depending on whether the future of AI is more “centralized” or “decentralized.”

In a “centralized” future, a small number of individuals, perhaps researchers themselves, contribute most innovation in AI, and the important mission is to influence them to pursue research in helpful rather than harmful directions.

In a “decentralized” future, progress in AI is spread over a broad population of institutions, and the important mission is to develop something like “industry best practices” — identifying which engineering practices are dangerous and instituting broadly shared standards that avoid them. This may involve producing new institutions focused on safety.

Basic research is an important prerequisite for both the “centralized” and “decentralized” strategies, because currently we do not know what kinds of progress in AI (if any) are dangerous.

The “centralized” strategy means promoting something like an intellectual culture, or philosophy, among the strongest researchers of the future; it is something like an educational mission. We would like future generations of AI researchers to have certain habits of mind: in particular, the ability to reason about the dramatic practical consequences of abstract concepts. The discoverers of quantum mechanics were able to understand that the development of the atomic bomb would have serious consequences for humanity, and to make decisions accordingly. We would like the future discoverers of major advances in AI to understand the same. This means that today, we will need to communicate (through books, schools, and other cultural institutions, traditional and new) certain intellectual and moral virtues, particularly to the brightest young people.

The “decentralized” strategy will involve taking the theoretical insights from basic AI research and making them broadly implementable. Are some types of “narrow AI” particularly likely to lead to strong AI? Are there some precautions which, on the margin, make harmful strong AI less likely? Which kinds of precautions are least costly in immediate terms and most compatible with the profit and performance needs of the tech industry? To the extent that AI progress is decentralized and incremental, the goal is to ensure that it is difficult to go very far in the wrong direction. Once we know what we mean by a “wrong direction”, this is a matter of building long-term institutions and incentives that shape AI progress towards beneficial directions.

The assumption that strong AI is a long-term rather than a short-term risk affects strategy significantly. Influencing current leading players is not particularly important; promoting basic research is very important; disseminating information and transmitting culture to future generations, as well as building new institutions, is the most effective way to prepare for AI advances decades from now.

In the event that AI never becomes a serious risk, developing institutions and intellectual cultures that can successfully reason about AI is still societally valuable. The skill (in institutions and individuals) of taking theoretical considerations seriously and translating them into practical actions for the benefit of humanity is useful for civilizational stability in general. What’s important is recognizing that this is a long-term strategy — i.e. thinking more than ten years ahead. Planning for future decades looks different from taking advantage of the current boom in funding and attention for AI and locally hill-climbing.

*Sarah Constantin blogs at [Otium](https://srconstantin.wordpress.com/). She recently graduated from Yale with a PhD in mathematics.*


Introducing research bounties

*By Katja Grace, 7 August 2015*

Sometimes we like to experiment with novel research methods and formats. Today we are introducing ‘[AI Impacts Research Bounties](http://aiimpacts.org/ai-impacts-research-bounties/)‘, in which you get money if you send us inputs to some of our research.

To start, we have two bounties: one for showing us instances of [abrupt technological progress](http://aiimpacts.org/cases-of-discontinuous-technological-progress/), and one for pointing us to instances of people acting to avert risks decades ahead of time. Rewards currently range from $20 to $500, and anyone can enter. We may add more bounties, or adjust prices, according to responses. We welcome feedback on any aspect of this experiment.

Thanks to John Salvatier for ongoing collaboration on this project.


AI Impacts research bounties

We are offering rewards for several inputs to our research, described below. These offers have no specific deadline except where noted. We may modify them or take them down, but will give at least one week’s notice here unless there is strong reason not to. To submit an entry, email katja@intelligence.org. There is currently a large backlog of entries to check, so new entries will not receive a rapid response.

### 1. An example of discontinuous technological progress ($50-$500)

*This bounty offer is no longer available after 3 November 2016.*

We are interested in finding more examples of large discontinuous technological progress to add to [our collection](http://aiimpacts.org/cases-of-discontinuous-technological-progress/). We’re offering a bounty of around $50-500 per good example.

We currently know of [two good examples](http://aiimpacts.org/cases-of-discontinuous-technological-progress/) (and one moderate example):

1. **Nuclear weapons** discontinuously increased the [relative effectiveness](https://en.wikipedia.org/wiki/Relative_effectiveness_factor) of explosives.
2. **High temperature superconductors**led to a dramatic increase in the highest temperature at which superconducting was possible.

To assess discontinuity, we’ve been using “number of years worth of progress at past rates”, as measured by any relevant metric of technological progress. For example, the discovery of nuclear weapons was equal to about 6,000 years worth of previous progress in the relative effectiveness of explosives. However, we are also interested in examples that seem intuitively discontinuous, even if they don’t exactly fit the criteria of being a large number of year’s progress in one go.

Things that make examples better:

1. **Size:** Better examples represent larger changes. More than 20 times normal annual progress is ideal.
2. **Sharpness:** Better examples happened over shorter periods. Over less than a year is ideal.
3. **Breadth:** Metrics that measure larger categories of things are better. For example, fast adoption curves for highly specific categories (say a particular version of some software) is much less interesting than fast adoption curves for much broader categories (say a whole category of software).
4. **Rarity:**As we receive more examples, the interestingness of each one will tend to decline.

AI Impacts is willing to pay more for better examples. Basically we will judge how interesting your example is and then reward you based on that. We will accept examples that violate our stated preferences but satisfy the spirit of the bounty. Our guess is that we would pay about $500 for another example as good nuclear weapons.

**How to enter:** all that is necessary to submit an example is to email us a paragraph describing the example, along with sources to verify your claims (such sources are likely to involve at least one time series of success on a particular metric). Note that an example should be of the form ‘A caused abrupt progress in metric B’. For instance, ‘The boliolicopter caused abrupt progress in the maximum rate of fermblangling at sub-freezing temperatures’.

### 2. An example of early action on a risk ($20-$100)

*This bounty offer is no longer available after 3 November 2016.*

**We want:** a one sentence description of a case where at least one person acted to avert a risk that was least fifteen years away, along with a link or citation supporting the claim that the action preceded the risk by at least fifteen years.

**We will give:** up $100, with higher sums for examples that are better according to our judgment (see criteria for betterness below), and which we don’t already know about. We might go over $100 for exceptionally good examples.

**Further details**

Examples are better if:

1. **The risk is more novel:** relatively similar problems have not arisen before, and would probably not arise sooner than fifteen years in the future. e.g. Poverty in retirement is a risk people often prepare for more than fifteen year before it befalls them, however it is not very novel because other people already face an essentially identical risk, and have done many times before.
2. **The solution is more specific:** the action taken would not be nearly as useful if the risk disappeared. e.g. Saving money to escape is a reasonable solution to expecting your country to face civil war soon. However saving money is fairly useful in any case, so this solution is not very specific.
3. **We haven’t received a lot of examples:** as we collect more examples, the value of each one will tend to decline.

Some examples:

1. **Leo Szilard’s secret nuclear patent**: the threat of nuclear weapons was quite novel. It’s unclear when Szilard expected such weapons, but quite plausibly at least fifteen years later in 1934. The secret patent does not seem broadly useful, though useful for encouraging more local nuclear research, which is somewhat more broadly useful than secrecy per se. More details in [this report](https://intelligence.org/files/SzilardNuclearWeapons.pdf). This is a reasonably good example.
2. **The Asilomar Conference on recombinant DNA**: the risk of was arguably quite novel (genetically engineered pandemics), and the solution was reasonably specific (safety rules for dealing with recombinant DNA). However the risks people were concerned about were immediate, rather than decades hence. More details [here](https://intelligence.org/2015/06/30/new-report-the-asilomar-conference-a-case-study-in-risk-mitigation/). This is not a good example.

Evidence that the example is better in the above ways is also welcome, though we reserve the right not to explore it fully.


Time flies when robots rule the earth

*By Katja Grace, 28 July 2015*

This week Robin Hanson is finishing off his much anticipated book, [The Age of Em: Work, Love and Life When Robots Rule the Earth](http://www.overcomingbias.com/2015/03/oxford-to-publish-the-age-of-em.html). He recently told me that it would be helpful to include rough numbers for the brain’s memory and computing capacity in the book, so I agreed to prioritize finding the ones AI Impacts didn’t already have. Consequently we just put up [new](http://aiimpacts.org/costs-of-information-storage/) [pages](http://aiimpacts.org/information-storage-in-the-brain/) [about](http://aiimpacts.org/cost-of-human-level-information-storage/) information storage in the brain. We also made a [summary](http://aiimpacts.org/costs-of-human-level-hardware/) of related ‘human-level’ hardware pages, and an [index](http://aiimpacts.org/index-of-hardware-articles/) of pages about hardware in general.

Robin’s intended use for these numbers is interesting. The premise of his book is that one day (perhaps in the far future) human minds might be [emulated](https://en.wikipedia.org/wiki/Mind_uploading) on computers, and that this would produce a society that is somewhere between recognizably human and predictably alien. Robin’s project is a detailed account of life in such a society, as far as it can be discerned by peering through social science and engineering theory.

One prediction Robin makes is that these emulated minds (’ems’) will run at a wide variety of speeds, depending on their purposes and the incentives. So some ems will have whole lifetimes while others are getting started on a single thought. Robin wanted to know how slow the very slowest ems would run. And for this,  he wanted to know much memory the brain uses, and how much computing it does.

His reasoning is as follows. The main costs of running emulations are computing hardware and memory. If Anna is running twice as fast as Ben, then Anna needs about twice as much computing power to run, which will cost about twice as much. However Anna still uses around as much memory as Ben to store the contents of her brain over time. So if most of the cost of an emulation is in computation, then halving the speed would halve the cost, and would often be worth it. But once an emulation is moving so slowly that memory becomes the main cost, slowing down by half makes little difference, and soon stops being worth it. So the slowest emulations should run at around the speed at which computing hardware and memory contribute similarly to cost.

[Hardware](http://aiimpacts.org/trends-in-the-cost-of-computing/) and [memory](http://aiimpacts.org/costs-of-information-storage/) costs have been falling at roughly similar rates in the past, so if this continues, then the ratio between their costs now is a reasonable (if noisy) predictor of their ratio in several decades time. Given [our numbers](http://aiimpacts.org/costs-of-human-level-hardware/), Robin estimates that the slowest emulations will operate at between a one hundred trillionth of human speed and one millionth of human speed, with a middle estimate of one tenth of a billionth of human speed.

At these rates, immortality looks a lot like dying. If you had been experiencing the world at these speeds since the beginning of the universe, somewhere between an hour and a thousand years would seem to have passed, with a middle estimate of a year. Even if the em economy somehow lasts for a thousand years, running this slowly would mean immediately jumping into whatever comes next.

These things are rough of course, but it seems pretty cool to me that we can make reasonable guesses at all about such exotic future scenarios, using clues from our everyday world, like the prevailing prices of hard disks and memory.

If you are near Berkeley, CA and want to think more about this kind of stuff, or in this kind of style, remember you can come and meet Robin and partake in more economic futurism at our event [this Thursday](http://aiimpacts.org/event-exercises-in-economic-futurism/). We expect a good number of attendees already, but could squeeze in a few more.

*Image: [Robin Hanson in a field](https://commons.wikimedia.org/wiki/File:Robin_Hanson_in_a_field.jpg), taken by Katja Grace*


Costs of human-level hardware

Computing hardware which is equivalent to the brain –

* in terms of FLOPS probably costs between $1 x 105 and $3 x 1016, or $2/hour-$700bn/hour.
* in terms of TEPS probably costs $200M – $7B, or or $4,700 – $170,000/hour (including energy costs in the hourly rate).
* in terms of secondary memory probably costs $300-3,000, or $0.007-$0.07/hour.

Details
-------

### Partial costs

#### Computation

*Main articles: [Brain performance in FLOPS](http://aiimpacts.org/brain-performance-in-flops/), [Current FLOPS prices](http://aiimpacts.org/current-flops-prices/), [Trends in the costs of computing](http://aiimpacts.org/trends-in-the-cost-of-computing/)*

[FLoating-point Operations Per Second](https://en.wikipedia.org/wiki/FLOPS) (FLOPS) is a measure of computer performance that emphasizes computing capacity. The human brain is estimated to perform between 1013.5 and 1025FLOPS. Hardware [currently costs](http://aiimpacts.org/current-flops-prices/) around $3 x 10-9/FLOPS, or $7 x 10-14/FLOPShour. This makes the current price of hardware which has equivalent computing capacity to the human brain between $1 x 105 and $3 x 1016, or $2/hour-$700bn/hour if hardware is used for five years.

The price of FLOPS [has probably](http://aiimpacts.org/trends-in-the-cost-of-computing/) decreased by a factor of ten roughly every four years in the last quarter of a century.

#### Communication

*Main articles: [Brain performance in TEPS](http://aiimpacts.org/brain-performance-in-teps/), [The cost of TEPS](http://aiimpacts.org/cost-of-teps/)*

[Traversed Edges Per Second](https://en.wikipedia.org/wiki/Traversed_edges_per_second) (TEPS) is a measure of computer performance that emphasizes communication capacity. The human brain [is estimated](http://aiimpacts.org/brain-performance-in-teps/) to perform at 0.18 – 6.4 x 105 GTEPS. Communication capacity [costs](http://aiimpacts.org/cost-of-teps/) around $11,000/GTEP or $0.26/GTEPShour in 2015, when amortized over five years and combined with energy costs. This makes the current price of hardware which has equivalent communication capacity to the human brain around $200M – $7B in total, or $4,700 – $170,000/hour including energy costs.

[We estimate](http://aiimpacts.org/cost-of-teps/) that the price of TEPS falls by a factor of ten every four years, based the relationship between TEPS and FLOPS.

#### Information storage

*Main articles: [Information storage in the brain](http://aiimpacts.org/information-storage-in-the-brain/), [Costs of information storage](http://aiimpacts.org/costs-of-information-storage/), [Costs of human-level information storage](http://aiimpacts.org/cost-of-human-level-information-storage/)*

[Computer memory](https://en.wikipedia.org/wiki/Computer_memory) comes in primary and secondary forms. Primary memory (e.g. RAM) is intended to be accessed frequently, while secondary memory is slower to access but has higher capacity. Here we estimate the secondary memory requirements ofthe brain. The human brain [is estimated](http://aiimpacts.org/information-storage-in-the-brain/) to store around 10-100TB of data. Secondary storage [costs around $30/TB](http://aiimpacts.org/costs-of-information-storage/) in 2015. [This means](http://aiimpacts.org/cost-of-human-level-information-storage/) it costs $300-3,000 for enough storage to store the contents of a human brain, or $0.007-$0.07/hour if hardware is used for five years.

In the long run [the price of secondary memory has declined](http://aiimpacts.org/costs-of-information-storage/) by an order of magnitude roughly every 4.6 years. However the rate has declined so much that prices haven’t substantially dropped since 2011 (in 2015).

### Interpreting partial costs

Calculating the total cost of hardware that is relevantly equivalent to the brain is not as simple as adding the partial costs as listed. FLOPS and TEPS are measures of different capabilities of the same hardware, so if you pay for TEPS at the aforementioned prices, you will also receive FLOPS.

The above list is also not exhaustive: there may be substantial hardware costs that we haven’t included.


Brain performance in FLOPS

The computing power needed to replicate the human brain’s relevant activities has been estimated by various authors, with answers ranging from 1012 to 1028 FLOPS.

Details
-------

### Notes

We have not investigated the brain’s performance in FLOPS in detail, nor substantially reviewed the literature since 2015. This page summarizes others’ estimates that we are aware of, as well as the implications of our investigation into brain performance in TEPS.

### Estimates

#### Sandberg and Bostrom 2008: estimates and review

[Sandberg and Bostrom](http://www.fhi.ox.ac.uk/brain-emulation-roadmap-report.pdf) project the processing required to emulate a human brain at different levels of detail.[1](https://aiimpacts.org/brain-performance-in-flops/#easy-footnote-bottom-1-596 "From Sandberg and Bostrom, table 9: Processing demands (emulation only, human brain)(p80):

spiking neural network: 1018 FLOPS (Earliest year, $1 million: commodity computer estimate: 2042, supercomputer estimate: 2019)

electrophysiology: 1022 FLOPS (Earliest year, $1 million: commodity computer estimate: 2068, supercomputer estimate: 2033)

metabolome: 1025 FLOPS (Earliest year, $1 million: commodity computer estimate: 2087, supercomputer estimate: 2044)

") For the three levels that their workshop participants considered most plausible, their estimates are 1018, 1022, and 1025 FLOPS.

They also summarize other brain compute estimates, as shown below (we reproduce their Table 10).[2](https://aiimpacts.org/brain-performance-in-flops/#easy-footnote-bottom-2-596 "See appendix A, Nick Bostrom and Anders Sandberg, “Whole Brain Emulation: A Roadmap,” 2008, 130.") We have not reviewed these estimates, and some do not appear superficially credible to us.

[![](http://aiimpacts.org/wp-content/uploads/2015/07/Screen-Shot-2019-07-06-at-4.47.04-PM.png)](http://aiimpacts.org/wp-content/uploads/2015/07/Screen-Shot-2019-07-06-at-4.47.04-PM.png)

[![](http://aiimpacts.org/wp-content/uploads/2015/07/Screen-Shot-2019-07-06-at-4.47.27-PM.png)](http://aiimpacts.org/wp-content/uploads/2015/07/Screen-Shot-2019-07-06-at-4.47.27-PM.png)

#### Drexler 2018

Drexler looks at multiple comparisons between narrow AI tasks and neural tasks, and finds that they suggest the ‘basic functional capacity’ of the human brain is less than one petaFLOPS (1015).[3](https://aiimpacts.org/brain-performance-in-flops/#easy-footnote-bottom-3-596 "“Multiple comparisons between narrow AI tasks and narrow neural tasks concur in suggesting that PFLOP/s computational systems exceed the basic functional capacity of the human brain.”

K Eric Drexler, “Reframing Superintelligence,” 2019, 182.")

#### Conversion from brain performance in TEPS

Among a small number of computers we compared[4](https://aiimpacts.org/brain-performance-in-flops/#easy-footnote-bottom-4-596 "“The [eight] supercomputers measured here consistently achieve around 1-2 GTEPS per scaled TFLOPS (see Figure 3). The median ratio is 1.9 GTEPS/TFLOPS, the mean is 1.7 GTEPS/TFLOP, and the variance 0.14 GTEPS/TFLOP. ” See Relationship between FLOPS and TEPS here for more details"), FLOPS and TEPS seem to vary proportionally, at a rate of around 1.7 GTEPS/TFLOP. We also [estimate](http://aiimpacts.org/brain-performance-in-teps/) that the human brain performs around  0.18 – 6.4 \* 1014 TEPS. Thus if the FLOPS:TEPS ratio in brains is similar to that in computers, a brain would perform around 0.9 – 33.7 \* 1016 FLOPS.[5](https://aiimpacts.org/brain-performance-in-flops/#easy-footnote-bottom-5-596 "0.18 – 6.4 * 1014 TEPS =0.18 – 6.4 * 105 GTEPS =0.18 – 6.4 * 105 GTEPS * 1TFLOPS/1.9GTEPS = 9,000-337,000 TFLOPS = 0.9 – 33.7 * 1016FLOPS") We have not investigated how similar this ratio is likely to be.

Notes
-----


Index of articles about hardware

### Hardware in terms of computing capacity (FLOPS and MIPS)

[Brain performance in FLOPS](http://aiimpacts.org/brain-performance-in-flops/)

[2019 recent trends in GPU price per FLOPS](https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-flops/)

[Electrical efficiency of computing](https://aiimpacts.org/electrical-efficiency-of-computing/)

[2018 price of performance by Tensor Processing Units](https://aiimpacts.org/2018-price-of-performance-by-tensor-processing-units/)

[2017 trend in the cost of computing](https://aiimpacts.org/recent-trend-in-the-cost-of-computing/)

[Price-performance trend in top supercomputers](https://aiimpacts.org/price-performance-trend-in-top-supercomputers/)

[2017 FLOPS prices](http://aiimpacts.org/current-flops-prices/)

[Trends in the cost of computing](http://aiimpacts.org/trends-in-the-cost-of-computing/)

[Wikipedia history of GFLOPS costs](http://aiimpacts.org/wikipedia-history-of-gflops-costs/)

### Hardware in terms of communication capacity (TEPS)

[Brain performance in TEPS](http://aiimpacts.org/brain-performance-in-teps/) (includes the cost of brain-level TEPS performance on current hardware)

[The cost of TEPS](http://aiimpacts.org/cost-of-teps/) (includes current costs, trends and relationship to other measures of hardware price)

### Information storage

[Information storage in the brain](http://aiimpacts.org/information-storage-in-the-brain/)

[Costs of information storage](http://aiimpacts.org/costs-of-information-storage/)

[Costs of human-level information storage](http://aiimpacts.org/cost-of-human-level-information-storage/)

### Other

[Costs of human-level hardware](http://aiimpacts.org/costs-of-human-level-hardware/)

[2019 recent trends in Geekbench score per CPU price](https://aiimpacts.org/2019-recent-trends-in-geekbench-score-per-cpu-price/)

[Trends in DRAM price per gigabyte](https://aiimpacts.org/trends-in-dram-price-per-gigabyte/)

[Effect of marginal hardware on artificial general intelligence](https://aiimpacts.org/effect-of-marginal-hardware-on-artificial-general-intelligence/)

[Research topic: hardware, software and AI](http://aiimpacts.org/research-topic-hardware-software-and-ai/)

[Index of articles about hardware](http://aiimpacts.org/index-of-hardware-articles/)

### Related blog posts

*[Preliminary prices for human level hardware](http://aiimpacts.org/preliminary-prices-for-human-level-hardware/) (4 April 2015)*

*[A new approach to predicting brain-computer parity](http://aiimpacts.org/tepsbrainestimate/) (7 May 2015)*

*[Time flies when robots rule the earth](http://aiimpacts.org/time-flies-when-robots-rule-the-earth/) (28 July 2015)*


Cost of human-level information storage

It costs roughly $300-$3000 to buy enough storage space to store all information contained by a human brain.

Support
-------

The human brain probably stores around [10-100TB of data](http://aiimpacts.org/information-storage-in-the-brain/). Data storage costs around $30/TB. Thus it costs roughly $300-$3000 to buy enough storage space to store all information contained by a human brain.

If we suppose that one wants to replace the hardware every five years, this is $0.007-$0.07/hour.[1](https://aiimpacts.org/cost-of-human-level-information-storage/#easy-footnote-bottom-1-592 "$300 to $3000 / (5 * 365 * 24)")

For reference, we have estimated that the computing hardware and electricity required to do the computation the brain does would cost around $4,700 – $170,000/hour at present (using an estimate based on [TEPS](http://aiimpacts.org/brain-performance-in-teps/), and assuming computers last for five years). Estimates based on computation rather than communication capabilities (like TEPS) appear to be spread between $3/hour and $1T/hour.[2](https://aiimpacts.org/cost-of-human-level-information-storage/#easy-footnote-bottom-2-592 "“So it seems human-level hardware presently costs between $3/hour and $1T/hour. ” – our blog post, ‘preliminary prices for human-level hardware’.") On the TEPS-based estimate then, the cost of replicating the brain’s information storage using existing hardware would currently be between a twenty millionth and a seventy thousandth of the cost of replicating the brain’s computation using existing hardware.


Costs of information storage

*Posted 23 July 2015*

Cheap [secondary memory](https://en.wikipedia.org/wiki/Auxiliary_memory) appears to cost around $0.03/GB in 2015. In the long run the price has declined by an order of magnitude roughly every 4.6 years. However the rate has declined so much that prices haven’t substantially dropped since 2011 (in 2015).

Support
-------

Cheap [secondary memory](https://en.wikipedia.org/wiki/Auxiliary_memory) appears to cost around $0.03/GB in 2015.[1](https://aiimpacts.org/costs-of-information-storage/#easy-footnote-bottom-1-589 "John C. McCallum’s dataset includes a point at May 2015 for $0.0000317/MB, which is $0.03/GB. He says ‘In general, these are the lowest priced disk drives for which I could find prices at the time.’ Figure 1 shows a similar price, from a different dataset. We have not assessed how different the datasets are, however they look somewhat different.")

The price appears to have declined at an average rate of around an order of magnitude every five years in the long run, as illustrated in Figures 1 and 2. Figure 1 shows roughly six and a half orders of magnitude in the thirty years between 1985 and 2015, for around an order of magnitude every 4.6 years. Figure 2 shows thirteen orders of magnitude over the the sixty years between 1955 and 2015, for exactly the same rate. Both figures suggest the rate has been much slower in the past five years, seemingly as part of a longer term flattening. It appears that prices haven’t substantially dropped since 2011 (in 2015).

[![xxx](http://aiimpacts.org/wp-content/uploads/2015/07/cost-per-gigabyte-large-1024x539.png)](http://aiimpacts.org/wp-content/uploads/2015/07/cost-per-gigabyte-large.png)**Figure 1:** Historic prices of hard drive space, from [Matt Komorowski](http://www.mkomo.com/cost-per-gigabyte-update)

[![Figure 2:](http://aiimpacts.org/wp-content/uploads/2015/07/storage_memory_prices_large-_hblok.net_-1024x640.png)](http://aiimpacts.org/wp-content/uploads/2015/07/storage_memory_prices_large-_hblok.net_.png)Figure 2: Historical prices of information storage in various formats, from [Havard Blok](http://hblok.net/blog/storage/), mostly drawing on John C. McCallum’s [data](http://www.jcmit.com/diskprice.htm).

---


Information storage in the brain

*Last updated 9 November 2020*

The brain probably stores around 10-100TB of data.

Support
-------

According to Forrest Wickman, computational neuroscientists generally believe the brain stores 10-100 terabytes of data.[1](https://aiimpacts.org/information-storage-in-the-brain/#easy-footnote-bottom-1-587 "“…Most computational neuroscientists tend to estimate human storage capacity somewhere between 10 terabytes and 100 terabytes, though the full spectrum of guesses ranges from 1 terabyte to 2.5 petabytes. (One terabyte is equal to about 1,000 gigabytes or about 1 million megabytes; a petabyte is about 1,000 terabytes.)

The math behind these estimates is fairly simple. The human brain contains roughly 100 billion neurons. Each of these neurons seems capable of making around 1,000 connections, representing about 1,000 potential synapses, which largely do the work of data storage. Multiply each of these 100 billion neurons by the approximately 1,000 connections it can make, and you get 100 trillion data points, or about 100 terabytes of information.

Neuroscientists are quick to admit that these calculations are very simplistic. First, this math assumes that each synapse stores about 1 byte of information, but this estimate may be too high or too low…”

– Wickman 2012") He suggests that these estimates are produced by assuming that information is largely stored in synapses, and that each synapse stores around 1 byte. The number of bytes is then simply the number of synapses.

These assumptions are simplistic (as he points out). In particular:

* synapses may store more or less than one byte of information on average
* some information may be stored outside of synapses
* not all synapses appear to store information
* synapses do not appear to be entirely independent

[We estimate](http://aiimpacts.org/scale-of-the-human-brain/) that there are 1.8-3.2 x 10¹⁴ synapses in the human brain, so according to the procedure Wickman outlines, this suggests that the brain stores around 180-320TB of data. It is unclear from his article whether the variation in the views of computational neuroscientists is due to different opinions on the assumptions stated above, or on the number of synapses in the brain. This makes it hard to adjust our estimate well, so our best guess for now is that the brain can store around 10-100TB of data, based on this being the common view among computational neuroscientists.

---


Event: Exercises in Economic Futurism

*By Katja Grace, 15 July 2015*

On Thursday July 30th Robin Hanson is visiting [again](http://aiimpacts.org/event-multipolar-ai-workshop-with-robin-hanson/), and this time we will be holding an informal workshop on how to usefully answer questions about the future, with an emphasis on economic approaches. We will pick roughly three concrete futurism questions, then think about how to go about answering them together. We hope both to make progress on the questions at hand, and to equip attendees with a wider range of tools for effective futurism.

Topic suggestions are welcome in the comments, whether or not you hope to come.

Afternoon tea will be provided.

### Details summary

**Date:** 30 July 2015

**Location:** Berkeley, near College Ave and Ashby (ask for more detail)

**Timetable:**

2pm: Afternoon tea and chatting (it is best to show up somewhere in this hour)

3pm: Exercises

7pm: End (and transition into a party at the same location—attendees welcome to stay on)

We hope to keep the event to around twenty people, so RSVP required. If you would like to come, write to katja@intelligence.org.

[![Sortie_de_l'opéra_en_l'an_2000-2](http://aiimpacts.org/wp-content/uploads/2015/07/Sortie_de_lopéra_en_lan_2000-2-1024x661.jpg)](http://aiimpacts.org/wp-content/uploads/2015/07/Sortie_de_lopéra_en_lan_2000-2.jpg)

Image: [La Sortie de l’opéra en l’an 2000](https://en.wikipedia.org/wiki/File:Sortie_de_l%27op%C3%A9ra_en_l%27an_2000-2.jpg)


Steve Potter on neuroscience and AI

*By Katja Grace, 13 July 2015*

[![Dr Steve Potter](http://aiimpacts.org/wp-content/uploads/2015/07/Dr.-Potter-150x150.jpg)](http://aiimpacts.org/wp-content/uploads/2015/07/Dr.-Potter.jpg)Dr Steve Potter
[Prof. Steve Potter](https://neurolab.gatech.edu/labs/potter/steve-potter) works at the [Laboratory of Neuroengineering](https://neurolab.gatech.edu/) in Atlanta, Georgia. I wrote to him after coming across his old article, [‘What can AI get from Neuroscience?’](http://www.neurolab.gatech.edu/wp/wp-content/uploads/potter/publications/Potter-NeuroscienceForAIchapter.pdf) I wanted to know how neuroscience might contribute to AI in the future: for instance will ‘[reverse engineering the brain](http://aiimpacts.org/kurzweil-the-singularity-is-near/)‘ be a substantial contributor of software for general AI? To shed light on this, I talked to Prof. Potter about how neuroscience has helped AI in the past, how the fields interact now, and what he expects in the future. Summary notes on the conversation are [here](http://aiimpacts.org/conversation-with-steve-potter/).