Evidence on good forecasting practices from the Good Judgment Project: an accompanying blog post

*By Daniel Kokotajlo,* *2 July 2019*

[![](http://aiimpacts.org/wp-content/uploads/2019/02/image2.png)](http://aiimpacts.org/wp-content/uploads/2019/02/image2.png)

Figure 0: The “four main determinants of forecasting accuracy.” [1](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-1-1260 "This graph can be found here, the GJP’s list of academic literature on this topic. The graph illustrates approximate relative effects. It will be discussed more in Section 2.")
Experience and data from the Good Judgment Project (GJP) provide important evidence about how to make accurate predictions. For a concise summary of the evidence and what we learn from it, see [this page](http://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/). For a review of [*Superforecasting*](https://smile.amazon.com/Superforecasting-Science-Prediction-Philip-Tetlock/dp/0804136718/ref=sr_1_1?ie=UTF8&qid=1541990711&sr=8-1&keywords=superforecasting+the+art+and+science+of+prediction)*,* the popular book written on the subject, see [this blog](http://slatestarcodex.com/2016/02/04/book-review-superforecasting/).

This post explores the evidence in more detail, drawing from the book, the academic literature, the older [*Expert Political Judgment*](https://smile.amazon.com/Expert-Political-Judgment-Good-Know/dp/0691128715/ref=sr_1_1?ie=UTF8&qid=1541990738&sr=8-1&keywords=expert+political+judgment) book, and an interview with a superforecaster. Readers are welcome to skip around to parts that interest them:

1. The experiment
-----------------

[IARPA](https://www.iarpa.gov/) ran a forecasting tournament from 2011 to 2015, in which five teams plus a control group gave probabilistic answers to hundreds of questions. The questions were generally about potential geopolitical events more than a month but less than a year in the future, e.g. “Will there be a violent incident in the South China Sea in 2013 that kills at least one person?” The questions were carefully chosen so that a reasonable answer would be somewhere between 10% and 90%.[2](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-2-1260 "This is from my conversation with the superforecaster. ") The forecasts were scored using the *original* [Brier score](https://en.wikipedia.org/wiki/Brier_score)—more on that in Section 2.[3](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-3-1260 "They did this so that they could include occasional non-binary questions. They show here that their results are robust to using a logarithmic scoring rule instead.")

The winning team was the GJP, run by Philip Tetlock & Barbara Mellers. They recruited thousands of online volunteers to answer IARPA’s questions. These volunteers tended to be males (83%) and US citizens (74%). Their average age was forty. 64% of respondents held a bachelor’s degree, and 57% had postgraduate training.[4](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-4-1260 "These statistics come from this study. The dataset excludes individuals who signed up but failed to register at least 25 predictions in a given year.")

GJP made their official predictions by aggregating and extremizing the predictions of their volunteers.[5](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-5-1260 "The aggregation algorithm was elitist, meaning that it weighted more heavily forecasters with good track-records who had updated their forecasts more often. This description of elitism comes from the webpage. In these slides Tetlock describes the elitism differently: He says it gives weight to higher-IQ, more open-minded forecasters. The extremizing step pushes the aggregated judgment closer to 1 or 0, to make it more confident. The degree to which they extremize depends on how diverse and sophisticated the pool of forecasters is. The academic papers on this topic can be found here and here. Whether extremizing is a good idea is controversial; according to one expert I interviewed, more recent data suggests that the successes of the extremizing algorithm during the forecasting tournament were a fluke. After all, a priori one would expect extremizing to lead to small improvements in accuracy most of the time, but big losses in accuracy some of the time.") They identified the top 2% of predictors in their pool of volunteers each year, dubbing them “superforecasters,” and put them on teams in the next year so they could collaborate on special forums. They also experimented with a prediction market, and they did a [RCT](https://en.wikipedia.org/wiki/Randomized_controlled_trial) to test the effect of a one-hour training module on forecasting ability. The module included content about probabilistic reasoning, using the outside view, avoiding biases, and more. Attempts were made to find out which parts of the training were most helpful—see Section 4.

2. The results & their intuitive meaning
----------------------------------------

Here are some of the key results:

“In year 1 GJP beat the official control group by 60%. In year 2, we beat the control group by 78%. GJP also beat its university-affiliated competitors, including the University of Michigan and MIT, by hefty margins, from 30% to 70%.”[6](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-6-1260 "Superforecasting p18. On page 69: “Teams had to beat the combined forecast—the “wisdom of the crowd”—of the contro group, and by margins we all saw as intimidating. In the first year, IARPA wanted teams to beat that standard by 20%—and it wanted that margin of victory to grow to 50% by the fourth year.” In light of this, it is especially impressive that individual superforecasters in the first two years beat the wisdom-of-the-crowds-of-the-control-group by ~60% and that the GJP beat it by 78%. (p72)")

“The Good Judgment Project outperformed a prediction market inside the intelligence community, which was populated with professional analysts who had classified information, by 25 or 30 percent, which was about the margin by which the superforecasters were outperforming our own prediction market in the external world.”[7](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-7-1260 "Transcript of this seminar.")

“Teams of ordinary forecasters beat the wisdom of the crowd by about 10%. Prediction markets beat ordinary teams by about 20%. And [teams of superforecasters] beat prediction markets by 15% to 30%.”[8](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-8-1260 "Superforecasting p207") “On average, teams were 23% more accurate than individuals.”[9](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-9-1260 "Superforecasting p201")

What does Tetlock mean when he says that one group did X% better than another? By examining Table 4 (in Section 4)  it seems that he means X% *lower* Brier score. What is the Brier score? For more details, see the [Wikipedia article](https://en.wikipedia.org/wiki/Brier_score); basically, it measures the average squared distance from the truth. This is why it’s better to have a lower Brier score—it means you were on average closer to the truth.[10](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-10-1260 "The best possible Brier score is 0; the Brier score achieved by guessing randomly depends on which version of the score you use and how many possible outcomes each prediction chooses between. For binary predictions, which constituted the bulk of IARPA’s questions, the original version of the Brier score is effectively twice the squared distance from the truth, so always guessing 50% would yield a score of 0.5.")

Here is a bar graph of all the forecasters in Year 2, sorted by Brier score:[11](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-11-1260 "This is from this study. The data covers the first two years of the tournament.")

[![](http://aiimpacts.org/wp-content/uploads/2019/02/image9.png)](http://aiimpacts.org/wp-content/uploads/2019/02/image9.png)

For this set of questions, guessing randomly (assigning even odds to all possibilities) would yield a Brier score of 0.53. So most forecasters did significantly better than that. Some people—the people on the far left of this chart, the superforecasters—did much better than the average. For example, in year 2, the superforecaster Doug Lorch did best with 0.14. This was more than 60% better than the control group.[12](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-12-1260 "Superforecasting p93") Importantly, being a superforecaster in one year correlated strongly with being a superforecaster the next year; there was some regression to the mean but roughly 70% of the superforecasters maintained their status from one year to the next.[13](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-13-1260 "Superforecasting p104")

OK, but what does all this mean, in intuitive terms? Here are three ways to get a sense of how good these scores really are:

**Way One:** Let’s calculate some examples of prediction patterns that would give you Brier scores like those mentioned above. Suppose you make a bunch of predictions with 80% confidence and you are correct 80% of the time. Then your Brier score would be 0.32, roughly middle of the pack in this tournament. If instead it was 93% confidence correct 93% of the time, your Brier score would be 0.132, very close to the best superforecasters and to GJP’s aggregated forecasts.[14](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-14-1260 "To calculate this, I assumed binary questions and plugged the probability, p, into this formula: P(event_doesn’t_happen)(0-p)^2+P(event_happens)(1-p)^2 = (1-p)(0-p)^2+(p)(1-p)^2. I then doubled it, since we are using the original Brier score that ranges between 0-2 instead of 0-1. I can’t find stats on GJP’s Brier score, but recall that in year 2 it was 78% better than the control group, and Doug Lorch’s 0.14 was 60% better than the control group. (Superforecasting p93)") In these examples, you are perfectly calibrated, which helps your score—more realistically you would be imperfectly calibrated and thus would need to be right even more often to get those scores.

**Way Two:** “An alternative measure of forecast accuracy is the proportion of days on which forecasters’ estimates were on the correct side of 50%. … For all questions in the sample, a chance score was 47%. The mean proportion of days with correct estimates was 75%…”[15](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-15-1260 "This is from the same study, as are the two figures.") According to this chart, the superforecasters were on the right side of 50% almost all the time:[16](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-16-1260 "The correlation between average Brier score and how often you were on the right side of 50% was 0.89 (same study), so I think it’s safe to assume the superforecasters were somewhere on the right side of the peak in Figure 2. (I assume they mean being on the right side of 50% correlates with lower Brier scores; the alternative is crazy.) The high proportion of guesses on the right side of 50% is a puzzling fact—doesn’t it suggest that they were poorly calibrated, and that they could improve their scores by extremizing their judgments? I think what’s going on here is that the majority of forecasts made on most questions by superforecasters were highly (>90%) confident, and also almost always correct.")

[![](http://aiimpacts.org/wp-content/uploads/2019/02/image1.png)](http://aiimpacts.org/wp-content/uploads/2019/02/image1.png)

**Way Three:** “Across all four years of the tournament, *superforecasters looking out three hundred days were more accurate than regular forecasters looking out one hundred days*.”[17](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-17-1260 "Superforecasting p94, emphasis mine. Later, in the edge.org seminar, Tetlock says “In some other ROC curves—receiver operator characteristic curves, from signal detection theory—that Mark Steyvers at UCSD constructed—superforecasters could assign probabilities 400 days out about as well as regular people could about eighty days out.” The quote is accompanied by a graph; unfortunately, it’s hard to interpret.") (Bear in mind, this wouldn’t necessarily hold for a different genre of questions. For example, information about the weather decays in days, while information about the climate lasts for decades or more.)

3. Correlates of good judgment
------------------------------

The data from this tournament is useful in two ways: It helps us decide whose predictions to trust, and it helps us make better predictions ourselves. This section will focus on which kinds of people and practices best correlate with success—information which is relevant to both goals. Section 4 will cover the training experiment, which helps to address causation vs. correlation worries.

Feast your eyes on this:[18](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-18-1260 "This table is from the same study.")

[![](http://aiimpacts.org/wp-content/uploads/2019/02/image8-1024x576.png)](http://aiimpacts.org/wp-content/uploads/2019/02/image8.png)

This shows the correlations between various things.[19](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-19-1260 "“Ravens” is an IQ test, “Numeracy” is a mathematical aptitude test.") The leftmost column is the most important; it shows how each variable correlates with (standardized) Brier score. (Recall that Brier scores measure inaccuracy, so negative correlations are good.)

It’s worth mentioning that while intelligence correlated with accuracy, it didn’t steal the show.[20](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-20-1260 "That said, as Carl Shulman pointed out, the forecasters in this sample were probably above-average IQ, so the correlation between IQ and accuracy in this sample is almost certainly smaller than the “true” correlation in the population at large. See e.g. restriction of range and the Thorndike Correction.") The same goes for time spent deliberating.[21](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-21-1260 "“Deliberation time, which was only measured in Year 2, was transformed by a logarithmic function (to reduce tail effects) and averaged over questions. The average length of deliberation time was 3.60 min, and the average number of questions tried throughout the 2-year period was 121 out of 199 (61% of all questions). Correlations between standardized Brier score accuracy and effort were statistically significant for belief updating, … and deliberation time, … but not for number of forecasting questions attempted.” (study) Anecdotally, I spoke to a superforecaster who said that the best of the best typically put a lot of time into it; he spends maybe fifteen minutes each day making predictions but several hours per day reading news, listening to relevant podcasts, etc.") The authors summarize the results as follows: “The best forecasters scored higher on both intelligence and political knowledge than the already well-above-average group of forecasters. The best forecasters had more open-minded cognitive styles. They benefited from better working environments with probability training and collaborative teams. And while making predictions, they spent more time deliberating and updating their forecasts.”[22](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-22-1260 "This is from the same study")

That big chart depicts all the correlations individually. Can we use them to construct a model to take in all of these variables and spit out a prediction for what your Brier score will be? Yes we can:

[![](http://aiimpacts.org/wp-content/uploads/2019/02/image5.png)](http://aiimpacts.org/wp-content/uploads/2019/02/image5.png)Figure 3. Structural equation model with standardized coefficients.
This model has a multiple correlation of 0.64.[23](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-23-1260 "“Nonetheless, as we saw in the structural model, and confirm here, the best model uses dispositional, situational, and behavioral variables. The combination produced a multiple correlation of .64.” (study)  Yellow ovals are latent dispositional variables, yellow rectangles are observed dispositional variables, pink rectangles are experimentally manipulated situational variables, and green rectangles are observed behavioral variables. If this diagram follows convention, single-headed arrows represent hypothesized causation, whereas the double-headed arrow represents a correlation without any claim being made about causation.") Earlier, we noted that superforecasters typically remained superforecasters (i.e. in the top 2%), proving that their success wasn’t mostly due to luck. Across all the forecasters, the correlation between performance in one year and performance in the next year is 0.65.[24](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-24-1260 "Superforecasting p104") So we have two good ways to predict how accurate someone will be: Look at their past performance, and look at how well they score on the structural model above.

I speculate that these correlations underestimate the true predictability of accuracy, because the forecasters were all unpaid online volunteers, and many of them presumably had random things come up in their life that got in the way of making good predictions—perhaps they have a kid, or get sick, or move to a new job and so stop reading the news for a month, and their accuracy declines.[25](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-25-1260 "Of course, these things can happen in the real world too—maybe our AI timelines forecasters will get sick and stop making good forecasts. What I’m suggesting is that this data is inherently noisier than data from a group of full-time staff whose job it is to predict things would be. Moreover, when these things happen in the real world, we can see that they are happening and adjust our model accordingly, e.g. “Bob’s really busy with kids this month, so let’s not lean as heavily on his forecasts as we usually do.”") Yet still 70% of the superforecasters in one year remained superforecasters in the next.

Finally, what about superforecasters in particular? Is there anything to say about what it takes to be in the top 2%?

Tetlock devotes much of his book to this. It is hard to tell how much his recommendations come from data analysis and how much are just his own synthesis of the interviews he’s conducted with superforecasters. Here is his “Portrait of the modal superforecaster.”[26](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-26-1260 "Superforecasting p191")

**Philosophic outlook:**

* **Cautious:** Nothing is certain.
* **Humble:** Reality is infinitely complex.
* **Nondeterministic:** Whatever happens is not meant to be and does not have to happen.

**Abilities & thinking styles:**

* **Actively open-minded:** Beliefs are hypotheses to be tested, not treasures to be protected.
* **Intelligent and knowledgeable, with a “Need for Cognition”:** Intellectually curious, enjoy puzzles and mental challenges.
* **Reflective:** Introspective and self-critical.
* **Numerate:** Comfortable with numbers.

**Methods of forecasting:**

* **Pragmatic:** Not wedded to any idea or agenda.
* **Analytical:** Capable of stepping back from the tip-of-your-nose perspective and considering other views.
* **Dragonfly-eyed:** Value diverse views and synthesize them into their own.
* **Probabilistic:** Judge using many grades of maybe.
* **Thoughtful updaters:** When facts change, they change their minds.
* **Good intuitive psychologists:** Aware of the value of checking thinking for cognitive and emotional biases.

**Work ethic:**

* **Growth mindset:** Believe it’s possible to get better.
* **Grit:** Determined to keep at it however long it takes.

Additionally, there is experimental evidence that superforecasters are less prone to standard cognitive science biases than ordinary people.[27](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-27-1260 "From edge.org: Mellers: “We have given them lots of Kahneman and Tversky-like problems to see if they fall prey to the same sorts of biases and errors. The answer is sort of, some of them do, but not as many. It’s not nearly as frequent as you see with the rest of us ordinary mortals. The other thing that’s interesting is they don’t make the kinds of mistakes that regular people make instead of the right answer. They do something that’s a little bit more thoughtful. They integrate base rates with case-specific information a little bit more.”
Tetlock: “They’re closer to Bayesians.”
Mellers: “Right. They’re a little less sensitive to framing effects. The reference point doesn’t have quite the enormous role that it does with most people.”") This is particularly exciting because—we can hope—the same sorts of training that help people become superforecasters might also help overcome biases.

Finally, Tetlock says that “The strongest predictor of rising into the ranks of superforecasters is perpetual beta, the degree to which one is committed to belief updating and self-improvement. It is roughly three times as powerful a predictor as its closest rival, intelligence.”[28](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-28-1260 "Superforecasting p192") Unfortunately, I couldn’t find any sources or data on this, nor an operational definition of “perpetual beta,” so we don’t know how he measured it.[29](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-29-1260 "Moreover, a quick search through Google Scholar and library.unc.edu turned up nothing of interest. I reached out to Tetlock to ask questions but he hasn’t responded yet.")

4. The training and Tetlock’s commandments
------------------------------------------

This section discusses the surprising effect of the training module on accuracy, and finishes with Tetlock’s training-module-based recommendations for how to become a better forecaster.[30](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-30-1260 "“The guidelines sketched here distill key themes in this book and in training systems that have been experimentally demonstrated to boost accuracy in real-world forecasting tournaments.” (277)")

The training module, which was randomly given to some participants but not others, took about an hour to read.[31](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-31-1260 "This is from this study. Relevant quote: “Although the training lasted less than one hour, it consistently improved accuracy (Brier scores) by 6 to 11% over the control condition.”") The authors describe the content as follows:

“Training in year 1 consisted of two different modules: probabilistic reasoning training and scenario training. Scenario-training was a four-step process: 1) developing coherent and logical probabilities under the probability sum rule; 2) exploring and challenging assumptions; 3) identifying the key causal drivers; 4) considering the best and worst case scenarios and developing a sensible 95% confidence interval of possible outcomes; and 5) avoid over-correction biases. … Probabilistic reasoning training consisted of lessons that detailed the difference between calibration and resolution, using comparison classes and base rates (Kahneman & Tversky, 1973; Tversky & Kahneman, 1981), averaging and using crowd wisdom principles (Surowiecki, 2005), finding and utilizing predictive mathematical and statistical models (Arkes, 1981; Kahneman & Tversky, 1982), cautiously using time-series and historical data, and being self-aware of the typical cognitive biases common throughout the population.”[32](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-32-1260 "Same study.")

In later years, they merged the two modules into one and updated it based on their observations of the best forecasters. The updated training module is organized around an acronym:[33](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-33-1260 "Same study.")

[![](http://aiimpacts.org/wp-content/uploads/2019/02/image6.png)](http://aiimpacts.org/wp-content/uploads/2019/02/image6.png)

Impressively, this training had a lasting positive effect on accuracy in all four years:

[![](http://aiimpacts.org/wp-content/uploads/2019/02/image3-1024x423.png)](http://aiimpacts.org/wp-content/uploads/2019/02/image3.png)

One might worry that training improves accuracy by motivating the trainees to take their jobs more seriously. Indeed it seems that the trained forecasters made more predictions per question than the control group, though they didn’t make more predictions overall. Nevertheless it seems that the training also had a direct effect on accuracy as well as this indirect effect.[34](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-34-1260 "See sections 3.3, 3.5, and 3.6 of this study.")

Moving on, let’s talk about the advice Tetlock gives to his audience in *Superforecasting*, advice which is based on, though not identical to, the CHAMPS-KNOW training. The book has a few paragraphs of explanation for each commandment, a transcript of which is [here](https://www.lesswrong.com/posts/dvYeSKDRd68GcrWoe/ten-commandments-for-aspiring-superforecasters); in this post I’ll give my own abbreviated explanations:

TEN COMMANDMENTS FOR ASPIRING SUPERFORECASTERS

**(1) Triage:** Don’t waste time on questions that are “clocklike” where a rule of thumb can get you pretty close to the correct answer, or “cloudlike” where even fancy models can’t beat a dart-throwing chimp.

**(2) Break seemingly intractable problems into tractable sub-problems:** This is how Fermi estimation works. One related piece of advice is “be wary of accidentally substituting an easy question for a hard one,” e.g. substituting “Would Israel be willing to assassinate Yasser Arafat?” for “Will at least one of the tests for polonium in Arafat’s body turn up positive?”

**(3) Strike the right balance between inside and outside views:** In particular, *first* anchor with the outside view and *then* adjust using the inside view. (More on this in Section 5)

**(4) Strike the right balance between under- and overreacting to evidence:** “Superforecasters aren’t perfect Bayesian predictors but they are much better than most of us.”[35](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-35-1260 "Superforecasting p281") Usually do many small updates, but occasionally do big updates when the situation calls for it. Take care not to fall for things that seem like good evidence but aren’t; remember to think about P(E|H)/P(E|~H); remember to avoid the base-rate fallacy.

**(5) Look for the clashing causal forces at work in each problem:** This is the “dragonfly eye perspective,” which is where you attempt to do a sort of mental wisdom of the crowds: Have tons of different causal models and aggregate their judgments. Use “Devil’s advocate” reasoning. If you think that P, try hard to convince yourself that not-P. You should find yourself saying “On the one hand… on the other hand… on the third hand…” a lot.

**(6) Strive to distinguish as many degrees of doubt as the problem permits but no more:** Some people criticize the use of exact probabilities (67%! 21%!) as merely a way to pretend you know more than you do. There might be another post on the subject of why credences are better than hedge words like “maybe” and “probably” and “significant chance;” for now, I’ll simply mention that when the authors rounded the superforecaster’s forecasts to the nearest 0.05, their accuracy dropped.[36](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-36-1260 "This is from Friedman et al (2018), available here.") Superforecasters really were making use of all 101 numbers from 0.00 to 1.00! (EDIT: I am told this may be wrong; the number should be 0.1, not 0.05. See the discussion [here](https://www.metaculus.com/questions/4166/the-lightning-round-tournament-comparing-metaculus-forecasters-to-infectious-disease-experts/#comment-28756 "https://www.metaculus.com/questions/4166/the-lightning-round-tournament-comparing-metaculus-forecasters-to-infectious-disease-experts/#comment-28756") and [here](https://forum.effectivealtruism.org/posts/W94KjunX3hXAtZvXJ/evidence-on-good-forecasting-practices-from-the-good?commentId=RtjNoQMNRQvuYtsx5 "https://forum.effectivealtruism.org/posts/W94KjunX3hXAtZvXJ/evidence-on-good-forecasting-practices-from-the-good?commentId=RtjNoQMNRQvuYtsx5").)

**(7) Strike the right balance between under- and overconfidence, between prudence and decisiveness.**

**(8) Look for the errors behind your mistakes but beware of rearview-mirror hindsight biases.**

**(9) Bring out the best in others and let others bring out the best in you:** The book spent a whole chapter on this, using the Wehrmacht as an extended case study on good team organization.[37](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-37-1260 "Scott Alexander: “Later in the chapter, he admits that his choice of examples might raise some eyebrows, but says that he did it on purpose to teach us to think critically and overcome cognitive dissonance between our moral preconceptions and our factual beliefs. I hope he has tenure.”") One pervasive guiding principle is “Don’t tell people how to do things; tell them what you want accomplished, and they’ll surprise you with their ingenuity in doing it.” The other pervasive guiding principle is “Cultivate a culture in which people—even subordinates—are encouraged to dissent and give counterarguments.”[38](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-38-1260 "See e.g. page 284 of Superforecasting, and the entirety of chapter 9.")

**(10) Master the error-balancing bicycle:** This one should have been called practice, practice, practice. Tetlock says that reading the news and generating probabilities isn’t enough; you need to actually score your predictions so that you know how wrong you were.

**(11) Don’t treat commandments as commandments:** Tetlock’s point here is simply that you should use your judgment about whether to follow a commandment or not; sometimes they should be overridden.

It’s worth mentioning at this point that the advice is given at the end of the book, as a sort of summary, and may make less sense to someone who hasn’t read the book. In particular, Chapter 5 gives a less formal but more helpful recipe for making predictions, with accompanying examples. See the end of this blog post for a summary of this recipe.

5. On the Outside View & Lessons for AI Impacts
-----------------------------------------------

The previous section summarized Tetlock’s advice for how to make better forecasts; my own summary of the lessons I think we should learn is more concise and comprehensive and can be found at [this page](http://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/). This section goes into detail about one particular, more controversial matter: The importance of the “[outside view](https://www.mckinsey.com/business-functions/strategy-and-corporate-finance/our-insights/daniel-kahneman-beware-the-inside-view),” also known as [reference class forecasting](https://en.wikipedia.org/wiki/Reference_class_forecasting). This research provides us with strong evidence in favor of this method of making predictions; however, the situation is complicated by Tetlock’s insistence that other methods are useful as well. This section discusses the evidence and attempts to interpret it.

The GJP asked people who took the training to self-report which of the CHAMPS-KNOW principles they were using when they explained why they made a forecast; 69% of forecast explanations received tags this way. The only principle significantly positively correlated with successful forecasts was C: Comparison classes.[39](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-39-1260 "This is from this paper. One worry I have about it is that another principle, P, was strongly associated with inaccuracy, but the authors explain this away by saying that “Post-mortem analyses,” the P’s, are naturally done usually after bad forecasts. This makes me wonder if a similar explanation could be given for the success of the C’s: Questions for which a good reference class exists are easier than others.") The authors take this as evidence that the outside view is particularly important. Anecdotally, the superforecaster I interviewed agreed that reference class forecasting was perhaps the most important piece of the training. (He also credited the training in general with helping him reach the ranks of the superforecasters.)

Moreover, Tetlock did an earlier, much smaller forecasting tournament from 1987-2003, in which experts of various kinds made the forecasts.[40](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-40-1260 "The results and conclusions from this tournament can be found in the resulting book, Expert Political Judgment: How good is it? How can we know? See p242 for a description of the methodology and dates.") The results were astounding: Many of the experts did worse than random chance, and *all of them did worse than simple algorithms*:

[![](http://aiimpacts.org/wp-content/uploads/2019/02/image10.png)](http://aiimpacts.org/wp-content/uploads/2019/02/image10.png)

Figure 3.2, pulled from *Expert Political Judgment,* is a gorgeous depiction of some of the main results.[41](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-41-1260 "Page 77.") Tetlock used something very much like a Brier score in this tournament, but he broke it into two components: “Discrimination” and “Calibration.” This graph plots the various experts and algorithms on the axes of discrimination and calibration. Notice in the top right corner the “Formal models” box. I don’t know much about the model used but apparently it was significantly better than all of the humans. This, combined with the fact that simple case-specific trend extrapolations also beat all the humans, is strong evidence for the importance of the outside view.

So we should always use the outside view, right? Well, it’s a bit more complicated than that. Tetlock’s advice is to *start* with the outside view, and then *adjust* using the inside view.[42](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-42-1260 "Superforecasting p120") He even goes so far as to say that *hedgehoggery* and *storytelling* can be valuable when used properly.

First, what is hedgehoggery? Recall how the human experts fall on a rough spectrum in Figure 3.2, with “hedgehogs” getting the lowest scores and “foxes” getting the highest scores. What makes someone a hedgehog or a fox? Their answers to [these questions](http://www.overcomingbias.com/2006/11/quiz_fox_or_hed.html).[43](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-43-1260 "For the data on how these questions were weighted in determining foxyness, see Expert Political Judgment p74") Tetlock characterizes the distinction as follows:

Low scorers look like hedgehogs: thinkers who “know one big thing,” aggressively extend the explanatory reach of that one big thing into new domains, display bristly impatience with those who “do not get it,” and express considerable confidence that they are already pretty proficient forecasters, at least in the long term. High scorers look like foxes: thinkers who know many small things (tricks of their trade), are skeptical of grand schemes, see explanation and prediction not as deductive exercises but rather as exercises in flexible “ad hocery” that require stitching together diverse sources of information, and are rather diffident about their own forecasting prowess, and … rather dubious that the cloudlike subject of politics can be the object of a clocklike science.[44](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-44-1260 "Expert Political Judgment p75")

Next, what is storytelling? Using your domain knowledge, you think through a detailed scenario of how the future might go, and you tweak it to make it more plausible, and then you assign a credence based on how plausible it seems. By itself this method is unpromising.[45](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-45-1260 "There are several reasons to worry about this method. For one, it’s not what foxes do, and foxes score better than hedgehogs. Tetlock also says it’s not what superforecasters do. More insightfully, Tetlock says we are biased to assign more probability to more vivid and interesting stories, and as a result it’s easy for your probabilities to sum to much more than 1. Anecdote: I was answering a series of “Probability of extinction due to cause X” questions on Metaculus, and I soon realized that my numbers were going to add up to more than 100%, so I had to adjust them all down systematically to make room for the last few kinds of disaster on the list. If I hadn’t been assigning explicit probabilities, I wouldn’t have noticed the error. And if I hadn’t gone through the whole list of possibilities, I would have come away with an unjustifiably high credence in the few I had considered.")

Despite this, Tetlock thinks that storytelling and hedgehoggery are valuable if handled correctly. On hedgehogs, Tetlock says that hedgehogs provide a valuable service by doing the deep thinking necessary to build detailed causal models and raise interesting questions; these models and questions can then be slurped up by foxy superforecasters, evaluated, and aggregated to make good predictions.[46](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-46-1260 "Superforecasting p266. This is reminiscent of Yudkowsky’s perspective on what is essentially this same debate.") The superforecaster Bill Flack is quoted in agreement.[47](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-47-1260 " Superforecasting p271.") As for storytelling, see these slides from Tetlock’s [edge.org seminar](https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-v):

[![](http://aiimpacts.org/wp-content/uploads/2019/02/image4.jpg)](http://aiimpacts.org/wp-content/uploads/2019/02/image4.jpg)

As the second slide indicates, the idea is that we can sometimes “fight fire with fire” by using some stories to counter other stories. In particular, Tetlock says there has been success using stories about the past—about ways that the world could have gone, but didn’t—to “reconnect us to our past states of ignorance.”[48](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-48-1260 "Same seminar.") The superforecaster I interviewed said that it is common practice now on superforecaster forums to have a designated “red team” with the explicit mission of finding counter-arguments to whatever the consensus seems to be. This, I take it, is an example of motivated reasoning being put to good use.

Moreover, arguably the outside view simply isn’t useful for some questions.[49](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-49-1260 "For example, see Yudkowsky: “Where two sides disagree, this can lead to reference class tennis—both parties get stuck insisting that their own “outside view” is the correct one, based on diverging intuitions about what similarities are relevant. If it isn’t clear what the set of “similar historical cases” is, or what conclusions we should draw from those cases, then we’re forced to use an inside view—thinking about the causal process to distinguish relevant similarities from irrelevant ones. You shouldn’t avoid outside-view-style reasoning in cases where it looks likely to work, like when planning your Christmas shopping. But in many contexts, the outside view simply can’t compete with a good theory.”") People say this about lots of things—e.g. “The world is changing so fast, so the current situation in Syria is unprecedented and historical averages will be useless!”—and are proven wrong; for example, this research seems to indicate that the outside view is far more useful in geopolitics than people think. Nevertheless, maybe it is true for some of the things we wish to predict about advanced AI. After all, a major limitation of this data is that the questions were mainly on geopolitical events only a few years in the future at most. (Geopolitical events seem to be somewhat predictable up to two years out but much more difficult to predict five, ten, twenty years out.)[50](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-50-1260 "Tetlock admits that “there is no evidence that geopolitical or economic forecasters can predict anything ten years out beyond the excruciatingly obvious… These limits on predictability are the predictable results of the butterfly dynamics of nonlinear systems. In my EPJ research, the accuracy of expert predictions declined toward chance five years out.” (Superforecasting p243) I highly recommend the graphic on that page, by the way, also available here: “Thoughts for the 2001 Quadrennial Defense Review.”") So this research does not *directly* tell us anything about the predictability of the events AI Impacts is interested in, nor about the usefulness of reference-class forecasting for those domains.[51](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-51-1260 "The superforecaster I interviewed speculated that predicting things like the continued drop in price of computing hardware or solar panels is fairly easy, but that predicting the appearance of new technologies is very difficult. Tetlock has ideas for how to handle longer-term, nebulous questions. He calls it “Bayesian Question Clustering.” (Superforecasting 263) The idea is to take the question you really want to answer and look for more precise questions that are evidentially relevant to the question you care about. Tetlock intends to test the effectiveness of this idea in future research.")

That said, the forecasting best practices discovered by this research seem like general truth-finding skills rather than cheap hacks only useful in geopolitics or only useful for near-term predictions. After all, geopolitical questions are themselves a fairly diverse bunch, yet accuracy on some was highly correlated with accuracy on others.[52](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-52-1260 "“There are several ways to look for individual consistency across questions. We sorted questions on the basis of response format (binary, multinomial, conditional, ordered), region (Eurzone, Latin America, China, etc.), and duration of question (short, medium, and long). We computed accuracy scores for each individual on each variable within each set (e.g., binary, multinomial, conditional, and ordered) and then constructed correlation matrices. For all three question types, correlations were positive… Then we conducted factor analyses. For each question type, a large proportion of the variance was captured by a single factor, consistent with the hypothesis that one underlying dimension was necessary to capture correlations among response formats, regions, and question duration.” (from this study)") So despite these limitations I think we should do our best to imitate these best-practices, and that means using the outside view far more than we would naturally be inclined.

One final thing worth saying is that, remember, the GJP’s aggregated judgments did at least as well as the best superforecasters.[53](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-53-1260 " I haven’t found this said explicitly, but I infer this from Doug Lorch, the best superforecaster in Year 2, beating the control group by at least 60% when the GJP beat the control group by 78%. (Superforecasting 93, 18) That said, page 72 seems to say that in Year 2 exactly one person—Doug Lorch—managed to beat the aggregation algorithm. This is almost a contradiction; I’m not sure what to make of it. At any rate, it seems that the aggregation algorithm pretty reliably does better than the superforecasters in general, even if occasionally one of them beats it.") Presumably at least one of the forecasters in the tournament was using the outside view a lot; after all, half of them were trained in reference-class forecasting.  So I think we can conclude that straightforwardly using the outside view as often as possible wouldn’t get you better scores than the GJP, though it might get you close for all we know. Anecdotally, it seems that when the superforecasters use the outside view they often aggregate between different reference-class forecasts.[54](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-54-1260 "This is on page 304. Another example on 313.") The wisdom of the crowds is powerful; this is consistent with the wider literature on the cognitive superiority of groups, and the literature on ensemble methods in AI.[55](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-55-1260 "For more on these, see this page.")

Tetlock describes how superforecasters go about making their predictions.[56](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/#easy-footnote-bottom-56-1260 "This is my summary of Tetlock’s advice in Chapter 5: “Ultimately, it’s not the number crunching power that counts. It’s how you use it. … You’ve Fermi-ized the question, consulted the outside view, and now, finally, you can consult the inside view … So you have an outside view and an inside view. Now they have to be merged. …”") Here is an attempt at a summary:

1. Sometimes a question can be answered more rigorously if it is first “Fermi-ized,” i.e. broken down into sub-questions for which more rigorous methods can be applied.
2. Next, use the outside view on the sub-questions (and/or the main question, if possible). You may then adjust your estimates using other considerations (‘the inside view’), but do this cautiously.
3. Seek out other perspectives, both on the sub-questions and on how to Fermi-ize the main question. You can also generate other perspectives yourself.
4. Repeat steps 1 – 3 until you hit diminishing returns.
5. Your final prediction should be based on an aggregation of various models, reference classes, other experts, etc.

**Footnotes**
-------------


Evidence on good forecasting practices from the Good Judgment Project

According to experience and data from the Good Judgment Project, the following are associated with successful forecasting, in rough decreasing order of combined importance and confidence:

* Past performance in the same broad domain
* Making more predictions on the same question
* Deliberation time
* Collaboration on teams
* Intelligence
* Domain expertise
* Having taken a one-hour training module on these topics
* ‘Cognitive reflection’ test scores
* ‘Active open-mindedness’
* Aggregation of individual judgments
* Use of precise probabilistic predictions
* Use of ‘the outside view’
* ‘Fermi-izing’
* ‘Bayesian reasoning’
* Practice

**Details**
-----------

### **1. 1. Process**

The Good Judgment Project (GJP) was the winning team in IARPA’s 2011-2015 forecasting tournament. In the tournament, six teams assigned probabilistic answers to hundreds of questions about geopolitical events months to a year in the future. Each competing team used a different method for coming up with their guesses, so the tournament helps us to evaluate different forecasting methods.

The GJP team, led by Philip Tetlock and Barbara Mellers, gathered thousands of online volunteers and had them answer the tournament questions. They then made their official forecasts by aggregating these answers. In the process, the team collected data about the patterns of performance in their volunteers, and experimented with aggregation methods and improvement interventions. For example, they ran an RCT to test the effect of a short training program on forecasting accuracy. They especially focused on identifying and making use of the most successful two percent of forecasters, dubbed ‘superforecasters’.

Tetlock’s book *Superforecasting* describes this process and Tetlock’s resulting understanding of how to forecast well.

### **1.2. Correlates of successful forecasting**

#### 1.2.1. Past performance

Roughly 70% of the superforecasters maintained their status from one year to the next [1](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-1-1283 " Superforecasting p104 "). Across all the forecasters, the correlation between performance in one year and performance in the next year was 0.65 [2](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-2-1283 " Superforecasting p104 "). These high correlations are particularly impressive because the forecasters were online volunteers; presumably substantial variance year-to-year came from forecasters throttling down their engagement due to fatigue or changing life circumstances [3](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-3-1283 " Technically the forecasters were paid, up to $250 per season. (Superforecasting p72) However their payments did not depend on how accurate they were or how much effort they put in, beyond the minimum. ").

#### 1.2.2. Behavioral and dispositional variables

Table 2  depicts the correlations between measured variables amongst GJP’s volunteers in the first two years of the tournament [4](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-4-1283 " The table is from Mellers et al 2015. “Del time” is deliberation time. ").  Each is described in more detail below.

[![](http://aiimpacts.org/wp-content/uploads/2019/02/image2-1-1024x576.png)](http://aiimpacts.org/wp-content/uploads/2019/02/image2-1.png)

The first column shows the relationship between each variable and standardized [Brier score](https://en.wikipedia.org/wiki/Brier_score), which is a measure of inaccuracy: higher Brier scores mean less accuracy, so negative correlations are good. “Ravens” is an IQ test; “Del time” is deliberation time, and “teams” is whether or not the forecaster was assigned to a team. “Actively open-minded thinking” is an attempt to measure “the tendency to evaluate arguments and evidence without undue bias from one’s own prior beliefs—and with recognition of the fallibility of one’s judgment.” [5](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-5-1283 " “Nonetheless, as we saw in the structural model, and confirm here, the best model uses dispositional, situational, and behavioral variables. The combination produced a multiple correlation of .64.” This is from Mellers et al 2015. ")

The authors conducted various statistical analyses to explore the relationships between these variables. They computed a structural equation model to predict a forecaster’s accuracy:

[![](http://aiimpacts.org/wp-content/uploads/2019/02/image1-1.png)](http://aiimpacts.org/wp-content/uploads/2019/02/image1-1.png)

Yellow ovals are latent dispositional variables, yellow rectangles are observed dispositional variables, pink rectangles are experimentally manipulated situational variables, and green rectangles are observed behavioral variables. This model has a [multiple correlation](https://en.wikipedia.org/wiki/Multiple_correlation) of 0.64.[6](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-6-1283 " This is from Mellers et al 2015. ")

As these data indicate, domain knowledge, intelligence, active open-mindedness, and working in teams each contribute substantially to accuracy. We can also conclude that effort helps, because deliberation time and number of predictions made per question (“belief updating”) both improved accuracy. Finally, training also helps. This is especially surprising because the training module lasted only an hour and its effects persisted for at least a year. The module included content about probabilistic reasoning, using the outside view, avoiding biases, and more.

### **1.3. Aggregation algorithms**

GJP made their official predictions by aggregating and extremizing the predictions of their volunteers. The aggregation algorithm was elitist, meaning that it weighted more heavily people who were better on various metrics. [7](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-7-1283 " On the webpage, it says forecasters with better track-records and those who update more frequently get weighted more. In these slides, Tetlock describes the elitism differently: He says it gives weight to higher-IQ, more open-minded forecasters. ") The extremizing step pushes the aggregated judgment closer to 1 or 0, to make it more confident. The degree to which they extremize depends on how diverse and sophisticated the pool of forecasters is. [8](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-8-1283 " The academic papers on this topic are Satopaa et al 2013 and Baron et al 2014. ") Whether extremizing is a good idea is still controversial.  [9](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-9-1283 " According to one expert I interviewed, more recent data suggests that the successes of the extremizing algorithm during the forecasting tournament were a fluke. After all, a priori one would expect extremizing to lead to small improvements in accuracy most of the time, but big losses in accuracy some of the time. ")

GJP beat all of the other teams. They consistently beat the control group—which was a forecast made by averaging ordinary forecasters—by more than 60%. [10](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-10-1283 " Superforecasting p18. ") They  also beat a prediction market inside the intelligence community—populated by professional analysts with access to classified information—by 25-30%. [11](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-11-1283 " This is from this seminar. ")

That said, individual superforecasters did almost as well, so the elitism of the algorithm may account for a lot of its success.[12](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-12-1283 " For example, in year 2 one superforecaster beat the extremizing algorithm. More generally, as discussed in this seminar, the aggregation algorithm produces the greatest improvement with ordinary forecasters; the superforecasters were good enough that it didn’t help much. ")

### **1.4. Outside View**

The forecasters who received training were asked to record, for each prediction, which parts of the training they used to make it. Some parts of the training—e.g. “Post-mortem analysis”—were correlated with inaccuracy, but others—most notably “Comparison classes”—were correlated with accuracy. [13](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-13-1283 " This is from Chang et al 2016. The average brier score of answers tagged “comparison classes” was 0.17, while the next-best tag averaged 0.26.")  ‘Comparison classes’ is another term for [reference-class forecasting](https://en.wikipedia.org/wiki/Reference_class_forecasting), also known as ‘the outside view’. It is the method of assigning a probability by straightforward extrapolation from similar past situations and their outcomes.

### **1.5. Tetlock’s “Portrait of the modal superforecaster”**

This subsection and those that follow will lay out some more qualitative results, things that Tetlock recommends on the basis of his research and interviews with superforecasters. Here is Tetlock’s “portrait of the modal superforecaster:” [14](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-14-1283 " Superforecasting p191 ")

**Philosophic outlook:**

* **Cautious:** Nothing is certain.
* **Humble:** Reality is infinitely complex.
* **Nondeterministic:** Whatever happens is not meant to be and does not have to happen.

**Abilities & thinking styles:**

* **Actively open-minded:** Beliefs are hypotheses to be tested, not treasures to be protected.
* **Intelligent and knowledgeable, with a “Need for Cognition”:** Intellectually curious, enjoy puzzles and mental challenges.
* **Reflective:** Introspective and self-critical
* **Numerate:** Comfortable with numbers

**Methods of forecasting:**

* **Pragmatic:** Not wedded to any idea or agenda
* **Analytical:** Capable of stepping back from the tip-of-your-nose perspective and considering other views
* **Dragonfly-eyed:** Value diverse views and synthesize them into their own
* **Probabilistic:** Judge using many grades of maybe
* **Thoughtful updaters:** When facts change, they change their minds
* **Good intuitive psychologists:** Aware of the value of checking thinking for cognitive and emotional biases [15](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-15-1283 " There is experimental evidence that superforecasters are less prone to standard cognitive science biases than ordinary people. From edge.org: Mellers: “We have given them lots of Kahneman and Tversky-like problems to see if they fall prey to the same sorts of biases and errors. The answer is sort of, some of them do, but not as many. It’s not nearly as frequent as you see with the rest of us ordinary mortals. The other thing that’s interesting is they don’t make the kinds of mistakes that regular people make instead of the right answer. They do something that’s a little bit more thoughtful. They integrate base rates with case-specific information a little bit more.”  ")

**Work ethic:**

* **Growth mindset:** Believe it’s possible to get better
* **Grit:** Determined to keep at it however long it takes

### **1.6. Tetlock’s “Ten Commandments for Aspiring Superforecasters:”**

This advice is given at the end of the book, and may make less sense to someone who hasn’t read the book. A full transcript of these commandments can be found [here](https://www.lesswrong.com/posts/dvYeSKDRd68GcrWoe/ten-commandments-for-aspiring-superforecasters); this is a summary:

**(1) Triage:** Don’t waste time on questions that are “clocklike” where a rule of thumb can get you pretty close to the correct answer, or “cloudlike” where even fancy models can’t beat a dart-throwing chimp.

**(2) Break seemingly intractable problems into tractable sub-problems:** This is how Fermi estimation works. One related piece of advice is “be wary of accidentally substituting an easy question for a hard one,” e.g. substituting “Would Israel be willing to assassinate Yasser Arafat?” for “Will at least one of the tests for polonium in Arafat’s body turn up positive?”

**(3) Strike the right balance between inside and outside views:** In particular, *first* anchor with the outside view and *then* adjust using the inside view.

**(4) Strike the right balance between under- and overreacting to evidence:** Usually do many small updates, but occasionally do big updates when the situation calls for it. Remember to think about P(E|H)/P(E|~H); remember to avoid the base-rate fallacy. “Superforecasters aren’t perfect Bayesian predictors but they are much better than most of us.” [16](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-16-1283 " Superforecasting p281 ")

**(5) Look for the clashing causal forces at work in each problem:** This is the “dragonfly eye perspective,” which is where you attempt to do a sort of mental wisdom of the crowds: Have tons of different causal models and aggregate their judgments. Use “Devil’s advocate” reasoning. If you think that P, try hard to convince yourself that not-P. You should find yourself saying “On the one hand… on the other hand… on the third hand…” a lot.

**(6) Strive to distinguish as many degrees of doubt as the problem permits but no more.**

**(7) Strike the right balance between under- and overconfidence, between prudence and decisiveness.**

**(8) Look for the errors behind your mistakes but beware of rearview-mirror hindsight biases.**

**(9) Bring out the best in others and let others bring out the best in you.** The book spent a whole chapter on this, using the Wehrmacht as an extended case study on good team organization. One pervasive guiding principle is “Don’t tell people how to do things; tell them what you want accomplished, and they’ll surprise you with their ingenuity in doing it.” The other pervasive guiding principle is “Cultivate a culture in which people—even subordinates—are encouraged to dissent and give counterarguments.” [17](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-17-1283 " See e.g. page 284 of Superforecasting, and the entirety of chapter 9. ")

**(10) Master the error-balancing bicycle:** This one should have been called practice, practice, practice. Tetlock says that reading the news and generating probabilities isn’t enough; you need to actually score your predictions so that you know how wrong you were.

**(11) Don’t treat commandments as commandments:** Tetlock’s point here is simply that you should use your judgment about whether to follow a commandment or not; sometimes they should be overridden.

### **1.7. Recipe for Making Predictions**

Tetlock describes how superforecasters go about making their predictions. [18](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-18-1283 " See Chapter 5: “Ultimately, it’s not the number crunching power that counts. It’s how you use it. … You’ve Fermi-ized the question, consulted the outside view, and now, finally, you can consult the inside view … So you have an outside view and an inside view. Now they have to be merged. …” ") Here is an attempt at a summary:

1. Sometimes a question can be answered more rigorously if it is first “Fermi-ized,” i.e. broken down into sub-questions for which more rigorous methods can be applied.
2. Next, use the outside view on the sub-questions (and/or the main question, if possible). You may then adjust your estimates using other considerations (‘the inside view’), but do this cautiously.
3. Seek out other perspectives, both on the sub-questions and on how to Fermi-ize the main question. You can also generate other perspectives yourself.
4. Repeat steps 1 – 3 until you hit diminishing returns.
5. Your final prediction should be based on an aggregation of various models, reference classes, other experts, etc.

### **1.8.** **Bayesian reasoning & precise probabilistic forecasts**

Humans normally express uncertainty with terms like “maybe” and “almost certainly” and “a significant chance.” Tetlock advocates for thinking and speaking in probabilities instead. He recounts many anecdotes of misunderstandings that might have been avoided this way. For example:

In 1961, when the CIA was planning to topple the Castro government by landing a small army of Cuban expatriates at the Bay of Pigs, President John F. Kennedy turned to the military for an unbiased assessment. The Joint Chiefs of Staff concluded that the plan had a “fair chance” of success. The man who wrote the words “fair chance” later said he had in mind odds of 3 to 1 against success. But Kennedy was never told precisely what “fair chance” meant and, not unreasonably, he took it to be a much more positive assessment. [19](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-19-1283 " Superforecasting 44 ")

This example hints at another advantage of probabilistic judgments: It’s harder to weasel out of them afterwards, and therefore easier to keep score. Keeping score is crucial for getting feedback from reality, which is crucial for building up expertise.

A standard criticism of using probabilities is that they merely conceal uncertainty rather than quantify it—after all, the numbers you pick are themselves guesses. This may be true for people who haven’t practiced much, but it isn’t true for superforecasters, who are impressively well-calibrated and whose accuracy scores decrease when you round their predictions to the nearest 0.05. (EDIT: This should be 0.1)[20](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-20-1283 " The superforecasters had a calibration of 0.01, which means that the average difference between a probability they use and the true frequency of occurrence is 0.01. This is from Mellers et al 2015. The fact about rounding their predictions is from Friedman et al 2018. EDIT: Seems I was wrong, thanks to this commenter for noticing.https://www.metaculus.com/questions/4166/the-lightning-round-tournament-comparing-metaculus-forecasters-to-infectious-disease-experts/#comment-28756")

Bayesian reasoning is a natural next step once you are thinking and talking probabilities—it is the theoretical ideal in several important ways [21](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-21-1283 " For an excellent introduction to Bayesian reasoning and its theoretical foundations, see Strevens’ textbook-like lecture notes. Some of the facts summarized in this paragraph about Superforecasters and Bayesianism can be found on pages 169-172, 281, and 314 of Superforecasting. ") —and Tetlock’s experience and interviews with superforecasters seems to bear this out. Superforecasters seem to do many small updates, with occasional big updates, just as Bayesianism would predict. They recommend thinking in the Bayesian way, and often explicitly make Bayesian calculations. They are good at breaking down difficult questions into more manageable parts and chaining the probabilities together properly.

**2. Discussion: Relevance to AI Forecasting**
----------------------------------------------

### **2.1. Limitations**

A major limitation is that the forecasts were mainly on geopolitical events only a few years in the future at most. (Uncertain geopolitical events seem to be somewhat predictable up to two years out but much more difficult to predict five years out.) [22](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-22-1283 " Tetlock admits that “there is no evidence that geopolitical or economic forecasters can predict anything ten years out beyond the excruciatingly obvious… These limits on predictability are the predictable results of the butterfly dynamics of nonlinear systems. In my EPJ research, the accuracy of expert predictions declined toward chance five years out.” (Superforecasting p243) ") So evidence from the GJP may not generalize to forecasting other types of events (e.g. technological progress and social  consequences) or events further in the future.

That said, the forecasting best practices discovered by this research are not overtly specific to geopolitics or near-term events.  Also, geopolitical questions are diverse and accuracy on some was highly correlated with accuracy on others. [23](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-23-1283 " “There are several ways to look for individual consistency across questions. We sorted questions on the basis of response format (binary, multinomial, conditional, ordered), region (Eurzone, Latin America, China, etc.), and duration of question (short, medium, and long). We computed accuracy scores for each individual on each variable within each set (e.g., binary, multinomial, conditional, and ordered) and then constructed correlation matrices. For all three question types, correlations were positive… Then we conducted factor analyses. For each question type, a large proportion of the variance was captured by a single factor, consistent with the hypothesis that one underlying dimension was necessary to capture correlations among response formats, regions, and question duration.” From Mellers et al 2015. ")

Tetlock has ideas for how to handle longer-term, nebulous questions. He calls it “Bayesian Question Clustering.” (*Superforecasting* 263) The idea is to take the question you really want to answer and look for more precise questions that are evidentially relevant to the question you care about. Tetlock intends to test the effectiveness of this idea in future research.

### **2.2 Value**

The benefits of following these best practices (including identifying and aggregating the best forecasters) appear to be substantial: Superforecasters predicting events 300 days in the future were more accurate than regular forecasters predicting events 100 days in the future, and the GJP did even better. [24](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/#easy-footnote-bottom-24-1283 "  Superforecasting p94. Later, in the edge.org seminar, Tetlock says “In some other ROC curves—receiver operator characteristic curves, from signal detection theory—that Mark Steyvers at UCSD constructed—superforecasters could assign probabilities 400 days out about as well as regular people could about eighty days out.” The quote is accompanied by a graph; unfortunately, it’s hard to interpret. ") If these benefits generalize beyond the short-term and beyond geopolitics—e.g. to long-term technological and societal development—then this research is highly useful to almost everyone. Even if the benefits do not generalize beyond the near-term, these best practices may still be well worth adopting. For example, it would be extremely useful to have 300 days of warning before strategically important AI milestones are reached, rather than 100.

**3. Contributions**
--------------------

*Research, analysis, and writing were done by Daniel Kokotajlo. Katja Grace and Justis Mills contributed feedback and editing. Tegan McCaslin, Carl Shulman, and Jacob Lagerros contributed feedback.*

**4. Footnotes**
----------------

This post lists some of the key traits which are associated with successful forecasting, based on work from the Good Judgement Project (who won IARPA's forecasting tournament by a wide margin). The top 5: past performance in the same broad domain; making more predictions on the same question; deliberation time; collaboration on teams; and intelligence.  The authors also summarise various other ideas from the Superforecasting book.


Reinterpreting “AI and Compute”

*This is a guest post by Ben Garfinkel. We revised it slightly, at his request, on February 9, 2019.*

A recent OpenAI [blog post](https://blog.openai.com/ai-and-compute/), “AI and Compute,” showed that the amount of computing power consumed by the most computationally intensive machine learning projects has been doubling every three months. The post presents this trend as a reason to better prepare for “systems far outside today’s capabilities.” Greg Brockman, the CTO of OpenAI, has also used the trend to argue for the plausibility of “[near-term AGI](https://www.youtube.com/watch?v=YHCSNsLKHfM).” Overall, it seems pretty common to interpret the OpenAI data as evidence that we should expect extremely capable systems sooner than we otherwise would.

However, I think it’s important to note that the data can also easily be interpreted in the opposite direction. A more pessimistic interpretation goes like this:

1. If we were previously underestimating the rate at which computing power was increasing, this means we were *overestimating* the returns on it.
2. In addition, if we were previously underestimating the rate at which computing power was increasing, this means that we were *overestimating* how sustainable its growth is.[1](https://aiimpacts.org/reinterpreting-ai-and-compute/#easy-footnote-bottom-1-1250 "As Ryan Carey has argued, we should expect the trend to run up against physical and financial limitations within the decade.")
3. Let’s suppose, as the original post does, that increasing computing power is currently one of the main drivers of progress in creating more capable systems. Then — barring any major changes to the status quo — it seems like we should expect progress to slow down pretty soon and we should expect to be underwhelmed by how far along we are when the slowdown hits.

I actually think of this more pessimistic interpretation as something like the *default* one. There are many other scientific fields where R&D spending and other inputs are increasing rapidly, and, so far as I’m aware, these trends are nearly always interpreted as reasons for pessimism and concern about future research progress.[2](https://aiimpacts.org/reinterpreting-ai-and-compute/#easy-footnote-bottom-2-1250 "See, for example, the pharmaceutical industry’s concern about “Eroom’s Law”: the observation that progress in developing new drugs has been steady despite exponentially growing R&D spending and increasingly powerful drug discovery technologies. The recent paper “Are Ideas Getting Harder to Find?” (Bloom et al., 2018) also includes a pessimistic discussion of several other domains, including agriculture and semiconductor manufacturing.") If we are going to treat the field of artificial intelligence differently, then we should want clearly articulated and compelling reasons for doing so.

These reasons certainly might exist.[3](https://aiimpacts.org/reinterpreting-ai-and-compute/#easy-footnote-bottom-3-1250 "One way to argue for the bullish interpretation is to draw on work (briefly surveyed in Ryan’s post) that attempts to estimate the minimum quantity of computing power required to produce a system with the same functionality as the human brain. We can then attempt to construct an argument where: (a) we estimate this minimum quantity of computing power (using evidence unrelated to the present rate of return on computing power), (b) predict that the quantity will become available before growth trends hit their wall, and (c) argue that having it available would be nearly sufficient to rapidly train systems that can do a large portion of the things humans can do. In this case, the OpenAI data would be evidence that we should expect the computational “threshold” to be reached slightly earlier than we would otherwise have expected to reach it. For example, it might take only five years to reach the threshold rather than ten. However, my view is that it’s very difficult to construct an argument where parts (a)-(c) are all sufficiently compelling. In any case, it still doesn’t seem like the OpenAI data alone should substantially increase the probability anyone assigns to “near-term AGI” (rather than just shifting forward their conditional probability estimates of how “near-term” “near-term AGI” would be).") Still, whatever the case may be, I think we should not be too quick to interpret the OpenAI data as evidence for dramatically more capable systems coming soon.[4](https://aiimpacts.org/reinterpreting-ai-and-compute/#easy-footnote-bottom-4-1250 "As a final complication, it’s useful to keep in mind that the OpenAI data only describes the growth rate for the most well-resourced research projects. If you think about progress in developing more “capable” AI systems (whatever you take “capable” to mean) as mostly a matter of what the single most computationally unconstrained team can do at a given time, then this data is obviously relevant. However, if you instead think that something like the typical amount of computing power available to talented researchers is what’s most important — or if you simply think that looking at the amount of computing power available to various groups can’t tell us much at all — then the OpenAI data seems to imply relatively little about future progress.")

*Thank you to Danny Hernandez and Ryan Carey for comments on a draft of this post.*

[Data](https://blog.openai.com/ai-and-compute/) from OpenAI showed that the amount of compute used by the most expensive projects had been growing exponentially with a doubling time of three months. While it is easy to interpret this trend as suggesting that we will get AGI sooner than expected, it is also possible to interpret this trend as evidence in the opposite direction. A surprisingly high rate of increase in amount of compute used suggests that we have been _overestimating_ how helpful more compute is. Since this trend [can't be sustainable over decades](https://aiimpacts.org/interpreting-ai-compute-trends/), we should expect that progress will slow down, and so this data is evidence _against_ near-term AGI.

The surprising part of the data is how fast compute has been growing. One common part of AGI timelines is whether you think compute or algorithms are the bottleneck. Assuming you had a good sense of progress in AI, but were surprised by how fast compute grew, you should update against the relative benefits of compute.

This post seems to be about the way you relate compute to AGI timelines, ignoring algorithms altogether. If you think of AGI as requiring a specific amount of compute that is determined independently of current AI progress (for example, by estimating the compute used by a human brain), then the evidence should shorten your timelines. If you instead predict how close AGI is by looking at the rate of progress in AI and extrapolating over time, then to first order this data should not affect timelines (since compute is not part of the model), and to second order it should lengthen them for the reasons in this post.


Time for AI to cross the human performance range in diabetic retinopathy

In diabetic retinopathy, automated systems started out just below expert human level performance, and took around ten years to reach expert human level performance.

Details
-------

Diabetic retinopathy is a complication of diabetes in which the back of the eye is damaged by high blood sugar levels.[1](https://aiimpacts.org/diabetic-retinopathy-as-a-case-study-in-time-for-ai-to-cross-the-range-of-human-performance/#easy-footnote-bottom-1-1241 " See e.g. https://www.nhs.uk/conditions/diabetic-retinopathy/ ") It is the most common cause of blindness among working-age adults.[2](https://aiimpacts.org/diabetic-retinopathy-as-a-case-study-in-time-for-ai-to-cross-the-range-of-human-performance/#easy-footnote-bottom-2-1241 "See https://nei.nih.gov/health/diabetic/retinopathy") The disease is diagnosed by examining images of the back of the eye. The gold standard used for diabetic retinopathy diagnosis is typically some sort of pooling mechanism over several expert opinions. Thus, in the papers below, each time expert sensitivity/specificity (Se/Sp) is considered, it is the Se/Sp of individual experts graded against aggregate expert agreement.

As a rough benchmark for expert-level performance we’ll take the average Se/Sp of ophthalmologists from a few studies. Based on Google Brain’s work (detailed below), this paper [3](https://aiimpacts.org/diabetic-retinopathy-as-a-case-study-in-time-for-ai-to-cross-the-range-of-human-performance/#easy-footnote-bottom-3-1241 " See Results section before adjudication and consensus https://www.ncbi.nlm.nih.gov/pubmed/23494039 "), and this paper [4](https://aiimpacts.org/diabetic-retinopathy-as-a-case-study-in-time-for-ai-to-cross-the-range-of-human-performance/#easy-footnote-bottom-4-1241 " See Figure 3 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2911785/ ") , the average specificity of 14 opthamologists, which indicates expert human-level performance, is 95% and the average sensitivity is 82%.

As far as we can tell, 1996 is when the first algorithm automatically detecting diabetic retinopathy was developed. When compared to opthamologists’ ratings, the algorithm achieved 88.4% sensitivity and 83.5% specificity.

In late 2016 Google algorithms were on par with eight opthamologist diagnoses of diabetic retinopathy. See Figure 1.[5](https://aiimpacts.org/diabetic-retinopathy-as-a-case-study-in-time-for-ai-to-cross-the-range-of-human-performance/#easy-footnote-bottom-5-1241 " https://ai.googleblog.com/2016/11/deep-learning-for-detection-of-diabetic.html ") The high-sensitivity operating point (labelled on the graph) achieved 97.5/93.4 Se/Sp.

[![](http://aiimpacts.org/wp-content/uploads/2018/11/eyepacs1.png)](http://aiimpacts.org/wp-content/uploads/2018/11/eyepacs1.png)Figure 1: Performance comparison of a late 2016 Google algorithm, and eight opthalmologists, from here. The black curve represents the algorithm and the eight colored dots are opthamologists.

Many other papers were published in between 1996 and 2016. However, none of them achieved better than expert human-level performance on both specificity and sensitivity. For instance 86/77 Se/Sp was achieved in 2007, 97/59 in 2013, and 94/72 by another team in 2016. [6](https://aiimpacts.org/diabetic-retinopathy-as-a-case-study-in-time-for-ai-to-cross-the-range-of-human-performance/#easy-footnote-bottom-6-1241 " ‘Automated and semi-automated diabetic retinopathy evaluation has been previously studied by other groups. Abràmoff et al4 reported a sensitivity of 96.8% at a specificity of 59.4% for detecting referable diabetic retinopathy on the publicly available Messidor-2 data set.9Solanki et al12 reported a sensitivity of 93.8% at a specificity of 72.2% on the same data set. A study by Philip et al21 reported a sensitivity of 86.2% at a specificity of 76.8% for predicting disease vs no disease on their own data set of 14, 406 images.’ https://jamanetwork.com/journals/jama/fullarticle/2588763 ")

Thus it took about **ten years** to go from just below expert human level performance to slightly superhuman performance.

Contributions
-------------

*Aysja Johnson researched and wrote this page. Justis Mills and Katja Grace contributed feedback.*

Footnotes
---------


AGI-11 survey

The AGI-11 survey was a survey of 60 participants at the AGI-11 conference. In it:

* Nearly half of respondents believed that AGI would appear before 2030.
* Nearly 90% of respondents believed that AGI would appear before 2100.
* About 85% of respondents believed that AGI would be beneficial for humankind.

Details
-------

James Barrat and Ben Goertzel surveyed participants at the [AGI-11](http://agi-conference.org/2011/) conference on [AGI](http://en.wikipedia.org/wiki/Artificial_General_Intelligence) timelines. The survey had two questions, administered over email after the conference. The results were fairly similar to those from the more complex [AGI-09](https://aiimpacts.org/agi-09-survey/) survey, for which Ben Goertzel was also an author.

Sixty people total responded to the survey, out of [over 200](http://agi-conf.org/2011/2011/08/13/agi-11-a-smashing-success/) conference registrations. Nobody skipped either of the two questions.

The data in this post, and the results tables, are taken from the [write up on this survey](http://hplusmagazine.com/2011/09/16/how-long-till-agi-views-of-agi-11-conference-participants/) in h+ magazine.

### Question One

Question one was: “I believe that AGI (however I define it) will be effectively implemented in the following timeframe:”

The answer choices were:

* Before 2030
* 2030-2049
* 2050-2099
* After 2100
* Never

The results were:

![](http://hplusmagazine.com/wp-content/uploads/Barrat_survey_1_again1.png)

### Question Two

Question two was: “I believe that AGI (however I define it) will be a net positive event for humankind:”

The answer choices were:

* True
* False

The results were:

![](http://hplusmagazine.com/wp-content/uploads/Barrat_survey_2_again1.png)

### Comments

As well as the two survey questions, there was also a form where respondents could submit comments. These are recorded in the [h+ magazine](http://hplusmagazine.com/2011/09/16/how-long-till-agi-views-of-agi-11-conference-participants/) write up. Many of the comments expressed concern with the survey structure, suggesting that there could have been more or different options.

Contributions
-------------

*Justis Mills researched for and wrote this post. Katja Grace provided feedback on style and content. Thanks to James Barrat for linking us to the h+ magazine write up when we reached out to him about his survey.*

A survey of participants in the AGI-11 participants (with 60 respondents out of over 200 registrations) found that 43% thought AGI would appear before 2030, 88% thought it would appear before 2100, and 85% believed it would be beneficial for humankind.

Note there's a strong selection effect, as AGI is a conference specifically aimed at general intelligence.


On the (in)applicability of corporate rights cases to digital minds

*This is a guest [cross-post](https://cullenokeefe.com/blog/2018/9/25/on-the-inapplicability-of-corporate-rights-cases-to-digital-minds) by [Cullen O’Keefe,](https://cullenokeefe.com/) 28 September 2018*

High-Level Takeaway
-------------------

The extension of rights to corporations likely does *not* provide useful analogy to potential extension of rights to digital minds.

Introduction
------------

Examining how law can protect the welfare of possible future digital minds is part of my research agenda. I expect that study of historical efforts to secure legal protections (“rights”) for previously unprotected classes (e.g., formerly enslaved persons, nonhuman animals, young children) will be crucial to this line of research.

I recently read *We the Corporations: How American Businesses Won Their Civil Rights* by [UCLA constitutional law professor Adam Winkler](https://law.ucla.edu/faculty/faculty-profiles/adam-winkler/). The book chronicles how business corporations gradually won various constitutional and statutory civil rights, culminating in the (in)famous recent [*Citizens United*](https://www.oyez.org/cases/2008/08-205) and [*Hobby Lobby*](https://www.oyez.org/cases/2013/13-354) cases.

A key insight from Winkler’s book is that, contrary to some popular portrayals of corporate rights cases, these cases usually do *not* rely primarily on corporate personhood: “While the Supreme Court has on occasion said that corporations are people, the justices have more often relied upon a very different conception of the corporation, one that views it as an *association* capable of asserting the rights of its *members*.” *Id.* at xx. The Court, in other words, “pierced the corporate veil” to give corporations rights properly belonging to its members. *See id.* at 54–55.

The Supreme Court’s opinion in *Citizens United* is illustrative. In determining that the First Amendment’s free speech protections applied to corporations, the Court wrote: “[Under the challenged campaign finance statute,] certain disfavored associations of citizens—those that have taken on the corporate form—are penalized for engaging in [otherwise-protected] political speech.” 558 U.S. at 356. The Court held that this was impermissible: the shareholders’ right to free speech imbued the corporation—which it viewed as merely an association of rights-bearing shareholders—with those same rights. *See id.* at 365.

Earlier cases that, Winkler argues, exhibit this same pattern include:

1. *Bank of U.S. v. Deveaux*, holding that federal jurisdiction over corporations depends on jurisdiction over the individuals comprising the corporation;
2. *Trustees of Dartmouth Coll. v. Woodward*, holding that corporate charters gave trustees private rights therein, which were protected against state alteration by the Constitution;
3. *NAACP* v. *Alabama ex rel. Patterson*, holding that non-profit corporation could assert First Amendment rights of its members;
4. *Bains LLC v. Arco Prod. Co., Div. of Atl. Richfield Co.*, holding that a corporation had standing to bring racial discrimination claim for racial discrimination against its employees.

Implications
------------

I believe that this understanding of the corporate civil rights “struggle” has small-but-nontrivial implications for a potential future strategy to secure legal protections for digital minds. Specifically, I think Winkler’s thesis suggests that the extension of rights to corporations is *not* a useful historical or legal analogy for the potential extension of rights to digital minds. This is because Winkler’s book demonstrates that corporations gained rights primarily because their constitutive members (i.e., shareholders) *already had* rights. In the case of digital minds generally, I see no obvious analogy to shareholders: digital minds as such are not mere extensions or associations of entities already bearing rights.

More concretely, this suggests that securing [legal personhood for digital minds](https://www.politico.eu/article/europe-divided-over-robot-ai-artificial-intelligence-personhood/) for instrumental reasons is *not* likely, on its own, to increase the likelihood of legal protections for them.

Carrick Flynn suggested to me (and I now agree) that nonhuman animal protections probably provide the best analog for future digital mind protections. To the extent that it rules out another possible method of approaching the question, this post supports that thesis.

*This work was financially supported by the Berkeley Existential Risk Initiative.*


Hardware overhang

*Hardware overhang* refers to a situation where large quantities of computing hardware can be diverted to running powerful AI systems as soon as the software is developed.

Details
-------

### Definition

In the context of AI forecasting, *hardware overhang* refers to a situation where enough computing hardware to run many powerful AI systems already exists by the time the software to run such systems is developed. If such hardware is repurposed to AI, this would mean that as soon as one powerful AI system exists, probably a large number of them do. This might amplify the impact of the arrival of human-level AI.

The alternative to a hardware overhang is for software sufficient for powerful AI to be developed before the hardware to run it has become cheap. In that case, we might instead expect to see the first powerful AI be built when it is very expensive, and therefore continues to be rare.