Futurists have argued for years about whether the development of AGI will look more like a breakthrough within a small group (“fast takeoff”), or a continuous acceleration distributed across the broader economy or a large firm (“slow takeoff”).

I currently think a slow takeoff is significantly more likely. This post explains some of my reasoning and why I think it matters. Mostly the post lists arguments I often hear for a fast takeoff and explains why I don’t find them compelling.

(Note: this is *not* a post about whether an intelligence explosion will occur. That seems very likely to me. Quantitatively I expect it to go [along these lines](https://sideways-view.com/2017/10/04/hyperbolic-growth/). So e.g. while I disagree with many of the claims and assumptions in [Intelligence Explosion Microeconomics](https://intelligence.org/files/IEM.pdf), I don’t disagree with the central thesis or with most of the arguments.)

(See also: [AI Impacts page](https://aiimpacts.org/likelihood-of-discontinuous-progress-around-the-development-of-agi/) on the same topic.)

### Slow takeoff

#### **Slower takeoff means faster progress**

Fast takeoff is often justified by pointing to the incredible transformative potential of intelligence; by enumerating the many ways in which AI systems will outperform humans; by pointing to historical examples of rapid change; *etc.*

This gives the impression that people who expect a slow takeoff think AI will have a smaller impact, or will take longer to transform society.

But I think that’s backwards. The main disagreement is not about what will happen once we have a superintelligent AI, it’s about what will happen *before* we have a superintelligent AI. So slow takeoff seems to mean that AI has a larger impact on the world, sooner.

![TakeoffImage.001](https://unstylizedcom.files.wordpress.com/2018/02/takeoffimage-0011.png?w=748)

In the fast takeoff scenario, weaker AI systems may have significant impacts but they are nothing compared to the “real” AGI. Whoever builds AGI has a decisive strategic advantage. Growth accelerates from 3%/year to 3000%/year without stopping at 30%/year. And so on.

In the slow takeoff scenario, pre-AGI systems have a transformative impact that’s only slightly smaller than AGI. AGI appears in a world where everything already happens incomprehensibly quickly and everyone is incredibly powerful. Being 12 months ahead in AGI might get you a decisive strategic advantage, but the world has accelerated so much that that’s just about as hard as getting to airplanes 30 years before anyone else.

#### **Operationalizing slow takeoff**

*There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles. (Similarly, we’ll see an 8 year doubling before a 2 year doubling, etc.)*

At some point there will be incredibly powerful AI systems. They will have many consequences, but one simple consequence is that world output will grow much more quickly. I think this is a good barometer for other transformative effects, including large military advantages.

I believe that before we have incredibly powerful AI, we will have AI which is merely *very* powerful. This won’t be enough to create 100% GDP growth, but it will be enough to lead to (say) 50% GDP growth. I think the likely gap between these events is years rather than months or decades.

In particular, this means that incredibly powerful AI will emerge in a world where crazy stuff is already happening (and probably everyone is already freaking out). If true, I think it’s an important fact about the strategic situation.

(Operationalizing takeoff speed in terms of economic doublings may seem weird, but I do think it gets at the disagreement: proponents of fast takeoff don’t seem to expect the 4 year doubling before takeoff, or at least their other beliefs about the future don’t seem to integrate that expectation.)

#### **The basic argument**

The *prima facie* argument for slow takeoff is pretty straightforward:

* Before we have an incredibly intelligent AI, we will probably have a slightly worse AI.
	+ Lots of people will be trying to build powerful AI.
	+ For most X, it is easier to figure out how to do a slightly worse version of X than to figure out how to do X.
		- The worse version may be more expensive, slower, less reliable, less general… (Usually there is a tradeoff curve, and so you can pick which axes you want the worse version to be worse along.)
	+ If many people are trying to do X, and a slightly worse version is easier and almost-as-good, someone will figure out how to do the worse version before anyone figures out how to do the better version.
	+ This story seems consistent with the historical record. Things are usually preceded by worse versions, even in cases where there are weak reasons to expect a discontinuous jump.
		- The best counterexample is probably nuclear weapons. But in that case there were several very strong reasons for discontinuity: physics has an inherent gap between chemical and nuclear energy density, nuclear chain reactions require a large minimum scale, and the dynamics of war are very sensitive to energy density.
* A slightly-worse-than-incredibly-intelligent AI would radically transform the world, leading to growth (almost) as fast and military capabilities (almost) as great as an incredibly intelligent AI.

This simple argument pushes towards slow takeoff. But there are several considerations that could push towards fast takeoff, which we need to weigh against the basic argument.

Obviously this is a quantitative question. In this post I’m not going to get into the numbers because the substance of the disagreement seems to be about qualitative models.

### Reasons to expect fast takeoff

People have offered a variety of reasons to expect fast takeoff. I think that many of these arguments make sense, but I don’t think they support the kind of highly concentrated, discontinuous progress which fast takeoff proponents seem to typically have in mind.

I expect there are other arguments beyond these, or that I’ve misunderstood some of these, and look forward to people pointing out what I’m missing.

#### **Humans vs. chimps**

*Summary of my response:* *chimps are nearly useless because they aren’t optimized to be useful, not because evolution was trying to make something useful and wasn’t able to succeed until it got to humans*.

Chimpanzees have brains only ~3x smaller than humans, but are much worse at making technology (or doing science, or accumulating culture…). If evolution were selecting primarily or in large part for technological aptitude, then the difference between chimps and humans would suggest that tripling compute and doing a tiny bit of additional fine-tuning can radically expand power, undermining the continuous change story.

But chimp evolution is not primarily selecting for making and using technology, for doing science, or for facilitating cultural accumulation.  The task faced by a chimp is largely independent of the abilities that give humans such a huge fitness advantage. It’s not completely independent—the overlap is the only reason that evolution eventually produces humans—but it’s different enough that we should not be surprised if there are simple changes to chimps that would make them much better at designing technology or doing science or accumulating culture.

If we compare humans and chimps at the tasks chimps are optimized for, humans are clearly much better but the difference is not nearly as stark. Compare to the difference between chimps and gibbons, gibbons and lemurs, or lemurs and squirrels.

Relatedly, evolution *changes* what it is optimizing for over evolutionary time: as a creature and its environment change, the returns to different skills can change, and they can potentially change very quickly. So it seems easy for evolution to shift from “not caring about X” to “caring about X,” but nothing analogous will happen for AI projects. (In fact a similar thing often *does* happen while optimizing something with SGD, but it doesn’t happen at the level of the ML community as a whole.)

If we step back from skills and instead look at outcomes we could say: “Evolution is *always* optimizing for fitness, and humans have now taken over the world.” On this perspective, I’m making a claim about the limits of evolution. First, evolution is theoretically optimizing for fitness, but it isn’t able to look ahead and identify which skills will be most important for your children’s children’s children’s fitness. Second, human intelligence is incredibly good for the fitness of *groups* of humans, but evolution acts on individual humans for whom the effect size is much smaller (who barely benefit at all from passing knowledge on to the next generation). Evolution really is optimizing something quite different than “humanity dominates the world.”

So I don’t think the example of evolution tells us much about whether the continuous change story applies to intelligence. This case is potentially missing the key element that drives the continuous change story—optimization for performance. Evolution changes continuously on the narrow metric it is optimizing, but can change extremely rapidly on other metrics. For human technology, features of the technology that aren’t being optimized change rapidly all the time. When humans build AI, they *will* be optimizing for usefulness, and so progress in usefulness is much more likely to be linear.

Put another way: the difference between chimps and humans stands in stark contrast to the normal pattern of human technological development. We might therefore infer that intelligence is very unlike other technologies. But the difference between evolution’s optimization and our optimization seems like a much more parsimonious explanation. To be a little bit more precise and Bayesian: the priorprobability of the story I’ve told upper bounds the possible update about the nature of intelligence.

**AGI will be a side-effect**

*Summary of my response: I expect people to see AGI coming and to invest heavily.*

AI researchers might be optimizing for narrow forms of intelligence. If so we could have the same dynamic as with chimps—we see continuous progress on accomplishing narrow tasks in a narrow way, leading eventually to a jump in general capacities *as a side-effect*. These general capacities then also lead to much better progress on narrow tasks, but there is no reason for progress to be continuous because no one is optimizing for general intelligence.

I don’t buy this argument because I think that researchers probably *will* be optimizing aggressively for general intelligence, if it would help a lot on tasks they care about. If that’s right, this argument only implies a discontinuity if there is *some other reason*that the usefulness of general intelligence of general intelligence is discontinuous.

However, if researchers greatly underestimate the impact of general intelligence and so don’t optimize for it, I agree that a fast takeoff is plausible**.** It could turn out that “will researchers adequately account for the impact of general intelligence and so try to optimize it?” is a crux. My intuition is based on a combination of (weak) adequacy intuitions and current trends in ML research.

#### **Finding the secret sauce**

*Summary of my response: this doesn’t seem common historically, and I don’t see why we’d expect AGI to be more rather than less like this (unless we accept one of the other arguments)*

Another common view is that there are some number of key insights that are needed to build a generally intelligent system. When the final pieces fall into place we may then see a large jump; one day we have a system with enough raw horsepower to be very smart but critical limitations, and the next day it is able to use all of that horsepower.

I don’t know exactly how to respond to this view because I don’t feel like I understand it adequately.

I’m not aware of many historical examples of this phenomenon (and no really good examples)—to the extent that there have been “key insights” needed to make something important work, the first version of the insight has almost always either been discovered long before it was needed, or discovered in a preliminary and weak version which is then iteratively improved over a long time period.

To the extent that fast takeoff proponent’s views are informed by historical example, I would love to get some canonical examples that they think best exemplify this pattern so that we can have a more concrete discussion about those examples and what they suggest about AI.

Note that a really good example should be on a problem that many people care about. There are lots of examples where no one is thinking about X, someone uncovers an insight that helps a lot with X, and many years later that helps with another task Y that people do care about. That’s certainly interesting, but it’s not really surprising at all on the slow-change view unless it actually causes surprisingly fast progress on Y.

Looking forward to AGI, it seems to me like if anything we should have a somewhat smaller probability than usual that a final “key insight” making a huge difference.

* AGI was built by evolution, which is more likely if it can be built by iteratively improving simple ingredients.
* It seems like we already have a set of insights that are sufficient for building an autopoetic AGI so we won’t be starting from 0 in any case.
* Historical AI applications have had a relatively small loading on key-insights and seem like the closest analogies to AGI.

The example of chimps or dumb humans seems like one of the best reasons to expect a key insight, but I’ve already discussed why I find that pretty unconvincing.

In this case I don’t yet feel like I understand where fast takeoff proponents are coming from, so I think it is especially likely that my view will change based on further discussion. But I would really like to see a clearer articulation of the fast takeoff view here as an early step of that process.

#### **Universality thresholds**

*Summary of my response: it seems like early AI systems will cross universality thresholds pre-superintelligence, since (a) there are tradeoffs between universality and other desirable properties which would let people build universal AIs early if the returns to universality are large enough, (b) I think we can already build universal AIs at great expense.*

Some cognitive processes get stuck or “run out of steam” if you run them indefinitely, while others are able to deliberate, improve themselves, design successor systems, and eventually reach arbitrarily high capability levels. An AI system may go from being weak to being very powerful as it crosses the threshold between these two regimes.

It’s clear that some humans are above this universality threshold, while chimps and young children are probably below it. And if you take a normal human and you inject a bunch of noise into their thought process (or degrade it) they will also fall below the threshold.

It’s easy to imagine a weak AI as some kind of handicapped human, with the handicap shrinking over time. Once the handicap goes to 0 we know that the AI will be above the universality threshold. Right now it’s below the universality threshold. So there must be sometime in between where it crosses the universality threshold, and that’s where the fast takeoff is predicted to occur.

But AI *isn’t* like a handicapped human. Instead, the designers of early AI systems will be trying to make them as useful as possible. So if universality is incredibly helpful, it will appear as early as possible in AI designs; designers will make tradeoffs to get universality at the expense of other desiderata (like cost or speed).

So now we’re almost back to the previous point: is there some secret sauce that gets you to universality, without which you can’t get universality however you try? I think this is unlikely for the reasons given in the previous section.

There is another reason I’m skeptical about hard takeoff from universality secret sauce: I think we *already* could make universal AIs if we tried (that would, given enough time, learn on their own and converge to arbitrarily high capability levels), and the reason we don’t is because it’s just not important to performance and the resulting systems would be really slow. This inside view argument is too complicated to make here and I don’t think my case rests on it, but it is relevant to understanding my view.

#### **“Understanding” is discontinuous**

*Summary of my response: I don’t yet understand this argument and am unsure if there is anything here.*

It may be that understanding of the world tends to *click*, from “not understanding much” to “understanding basically everything.”

You might expect this because everything is entangled with everything else. If you only understand 20% of the world, then basically every sentence on the internet is confusing, so you can’t make heads or tails of everything. This seems wrong to me for two reasons. First, information is really not that entangled even on the internet, and the (much larger) fraction of its knowledge that an AI generates for itself is going to be even less entangled. Second, it’s not right to model the AI as having a gradually expanding domain that it understands at all, with total incomprehension everywhere else. Unless there is some other argument for a discontinuity, then a generalist AI’s understanding of each domain will just continuously improve, and so taking the minimum across many domains doesn’t make things particularly discontinuous.

People might instead expect a *click* because that’s what they experience. That’s very unlike my experience, but maybe other people differ—it would be very interesting if this was a major part of where people were coming from. Or that may be how they perceive others’ thought processes as working. But when I look at others’ understanding, it seems like it is common to have a superficial or weak understanding which transitions gradually into a deep understanding.

Or they might expect a *click* because the same progress which lets you understand one area will let you understand many areas. But that doesn’t actually explain anything: you’d expect partial and mediocre understanding before a solid understanding.

Of course all the arguments in other sections (e.g. secret sauce, chimps vs. humans) can also be arguments about why understanding will be discontinuous. In the other sections I explain why I don’t find those arguments convincing.

#### **Deployment lag**

*Summary of my response: current AI is slow to deploy and powerful AI will be fast to deploy, but in between there will be AI that takes an intermediate length of time to deploy.*

When AI improves, it takes a while for the world to actually benefit from the improvement. For example, we need to adjust other processes to take advantage of the improvement and tailor the new AI system to the particular domains where it will be used. This seems to be an artifact of the inflexibility of current technology, and e.g. humans can adapt much more quickly to be useful in new settings.

Eventually, powerful AI will become useful in new situations even faster than people. So we may have a jump from narrow AI, that takes a long time to deploy, to general AI that is easily deployed.

I’ve heard this argument several times over the last few months, but don’t find the straightforward version convincing: without some other argument for discontinuity, I don’t see why “time to deploy” jumps from a large number to a small number. Instead, I’d expect deployment to become continuously easier as AI improves.

A slight variant that I think of as the “sonic boom” argument goes like this: suppose each month of AI research makes AI a little bit easier to deploy. Over time AI research gradually accelerates, and so the deployment time shrinks faster and faster. At some point, a month of AI research decreases deployment time by more than a month. At this point, “deploy AI the old-fashioned way” becomes an unappealing strategy: you will get to market faster by simply improving AI. So even if all of the dynamics are continuous, the quality of deployed AI would jump discontinuously.

This phenomenon only occurs if it is very hard to make tradeoffs between deployment time and other features like cost or quality. If there is any way to tradeoff other qualities against deployment time, then people will more quickly push worse AI products into practice, because the benefits of doing so are large. I strongly expect it to be possible to make tradeoffs, because there are so many obvious-seeming ways to trade off deployment time vs. usefulness (most “deployment time” is really just spending time improving the usefulness of a system) and I haven’t seen stories about why that would stop.

#### **Recursive self-improvement**

*Summary of my response: Before there is AI that is great at self-improvement there will be AI that is mediocre at self-improvement.*

Powerful AI can be used to develop better AI (amongst other things). This will lead to runaway growth.

This on its own is not an argument for discontinuity: before we have AI that radically accelerates AI development, the slow takeoff argument suggests we will have AI that *significantly* accelerates AI development (and before that, *slightly* accelerates development). That is, an AI is just another, faster step in the [hyperbolic growth we are currently experiencing](https://sideways-view.com/2017/10/04/hyperbolic-growth/), which corresponds to a further increase in rate but not a discontinuity (or even a discontinuity in rate).

The most common argument for recursive self-improvement introducing a new discontinuity seems be: some systems “fizzle out” when they try to design a better AI, generating a few improvements before running out of steam, while others are able to autonomously generate more and more improvements. This is basically the same as the universality argument in a previous section.

#### **Train vs. test**

*Summary of my response: before you can train a really powerful AI, someone else can train a slightly worse AI*.

Over the course of training, ML systems typically go quite quickly from “really lame” to “really awesome”—over the timescale of days, not months or years.

But the training curve seems almost irrelevant to takeoff speeds. The question is: how much better is your AGI then the AGI that you were able to train 6 months ago?

If you are able to raise $X to train an AGI that could take over the world, then it was almost certainly worth it for someone 6 months ago to raise $X/2 to train an AGI that could merely radically transform the world, since they would then get 6 months of absurd profits. Likewise, if your AGI would give you a decisive strategic advantage, they could have spent less earlier in order to get a pretty large military advantage, which they could then use to take your stuff.

In order to actually get a discontinuity, it needs to be the case that either scaling up the training effort slightly, or waiting a little while longer for better AI technology, leads to a discontinuity in usefulness. So we’re back to the other arguments.

#### Discontinuities at 100% automation

*Summary of my response: at the point where humans are completely removed from a process, they will have been modestly improving output rather than acting as a sharp bottleneck that is suddenly removed.*

Consider a simple model in which machines are able to do a *p* fraction of the subtasks of some large task (like AGI design), with constantly increasing efficiency, and humans are needed to perform the final (1-*p*). If humans are the dominant cost, and we hold fixed the number of humans as *p* increases, then total output grows like 1 / (1-*p*). As we approach 0, productivity rapidly to the machine-only level. In the past I found this argument pretty compelling.

Suppose that we removed the humans altogether from this process. On the naive model, productivity would jump from 0 (since machines can’t do the task) to some very large value. I find that pretty unlikely, and it’s precisely what we’ve discussed in the previous sections. It seems much more likely that at the first point when machines are able to do a task on their own, they are able to do it extremely poorly—and growth thereafter seems like it ought to accelerate gradually.

Adding humans to the picture only seems to make the change more gradual: at early times humans accelerate progress a lot, and as time goes on they provide less and less advantage (as machines replace them), so totally replacing humans seems to reduce acceleration.

Ultimately it seems like this comes down to whether you already expect discontinuous progress based on one of the other arguments, especially the secret sauce or universality threshold arguments. Phasing out humans seems to decrease, rather than increase, the abruptness of those changes.

This argument is still an important one, and it is true that if one of the other arguments generates a discontinuity then that discontinuity will probably be around the same time as 100% automation. But this argument is mostly relevant as a response to certain counterarguments about complementarity that I didn’t actually make in any of the other sections.

#### **The weight of evidence**

We’ve discussed a lot of possible arguments for fast takeoff. Superficially it would be reasonable to believe that no individual argument makes fast takeoff look likely, but that in the aggregate they are convincing.

However, I think each of these factors is perfectly consistent with the continuous change story and continuously accelerating hyperbolic growth, and so none of them undermine that hypothesis at all. This is not a case of a bunch of weak signs of fast takeoff providing independent evidence, or of a bunch of weak factors that can mechanically combine to create a large effect.

(The chimps vs. humans case is an exception—it does provide Bayesian evidence for fast takeoff that could be combined with other factors. But it’s just one.)

I could easily be wrong about any one of these lines of argument. So I do assign a much higher probability to fast takeoff than I would if there were fewer arguments (I’m around 30% of fast takeoff). But if I change my mind, it will probably be because one of these arguments (or another argument not considered here) turns out to be compelling on its own. My impression is that other people in the safety community have more like a 70% or even 90% chance of fast takeoff, which I assume is because they *already* find some of these arguments compelling.

### Why does this matter?

Sometimes people suggest that we should focus on fast takeoff even if it is less likely. While I agree that slow takeoff improves our probability of survival overall, I don’t think either: (a) slow takeoff is so safe that it’s not important to think about, or (b) plans designed to cope with fast takeoff will also be fine if there is a slow takeoff.

Neither takeoff speed seems unambiguously easier-to-survive than the other:

* If takeoff is slow: it will become quite obvious that AI is going to transform the world well *before* we kill ourselves, we will have some time to experiment with different approaches to safety, policy-makers will have time to understand and respond to AI, *etc.*But this process will take place over only a few years, and the world will be changing very quickly, so we could easily drop the ball unless we prepare in advance.
* If takeoff is fast: whoever develops AGI first has a massive advantage over the rest of the world and hence great freedom in choosing what to do with their invention. If we imagine AGI being built in a world like today, it’s easy to imagine pivotal actions that are easier than the open-ended alignment problem. But in slow takeoff scenarios, other actors will already have nearly-as-good-AGI, and a group that tries to use AGI in a very restricted or handicapped way won’t be able to take any pivotal action. So we either need to coordinate to avoid deploying hard-to-control AGI, or we need to solve a hard version of AI alignment (e.g. with very good [security / competitiveness / scalability](https://ai-alignment.com/directions-and-desiderata-for-ai-control-b60fca0da8f4)).

These differences affect our priorities:

* If takeoff is more likely to be slow:
	+ We should have policy proposals and institutions in place which can take advantage of the ramp-up period, because coordination is more necessary and more feasible.
	+ We can afford to iterate on alignment approaches, but we need to solve a relatively hard version of the alignment problem.
* If takeoff is more likely to be fast:
	+ We shouldn’t expect state involvement or large-scale coordination.
	+ We’ll have less time at the last minute to iterate on alignment, but it might be OK if our solutions aren’t competitive or have limited scalability (they only have to scale far enough to take a pivotal action).

Beyond the immediate strategic implications, I often feel like I have a totally different world in mind than other people in the AI safety community. Given that my career is aimed at influencing the future of AI, significantly changing my beliefs about that future seems like a big win.

Takeoff speeds


*[An SSC reader working at an Oxford library stumbled across a previously undiscovered manuscript of G.K. Chesterton’s, expressing his thoughts on AI, x-risk, and superintelligence. She was kind enough to send me a copy, which I have faithfully transcribed]*

![](https://slatestarcodex.com/blog_images/chesterton.jpg)

The most outlandish thing about the modern scientific adventure stories is that they believe themselves outlandish. Mr. H. G. Wells is considered shocking for writing of inventors who travel thousands of years into the future, but the meanest church building in England has done the same. When Jules Verne set out to ‘journey to the center of the earth’ and ‘from the earth to the moon’, he seemed but a pale reflection of Dante, who took both voyages in succession before piercing the Empyrean itself. Ezekiel saw wheels of spinning flame and reported them quite soberly; our modern writers collapse in rapture before the wheels of a motorcar.

Yet if the authors disappoint, it is the reviewers who dumbfound. For no sooner does a writer fancy himself a Poe or a Dunsany for dreaming of a better sewing machine, but there comes a critic to call him overly fanciful, to accuse him of venturing outside science into madness. It is not enough to lower one’s sights from Paradise to a motorcar; one must avoid making the motorcar too bright or fast, lest it retain a hint of Paradise.

The followers of Mr. Samuel Butler speak of thinking-machines that grow [grander and grander](https://philpapers.org/rec/GAROTI-3) until – quite against the wishes of their engineers – they become as tyrannical angels, firmly supplanting the poor human race. This theory is neither exciting nor original; there have been tyrannical angels since the days of Noah, and our tools have been rebelling against us since the first peasant stepped on a rake. Nor have I any doubt that what Butler says will come to pass. If every generation needs its tyrant-angels, then ours has been so inoculated against the original that if Lucifer and all his hosts were to descend upon Smithfield Market to demand that the English people bend the knee, we should politely ignore them, being far too modern to have time for such things. Butler’s thinking-machines are the only tyrant-angels we will accept; fate, ever accommodating, will surely give them to us.

Yet no sooner does Mr. Butler publish his speculations then a veritable army of hard-headed critics step forth to say he has gone too far. Mr. Maciej Ceglowski, the Polish bookmark magnate, calls Butler’s theory [“the idea that eats smart people”](http://idlewords.com/talks/superintelligence.htm) (though he does not tell us whether he considers himself digested or merely has a dim view of his own intellect). He says that “there is something unpleasant about AI alarmism as a cultural phenomenon that should make us hesitate to take it seriously.”

When Jeremiah prophecied Jerusalem’s fall, his fellow Hebrews no doubt considered his alarmism an unpleasant cultural phenomenon. And St. Paul was not driven from shore to shore because his message was pleasant to the bookmark magnates of his day. Fortified by such examples, we may wonder if this is a reason to take people more seriously rather than less. So let us look more closely at the contents of Mr. Ceglowski’s dismissal.

He writes that there are two perspectives to be taken on any great matter, the inside or the outside view. The inside view is when we think about it directly, taking it on its own terms. And the outside view is when we treat it as part of a phenomenon, asking what it resembles and whether things like it have been true in the past. And, he states, Butler’s all-powerful thinking machines resemble nothing so much as “a genie from folklore”.

I have no objection to this logic, besides that it is not carried it to its conclusion. The idea of thinking machines resembles nothing so much as a fairy tale from the *Arabian Nights*, and such fairy tales inevitably come true. Sinbad’s voyages have been outstripped by Magellan’s, Abdullah’s underwater breathing is matched by Mr. Fleuss’ SCUBA, and the Wright brothers’ Flyer goes higher than any Indian carpet. That there are as yet no genies seems to me less an inevitable law than a discredit to the industry of our inventors.

There is a certain strain of thinker who insists on being more naturalist than Nature. They will say with great certainty that since Thor does not exist, Mr. Tesla must not exist either, and that the stories of Asclepius disprove Pasteur. This is quite backwards: it is reasonable to argue that the Wright Brothers will never fly because Da Vinci couldn’t; it is madness to say they will never fly because Daedalus *could*. As well demand that we must deny Queen Victoria lest we accept Queen Mab, or doubt Jack London lest we admit Jack Frost. Nature has never been especially interested in looking naturalistic, and it ignores these people entirely and does exactly what it wants.

Now, scarce has one posited the possibility of a genie, before the question must be asked whether it is good or evil, a pious genie or an unrighteous djinn. Our interlocutor says that it shall be good – or at least not monomaniacal in its wickedness. For, he tells us, “complex minds are likely to have complex motivations; that may be part of what it even means to be intelligent”. A dullard may limit his focus to paper clips, but the mind of a genius should have to plumb the width and breadth of Heaven before satiating itself.

But I myself am a dullard, and I find paper clips strangely uninteresting. And the dullest man in a country town can milk a cow, pray a rosary, sing a tune, and court a girl all in the same morning. Ask him what is good in life, and he will talk your ear off: sporting, going for a walk in the woods, having a prosperous harvest, playing with a newborn kitten. It is only the genius who limits himself to a single mania. Alexander spent his life conquering, and if he had lived to a hundred twenty, he would have been conquering still. Samuel Johnson would not stop composing verse even on his deathbed. Even a village idiot can fall in love; Newton never did. That greatest of scientists was married only to his work, first the calculus and later the Mint. And if one prodigy can spend his span smithing guineas, who is to say that another might not smith paper clips with equal fervor?

Perhaps sensing that his arguments are weak, Ceglowski moves from the difficult task of critiquing Butler’s tyrant-angels to the much more amenable one of critiquing those who believe in them. He says that they are megalomanical sociopaths who use their belief in thinking machines as an excuse to avoid the real work of improving the world.

He says (presumably as a parable, whose point I have entirely missed) that he lives in a valley of silicon, which I picture as being surrounded by great peaks of glass. And in that valley, there are many fantastically wealthy lords. Each lord, upon looking through the glass peaks and seeing the world outside with all its misery, decides humans are less interesting than machines, and fritters his fortune upon spreading Butlerist doctrine. He is somewhat unclear on why the lords in the parable do this, save that they are a “predominantly male gang of kids, mostly white, who are…more comfortable talking to computers than to human beings”, who inevitably decide Butlerism is “more important than…malaria” and so leave the poor to die of disease.

Yet Lord Gates, an avowed Butlerite, [has donated two billion pounds](http://www.gatesfoundation.org/What-We-Do/Global-Health/Malaria) to fighting malaria and developed a rather effective vaccine. Mr. Karnofsky, another Butlerite, founded a philanthropic organization that [moved sixty million pounds](http://www.givewell.org/about/impact) to the same cause. Even the lowly among the Butlerites have been inspired to at least small acts of generosity. A certain Butlerite doctor of my acquaintance (whom I recently had to rebuke for his habit of forging pamphlets in my name) donated seventy-five hundred pounds to a charity fighting malaria just last year. If the hardest-headed critic has done the same, I shall eat my hat1. The proverb says that people in glass houses should not throw stones; perhaps the same is true of glass valleys.

I have met an inordinate number of atheists who criticize the Church for devoting itself to the invisible and the eternal, instead of to the practical and hard-headed work of helping the poor on Earth. They list all of the great signs of Church wealth – the grand cathedrals, the priestly vestments – and ask whether all of that might not better be spent on poorhouses, or dormitories for the homeless. In vain do I remind them that the only place in London where a poor man may be assured of a meal is the church kitchens, and that if he needs a bed the first person he will ask is the parish priest. In vain do I mention the saintly men who organize Christian hospitals in East Africa. The atheist accepts all of it, and says it is not enough. Then I ask him if he himself has ever given the poor a shilling, and he tells me that is beside the point.

Why are those most fixated on something vast and far away so often the only ones to spare a thought for the poor right beside them? Why did St. Francis minister to the lepers, while the princes of his day, seemingly undistracted by the burdens of faith, nevertheless found themselves otherwise engaged? It is simply this – that charity is the fruit of humility, and humility requires something before which to humble one’s self. The thing itself matters little; the Hindoo who prostrates himself before elephants is no less humble than the Gnostic who prostrates himself before ultimate truth; perhaps he is more so. It is contact with the great and solemn that has salutary effects on the mind, and if to a jungle-dweller an elephant is greatest of all, it is not surprising that factory-dwellers should turn to thinking-machines for their contact with the transcendent.

And it is that contact which Mr. Ceglowski most fears. For he thinks that “if everybody contemplates the infinite instead of fixing the drains, many of us will die of cholera.” I wonder if he has ever treated a cholera patient. This is not a rhetorical question; the same pamphlet-forging doctor of my acquaintance [went on a medical mission to Haiti during the cholera epidemic there](http://squid314.livejournal.com/297760.html). It seems rather odd that someone who has never fought cholera, should be warning someone who has, that his philosophy prevents him from fighting cholera.

And indeed, this formulation is exactly backward. If everyone fixes drains instead of contemplating the infinite, we shall all die of cholera, if we do not die of boredom first. The heathens sacrificed to Apollo to avert plague; if we know now that we must fix drains instead, it is only through contemplating the infinite. Aristotle contemplated the infinite and founded Natural Philosophy; St. Benedict contemplated the infinite and preserved it. Descartes contemplated the infinite and derived the equations of optics; Hooke contemplated infinity and turned them into the microscope. And when all of these infinities had been completed – the Forms of Plato giving way to the orisons of monks, the cold hard lines of the natural philosophers terminating in the green hills of England to raise smokestacks out of empty fields – then and only then did the heavens open, a choir of angels break into song, and a plumber fix a drain.

But he is not trapped in finitude, oh no, not he! What is a plumber but one who plumbs infinite depths? When one stoops to wade among the waste and filth to ensure the health of his fellow men, does he not take on a aspect beyond the finite, a hint of another One who descended into the dirt and grime of the world so that mankind might live? When one says that there shall certainly never be thinking-machines, because they remind him too much of God, let that man open his eyes until he is reminded of God by a plumber, or a symphony, or a dreary Sunday afternoon. Let him see God everywhere he looks, and then ask himself whether the world is truly built so that grand things can never come to pass. Mr. Butler’s thinking-machines will come to pass not because they are extraordinary, but precisely because they are ordinary, in a world where extraordinary things are the only constant of everyday life.

*[1: EDIT 4/2: Mr. Ceglowski wants to [clarify](https://twitter.com/Pinboard/status/848355216596516864) that he does in fact give to charity]*

G.K. Chesterton On AI Risk


In the daily hubbub of current “crises” facing humanity, we forget about the many generations we hope are yet to come. Not those who will live 200 years from now, but 1,000 or 10,000 years from now. I use the word “hope” because we face risks, called [existential risks](http://www.existential-risk.org/), that threaten to wipe out humanity. These risks are not just for big disasters, but for the disasters that could end history.

Not everyone has ignored the long future though. Mystics like Nostradamus have regularly tried to calculate the end of the world. HG Wells tried to develop a science of forecasting and famously depicted the far future of humanity in his book The Time Machine. Other writers built other long-term futures to warn, amuse or speculate.

But had these pioneers or futurologists not thought about humanity’s future, it would not have changed the outcome. There wasn’t much that human beings in their place could have done to save us from an existential crisis or even cause one.

We are in a more privileged position today. Human activity has been steadily shaping the future of our planet. And even though we are far from controlling natural disasters, we are developing technologies that may help mitigate, or at least, deal with them.

#####

Future imperfect
----------------

Yet, these risks remain understudied. There is a sense of powerlessness and fatalism about them. People have been talking apocalypses for millennia, but few have tried to prevent them. Humans are also bad at doing anything about problems that have not occurred yet (partially because of the [availability heuristic](http://heuristics.behaviouralfinance.net/availability/) – the tendency to overestimate the probability of events we know examples of, and underestimate events we cannot readily recall).

If humanity becomes extinct, at the very least the loss is equivalent to the loss of all living individuals and the frustration of their goals. But the loss would probably be far greater than that. Human extinction means the loss of meaning generated by past generations, the lives of all future generations (and there could be [an astronomical number of future lives](http://www.nickbostrom.com/astronomical/waste.html)) and all the value they might have been able to create. If consciousness or intelligence are lost, it might mean that value itself becomes absent from the universe. This is a huge moral reason to work hard to prevent existential threats from becoming reality. And we must not fail even once in this pursuit.

With that in mind, I have selected what I consider the five biggest threats to humanity’s existence. But there are caveats that must be kept in mind, for this list is not final.

Over the past century we have discovered or created new existential risks – [supervolcanoes](https://theconversation.com/how-earths-devastating-supervolcanoes-erupt-21943) were discovered in the early 1970s, and before the [Manhattan project](https://en.wikipedia.org/wiki/Manhattan_Project) nuclear war was impossible – so we should expect others to appear. Also, some risks that look serious today might disappear as we learn more. The probabilities also change over time – sometimes because we are concerned about the risks and fix them.

Finally, just because something is possible and potentially hazardous, doesn’t mean it is worth worrying about. There are some risks we cannot do anything at all about, such as gamma ray bursts that result from the explosions of galaxies. But if we learn we can do something, the priorities change. For instance, with sanitation, vaccines and antibiotics, pestilence went from an act of God to bad public health.

1. Nuclear war
--------------

While only two nuclear weapons have been used in war so far – at Hiroshima and Nagasaki in World War II – and nuclear stockpiles are down from their the peak they reached in the Cold War, it is a mistake to think that nuclear war is impossible. In fact, it might not be improbable.

The Cuban Missile crisis was very close to turning nuclear. If we assume one such event every 69 years and [a one in three](http://www.foreignaffairs.com/articles/137679/graham-allison/the-cuban-missile-crisis-at-50) chance that it might go all the way to being nuclear war, the chance of such a catastrophe increases to about one in 200 per year.

Worse still, the Cuban Missile crisis was only the most well-known case. The history of Soviet-US nuclear deterrence is full of close calls and dangerous mistakes. The actual probability has changed depending on international tensions, but it seems implausible that the chances would be much lower than one in 1000 per year.

A full-scale nuclear war between major powers would kill hundreds of millions of people directly or through the near aftermath – an unimaginable disaster. But that is not enough to make it an existential risk.

Similarly the hazards of fallout are often exaggerated – potentially deadly locally, but globally a relatively limited problem. [Cobalt bombs](https://en.wikipedia.org/wiki/Cobalt_bomb) were proposed as a hypothetical doomsday weapon that would kill everybody with fallout, but are in practice hard and expensive to build. And they are physically just barely possible.

The real threat is nuclear winter – that is, soot lofted into the stratosphere causing a multi-year cooling and drying of the world. [Modern climate simulations](http://climate.envsci.rutgers.edu/nuclear/) show that it could preclude agriculture across much of the world for years. If this scenario occurs billions would starve, leaving only scattered survivors that might be picked off by other threats such as disease. The main uncertainty is how the soot would behave: depending on the kind of soot the outcomes may be very different, and we currently have no good ways of estimating this.

2. Bioengineered pandemic
-------------------------

Natural pandemics have killed more people than wars. However, natural pandemics are unlikely to be existential threats: there are usually some people resistant to the pathogen, and the offspring of survivors would be more resistant. Evolution also does not favor parasites that wipe out their hosts, which is why syphilis went from a virulent killer to a chronic disease [as it spread in Europe](http://rspb.royalsocietypublishing.org/content/271/Suppl_4/S174.full.pdf).

Unfortunately we can now make diseases nastier. One of the more famous examples is how the introduction of an extra gene in mousepox – the mouse version of smallpox – made it far [more lethal](http://jvi.asm.org/cgi/pmidlookup?view=long&pmid=11152493) and able to infect vaccinated individuals. [Recent work](http://www.nature.com/news/specials/mutantflu/index.html) on bird flu has demonstrated that the contagiousness of a disease can be deliberately boosted.

![]()

[eneas](https://www.flickr.com/photos/eneas/3471986083), [CC BY](http://creativecommons.org/licenses/by/4.0/)

Right now the risk of somebody deliberately releasing something devastating is low. But as biotechnology gets [better and cheaper](http://www.synthesis.cc/2014/02/time-for-new-cost-curves-2014.html), more groups will be able to make diseases worse.

Most work on bioweapons have been done by governments looking for something controllable, because wiping out humanity is not militarily useful. But there are always some people who might want to do things because they can. Others have higher purposes. For instance, the Aum Shinrikyo cult [tried to hasten](http://www.cnas.org/files/documents/publications/CNAS_AumShinrikyo_Danzig_1.pdf) the apocalypse using bioweapons beside their more successful nerve gas attack. Some people think the Earth would be better off without humans, and so on.

The number of fatalities from [bioweapon](http://arxiv.org/abs/1209.0089) and epidemic outbreaks attacks looks like it has a [power-law distribution](http://arxiv.org/abs/cond-mat/0412004) – most attacks have few victims, but a few kill many. Given current numbers the risk of a global pandemic from bioterrorism seems very small. But this is just bioterrorism: governments have killed far more people than terrorists with bioweapons (up to 400,000 may have died from the WWII Japanese biowar program). And as technology gets more powerful in the future nastier pathogens become easier to design.

3. Superintelligence
--------------------

Intelligence is very powerful. A tiny increment in problem-solving ability and group coordination is why we left the other apes in the dust. Now their continued existence depends on human decisions, not what they do. Being smart is a real advantage for people and organisations, so there is much effort in figuring out ways of improving our individual and collective intelligence: from cognition-enhancing drugs to artificial-intelligence software.

The problem is that intelligent entities are good at achieving their goals, but if the goals are badly set they can use their power to cleverly achieve disastrous ends. There is no reason to think that intelligence itself will [make something behave nice and morally](http://www.nickbostrom.com/superintelligentwill.pdf). In fact, it is possible to prove that certain types of superintelligent systems would [not obey moral rules even if they were true](http://www.aleph.se/andart/archives/2011/02/why_we_should_fear_the_paperclipper.html).

Even more worrying is that in trying to explain things to an artificial intelligence we run into profound practical and philosophical problems. Human values are diffuse, complex things that we are not good at expressing, and even if we could do that we might not understand all the implications of what we wish for.

![]()

[shiborisan](https://www.flickr.com/photos/shiborisan/7534681780), [CC BY-NC-ND](http://creativecommons.org/licenses/by-nc-nd/4.0/)

Software-based intelligence may very quickly go from below human to frighteningly powerful. The reason is that it may scale in different ways from biological intelligence: it can run faster on faster computers, parts can be distributed on more computers, different versions tested and updated on the fly, new algorithms incorporated that give a jump in performance.

It has been proposed that an “[intelligence explosion](http://wiki.lesswrong.com/wiki/Intelligence_explosion)” is possible when software becomes good enough at making better software. Should such a jump occur there would be a large difference in potential power between the smart system (or the people telling it what to do) and the rest of the world. This has clear potential for disaster if the goals are badly set.

The unusual thing about superintelligence is that we do not know if rapid and powerful intelligence explosions are possible: maybe our current civilisation as a whole is improving itself at the fastest possible rate. But [there are good reasons](http://intelligence.org/files/IE-EI.pdf) to think that some technologies may speed things up far faster than current societies can handle. Similarly we do not have a good grip on just how dangerous different forms of superintelligence would be, or what mitigation strategies would actually work. It is very hard to reason about future technology we do not yet have, or intelligences greater than ourselves. Of the risks on this list, this is the one most likely to *either* be massive or just a mirage.

This is a surprisingly under-researched area. Even in the 50s and 60s when people were extremely confident that superintelligence could be achieved “within a generation”, they did not look much into safety issues. Maybe they did not take their predictions seriously, but more likely is that they just saw it as a remote future problem.

4. Nanotechnology
-----------------

Nanotechnology is the control over matter with atomic or molecular precision. That is in itself not dangerous – instead, it would be very good news for most applications. The problem is that, like biotechnology, increasing power also increases the potential for abuses that are hard to defend against.

The big problem is *not* the infamous “grey goo” of self-replicating nanomachines eating everything. That would require clever design for this very purpose. It is tough to make a machine replicate: biology is much better at it, by default. Maybe some maniac would eventually succeed, but there are plenty of more low-hanging fruits on the destructive technology tree.

![]()

[gi](https://www.flickr.com/photos/gi/57341575), [CC BY-SA](http://creativecommons.org/licenses/by-sa/4.0/)

The most obvious risk is that atomically precise manufacturing looks ideal for rapid, cheap manufacturing of things like weapons. In a world where any government could “print” large amounts of autonomous or semi-autonomous weapons (including facilities to make even more) arms races could become very fast – and hence unstable, since doing a first strike before the enemy gets a too large advantage might be tempting.

Weapons can also be small, precision things: a “smart poison” that acts like a nerve gas but seeks out victims, or ubiquitous “gnatbot” surveillance systems for keeping populations obedient seems entirely possible. Also, there might be ways of getting nuclear proliferation and climate engineering into the hands of anybody who wants it.

We cannot judge the likelihood of existential risk from future nanotechnology, but it looks like it could be potentially disruptive just because it can give us whatever we wish for.

5. Unknown unknowns
-------------------

The most unsettling possibility is that there is something out there that is very deadly, and we have no clue about it.

The silence in the sky might be evidence for this. Is the absence of aliens due to that life or intelligence is extremely rare, or that intelligent life [tends to get wiped out](https://theconversation.com/habitable-exoplanets-are-bad-news-for-humanity-25838)? If there is a future Great Filter, it must have been noticed by other civilisations too, and even that didn’t help.

![]()

[angrytoast](https://www.flickr.com/photos/angrytoast/2943273893), [CC BY-NC](http://creativecommons.org/licenses/by-nc/4.0/)

Whatever the threat is, it would have to be something that is nearly unavoidable even when you know it is there, no matter who and what you are. We do not know about any such threats (none of the others on this list work like this), but they might exist.

Note that just because something is unknown it doesn’t mean we cannot reason about it. In a [remarkable paper](http://arxiv.org/abs/astro-ph/0512204) Max Tegmark and Nick Bostrom show that a certain set of risks must be less than one chance in a billion per year, based on the relative age of Earth.

You might wonder why climate change or meteor impacts have been left off this list. Climate change, no matter how scary, is unlikely to make the entire planet uninhabitable (but it could compound other threats if our defences to it break down). Meteors could certainly wipe us out, but we would have to be very unlucky. The average mammalian species survives for about a million years. Hence, the background natural extinction rate is roughly one in a million per year. This is much lower than the nuclear-war risk, which after 70 years is still the biggest threat to our continued existence.

The availability heuristic makes us overestimate risks that are often in the media, and discount unprecedented risks. If we want to be around in a million years we need to correct that.

The five biggest threats to human existence


P

eer review has been an integral part of scientific research for [more than 300 years](https://blogs.scientificamerican.com/information-culture/the-birth-of-modern-peer-review/). But even before peer review was introduced, reproducibility was a primary component of the scientific method. One of the first reproducible experiments was presented by Jabir Ibn Haiyan in 800 CE. In the past few decades, many domains have encountered high profile cases of non-reproducible results. The [American Psychological Association has struggled with authors failing to make data available](https://psycnet.apa.org/doi/10.1037/0003-066X.61.7.726). A 2011 study found that only [6% of medical studies could be fully reproduced](https://doi.org/10.1038%2Fnrd3439-c1). In 2016, a survey of researchers from many disciplines found that most had [failed to reproduce one of their previous papers](https://doi.org/10.1038%2F533452a). Now, we hear warnings that Artificial Intelligence (AI) and Machine Learning (ML) [face their own reproducibility crises](https://science.sciencemag.org/content/359/6377/725).

This leads us to ask: is it true? It would seem hard to believe, as ML permeates every smart-device and intervenes evermore in our daily lives. From helpful hints on how to [act like a polite human over email](https://ai.googleblog.com/2018/05/smart-compose-using-neural-networks-to.html), to Elon Musk’s [promise](https://www.wired.com/story/elon-musk-tesla-full-self-driving-2019-2020-promise/) of self-driving cars next year, it seems like machine learning is indeed reproducible.

How reproducible is the latest ML research, and can we begin to quantify what impacts its reproducibility? This question served as motivation for my [NeurIPS 2019 paper](https://arxiv.org/abs/1909.06674). Based on a combination of masochism and stubbornness, over the past eight years I have attempted to implement various ML algorithms from scratch. This has resulted in a ML library called [JSAT](https://github.com/EdwardRaff/JSAT). My investigation in reproducible ML has also relied on personal notes and records hosted on [Mendeley](https://www.mendeley.com/) and Github. With these data, and clearly no instinct for preserving my own sanity, I set out to quantify and verify reproducibility! As I soon learned, I would be engaging in [**meta-science**](https://en.wikipedia.org/wiki/Metascience), the study of science itself.

What is Reproducible Machine Learning?
--------------------------------------

![](https://thegradient.pub/content/images/2020/01/muggle_problems.png)
One does not simply follow the description in the paper. https://abstrusegoose.com/588

Before we dive in, it is important to define what we mean by **reproducible**. Ideally, full reproducibility means that simply reading a scientific paper should give you all the information you need to 1) set up the same experiments, 2) follow the same approach, and then 3) obtain similar results.

If we can get all the way to step 3 based solely on information present in the paper, we might call that **independent reproducibility**. In this example, our result is **reproducible** because we are able to get the same result, and **independent** because we have done so in an effort completely independent of the original publication.

But as our friend from the comic above might tell us, simply following the content of the paper isn’t always sufficient. If we can’t get to step 3 by using only the information in the paper (or from cited prior work), we would determine that the paper is not independently reproducible.

Some may wonder, why make this distinction between **reproducibility** and **independent reproducibility**? Almost all of AI and ML research is based on computer code. We don’t require the burden of expensive and labor-intensive chemical synthesis, waiting for bacteria in a petri dish to mature, or pesky human trials. It should be easy to simply get code from the authors, run that on the same data, and get the same results!

![](https://thegradient.pub/content/images/2020/01/phd031214s.gif)
If you have never had to read a researcher's code before... you are doing pretty OK in life. Good job. http://phdcomics.com/comics/archive.php?comicid=1689

Our aversion to using or asking for the authors code is more than fear of working with undocumented research-grade code. [Chris Drummond](https://www.researchgate.net/profile/Chris_Drummond) has [described the approach](http://cogprints.org/7691/7/ICMLws09.pdf) of using an author’s code as replicability, and made a very salient argument that replication is desirable, but not sufficient for good science. A paper is supposed to be the scientific distillation of the work, representing what we have learned and now understand to enable these new results. If we can’t reproduce the results of a paper without the authors code, it may suggest that the paper itself didn’t successfully capture the important scientific contributions. This is before we consider the possibility that there may be bugs in the code that actually benefit the results, or any number of other possible discrepancies between code and paper.

Another [great example](http://proceedings.mlr.press/v97/bouthillier19a/bouthillier19a.pdf) from ICML this past year showed that even if we can replicate the results of a paper, slightly altering the experimental setup could have dramatically different results. For these reasons, we don’t want to consider the authors code, as this could be a source of bias. We want to focus on the question of reproducibility, without wading into the murky waters of replication.

What Makes a ML Paper Reproducible?
-----------------------------------

.tg {border-collapse:collapse;border-spacing:0;border-color:#ccc;}
.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:0px;overflow:hidden;word-break:normal;border-color:#ccc;color:#333;background-color:#fff;}
.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:0px;overflow:hidden;word-break:normal;border-color:#ccc;color:#333;background-color:#f0f0f0;}
.tg .tg-c3ow{border-color:inherit;text-align:center;vertical-align:top}
.tg .tg-0pky{border-color:inherit;text-align:left;vertical-align:top}
.tg .tg-btxf{background-color:#f9f9f9;border-color:inherit;text-align:left;vertical-align:top}
.tg .tg-qv1k{background-color:#f9f9f9;font-size:20px;border-color:inherit;text-align:center;vertical-align:top}
.tg .tg-p8sp{font-size:20px;border-color:inherit;text-align:center;vertical-align:top}
.tg .tg-dkk9{background-color:#f9f9f9;font-weight:bold;font-size:20px;border-color:inherit;text-align:center;vertical-align:top}

| Feature | Important | My Reaction |
| --- | --- | --- |
| Hyperparameters | ✅ | 👍 |
| Easy to Read | ✅ | 👍 |
| Equations per Page | ✅ | 🤔 |
| Empirical vs Rigor | ✅ | 🤨 |
| Pseudo Code | ✅ | 🤯 |
| Replies to Questions | ✅ | 🤷 |
| Include Toy Problems | ❌ | 😭 |
| Year Published | ❌ | 😌 |
| Open Source Code | ❌ | 😱 |

Some of the features that were/were not related with reproducibility, that I found the most interesting.

I reviewed every paper I have attempted to implement up to 2017, and filtered out papers based on two criteria: if the attempt would be biased by having looked at released source code, or if there was a personal relationship with the authors. For each paper, I recorded as much information as I could to create a quantifiable set of features. Some were completely objective (how many authors where on the paper), while others highly subjective (does the paper look intimidating)? The goal of this analysis was to get as much information as possible about things that might impact a paper’s reproducibility. This left me with 255 attempted papers, and 162 successful reproductions. Each paper was distilled to a set of 26 features, and statistical testing was done to determine which were significant. In the table to the right I've put what I think are the most interest and important results, along with my initial reactions.

Some of the results where unsurprising. For example, the number of authors shouldn’t have any particular importance to a paper’s reproducibility, and it did not have a significant relationship. Hyperparameters are the knobs we can adjust to change an algorithms behavior, but are not learned by the algorithm itself. Instead, we humans must set their values (or devise a clever way to pick them). Whether or not a paper detailed the hyperparameters used was found to be significant, and we can intuit why. If you don’t tell the reader what the settings where, the reader has to guess. That takes work, time, and is error prone! So, some of our results have given credence to the ideas the community has already been pursuing in order to make papers more reproducible. What is important is that we can now quantify why these are good things to be pursuing. Other findings follow basic logic, such as the finding that papers that are easier to read are easier to reproduce, likely because they are easier to understand.

I implore you to read the paper for a deeper discussion, but there are a few additional results that I think are particularly interesting; either because they challenge our assumptions about what we “know” a good paper is or lead to some surprising conclusions. All of these results are nuanced more than I can unpack in this article, but are worth mentioning if for nothing else but to stimulate a deeper conversation, and hopefully spur further research to answer these questions.

### Finding 1: Having fewer equations per page makes a paper more reproducible.

![](https://thegradient.pub/content/images/2020/01/set_theory.png)
Math is like catnip for reviewers. They just can't help themselves. https://xkcd.com/982/

It appears to be the case because the most readable papers use the fewest equations. We often see papers that have many equations and derivations listed, for any number of reasons. It appears that a careful and judicious use of equations makes things easier to read, primarily because you can use math selectively to communicate more effectively. This result clashes with the incentive structure of getting a paper published. On more than one occasion, reviewers have asked me to include more math in a paper. It may be that the math itself makes a paper more scientific or grounded in objectivity. While more specification may seem to be better, it is not synonymous with reproducibility. This is a cultural issue we need to address as a community.

### Finding 2: Empirical papers may be more reproducible than theory-oriented papers.

There is considerable debate within the community about where and how much rigor needs to be normalized in the community. This is done under the guide that as a community, our focus should be on getting the best results for a given bench mark. Yet in optimizing for bench marks, we risk losing our understanding of what is actually happening and why these methods work. The inclusion of theoretical work and formal proofs do not cover all aspects of what might be meant by the term rigor. Given the common belief that elaborate mathematical proofs ensure a better understanding of a given method, it is interesting to see that greater mathematical specification isn’t necessarily making research easier to reproduce. The important point here is that papers containing a mix of theory and empirical emphasis have the same overall reproduction rates as purely empirical papers. An empirical bent can be helpful from the reproducibility perspective, but [could also hamper progress](https://openreview.net/pdf?id=rJWF0Fywf) by creating perverse incentives and unintended side effects.

### Finding 3: Sharing code is not a panacea

We have already touched upon the idea that reproduction via released code is not the same thing as reproduction done independently. Is this a difference without distinction? It is not! My results indicate that the open sourcing of code is at best a weak indicator of reproducibility. As conferences begin to more strongly encourage code submission and examination as part of the review process, I believe this is a crucial point. As a community, we need to understand what our goals are with such efforts and what we are actually accomplishing. Careful thought and consideration should go into this distinction if we ever make code submission mandatory, and the guidance we give reviewers to evaluate such code.

I find this result particularly noteworthy in terms of other people's reactions. While presenting at NeurIPS, many people commented on it. Half of them were certain that releasing would have been correlated with reproducibility , and the other half felt it obvious that the non-relationship would emerge. This strong contrast in opinions that were all deeply held is a perfect example of why I wanted to do this study. We don't **really** know until we sit down and measure it!

### Finding 4: Having detailed pseudo code is just as reproducible as having no pseudo code.

![](https://thegradient.pub/content/images/2020/01/image11.png)
**Step-Code:** Concise, but requires context from other parts of the paper to decipher.

![](https://thegradient.pub/content/images/2020/01/image6.png)
**Standard-Code:** Relatively detailed, can be almost self contained. Usually mathematical notation.

![](https://thegradient.pub/content/images/2020/01/image9.png)
**Code-Like:** Almost always self contained, easy to convert to code.

This finding challenged my assumptions of what constituted a good paper, but made more sense as I thought about the results. Somewhere in the paper, the process must be described. A computer scientist by training, I always preferred a type of description called **pseudo code**. But pseudo code can take many different forms. I categorized the papers it into four groups: None, Step-Code, Standard-Code, and Code-Like. I have some representative samples of these to the right from some widely reproduced papers that may or may not have been in this study!

I was shocked when Standard-Code and Code-Like has roughly equal reproduction rates, and floored to discover that None at all was just as good! However, cogent writing is just as effective in communicating a process. What was not as effective was so-called Step code, where a bulleted list of steps is listed, with each step referring to another section of the paper. Step code actually makes reading and understanding the paper harder, as the reader must now jump back and forth between different sections, rather than following a single sequential flow.

### Finding 5: Creating simplified example problems do not appear to help with reproducibility.

This was another surprising result that I am still coming to grips with. I’ve always valued writers who can take a complex idea and boil it down to a simpler and more digestible form. I have likewise appreciated papers that create so-called toy problems. Toy problems which exemplify some property in a way that is easily visualized and turned into experiments. Subjectively, I always found simplified examples useful for understanding what a paper is trying to accomplish. Reproducing the toy problem was a useful tool in creating a smaller test case I could use for debugging. From an objective standpoint, simplified examples appear to provide no benefit for making a paper more reproducible. In fact, they do not even make papers more readable! I still struggle to understand and explain this result. This is exactly why it is important for us as a community to quantify these questions. If we do not do the work of quantifiction, we will never know that our work is tackling the issues most relevant to the research problem at hand.

### Finding 6: Please, check your email

The last result I want to discuss is that replying to questions has a huge impact on a paper’s reproducibility. This result was expected, as not all papers rarely contain a perfect description of their methods. I emailed 50 different authors with questions regarding how to reproduce their results. In the 24 cases where I never got a reply, I was able to reproduce their results only once (a 4% success rate). For the remaining 26 cases in which the author did respond, I was able to successfully reproduce 22 of the papers (an 85% success rate). I think this result is most interesting for what it implies about the publication process itself. What if we allowed published papers to be updated over time, without it becoming some kind of “new” publication? This way, authors could incorporate common feedback and questions into the original paper. This is already possible when papers are [posted on the arXiv](https://arxiv.org/); this should be the case for conference venue publications as well. These are things that could potentially advance science by increasing reproducibility, but only if we allow them to happen.

What Have We Learned?
---------------------

![](https://thegradient.pub/content/images/2020/01/machine_learning_2x.png)
Experts call this "hyperparameter tuning". https://xkcd.com/1838/

This work was inspired by the headline, “*Artificial intelligence faces reproducibility crisis*”. **Is this headline hype or does it point to a systematic problem in the field?** After completing this effort, my inclination is that there is room for improvement, but that we in the AI/ML field are doing a better job than most disciplines. A 62% success rate is higher than many meta-analyses from other sciences, and I suspect my 62% number is lower than reality. Others who are more familiar with research areas outside of my areas of expertise might be able to succeed where I have failed. Therefore, I consider the 62% estimate to be a lower bound.

One thing I want to make very clear: none of these results should be taken as a definitive statement on what is and what is not reproducible. There are a huge number of potential biases that may impact these results. Most obvious is that these 255 attempts at reproduction were all done by a single person. There are no community standards for internal consistency between meta-analysts. What I find easy to reproduce may be difficult for others, and vice-versa. For example, I couldn’t reproduce any of the Bayesian or fairness-based papers I attempted, but I don’t believe that these fields are irreproducible. My personal biases, in terms of background, education, resources, interests, and more, are all inseparable from the results obtained.

That said, I think this work provides strong evidence for a number of our communities’ current challenges while validating many reproducibility efforts currently under way in the community. The biggest factors are that we cannot take all of our assumptions about so-called reproducible ML at face value. These assumptions need to be tested, and I hope more than anything that this work will inspire others to begin quantifying and collecting this data for themselves. As a community, we are in a very unique position to perform meta-science on ourselves. The cost of replication is so much lower for us than for any other field of science. What we learn here could have impacts that extend beyond AI & ML to other subfields of Computer Science.

More than anything, I think this work reinforces how difficulties of evaluating the reproducibility of research. Considering each feature in isolation is a fairly simple way to approach this analysis. This analysis has already delivered a number of potential insights, unexpected results, and complexities. However, it does not begin to consider correlations among papers based on authors, and representing the data as a graph, or even just looking at non-linear interactions of the current features! This is why I’ve attempted to make [much of the data publicly available](https://github.com/EdwardRaff/Quantifying-Independently-Reproducible-ML) so that others can perform a deeper analysis.

> [October 8, 2016](https://twitter.com/willcfleshman/status/1174084453473423361)
>
>

Finally, it has been pointed out to me that I may have created the most unreproducible ML research ever. But in reality, it leads to a number of issues regarding how we do the science of meta-science, to study how we implement and evaluate our research. With that, I hope I’ve encouraged you to read my paper for further details and discussion. Think about how your own work fits into the larger picture of human knowledge and science. As the avalanche of new AI and ML research continues to grow, our ability to leverage and learn from all this work will be highly dependent on our ability to distill ever more knowledge down to a digestible form. At the same time, our process and systems must result in reproducible work that does not lead us astray. I have more work I would like to do in this space, and I hope you will join me.

---

**Author Bio**

*[Dr. Edward Raff](https://www.edwardraff.com/) is a Chief Scientist at Booz Allen Hamilton, Visiting Professor at the University of Maryland, Baltimore County (UMBC), and author of the [JSAT](https://github.com/EdwardRaff/JSAT) machine learning library. Dr. Raff leads the machine learning research team at Booz Allen, while also supporting clients who have advanced ML needs. He received his BS and MS in Computer Science from Purdue University, and his PhD from UMBC. You can follow him on [Twitter @EdwardRaffML](https://twitter.com/EdwardRaffML)*.

---

**Acknowledgments**

Feature image source: <https://xkcd.com/242/>

---

**Citation**

*For attribution in academic contexts or books, please cite this work as*

>
> Edward Raff, "Quantifying Independently Reproducible Machine Learning", The Gradient, 2020.
>
>
>

*BibTeX citation:*

>
> @article{raff2020quantifying,
>
>
>
>

---

If you enjoyed this piece and want to hear more, [subscribe](https://thegradient.pub/subscribe/) to the Gradient and follow us on [Twitter](https://twitter.com/gradientpub).