Forecasting transformative AI: what's the burden of proof?

*Audio also available by searching Stitcher, Spotify, Google Podcasts, etc. for "Cold Takes Audio"*

[Today’s world

Transformative AI

Digital people
World of

Misaligned AI
World run by
Something else

or
or

Stable, galaxy-wide
civilization](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/)

This is one of 4 posts summarizing hundreds of pages of technical reports focused almost entirely on forecasting one number: the year by which transformative AI will be developed.[1](#fn1)

By "transformative AI," I mean "AI powerful enough to bring us into a new, qualitatively different future." I specifically focus on what I'm calling [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/): AI systems that can essentially automate all of the human activities needed to speed up scientific and technological advancement.

The sooner PASTA might be developed, the sooner the world could change [radically](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#impacts-of-pasta), and the more important it seems to be thinking today about how to make that change go well vs. poorly.

In future pieces, I'm going to lay out two methods of making a "best guess" at when we can expect transformative AI to be developed. But first, in this piece, I'm going to address the question: **how good do these forecasting methods need to be in order for us to take them seriously?** In other words, what is the "burden of proof" for forecasting transformative AI timelines?

When someone forecasts transformative AI in the 21st century - especially when they are clear about the [full consequences](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#impacts-of-pasta) it would bring - a common intuitive response is something like: **"It's really out-there and wild to claim that transformative AI is coming this century. So your arguments had better be really good."**

I think this is a very reasonable *first reaction* to forecasts about transformative AI (and it matches my own initial reaction). But I've tried to examine what's driving the reaction and how it might be justified, and having done so, **I ultimately don't agree with the reaction.**

* I think there are a number of reasons to think that transformative AI - or something equally momentous - is *somewhat* likely this century, even before we examine details of AI research, AI progress, etc.

* I also think that on the kinds of multi-decade timelines I'm talking about, we should generally be quite open to very wacky, disruptive, even revolutionary changes. With this backdrop, I think that **specific well-researched estimates of when transformative AI is coming can be credible, even if they involve a lot of guesswork and aren't rock-solid.**

This post tries to explain where I'm coming from.

Below, I will (a) get a bit more specific about which transformative AI forecasts I'm defending; then (b) discuss how to formalize the "That's too wild" reaction to such forecasts; then (c) go through each of the rows below, each of which is a different way of formalizing it.

|  |  |  |
| --- | --- | --- |
| **"Burden of proof" angle** | **Key in-depth pieces (abbreviated titles)** | **My takeaways** |
| It's unlikely that any given century would be the "most important" one. ([More](#most-important-century-skepticism))
  | [Hinge](https://static1.squarespace.com/static/5506078de4b02d88372eee4e/t/5f36b015d9a3691ba8e1096b/1597419543571/Are+we+living+at+the+hinge+of+history.pdf); [Response to Hinge](https://forum.effectivealtruism.org/posts/j8afBEAa7Xb2R9AZN/thoughts-on-whether-we-re-living-at-the-most-influential) | We have many reasons to think this century is a "special" one before looking at the details of AI. Many have been covered in previous pieces; another is covered in the next row.
  |
| What would you forecast about transformative AI timelines, based **only** on basic information about (a) how many years people have been trying to build transformative AI; (b) how much they've "invested" in it (in terms of the number of AI researchers and the amount of computation used by them); (c) whether they've done it yet (so far, they haven't)? ([More](#semi-informative-priors))
  | [Semi-informative Priors](https://www.openphilanthropy.org/blog/report-semi-informative-priors) | Central estimates: 8% by 2036; 13% by 2060; 20% by 2100.[2](#fn2) In my view, this report highlights that the history of AI is short, investment in AI is increasing rapidly, and so we shouldn't be too surprised if transformative AI is developed soon.
  |
| Based on analysis of economic models and economic history, how likely is 'explosive growth' - defined as >30% annual growth in the world economy - by 2100? Is this far enough outside of what's "normal" that we should doubt the conclusion? ([More](#economic-growth))
  | [Explosive Growth](https://www.openphilanthropy.org/could-advanced-ai-drive-explosive-economic-growth), [Human Trajectory](https://www.openphilanthropy.org/blog/modeling-human-trajectory) | [Human Trajectory](https://www.openphilanthropy.org/blog/modeling-human-trajectory) projects the past forward, implying explosive growth by 2043-2065.

[Explosive Growth](https://www.openphilanthropy.org/could-advanced-ai-drive-explosive-economic-growth) concludes: "I find that economic considerations don’t provide a good reason to dismiss the possibility of TAI being developed in this century. In fact, there is a plausible economic perspective from which sufficiently advanced AI systems are expected to cause explosive growth."
  |
| "How have people predicted AI ... in the past, and should we adjust our own views today to correct for patterns we can observe in earlier predictions? ... We’ve encountered the view that AI has been prone to repeated over-hype in the past, and that we should therefore expect that today’s projections are likely to be over-optimistic." ([More](#history-of-))
  | [Past AI Forecasts](https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/what-should-we-learn-past-ai-forecasts) | "The peak of AI hype seems to have been from 1956-1973. Still, the hype implied by some of the best-known AI predictions from this period is commonly exaggerated."
  |

*For transparency, note that the reports for the latter three rows are all [Open Philanthropy](https://www.openphilanthropy.org/) analyses, and I am co-CEO of Open Philanthropy.*
Some rough probabilities
------------------------

Here are some things I believe about transformative AI, which I'll be trying to defend:

* I think there's more than a 10% chance we'll see something PASTA-like enough to qualify as "transformative AI" within 15 years (by 2036); a ~50% chance we'll see it within 40 years (by 2060); and a ~2/3 chance we'll see it this century (by 2100).

* *Conditional on* the above, I think there's at least a 50% chance that we'll soon afterward see a world run by [digital people](https://www.cold-takes.com/how-digital-people-could-change-the-world/) or [misaligned AI](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#misaligned-ai-mysterious-potentially-dangerous-objectives) or something else that would make it fair to say we have "transitioned to a state in which humans as we know them are no longer the main force in world events." (This corresponds to point #1 in my "most important century" definition in the [roadmap](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/).)

* And *conditional on* the above, I think there's at least a 50% chance that whatever *is* the main force in world events will be able to create [a stable galaxy-wide civilization](https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/) for billions of years to come. (This corresponds to point #2 in my "most important century" definition in the [roadmap](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/).)

I've also put a bit more detail on what I mean by the "most important century" [here](https://www.cold-takes.com/some-additional-detail-on-what-i-mean-by-most-important-century/).

Formalizing the "That's too wild" reaction
------------------------------------------

Often, someone states a view that I can't immediately find a concrete flaw in, but that I instinctively think is "just too wild" to be likely. For example, "My startup is going to be the next Google" or "College is going to be obsolete in 10 years" or "As President, I would bring both sides together rather than just being partisan."

I hypothesize that the "This is too wild" reaction to statements like these can *usually* be formalized along the following lines: "Whatever your arguments for X being likely, **there is *some* salient way of looking at things (often oversimplified, but relevant) that makes X look very unlikely.**"

For the examples I just gave:

* *"My startup is going to be the next Google."* There are large numbers of startups (millions?), and the *vast* majority of them don't end up anything like Google. (Even when their founders think they will!)

* *"College is going to be obsolete in 10 years.*" College has been very non-obsolete for hundreds of years.

* *"As President, I would bring both sides together rather than just being partisan."* This is a common thing for would-be US Presidents to say, but partisanship seems to have been getting worse for at least a couple of decades nonetheless.

Each of these cases establishes a sort of **starting point (or "prior" probability) and "burden of proof,"** and we can then **consider further evidence that might overcome the burden.** That is, we can ask things like: what makes this startup different from the many other startups that think they can be the next Google? What makes the coming decade different from all the previous decades that saw college stay important? What's different about this Presidential candidate from the last few?

**There are a number of different ways to think about the burden of proof for my [claims above](#some-rough-probabilities):** a number of ways of getting a prior ("starting point") probability, that can then be updated by further evidence.

Many of these capture different aspects of the "That's too wild" intuition, by generating prior probabilities that (at least initially) make the probabilities I've given look too high.

Below, I will go through a number of these "prior probabilities," and examine what they mean for the "burden of proof" on forecasting methods I'll be discussing in later posts.

Different angles on the burden of proof
---------------------------------------

### "Most important century" skepticism

One angle on the burden of proof is along these lines:

* *Holden claims a 15-30% chance that this is the "most important century" in one sense or another.*[3](#fn3)* *But there are a lot of centuries, and by definition most of them can't be the most important. Specifically:*
	+ *Humans have been around for 50,000 to ~5 million years, depending on how you define "humans."*[4](#fn4) *That's 500 to 50,000 centuries.* + *If we assume that our future is about as long as our past, then there are **1,000 to 100,000 total centuries.*** + *So the prior (starting-point) probability for the "most important century" is **1/100,000 to 1/1,000.**** *It's actually worse than that: Holden has talked about [civilization lasting for billions of years](https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/). That's tens of millions of centuries, so the prior probability of "most important century" is less than **1/10,000,000.***

([Are We Living at the Hinge of History?](https://static1.squarespace.com/static/5506078de4b02d88372eee4e/t/5f36b015d9a3691ba8e1096b/1597419543571/Are+we+living+at+the+hinge+of+history.pdf) argues along these general lines, though with some differences.[5](#fn5))

This argument feels like it is pretty close to capturing my biggest source of past hesitation about the "most important century" hypothesis. However, I think there are **plenty of markers that this is not an average century, even before we consider specific arguments about AI.**

One key point is emphasized in my earlier post, [All possible views about humanity's future are wild](https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/). If you think humans (or our descendants) have billions of years ahead of us, you should think that we are among the very earliest humans, which makes it much more plausible that our time is among the most important. (This point is also emphasized in [Thoughts on whether we're living at the most influential time in history](https://forum.effectivealtruism.org/posts/j8afBEAa7Xb2R9AZN/thoughts-on-whether-we-re-living-at-the-most-influential) as well as the comments on an [earlier version of "Are We Living at the Hinge of History?".](https://forum.effectivealtruism.org/posts/XXLf6FmWujkxna3E6/are-we-living-at-the-most-influential-time-in-history-1))

Additionally, while humanity has existed for a few million years, for most of that time we had extremely low populations and very little in the way of compounding technological progress. Human *civilization* [started about 10,000 years ago](https://en.wikipedia.org/wiki/Neolithic_Revolution), and since then we've already gotten to the point of building digital programmable computers and exploring our solar system.

With these points in mind, it seems reasonable to think we will eventually launch a stable galaxy-wide civilization, sometime in the next 100,000 years (1000 centuries). Or to think there's a 10% chance we will do so sometime in the next 10,000 years (100 centuries). Either way, this implies that a given century has a **~1/1,000** chance of being the most important century for the launch of that civilization - much higher than the figures given earlier in this section. It's still ~100x off from the numbers I [gave above](#some-rough-probabilities), so there's still a burden of proof.

There are further reasons to think this particular century is unusual. For example, see [This Can't Go On](https://www.cold-takes.com/this-cant-go-on/):

* The total size of the world economy has grown more in the last **2** centuries than in all of the rest of history combined.

* The current economic growth rate can't be sustained for more than another **80** centuries or so. (And as discussed below, if its past accelerating trend resumed, it would imply explosive growth and hitting the limits of what's possible this century.)

* It's plausible that science has advanced more in the last **5** centuries than in the rest of history combined.

A final point that makes our time special: we're talking about when to expect transformative AI, and we're living very close in time to the very beginnings of efforts on AI*.* In well under **1** century, we've gone from the first [programmable electronic general-purpose computer](https://en.wikipedia.org/wiki/ENIAC) to AI models that can compete with humans at speech recognition,[6](#fn6) [image classification](http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/) and much more.

More on the implications of this in the next section.

![](https://www.cold-takes.com/content/images/size/w1000/2021/09/burden-of-proof-timeline-3.jpg)Thanks to María Gutiérrez Rojas for this graphic. The top timeline illustrates how recent major milestones for computing and AI are. Below it are (cropped) other timelines showing how significant this few-hundred-year period (more at [This Can't Go On](https://www.cold-takes.com/this-cant-go-on/)), and this era (more at [All Possible Views About Humanity's Future Are Wild](https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/)), appear to be.

### Semi-informative priors

[Report on Semi-informative Priors](https://www.openphilanthropy.org/blog/report-semi-informative-priors) (abbreviated in this piece as "Semi-informative Priors") is an extensive attempt to forecast transformative AI timelines while using as little information about the specifics of AI as possible. So it is one way of providing an angle on the "burden of proof" - that is, establishing a prior (starting-point) set of probabilities for when transformative AI will be developed, before we look at the detailed evidence.

The central information it uses is about *how much effort has gone into developing AI so far.* The basic idea:

* If we had been trying and failing at developing transformative AI for thousands of years, the odds of succeeding in the coming decades would be low.

* But if we've only been trying to develop AI systems for a few decades so far, this means the coming decades could contain a large fraction of all the effort that has ever been put in. The odds of developing it in that time are not all that low.

* One way of thinking about this is that before we look at the details of AI progress, we should be somewhat agnostic about whether developing transformative AI is relatively "easy" (can be done in a few decades) or "hard" (takes thousands of years). Since things are still early, the possibility that it's "easy" is still open.

A bit more on the report's approach and conclusions:

**Angle of analysis.** The report poses the following question (paraphrased): "Suppose you had gone into isolation on the day that people started investing in building AI systems. And now suppose that you've received annual updates on (a) how many years people have been trying to build transformative AI; (b) how much they've 'invested' in it (in terms of time and money); (c) whether they've succeeded yet (so far, they haven't). What can you forecast about transformative AI timelines, having only that information, as of 2021?"

Its methods take inspiration from the [Sunrise Problem](https://en.wikipedia.org/wiki/Sunrise_problem): "Suppose you knew nothing about the universe except whether, on each day, the sun has risen. Suppose there have been N days so far, and the sun has risen on all of them. What is the probability that the sun will rise tomorrow?" You don't need to know anything about astronomy in order to get a decent answer to this question - there are simple mathematical methods for estimating the probability that X will happen tomorrow, based on the fact that X has happened each day in the past. "Semi-informative Priors" extends these mathematical methods in order to adapt them to transformative AI timelines. (In this case, "X" is "Failing to develop transformative AI, as we have in the past.")

**Conclusions.** I'm not going to go heavily into the details of how the analysis works (see the [blog post summarizing the report](https://www.openphilanthropy.org/blog/report-semi-informative-priors) for more detail), but the report's conclusions include the following:

* It puts the probability of artificial general intelligence (AGI, which would include PASTA) by 2036 between 1-18%, with a best guess of 8%.

* It puts the probability of AGI by 2060 at around 3-25% (best guess ~13%), and the probability of AGI by 2100 at around 5-35%, best guess 20%.

These are lower than the probabilities I give [above](#some-rough-probabilities), but not *much* lower. This implies that there **isn't an enormous burden of proof** when bringing in additional evidence about the specifics of AI investment and progress.

**Notes on regime start date.** Something interesting here is that the **report is less sensitive than one might think about how we define the "start date" for trying to develop AGI.** (See [this section of the full report](https://www.openphilanthropy.org/semi-informative-priors#5.3).) That is:

* By default, "Semi-informative Priors" models the situation as if humanity started "trying" to build AGI in 1956.[7](#fn7) This implies that efforts are only ~65 years old, so the coming decades will represent a large fraction of the effort.

* But the report also looks at other measures of "effort to build AGI" - notably, researcher-time and "compute" (processing power). Even if you want to say that we've been implicitly trying to build AGI since the beginning of human civilization ~10,000 years ago, the coming decades will contain a large chunk of the research effort and computation invested in trying to do so.

**Bottom line on this section.**

* Occasionally I'll hear someone say something along the lines of "We've been trying to build transformative AI for decades, and we haven't yet - why do you think the future will be different?" At a minimum, this report reinforces what I see as the common-sense position that a few decades of "no transformative AI yet, despite efforts to build it" doesn't do much to argue against the possibility that transformative AI will arrive in the next decade or few.

* In fact, in the scheme of things, we live extraordinarily close in time to the beginnings of attempts at AI development - **another way in which our century is "special,"** such that we shouldn't be too surprised if it turns out to be the key one for AI development.

### Economic growth

Another angle on the burden of proof is along these lines:

*If PASTA were to be developed anytime soon, and if it were to have the consequences outlined in this series of posts, this would be a massive change in the world - and **the world simply doesn't change that fast.***

*To quantify this: the world economy has grown at a few percent per year for the last 200+ years, and PASTA would [imply](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#impacts-of-pasta) a much faster growth rate, possibly 100% per year or above.*

*If we **were** moving toward a world of explosive economic growth, economic growth should be speeding up today. It's not - it's stagnating, at least in the most developed economies. If AI were really going to revolutionize everything, the least it could be doing now is creating enough value - enough new products, transactions and companies - to make overall US economic growth speed up.*

*AI may lead to cool new technologies, but there's no sign of anything nearly as momentous as PASTA would be. Going from where we are to where PASTA would take us is the kind of sudden change that hasn't happened in the past, and is unlikely to happen in the future.*

(If you aren't familiar with economic growth, you may want to read [my brief explainer](https://www.cold-takes.com/what-is-economic-growth/) before continuing.)

I think this is a reasonable perspective, and it especially makes me skeptical of *very* imminent forecasts for transformative AI (2036 and earlier).

My main response is that the picture of steady growth - "the world economy growing at a few percent per year" - gets a lot more complicated when we pull back and look at all of economic history, as opposed to just the last couple of centuries. From that perspective, economic growth has mostly been accelerating,[8](#fn8) and projecting the acceleration forward could lead to very rapid economic growth in the coming decades.

I wrote about this previously in [The Duplicator](https://www.cold-takes.com/the-duplicator/) and [This Can't Go On](https://www.cold-takes.com/this-cant-go-on/); here I'll very briefly recap the key reports that I cited there.

**[Could Advanced AI Drive Explosive Economic Growth?](https://www.openphilanthropy.org/could-advanced-ai-drive-explosive-economic-growth)** explicitly asks the question, "How likely is 'explosive growth' - defined as >30% annual growth in the world economy - by 2100?" It considers arguments on both sides, including both (a) the long view of history that shows accelerating growth; (b) the fact that growth has been remarkably stable over the last ~200 years, implying that something may have changed.

It concludes: "the possibilities for long-run growth are wide open. Both explosive growth and stagnation are plausible."

**[Modeling the Human Trajectory](https://www.openphilanthropy.org/blog/modeling-human-trajectory)** asks what future we can expect if we extrapolate out existing trends over the course of economic history. The answer is explosive growth by 2043-2065 - not too far from what my [probabilities above suggest](#some-rough-probabilities). This implies to me that the lack of economic acceleration over the last ~200 years could be a "blip" - soon to be resolved by technology development that restores the feedback loop (discussed in [The Duplicator](https://www.cold-takes.com/the-duplicator/)) that can cause acceleration to continue.

To be clear, there are also good reasons not to put too much weight on this as a projection,[9](#fn9) and I am presenting it more as a perspective on the "burden of proof" than as a mainline forecast for when PASTA will be developed.

### History of "AI hype"

Another angle on the burden of proof: I sometimes hear comments along the lines of "AI has been overhyped many times in the past, and transformative AI[10](#fn10) is constantly 'just around the corner' according to excited technologists. Your estimates are just the latest in this tradition. Since past estimates were wrong, yours probably are too."

However, I don't think the history of "AI hype" bears out this sort of claim. [What should we learn from past AI forecasts?](https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/what-should-we-learn-past-ai-forecasts) reviewed histories of AI to try to understand what the actual historical pattern of "AI hype" has been.

Its summary gives the following impressions (note that "HLMI," or "human-level machine intelligence," is a fairly similar idea to PASTA):

* *The peak of AI hype seems to have been from 1956-1973. Still, the hype implied by some of the best-known AI predictions from this period is commonly exaggerated.** *After ~1973, few experts seemed to discuss HLMI (or something similar) as a medium-term possibility, in part because many experts learned from the failure of the field’s earlier excessive optimism.** *The second major period of AI hype, in the early 1980s, seems to have been more about the possibility of commercially useful, narrow-purpose “expert systems,” not about HLMI (or something similar) ...* * *It’s unclear to me whether I would have been persuaded by contemporary critiques of early AI optimism, or whether I would have thought to ask the right kinds of skeptical questions at the time. The most substantive critique during the early years was by Hubert Dreyfus, and my guess is that I would have found it persuasive at the time, but I can’t be confident of that.*

My summary is that it isn't particularly fair to say that there have been many waves of separate, over-aggressive forecasts about transformative AI. Expectations were probably too high in the 1956-1973 period, but I don't think there is much reason here to impose a massive "burden of proof" on well-researched estimates today.

### Other angles on the burden of proof

Here are some other possible ways of capturing the ["That's too wild" reaction](#formalizing-the-):

**"My cause is very important" claims.** Many people - throughout the world today, and throughout history - claim or have claimed that whatever issue they're working on is hugely important, often that it could have global or even galaxy-wide stakes. Most of them have to be wrong.

Here I think the key question is whether this claim is supported by better arguments, and/or more trustworthy people, than other "My cause is very important" claims. If you're this deep into reading about the "most important century" hypothesis, I think you're putting yourself in a good position to answer this question for yourself.

**Expert opinion** will be covered extensively in future posts. For now, my main position is that the claims I'm making neither *contradict* a particular expert consensus, nor are *supported* by one. They are, rather, claims about topics that simply have no "field" of experts devoted to studying them. Some people might choose to ignore any claims that aren't actively supported by a robust expert consensus; but given the stakes, I don't think that is what we should be doing in this case.

(That said, the best available survey of AI researchers has conclusions that seem broadly consistent with [mine](#some-rough-probabilities), as I'll discuss in the next post.)

**Uncaptured "That's too wild" reactions.** I'm sure this piece hasn't captured every possible angle that could be underlying a "That's too wild" reaction. (Though not for lack of trying!) Some people will simply have irreducible intuitions that the claims in this series are too wild to take seriously.

**A general take on these angles.** Something that bugs me about most of the angles in this section is that they seem **too general.** If you simply refuse (absent overwhelming evidence) to believe any claim that fits a "my cause is very important" pattern, or isn't already backed by a robust expert consensus, or simply sounds wild, that seems like a dangerous reasoning pattern. Presumably some people, sometimes, *will* live in the most important century; we should be suspicious of any reasoning patterns that would reliably[11](#fn11) make these people conclude that they don't.

**Next in series:** [Are we "trending toward" transformative AI? (How would we know?)](https://www.cold-takes.com/are-we-trending-toward-transformative-ai-how-would-we-know/)

<

---

Footnotes
---------

1. Of course, the answer could be "A kajillion years from now" or "Never." [↩](#fnref1)- Technically, these probabilities are for “artificial general intelligence”, not transformative AI. The probabilities for transformative AI could be higher if it’s possible to have transformative AI without artificial general intelligence, e.g. by via something like PASTA. [↩](#fnref2)- This corresponds to the second two bullet points from [this section](#some-rough-probabilities). [↩](#fnref3)- From [Wikipedia](https://en.wikipedia.org/wiki/Human_history): "Genetic measurements indicate that the ape lineage which would lead to Homo sapiens diverged from the lineage that would lead to chimpanzees and bonobos, the closest living relatives of modern humans, around 4.6 to 6.2 million years ago.[23] Anatomically modern humans arose in Africa about 300,000 years ago,[24] and reached behavioural modernity about 50,000 years ago.[25]" [↩](#fnref4)- E.g., it emphasizes the odds of being among the most important "people" instead of "centuries." [↩](#fnref5)- I don't have a great single source for this, although you can see [this paper](https://arxiv.org/abs/1610.05256). My informal impression from talking to people in the field is that AI speech recognition is at least quite close to human-level, if not better. [↩](#fnref6)- "The field of AI is largely held to have begun in Dartmouth in 1956" [↩](#fnref7)- There is an [open debate](https://forum.effectivealtruism.org/posts/CWFn9qAKsRibpCGq8/does-economic-history-point-toward-a-singularity) on whether past economic data actually shows sustained acceleration, as opposed to a series of very different time periods with increasing growth rates. I discuss how the debate could change my conclusions [here](https://www.cold-takes.com/a-note-on-historical-economic-growth/). [↩](#fnref8)- "Modeling the Human Trajectory" emphasizes that the model that generates these numbers “is not flexible enough to fully accommodate events as large and sudden as the industrial revolution.” The author adds: "Especially since it imperfectly matches the past, its projection for the future should be read loosely, as merely adding plausibility to an upswing in the next century. Davidson (2021) ["Could Advanced AI Drive Explosive Economic Growth?"] points at one important way the projections could continue to be off for many decades: while the model’s dynamics are dominated by a spiraling economic acceleration, people are still an important input to production, and, if anything becoming wealthy has led to people having fewer children. In the coming decades, that could hamper the predicted acceleration, to the degree we can’t or don’t substitute robots for workers."  [↩](#fnref9)- These comments usually refer to [AGI](https://en.wikipedia.org/wiki/Artificial_general_intelligence) rather than transformative AI, but the concepts are similar enough that I'm using them interchangeably here. [↩](#fnref10)- (Absent overwhelming evidence, which I don't think we should generally assume will always be present when it is "needed.") [↩](#fnref11)


A few quick links re: COVID-19/Delta

There is a lot of confusing and/or misleading information out there about the recent Delta-driven COVID-19 surges.[1](#fn1) Here are some of the links I've found most helpful for getting a handle on the current situation: how risky things are at the moment and how much vaccines are helping. I've prioritized links that are pretty simple and digestible but (IMO) credible. (This isn't a post about all the ways the COVID-19 policy response could be better. I generally recommend [Marginal Revolution](https://www.marginalrevolution.com) for that.)

[microCOVID Project](https://www.microcovid.org/):[2](#fn2) synthesizes lots of research about the transmission risks and provides a calculator for how risky specific activities are. I highly recommend this site on the whole, especially if you are trying to reach consensus with other people on which activities are too risky, but also if you just want to get some quantitative information on COVID-19 risks that was created via scrupulous literature review instead of via a quest for clicky headlines. Its [update on the Delta variant](https://www.microcovid.org/blog/delta) implies that Delta is about 1.5x as contagious per hour of exposure; that vaccines are less effective but still very effective against it (83% risk reduction vs. 90% for Pfizer/Moderna, consistent with [Youyang Gu's rough estimate](https://twitter.com/youyanggu/status/1420803109278916609)). It also [discusses the sources it used to arrive at this conclusion](https://www.microcovid.org/paper/14-research-sources#delta-variant).

[July 13 overview from Tomas Pueyo](https://unchartedterritories.tomaspueyo.com/p/delta-variant-everything-you-need), whom I've generally found to be one of the most careful+clear writers on this topic. I didn't find his [more recent update](https://unchartedterritories.tomaspueyo.com/p/covid-faq-summer-2021-part-1) to add a ton to that, but I'm very much looking forward to his upcoming analysis of "long COVID." He seems to be estimating something like 50% protection against infection and 90% protection against hospitalization, from most vaccines (partially offset by Delta being ~2x more infectious).

[Covidestim.org](https://covidestim.org/) provides something that has been really undersupplied IMO: estimates of true (not just reported) COVID-19 prevalence, up to date, by granular geographic area. A very hacky way to estimate your risk is to assume that in a given area, if you are taking the average level of precautions, you'll have the same odds as the average person of contracting COVID-19, which means you can simply look at "new cases per 100k people" and divide by 100k to get your daily probability of infection. This is especially useful for deciding where to travel. I haven't vetted the methodology for Covidestim.org, but it is recommended by [Youyang Gu](https://youyanggu.com/covid19-resources) (his [other recommendation](https://www.covid19.uga.edu/nowcast.html) is not up to date, not as geographically granular, and only estimates total cases, not cases per 100k population, making it harder to use).

[Nice simple presentation of lots of charts on deaths and vaccinations](https://www.jefftk.com/p/delta-deaths-and-vaccinations) from Jeff Kaufman.

Finally, my favorite links for cutting COVID-19 risk for you and those around you: rapid tests. Worth checking both Amazon and Wal-Mart as availability varies a lot; here's [BinaxNOW at Wal-Mart](https://www.walmart.com/ip/BinaxNOW-COVID-19-Antigen-Self-Test-2-Tests/142089281), [BinaxNOW at Amazon](https://www.amazon.com/BinaxNOW-COVID-19-15-Minute-Emergency-Authorization/dp/B094LVQZJL/ref=sr_1_1?dchild=1&keywords=binaxnow&qid=1628789986&sr=8-1), [QuickVue at Wal-Mart](https://www.walmart.com/ip/Quidel-QuickVue-At-Home-COVID-19-Test-10-Minute-Results-at-Home/929494774), [QuickVue at Amazon](https://www.amazon.com/Quidel-QuickVue-at-Home-COVID-19-Test/dp/B093ND7LX4/ref=sr_1_3?dchild=1&keywords=quickvue&qid=1628789994&sr=8-3). Aside from getting vaccinated, rapid testing seems like by far the least life-disrupting way to cut your risk.

---

Footnotes
---------

1. [This article](https://www.medscape.com/viewarticle/955399?src=mkm_covid_update_210727_MSCPEDIT&uac=392736EN&impID=3531827&faf=1#vp_2) is an example of reporting lots of statistics like "Israeli health officials have said 60% of current hospitalized COVID-19 cases are in vaccinated people" without addressing things like what percentage of all people are vaccinated in the relevant areas, possible sources of reporting bias, etc. [↩](#fnref1)- Disclosure: Catherine Olsson, a former coworker, is one of the [collaborators](https://www.microcovid.org/paper/1-intro#authors) on this project. [↩](#fnref2)
For email filter: florpschmop


Give Sports a Chance

[![](https://www.cold-takes.com/content/images/2021/08/mic-diagram---sports-2.png)](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/)

Around now, the [Most Important Century](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/) series is going to be getting a bit dryer, so I'm going to try making some of the other posts a bit lighter. Specifically, I'm going to try something I call **"Cold Links":** links that I like a lot, that are so old you can't believe I'm posting them now. I think this is a more useful/enjoyable service than it might sound like: it's fun to get collections of links on a theme that are more memorable than "best of the week," and even if you've seen some before, you might enjoy seeing them again. If you end up hating this, [let me know](https://www.guidedtrack.com/programs/4kal2ue/run?posttitle=Give%20Sports%20a%20Chance).

Now: a lot of the links I post here will be about sports. "Boooo I hate sportsball!" you're probably thinking, if you're the kind of person I imagine reading this blog. But try to keep an open mind. I'm here to filter out all the "My team won, be excited for me!" and "Isn't this player incredible, check out [stats that are basically the same stats all top players have]" and "Player X isn't just an athlete, they're a LEADER [this roughly just means their team is good]" and "Player Y might be talented, but they never come through when it counts [this roughly just means their team isn't good]," and get you the links that are truly interesting, inspiring or just amazing.

For someone who doesn't care about who wins, what do sports have to offer? High on my list is **getting to closely observe people being incredibly (like world-outlier-level) intense about something.** I am generally somewhat obsessed with obsession (I think it is a key ingredient in almost every case of someone accomplishing something remarkable). And with sports, you can easily identify which players are in the top-5 in the world at the incredibly competitive things they do; you can safely assume that their level of obsession and competitiveness is beyond what you'll ever be able to wrap your head around; and you can see them in action. A few basketball links that illustrate this:* Kobe Bryant's ["Dear Basketball" poem](https://www.theplayerstribune.com/articles/dear-basketball) that he put out when he retired in 2015. Very short and seriously moving. I wish I felt about any activity the way he feels about basketball. This was turned into an [animated short that won an Oscar](https://www.youtube.com/watch?v=lUcdx4W8Xes&ab_channel=NikosElite) (but I'd recommend just reading the poem).
* LeBron James in a [rare informative interview](https://www.youtube.com/watch?v=sgJE-YKP2Vo&t=145s&ab_channel=ESPN), claiming that he watches "all the games ... at the same time," rattling off 5-6 straight plays from one of them from memory, and glaring beautifully as he says "I don't watch basketball for entertainment."
* The memory thing is real: here's [Stephen Curry succeeding at a game](https://www.youtube.com/watch?v=7y0zejRZXXk&ab_channel=DavidTedla) where they show him clips from basketball games he played in (sometimes years ago) and ask him what happened next.
* There are a lot of stories about how competitive Michael Jordan was; my favorite one is just his [Hall of Fame acceptance speech](https://www.youtube.com/watch?v=XLzBMGXfK4c&ab_channel=OfficialHoophall). (For those of you who don't follow sports, just think of Michael Jordan as "if Jesus were also The Beatles.") At a time when anyone else would be happy, peaceful and grateful, MJ is still settling old scores and smarting under every imagined insult from decades ago. Highlights include 6:20 (where he reveals that he's invited the person who was picked over him to make the team in high school, to reinforce that this was a mistake); 12:00 (where he criticizes his general manager for saying "organizations win championships," as opposed to players); 14:40 (where he thanks a group of other players for "freezing him out" during his rookie season and getting him angry and motivated, then admits that the "freeze-out" may have been a rumor); and 15:35 (an extended "Thank you" to Pat Riley for ... basically being a jerk?). That's the most competitive person in the world right there, and maybe the one person on earth who's above not being petty.

What else is good about sports:

* I think it's fun when people care so deeply about something so intrinsically meaningless. It means we can enjoy their emotional journeys without all the baggage of whether we're endorsing something "good" or "bad." (My wife also loves this about sports - her thing is watching [Last Chance U](https://www.netflix.com/title/80091742) while crying her eyes out.) My next sports post will be a [collection of "heartwarming" links and stories](https://www.cold-takes.com/cold-links-heartwarming-sports-stuff/).

- There's a lot of sports analysis, and I kind of think sports is to social science what the laboratory is to natural sciences. Sports statistics have high sample sizes, stable environments and are exhaustively captured on video, so it's often possible to actually figure out what's going on. It's therefore unusually easy to form your own judgment about whether someone's analysis is good or bad, and that can have lessons for what patterns to look for on other topics. (My view: academic analysis of sports is often almost unbelievably bad, as you can see from some of the [Phil Birnbaum eviscerations](https://www.cold-takes.com/phil-birnbaums-regression-analysis/), whereas average sportswriting and TV commentating is worse than language can convey. Nerdy but non-academic sports analysis websites like [Cleaning the Glass](https://cleaningtheglass.com/), [Football Outsiders](https://www.footballoutsiders.com/) and [FiveThirtyEight](https://fivethirtyeight.com/) are good.)
I'll leave you with this [absurd touchdown run](https://www.youtube.com/watch?v=RnTzBTrGlpw&ab_channel=GurinderBual) by Marshawn Lynch (if you haven't watched much football, keep in mind that usually when someone gets tackled, they fall down), and [Marshawn Lynch's life philosophy](https://kottke.org/16/06/a-deeper-metaphor). If you didn't enjoy that pair of links, go ahead and tune out future sports posts from this blog.


Forecasting Transformative AI, Part 1: What Kind of AI?

*Audio also available by searching Stitcher, Spotify, Google Podcasts, etc. for "Cold Takes Audio"*

[Today’s world

Transformative AI

Digital people
World of

Misaligned AI
World run by
Something else

or
or

Stable, galaxy-wide
civilization](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/)

This is the first of four posts summarizing hundreds of pages of technical reports focused almost entirely on forecasting one number. It's the single number I'd probably most value having a good estimate for: the **year by which transformative AI will be developed.**[1](#fn1)

By "transformative AI," I mean "AI powerful enough to bring us into a new, qualitatively different future." The [Industrial Revolution](https://www.vox.com/future-perfect/2018/11/8/18052076/human-history-in-one-chart-industrial-revolution) is the most recent example of a transformative event; others would include the Agricultural Revolution and the emergence of humans.[2](#fn2)

This piece is going to focus on exploring a particular kind of AI I believe could be transformative: **AI systems that can essentially automate all of the human activities needed to speed up scientific and technological advancement.** I will call this sort of technology Process for Automating Scientific and Technological Advancement, or **PASTA.**[3](#fn3) (I mean PASTA to refer to either a single system or a collection of systems that can collectively do this sort of automation.)

PASTA could resolve the same sort of bottleneck discussed in [The Duplicator](https://www.cold-takes.com/the-duplicator/) and [This Can't Go On](https://www.cold-takes.com/this-cant-go-on/) - the **scarcity of human minds (or something that plays the same role in innovation)**.

PASTA could therefore lead to **[explosive science](https://www.cold-takes.com/this-cant-go-on/#scientific-and-technological-advancement)**, culminating in technologies as impactful as [digital people](https://www.cold-takes.com/how-digital-people-could-change-the-world/). And depending on the details, PASTA systems could have objectives of their own, which could be **dangerous for humanity** and could matter a great deal for [what sort of civilization ends up expanding through the galaxy](https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/).

By talking about PASTA, I'm partly trying to get rid of some unnecessary baggage in the debate over "artificial general intelligence." I don't think we need artificial *general* intelligence in order for this century to be the most important in history. Something narrower - as PASTA might be - would be plenty for that.

To make this idea feel a bit more concrete, the rest of this post will discuss:

* How PASTA could (hypothetically) be developed via roughly modern-day machine learning methods.

* Why this could lead to explosive scientific and technological progress - and why it could be dangerous via PASTA systems having objectives of their own.

Future pieces will discuss how soon we might expect something like PASTA to be developed.

Making PASTA
------------

I'll start with a very brief, simplified characterization of machine learning, which you can skip by clicking [here](#SkipML).

There are essentially two ways to "teach" a computer to do a task:

**Traditional programming.** In this case, you code up extremely specific, step-by-step instructions for completing the task. For example, the chess-playing program [Deep Blue](https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer)) is essentially executing instructions[4](#fn4) along the lines of:

* Receive a digital representation of a chessboard, with numbers indicating (a) which chess piece is on each square; (b) which moves would be legal; (c) which board positions would count as checkmate.

* Check how each legal move would modify the board. Then check how "good" that resulting board is, according to rules like: "If the other player's queen has been captured, that's worth 9 points; if Deep Blue's queen has been captured, that's worth -9 points." These rules could be quite complex,[5](#fn5) but they've all been coded in precisely by humans.

**Machine learning.** This is essentially "training" an AI to do a task by trial and error, rather than by giving it specific instructions. Today, the most common way of doing this is by using an "artificial neural network" (ANN), which you might think of sort of like a "digital brain" that starts in an empty (or random) state: it hasn't yet been wired to do specific things.

For example, [AlphaZero](https://en.wikipedia.org/wiki/AlphaZero) - an AI that has been used to master multiple board games including chess and Go - does something more like this (although it has important elements of "traditional programming" as well, which I'm ignoring for simplicity):

* Plays a chess game against itself (by choosing a legal move, modifying the digital game board accordingly, and then choosing another legal move, etc.) Initially, it's playing by making random moves.

* Every time White wins, it "learns" a small amount, by tweaking the wiring of the ANN ("digital brain") - literally by strengthening or weakening the connections between some "artificial neurons" and others. The tweaks cause the ANN to form a stronger association between game states like what it just saw and "White is going to win." And vice versa when Black wins.

* After a very large number of games, the ANN has become very good at determining - from a digital board game state - which side is likely to win. The ANN can now select moves that make its own side more likely to win.

* The process of "training" the ANN takes a very large amount of trial-and-error: it is initially terrible at chess, and it needs to play a lot of games to "wire its brain correctly" and become good. Once the ANN has been trained once, though, its "digital brain" is now consistently good at the board game it's learned; it can beat its opponents repeatedly.

The latter approach is central for a lot of the recent progress in AI. This is especially true for tasks that are hard to “write down all the instructions” for. For example, humans are able to write down some reasonable guidelines for succeeding at chess, but we know very little about how we ourselves classify images (determine whether some image is of a dog, cat, or something else). So machine learning is particularly essential for tasks like classifying images.

Could PASTA be developed via machine learning? One obvious (but unrealistic) way of doing this might be something like this:

* Instead of playing chess, an AI could play a game called "Cause scientific and technological advancement." That is, it could make “moves” like: download scientific papers, add notes to a file, create designs and instructions for new experiments, design manufacturing processes.

* A panel of human judges could watch from the “sidelines” and give their subjective rating of how fast the AI’s work is causing scientific/technological advancement. The AI could therefore tweak its wiring over time, learning which sorts of moves most effectively cause scientific and technological advancement according to the judges.

This would be wildly impractical, at least compared to how I think things are more likely to play out, but it hopefully gives a starting intuition for what a training process could be trying to accomplish: by providing a signal of "how the AI is doing," it could allow an AI to get good at the goal via trial-and-error and tweaking its internal wiring.

In reality, I'd expect training to be faster and more practical due to things like:

* Different AIs could be trained to perform different sorts of roles related to speeding up science and technology: writing academic papers, designing and critiquing blueprints and manufacturing processes, etc. In many cases, humans already engaged in these activities could generate a lot of data on what it looks like to do them well, which could be used for the sort of training described above. Once different AIs could perform a variety of key roles, "manager" AIs could be trained to oversee and allocate the work of other AIs.

* AIs could also be trained as *judges*. Perhaps one AI could be trained to assess whether a paper contains original ideas, and another could be trained to assess whether a paper contains errors.[6](#fn6) These "judge" AIs could then be used to more efficiently train a third AI learning to write original, correct papers.

* More generally, AIs could learn to do all sorts of other human activities, gaining generic human abilities like the ability to learn from textbooks and the ability to "brainstorm creative solutions to a problem." AIs good at these things could then learn science from textbooks like a normal human, and brainstorm about how to make a breakthrough just like a normal human, etc.
	+ The distinction here is between "using huge numbers of examples to wire a brain" and "an already-wired brain using small amounts of examples to learn quickly, as a human brain does."

	+ Here it would take lots of trial and error for the ANN to become good at "generic" human abilities, but after that the trained ANN could learn how to do specifically *scientific* work as efficiently as a human learns to do it. (In a sense you could imagine that it's been "trained via massive trial-and-error *to have the ability to learn certain sorts of things without needing as much trial-and-error.*")

	+ There is some preliminary evidence (for example, [here](https://openai.com/blog/improving-language-model-behavior/)) that AI systems could go through this pattern of "Learning 'the basics' using a ton of trial-and-error, and learning specific sub-skills using less trial-and-error."[7](#fn7)* I don't particularly expect all of this to happen as part of a single, deliberate development process. Over time, I expect different AI systems to be used for different and increasingly broad tasks, including and especially tasks that help complement human activities on scientific and technological advancement. There could be many different types of AI systems, each with its own revenue model and feedback loop, and their collective abilities could grow to the point where at some point, some set of them is able to do everything (with respect to scientific and technological advancement) that formerly required a human. (For convenience, though, I'll sometimes refer to such a set as PASTA in the singular.)

Developing PASTA will almost certainly be hugely harder and more expensive than it was for AlphaZero. It may require a lot of ingenuity to get around obstacles that exist today (the picture above is surely radically oversimplified, and is there to give basic intuitions). But AI research is simultaneously getting cheaper[8](#fn8) and better-funded. I'll argue in future pieces that the odds of developing PASTA in the coming decades are substantial.

Impacts of PASTA
----------------

### Explosive scientific and technological advancement

I've previously talked about the idea of a potential [explosion in scientific and technological advancement](https://www.cold-takes.com/this-cant-go-on/#scientific-and-technological-advancement), which could lead to a [radically unfamiliar future](https://www.cold-takes.com/how-digital-people-could-change-the-world/).

I've emphasized that such an explosion could be caused by a technology that "dramatically increased the number of 'minds' (humans, or [digital people](https://www.cold-takes.com/how-digital-people-could-change-the-world/), or advanced AIs) pushing forward scientific and technological advancement."

PASTA would fit this bill well, particularly if it were as good as humans (or better) at finding better, cheaper ways to make more PASTA systems. PASTA would have **all of the tools for a productivity explosion that I previously laid out for [digital people](https://www.cold-takes.com/how-digital-people-could-change-the-world/)**:

* PASTA systems could make copies of themselves, including temporary copies, and run them at different speeds.

* They could engage in the sort of loop described in [The Duplicator](https://www.cold-takes.com/the-duplicator/): "more ideas [including ideas for making more/better PASTA systems] → more people [in this case more PASTA systems] → more ideas→..."

![](https://www.cold-takes.com/content/images/size/w1000/2021/09/pasta-stills-1.png)

![](https://www.cold-takes.com/content/images/size/w1000/2021/09/pasta-stills-2.png)

![](https://www.cold-takes.com/content/images/size/w1000/2021/09/pasta-stills-3.png)Thanks to María Gutiérrez Rojas for these graphics, a variation on similar graphics from [The Duplicator](https://www.cold-takes.com/the-duplicator/) and [Digital People Would Be An Even Bigger Deal](https://www.cold-takes.com/how-digital-people-could-change-the-world/#productivity) illustrating the dynamics of explosive growth. Here, instead of people having ideas that increase productivity, it's AI algorithms (denoted by neural network icons).

Why doesn't this feedback loop apply to today's computers and AIs? Because today's computers and AIs aren't able to do *all* of the things required to have new ideas and get themselves copied more efficiently. They play a role in innovation, but innovation is ultimately bottlenecked by humans, whose population is only growing so fast. This is what PASTA would change (it is also what [digital people](https://www.cold-takes.com/how-digital-people-could-change-the-world/) would change).

Additionally: unlike digital copies of humans, PASTA systems might not be attached to their existing identity and personality. A PASTA system might quickly make any edits to its "mind" that made it more effective at pushing science and technology forward. This might (or might not, depending on a lot of details) lead to [recursive self-improvement and an "intelligence explosion."](https://en.wikipedia.org/wiki/Technological_singularity#Background) But even if this *didn't* pan out, simply being as good as humans at making more PASTA systems could cause explosive advancement for the same reasons the [digital people could](https://www.cold-takes.com/how-digital-people-could-change-the-world/#productivity).

### Misaligned AI: mysterious, potentially dangerous objectives

If PASTA were developed as outlined [above](#making-pasta), it's possible that we might know *extremely* little about its inner workings.

AlphaZero - like other modern deep learning systems - is in a sense very poorly understood. We know that it "works." But we don't really know "what it's thinking."

If we want to know why AlphaZero made some particular chess move, we can't look inside its code to find ideas like "Control the center of the board" or "Try not to lose my queen." Most of what we see is just a vast set of numbers, denoting the strengths of connections between different artificial neurons. As with a human brain, we can mostly only guess at what the different parts of the "digital brain" are doing[9](#fn9) (although there are some [early attempts](https://distill.pub/2020/circuits/zoom-in/) to do what one might call "digital neuroscience.")

The "designers" of AlphaZero (discussed above) didn't need much of a vision for how its thought processes would work. They mostly just set it up so that it would get a lot of trial and error, and evolve to get a particular result (win the game it’s playing). Humans, too, evolved primarily through trial and error, with selection pressure to get particular results (survival and reproduction - although the selection worked differently).

![](https://www.cold-takes.com/content/images/2021/08/tired-terminator-inspired-pasta.png)

Like humans, PASTA systems might be good at getting the results they are under pressure to get. But like humans, they might learn along the way to think and do all sorts of other things, and it won't necessarily be obvious to the designers whether this is happening.

Perhaps, due to being optimized for pushing forward scientific and technological advancement, PASTA systems will be in the habit of taking every opportunity to do so. This could mean that they would - given the opportunity - seek to [fill the galaxy with long-lasting space settlements](https://www.cold-takes.com/how-digital-people-could-change-the-world/#lock-in) devoted to science.

Perhaps PASTA will emerge as some byproduct of another objective. For example, perhaps humans will be trying to train systems to make money or amass power and resources, and setting them up to do scientific and technological advancement will just be part of that. In which case, perhaps PASTA systems will just end up as power-and-resources seekers, and will seek to bring the whole galaxy under their control.

Or perhaps PASTA systems will end up with very weird, "random" objectives. Perhaps some PASTA system will observe that it "succeeds" (gets a positive training signal) whenever it does something that causes it to have direct control over an increased amount of electric power (since this is often a result of advancing technology and/or making money), and it will start directly aiming to increase its supply of electric power as much as possible - with the difference between these two objectives not being noticed until it becomes quite powerful. (Analogy: humans have been under selection pressure to pass their genes on, but many have ended up caring more about power, status, enjoyment, etc. than about genes.)

These are scary possibilities if we are talking about AI systems (or collections of systems) that may be more capable than humans in at least some domains.

* PASTA systems might try to fool and defeat humans in order to achieve their goals.

* They might succeed entirely, if they were able to outsmart and/or [outnumber](#explosive-scientific-and-technological-advancement) humans, hack critical systems, and/or develop more powerful weapons. (Just as humans have generally been able to defeat other animals to achieve our goals.)

* Or there might be conflict between different PASTA systems with different goals, perhaps partially (but not fully) controlled by humans with goals of their own. This could lead to general chaos and a hard-to-predict, possibly very bad long-run outcome.

If you're interested in more discussion of whether an AI could or would have its own goals, I'd suggest checking out [Why AI alignment could be hard with modern deep learning](https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/) (Cold Takes guest post), [Superintelligence (book)](https://smile.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom-ebook/dp/B00LOOCGB2/), [The case for taking AI seriously as a threat to humanity (Vox article)](https://www.vox.com/future-perfect/2018/12/21/18126576/ai-artificial-intelligence-machine-learning-safety-alignment), [Draft report on existential risk from power-seeking AI (Open Philanthropy analysis)](https://www.alignmentforum.org/posts/HduCjmXTBD4xYTegv/draft-report-on-existential-risk-from-power-seeking-ai) or one of the many other pieces on this topic.[10](#fn10)

Conclusion
----------

It's hard to predict what a world with PASTA might look like, but two salient possibilities would be:

* PASTA could - by causing an explosion in the rate of scientific and technological advancement - lead quickly to something like digital people, and hence to the sorts of changes to the world described in [Digital People Would Be An Even Bigger Deal](https://www.cold-takes.com/how-digital-people-could-change-the-world/).

* PASTA could lead to technology capable of wiping humans out of existence, such as devastating bioweapons or robot armies. This technology could be wielded by humans for their own purposes, or humans could be manipulated into using it to help PASTA pursue its own ends. Either way could lead to dystopia or human extinction.

The next 3 posts will argue that PASTA is more likely than not to be developed this century.

**Next in series:** [Why AI alignment could be hard with modern deep learning](https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/)

---

Footnotes
---------

1. Of course, the answer could be "A kajillion years from now" or "Never." [↩](#fnref1)- See [this section of](https://docs.google.com/document/d/1IJ6Sr-gPeXdSJugFulwIpvavc0atjHGM82QjIfUSBGQ/edit#heading=h.6t4rel10jbcj) "Forecasting TAI with Biological Anchors" (Cotra (2020)) for a more full definition of "transformative AI." [↩](#fnref2)- I'm sorry. But I do think the rest of the series will be slightly more fun to read this way. [↩](#fnref3)- The examples here are of course simplified. For example, both Deep Blue and AlphaGo incorporate substantial amounts of "tree search," a traditionally-programmed algorithm that has its own "trial and error" process. [↩](#fnref4)- And they can include simulating long chains of future game states. [↩](#fnref5)- Some AIs could be used to determine whether papers are original contributions *based on how they are later cited*; others could be used to determine whether papers are original contributions *based only on the contents of the paper and on previous literature.* The former could be used to train the latter, by providing a "That's correct" or "That's wrong" signal for judgments of originality. Similar methods could be used for training AIs to assess the correctness of papers. [↩](#fnref6)- E.g., <https://openai.com/blog/improving-language-model-behavior/>  [↩](#fnref7)- Due to improvements in hardware and software. [↩](#fnref8)- It's even worse than [spaghetti code](https://en.wikipedia.org/wiki/Spaghetti_code). [↩](#fnref9)- More books: [Human Compatible](https://smile.amazon.com/Human-Compatible-Artificial-Intelligence-Problem-ebook/dp/B07N5J5FTS), [Life 3.0](https://smile.amazon.com/Life-3-0-Being-Artificial-Intelligence-ebook/dp/B06WGNPM7V), and [The Alignment Problem](https://smile.amazon.com/Alignment-Problem-Machine-Learning-Values-ebook/dp/B085T55LGK/).  [↩](#fnref10)


Why talk about 10,000 years from now?

It seems a common reaction to [This Can't Go On](https://www.cold-takes.com/this-cant-go-on) is something like: "OK, so ... you're saying the current level of economic growth can't go on for another 10,000 years. So?? Call me in a few thousand years I guess?"

In general, this blog will often talk about "long" time frames (decades, centuries, millennia) as if they're "short" (compared to the billions of years our universe has existed, millions of years our species has existed, and billions of years that could be in our civilization's future). I sort of try to **imagine myself as a billions-of-years-old observer,** looking at charts like [this](https://www.cold-takes.com/content/images/size/w1000/2021/08/long-vs-short-view.png) and thinking things like "The current economic growth level just got started!" even though it got started several lifetimes ago.

Why think this way?

One reason is that it's just a way of thinking about the world that feels (to me) refreshing/different.

But here are a couple more important reasons.

Effective altruism
------------------

My main obsession is with [effective altruism](https://en.wikipedia.org/wiki/Effective_altruism), or doing as much good as possible. I generally try to pay more attention to things when they "matter more," and I think things "matter more" when they affect larger numbers of persons.[1](#fn1)

I think there will be a LOT more persons[2](#fn2) over the coming billions of years than over the coming generation or few. So I think the long-run future, in some sense, "matters more" than whatever happens over the next generation or few. Maybe it doesn't matter more for me and my loved ones, but it matters more from an "all persons matter equally" perspective.[3](#fn3)

An obvious retort is "But there's nothing we can do that will affect ALL of the people who live over the coming billions of years. We should focus on what we can actually change - that's the next generation or few."

But I'm not convinced of that.

I think we could be in the [most important century of all time](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/), and I think things we do today could end up mattering for billions of years (an obvious example is [reducing risk of existential catastrophes](https://www.amazon.com/dp/B07V9GHKYP/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1)).

And more broadly, if I *couldn't* think of specific ways our actions might matter for billions of years, I'd still be very interested in *looking for them.* I'd still find it useful to try to step back and ask: "Is what I'm reading about in the news important [in the grand scheme of things](https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/)? Could these events matter for whether we end up with [explosion, stagnation or collapse](https://www.cold-takes.com/this-cant-go-on/)? For [what kind of digital civilization we create for the long run](https://www.cold-takes.com/how-digital-people-could-change-the-world/#would-these-impacts-be-a-good-or-bad-thing)? And if not ... what could?"

Appreciating the weirdness of the time we live in
-------------------------------------------------

I think we live in a very weird period of time. It looks really weird on various charts (like [this one](https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/#fnref4), [this one](https://www.cold-takes.com/this-cant-go-on/#fnref4), and [this one](https://www.cold-takes.com/content/images/size/w1000/2021/08/Timeline-10KBCE-10KCE-Aug-3.png)). The vast bulk of scientific and technological advancement, and growth in the economy, has happened in a tiny sliver of time that we are sitting in. And billions of years from now, it will probably *still* be the case that this tiny sliver of time looks like an [outlier in terms of growth and change](https://www.cold-takes.com/this-cant-go-on/#why-cant-this-go-on).

Again, it doesn't *feel* like a tiny sliver, it feels like lifetimes. It's hundreds of years. But that's out of millions (for our species) or billions (for life on Earth).

Sometimes, when I walk down the street, I just look around and think: "This is all SO WEIRD. Whooshing by me are a bunch of people calmly operating steel cars at 40 mph, and over there I see a bunch of people calmly operating a massive crane building a skyscraper, and up in the sky is a plane flying by ... and out of billions of years of life on Earth, it's only us - the humans of the last hundred-or-so years - who have ever been able to do any of this kind of stuff. Practically everything I look at is some crazy futurist technology we just came up with and haven't really had time to adapt to, and we won't have adapted before the next crazy thing comes along.

"And everyone is being very humdrum about their cars and skyscrapers and planes, but this is *not* normal, this is *not* 'how it usually is,' this is not part of a plan or a well-established pattern, this is crazy and weird and short-lived, and it's anyone's guess where it's going next."

I think many of us are instinctively, intuitively dismissive of [wild claims about the future](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/). I think we naturally imagine that there's more stability, solidness and hidden wisdom in "how things have been for generations" than there is.

By trying to **imagine the perspective of someone who's been alive for the whole story** - billions of years, not tens - maybe we can be more open to strange future possibilities. And then, maybe we can be better at noticing the ones that actually might happen, and that our actions today might affect.

So that's why I often try on the lens of saying things like "X has been going on for 200 years and could maybe last another few thousand - bah, that's the blink of an eye!"

---

Footnotes
---------

1. I generally use the term "persons" instead of "people" to indicate that I am trying to refer to every person, animal or thing (AI?) that we should care about the welfare of. [↩](#fnref1)- Even more than you'd intuitively guess, as outlined [here](https://www.cold-takes.com/how-digital-people-could-change-the-world/#space-expansion). [↩](#fnref2)- I wrote a bit about this perspective several years ago, [here](https://www.openphilanthropy.org/blog/radical-empathy). [↩](#fnref3)


The Great Depression, Recession and Stagnation in Full Historical Context

While we're on the subject of [long-view economic history](https://www.cold-takes.com/this-cant-go-on), let's look at the [Great Depression](https://en.wikipedia.org/wiki/Great_Depression), [Great Recession](https://en.wikipedia.org/wiki/Great_Recession) and [Great Stagnation](https://en.wikipedia.org/wiki/The_Great_Stagnation) in full global historical context.

![](https://www.cold-takes.com/content/images/size/w1000/2021/08/the-great-3-3.png)*This chart uses the same (world) economic data as the [previous post](https://www.cold-takes.com/this-cant-go-on). I used thin lines to mark the beginning and end of each "Great." Sorry it's so hard to see, but these are just really tiny periods of time. The blue line is technically slowing or even going down a tiny amount in these periods, but not by enough to be visible.*
I don't recommend putting things in perspective like this *too* often. Occasionally seems good.

For email filter: florpschmop


This Can't Go On

*Audio also available by searching Stitcher, Spotify, Google Podcasts, etc. for "Cold Takes Audio"*

> This piece starts to make the case that **we live in a remarkable century, not just a remarkable era.**  Previous pieces in this [series](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/) talked about the strange future that could be ahead of us *eventually* (maybe 100 years, maybe 100,000).
>
>
>
>
>
> Summary of this piece:* We're used to the world economy growing a few percent per year. This has been the case for many generations.
>
> * However, this is a very unusual situation. Zooming out to all of history, we see that growth has been accelerating; that it's near its historical high point; and that it's faster than it can be for all that much longer (there aren't enough atoms in the galaxy to sustain this rate of growth for even another 10,000 years).
>
> * The world can't just keep growing at this rate indefinitely. We should be ready for other possibilities: stagnation (growth slows or ends), explosion (growth accelerates even more, before hitting its limits), and collapse (some disaster levels the economy).
>
>
>
> The times we live in are unusual and unstable. We shouldn't be surprised if something wacky happens, like an explosion in economic and scientific progress, leading to [technological maturity](https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/). In fact, such an explosion would arguably be right on trend.
>
>
>
>

For as long as any of us can remember, the world economy has grown[1](#fn1) a few percent per year, on average. Some years see more or less growth than other years, but growth is pretty steady overall.[2](#fn2) I'll call this the **Business As Usual** world.

In Business As Usual, the world is constantly changing, and the change is noticeable, but it's not overwhelming or impossible to keep up with. There is a constant stream of new opportunities and new challenges, but if you want to take a few extra years to adapt to them while you mostly do things the way you were doing them before, you can usually (personally) get away with that. In terms of day-to-day life, 2019 was pretty similar to 2018, noticeably but not hugely different from 2010, and hugely but not crazily different from 1980.[3](#fn3)

If this sounds right to you, and you're used to it, and you picture the future being like this as well, then you live in the Business As Usual headspace. When you think about the past and the future, you're probably thinking about something kind of like this:

![Chart of the world economy starting in about 1950. It looks like a straight line (on a logarithmic y-axis), with a dotted straight line projecting more of the same for the future.](https://www.cold-takes.com/content/images/size/w1000/2021/07/business-as-usual.png)Business As Usual

I live in a different headspace, one with a more turbulent past and a more uncertain future. I'll call it the **This Can't Go On** headspace. Here's my version of the chart:

![Chart of the world economy starting in about 5000 BC. Unlike the previous chart, it looks very jagged and irregular, with the line getting steeper and steeper over time. Instead of a single dotted line projecting more of the same, I have several dotted lines projecting different possible futures.](https://www.cold-takes.com/content/images/size/w1000/2021/07/this-cant-go-on.png)This Can't Go On[4](#fn4)

Which chart is the right one? Well, they're using exactly the same historical data - it's just that the Business As Usual chart starts in 1950, whereas This Can't Go On starts all the way back in 5000 BC. **"This Can't Go On" is the whole story; "Business As Usual" is a tiny slice of it.**

![Graphic illustrating that when you zoom in on the 'this can't go on' chart, you get the 'business as usual chart.'](https://www.cold-takes.com/content/images/size/w1000/2021/08/long-vs-short-view.png)

Growing at a few percent a year is what we're all used to. But in full historical context, growing at a few percent a year is crazy. (It's the part where the blue line goes near-vertical.)

This growth has gone on for longer than any of us can remember, but that isn't very long in the scheme of things - just a couple hundred years, out of thousands of years of human civilization. It's a huge acceleration, and it can't go on all that much longer. (I'll flesh out "it can't go on all that much longer" [below](#why-cant-this-go-on).)

The first chart suggests regularity and predictability. The second suggests volatility and dramatically different possible futures.

One possible future is **stagnation:** we'll reach the economy's "maximum size" and growth will essentially stop. We'll all be concerned with how to divide up the resources we have, and the days of a growing pie and a dynamic economy will be over forever.

Another is **explosion:** growth will accelerate further, to the point where the world economy is doubling every year, or week, or hour. A [Duplicator](https://www.cold-takes.com/the-duplicator)-like technology (such as [digital people](https://www.cold-takes.com/how-digital-people-could-change-the-world/) or, as I’ll discuss in future pieces, advanced AI) could drive growth like this. If this happens, everything will be changing far faster than humans can process it.

Another is **collapse:** a global catastrophe will bring civilization to its knees, or wipe out humanity entirely, and we'll never reach today's level of growth again.

Or maybe something else will happen.

Why can't this go on?
---------------------

A good starting point would be [this analysis from Overcoming Bias](https://www.overcomingbias.com/2009/09/limits-to-growth.html), which I'll give my own version of here:

* Let's say the world economy is currently getting 2% bigger each year.[5](#fn5) This implies that the economy would be doubling in size about every 35 years.[6](#fn6)* If this holds up, then 8200 years from now, the economy would be about 3\*1070 times its current size.

 * There are likely fewer than 1070 atoms in our galaxy,[7](#fn7) which we would not be able to travel beyond within the 8200-year time frame.[8](#fn8)* So if the economy were 3\*1070 times as big as today's, and could only make use of 1070 (or fewer) atoms, we'd need to be sustaining **multiple economies as big as today's entire world economy *per atom.***

8200 years might sound like a while, but it's far less time than humans have been around. In fact, it's less time than human (agriculture-based) civilization has been around.

Is it *imaginable* that we could develop the technology to support multiple equivalents of today's entire civilization, per atom available? Sure - but this would require a radical degree of transformation of our lives and societies, far beyond how much change we've seen over the course of human history to date. And I wouldn't exactly *bet* that this is how things are going to go over the next several thousand years. (**Update:** for people who aren't convinced yet, I've [expanded on this argument in another post](https://www.cold-takes.com/more-on-multiple-world-size-economies-per-atom/).)

It seems much more likely that we will "run out" of new scientific insights, technological innovations, and resources, and the regime of "getting richer by a few percent a year" will come to an end. After all, this regime is only a couple hundred years old.

([This post](https://dothemath.ucsd.edu/2011/07/galactic-scale-energy/) does a similar analysis looking at energy rather than economics. It projects that the limits come even sooner. It assumes 2.3% annual growth in energy consumption (less than the historical rate for the USA since the 1600s), and estimates this would use up as much energy as is produced by all the stars in our galaxy within 2500 years.[9](#fn9))

Explosion and collapse
----------------------

So one possible future is stagnation: growth gradually slows over time, and we eventually end up in a no-growth economy. But I don't think that's the most likely future.

The chart above **doesn't show growth slowing down - it shows it accelerating dramatically.** What would we expect if we simply projected that same acceleration forward?

[Modeling the Human Trajectory](https://www.openphilanthropy.org/blog/modeling-human-trajectory) (by Open Philanthropy’s David Roodman) tries to answer exactly this question, by “fitting a curve” to the pattern of past economic growth.[10](#fn10) Its extrapolation implies ***infinite growth* this century**. Infinite growth is a mathematical abstraction, but you could read it as meaning: "We'll see the fastest growth possible before we hit the limits."

In [The Duplicator](https://www.cold-takes.com/the-duplicator), I summarize a broader discussion of this possibility. The upshot is that a growth explosion could be possible, *if* we had the technology to “copy” human minds - or something else that fulfills the same effective purpose, such as [digital people](https://www.cold-takes.com/how-digital-people-could-change-the-world/) or advanced enough AI.

In a growth explosion, the annual growth rate could hit 100% (the world economy doubling in size every year) - which could go on for at most ~250 years before we hit the kinds of limits discussed above.[11](#fn11) Or we could see even faster growth - we might see the world economy double in size every month (which we could sustain for at most 20 years before hitting the limits[12](#fn12)), or faster.

That would be a wild ride: blindingly fast growth, perhaps driven by AIs producing output beyond what we humans could meaningfully track, quickly approaching the limits of what's possible, at which point growth would have to slow.

In addition to stagnation or explosive growth, there's a third possibility: **collapse**. A global catastrophe could cut civilization down to a state where it never regains today's level of growth. Human extinction would be an extreme version of such a collapse. This future isn't suggested by the charts, but we know it's possible.

As Toby Ord’s [The Precipice](https://smile.amazon.com/Precipice-Existential-Risk-Future-Humanity/dp/0316484911?sa-no-redirect=1) argues, asteroids and other "natural" risks don't seem likely to bring this about, but there are a few risks that seem serious and very hard to quantify: climate change, nuclear war (particularly nuclear winter), pandemics (particularly if advances in biology lead to nasty bioweapons), and risks from advanced AI.

With these three possibilities in mind (stagnation, explosion and collapse):

* We live in one of the (two) fastest-growth centuries in all of history so far. (The 20th and 21st.)

* It seems likely that this will at least be one of the ~80 fastest-growing centuries of all time.[13](#fn13)* If the right technology comes along and drives explosive growth, it could be the #1 fastest-growing century of all time - by a lot.

 * If things go badly enough, it could be our last century.

So it seems like this is a quite remarkable century, with some chance of being the most remarkable. This is all based on pretty basic observations, not detailed reasoning about AI (which I will get to in future pieces).

Scientific and technological advancement
----------------------------------------

It’s hard to make a simple chart of how fast science and technology are advancing, the same way we can make a chart for economic growth. But I think that if we could, it would present a broadly similar picture as the economic growth chart.

A fun book I recommend is [Asimov's Chronology of Science and Discovery](https://smile.amazon.com/Asimovs-Chronology-Science-Discovery-Asimov/dp/0060156120/). It goes through the most important inventions and discoveries in human history, in chronological order. The first few entries include "stone tools," "fire," "religion" and "art"; the final pages include "Halley's comet" and "warm superconductivity."

An interesting fact about this book is that **553 out of its 654 pages take place after the year 1500** - even though it starts in the year 4 million BC. I predict other books of this type will show a similar pattern,[14](#fn14) and I believe there were, in fact, more scientific and technological advances in the last ~500 years than the previous several million.[15](#fn15)

![](https://www.cold-takes.com/content/images/size/w1000/2021/08/Timeline-10KBCE-10KCE-Aug-3.png)*In a [previous piece](https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/), I argued that the most significant events in history seem to be clustered around the time we live in, illustrated with [this timeline](https://www.cold-takes.com/content/images/2021/06/fermi-aggressive-1.png). That was looking at billions-of-years time frames. If we zoom in to thousands of years, though, we see something similar: the biggest scientific and technological advances are clustered very close in time to now. To illustrate this, here's a timeline focused on transportation and energy (I think I could've picked just about any category and gotten a similar picture).*

So as with economic growth, the rate of scientific and technological advancement is extremely fast compared to most of history. As with economic growth, presumably there are limits at some point to how advanced technology can become. And as with economic growth, from here scientific and technological advancement could:

* **Stagnate**, as [some are concerned is happening](https://www.theatlantic.com/science/archive/2018/11/diminishing-returns-science/575665/).

* **Explode**, if some technology were developed that dramatically increased the number of "minds" (people, or [digital people](https://www.cold-takes.com/how-digital-people-could-change-the-world/), or advanced AIs) pushing forward scientific and technological development.[16](#fn16)* **Collapse** due to some global catastrophe.

Neglected possibilities
-----------------------

I think there should be some people in the world who inhabit the Business As Usual headspace, thinking about how to make the world better if we basically assume a stable, regular background rate of economic growth for the foreseeable future.

And some people should inhabit the This Can’t Go On headspace, thinking about the ramifications of stagnation, explosion or collapse - and whether our actions could change which of those happens.

But today, it seems like things are far out of balance, with almost all news and analysis living in the Business As Usual headspace.

One metaphor for my headspace is that it feels as though the world is a set of people on a plane blasting down the runway:

We're going much faster than normal, and there isn't enough runway to do this much longer ... and we're accelerating.

And every time I read commentary on what's going on in the world, people are discussing how to arrange your seatbelt as comfortably as possible given that wearing one is part of life, or saying how the best moments in life are sitting with your family and watching the white lines whooshing by, or arguing about whose fault it is that there's a background roar making it hard to hear each other.

If I were in this situation and I didn't know what was next (liftoff), I wouldn't necessarily get it right, but I hope I'd at least be thinking: "This situation seems kind of crazy, and unusual, and temporary. We're either going to speed up even more, or come to a stop, or something else weird is going to happen."

*Thanks to María Gutiérrez Rojas for the graphics in this piece, and Ludwig Schubert for an earlier [timeline graphic](https://www.cold-takes.com/content/images/2021/06/fermi-aggressive-1.png) that this piece's timeline graphic is based on.*

**Next in series:** [Forecasting Transformative AI, Part 1: What Kind of AI?](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/)

---

Footnotes
---------

1. If you have no idea what that means, try my short [economic growth explainer](https://www.cold-takes.com/what-is-economic-growth). [↩](#fnref1)- Global real growth has generally ranged from slightly negative to ~7% per year. [↩](#fnref2)- I'm skipping over 2020 here since it was unusually different from past years, due to the global pandemic and other things. [↩](#fnref3)- For the historical data, see [Modeling the Human Trajectory](https://www.openphilanthropy.org/blog/modeling-human-trajectory). The projections are rough and meant to be visually suggestive rather than using the best modeling approaches.  [↩](#fnref4)- This refers to real GDP growth (adjusted for inflation). 2% is lower than the current world growth figure, and using the world growth figure would make my point stronger. But I think that 2% is a decent guess for "frontier growth" - growth occurring in the already-most-developed economies - as opposed to total world growth, which includes “catchup growth” (previously poor countries growing rapidly, such as China today).

 To check my 2% guess, I downloaded [this US data](https://fred.stlouisfed.org/series/GDPC1) and looked at the annualized growth rate between 2000-2020, 2010-2020, and 2015-2020 (all using July since July was the latest 2020 point). These were 2.5%, 2.2% and 2.05% respectively.  [↩](#fnref5)- 2% growth over 35 years is (1 + 2%)^35 = 2x growth [↩](#fnref6)- [Wikipedia](https://en.wikipedia.org/wiki/Milky_Way)'s highest listed estimate for the Milky Way's mass is 4.5\*10^12 solar masses, each of which [is](https://en.wikipedia.org/wiki/Solar_mass) about 2\*10^30 kg. The mass of a (hydrogen) atom [is](https://en.wikipedia.org/wiki/Observable_universe#Matter_content%E2%80%94number_of_atoms) estimated as the equivalent of about 1.67\*10^-27 kg. (Hydrogen atoms have the lowest mass, so assuming each atom is hydrogen will overestimate the total number of atoms.) So a high-end estimate of the total number of atoms in the Milky Way would be (4.5\*10^12 \* 2\*10^30)/(1.67\*10^-27) =~ 5.4\*10^69. [↩](#fnref7)- [Wikipedia](https://en.wikipedia.org/wiki/Milky_Way#Size_and_mass): "In March 2019, astronomers reported that the mass of the Milky Way galaxy is 1.5 trillion solar masses within a radius of about 129,000 light-years." I'm assuming we can't travel more than 129,000 light-years in the next 8200 years, because this would require far-faster-than-light travel. [↩](#fnref8)- This calculation isn't presented straightforwardly in the post. The key lines are "No matter what the technology, a sustained 2.3% energy growth rate would require us to produce as much energy as the entire sun within 1400 years" and "The Milky Way galaxy hosts about 100 billion stars. Lots of energy just spewing into space, there for the taking. Recall that each factor of ten takes us 100 years down the road. One-hundred billion is eleven factors of ten, so 1100 additional years." 1400 + 1100 = 2500, the figure I cite. This relies on the assumption that the average star in our galaxy offers about as much energy as the sun; I don't know whether that's the case. [↩](#fnref9)- There is an [open debate](https://forum.effectivealtruism.org/posts/CWFn9qAKsRibpCGq8/does-economic-history-point-toward-a-singularity) on whether [Modeling the Human Trajectory](https://www.openphilanthropy.org/blog/modeling-human-trajectory) is fitting the right sort of shape to past historical data. I discuss how the debate could change my conclusions [here](https://www.cold-takes.com/a-note-on-historical-economic-growth/). [↩](#fnref10)- 250 doublings would be a growth factor of about 1.8\*10^75, over 10,000 times the number of atoms in our galaxy. [↩](#fnref11)- 20 years would be 240 months, so if each one saw a doubling in the world economy, that would be a growth factor of about 1.8\*10^72, over 100 times the number of atoms in our galaxy. [↩](#fnref12)- That’s because of the above observation that today’s growth rate can’t last for more than another 8200 years (82 centuries) or so. So the only way we could have more than 82 more centuries with growth equal to today’s is if we also have a lot of centuries with negative growth, ala the zig-zag dotted line in the "This Can't Go On" chart. [↩](#fnref13)- [This dataset](https://osf.io/h3867/) assigns significance to historical figures based on how much they are covered in reference works. It has over 10x as many "Science" entries after 1500 as before; the data set starts in 800 BC. I don't endorse the book that this data set is from, as I think it draws many unwarranted conclusions from the data; here I am simply supporting my claim that most reference works will disproportionately cover years after 1500. [↩](#fnref14)- To be fair, reference works like this may be biased toward the recent past. But I think the big-picture impression they give on this point is accurate nonetheless. Really supporting this claim would be beyond the scope of this post, but the evidence I would point to is (a) the works I'm referencing - I think if you read or skim them yourselves you'll probably come out with a similar impression; (b) the fact that economic growth shows a similar pattern (although the explosion starts more recently; I think it makes intuitive sense that economic growth would follow scientific progress with a lag). [↩](#fnref15)- The papers cited in [The Duplicator](https://www.cold-takes.com/the-duplicator) on this point specifically model an explosion in innovation as part of the dynamic driving explosive economic growth. [↩](#fnref16)