Forecasting transformative AI: the "biological anchors" method in a nutshell

*Audio also available by searching Stitcher, Spotify, Google Podcasts, etc. for "Cold Takes Audio"*

[Today’s world

Transformative AI

Digital people
World of

Misaligned AI
World run by
Something else

or
or

Stable, galaxy-wide
civilization](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/)

> This is one of 4 posts summarizing hundreds of pages of technical reports focused almost entirely on forecasting one number: the year by which transformative AI will be developed.[1](#fn1)
>

By "transformative AI," I mean "AI powerful enough to bring us into a new, qualitatively different future." I specifically focus on what I'm calling [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/): AI systems that can essentially automate all of the human activities needed to speed up scientific and technological advancement.

The sooner PASTA might be developed, the sooner the world could change [radically](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#impacts-of-pasta), and the more important it seems to be thinking today about how to make that change go well vs. poorly.

This post is a layperson-compatible summary of [Ajeya Cotra's "Forecasting Transformative AI with Biological Anchors](https://www.lesswrong.com/posts/KrJfoZzpSDpnrv9va/draft-report-on-ai-timelines)" (which I'll abbreviate below as **"Bio Anchors"**), and its pros and cons.[2](#fn2) It is the forecast I find most informative for transformative AI, with some caveats:

* This approach is relatively complex, and it requires a fairly large number of assumptions and uncertain estimates. These qualities make it relatively difficult to explain, and they are also a mark against the method's reliability.

* Hence, as of today, I don't think this method is as trustworthy as the [examples I gave previously](https://www.cold-takes.com/are-we-trending-toward-transformative-ai-how-would-we-know/#what-kind-of-forecast-am-i-going-for) for forecasting a qualitatively different future. It does not have the simplicity and directness of some of those examples, such as modeling COVID-19's spread. And while climate modeling is also very complex, climate modeling has been worked on by a large number of experts over decades, whereas the Bio Anchors methodology doesn't have much history.

Nonetheless, I think it is the best available "best guess estimate" methodology for transformative AI timelines as of today. And as discussed in the [final section](#pros-and-cons-of-the-biological-anchors-method-for-forecasting-transformative-ai-timelines), one can **step back from a lot of the details to see that this century will likely see us hit some of the more "extreme" milestones in the report that strongly suggest the feasibility of transformative AI.**

(Note: I've also written up a follow-up post about this framework for skeptical readers. See [“Biological anchors” is about bounding, not pinpointing, AI timelines](https://www.cold-takes.com/biological-anchors-is-about-bounding-not-pinpointing-ai-timelines/).)

The basic idea is:

* Modern AI models can "learn" to do tasks via a (financially costly) process known as "training." You can think of training as a massive amount of trial-and-error. For example, voice recognition AI models are given an audio file of someone talking, take a guess at what the person is saying, then are given the right answer. By doing this millions of times, they "learn" to reliably translate speech to text. More: [Training](#training)* The bigger an AI model and the more complex the task, the more the training process costs. Some AI models are bigger than others; to date, none are anywhere near "as big as the human brain" (what this means will be elaborated below). More: [Model size and task type](#model-size-and-task-type)* The biological anchors method asks: **"Based on the usual patterns in how much training costs, how much would it cost to train an AI model as big as a human brain to perform the hardest tasks humans do? And when will this be cheap enough that we can expect someone to do it?"** More: [Estimating the expense](#estimating-the-expense)

Bio Anchors models a broad variety of different ways of approaching this question, generating estimates in a wide range from "aggressive" (projecting transformative AI sooner) to "conservative" (later). But from essentially all of these angles, it places a high probability on transformative AI this century.

![](https://www.cold-takes.com/content/images/size/w1000/2021/08/bio-anchors-probability-chart.png)This chart is from the report. You can roughly read the y-axis as the probability that transformative AI is developed by the year in question, although there is some additional nuance in the report. I won't be explaining what each of the different "Conditional on" models means; it's enough to know that each represents a different angle on forecasting transformative AI.

![](https://www.cold-takes.com/content/images/2021/08/bio-anchors-timeline-hack.png)Thanks to María Gutiérrez Rojas for this graphic. The top timeline gives major milestones for AI computing, past and future (the future ones are projected by Bio Anchors). Below it are (cropped) other timelines showing how significant this few-hundred-year period (more at [This Can't Go On](https://www.cold-takes.com/this-cant-go-on/)), and this era (more at [All Possible Views About Humanity's Future Are Wild](https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/)), appear to be.

I'll now elaborate on each of these a bit more. This is the densest part of this series, and some people might prefer to stick with the above summary and skip to the next post.

Note that Bio Anchors uses a number of different approaches (which it calls "anchors") to estimate transformative AI timelines, and combines them into one aggregate view. In this summary, I'm most focused on a particular set of these - called the "neural net anchors" - which are driving most of the report's aggregate timelines. Some of what I say applies to all anchors, but some applies only to the "neural net anchors."

Training
--------

As discussed [previously](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#making-pasta), there are essentially two ways to "teach" a computer to do a task:

1. **"Program" in extremely specific, step-by-step instructions for completing the task.** When this can be done, the computer can generally execute the instructions very quickly, reliably and cheaply. For example, you might program a computer to examine each record in a database and print the ones that match a user's search terms - you would "instruct" it in exactly how to do this, and it would be able to do the task very well.

- **"Train" an AI to do the task purely by trial and error.** Today, the most common way of doing this is by using a "neural network," which you might think of sort of like a "digital brain" that starts in a random state: it hasn't yet been wired to do specific things. For example, say we want an AI to be able to say whether a photo is of a dog or a cat. It's hard to give fully specific step-by-step instructions for doing this; instead, we can take a neural network and send in a million example images (each one labeled as a "dog" or a "cat"). Each time it sees an example, it will tweak its internal wiring to make it more likely to get the right answer on similar cases in the future. After enough examples, it will be wired to correctly recognize dogs vs. cats.

(We could maybe also move up another level of meta, and try to "train" models to be able to learn from "training" itself as efficiently as possible. This is called "meta-learning," but my understanding is that it hasn't had great success yet.)

"Training" is a sort of brute-force, expensive alternative to "programming." The advantage is that we don't need to be able to provide specific instructions - we can just give an AI lots of examples of doing the task right, and it will learn to do the task. The disadvantage is that we need a ***lot* of examples, which requires a lot of processing power, which costs money.**

How much? This depends on the size of the model (neural network) and the nature of the task itself. For some tasks AIs have learned as of 2021, training a single model could cost millions of dollars. For more complex tasks (such as "[do innovative scientific research](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#making-pasta)") and bigger models (reaching the size of the human brain), training a model could cost far more than that.

Bio Anchors is interested in the question: **"When will it be affordable to train a model, using a relatively crude trial-and-error-based approach, to do the hardest tasks humans can do?"**

These tasks could include the tasks necessary for [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/), such as:

* Learn about science from teachers, textbooks and homework as effectively as a human can.

* Push the frontier of science by asking questions, doing analyses and writing papers, as effectively as a human can.

The next section will discuss how Bio Anchors fleshes out the idea of the "hardest tasks humans can do" (which it assumes would require a "human-brain-sized" model).

Model size and task type
------------------------

Bio Anchors hypothesizes that we can estimate "how expensive it is to train a model" based on two basic parameters: the **model size** and the **task type.**

**Model size.** As stated above, you might think of a neural network as a "digital brain" that starts in a random state. In general, a *larger* "digital brain" - with more digital-versions-of-neurons and digital-versions-of-synapses[3](#fn3) - can learn more complex tasks. A larger "digital brain" also requires more computations - and is hence more expensive - each time it is used (for example, for each example it is learning from).

Drawing on the analysis in [Joe Carlsmith's "How Much Computational Power Does It Take to Match the Human Brain?"](https://www.openphilanthropy.org/brain-computation-report) (abbreviated in this piece as "Brain Computation"), Bio Anchors estimates comparisons between the size of "digital brains" (AI models) and "animal brains" (bee brains, mouse brains, human brains). These estimates imply that **today's AI systems are sometimes as big as insect brains, but never quite as big as mouse brains** - as of this writing, the largest known language model was the first to come reasonably close[4](#fn4) - and **not yet even 1% as big as human brains.**[5](#fn5)

The bigger the model, the more processing power it takes to train. Bio Anchors assumes that a **transformative AI model would need to be about 10x the size of a human brain,** so a lot bigger than any current AI model. (The 10x is to leave some space for the idea that "digital brains" might be less efficient than human brains; see [this section](https://docs.google.com/document/d/1IJ6Sr-gPeXdSJugFulwIpvavc0atjHGM82QjIfUSBGQ/edit#heading=h.z8ucahktj3ug) of the report.) This is one of the reasons it would be very expensive to train.

It could turn out that a smaller AI model is still big enough to learn the above sort of tasks. Or it could turn out that the needed model size is bigger than Bio Anchors estimates, perhaps because Bio Anchors has underestimated the effective "size" of the human brain, or because the human brain is better-designed than "digital brains" by more than Bio Anchors has guessed.

**Task type.** In order to learn a task, an AI model needs to effectively "try" (or "watch") the task a large number of times, learning from trial-and-error. The more costly (in processing power, and therefore money) the task is to try/watch, the more costly it will be for the AI model to learn it.

It's hard to quantify how costly a task is to try/watch. Bio Anchors's attempt to do this is the most contentious part of the analysis, according to the technical reviewers who have reviewed it so far.

You can roughly think of the Bio Anchors framework as saying:

* There are some tasks that a human can do with only a second of thought, such as classifying an image as a cat or dog.

* There are other tasks that might take a human several minutes of thought, such as solving a logic puzzle.

* Other tasks could take hours, days, months or even years, and require not just thinking, but interacting with the environment. For example, writing a scientific paper.

* The tasks on the longer end of this spectrum will be more costly to try/watch, so it will be more costly to train an AI model to do them. For example, it's more costly (takes more time, and more money) to have a million "tries" at a task that takes an hour than it is to have a million "tries" at a task that takes a second.

* However, the framework isn't as simple as this sounds. Many tasks that seem like "long" tasks (such as writing an essay) could in fact be broken into a series of "shorter" tasks (such as writing individual sentences).
	+ If an AI model can be trained to do a shorter "sub-task,", it might be able to do the longer task by simply repeating the shorter sub-task over and over again - without ever needing to be explicitly "trained" to do the longer task.

	+ For example, an AI model might get a million "tries" at the task: "Read a partly-finished essay and write a good next sentence." If it then learns to do this task well, it could potentially write a long essay by simply repeating this task over and over again. It wouldn't need to go into a separate training process where it gets a million "tries" at the more time-consuming task of writing an entire essay.

	+ So it becomes crucial whether the hardest and most important tasks (such as those listed above) are the kind that can be "decomposed" into short/easy tasks.

Estimating the expense
----------------------

Bio Anchors looks at how expensive existing AI models were to train, depending on model size and task type (as defined above). It then extrapolates this to see how expensive an AI model would be to train if it:

* Had a size 10x larger than a human brain.[6](#fn6)* Trained on a task where each "try" took days, weeks, or months of intensive "thinking."

As of today, this sort of training would cost in the ballpark of a million trillion dollars, which is enormously more than total world wealth. So it isn't surprising that nobody has tried to train such a model.

However, Bio Anchors also projects the following trends out into the future:

* Advances in both hardware and software that could make computing power cheaper.

* A growing economy, and a growing role of AI in the economy, that could increase the amount AI labs are able to spend training large models to $1 trillion and beyond.

According to these projections, at some point the "amount AI labs are able to spend" becomes equal to the "expense of training a human-brain-sized model on the hardest tasks." Bio Anchors bases its projections for "when transformative AI will be developed" on when this happens.

Bio Anchors also models uncertainty in all of the parameters above, and considers alternative approaches to the "model size and task type" parameters.[7](#fn7) By doing this, it estimates the probability that transformative AI will be developed by 2030, 2035, etc.

Aggressive or conservative?
---------------------------

Bio Anchors involves a number of simplifications that could cause it to be too aggressive (expecting transformative AI to come sooner than is realistic) or too conservative (expecting it to come later than is realistic).

The argument I most commonly hear that it is "**too aggressive**" is along the lines of: "There's no reason to think that a modern-methods-based AI can learn everything a human does, using trial-and-error training - no matter how big the model is and how much training it does. Human brains can reason in unique ways, unmatched and unmatchable by any AI unless we come up with fundamentally new approaches to AI." This kind of argument is often accompanied by saying that AI systems don't "truly understand" what they're reasoning about, and/or that they are merely imitating human reasoning through pattern recognition.

I think this may turn out to be correct, but I wouldn't bet on it. A full discussion of why is outside the scope of this post, but in brief:

* I am unconvinced that there is a deep or stable distinction between "pattern recognition" and "true understanding" ([this Slate Star Codex piece](https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-general-intelligence/) makes this point). "True understanding" might just be what really good pattern recognition looks like. Part of my thinking here is an intuition that even when people (including myself) superficially appear to "understand" something, their reasoning often (I'd even say usually) breaks down when considering an unfamiliar context. In other words, I think what we think of as "true understanding" is more of an ideal than a reality.

* I feel underwhelmed with the track record of those who have made this sort of argument - I don't feel they have been able to pinpoint what "true reasoning" looks like, such that they could make robust predictions about what would prove difficult for AI systems. (For example, see [this discussion of Gary Marcus's latest critique of GPT3](https://nostalgebraist.tumblr.com/post/628024664310136832/gary-marcus-has-co-authored-a-brief-critique-of), and [similar discussion on Astral Codex Ten](https://astralcodexten.substack.com/p/my-bet-ai-size-solves-flubs?s=r)).

* "Some breakthroughs / fundamental advances are needed" might be true. But for Bio Anchors to be overly aggressive, it isn't enough that *some* breakthroughs are needed; the breakthroughs needed have to be *more than what AI scientists are capable of in the coming decades*, the time frame over which Bio Anchors forecasts transformative AI. It seems hard to be confident that things will play out this way - especially because:
	+ Even moderate advances in AI systems could bring more talent and funding into the field (as is already happening[8](#fn8)).

	+ If money, talent and processing power are plentiful, and progress toward PASTA is primarily held up by some particular weakness of how AI systems are designed and trained, a sustained attempt by researchers to fix this weakness could work. When we're talking about multi-decade timelines, that might be plenty of time for researchers to find whatever is missing from today's techniques.

More broadly, Bio Anchors could be too aggressive due to its assumption that "computing power is the bottleneck":

* It assumes that *if* one could pay for all the computing power to do the brute-force "training" described above for the key tasks (e.g., automating scientific work), transformative AI would (likely) follow.

* Training an AI model doesn't just require purchasing computing power. It requires hiring researchers, running experiments, and perhaps most importantly, finding a way to set up the "trial and error" process so that the AI can get a huge number of "tries" at the key task. It may turn out that doing so is prohibitively difficult.

On the other hand, there are several ways in which Bio Anchors could be **too conservative** (underestimating the likelihood of transformative AI being developed soon).

* Perhaps with enough ingenuity, one could create a transformative AI by "programming" it to do key tasks, rather than having to "train" it (see [above](#training) for the distinction). This could require far less computation, and hence be far less expense. Or one could use a combination of "programming" and "training" to achieve better efficiency than Bio Anchors implies, while still not needing to capture everything via "programming."

* Or one could find far superior approaches to AI that can be "trained" much more efficiently. One possibility here is "meta-learning": effectively training an AI system on the "task" of being trained, itself.

* Or perhaps most likely, over time AI might become a bigger and bigger part of the economy, and there could be a proliferation of different AI systems that have each been customized and invested in to do different real-world tasks. The more this happens, the more opportunity there is for individual ingenuity and luck to result in more innovations, and more capable AI systems in particular economic contexts.
	+ Perhaps at some point, it will be possible to integrate many systems with different abilities in order to tackle some particularly difficult task like "automating science," without needing a dedicated astronomically expensive "training run."

	+ Or perhaps AI that falls short of PASTA will still be useful enough to generate a lot of cash, and/or help researchers make compute cheaper and more efficient. This in turn could lead to still bigger AI models that further increase availability of cash and efficiency of compute. That, in turn, could cause a PASTA-level training run to be affordable earlier than Bio Anchors projects.* Additionally, some technical reviewers of Bio Anchors feel that its treatment of [task type](#model-size-and-task-type) is too conservative. They believe that the most important tasks (and perhaps all tasks) that AI needs to be trained on will be on the "easier/cheaper" end of the spectrum, compared to what Bio Anchors assumes. (See the [above section](#model-size-and-task-type) for what it means for a task to be "easier/cheaper" or "harder/more expensive"). For a related argument, see [Fun with +12 OOMs of Compute](https://www.lesswrong.com/posts/rzqACeBGycZtqCfaX/fun-with-12-ooms-of-compute), which makes the intuitive point that Bio Anchors is imagining a truly massive amount of computation needed to create PASTA, and less could easily be enough.

I don't think it is obvious whether, overall, Bio Anchors is too aggressive (expecting transformative AI to come sooner than is realistic) or too conservative (expecting it to come later). The report itself states that it's likely to be too aggressive over the next few years and too conservative >50 years out, and likely most useful in between.[9](#fn9)

Intellectually, it feels to me as though the report is more likely to be too conservative. I find its [responses](https://docs.google.com/document/d/1cCJjzZaJ7ATbq8N2fvhmsDOUWdm7t3uSSXv6bD0E_GM/edit#heading=h.hu3zy1xzvvo) to the "Too aggressive" points above fairly compelling, and I think the "Too conservative" points are more likely to end up being correct. In particular, I think it's hard to rule out the possibility of ingenuity leading to transformative AI in some far more efficient way than the "brute-force" method contemplated here. And I think the treatment of "task type" is definitely erring in a conservative direction.

However, I also have an intuitive preference (which is related to the "burden of proof" analyses given [previously](https://www.cold-takes.com/forecasting-transformative-ai-whats-the-burden-of-proof/)) to err on the conservative side when making estimates like this. Overall, my best guesses about transformative AI timelines are similar to those of Bio Anchors.

Conclusions of Bio Anchors
--------------------------

Bio Anchors estimates a **>10% chance of transformative AI by 2036, a ~50% chance by 2055, and an ~80% chance by 2100.**

It's also worth noting what the report says about AI systems today. It estimates that:

* Today's largest AI models, such as [GPT-3](https://en.wikipedia.org/wiki/GPT-3), are a **bit smaller than mouse brains, and are starting to get within range (if they were to grow another 100x-1000x) of human brains.** So we might soon be getting close to AI systems that can be trained to do anything that humans can do with ~1 second of thought. Consistent with this, it seems to me that we're just starting to reach the point where language models *sound* like humans who are talking without thinking very hard.[10](#fn10) If anything, "human who puts in no more than 1 second of thought per word" seems somewhat close to what GPT-3 is doing, even though it's much smaller than a human brain.

* It's only very recently that AI models have gotten this big. A "large" AI model before 2020 would be more in the range of a honeybee brain. So for models even in the very recent past, we should be asking whether AI systems seem to be "as smart as insects." Here's [one attempt to compare AI and honeybee capabilities](https://www.lesswrong.com/posts/yW3Tct2iyBMzYhTw7/how-does-bee-learning-compare-with-machine-learning) (by Open Philanthropy intern Guille Costa), concluding that the most impressive honeybee capabilities the author was able to pinpoint do appear to be doable for AI systems.[11](#fn11)

I include these notes because:

* The Bio Anchors analysis seems fully consistent with what we're observing from AI systems today (and have over the last decade or two), while also implying that we're likely to see more transformative abilities in the coming decades.

* I think it's particularly noteworthy that we're getting close to the time when an AI model is "as big as a human brain" (according to the Bio Anchors / [Brain Computation](https://www.openphilanthropy.org/brain-computation-report) estimation method). It may turn out that such an AI model is able to "learn" a lot about the world and produce a lot of economic value, even if it can't yet do the hardest things humans do. And this, in turn, could kick off skyrocketing investment in AI (both money and talent), leading to a lot more innovation and further breakthroughs. This is a simple reason to believe that transformative AI by 2036 is plausible.

Finally, I note that Bio Anchors includes an "evolution" analysis among the different approaches it considers. This analysis hypothesizes that in order to produce transformative AI, one would need to do about as many computations as all animals in history combined, in order to re-create the progress that was made by natural selection.

I consider the "evolution" analysis to be *very* conservative, because machine learning is capable of much faster progress than the sort of trial-and-error associated with natural selection. Even if one believes in something along the lines of "Human brains reason in unique ways, unmatched and unmatchable by a modern-day AI," it seems that whatever is unique about human brains should be re-discoverable if one is able to essentially re-run the whole history of natural selection. And even this very conservative analysis estimates a ~50% chance of transformative AI by 2100.

Pros and cons of the biological anchors method for forecasting transformative AI timelines
------------------------------------------------------------------------------------------

**Cons.** I'll start with what I see as the biggest downside: this is a very complex forecasting framework, which relies crucially on multiple extremely uncertain estimates and assumptions, particularly:

* Whether it's reasonable to believe that an AI system could learn the key tasks listed above (the ones required for PASTA) given enough trial-and-error training.

* How to compare the size of AI models with the size of animal/human brains.

* How to characterize "task type," estimating how "difficult" and expensive a task is to “try” or “watch” once.

* How to use the model size and task type to estimate how expensive it would be to train an AI model to do the key tasks.

* How to estimate future advances in both hardware and software that could make computing power cheaper.

* How to estimate future increases in how much AI labs could be able to spend training models.

This kind of complexity and uncertainty means (IMO) that we shouldn't consider the forecasts to be highly reliable, especially today when the whole framework is fairly new. If we got to the point where as much scrutiny and effort had gone into AI forecasting as climate forecasting, it might be a different matter.

**Pros.** That said, the biological anchors method is essentially the only one I know of that estimates transformative AI timelines from **objective facts** (where possible) **and explicit assumptions** (elsewhere)**.**[12](#fn12)It does not rely on any concepts as vague and intuitive as "how fast AI systems are getting more impressive" (discussed [previously](https://www.cold-takes.com/are-we-trending-toward-transformative-ai-how-would-we-know/#subjective-extrapolations-and-)). Every assumption and estimate in the framework can be explained, discussed, and - over time - tested.

Even in its current early stage, I consider this a valuable property of the biological anchors framework. It means that the framework can give us timelines estimates that aren't simply rehashes of intuitions about whether it feels as though transformative AI is approaching.[13](#fn13)

I also think it's encouraging that even with all the guesswork, the testable "predictions" the framework makes as of today seem reasonable (see previous section). **The framework provides a way of thinking about how it could be simultaneously true that (a) the AI systems of a decade ago didn't seem very impressive at all; (b) the AI systems of today can do many impressive things but still feel far short of what humans are able to do; (c) the next few decades - or even the next 15 years - could easily see the development of transformative AI.**

Additionally, I think it's worth noting a **couple of high-level points** from Bio Anchors that **don't depend on quite so many estimates and assumptions:**

* In the coming decade or so, we're likely to see - for the first time - AI models with comparable "size" to the human brain.

* If AI models continue to become larger and more efficient at the rates that Bio Anchors estimates, it will probably become **affordable this century to hit some pretty extreme milestones - the "high end" of what Bio Anchors thinks might be necessary.** These are hard to summarize, but see the "long horizon neural net" and "evolution anchor" frameworks in the report.

* One way of thinking about this is that the next century will likely see us go from "not enough compute to run a human-sized model at all" to "extremely plentiful compute, as much as even quite conservative estimates of what we might need." Compute isn't the only factor in AI progress, but to the extent other factors (algorithms, training processes) became the new bottlenecks, there will likely be powerful incentives (and multiple decades) to resolve them.

A final advantage of Bio Anchors is that we can continue to watch AI progress over time, and compare what we see to the report's framework. For example, we can watch for:

* Whether there are some tasks that just can't be learned, even with plenty of trial and error - or whether some tasks require amounts of training very different from what the report estimates.

* How AI models' capabilities compare to those of animals that we are currently modeling as "similarly sized." If AI models seem more capable than such animals, we may be overestimating how large a model we would need to be in order to e.g. automate science. If they seem less capable, we may be underestimating it.

* How hardware and software are progressing, and whether AI models are getting bigger at the rate the report currently projects.

The next piece will summarize all of the different analyses so far about transformative AI timelines. It will then discuss a remaining reservation: that there is no robust expert consensus on this topic.

**Next in series:** [AI Timelines: Where the Arguments, and the "Experts," Stand](https://www.cold-takes.com/where-ai-forecasting-stands-today/)

---

Footnotes
---------

1. Of course, the answer could be "A kajillion years from now" or "Never." [↩](#fnref1)- For transparency, note that this is an [Open Philanthropy](https://www.openphilanthropy.org) analysis, and I am co-CEO of Open Philanthropy. [↩](#fnref2)- I (like Bio Anchors) generally consider the synapse count more important than the neuron count, for reasons I won't go into here. [↩](#fnref3)- [Wikipedia](https://en.wikipedia.org/wiki/GPT-3): "GPT-3's full version has a capacity of 175 billion machine learning parameters ... Before the release of GPT-3, the largest language model was Microsoft's Turing NLG, introduced in February 2020, with a capacity of 17 billion parameters." Wikipedia doesn't state this, but I don't believe there are publicly known AI models larger than these language models (with the exception of "[mixture-of-experts models](https://arxiv.org/abs/1701.06538)" that I think we should disregard for these purposes, for reasons I won't go into here). [Wikipedia estimates](https://en.wikipedia.org/wiki/List_of_animals_by_number_of_neurons) about 1 trillion synapses for a house mouse's brain; Bio Anchors's methodology for brain comparisons (based on [Brain Computation](https://www.openphilanthropy.org/brain-computation-report)) essentially equates synapses to parameters. [↩](#fnref4)- Bio Anchors estimates about 100 trillion parameters for the human brain, based on the fact that it has about 100 trillion synapses. [↩](#fnref5)- As noted above, the 10x is to leave some space for the idea that "digital brains" might be less efficient than human brains. See [this section](https://docs.google.com/document/d/1IJ6Sr-gPeXdSJugFulwIpvavc0atjHGM82QjIfUSBGQ/edit#heading=h.z8ucahktj3ug) of the report. [↩](#fnref6)- For example, one approach hypothesizes that training could be made cheaper by "meta-learning," discussed above; another approach hypothesizes that in order to produce transformative AI, one would need to do about as many computations as all animals in history combined, in order to re-create the progress that was made by natural selection.) [↩](#fnref7)- See charts from the early sections of the [2021 AI Index Report](https://aiindex.stanford.edu/wp-content/uploads/2021/03/2021-AI-Index-Report_Master.pdf), for example. [↩](#fnref8)- See [this section](https://docs.google.com/document/d/1IJ6Sr-gPeXdSJugFulwIpvavc0atjHGM82QjIfUSBGQ/edit#heading=h.y045l51rb826). [↩](#fnref9)- For a collection of links to GPT-3 demos, see [this post](https://www.lesswrong.com/posts/6Hee7w2paEzHsD6mn/collection-of-gpt-3-results). [↩](#fnref10)- In fact, he estimates that AI systems appear to use about 1000x less compute, which would match the above point in terms of suggesting that AI systems might be more efficient than animal/human brains and that the Bio Anchors estimates might be too conservative. However, he doesn't address the fact that bees arguably perform a more diverse set of tasks than the AI systems they're being compared to. [↩](#fnref11)- Other than the "semi-informative priors" method discussed [previously](https://www.cold-takes.com/forecasting-transformative-ai-whats-the-burden-of-proof/). [↩](#fnref12)- Of course, this isn't to say the estimates are *completely independent* of intuitions - intuitions are likely to color our choices of estimates for many of the difficult-to-estimate figures. But the ability to scrutinize and debate each estimate separately is helpful here. [↩](#fnref13)


More on “multiple world-size economies per atom”

In [This Can’t Go On](https://www.cold-takes.com/this-cant-go-on/), I [argued](https://www.cold-takes.com/this-cant-go-on/#why-cant-this-go-on) that 8200 more years of today’s growth rate would require us to sustain “multiple economies as big as today's entire world economy *per atom*.”

Feedback on this bit was split between “That is so obviously impossible, 8200 years of 2% growth is an absurd idea - growth will have to slow much before then” and “Why is that impossible? With ever-increasing creativity, we could increase quality of life higher and higher, without needing to keep using more and more material resources.”

Here I’m going to respond to the latter point, which means expanding on why 8200 years of 2% growth doesn’t look like a reasonable thing to expect. I’m going to make lots of extremely wild assumptions and talk about all kinds of weird possibilities just so that I cover even far-fetched ways for 2% growth to continue.

If you are already on team “Yeah, I don’t see the world economy growing that much,” you should skip this post unless you'd enjoy seeing the case made in a fair amount of detail.

How we COULD support “multiple world-size economies per atom”
-------------------------------------------------------------

I do think it’s *conceivable* that we could support multiple world-size economies per atom. Here’s one way:

Say that we discover some new activity, or experience, or drug, that people really, really, REALLY value.

Specifically, the market values it at 10^85 of today’s US dollars (that’s ten trillion trillion trillion trillion trillion trillion trillion dollars). That means it's valued about 10^71 times as much as everything the world produces in a year right now (combined).[1](https://www.cold-takes.com/p/864b10db-9ba3-4e6e-add3-5cad875af1b0#fn1)

Then, one person having this experience[2](https://www.cold-takes.com/p/864b10db-9ba3-4e6e-add3-5cad875af1b0#fn2) would mean the size of the economy is at least $10^85. And that would, indeed, be the equivalent of multiple of today’s world economies per atom.[3](https://www.cold-takes.com/p/864b10db-9ba3-4e6e-add3-5cad875af1b0#fn3)

To be clear, it’s not that we would’ve crammed multiple of today’s world economies into each atom. It’s that we would’ve crammed something 10^71 times as valuable as today’s world economy into a mere [10^28 atoms](https://science.howstuffworks.com/atoms-in-person.htm#:~:text=A%20human%20body%20weighing%20154,7%20followed%20by%2027%20zeros.) that make up a human being.

What would it mean, though, to value a single experience 10^71 times as much as today’s entire world economy?

One way of thinking about it might be:

* “A 1 in 10^71 chance of this thing being experienced would be as valuable as all of today’s world economy.”

* Or to make it a bit easier to intuit (while needing to oversimplify), “If I were risk-neutral, I’d be thrilled to accept a gamble where I would die immediately, with near certainty, in exchange for a 1 in 10^71 chance of getting to have this experience.”[4](https://www.cold-takes.com/p/864b10db-9ba3-4e6e-add3-5cad875af1b0#fn4)* How near-certain would death be? Well, for starters, if all the people who have ever lived to date accepted this gamble, it would be approximately certain that they would *all* lose and end up with immediate death.[5](https://www.cold-takes.com/p/864b10db-9ba3-4e6e-add3-5cad875af1b0#fn5)* But this really isn’t coming anywhere close to communicating how bad the odds would be for this gamble. It’s more like: if there were one person for each atom in the galaxy, and each of them took the gamble, they'd probably still **all** lose.[6](https://www.cold-takes.com/p/864b10db-9ba3-4e6e-add3-5cad875af1b0#fn6)* So to personally take a gamble with those kinds of odds … the experience had better be REALLY good to compensate.
	+ We’re not talking about “the best experience you’ve ever had” level here - it wouldn’t be sensible to value that more than an entire life, and the idea that it’s worth as much as today’s world economy seems pretty clearly wrong.

	+ We’re talking about something just unfathomably beyond anything any human has ever experienced.

Blowing out the numbers more
----------------------------

Imagine the single best second of your life, the kind of thing evoked by [Letter from Utopia](https://www.nickbostrom.com/utopia.html):

> Have you ever experienced a moment of bliss? On the rapids of inspiration maybe, your mind tracing the shapes of truth and beauty? Or in the pulsing ecstasy of love? Or in a glorious triumph achieved with true friends? Or in a conversation on a vine-overhung terrace one star-appointed night? Or perhaps a melody smuggled itself into your heart, charming it and setting it alight with kaleidoscopic emotions? Or when you prayed, and felt heard?
>

If you have experienced such a moment – experienced *the best type* of such a moment – then you may have discovered inside it a certain idle but sincere thought: “Heaven, yes! I didn’t realize it could be like this. This is so right, on whole different level of right; so real, on a whole different level of real. Why can’t it be like this always? Before I was sleeping; now I am awake.”

Yet a little later, scarcely an hour gone by, and the ever-falling soot of ordinary life is already covering the whole thing. The silver and gold of exuberance lose their shine, and the marble becomes dirty.

Now imagine, implausibly, that this single second was worth as much as the entire world economy outputs in a year today. (It doesn’t seem possible that it could be worth more, since the world economy that year *included* that second of your life, plus the rest of your year and many other people’s years.)

And now imagine a *full year* in which *every second* is as good as *that second.* We’ll call this the “perfect year.” According to the assumptions above, the perfect year would be no more than about 3\*10^8 times as valuable as the world economy (there are about 3\*10^8 seconds in a year).

And now imagine that *every atom in the galaxy* could be a person having the perfect year. This would now be about 10^70 \* (3 \* 10^8) = 3\*10^78 as much value as today’s world economy. **2% growth would get us there in 9150 years.**

(A crucial and perhaps counterintuitive assumption I'm making here, throughout, is that "2% growth" means "2% *really real* growth" - that whatever is valuable, holistically speaking, about annual world output today, we'll get 2% more of it each year. I think this is already the kind of assumption many people are making when they say we don't need more material to have ever-increasing wealth. If you think the 2% growth of the recent past is more "fake" than this and that it will continue in a "fake" way, that would be a debate for another time.)

And 1200 years after *that*, if each year still had 2% growth, the economy would be another ~20 billion times bigger. So now, for every atom in the galaxy, there’d have to be someone whose year was in some sense ~20 billion times *better* (or "more valuable") than the perfect year.

We’re still only talking about ~10,000 years of 2% growth.

New life forms
--------------

It’s still conceivable! Who knows what the future will bring.

But at this point it’s very intuitive to me that we are not talking about anything that looks like “Humans in human bodies having human kinds of fun and fulfillment.” An economy of this value seems to require fundamentally re-engineering something about the human experience - finding some way of arranging matter that creates far more happiness, or fulfillment, or something, that we would value so astronomically more than even the heights of human experience today.

And I think the most natural way for that to happen is something like: “Discovering fundamental principles behind what we value, and fundamental principles of how to arrange matter to get the most of it.” Which in turn suggests something more like “Once we have that level of understanding, we start to arrange the matter in the galaxy optimally, and quickly get close to the limits of what’s possible” than like “We grow at 2%, every year, for continuing thousands of years, even as (as would happen with e.g. [digital people](https://www.cold-takes.com/how-digital-people-could-change-the-world/)) we become beings who can do as much in a year as humans could do in hundreds or thousands of years.”

But it could still happen?
--------------------------

I guess? This was never meant to be a mathematical proof of the impossibility of 2%/year growth. It’s possible in theory.

But at this point, seeing what a funky and fundamentally transformed galaxy it would require within 10,000 years, what is the *affirmative* reason to expect 2%/year growth for that long a period of time? Is it that “This is the trendline, and by default I expect the trendline to continue?”

But that trendline is only a couple hundred years old - why expect it to continue for another 10,000?

Why not, instead, expect the *[longer-term pattern of accelerating economic growth](https://www.cold-takes.com/this-cant-go-on/#explosion-and-collapse)* to be what continues, until we approach some sort of fundamental limit on how much value we can cram into a given amount of matter? Or expect growth to fall gradually from here and never reach today's level again?

The last couple of centuries have been a wild ride, with wealth and living conditions improving at a historically high rate. But I don’t think that gives us reason to think that this trend goes to infinity. I believe the limits are somewhere, and it looks like sometime in the next 10,000 years, we’re either going to have to approach those limits, or [stagnate or collapse](https://www.cold-takes.com/this-cant-go-on/#why-cant-this-go-on).

Hopefully I’ve given a sense for why it seems so unlikely that there will be 10,000 more years in the future that each have 2% or greater growth. Which would imply that *each* of the last 100+ years will turn out to be one of the fastest-growing 10,000 years of all time.

If you'd like to comment on this post, [this](https://forum.effectivealtruism.org/posts/pFHN3nnN9WbfvWKFg/this-can-t-go-on?commentId=K3ESWKD3Kk5ywN7MP) would be a good place to do so.

---

Footnotes
---------

1. Today’s economy is a bit less than $10^14 per year ([source](https://en.wikipedia.org/wiki/Gross_world_product)). $10^85 = $10^14 \* 10^71. [↩](#fnref1)- (And paying full price for it, in a way that gets recorded by GDP statistics, which could get a bit hairy.) [↩](#fnref2)- See [previous estimate](https://www.cold-takes.com/this-cant-go-on/#fn7) of 10^70 atoms in the galaxy. [↩](#fnref3)- This assumes that one values one’s own life not much more than a year of the world economy’s output. I do not expect that I will see enough disagreement on this point to want to write another post on the matter, but it’s possible.

It is also making an iffy assumption about "risk-neutrality." In reality, one might personally value this experience much less than 10^71 times as much as one's own life, while still paying resources for it that would be sufficient to save an extraordinarily large number of *other* people's lives. It's hard to convey the same kind of magnitudes by appealing to impartiality, so I went with this intuition pump anyway; I think it does give the right basic sense of how mind-bogglingly large the value of this experience would be. [↩](#fnref4)- The calculation here would be: if there are 10^10 people alive today (this is "rounding up" from ~8 billion to 10 billion), and each has a 10^-71 (1 in 10^71) chance of winning the gamble, then each has a (1-10^-71) chance of losing the gamble. So the probability that they **all** lose the gamble is (1-10^-71)^(10^10), which is almost exactly 100%. [↩](#fnref5)- Similar calculation to the previous footnote, but with a population of 10^70 (one for each [atom in the galaxy](https://www.cold-takes.com/this-cant-go-on/#fn7)), so the probability that they all lose the gamble is (1-10^-71)^(10^70), which I think is around 90% (Excel can't actually handle numbers this big but this is what similar calculations imply). [↩](#fnref6)- (Footnote deleted) [↩](#fnref7)
For email filter: florpschmop


The gloves are off, the pants are on

*A lie gets halfway around the world before the truth has a chance to get its pants on. - [Winston Churchill](https://www.brainyquote.com/quotes/winston_churchill_103564)*

*(...[Not really, though](https://professorbuzzkill.com/twain-lie-travels/))*

I collect links that are good examples of "how much BS is floating around out there unchecked." Though in many cases I haven't run everything to ground, and it could be the debunking that's BS, or both the original and the debunking ... so I do want the lesson to be "Don't trust things you've heard," as opposed to "Trust this debunking." (I will probably make a future post devoted to debunking debunkings.)

[Nice piece debunking various memes that are supposedly based on studies](http://danluu.com/dunning-kruger/): "the less someone knows about a subject, the more they think they know" (not actually Dunning-Kruger at all), "money doesn’t make people happy" (it seems like it does when making some basic adjustments to the data - I think this point is well known by now), "people bounce back from setbacks (as well as positive events) and return to a fixed level of happiness" (guess not) and "type systems help in programming" (don't know what this one is about).

You may have come across most of these, but here in one place are debunkings (of varying convincingness) of pretty much all of the famous old social psychology experiments that blew my mind when I was in my 20s:

* [Disappointing replication of the "marshmallow" experiment](http://journals.sagepub.com/doi/abs/10.1177/0956797618761661?journalCode=pssa&).

* One person argues that [the Prison Experiment was a case of subjects behaving as their experimenters clearly wanted them to](https://www.psychologytoday.com/us/blog/freedom-learn/201310/why-zimbardo-s-prison-experiment-isn-t-in-my-textbook).

* The Robbers Cave experiment [backstory sounds particularly dicey](https://www.theguardian.com/science/2018/apr/16/a-real-life-lord-of-the-flies-the-troubling-legacy-of-the-robbers-cave-experiment): the experimenter tried to get two bands of boys to fight each other ala Lord of the Flies, failed miserably, tried again, succeeded, never mentioned the first time, and gained fame (despite even the 2nd try having the same issue as the Prison Experiment mentioned above).

* Finally, someone reports that many of the Milgram participants' own reports of why they had administered electric shocks provides more support for "they didn't think the person was really being harmed" than for Milgram's theory ([link](https://digest.bps.org.uk/2017/12/12/interviews-with-milgram-participants-provide-little-support-for-the-contemporary-theory-of-engaged-followership/)). I think this is the least compelling of critiques listed here, as people will rationalize their behavior with all kinds of stuff.

 Via an old [Marginal Revolution post](https://marginalrevolution.com/marginalrevolution/2018/07/happiness-results-robust.html), here's a [study claiming that none of the famous findings about happiness are robust](https://www.nber.org/system/files/working_papers/w19950/w19950.pdf). As far as I can tell, the central claim is basically this general idea: say that Group A consists of two people who each rate their happiness 6/10. And Group B consists of one person rating their happiness 4/10, and another 7/10. In some fundamental sense, we don't know which group has higher 'average' happiness, because for all we know, each increment on the 1-10 scale could represent an extra '10 units' of happiness or an extra '10 times as many units' of happiness, or something else. Now, sometimes we might be able to know which group has higher average happiness despite this issue (for example, a two 7's and a 6 vs. two 8's and a 7), but the authors here argue that the famous happiness findings are not robust in this way. Which I think makes sense, though I hope to read this more closely later on.

One of the studies I've found most mind-blowing (with video evidence!) was [this seeming demonstration that chimpanzees have better working memory than humans](https://twitter.com/BrianRoemmele/status/1213860120058220546) (at least for a particular, surprising task). But oops, [here is a very believable-sounding debunking](https://twitter.com/KrisSabbi/status/1214260415862956042) claiming that chimpanzees were trained extensively on the task, and similarly trained humans keep up just fine.

Finally, here's a 2008 [study](https://jamanetwork.com/journals/jamapsychiatry/fullarticle/210112) questioning the connection between exercise and certain mental health benefits in a convincing-seeming way. "Regular exercise is associated with reduced anxious and depressive symptoms in the population at large, but the association is not because of causal effects of exercise." The usual result is that exercise does cause such benefits, but this one looked at twin pairs and over-time changes: "Cross-sectional and longitudinal associations were small and were best explained by common genetic factors with opposite effects on exercise behavior and symptoms of anxiety and depression. In genetically identical twin pairs, the twin who exercised more did not display fewer anxious and depressive symptoms than the co-twin who exercised less. Longitudinal analyses showed that increases in exercise participation did not predict decreases in anxious and depressive symptoms."


Are we "trending toward" transformative AI? (How would we know?)

*Audio also available by searching Stitcher, Spotify, Google Podcasts, etc. for "Cold Takes Audio"*

[Today’s world

Transformative AI

Digital people
World of

Misaligned AI
World run by
Something else

or
or

Stable, galaxy-wide
civilization](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/)

> This is one of 4 posts summarizing hundreds of pages of technical reports focused almost entirely on forecasting one number: the year by which transformative AI will be developed.[1](#fn1)
>
>
>
>
> By "transformative AI," I mean "AI powerful enough to bring us into a new, qualitatively different future." I specifically focus on what I'm calling [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/): AI systems that can essentially automate all of the human activities needed to speed up scientific and technological advancement.
>
>
>
>
> The sooner PASTA might be developed, the sooner the world could change [radically](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#impacts-of-pasta), and the more important it seems to be thinking today about how to make that change go well vs. poorly.
>
>

In this post and the next, I will talk about the forecasting methods underlying my current view: I believe there's **more than a 10% chance we'll see something [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/)-like enough to qualify as "transformative AI" within 15 years (by 2036); a ~50% chance we'll see it within 40 years (by 2060); and a ~2/3 chance we'll see it this century (by 2100).**

Below, I will:

* Discuss [what kind of forecast I'm going for](#what-kind-of-forecast-am-i-going-for).
	+ I'm not sure whether it will feel as though transformative AI is "on the way" long before it arrives. I'm hoping, instead, that we can use trends in key underlying facts about the world (such as AI capabilities, model size, etc.) to forecast a qualitatively unfamiliar future.

	+ An analogy for this sort of forecasting would be something like: "This water isn't bubbling, and there are no signs of bubbling, but the temperature has gone from 70° Fahrenheit[2](#fn2) to 150°, and if it hits 212°, the water will bubble." Or: "It's like forecasting school closures and overbooked hospitals, when there aren't any yet, based on trends in reported infections."* Discuss whether we can look for [trends in how "impressive" or "capable" AI systems are](#subjective-extrapolations-and-). I think this approach is unreliable: (a) AI progress may not "trend" in the way we expect; (b) in my experience, different AI researchers have radically different intuitions about which systems are impressive or capable, and how progress is going.

* Briefly discuss [Grace et al 2017](https://arxiv.org/pdf/1705.08807.pdf), the best existing survey of AI researchers on transformative AI timelines. Its conclusions broadly seem in line with my own forecasts, though there are signs the researchers weren't thinking very hard about the questions.

The next piece in this series will focus on [Ajeya Cotra's "Forecasting Transformative AI with Biological Anchors](https://www.lesswrong.com/posts/KrJfoZzpSDpnrv9va/draft-report-on-ai-timelines)" (which I'll abbreviate below as "Bio Anchors"), the forecast I find most informative for transformative AI.

What kind of forecast am I going for?
-------------------------------------

There are a couple of ways in which forecasting transformative AI is different from the kind of forecasting we might be used to.

First, I'm forecasting over very long time horizons (decades), unlike e.g. a weather forecast (days) or an election forecast (months). This makes the task quite a bit harder,[3](#fn3) and harder for outsiders to evaluate since I don't have a clearly relevant [track record](https://www.cold-takes.com/prediction-track-records-i-know-of/) of making forecasts on similar topics.

Second, I lack rich, clearly relevant data sources, and I can't look back through a bunch of similar forecasts from the past. FiveThirtyEight's [election](https://projects.fivethirtyeight.com/2020-election-forecast/) forecasts look at hundreds of polls, and they have a model of how well polls have predicted elections in the past. Forecasting transformative AI needs to rely more on intuition, guesswork and judgment, in terms of determining what data is most relevant and how it's relevant.

Finally, I'm trying to forecast a **qualitatively unfamiliar future**. Transformative AI - and the strange future it comes with - doesn't *feel* like something we're "trending toward" year to year.

* If I were trying to forecast when the world population would hit 10 billion, I could simply extrapolate [existing trends](https://ourworldindata.org/world-population-growth#future-population-growth) of world population. World population itself is known to be growing and can be directly estimated. In my view, extrapolating out a long-running trend is one of the better ways to make a forecast.

* When FiveThirtyEight makes election forecasts, there's a background understanding that there's going to be an election on a certain date, and whoever wins will take office on another date. We all buy into that basic framework, and there's a general understanding that better polling means a better chance of winning.

* By contrast, transformative AI - and the strange future it comes with - isn't something we're "headed for" in any clearly measurable way. There's no clear metric like "transformativeness of AI" or "weirdness of the world" that's going up regularly every year such that we can project it out into the future and get the date that something like [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/) will be developed.

Perhaps for some, these points gives enough reason to ignore the whole possibility of transformative AI, or assume it's very far away. But I don't think this is a good idea, for a couple of reasons.

First, I have a background view that something like [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/) is in a sense "inevitable," assuming continued advances in society and computing. The basic intuition here - which I could expand on if there's [interest](https://www.guidedtrack.com/programs/4kal2ue/run?posttitle=Are%20we%20%22trending%20toward%22%20transformative%20AI%3F%20(How%20would%20we%20know%3F)) - is that human brains are numerous and don't seem to need particular rare materials to produce, so it should be possible at some point to synthetically replicate the key parts of their functionality.[4](#fn4)

At the same time, I'm not confident that PASTA will feel qualitatively as though it's "on the way" well before it arrives. (More on this [below](#subjective-extrapolations-and-).) So I'm inclined to look for ways to estimate when we can expect this development, despite the challenges, and despite the fact that it doesn't feel today as though it's around the corner.

I think there are plenty of example cases where a **qualitatively unfamiliar future could be seen in advance by plotting the trend in some underlying, related facts about the world.** A few that come to mind:

* When COVID-19 first emerged, a lot of people had trouble taking it seriously because it didn't feel as though we were "trending toward" or "headed for" a world full of overflowing hospitals, office and school closures, etc. At the time (say, January 2020), there were a relatively small number of cases, an even smaller number of deaths, and no qualitative sense of a global emergency. The only thing alarming about COVID-19, at first, was that case counts were growing at a fast exponential rate (though the overall number of cases was still small). But it was possible to extrapolate from the fast growth in case counts to a risk of a global emergency, and [some people did](https://80000hours.org/podcast/episodes/howie-rob-coronavirus-february-3rd/). (And [some didn't](https://i.insider.com/5e59596efee23d0fb873eb46?width=750&format=jpeg&auto=webp).)

* Climatologists forecast a global rise in temperatures that's significantly more than what we've seen over the past few decades, and could have major consequences far beyond what we're seeing today. They do this by forecasting trends in greenhouse gas emissions and extrapolating *from there* to temperature and consequences. If you simply tried to ask "How fast is the temperature rising?" or "Are hurricanes getting worse?", and based all your forecasts of the future on those, you probably wouldn't be forecasting the same kinds of extreme events around 2100.[5](#fn5)* To give a more long-run example, we can project a date by which the sun will burn out, and conclude that the world will look very different by that date than it does now, even though there's no trend of things getting colder or darker today.

![](https://www.cold-takes.com/content/images/2021/08/covid-vs-workplace-closures-3.png)COVID-19 cases from [WHO](https://portal.who.int/report/eios-covid19-counts/#display=Global&nrow=1&ncol=1&arr=row&pg=1&labels=view_who_regions,view_continents&sort=global_code;asc&filter=&sidebar=-1&fv=). Workplace closures are from [this OWiD data](https://ourworldindata.org/grapher/workplace-closures-covid), simply scored as 1 for "recommended," 2 for "required for some," 3 for "required for all but key workers" and summed across all countries.

An analogy for this sort of forecasting would be something like: "This water isn't bubbling, and there are no signs of bubbling, but the temperature has gone from 70° Fahrenheit[6](#fn6) to 150°, and if it hits 212°, the water will bubble."

Ideally, I can find some underlying factors that are changing regularly enough for us to predict them (such as growth in the [size and cost of AI models](https://openai.com/blog/ai-and-compute/)), and then argue that if those factors reach a certain point, the odds of transformative AI will be high.

You can think of this approach as answering the question: "If I think something like PASTA is inevitable, and I'm trying to guess the timing of it using a few different analysis methods, what do I guess?" We can separately ask "And is there reason that this guess is implausible, untrustworthy, or too 'wild?'" - this was addressed in the [previous piece in this series](https://www.cold-takes.com/forecasting-transformative-ai-whats-the-burden-of-proof/).

Subjective extrapolations and "AI impressiveness"
-------------------------------------------------

*For a different presentation of some similar content, see [this section](https://docs.google.com/document/d/1cCJjzZaJ7ATbq8N2fvhmsDOUWdm7t3uSSXv6bD0E_GM/edit#heading=h.njuz93bimqty) of [Bio Anchors](https://drive.google.com/drive/u/1/folders/15ArhEPZSTYU8f012bs6ehPS6-xmhtBPP).*

If we're looking for some underlying factors in the world that predict when transformative AI is coming, perhaps the first thing we should look for is trends in how "impressive" or "capable" AI systems are.

The easiest version of this would be if the world happened to shake out such that:

* One day, for the first time, an AI system managed to get a passing grade on a 4th-grade science exam.

* Then we saw the first AI passing (and then acing) a 5th grade exam, then 6th grade exam, etc.

* Then we saw the first AI earning a PhD, then the first AI writing a published paper, etc. all the way up to the first AI that could do Nobel-Prize-worthy science work.

* This all was spread out regularly over the decades, so we could clearly see the state of the art advancing from 4th grade to 5th grade to 6th grade, all the way up to "postdoc" and beyond. And all of this happened slowly and regularly enough that we could start putting a date on "full-blown scientist AI" several decades in advance.

It would be very convenient - I almost want to say "polite" - of AI systems to advance in this manner. It would also be "polite" if AI advanced in the way that some people seem to casually imagine it will: first taking over jobs like "truck driver" and "assembly line worker," then jobs like "teacher" and "IT support," and then jobs like "doctor" and "lawyer," before progressing to "scientist."

Either of these would give us plenty of lead time and a solid basis to project when science-automating AI is coming. Unfortunately, I don't think we can count on such a thing.

* AI seems to progress very differently from humans. For example, there were superhuman AI chess players[7](#fn7) long before there was AI that could reliably tell apart pictures of dogs and cats.[8](#fn8)* One possibility is that AI systems will be capable of the hardest intellectual tasks insects can do, then of the hardest tasks mice and other small mammals can do, then monkeys, then humans - effectively matching the abilities of larger and larger brains. If this happened, we wouldn't necessarily see many signs of AI being able to e.g. do science until we were *very* close. Matching a 4th-grader might not happen until the very end.

* Another possibility is that AI systems will be able to do anything that a human can do within 1 second, then anything that a human can do within 10 seconds, etc. This could also be quite a confusing progression that makes it non-obvious how to forecast progress.

Actually, if we didn't already know how humans tend to mature, we might find a child's progress to be pretty confusing and hard to extrapolate. **Watching someone progress from birth to age 8 wouldn't necessarily give you any idea that they were, say, 1/3 of the way to being able to start a business, make an important original scientific discovery, etc.** (Even *knowing* the usual course of human development, it's hard to tell from observing an 8-year-old what professional-level capabilities they could/will end up with in adulthood.)

Overall, it's quite unclear how we should think about the spectrum from "not impressive/capable" to "very impressive/capable" for AI. And indeed, in my experience, different AI researchers have radically different intuitions about which systems are impressive or capable, and how progress is going. I've often had the experience of seeing one AI researcher friend point to some new result and say "This is huge, how can anyone not see how close we're getting to powerful AI?" while another says "This is a minor advance with little significance."[9](#fn9)

It would be great if we could forecast the year transformative AI will be developed, by using a chart like this (from [Bio Anchors](https://drive.google.com/drive/u/1/folders/15ArhEPZSTYU8f012bs6ehPS6-xmhtBPP); "TAI" means "transformative AI"):

![Hypothetical chart showing a projection of 'AI impressiveness' that crosses a line for transformative AI.](https://www.cold-takes.com/content/images/size/w1000/2021/08/impressiveness-extrap-chart.png)

But as far as I can tell, there's no way to define the y-axis that wouldn't be fiercely debated between experts.

Surveying experts
-----------------

One way to deal with this uncertainty and confusion would be to survey a large number of experts and simply ask them when they expect transformative AI to be developed. We might hope that each of the experts (or at least, many of them) is doing their own version of the "impressiveness extrapolation" above - or if not, that they're doing something else that can help them get a reasonable estimate. By averaging many estimates, we might get an aggregate that reflects the "wisdom of crowds."[10](#fn10)

I think the best version of this exercise is [Grace et al 2017](https://arxiv.org/pdf/1705.08807.pdf), a survey of 352 AI researchers that included a question about “when unaided machines can accomplish every task better and more cheaply than human workers" (which would presumably include tasks that advance scientific and technological development, and hence would qualify as [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/)). The two big takeaways from this survey, according to [Bio Anchors](https://drive.google.com/drive/u/1/folders/15ArhEPZSTYU8f012bs6ehPS6-xmhtBPP) and me, are:

* **A ~20% probability of this sort of AI by 2036; a ~50% probability by 2060; a ~70% probability by 2100. These match the figures I give in the introduction.*** Much later estimates for slightly differently phrased questions (posed to a smaller subset of respondents), implying (to me) that the researchers simply weren't thinking very hard about the questions.[11](#fn11)

My bottom line: this evidence is consistent with my current probabilities, though potentially not very informative. The next piece in this series will be entirely focused on [Ajeya Cotra's "Forecasting Transformative AI with Biological Anchors,"](https://www.lesswrong.com/posts/KrJfoZzpSDpnrv9va/draft-report-on-ai-timelines) the forecasting method I find most informative here.

**Next in series:** [Forecasting transformative AI: the "biological anchors" method in a nutshell](https://www.cold-takes.com/forecasting-transformative-ai-the-biological-anchors-method-in-a-nutshell/)

---

Footnotes
---------

1. Of course, the answer could be "A kajillion years from now" or "Never." [↩](#fnref1)- Centigrade equivalents for this sentence: 21°, 66°, 100° [↩](#fnref2)- Some notes on longer-term forecasting [here](https://www.openphilanthropy.org/blog/how-feasible-long-range-forecasting#Tetlock_long-range_forecasting_and_questions_of_relevance). [↩](#fnref3)- See also [this piece](https://www.lesswrong.com/posts/HhWhaSzQr6xmBki8F/birds-brains-planes-and-ai-against-appeals-to-the-complexity) for a bit of a more fleshed out argument along these lines, which I don't agree with fully as stated (I don't think it presents a strong case for transformative AI soon), but which I think gives a good sense of my intuitions about in-principle feasibility. Also see [On the Impossibility of Supersized Machines](https://arxiv.org/abs/1703.10987) for some implicit (joking) responses to many common arguments for why transformative AI might be impossible to create.  [↩](#fnref4)- For example, see the temperature chart [here](https://www.climate.gov/news-features/understanding-climate/climate-change-global-temperature-projections#:~:text=Results%20from%20a%20wide%20range,gases%20that%20human%20activities%20produce.) - the lowest line seems like it would be a reasonable projection, if temperature were the only thing you were looking at. [↩](#fnref5)- Centigrade equivalents for this sentence: 21°, 66°, 100° [↩](#fnref6)- [1997](https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer)#Deep_Blue_versus_Kasparov). [↩](#fnref7)- The Kaggle "dogs vs. cats" challenge was [created in 2013](https://www.kaggle.com/c/dogs-vs-cats/leaderboard). [↩](#fnref8)- From [Bio Anchors](https://drive.google.com/drive/u/1/folders/15ArhEPZSTYU8f012bs6ehPS6-xmhtBPP): "We have heard ML experts with relatively short timelines argue that AI systems today can essentially see as well as humans, understand written information, and beat humans at almost all strategy games, and the set of things they can do is expanding rapidly, leading them to expect that transformative AI would be attainable in the next decade or two by training larger models on a broader distribution of ML problems that are more targeted at generating economic value. Conversely, we have heard ML experts with relatively long timelines argue that ML systems require much more data to learn than humans do, are unable to transfer what they learn in one context to a slightly different context, and don’t seem capable of much structured logical and causal reasoning; this leads them to believe we would need to make multiple major breakthroughs to develop TAI. At least one Open Philanthropy technical advisor has advanced each of these perspectives." [↩](#fnref9)- [Wikipedia](https://en.wikipedia.org/wiki/Wisdom_of_the_crowd): "The classic wisdom-of-the-crowds finding ... At a 1906 country fair in Plymouth, 800 people participated in a contest to estimate the weight of a slaughtered and dressed ox. Statistician Francis Galton observed that the median guess, 1207 pounds, was accurate within 1% of the true weight of 1198 pounds." [↩](#fnref10)- [Bio Anchors](https://drive.google.com/drive/u/1/folders/15ArhEPZSTYU8f012bs6ehPS6-xmhtBPP):

	* *Some researchers were asked to forecast “HLMI” as defined above [high-level machine intelligence, which I would take to include something like PASTA], while a randomly-selected subset was instead asked to forecast “full automation of labor”, the time when “all occupations are fully automatable.” Despite the fact that achieving HLMI seems like it should quickly lead to full automation of labor, the median estimate for full automation of labor was ~2138 while the median estimate for HLMI was ~2061, almost 80 years earlier.* * *Random subsets of respondents were asked to forecast when individual milestones (e.g. laundry folding, human-level StarCraft, or human-level math research) would be achieved. The median year by which respondents expected machines to be able to automate AI research was ~2104, while the median estimate for HLMI was ~2061 -- another clear inconsistency because “AI research” is a task done by human workers.* [↩](#fnref11)


Cold Links: heartwarming sports stuff

Apparently Marcus Smart is [really good at "taking charges,"](https://www.espn.com/blog/boston/celtics/post/_/id/4725417/marcus-smart-taking-charge-for-boston-celtics) which means positioning himself so that an offensive player will run into him and foul him. This requires getting into the right position and then specifically not moving for a second (the latter is necessary to trigger the ref's classification as an offensive foul). It's very funny to watch him do this (he's #36, the guy who takes a crotch to the face):

![](https://www.cold-takes.com/content/images/2021/08/smart-takes-charge.gif)

Marcus Smart has become my new central example of someone who enthusiastically takes one for the team.

A friend summarizes something he saw on ESPN (see also the [Wikipedia entry](https://en.wikipedia.org/wiki/1994_John_Tyler_vs._Plano_East_high_school_football_game), and [video](https://www.youtube.com/watch?v=ZHkABO0VwCg&ab_channel=jodyvancevids)):

> this is an amazing sports story i never knew about. Texas HS football. It was on ESPN b/c it's the 20th anniversary.
>

 "... giving the Lions a seemingly insurmountable 41–17 lead with only 3:03 remaining.

 However, on a two-play 70-yard drive, the Panthers scored a touchdown to bring the score to 41–23 (after a failed two-point conversion) with 2:36 on the clock. The Panthers then successfully executed three onside kicks in a row, recovering the ball each time and then driving down the field for a touchdown on each occasion.

 ... giving the Panthers a 44–41 comeback lead with only 24 seconds remaining.

 In a final twist, however, after the Panthers did a regular kickoff, the Lions' returner Roderick Dunn caught the ball at his own three-yard line and took it 97 yards for a touchdown at 0:11 and a 48–44 Lions victory.

 He was the very same player who had muffed the reception of the final two onside kicks.'

 ... interviews with the players from today [were on ESPN, not the Wikipedia page]:

 -- the guys from the team that lost were still crying about it

 -- the guy that ran back the kick said it was one of the greatest moments in his life and he still thinks about when he's down. the lesson, he says, is 'never give up.'

I wasn't able to easily verify a lot of this, but here is a [very short, sweet story about Nav Bhatia](https://www.mansworldindia.com/sports/story-of-nav-bhatia-nba-hall-of-fame/), perhaps the first person inducted into the NBA Hall of Fame and given a championship ring for ... being a really dedicated fan? Apparently he hasn't missed a Toronto Raptors home game in 25 years.

Watch this kid's [reaction to getting a spare racket from tennis legend Novak Djokovich](https://i.imgur.com/GSt0HWL.gifv). Sports!

[Super Nice Soccer Guy Rewarded For His Compassion With Easiest Goal Of His Life](https://deadspin.com/super-nice-soccer-guy-rewarded-for-his-compassion-with-1830382771).

[Very funny article on now-retired former star NFL quarterback Andrew Luck](http://deadspin.com/andrew-luck-is-an-affable-weirdo-1671351814): apparently he sincerely congratulated people who tackle him hard, and this was completely unique for a quarterback and was seen as extremely unnerving by the defenders. Doesn't mean he enjoyed the tackles though - he [retired at 29](https://en.wikipedia.org/wiki/Andrew_Luck#Retirement).

The article completely delivers on the headline: [36-Year-Old Accountant Called In As Emergency NHL Goalie — And He Crushed It](https://www.npr.org/sections/thetwo-way/2018/03/30/598263399/36-year-old-accountant-called-in-as-emergency-nhl-goalie-and-he-crushed-it?utm_source=facebook.com&utm_medium=social&utm_campaign=npr&utm_term=nprnews&utm_content=20180330).

That [story about two athletes (friends) who shared the gold medal](https://www.aljazeera.com/sports/2021/8/2/best-moment-of-tokyo-2020-world-reacts-to-shared-olympic-gold) is too recent and too popular for Cold Links, so maybe I'll link to it in a year or two. I did manage to find someone [complaining about it](https://www.foxsports.com.au/tokyo-olympics-2021/tokyo-olympics-2021-high-jump-final-shared-gold-medal-mutaz-essa-barshim-and-gianmarco-tamberi-win-explained-rule/news-story/e5917258d01197da640822c55b2f420a).

Readers sent in cool new links on "intensity" in sports, which I'll put out another time (only heartwarming stuff allowed in this one).