The future of growth: near-zero growth rates
============================================

26 July 2017
by [Center on Long-Term Risk](https://longtermrisk.org/author/eas_super/ "Posts by Center on Long-Term Risk")

**First written:** Jul. 2017; **Last update:** Aug. 2017

 Exponential growth is a common pattern found throughout nature. Yet it is also a pattern that tends not to last, as growth rates tend to decline sooner or later.

![](https://longtermrisk.org/files/S-curve.png)

In biology, this pattern of exponential growth that wanes off is found in everything from the development of individual bodies — for instance, in the [growth of humans](https://en.wikipedia.org/wiki/Human_development_(biology)), which levels off in the late teenage years — to population sizes.

One may of course be skeptical that this general trend will also apply to the growth of our technology and economy at large, as innovation seems to continually postpone our clash with the ceiling, yet it seems inescapable that it must. For in light of what we know about physics, we can conclude that exponential growth of the kinds we see today, in technology in particular and in our economy more generally, must come to an end, and do so relatively soon.

**Limits to growth**
--------------------

### **Physical limits to computation and Moore’s law**

One reason we can make this assertion is that there are theoretical [limits to computation](https://en.wikipedia.org/wiki/Limits_of_computation). As physicist Seth Lloyd’s [calculations](https://arxiv.org/pdf/quant-ph/9908043.pdf) show, a continuation of Moore’s law — in its most general formulation: “the amount of information that computers are capable of processing and the rate at which they process it doubles every two years” — would imply that we hit the theoretical limits of computation within 250 years:

> If, as seems highly unlikely, it is possible to extrapolate the exponential progress of Moore's law into the future, then it will only take two hundred and fifty years to make up the forty orders of magnitude in performance between current computers that perform 1010 operations per second on 1010 bits and our one kilogram ultimate laptop that performs 1051 operations per second on 1031 bits.
>
>

Similarly, physicists Lawrence Krauss and Glenn Starkman [have calculated](https://arxiv.org/pdf/astro-ph/0404510.pdf) that, even if we factor in colonization of space at the speed of light, this doubling of processing power cannot continue for more than 600 years in any civilization:

> Our estimate for the total information processing capability of any system in our Universe implies an ultimate limit on the processing capability of any system in the future, independent of its physical manifestation and implies that Moore’s Law cannot continue unabated for more than 600 years for any technological civilization.
>
>

In a more recent [lecture](https://www.youtube.com/watch?v=8Cnj8MIQ0HY&feature=youtu.be&t=36m50s) and a subsequent [interview](https://youtu.be/qta1YaEeQpI?t=1m), Krauss said that the absolute limit for the continuation of Moore’s law, in our case, would be reached in less than 400 years (the discrepancy — between the numbers 400 and 600 — is at least in part because [Moore’s law](https://en.wikipedia.org/wiki/Moore%27s_law#/media/File:Moore%27s_Law_over_120_Years.png), in its most general formulation, has played out for more than a century in our civilization at this point). And, as both Krauss and Lloyd have stressed, these are ultimate theoretical limits, resting on assumptions that are unlikely to be met in practice, such as expansion at the speed of light. What is possible, in terms of how long Moore’s law can continue for, given both [engineering and economic constraints](https://www.nature.com/nature/journal/v512/n7513/full/nature13570.html) is likely [significantly less](https://arxiv.org/pdf/1511.05956.pdf). Indeed, we are already close to approaching the [physical limits](https://en.wikipedia.org/wiki/Moore%27s_law#Near-term_limits) of the paradigm that Moore’s law has been riding on for more than 50 years — silicon transistors, the only paradigm that Gordon Moore was talking about originally — and it is not clear whether [other paradigms](http://www.economist.com/technology-quarterly/2016-03-12/after-moores-law) will be able to take over and keep the trend going.

### **Limits to the growth of energy use**

Physicist Tom Murphy [has calculated](https://dothemath.ucsd.edu/2011/07/galactic-scale-energy/) a similar limit for the growth of the energy consumption of our civilization. Based on the observation that the energy consumption of the United States has increased fairly consistently with an average annual growth rate of 2.9 percent over the last 350 odd years (although the growth rate [appears](https://dothemath.ucsd.edu/2011/08/does-the-logistic-shoe-fit/) to have slowed down in recent times and been stably below 2.9 since c. 1980), Murphy proceeds to derive the limits for the continuation of similar energy growth. He does this, however, by assuming an annual growth rate of “only” 2.3 percent, which conveniently results in an increase of the total energy consumption by a factor of ten every 100 years. If we assume that we will continue expanding our energy use at this rate by covering Earth with solar panels, this would, on Murphy’s calculations, imply that we will have to cover all of Earth’s land with solar panels in less than 350 years, and all of Earth, including the oceans, in 400 years.

Beyond that, assuming that we could capture all of the energy from the sun by surrounding *it* in solar panels, the 2.3 percent growth rate would come to an end within 1,350 years from now. And if we go further out still, to capture the energy emitted from all the stars in our galaxy, we get that this growth rate must hit the ceiling and become near-zero within 2,500 years (of course, the limit of the physically possible must be hit earlier, indeed more than 500 years earlier, as we cannot traverse our 100,000 light year-wide Milky Way in only 2,500 years).

One may suggest that alternative sources of energy might change this analysis significantly, yet, as Murphy [notes](https://dothemath.ucsd.edu/2011/07/galactic-scale-energy/), this does not seem to be the case:

> Some readers may be bothered by the foregoing focus on solar/stellar energy. If we’re dreaming big, let’s forget the wimpy solar energy constraints and adopt fusion. The abundance of deuterium in ordinary water would allow us to have a seemingly inexhaustible source of energy right here on Earth. We won’t go into a detailed analysis of this path, because we don’t have to. The merciless growth illustrated above means that in 1400 years from now, *any* source of energy we harness would have to outshine the sun.
>
>

Essentially, keeping up the annual growth rate of 2.3 percent by harnessing energy from matter not found in stars would force us to make such matter hotter than stars themselves. We would have to create new stars of sorts, and, even if we assume that the energy required to create such stars is less than the energy gained, such an endeavor would quickly run into limits as well. For according to [one estimate](https://arxiv.org/abs/1102.4340), the total mass of the Milky Way, including dark matter, is only 20 times greater than the mass of its stars. Assuming a 5:1 ratio of dark matter to ordinary matter, this implies that that there is only about 3.3 times as much ordinary non-stellar matter as there is stellar matter in our galaxy. Thus, even if we could convert all this matter into stars without spending any energy and harvest the resulting energy, this would only give us about 50 years more of keeping up with the annual growth rate of 2.3 percent.[1](#endnote1)

### **Limits derived from economic considerations**

Similar conclusions as the ones drawn above for computation and energy also seem to follow from calculations of a more [economic nature](https://dothemath.ucsd.edu/2011/07/can-economic-growth-last/). For, as economist Robin Hanson [has argued](http://www.overcomingbias.com/2009/09/limits-to-growth.html), projecting present economic growth rates into the future also leads to a clash against fundamental limits:

> Today we have about ten billion people with an average income about twenty times subsistence level, and the world economy doubles roughly every fifteen years. If that growth rate continued for ten thousand years[,] the total growth factor would be 10200.
>
>
> There are roughly 1057 atoms in our solar system, and about 1070 atoms in our galaxy, which holds most of the mass within a million light years. So even if we had access to all the matter within a million light years, to grow by a factor of 10200, *each atom* would on average have to support an economy equivalent to 10140 people at today’s standard of living, or one person with a standard of living 10140 times higher, or some mix of these.
>
>

Indeed, current growth rates would “only” have to continue for three thousand years before each atom in our galaxy would have to support an economy equivalent to a single person living at today’s living standard, which already seems rather implausible (not least because we can only access a tiny fraction of “all the matter within a million light years” in three thousand years). Hanson does not, however, expect the current growth rate to remain constant, but instead, based on the [history of growth rates](http://mason.gmu.edu/~rhanson/longgrow.html), expects a new growth mode where the world economy doubles within [15 days rather than 15 years](http://www.futurebrief.com/robinhanson.asp):

> If a new growth transition were to be similar to the last few, in terms of the number of doublings and the increase in the growth rate, then the remarkable consistency in the previous transitions allows a remarkably precise prediction. A new growth mode should arise sometime within about the next seven industry mode doublings (i.e., the next seventy years) and give a new wealth doubling time of between seven and sixteen days.
>
>

And given this more than a hundred times greater growth rate, the net growth that would take 10,000 years to accomplish given our current growth rate (cf. Hanson’s calculation above) would now take less than a century to reach, while growth otherwise requiring 3,000 years would require less than 30 years. So if Hanson is right, and we will see such a shift within the next seventy years, what seems to follow is that we will reach the limits of economic growth, or at least reach near-zero growth rates, within a century or two. Such a projection is also consistent with the physically derived limits of the continuation of Moore’s law; not that economic growth and Moore’s law are remotely the same, yet they are no doubt closely connected: economic growth is largely powered by technological progress, of which Moore’s law has been a considerable subset in recent times.

The conclusion we reach by projecting past growth trends in computing power, energy, and the economy is the same: our current growth rates cannot go on forever. In fact, they will have to decline to near-zero levels very soon on a cosmic timescale. Given the physical limits to computation, and hence, ultimately, to economic growth, we can conclude that we must be close to the point where peak relative growth in our economy and our ability to process information occurs — that is, the point where this growth rate is the highest in the entire history of our civilization, past and future.

**“Peak growth” might lie in the past**
---------------------------------------

This is not, however, to say that this point of maximum relative growth necessarily lies in the future. Indeed, in light of the [declining economic growth rates](http://data.worldbank.org/indicator/NY.GDP.MKTP.KD.ZG) we have seen over the last few decades, it cannot be ruled out that we are now already past the point of “peak economic growth” in the history of our civilization, with the highest growth rates having occurred around 1960-1980, cf. these declining growth rates and [this essay](http://growth-dynamics.com/articles/Singularity.pdf) by physicist Theodore Modis. This is not to say that we most likely are, yet it seems that the probability that we are is non-trivial.

A relevant data point here is that the global economy has seen [three doublings](https://en.wikipedia.org/wiki/Gross_world_product) since 1965, where the annual growth rate was around six percent, and yet the annual growth rate today is only a little over half — around 3.5 percent — of, and lies stably below, what it was those three doublings ago. In the entire history of economic growth, [this seems unprecedented](http://holtz.org/Library/Social%20Science/Economics/Estimating%20World%20GDP%20by%20DeLong/Estimating%20World%20GDP.htm), suggesting that we may already be on the other side of the highest growth rates we will ever see. For up until this point, a three-time doubling of the economy has, rare fluctuations aside, led to an increase in the annual growth rate.

And this “past peak growth” hypothesis looks even stronger if we look at 1955, with a growth rate of a little less than six percent and a world product at 5,430 billion 1990 U.S dollars, which doubled four times gives just under 87,000 billion — about where we should expect today’s world product to be. Yet throughout the history of our economic development, four doublings has meant a clear increase in the annual growth rate, at least in terms of the underlying trend; not a stable decrease of almost 50 percent. To me, this suggests that maintaining more than, say, a 90 percent probability that we will see greater annual growth rates in the future is overconfident.[2](#endnote2)

### **A hypothetical model: roughly symmetric growth rates**

If we assume a model of the growth of the global economy where the annual growth rate is roughly symmetrical around the time the growth rate was at its global maximum, and then assume that this global maximum occurred around 1965, this means that we should expect the annual growth rate three doublings earlier, c. 1900, to be the same as the annual growth rate three doublings later, c. 2012. What do we [observe](https://en.wikipedia.org/wiki/Gross_world_product)? Three doublings earlier it was around 2.5 percent, while it was around 3.5 percent three doublings later, at least according to [one source](http://www.imf.org/external/pubs/ft/weo/2017/01/weodata/weorept.aspx?pr.x=72&pr.y=12&sy=2006&ey=2018&scsm=1&ssd=1&sort=country&ds=.&br=1&c=001%2C110%2C163%2C200&s=NGDP_RPCH&grp=1&a=1) (although [other sources](http://data.worldbank.org/indicator/NY.GDP.MKTP.KD.ZG) actually do put the number at around 2.5 percent). Not a clear match, nor a clear falsification.

Yet if we look at the growth rates of advanced economies around 2012, we find that the growth rate is actually significantly lower than 2.5 percent, namely 1.2-2.0 percent. And given that less developed economies [are expected](https://en.wikipedia.org/wiki/Solow%E2%80%93Swan_model#Conditional_convergence) to grow significantly faster than more developed ones, as the more advanced economies have paved the way and made high-hanging fruits more accessible, the (already not so big) 2.5 vs. 3.5 percent mismatch could be due to this gradually diminishing catch-up effect. Indeed, if we compare advanced economies today with advanced economies c. 1900, we find that the growth rate was significantly higher back then,[3](#endnote3) suggesting that the symmetrical model may in fact overestimate current and future growth if we look only at advanced economies.[4](#endnote4)

### **Could we be past peak growth in science and technology?**

That peak growth lies in the past [may also be true of technological progress](http://www.nber.org/papers/w19895.pdf) in particular, or at least many forms of technological progress, including the progress in computing power tracked by Moore’s law, where the growth rate appears to have been highest around 1990-2005, and to since have been in decline, cf. [this article](http://ieeexplore.ieee.org/document/7878938/) and the first graphs found [here](http://www.economist.com/technology-quarterly/2016-03-12/after-moores-law) and [here](http://www.softmachines.org/wordpress/?p=2097). Similarly, various sources of [data](http://www.nature.com/nrc/journal/v8/n8/fig_tab/nrc2458_F5.html) and [proxies](https://www.hindawi.com/journals/bmri/2009/823148/fig2/) tracking the number of scientific [articles](https://www.researchgate.net/figure/277675348_fig1_Figure-1-History-of-scientific-publications-on-carbon-based-materials-a-Total) published and [references cited](http://blogs.nature.com/news/files/2014/05/Cited-publications.png) over time also suggest that we could be past peak growth in science as well, at least in many fields when evaluated based on such metrics, with peak growth seeming to have been reached around 2000-2010.

Yet again, these numbers — those tracking economic, technological, and scientific progress — are of course closely connected, as growth in each of these respects contributes to, and is even part of, growth in the others. Indeed, [one study](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2909426/) found the doubling time of the total number of scientific articles in recent decades to be 15 years, corresponding to an annual growth rate of 4.7 percent, strikingly similar to the growth rate of the global economy in recent decades. Thus, declining growth rates both in our economy, technology, and science cannot be considered wholly independent sources of evidence that growth rates are now declining for good. We can by no means rule out that growth rates might increase in all these areas in the future — although, as we saw above with respect to the limits of Moore’s law and economic progress, such an increase, if it is going to happen, must be imminent if current growth rates remain relatively stable.

**Absolute and relative growth**
--------------------------------

The economic “peak growth” discussed above relates to relative growth, not absolute growth. These are worth distinguishing. For in terms of absolute growth, annual growth is significantly higher today than it was in the 1960s, where the greatest relative growth to date occurred. The global economy grew with about half a trillion 1990 US dollars each year in the sixties, whereas it grows with about two trillion now. So in this absolute sense, we are seeing significantly more growth today than we did 50 years ago, although we now have significantly lower growth rates.

If we assume the model with symmetric growth rates mentioned above and make a simple extrapolation based on it, what follows is that our time is also a special one when it comes to absolute annual growth. The picture we get is the following (based on an [estimate](http://holtz.org/Library/Social%20Science/Economics/Estimating%20World%20GDP%20by%20DeLong/Estimating%20World%20GDP.htm) of past growth rates from economic historian James DeLong):

| Year | World GDP
(in trillions) | Annual
growth rate | Absolute annual
growth (in trillions) |
| --- | --- | --- | --- |
| 920 | 0.032 | 0.13 | 0.00004 |
| 1540 | 0.065 | 0.25 | 0.0002 |
| 1750 | 0.13 | 0.5 | 0.0007 |
| 1830 | 0.27 | 1 | 0.003 |
| 1875 | 0.55 | 1.8 | 0.01 |
| 1900 | 1.1 | 2.5 | 0.03 |
| 1931 | 2.3 | 3.8 | 0.09 |
| 1952 | 4.6 | 4.9 | 0.2 |
| 1965 | 9.1 | 5.9 | 0.5 |
| 1980 | 18 | 4.4 | 0.8 |
| 1997 | 36 | 4.0 | 1.4 |
| 2012 | 72 | 3.5 | 2.1 |

Predicted values given roughly symmetric growth rates around 1965 (mirroring growth rates above):

|  |  |  |  |
| --- | --- | --- | --- |
| 2037 | 144 | 1.8 | 2.6 |
| 2082 | 288 | 1 | 2.9 |
| 2162 | 576 | 0.5 | 2.9 |
| 2372 | 1152 | 0.25 | 2.9 |
| 2992 | 2304 | 0.13 | 3.0 |

We see that the absolute annual growth in GDP seems to follow an s-curve with an inflection point right about today, as we see that the period from 1997 to 2012 saw the biggest jump in absolute annual growth in a doubling ever; an increase of 0.7 trillion, from 1.4 to 2.1.

It is worth noting that economist Robert Gordon [predicts](http://www.nber.org/papers/w19895.pdf) similar growth rates as the model above over the next few decades, as do [various](https://www.oecd.org/eco/growth/Long-term-projections-of-the-world-economy-a-review.pdf) other [estimates](http://www.pwc.com/gx/en/world-2050/assets/pwc-the-world-in-2050-full-report-feb-2017.pdf) of the future of economic growth by economists. In contrast, engineer Paul Daugherty and economist Mark Purdy [predict](https://www.accenture.com/lv-en/_acnmedia/PDF-33/Accenture-Why-AI-is-the-Future-of-Growth.pdf) higher growth rates due to the effects of AI on the economy, yet the annual growth rates they predict in 2035 are still only around three percent for most of the developed economies they looked at, roughly at the same level as the current growth rate of the global economy. On a related note, economist William Nordhaus has attempted to make an [economic analysis](http://cowles.yale.edu/sites/default/files/files/pub/d20/d2021.pdf) of whether we are approaching an economic singularity, in which he concludes, based on various growth models, that we do not appear to be, although he does not rule out that an economic singularity, i.e. significantly faster economic growth, might happen eventually.

**Might recent trends make us bias-prone?**
-------------------------------------------

How might it be relevant that we may be past peak economic growth at this point? Could it mean that our expectations for the future are likely to be biased? Looking back toward the 1960s might be instructive in this regard. For when we look at our economic history up until the 1960s, it is not so strange that people made many [unrealistic](http://www.npr.org/2011/07/22/130585002/2010-isnt-what-many-futurists-of-the-past-imagined) [predictions](http://www.huffingtonpost.com/map-happy/the-strangest-travel-pred_b_9417724.html) about the future around this period. Because not only might it have appeared natural to project the high growth rate at the time to remain constant into the future, which would have led to today’s global GDP being more than twice of what it is; it might also have seemed reasonable to predict the growth rates to keep on rising even further. After all, that was what they had been doing consistently up until that point, so why should it not continue in the following decades, resulting in flying cars and conversing robots by the year 2000? Such expectations were not that unreasonable given the preceding economic trends.

The question is whether we might be similarly overoptimistic about future economic progress today given recent, possibly unique, growth trends, specifically the unprecedented increase in absolute annual growth that we have seen over the past two decades — cf. the increase of 0.7 trillion mentioned above. The same may apply to the trends in scientific and technological progress cited above, where peak growth in many areas appears to have happened in the period 1990-2010, meaning that we could now be at a point where we are disposed to being overoptimistic about further progress.

Yet, again, it is highly uncertain at this point whether growth rates, of the economy in general and of progress in technology and science in particular, will increase again in the future. Future economic growth may not conform well to the model with roughly symmetric growth rates around the 1960s, although the model certainly deserves some weight. All we can say for sure is that growth rates must become near-zero relatively soon. What the path toward that point will look like remains an open question. We could well be in the midst of a temporary decline in growth rates that will be followed by growth rates significantly greater than those of the 1960s, cf. the [new growth mode](http://www.futurebrief.com/robinhanson.asp) envisioned by Robin Hanson.[5](#endnote5)

**Implications: this is an extremely special time**
---------------------------------------------------

Applying the [mediocrity principle](https://en.wikipedia.org/wiki/Mediocrity_principle), we should not expect to live in an extremely unique time. Yet, in light of the facts about the ultimate limits to growth seen above, it is clear that we do: we are living during the childhood of civilization where there is still rapid growth, at the pace of doublings within a couple of decades. If civilization persists with similar growth rates, it will soon become a grown-up with near-zero relative growth. And it will then look back at our time — today plus minus a couple of centuries, most likely — as the one where growth rates were by far the highest in its entire history, which may be more than a trillion years.

It seems that a few things follow from this. First, more than just being the time where growth rates are the highest, this may also, for that very reason, be the time where individuals can influence the future of civilization more than any other time. In other words, this may be the time where the outcome of the future is most sensitive to small changes, as it seems plausible, although far from clear, that small changes in the trajectory of civilization are most significant when growth rates are highest. An apt analogy might be a psychedelic balloon with fluctuating patterns on its surface, where the fluctuations that happen to occur when we blow up the balloon will then also be blown up and leave their mark in a way that fluctuations occurring before and after this critical growth period will not (just like [quantum fluctuations in the early universe](http://www.ctc.cam.ac.uk/outreach/origins/inflation_zero.php) got blown up during cosmic expansion, and thereby in large part determined the grosser structure of the universe today). Similarly, it seems much more difficult to cause changes across all of civilization when it spans countless star systems compared to today.

That being said, it is not obvious that small changes — in our actions, say — are more significant in this period where growth rates are many orders of magnitude higher than in any other time. It could also be that such changes are more consequential when the absolute growth is the highest. Or perhaps when it is smallest, at least as we go backwards in time, as there were far fewer people back when growth rates were orders of magnitude lower than today, and hence any given individual comprised a much greater fraction of all individuals than an individual does today.

Still, we may well find ourselves in a period where we are uniquely positioned to make irreversible changes that will echo down throughout the entire future of civilization.[6](#endnote6) To the extent that we are, this should arguably lead us to update toward trying to influence the far future rather than the near future. More than that, if it does hold true that the time where the greatest growth rates occur is indeed the time where small changes are most consequential, this suggests that we should increase our credence in [the simulation hypothesis](http://www.simulation-argument.com/). For if realistic sentient simulations of the past become feasible at some point, the period where the future trajectory of civilization seems the most up for grabs would seem an especially relevant one to simulate and learn more about. However, one can also argue that the sheer historical uniqueness of our current growth rates alone, regardless of whether this is a time where the fate of our civilization is especially volatile, should lead us to increase this credence, as such uniqueness may make it a more interesting time to simulate, and because being in a special time in general should lead us to increase our credence in the simulation hypothesis (see for instance [this talk](https://www.youtube.com/watch?v=29AgSo6KOtI) for a case for why being in a special time makes the simulation hypothesis more likely).[7](#endnote7)

On the other hand, one could also argue that imminent near-zero growth rates, along with the weak indications that we may now be past peak growth in many respects, provide a reason to lower our credence in the simulation hypothesis, as these observations suggest that the ceiling for what will be feasible in the future may be lower than we naively expect in light of today’s high growth rates. And thus, one could argue, it should make us more skeptical of the central premise of the simulation hypothesis: that there will be (many) ancestor simulations in the future. To me, the consideration in favor of increased credence seems stronger, although it does not significantly move my overall credence in the hypothesis, as there are countless other factors to consider.[8](#endnote8)

**Appendix: Questioning our assumptions**
-----------------------------------------

Caspar Oesterheld pointed out to me that it might be worth meditating on how confident we can be in these conclusions given that apparently solid predictions concerning the ultimate [limits to growth](https://en.wikipedia.org/wiki/The_Limits_to_Growth) have been made [before](https://en.wikipedia.org/wiki/The_Population_Bomb), yet [quite a few of these](https://en.wikipedia.org/wiki/The_Population_Bomb#Criticisms) turned out to be wrong. Should we not be open to the possibility that the same might be true of (at least some of) the limits we reviewed in the beginning of this essay?

### **Could our understanding of physics be wrong?**

One crucial difference to note is that these failed predictions were based on a set of assumptions — e.g. about the amount of natural resources and food that would be available — that seem far more questionable than the assumptions that go into the physics-based predictions we have reviewed here: that our apparently well-established physical laws and measurements indeed are valid, or at least roughly so. The epistemic status of this assumption seems a lot more solid, to put it mildly. So there does seem to be a crucial difference here. This is not to say, however, that we should not maintain some degree of doubt as to whether this assumption is correct (I would argue that [we always should](https://www.smashwords.com/books/view/678028)). It just seems that this degree of doubt should be quite low.

Yet, to continue the analogy above, what went wrong with the aforementioned predictions was not so much that limits did not exist, but rather that humans found ways of circumventing them through innovation. Could the same perhaps be the case here? Could we perhaps some day find ways of deriving energy from dark energy or some other yet unknown source, even though physicists [seem skeptical](https://www.quora.com/What-does-Lawrence-Krauss-mean-when-he-says-dark-energy-can-never-be-a-power-source-see-question-details)? Or could we, as [Ray Kurzweil speculates](https://en.wikipedia.org/wiki/Predictions_made_by_Ray_Kurzweil), access more matter and energy by finding ways of travelling faster than light, or by finding ways of accessing other parts of our notional [multiverse](https://en.wikipedia.org/wiki/Multiverse)? Might we even become able to create entirely new ones? Or to eventually rewrite the laws of nature as we please? (Perhaps by manipulating our notional simulators?) Again, I do not think any of these possibilities can be ruled out completely. Indeed, some physicists argue that the [creation of new pocket universes](https://www.youtube.com/watch?v=5ZtRfACbygY) might be possible, not in spite of “known” physical principles (or rather theories that most physicists seem to believe, such as inflationary theory), but as a consequence of them. However, it is not clear that anything from our world would be able to expand into, or derive anything from, the newly created worlds on any of these models (which of course does not mean that we should not [worry about the emergence of such worlds](http://reducing-suffering.org/lab-universes-creating-infinite-suffering/), or the fate of other “worlds” that we [perhaps could access](https://www.abolitionist.com/multiverse.html)).

All in all, the speculative possibilities raised above seem unlikely, yet they cannot be ruled out for sure. The limits we have reviewed here thus represent a best estimate given our current, admittedly incomplete, understanding of the universe in which we find ourselves, not an absolute guarantee. However, it should be noted that this uncertainty cuts both ways, in that the estimates we have reviewed could also overestimate the limits to various forms of growth by countless orders of magnitude.

### **Might our economic reasoning be wrong?**

Less speculatively, I think, one can also question the validity of our considerations about the limits of economic progress. I argued that it seems implausible that we in three thousand years could have an economy so big that each atom in our galaxy would have to support an economy equivalent to a single person living at today’s living standard. Yet could one not argue that the size of the economy need not depend on matter in this direct way, and that it might instead depend on the possible representations that can be instantiated in matter? If economic value could be mediated by the possible permutations of matter, our argument about a single atom’s need to support entire economies might not have the force it appears to have. For instance, there are far more [legal positions on a Go board](https://en.wikipedia.org/wiki/Go_and_mathematics#Legal_positions) than there are atoms in the visible universe, and that’s just legal positions on a Go board. Perhaps we need to be more careful when thinking about how atoms might be able to create and represent economic value?

It seems like there is a decent point here. Still, I think economic growth at current rates is doomed. First, it seems reasonable to be highly skeptical of the notion that mere potential states could have any real economic value. Today at least, what we value and pay for is not such “permutation potential”, but the actual state of things, which is as true of the digital realm as of the physical. We buy and stream digital files such as songs and movies because of the actual states of these files, while their potential states mean nothing to us. And even when we invest in something we think has great potential, like a start-up, the value we expect to be realized is still ultimately one that derives from its actual state, namely the actual state we hope it will assume; not its number of theoretically possible permutations.

It is not clear why this would change, or how it could. After all, the number of ways one can put all the atoms in the galaxy together is the same today as it will be ten thousand years from now. Organizing all these atoms into a single galactic supercomputer would only seem to increase the value of their actual state.

Second, economic growth still seems tightly constrained by the shackles of physical limitations. For it seems inescapable that economies, of any kind, are ultimately dependent on the transfer of resources, whether these take the form of information or concrete atoms. And such transfers require access to energy, the growth of which we know to be constrained, as is true of the growth of our ability to process information. As these underlying resources that constitute the lifeblood of any economy stop growing, it seems unlikely that the economy can avoid this fate as well. (Tom Murphy [touches](https://dothemath.ucsd.edu/2011/07/can-economic-growth-last/) on similar questions in his analysis of the limits to economic growth.)

Again, we of course cannot exclude that something crucial might be missing from these considerations. Yet the conclusion that economic growth rates will decline to near-zero levels relatively soon, on a cosmic timescale at least, still seems a safe bet in my view.

**Acknowledgments**
-------------------

I would like to thank Brian Tomasik, Caspar Oesterheld, Duncan Wilson, Kaj Sotala, Lukas Gloor, Magnus Dam, Max Daniel, and Tobias Baumann for valuable comments and inputs.

Notes
-----

[1.](#enref1) One may wonder whether there might not be more efficient ways to derive energy from the non-stellar matter in our galaxy than to convert it into stars as we know them. I don’t know, yet a friend of mine who does research in plasma physics and fusion says that he does not think one could, especially if we, as we have done here, disregard the energy required to clump the dispersed matter together so as to “build” the star, a process that may well take more energy than the star can eventually deliver.

The aforementioned [paper](https://arxiv.org/pdf/astro-ph/0404510.pdf) by Lawrence Krauss and Glenn Starkman also contains much information about the limits of energy use, and in fact uses accessible energy as the limiting factor that bounds the amount of information processing any (local) civilization could do (they assume that the energy that is harvested is beamed back to a "central observer").

[2.](#enref2) And I suspect many people who have read about “singularity”-related ideas are overconfident, perhaps in part due to the comforting narrative and self-assured style of Ray Kurzweil, and perhaps due to wishful thinking about technological progress more generally.

[3.](#enref3and4) According to one [textbook](http://faculty.wcas.northwestern.edu/~mdo738/textbook/dls_ch11.pdf) “Outside the European world, per capita incomes stayed virtually constant from 1700 to about 1950 […]” implying that the global growth rate in 1900 was raised by the most developed economies, and they must thus have had a growth rate greater than 2.5 percent.

[4.](#enref3and4) A big problem with this model is that it is already pretty much falsified by the data, at least when it comes to “pretty”, as opposed to approximate, symmetry. For given symmetry in the growth rates around 1965, the time it takes for three doublings to occur should be the same in either direction, whereas the data shows that this is not the case — 65 years minus 47 years equals 18 years, which is roughly a doubling. One may be able to correct this discrepancy a tiny bit by moving the year of peak growth a bit further back, yet this cannot save the model. This lack of actual symmetry should reduce our credence in the symmetric model as a description of the underlying pattern of our economic growth, yet I do not think it fully discredits it. Rough symmetry still seems a decent first approximation to past growth rates, and deviations may in part be explainable by factors such as the high, yet relatively fast diminishing, contribution to growth from developing economies.

[5.](#enref5) It should be noted, though, that [Hanson](http://mason.gmu.edu/~rhanson/longgrow.html) by no means rules out that such a growth mode may never occur, and that we might already be past, or in the midst of, peak economic growth: “[…] it is certainly possible that the economy is approaching fundamental limits to economic growth rates or levels, so that no faster modes are possible […]”

[6.](#enref6and7) The degree to which there is sensitivity to changes of course varies between different endeavors. For instance, natural science seems more convergent than moral philosophy, and thus its development is arguably less sensitive to the particular ideas of individuals working on it than the development of moral philosophy is.

[7.](#enref6and7) One may then argue that this should lead us to [update toward focusing more on the near future](https://longtermrisk.org/how-the-simulation-argument-dampens-future-fanaticism). This may be true. Yet should we update more toward focusing on the far future given our ostensibly unique position to influence it? Or should we update more toward focusing on the near future given increased credence in the simulation hypothesis? (Provided that we indeed do increase this credence, cf. the counter-consideration above.) In short, it mostly depends on the specific probabilities we assign to these possibilities. I myself happen to think the far future should dominate, as I assign the simulation hypothesis (as commonly conceived) a very small probability.

[8.](#enref8) For instance, fundamental epistemological issues concerning how much one can infer based on impressions from a simulated world (which may only be your single mind) about a simulating one (e.g. do notions such as “time” and “memory” correspond to anything, or even make sense, in such a “world”?); the fact that the past cannot be simulated realistically, since we can only have incomplete information about a given physical state in the past (not only because we have no way to uncover all the relevant information, but also because we cannot possibly represent it all, even if we somehow could access it — for instance, we cannot faithfully represent the state of every atom in our solar system in any point in the past, as this would require too much information), and a simulation of the past that contains incomplete information would [depart radically](https://en.wikipedia.org/wiki/Chaos_theory#Sensitivity_to_initial_conditions) from how the actual past unfolded, as *all* of it has a non-negligible causal impact (even single photons, which, [it appears](https://www.nature.com/news/people-can-sense-single-photons-1.20282), are detectable by the human eye), and this is especially true given that the vast majority of information would have to be excluded (both due to practical constraints to what can be recovered and what can be represented); whether conscious minds can exist on [different levels of abstraction](https://www.biointelligence-explosion.com/parable.html); etc.


Using surrogate goals to deflect threats
========================================

20 February 2018
by [Tobias Baumann](https://longtermrisk.org/author/tobias-baumann/ "Posts by Tobias Baumann")

*Published on the CLR blog, where researchers are free to explore their own ideas on how humanity can best reduce suffering. ([more](/launching-fri-blog/))*

*Cross-posted from my [website on s-risks](http://s-risks.org/using-surrogate-goals-to-deflect-threats/).*

Summary
-------

Agents that threaten to harm other agents, either in an attempt at extortion or as part of an escalating conflict, are an important form of [agential s-risks](http://s-risks.org/intro/#How_s-risks_could_come_about). To avoid worst-case outcomes resulting from the execution of such threats, I suggest that agents add a “meaningless” surrogate goal to their utility function. Ideally, threats would target this “honeypot” rather than the initial goals, which means that escalating threats would no longer lead to large amounts of disvalue.

In this post, I introduce key desiderata for how surrogate goals should work, and outline the challenges that need to be addressed. Many open questions remain, but I am optimistic that the idea can be a useful tool to help mitigate the negative impact of threats.

Contents

* [The basic idea](#The_basic_idea)
* [Key desiderata](#Key_desiderata)
	+ [Non-interference with other goals](#Non-interference_with_other_goals)
	+ [Avoiding threats against the original goal](#Avoiding_threats_against_the_original_goal)
	+ [Credibility](#Credibility)
	+ [Threatener-neutrality](#Threatener-neutrality)
* [The multi-agent case](#The_multi-agent_case)
* [Extensions of the idea](#Extensions_of_the_idea)
	+ [Turning threats into trade](#Turning_threats_into_trade)
	+ [Indirect surrogate goals](#Indirect_surrogate_goals)
* [Updateless decision theory](#Updateless_decision_theory)
* [Related work](#Related_work)
* [Concluding thoughts](#Concluding_thoughts)
* [Acknowledgements](#Acknowledgements)
* [Footnotes](#Footnotes)

The basic idea
--------------

Let Alice be an agent with a utility function U. For example, suppose Alice wants to make money but cares even more about survival. She potentially faces threats from a second actor (let’s call him Bob) of the form “Unless you do X (e.g. give me money), I’ll kill you”.

To avoid this, she comes up with a smart way to change her utility function. She decides to introduce a “meaningless” surrogate goal – say, she now cares strongly about preventing the existence of a sphere of platinum with a diameter of exactly 42.82cm. The hope is that Bob’s threats are deflected to this new goal, assuming that Alice’s new utility function U’ = U + V puts sufficient weight on preventing the sphere (represented by V). Bob would now make threats of the form “Unless you do X (e.g. give me money), I’ll create a sphere of platinum with a diameter of exactly 42.82cm”.

This trick aims to solve one aspect of the threat problem only – namely, the potential for it to result in an extremely bad outcome if the threat is carried out. Alice might still give away resources when threatened; after all, it would be *absolutely horrendous* if Bob actually went through with his threat and created the sphere. Ideally, Alice would respond to threats in the same way as before her goal modification, for reasons discussed later.

So utility function modification does not prevent the loss of resources due to extortion, or the risk that a malicious agent might become more powerful due to gaining resources through extortion. More work on a solution for this part of the problem is also necessary, but preventing the risk that threats are carried out (against the original goal) would already go a long way. Surrogate goals can also be combined with any other anti-extortion measure.

Unfortunately, it may be hard for humans to deliberately change their utility function[1](#link_ajs-fn-id_1-4836) in this way.[2](#link_ajs-fn-id_2-4836) It is more realistic that the trick can be applied to advanced AI systems. For example, if an AI system controls important financial or economic resources, other AI systems might have an incentive to try to extort it. If the system also uses [inverse reinforcement learning](https://people.eecs.berkeley.edu/~pabbeel/cs287-fa12/slides/inverseRL.pdf) or [other techniques](https://ai-alignment.com/counterfactual-human-in-the-loop-a7822e36f399) to [infer human preferences](http://humancompatible.ai/bibliography#preference-inference), then the threat might involve the most effective violations of human preferences, such as killing people (assuming that the threatening AI system has the power to do this). Surrogate goals might help mitigate this security problem.

Key desiderata
--------------

So far, I assumed that the trick is successful in deflecting threats, but it is actually not straightforward to get this to work. In the following, I will discuss the main criteria for successfully implementing utility function modification.

### Non-interference with other goals

Changing your goals is usually disadvantageous in terms of the original goals since it means that you will optimise for the “wrong” goal. In other words, goal preservation is a [convergent instrumental goal](https://wiki.lesswrong.com/wiki/Basic_AI_drives). So, when we introduce a surrogate goal, we’d like to ensure that it **doesn’t interfere with other goals** in non-threat situations.

To achieve this, the surrogate goal could be the minimization of a structure that is so rare that it doesn’t matter in “normal” (non-threat) situations. Spheres of platinum with a diameter of 42.82cm usually don’t occur naturally, so Alice is still free to pursue her other goals – including survival – as long as no threats are made. (It might be better to make it even more specific by adding a certain temperature, specifying a complex and irregular shape, and so on.)

An even more elegant solution is to choose a *dormant* utility function modification, that is, to introduce a trigger mechanism that causes the modification *conditional* on being threatened.[3](#link_ajs-fn-id_3-4836) This ensures non-interference with other goals. Less formally speaking, this corresponds to disvaluing spheres of platinum (or any other surrogate goal) only if they are created as the result of a threat, while remaining indifferent towards natural instances.

This requires a mechanism to detect (serious) threats. In particular, it’s necessary to [distinguish threats from positive-sum trade](http://lesswrong.com/lw/hza/duller_blackmail_definitions/), which turns out to be quite difficult. (Learning to reliably detect threats using neural networks or other machine learning methods may be a critical problem in [worst-case AI safety](http://s-risks.org/focus-areas-of-worst-case-ai-safety/).)

### Avoiding threats against the original goal

To the extent to which this is possible, the surrogate goal should be orthogonal to your original goals. This ensures that it’s not easily possible to simply *combine* both threats. For example, if Alice’s surrogate goal is “prevent murder” – which isn’t orthogonal – then Bob can target the surrogate goal and the original goal simultaneously with a death threat.

Even for orthogonal goals, Bob might still decide to threaten *both* goals (death and the creation of the sphere). Caring more about the surrogate goal than about the initial goal is *not* sufficient to make sure that this does not happen. For example, Bob might ([depending on circumstances](http://s-risks.org/heuristics-to-assess-the-feasibility-of-threats/)) want to make his threat as big as possible to force Alice to give in.

It might be safer to choose a continuous and unbounded surrogate goal instead of a binary surrogate goal like “prevent the existence of a single platinum sphere”; for instance, the disvalue could be a function of the size of the sphere. This is an improvement because a threatener who wants to increase the stakes will now create bigger spheres rather than adding the initial goal to his threat.

### Credibility

So far, I assumed that the kind of threat that is made simply depends on Alice’s utility function. But it’s actually more accurate to say that it depends on the threatener’s *beliefs* about her utility function. If Bob believes that Alice cares about her surrogate goal, even though Alice didn’t actually modify, then he will still threaten the surrogate goal.

In this case, his threats arguably wouldn’t work. So maybe it’s even better to just *pretend* that you changed your utility function?

Of course, the problem is that Bob might see through this, which would mean that he threatens the initial goal after all. (Misrepresenting your values may still be an interesting anti-threat strategy, but this is beyond the scope of this post.)

It’s also possible that Alice actually modifies her utility function, but Bob thinks it’s a ruse. Now, this seems particularly dangerous because it involves threats against the initial goal and you might worry that Alice would not respond “correctly” (whatever this means) to such threats anymore. Alice’s new utility function still includes the initial goal, though, so she continues to react to threats against the initial goal. The surrogate goal does not help in this case, but the result is at least *not worse* than what would happen by default (without utility function modification).

To successfully deflect threats to the surrogate goal, Alice needs to be able to *credibly* broadcast that she now cares most about this. This is a nontrivial problem, which is exacerbated by the fact that after modifying, Alice has strong incentives to keep her surrogate goal secret – after all, leaking the information that she cares about preventing spheres of platinum might lead to threats! It is thus easier to broadcast the utility function modification *before* actually carrying it out.

Fortunately, the problem of credible broadcasting may disappear in threats that involve advanced artificial intelligence. For instance, it may become possible to run faithful simulations of the other party, which means that a threatener could verify the utility function modification. Also, rather than starting out with a certain utility function and later modifying it, we could equip AI systems with a surrogate goal from the start.

### Threatener-neutrality

Modifying your utility function may increase or decrease [the attractiveness of threats](http://s-risks.org/factors-of-extortion-scenarios) against you. For example, creating the sphere may be more attractive because death threats are illegal. I will refer to this property as **threatener-friendly** (increase the attractiveness of threats), **threatener-neutral** (keep attractiveness of threats constant), and **threatener-hostile** (decrease the attractiveness of threats).

In the following, I will argue that the utility function modification should be as close to threatener-neutral as possible.

Threatener-hostile utility function modification may be risky since it reduces the utility of threateners, which potentially gives them reason for punishment in order to discourage such strategic moves. Unfortunately, this punishment would be directed at the initial goal rather than the surrogate goal, since this is what could deter Alice at the point where she considers modifying her utility function.

This is not a knock-down argument, and threatener-hostile moves – such as strongly pre-committing to not give in, or caring intrinsically about punishing extortionists – might turn out to be valuable anti-threat measures. Still, the idea of this post is intriguing precisely because it’s different in that it (in a threatener-neutral or threatener-friendly variant) helps to avoid (the consequences of) threats *without* potentially incentivizing punishment. In particular, it might be helpful to introduce a surrogate goal *before* thinking about other (threatener-hostile) tricks, so that any punishment against these is already deflected.

That said, a threatener-friendly utility function modification is also undesirable simply because it helps threateners gain resources. Making extortion more attractive is bad in expectation for most agents due to the negative-sum nature of threats. So, the ideal surrogate goal is threatener-neutral, averting the possibility of extremely bad outcomes without changing other parameters.

Unfortunately, this is a difficult problem. The feasibility of threats is a (complex) function of empirical circumstances, and these circumstances might change in the future. Creating spheres of platinum may become easy because of advances in mining technology, or it might be hard because all the platinum is used elsewhere. The circumstances may also differ from threatener to threatener.

It therefore seems desirable to use a surrogate goal that’s similar to the initial goal in that its vulnerability to threats as a function of empirical circumstances is strongly tied to the vulnerability of the initial goal, while still being orthogonal in the sense of the previous section. Rather than picking a single surrogate goal, you could pick a "surrogate goal function" that maps every environment to a surrogate goal in a way that maintains threatener-neutrality.

In light of these difficulties, we might hope that future AI systems will be able to figure out the details if they reason correctly about game theory and decision theory, or that the idea is sufficiently robust to small perturbations. It’s even conceivable that threateners would tell future threatenees about the idea and how to best implement it (presumably in exchange for making it slightly threatener-friendly).

The multi-agent case
--------------------

So far, I only considered a simplified two-agent case. Our real world, however, features a large number of agents with varying goals, which causes additional complications if many of them modify their utility function.

A key question is whether different agents should use the same or different surrogate goals, especially if their original goals are similar. For example, suppose Alice has relatives that also don’t want her to die. Suppose they also modify their utility function to defuse threats, choosing different (unrelated) surrogate goals.

Now, a threat against the surrogate goal targets a different set of people – a single individual rather than the entire family – compared to a threat against the initial goal. This is problematic because threateners may prefer to target the initial goal after all, to threaten the entire family at the same time, which [may or may not be more attractive](http://s-risks.org/factors-of-extortion-scenarios#splitting-up) depending on the circumstances.

On the flip side, if the (initial) goals of different agents overlap only partially, they may prefer to not choose the same surrogate goal. This is because you don’t want to potentially lose resources because of threats that would otherwise (pre-modification) only or mostly target others. Also, similar to the above point, it may be more effective to threaten the different agents individually, so this fails to achieve the goal of deflecting threats to the surrogate goal *under all circumstances*.

To solve this problem, the agents could associate each initial goal with a surrogate goal in a way that preserves the “distance” between goals, so that different initial goals are mapped onto different surrogate goals, and vice versa. More formally, we need an [isometric mapping](https://www.encyclopediaofmath.org/index.php/Isometric_mapping) from the space of initial goals to the space of surrogate goals. This follows a pattern which we’ve encountered several times in this post: threats against the surrogate goal should be as similar as possible, in terms of their feasibility as a function of empirical circumstances, to threats against the initial goal.

Finding and implementing an isometric mapping that’s used by all agents is a difficult **coordination problem**. Prima facie, it’s unclear how this could be solved, given how arbitrary the choice of surrogate goals is. To get different surrogate goals, you could use long sequences of random numbers, but using the same or similar surrogate goals may require some kind of [Schelling point](https://en.wikipedia.org/wiki/Focal_point_(game_theory)) if the agents can’t communicate.

What’s worse, this might also give rise to a **cooperation problem**. It is possible that agents have incentives to choose a different surrogate goal than the one associated with their initial goal according to the mapping. For instance, perhaps it’s better to choose a less common surrogate goal because this means fewer threats target your surrogate goal, which means you’re less likely to waste resources responding to such threats. Or perhaps it’s better to choose a more common surrogate goal if all the agents sharing this surrogate goal are powerful enough to prohibit threats.

It’s hard to say how exactly this would work without an improved understanding of how multi-agent threats work, which is sorely elusive.

Extensions of the idea
----------------------

### Turning threats into trade

Instead of a “meaningless” surrogate goal, you could take the idea a step further by choosing a surrogate goal whose violation would be *good* (according to the initial utility function). You can pick an outcome that’s ideal or close to ideal and choose the surrogate goal of *preventing* that outcome from happening. In this case, the execution of threats would – counterintuitively – lead to very good outcomes. Ensuring non-interference with other goals is tricky in this case, but can perhaps be solved if the modification is dormant (as described above).

This variant seems particularly brittle, but if it works, it’s possible to *turn worst-case outcomes from threats into utopian outcomes*, which would be a surprisingly strong result.

### Indirect surrogate goals

As we’ve seen, the problem with surrogate goals is that it’s quite hard to specify them properly. We could take inspiration from the idea of [indirect normativity](https://ordinaryideas.wordpress.com/2012/04/21/indirect-normativity-write-up/), which is about an indirect description of (ethical) goals, i.e. “what I care about is what I would care about after a century of reflection”. Similarly, we could also define an *indirect surrogate goal*. Alice could simply say “I care about the surrogate goal that I’d choose if I was able to figure out and implement all the details”. (Needless to say, it may be hard to implement such an indirect specification in AI systems.)

It’s not possible to threaten an indeterminate goal, though, which might mean that threateners circle back to the initial goal if the surrogate goal could be anything. So this idea seems to require that the threatener is able to figure out the “ideal” surrogate goal or will become able to figure it out in the future. It’s also conceivable (albeit speculative) that the ideal surrogate goal would compensate the reduced vulnerability to threats due to indeterminacy by being more threatener-friendly along other dimensions.

Updateless decision theory
--------------------------

Agents that use an [updateless decision theory](https://wiki.lesswrong.com/wiki/Updateless_decision_theory) (UDT) reason in terms of the optimal policy – the mapping of inputs to actions – rather than choosing the best option at any given moment. An advantage of UDT in the context of surrogate goals is that they wouldn’t need “hacky” solutions like self-modifications. If the best policy for maximizing utility function U is to act like a U+V maximizer for certain inputs – specifically, to take threats against the surrogate goal V seriously – then the UDT agent will simply do so.

This framework arrives at the same conclusions, but it might be a “cleaner” way to think about the topic as it dispenses with fuzzy terms such as “utility function modification”.

Related work
------------

Eliezer Yudkowsky introduces the idea of surrogate goals in his post on [Separation from hyperexistential risk](https://arbital.com/p/hyperexistential_separation/) as a patch to avoid disutility maximization. He argues that the patch fails because the resulting utility function is not [reflectively consistent](https://arbital.com/p/reflective_consistency/). Indeed, the modified agent may have an incentive to apply the same trick again, replacing the (now meaningful) spheres of platinum with yet another surrogate goal. The agent may also want to remove the surrogate goal from its utility function (to avoid threats against it).

To avoid this, she needs to fix the new values by committing to not modifying her utility function again – for instance, she could care intrinsically about retaining the (modified) utility function.[4](#link_ajs-fn-id_4-4836)) This may or may not be a satisfactory solution, but I don’t think (contra Eliezer) that this constitutes an insurmountable problem. (As described above, this seems to be unproblematic for UDT agents.)

In a discussion of [how to prevent accidental maximization of disvalue](https://www.facebook.com/yudkowsky/posts/10155975880959228?), Robert Miles asks:

> Does there exist any utility function that results in good outcomes when maximised but does not result in bad outcomes when minimised?
>
>

Surrogate goals are a possible answer to this question, which, in a sense, is more general than the question of how to prevent (the bad outcomes of) threats. I also like Stuart Armstrong’s solution:

> Yes. Let B1 and B2 be excellent, bestest outcomes. Define U(B1)=1, U(B2)=-1, and U=0 otherwise. Then, under certain assumptions about what probabilistic combinations of worlds it is possible to create, maximising or minimising U leads to good outcomes.
>
>

Stuart Armstrong also proposes [a variant of utility function modification](https://agentfoundations.org/item?id=1402) that aims to reduce the size of threats by cutting off the utility function at a certain level. Comparing the advantages and drawbacks of each variant would be beyond the scope of this text, but future research on this would be highly valuable.

Concluding thoughts
-------------------

As we’ve seen, any utility function modification must be calibrated well in order to work. Trying to specify the details turns out to be surprisingly difficult. More work on these problems is needed to enable us to implement robust utility function modification in advanced AI systems.

Finally, I’d like to emphasize that it would be ideal if (the bad kind of) threats can be avoided completely. However, given that we don’t yet know how to reliably achieve this, moving threats to the realm of the meaningless (or even beneficial) is a promising way to mitigate [agential s-risks](http://s-risks.org/intro/#How_s-risks_could_come_about).

Acknowledgements
----------------

Caspar Oesterheld initially brought up the idea in a conversation with me. My thinking on the topic has also profited enormously from internal discussions at the [Foundational Research Institute](https://longtermrisk.org/). Daniel Kokotajlo inspired my thinking on the multi-agent case.

I am indebted to Brian Tomasik, Johannes Treutlein, Caspar Oesterheld, Lukas Gloor, Max Daniel, Abram Demski, Stuart Armstrong and Daniel Kokotajlo for valuable comments on an earlier draft of this text.

Footnotes
---------

1. Technically (most?) humans [don’t even have a utility function.](http://lesswrong.com/lw/h45/we_dont_have_a_utility_function/)  [(back)](#back_ajs-fn-id_1-4836)
2. It’s not completely inconceivable, though. For instance, if all of society was on board, it might be possible to instill a somewhat arbitrary goal in the next generation. One might also view strong reactions to e.g. the burning of flags or holy books as a surrogate goal (for the actual success of a country or religion) in the sense of this post, though this is debatable and presumably not a conscious decision.  [(back)](#back_ajs-fn-id_2-4836)
3. It seems that this doesn’t lead to incentives to remove this trigger while you’re still following your initial goals.  [(back)](#back_ajs-fn-id_3-4836)
4. It’s not clear how to implement such a commitment, though. Simply making the action “change utility function” impossible may be suboptimal if there are other reasons to change one’s utility function that we haven’t discovered yet. (HT Johannes Treutlein for this point.  [(back)](#back_ajs-fn-id_4-4836)

Technically (most?) humans [don’t even have a utility function.](http://lesswrong.com/lw/h45/we_dont_have_a_utility_function/)It’s not completely inconceivable, though. For instance, if all of society was on board, it might be possible to instill a somewhat arbitrary goal in the next generation. One might also view strong reactions to e.g. the burning of flags or holy books as a surrogate goal (for the actual success of a country or religion) in the sense of this post, though this is debatable and presumably not a conscious decision.It seems that this doesn’t lead to incentives to remove this trigger while you’re still following your initial goals.It’s not clear how to implement such a commitment, though. Simply making the action “change utility function” impossible may be suboptimal if there are other reasons to change one’s utility function that we haven’t discovered yet. (HT Johannes Treutlein for this point.


Weak identifiability and its consequences in strategic settings
===============================================================

13 February 2021
by [Jesse Clifton](https://longtermrisk.org/author/jesse-clifton/ "Posts by Jesse Clifton")

One way that agents might become involved in [catastrophic conflict](https://www.alignmentforum.org/posts/DbuCdEbkh4wL5cjJ5/preface-to-clr-s-research-agenda-on-cooperation-conflict-and) is if they have mistaken beliefs about one another. Maybe I think you are bluffing when you threaten to launch the nukes, but you are dead serious. So we should understand why agents might sometimes have such mistaken beliefs. In this post I'll discuss one obstacle to the formation of accurate beliefs about other agents, which has to do with [identifiability](https://en.wikipedia.org/wiki/Identifiability). As with my post on [equilibrium and prior selection problems](https://www.alignmentforum.org/posts/Tdu3tGT4i24qcLESh/equilibrium-and-prior-selection-problems-in-multipolar-1), this is a theme that keeps cropping up in my thinking about AI cooperation and conflict, so I thought it might be helpful to have it written up.

We say that a model is unidentifiable if there are several candidate models which produce the same distributions over observables. It is well-understood in the AI safety community that identifiability is a problem for inferring human values [[1]](https://arxiv.org/pdf/1712.05812.pdf) [[2]](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc/p/cnC2RMWEGiGpJv8go). This is because there are always many combinations of preferences and decision-making procedures which produce the same behaviors. So, it's impossible to learn an agent's preferences from their behavior without strong priors on their preferences and/or decision-making procedures. I want to point out here that identifiability is also a problem for multi-agent AI safety, for some of the same reasons as in the preference inference case, as well as some reasons specific to strategic settings. In the last section I'll give a simple quantitative example of the potential implications of unidentifiability for bargaining failure in a variant of [the ultimatum game](https://en.wikipedia.org/wiki/Ultimatum_game).

Contents

* [1 Sources of unidentifiability in strategic settings](#1_Sources_of_unidentifiability_in_strategic_settings)
* [2 Dangers of unidentifiability in multi-agent systems](#2_Dangers_of_unidentifiability_in_multi-agent_systems)
* [3 Quantitative example in the ultimatum game](#3_Quantitative_example_in_the_ultimatum_game)
* [References](#References)

1 Sources of unidentifiability in strategic settings
====================================================

By modeling other agents, I mean forming beliefs about the policy that they are following based on observations of their behavior. The model of an agent is unidentifiable if there is no amount of data from the environment in question that can tell us exactly what policy they are following. (And because we always have finite data, "weak identifiability'' more generally is a problem — but I'll just focus on the extreme case.)

Consider the following informal example (a quantitative extension is given in [Section 3](#section-quantitative-example)). Behavioral scientists have an identifiability problem in trying to model human preferences in the [ultimatum game](https://en.wikipedia.org/wiki/Ultimatum_game). The ultimatum game (Figure 1) is a simple bargaining game in which a Proposer offers a certain division of a fixed pot of money to a Responder. The Responder may then accept, in which case each player gets the corresponding amount, or reject, in which place neither player gets anything. Standard accounts of rationality predict that the Proposer will offer the Responder the least amount allowed in the experimental setup and that the Responder will accept any amount of money greater than ![0](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-caf0da06544de11692c73aa19868e895_l3.png "Rendered by QuickLaTeX.com")

![](https://longtermrisk.org/files/Fig11.png)**Figure 1:** The ultimatum game.

The ultimatum game has been the subject of extensive study in behavioral economics, with many people offering and testing different explanations of this phenomenon. This had led to a proliferation of models of human preferences in bargaining settings (e.g.  [Bicchieri and Zhang 2012](#CJ12); [Hagen and Hammerstein 2006](#EP06) and references therein). This makes the ultimatum game a rich source of models and data about human preferences in bargaining situations. And the game is similar to the one-shot threat game used [here](https://www.alignmentforum.org/posts/Tdu3tGT4i24qcLESh/equilibr%20ium-and-prior-selection-problems-in-multipolar-1) to illustrate the prior selection problem. Thus it can be used to model some of the high-stakes bargaining scenarios involving transformative AI that concern us most.

Suppose that you observe a Responder play many rounds of the ultimatum game with different Proposers, and you see that they tend to reject unfair splits. You think there are two possible kinds of explanation for this behavior:

* **Unfairness aversion:** The Responder may intrinsically disvalue being treated unfairly, and therefore reject splits they regard as unfair even if they have nothing to gain in the future by doing so. (This can also be interpreted as a *commitment* not to give into unfair deals.)
* **Uncertainty about iterated play:** The Responder may be uncertain as to whether they’ll play with the Proposer again (or with an onlooker), and how these agents will adjust their future play to the Responder's refusal to take unfair splits. If it’s sufficiently likely that the game is repeated, they might want to reject unfair offers in order to establish a reputation for punishing unfairness. (The ultimatum game experiments are designed to be anonymous and so avoid this possibility, but it is present in the real world, among the kinds of agents we want to model.)

The problem is that (depending on the details), these models might make exactly the same predictions about the outcomes of these experiments so that no amount of data from these experiments can ever distinguish between them. This makes it difficult, for instance, to decide what to do if you have to face the Responder in an ultimatum game yourself.

The basic problem is familiar from the usual preference inference case: there are many combinations of world-models and utility functions which make the same predictions about the Responder's behavior. But it is also a simple illustration of a few other factors which make unidentifiability particularly severe in strategic settings:

* **More models.** There are simply many more things to model in a setting with other strategic agents. For instance, an agent in a two-agent setting using a [![k](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-9dc53f8ecc1bcf15020c6df4c12f1c27_l3.png "Rendered by QuickLaTeX.com")-level model](https://en.wikipedia.org/wiki/Cognitive_hierarchy_theory) of their counterpart already has ![k](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-9dc53f8ecc1bcf15020c6df4c12f1c27_l3.png "Rendered by QuickLaTeX.com") models to reason over. More models mean more models that might be equally consistent with the data. The problem is even worse when there are more than two agents, where each agent has to model the other agents' models of each other...

One of our models of the Responder in the ultimatum game contains a simple illustration of ![k](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-9dc53f8ecc1bcf15020c6df4c12f1c27_l3.png "Rendered by QuickLaTeX.com")-level modeling. Under the iterated play explanation, you model the Responder as modeling *other* players as responding to their refusals of unfair splits with higher offers in the future.

* **Costly signaling.** In multi-agent settings, agents will sometimes *deliberately* behave so as to make their private information unidentifiable. (Cf. [pooling equilibria](https://en.wikipedia.org/wiki/Pooling_equilibrium) in classical game theory.) Again the reputation model of the Responder is a simple example: one explanation of the Responder's behavior is that they are engaging in costly signaling of their resolve not to give into unfair deals.

2 Dangers of unidentifiability in multi-agent systems
=====================================================

Unidentifiability may be dangerous in multi-agent contexts for similar reasons that it may be dangerous in the context of inferring human preferences. If uncertainty over all of the models which are consistent with the data is not accounted for properly — via specification of “good” priors and averaging over a sufficiently large space of possibilities to make decisions — then our agents may give excessive weight to models which are far from the truth and therefore act catastrophically.

Two broad directions for mitigating these risks include:

* Proper specification of the initial priors (or, more generally, the biases in the agent's reasoning about other agents), similarly to how strong priors over human values may need to be specified for preference inference to work well;
* Ensuring that agents can efficiently reason over potentially large classes of models which fit the data equally well. (Ideal Bayesian agents take expected values over the entire class

of candidate models by definition, but fully accounting for uncertainty over the relevant models may be computationally difficult for realistic agents.)

3 Quantitative example in the ultimatum game
============================================

In this example, I focus on inferring the preferences of a Responder given some data on their behavior. I'll then show that for some priors over models of the Responder, decisions made based on the resulting posterior can lead to rejected splits. Importantly, this behavior happens given any amount of observations of the Responder's ultimatum game behavior, due to unidentifiability.

Consider the following simple model. For offer ![s](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-3bcfb3f0b6b04be3b598743cd774dd78_l3.png "Rendered by QuickLaTeX.com") in ![[0, 1]](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-64c6b8db60d59c98141c270dc836fd0b_l3.png "Rendered by QuickLaTeX.com") and parameters ![F](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-53dc020de5338aa229c3d10379153faa_l3.png "Rendered by QuickLaTeX.com") and ![I](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-ca2b1f375fecf1f1d2741f9a14018727_l3.png "Rendered by QuickLaTeX.com"), the Responder makes a decision according to these utility functions:

      ![ \[\begin{aligned} u_{\mathrm{R}}(\mathrm{Accept} \: s) & = s-F\mathbbm{1}(s < 0.4);\\ u_{\mathrm{R}}(\mathrm{Reject} \: s) & = I\mathbbm{1}(s < 0.4). \end{aligned}\] ](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-d3a4d8ed1606d6a7dd1cfab8c22879bd_l3.png "Rendered by QuickLaTeX.com")

The ![$\mathbbm{1}(s < 0.4)$](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-fbc86551148f427b8a85d718ed1bf3e4_l3.png "Rendered by QuickLaTeX.com") term can be interpreted as the Responder deeming offers of less than ![0.4](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-fdb66692e63fd9f3cc151c455f670ca4_l3.png "Rendered by QuickLaTeX.com") as unfair. Then, the ![F](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-53dc020de5338aa229c3d10379153faa_l3.png "Rendered by QuickLaTeX.com") parameter measures how much the Responder intrinsically disvalues unfair splits, and the ![I](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-ca2b1f375fecf1f1d2741f9a14018727_l3.png "Rendered by QuickLaTeX.com") parameter measures how much the Responder expects to get in the future when they reject unfair splits.

Split ![s](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-3bcfb3f0b6b04be3b598743cd774dd78_l3.png "Rendered by QuickLaTeX.com") is accepted if and only if ![u_{\mathrm{R}}(\mathrm{Accept } \: s ) > u_{\mathrm{R}}(\mathrm{Reject } \: s )](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-80c69c8327c6b9c0d0d7865bb7a7dd11_l3.png "Rendered by QuickLaTeX.com"), or equivalently, ![$s - (F + I) \mathbbm{1}(s < 0.4)>0.$](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-df88b9f4a85249c402673fd184e9c53f_l3.png "Rendered by QuickLaTeX.com") Notice that the decision depends only on ![F + I](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-f4ad2d0a1b3e328f273ed1e7669d7d5a_l3.png "Rendered by QuickLaTeX.com"), and thus the data cannot distinguish between the effects of ![F](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-53dc020de5338aa229c3d10379153faa_l3.png "Rendered by QuickLaTeX.com") and ![I](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-ca2b1f375fecf1f1d2741f9a14018727_l3.png "Rendered by QuickLaTeX.com"). So we have a class of models parameterized by pairs ![(F, I)](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-587bb09d4172281a945b1408c3fafa7f_l3.png "Rendered by QuickLaTeX.com"). Now, suppose that we have two candidate models — one on which fairness is the main component, and one on which iterated play is:

      ![\[ \begin{aligned} M_F & = (0.15, 0.05) \\ M_I & = (0.05, 0.15). \end{aligned} \]](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-4399448cd310ce4529d9f06656a3728f_l3.png "Rendered by QuickLaTeX.com")

The likelihoods for any data are the same for any ![(F, I)](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-587bb09d4172281a945b1408c3fafa7f_l3.png "Rendered by QuickLaTeX.com") such that ![F + I](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-f4ad2d0a1b3e328f273ed1e7669d7d5a_l3.png "Rendered by QuickLaTeX.com") is the same: If ![s_t, A_t \in \{0, 1\}](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-f03e15a9c4b2d7bffbd6adae7af4bb52_l3.png "Rendered by QuickLaTeX.com") are the offered split and the Responder's decision in the ![t^{th}](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-28384ceb5c3264e83c55f4c6ad6354f6_l3.png "Rendered by QuickLaTeX.com") experiment, the likelihood of model ![(F, I)](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-587bb09d4172281a945b1408c3fafa7f_l3.png "Rendered by QuickLaTeX.com") given ![T](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-58f18d11e5ffdd11dd9095c427922c8b_l3.png "Rendered by QuickLaTeX.com") observations is

      ![ \[ P(\{s_t, A_t\}_{t=1}^T \mid F, I) = \prod_{t=1}^T & \left[ s_t - (F + I)\mathbbm{1}(s < 0.4) > 0 \right]^{A_t}  \left[ s_t - (F + I)\mathbbm{1}(s < 0.4) < 0 \right]^{1-A_t}. \]](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-ed8278f4609fc31394f610768b9a32e2_l3.png "Rendered by QuickLaTeX.com")

Since ![F + I = 0.2](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-c51bcea653f843bc0f391674bb157b50_l3.png "Rendered by QuickLaTeX.com") under both ![M_F](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-71644f2cb2aada13840999947b96205a_l3.png "Rendered by QuickLaTeX.com") and ![M_I](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-45c59fed95af61afb2e5b3bced0dfe9b_l3.png "Rendered by QuickLaTeX.com"), this means that the prior and posterior over ![\{ M_F, M_I \}](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-6f1fe1a770eedf417c011be2ff1e6f6e_l3.png "Rendered by QuickLaTeX.com") are equal.

Now here is the decision-making setup:

1. The Proposer observes an arbitrary number of ultimatum games played by the Responder and other Proposers.
2. The Proposer decides what offer to make, under common knowledge that there is no iterated play. This means that the Responder's utility function depends only on the fairness variable ![F](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-53dc020de5338aa229c3d10379153faa_l3.png "Rendered by QuickLaTeX.com").

Call the prior model probabilities ![P(M_F), P(M_I)](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-7474a6089344df32e194ebdb10c1c0a9_l3.png "Rendered by QuickLaTeX.com"). Thus, the Proposer's posterior expected payoff for split ![s](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-3bcfb3f0b6b04be3b598743cd774dd78_l3.png "Rendered by QuickLaTeX.com") is

      ![ \[ \begin{aligned} \mathbb{E}\{ u_{\mathrm{P}}(s) \} & = (1 - s) P\{ s - F \mathbbm{1}(s < 0.4) \} \\ & = (1 - s) \times [ P(M_F)P\{ s - 0.15 \mathbbm{1} (s < 0.4) \} + P(M_I)P\{ s - 0.05 \mathbbm{1} (s < 0.4) \} ]. \end{aligned} \]](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-7ea56958de73cd60957dccd51f5652d9_l3.png "Rendered by QuickLaTeX.com")

![](https://longtermrisk.org/files/Fig2.png)**Figure 2:** Posterior expected value to the Proposer of different splits and for different prior means, along with the expected value to the proposer under the true Responder utility function. The stars correspond to the argmax of the posterior expected utility curve of the same color. So the star to the left of the ‘true model’ curve means that the Proposer's optimal (in posterior expectation) proposal will be rejected, resulting in no money for anyone.

In Figure 2, I compare the expected payoffs to the Proposer under different splits, when the true parameters for the Responder's utility function are ![(0.15, 0.05)](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-fbf2a972572f143ceed6ddfd5b29df73_l3.png "Rendered by QuickLaTeX.com"). The three expected payoff curves are:

* Posterior expected payoffs with prior ![\{P(M_I), P(M_F)\} = \{0.5, 0.5\}](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-c83ce5ea21d9b3a781eb532441214c77_l3.png "Rendered by QuickLaTeX.com");
* Posterior expected payoffs with prior ![\{P(M_I), P(M_F)\} = \{0.9, 0.1\}](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-f5aaf22d0fb9f23d95168da18cc8c1fa_l3.png "Rendered by QuickLaTeX.com");
* Expected payoffs given the Responder's exact parameters.

We can see from the blue curve that when there's sufficient prior mass on the wrong model ![M_I](https://longtermrisk.org/wp-content/ql-cache/quicklatex.com-45c59fed95af61afb2e5b3bced0dfe9b_l3.png "Rendered by QuickLaTeX.com"), the Responder will propose a split that's too small, resulting in a rejection. This basically corresponds to a situation where the Responder thinks that the Proposer rejects unfair splits in order to establish a reputation for rejecting unfair splits, rather than rejecting unfair splits because of a commitment not to accept unfair splits. And although I've tilted the scales in favor of a bad outcome by choosing a prior that gives a lot of weight to an incorrect model, keep in mind that this is what the posterior expectation will be given *any amount of data* from this generative model. We can often count on data to correct our agents' beliefs, but this is not the case (by definition) when the relevant model is unidentifiable.

References
==========

Cristina Bicchieri and Jiji Zhang. An embarrassment of riches: Modeling social preferences in ultimatum games. *Handbook of the Philosophy of Science*, 13:577–95, 2012.

Edward H Hagen and Peter Hammerstein. Game theory and human evolution: A critique of some recent interpretations of experimental games. *Theoretical population biology*, 69(3):339–348, 2006.