Generalization in Reward Learning
---------------------------------

An overview of reinforcement learning, generalization, and reward learning
--------------------------------------------------------------------------

[![Max Chiswick](https://miro.medium.com/v2/resize:fill:88:88/1*9bQVM9sOQq-c_glaRi6_8A.jpeg)](https://chisness.medium.com/?source=post_page-----da6c99d9e48--------------------------------)[![Towards Data Science](https://miro.medium.com/v2/resize:fill:48:48/1*CJe3891yB1A1mzMdqemkdg.jpeg)](https://towardsdatascience.com/?source=post_page-----da6c99d9e48--------------------------------)[Max Chiswick](https://chisness.medium.com/?source=post_page-----da6c99d9e48--------------------------------)

·[Follow](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F_%2Fsubscribe%2Fuser%2F98505f8c082&operation=register&redirect=https%3A%2F%2Ftowardsdatascience.com%2Fassessing-generalization-in-reward-learning-intro-and-background-da6c99d9e48&user=Max+Chiswick&userId=98505f8c082&source=post_page-98505f8c082----da6c99d9e48---------------------post_header-----------)

Published in[Towards Data Science](https://towardsdatascience.com/?source=post_page-----da6c99d9e48--------------------------------)·16 min read·Sep 30, 2020--

Listen

Share

**Authors: Anton Makiievskyi, Liang Zhou, Max Chiswick**

*Note: This is the* ***first*** *of* ***two*** *blog posts (part* [*two*](https://chisness.medium.com/assessing-generalization-in-reward-learning-implementations-and-experiments-de02e1d08c0e)*). In these posts, we describe a project we undertook to assess the ability of reward learning agents to generalize. The implementation for this project is* [*available*](https://github.com/lzil/procedural-generalization) *on GitHub.*

*This first post will provide a background on reinforcement learning, reward learning, and generalization, as well as summarize the main aims and inspirations for our project. If you have the requisite technical background, feel free to skip the first couple sections.*

About Us
========

We are a team that participated in the 2020 [AI Safety Camp](https://aisafety.camp/) (AISC), a program in which early career researchers collaborate on research proposals related to AI safety. In short, AI safety is a field that aims to ensure that as AI continues to develop, it does not harm humanity.

Given our team’s mutual interests in technical AI safety and reinforcement learning, we were excited to work together on this project. The idea was originally suggested by Sam Clarke, another AISC participant with whom we had fruitful conversations over the course of the camp.

Reinforcement Learning
======================

In reinforcement learning (RL), an agent interacts with an environment with the goal of earning rewards. **Ultimately, the agent wants to learn a strategy in order to maximize the rewards it obtains over time.** First things first, though: what exactly is an agent, and what are rewards? An *agent* is a character that interacts with some world, also known as an *environment***,** by taking actions. For instance, an agent could be a character playing a video game, the car in a self-driving car simulation, or a player in a poker game. The rewards are simply numbers that represent the goal of the agent, whether what happens to the agent is preferable or not. For example picking up a coin may give a positive reward, while getting hit by an enemy a negative reward.

In RL, the *state* represents everything about the current situation in the environment. What the agent can actually see, however, is an *observation*. For example, in a poker game, the observation may be the agent’s own cards and the previous actions of the opponent, while the state also includes the cards of the opponent and the sequence of cards in the deck (i.e., things the agent can’t see). In some environments like chess where there is no hidden information, the state and the observation are the same.

Given observations, the agent takes *actions*. After each action, the agent will get feedback from the environment in the form of:

1. **Rewards:** Scalar values, which can be positive, zero, or negative
2. **A new observation:** The result of taking the action from the previous state, which moves the agent to a new state and results in this new observation. (Also, whether or not the new state is “terminal”, meaning whether the current interaction is finished or still in progress. For example, completing a level or getting eaten by an opponent will terminate many games.)

In RL, our goal is to *train* the agent to be really good at a task by using rewards as feedback. Through one of many possible training algorithms, the agent gradually learns a strategy (also known as a *policy*) that defines what action the agent should take in any state to maximize reward. The goal is to maximize reward over an entire *episode*, which is a sequence of states that an agent goes through from the beginning of an interaction to the terminal state.

Hugely successful agents have been trained to superhuman performance in domains such as [Atari](https://medium.com/@jonathan_hui/rl-dqn-deep-q-network-e207751f7ae4) and the game of [Go](https://deepmind.com/blog/article/alphago-zero-starting-scratch).

![]()The reinforcement learning process (Image by Authors)Let’s look at how a sample algorithm might work, using the video game Mario as an example. Let’s say that Mario has an enemy to his right, a mushroom to his left, and nothing above (see figure below). With those three action options, he might get a reward of +2 if he goes left, -10 if he goes right, and 0 if he goes up. After Mario takes an action, he will be in a new state with a new observation, and will have earned a reward based on his action. Then it will be time for another action, and the process goes on. Recall that the intention is to maximize rewards for an entire episode, which in this context is the series of states from the beginning of the game for the duration of Mario’s life.

![]()Mario learning to eat mushrooms (Image by Authors)The first time the algorithm sees this situation, it might select an option randomly since it doesn’t yet understand the consequences of the available actions. As it sees this situation more and more, it will learn from experience that in situations like these, going right is bad, going up is ok, and going left is best. We wouldn’t directly teach a dog how to fetch a ball, but by giving treats (rewards) for doing so, the dog would learn by reinforcement. Similarly, Mario’s actions are reinforced by feedback from experience that mushrooms are good and enemies are not.

How does the algorithm work to maximize the rewards? Different RL algorithms work in different ways, but one might keep track of the results of taking each action from this position, and the next time Mario is in this same position, he would select the action expected to be the most rewarding according to the prior results. Many algorithms select the best action most of the time, but also sometimes select randomly to make sure that they are exploring all of the options. (Note that at the beginning, the agent usually acts randomly because it hasn’t yet learned anything about the environment.)

It’s important to keep exploring all of the options to make sure that the agent doesn’t find something decent and then stick with it forever, possibly ignoring much better alternatives. In the Mario game, if Mario first tried going right and saw it was -10 and then tried up and saw it was 0, it wouldn’t be great to always go up from that point on. It would be missing out on the +2 reward for going left that hadn’t been explored yet.

Imagine that you tried cooking at home and didn’t like the food, and then went to McDonald’s and had a fantastic meal. You found a good “action” of going to McDonald’s, but it would be a shame (and not great health-wise) if you kept eating at McDonald’s forever and didn’t try other restaurants that may end up giving better “rewards”.

Generalization
==============

RL is often used in game settings like [Atari](https://gym.openai.com/envs/#atari). One problem with using RL in Atari games (which are similar to Mario-style games) is the *sequential* nature of these games. After winning one level, you advance to the next level, and keep going through levels in the same order. **The algorithms may simply memorize exactly what happens in each level and then fail miserably when facing the slightest change in the game.** This means that the algorithms may not actually be understanding the game, but instead learning to memorize a sequence of button presses that leads to high rewards for particular levels. A better algorithm, instead of learning to memorize a sequence of button presses, is able to “understand” the structure of the game and thus is able to adapt to unseen situations, or *generalize*.

Successful generalization means performing well in situations that haven’t been seen before. If you learned that 2\*2 = 4, 2\*3 = 6, and 2\*4 = 8, and then could figure out that 2\*6 = 12, that means that you were able to “understand” the multiplication and not just memorize the equations.

![]()Atari Breakout ([source](/atari-reinforcement-learning-in-depth-part-1-ddqn-ceaa762a546f))Let’s look at a generalization example in the context of an email spam filter. These usually work by collecting data from users who mark emails in their inbox as spam. If a bunch of people marked the message “EARN $800/DAY WITH THIS ONE TRICK” as spam, then the algorithm would learn to block all of those messages for all email users in the future. But what if the spammer noticed his emails were being blocked and decided to outsmart the filter? The next day he might send a new message, “EARN $900/DAY WITH THIS ONE OTHER TRICK”. An algorithm that is only memorizing would fail to catch this because it was just memorizing exact messages to block, rather than learning about spam in general. A generalizing algorithm would learn patterns and effectively understand what constitutes a piece of spam mail.

Reward Learning
===============

Games generally have very well-defined rewards built into them. In a card game like [Blackjack](https://gym.openai.com/envs/Blackjack-v0/), rewards correspond to how much the agent wins or loses each hand. InAtari, rewards are game dependent, but are well-specified, such as earning points for defeating enemies or finishing levels and losing points for getting hit or dying.

The image below is from a classic reinforcement learning environment called [CartPole](https://gym.openai.com/envs/CartPole-v0/), where the goal is to keep a pole upright on a track, and where a reward of +1 is provided for every second that the pole stays upright. The agent moves the cart left or right to try to keep the pole balanced and the longer it can keep it balanced, the more +1 rewards it receives.

![]()CartPole ([source](https://medium.com/@tuzzer/cart-pole-balancing-with-q-learning-b54c6068d947))**However, many tasks in the real world do not have such clearly defined rewards, which leads to limitations in the possible applications of reinforcement learning.** This problem is compounded by the fact that even attempting to specify clearly defined rewards is often difficult, if not impossible. A human could provide direct feedback to an RL agent during training, but this would require too much human time.

One approach called inverse reinforcement learning involves “reverse engineering” a reward function from demonstrations. For complex tasks, figuring out the reward function from demonstrations is very difficult to do well.

***Reward learning* involves learning a reward function, which describes how many rewards are earned in each situation in the environment, i.e. a mapping of the current state and action to the rewards received.** The goal is to learn a reward function that encourages the desired behavior. To train the algorithm to learn a reward function, we need another source of data such as demonstrations of successfully performing the task. The reward function outputs reward predictions for each state, after which standard RL algorithms can be used to learn a strategy by simply substituting these approximate rewards in place of the usually known rewards.

![]()The reinforcement learning process with a reward function in place of known rewards (Image by Authors)Prior work (described below as Christiano et al. 2017) provides an example that illuminates how difficult learning the reward function can be. Imagine teaching a robot to do a backflip. If you aren’t a serious gymnast, it would be challenging to give a demonstration of successfully performing the task yourself. **One could attempt to design a reward function that an agent could learn from, but this approach often falls victim to non-ideal reward design and reward hacking.** Reward hacking means that the agent can find a “loophole” in the reward specification. For example, if we assigned too much reward for getting in the proper initial position for the backflip, then maybe the agent would learn to repeatedly move into that bent over position forever. It would be maximizing rewards based on the reward function that we gave it, but wouldn’t actually be doing what we intended!

A human could supervise every step of an agent’s learning by manually giving input on the reward function at each step, but this would be excessively time consuming and tedious.

The difficulty in specifying the rewards points towards the larger issue of human-AI alignment whereby humans want to align AI systems to their intentions and values, but specifying what we actually want can be surprisingly difficult (recall how every single genie story ends!).

Relevant Papers
===============

We’d like to look at several recent reward learning algorithms to evaluate their ability to learn rewards. We are specifically interested in how successful the algorithms are when faced with previously unseen environments or game levels, which tests their ability to generalize.

To do this, we leverage a body of prior work:

1. [*Deep reinforcement learning from human preferences*](https://arxiv.org/abs/1706.03741) — 2017 by Christiano et al.
2. [*Reward learning from human preferences and demonstrations in Atari*](https://arxiv.org/abs/1811.06521) — 2018 by Ibarz et al.
3. [*Leveraging Procedural Generation to Benchmark Reinforcement Learning*](https://arxiv.org/abs/1912.01588) — 2019 by Cobbe et al.
4. [*Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations*](https://arxiv.org/abs/1904.06387) — 2019 by Brown, Goo et al.

The first two papers were impactful in utilizing reward learning alongside deep reinforcement learning, and the third introduces the OpenAI Procgen Benchmark, a useful set of games for testing algorithm generalization. The fourth paper proposed an efficient alternative to the methods of the first two works.

Deep reinforcement learning from human preferences (Christiano et al. 2017)
---------------------------------------------------------------------------

The key idea of this paper is that **it’s a lot easier to recognize a good backflip than to perform one.** The paper shows that it is possible to learn a predicted reward function for tasks in which we can only recognize a desired behavior, even if we can’t demonstrate it.

The proposed algorithm is shown below. It alternates between learning the reward function through human preferences and learning the strategy, which are both initially random.

*Repeat until the agent is awesome:*

> *1. Show two short video clips of the agent acting in the environment with its current strategy*
>
> *2. Ask a human in which video clip the agent was better*
>
> *3. Update the reward function given the human’s feedback*
>
> *4. Update the strategy based on the new reward function*
>
>

The simulated robot (shown in the below figure) was trained to perform a backflip from 900 queries in under an hour, a task that would be very difficult to demonstrate or to manually create rewards for.

![]()Training a backflip from human preferences ([source](https://github.com/nottombrown/rl-teacher))Experiments were performed in the physics simulator called MuJoCo and also in Atari games. Why run these experiments in Atari when we already know the true rewards in Atari for the games? This gives the opportunity to assign preferences automatically instead of having a human manually give feedback about two video clip demonstrations. We can get automatic (synthetic) feedback by simply ranking the clip with higher true reward as the better one. This enables us to run experiments very quickly because no human effort is needed. Furthermore in this case we can assess the performance of the algorithm by comparing the learned reward function to the true rewards given in the game.

![]()Backflip in motion ([source](https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/))Reward learning from human preferences and demonstrations in Atari (Ibarz et al. 2018)
--------------------------------------------------------------------------------------

This paper built on the prior paper by performing additional experiments in the Atari domain with a different setup and a different RL algorithm. Their main innovation is to utilize human demonstrations at the beginning in order to start with a decent strategy, whereas the original algorithm would have to start with an agent acting completely randomly since no rewards are known at the beginning. The addition of these human demonstrations improved learning significantly in three of nine tested Atari games relative to the no-demos method used by Christiano.

Leveraging Procedural Generation to Benchmark Reinforcement Learning (Cobbe et al. 2019)
----------------------------------------------------------------------------------------

[OpenAI](https://openai.com/), an AI research lab, developed reinforcement learning testbed game environments called the [Procgen Benchmark](https://openai.com/blog/procgen-benchmark/), which includes 16 unique games. **Within each game, all levels are similar and share the same goal, but the actual components like the locations of enemies and hazards are randomly generated** and therefore can be different in each level.

This means that we can **train our agent on many random levels and then test it on completely new levels**, allowing us to understand whether the agent is able to generalize its learning. Note the contrast to Atari games in which training is done on sequential game levels where the enemies and rewards and game objects are always in the same places. Furthermore, when testing the agent’s abilities in sequential and non-randomly generated games, they are tested on those same levels in the same order. An important machine learning principle is to train with one set of data and test with another set to truly evaluate the agent’s ability to learn/generalize.

We looked primarily at four environments from Procgen in our work:

1. **CoinRun:** Collect the coin at the end of the level while dodging enemies
2. **FruitBot:** Eat fruit and avoid non-fruit foods like eggs and ice cream
3. **StarPilot:** Side scrolling shooter game
4. **BigFish:** Begin as a small fish and eat other strictly smaller fish to get bigger

Below are screenshots from each of the games. The agent view uses a lower resolution to optimize for the algorithm to require less computation. The human view is how the game would look if a human were playing.

![]()CoinRun, FruitBot, StarPilot, and BigFish with agent view (Image by Authors)![]()CoinRun, FruitBot, StarPilot, and BigFish with human view (Image by Authors)The main experiments in the paper involved training agents in all 16 unique games over a range of 100 to 100,000 training levels each, while keeping the training time fixed. These agents were then tested on levels that they had never played before (this is possible because each level is uniquely auto-generated). They found that agents need to be trained on as many as 10,000 levels of the game (training levels) before they are able to demonstrate good performance on the test levels.

The StarPilot game plot below shows the training performance in blue and the testing performance in red. The y-axis is the rewards and the x-axis is the number of levels used for training. Note that the x-axis is on a logarithmic scale.

![]()StarPilot training (blue) and testing (red) ([source](https://openai.com/blog/procgen-benchmark/))We see that the agent does very well immediately during training and that training performance then goes down and then back up slightly. Why would the agent get worse as it trains more? Since the training time is fixed, by training on only 100 levels, the agent would be repeating the same levels over and over and could easily memorize everything (but do very poorly at test time in the unseen levels). With 1,000 levels, the agent would have to spread its time over more levels and therefore wouldn’t be able to learn the levels as well. As we get to 10,000 and more levels, the agent is able to see such a diversity of levels, that it can perform well as it has begun to generalize its understanding. We also see that the test performance quickly improves to nearly the level of the training performance, suggesting that the agent is able to generalize well to unseen levels.

Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations (Brown, Goo et al. 2019)
----------------------------------------------------------------------------------------------------------------------------

The algorithm proposed in this paper, called *T-REX*, is different from the previously mentioned reward learning methods in that it **doesn’t require ongoing human feedback during the learning procedure**. While the other algorithms require relatively little human time compared to supervising every agent action, they still require a person to answer thousands of preference queries. A key idea of T-REX is that the human time commitment can be significantly reduced by completing all preference feedback at the beginning, rather than continuously throughout the learning procedure.

The first step is to generate demonstrations of the game or task that is being learned. The demonstrations can either come from a standard reinforcement learning agent or from a human.

The main idea is that we can get a lot of preference data by extracting short video clips from these demonstrations and assigning preferences to them **based only on a ranking of the demonstrations that they came from**. For example, with 20 demonstrations, each demo would get a rank of 1 through 20. A large number of short video clips would be taken from each of these demonstrations and each clip would be assigned the ranking of the demo that it came from, so when they face each other, the preference would go to the clip that came from the better demo. The reward function would then be based on these preferences.

This is in contrast to the approach of the prior works that require human preference input over each and every pair of 1–2 second clips. Here, we only require human preference input to rank the initial demonstrations. T-REX’s disadvantage, however, is that it is using an approximation. Not all clips from a higher ranked demonstration should be preferred to clips from a lower ranked demonstration, but the idea is that on average, the preferences would work well and the procedure would suffice to learn a reward model.

Providing a ranking over demonstrations is the same as giving preferences between every pair of them. For example, if we had three demos and ranked them 3>1>2, this means that we would generate the rankings of 3>1, 3>2, and 1>2. Then randomly generated clips from the demos would be given the same preference ranking based on which demos the clips came from.

The T-REX paper showed that having just 12 demonstrations was sufficient to learn a useful reward model. There are 12 \* 11 / 2 = 66 distinct pairs for any 12 objects, so ranking 12 demonstrations from 1 to 12 is equivalent to answering up to 66 queries about which demo is better, which is ~100 times fewer queries than required by the algorithm by Christiano et al. Again, although the T-REX ranked demonstrations method is more efficient, it is sacrificing precision due to the simplifying assumption that all clips from a better demo are better than all clips from a worse demo.

Brown and Goo et al.’s Atari-based experiments showed that T-REX was competitive against the Ibarz et al. method that was previously described. It was able to learn better-than-demonstrator quality agents using only 12 demonstrations and their corresponding preference (rank) labels.

The figure below shows a comparison between the scores from human demonstrations and the scores from the T-REX algorithm in five Atari games. T-REX was able to soundly outperform 3 of the 5 human scores, though was not able to earn any points in the Montezuma’s Revenge game.

![]()T-REX algorithm vs. humans ([source](https://arxiv.org/abs/1904.06387))T-REX also exceeded the performance a state-of-the-art behavioral cloning algorithm (BCO) and imitation learning algorithm (GAIL) in 7 out of 8 games, as shown in the chart below, while also beating the best available demonstrations in 7 out of 8 games. (Behavioral cloning algorithms try to act as close to demonstrations as possible and inverse reinforcement learning algorithms attempt to recover a reward function from an expert demonstration.)

![]()T-REX algorithm vs. other state-of-the-art methods ([source](https://arxiv.org/abs/1904.06387))Next: Implementations and Experiments
=====================================

Based on T-REX’s strong results and simple idea, we decided to base our initial experiments on combining this algorithm with the Procgen game environments, which would give us a highly efficient reward learning algorithm and a variety of benchmark games to test generalization. We will explain the details of our implementation and the experimental results and issues that we faced in the [second blog post](https://chisness.medium.com/assessing-generalization-in-reward-learning-implementations-and-experiments-de02e1d08c0e) of this series.

Assessing Generalization in Reward Learning: Intro and Background

Max Chiswick
Anton Makiievskyi
Liang Zhou


[epistemic status: that’s just my opinion, man. I have highly suggestive evidence, not deductive proof, for a belief I sincerely hold]

*“If you see fraud and do not say fraud, you are a fraud.”* — [Nasim Taleb](https://en.wikiquote.org/wiki/Nassim_Nicholas_Taleb)

I was talking with a colleague the other day about an AI organization that claims:

1. AGI is probably coming in the next 20 years.
2. Many of the reasons we have for believing this are secret.
3. They’re secret because if we told people about those reasons, they’d learn things that would let them make an AGI even sooner than they would otherwise.

His response was (paraphrasing): “Wow, that’s a really good lie! A lie that can’t be disproven.”

I found this response refreshing, because he *immediately* jumped to the most likely conclusion.

Near predictions generate more funding
--------------------------------------

Generally, entrepreneurs who are optimistic about their project get more funding than ones who aren’t. AI is no exception. For a recent example, see the [Human Brain Project](https://en.wikipedia.org/wiki/Blue_Brain_Project). The founder, Henry Makram, predicted in 2009 that the project would succeed in simulating a human brain by 2019, and the project [was already widely considered a failure by 2013](https://www.scientificamerican.com/article/why-the-human-brain-project-went-wrong-and-how-to-fix-it/). (See [his TED talk](https://www.ted.com/talks/henry_markram_supercomputing_the_brain_s_secrets#t-856603), at 14:22)

The Human Brain project got [1.3 *billion* Euros](https://www.popsci.com/science/article/2013-02/how-simulate-human-brain-one-neuron-time-13-billion/) of funding from the EU.

It’s not hard to see why this is. To justify receiving large amounts of money, the leader must make a claim that the project is actually worth that much. And, AI projects are more impactful if it is, in fact, possible to develop AI soon. So, there is an economic pressure towards inflating estimates of the chance AI will be developed soon.

Fear of an AI gap
-----------------

The [missile gap](https://en.wikipedia.org/wiki/Missile_gap) was a lie by the US Air Force to justify building more nukes, by falsely claiming that the Soviet Union had more nukes than the US.

Similarly, there’s historical precedent for an AI gap lie used to justify more AI development. [Fifth Generation Computer Systems](https://en.wikipedia.org/wiki/Fifth_generation_computer) was an ambitious 1982 project by the Japanese government (funded for $400 million in 1992, or $730 million in 2019 dollars) to create artificial intelligence through massively parallel logic programming.

The project is widely considered to have failed.  From a [1992 New York Times article](http://www.nytimes.com/1992/06/05/business/fifth-generation-became-japan-s-lost-generation.html):

> A bold 10-year effort by Japan to seize the lead in computer technology is fizzling to a close, having failed to meet many of its ambitious goals or to produce technology that Japan’s computer industry wanted.
>
> …
>
> That attitude is a sharp contrast to the project’s inception, when it spread fear in the United States that the Japanese were going to leapfrog the American computer industry. In response, a group of American companies formed the Microelectronics and Computer Technology Corporation, a consortium in Austin, Tex., to cooperate on research. And the Defense Department, in part to meet the Japanese challenge, began a huge long-term program to develop intelligent systems, including tanks that could navigate on their own.
>
> …
>
> **The Fifth Generation effort did not yield the breakthroughs to make machines truly intelligent, something that probably could never have realistically been expected anyway.** Yet the project did succeed in developing prototype computers that can perform some reasoning functions at high speeds, in part by employing up to 1,000 processors in parallel. The project also developed basic software to control and program such computers. Experts here said that some of these achievements were technically impressive.
>
> …
>
> In his opening speech at the conference here, Kazuhiro Fuchi, the director of the Fifth Generation project, made an impassioned defense of his program.
>
> “Ten years ago we faced criticism of being too reckless,” in setting too many ambitious goals, he said, adding, “Now we see criticism from inside and outside the country because we have failed to achieve such grand goals.”
>
> Outsiders, he said, initially exaggerated the aims of the project, with the result that the program now seems to have fallen short of its goals.
>
> **Some American computer scientists say privately that some of their colleagues did perhaps overstate the scope and threat of the Fifth Generation project. Why? In order to coax more support from the United States Government for computer science research.**
>
>

(emphasis mine)

This bears similarity to some conversations on AI risk I’ve been party to in the past few years. The fear is that Others (DeepMind, China, whoever) will develop AGI soon, so We have to develop AGI first in order to make sure it’s safe, because Others won’t make sure it’s safe and We will. Also, We have to discuss AGI strategy in private (and avoid public discussion), so Others don’t get the wrong ideas. (Generally, these claims have little empirical/rational backing to them; they’re based on scary stories, not historically validated threat models)

The claim that others will develop weapons and kill us with them by default implies a moral claim to resources, and a moral claim to be justified in making weapons in response. Such claims, if exaggerated, justify claiming more resources and making more weapons. And they weaken a community’s actual ability to track and respond to real threats (as in The Boy Who Cried Wolf).

How does the AI field treat its critics?
----------------------------------------

Hubert Dreyfus, probably the most famous historical AI critic, published [“Alchemy and Artificial Intelligence”](https://www.rand.org/content/dam/rand/pubs/papers/2006/P3244.pdf) in 1965, which argued that the techniques popular at the time were insufficient for AGI. Subsequently, he was [shunned by other AI researchers](https://en.wikipedia.org/wiki/History_of_artificial_intelligence#Critiques_from_across_campus):

> The paper “caused an uproar”, according to Pamela McCorduck.  The AI community’s response was derisive and personal.  Seymour Papert dismissed one third of the paper as “gossip” and claimed that every quotation was deliberately taken out of context.  Herbert A. Simon accused Dreyfus of playing “politics” so that he could attach the prestigious RAND name to his ideas. Simon said, “what I resent about this was the RAND name attached to that garbage.”
>
> Dreyfus, who taught at MIT, remembers that his colleagues working in AI “dared not be seen having lunch with me.”  Joseph Weizenbaum, the author of ELIZA, felt his colleagues’ treatment of Dreyfus was unprofessional and childish.  Although he was an outspoken critic of Dreyfus’ positions, he recalls “I became the only member of the AI community to be seen eating lunch with Dreyfus. And I deliberately made it plain that theirs was not the way to treat a human being.”
>
>

This makes sense as anti-whistleblower activity: ostracizing, discrediting, or punishing people who break the conspiracy to the public. Does this still happen in the AI field today?

[Gary Marcus](https://en.wikipedia.org/wiki/Gary_Marcus) is a more recent AI researcher and critic. In 2012, [he wrote](https://medium.com/@GaryMarcus/the-deepest-problem-with-deep-learning-91c5991f5695):

> Deep learning is important work, with immediate practical applications.
>
> …
>
> Realistically, deep learning is only part of the larger challenge of building intelligent machines. Such techniques lack ways of representing causal relationships (such as between diseases and their symptoms), and are likely to face challenges in acquiring abstract ideas like “sibling” or “identical to.” They have no obvious ways of performing logical inferences, and they are also still a long way from integrating abstract knowledge, such as information about what objects are, what they are for, and how they are typically used. The most powerful A.I. systems … use techniques like deep learning as just one element in a very complicated ensemble of techniques, ranging from the statistical technique of Bayesian inference to deductive reasoning.
>
>

In 2018, he [tweeted](https://twitter.com/GaryMarcus/status/1065280340669816832?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed&ref_url=https%3A%2F%2Fcdn.embedly.com%2Fwidgets%2Fmedia.html%3Ftype%3Dtext%252Fhtml%26key%3Da19fcc184b9711e1b4764040d3dc5c07%26schema%3Dtwitter%26url%3Dhttps%253A%2F%2Ftwitter.com%2Fgarymarcus%2Fstatus%2F1065280340669816832%26image%3Dhttps%253A%2F%2Fpbs.twimg.com%2Fprofile_images%2F850347800789372929%2FVg_HrEun_400x400.jpg) an article in which Yoshua Bengio (a deep learning pioneer) seemed to agree with these previous opinions. This tweet received a number of mostly-critical replies. Here’s one, by AI professor Zachary Lipton:

> There’s a couple problems with this whole line of attack. 1) Saying it louder ≠ saying it first. You can’t claim credit for differentiating between reasoning and pattern recognition. 2) Saying X doesn’t solve Y is pretty easy. But where are your concrete solutions for Y?
>
>

The first criticism is essentially a claim that [everybody knows](https://thezvi.wordpress.com/2019/07/02/everybody-knows/) that deep learning can’t do reasoning. But, this is essentially admitting that Marcus is correct, while still criticizing him for saying it [ED NOTE: the phrasing of this sentence is off (Lipton publicly agrees with Marcus on this point), and there is more context, see [Lipton’s reply](https://www.greaterwrong.com/posts/KnQs55tjxWopCzKsk/the-ai-timelines-scam/comment/nGS8nPe8rA4Ay4uFN)].

The second is a claim that Marcus shouldn’t criticize if he doesn’t have a solution in hand. This policy deterministically results in the short AI timelines narrative being maintained: to criticize the current narrative, you must present your own solution, which constitutes another narrative for why AI might come soon.

Deep learning pioneer Yann LeCun’s response is similar:

> Yoshua (and I, and others) have been saying this for a long time.
> The difference with you is that we are actually trying to do something about it, not criticize people who don’t.
>
>

Again, the criticism is not that Marcus is wrong in saying deep learning can’t do certain forms of reasoning, the criticism is that he isn’t presenting an alternative solution. (Of course, the claim could be correct even if Marcus doesn’t have an alternative!)

Apparently, it’s considered *bad practice* in AI to criticize a proposal for making AGI without presenting on alternative solution. Clearly, such a policy causes large distortions!

Here’s another response, by Steven Hansen (a research scientist at DeepMind):

> Ideally, you’d be saying this through NeurIPS submissions rather than New Yorker articles. A lot of the push-back you’re getting right now is due to the perception that you haven’t been using the appropriate channels to influence the field.
>
>

That is: to criticize the field, you should go through the field, not through the press. This is standard guild behavior. In the words of [Adam Smith](https://www.goodreads.com/quotes/420123-people-of-the-same-trade-seldom-meet-together-even-for): “People of the same trade seldom meet together, even for merriment and diversion, but the conversation ends in a conspiracy against the public, or in some contrivance to raise prices.”

(Also see Marcus’s [medium article](https://medium.com/@GaryMarcus/the-deepest-problem-with-deep-learning-91c5991f5695) on the Twitter thread, and on the limitations of deep learning)

[ED NOTE: I’m not saying these critics on Twitter are publicly promoting short AI timelines narratives (in fact, some are promoting the opposite), I’m saying that the norms by which they criticize Marcus result in short AI timelines narratives being maintained.]

Why model sociopolitical dynamics?
----------------------------------

This post has focused on sociopolotical phenomena involved in the short AI timelines phenomenon. For this, I anticipate criticism along the lines of “why not just model the technical arguments, rather than the credibility of the people involved?” To which I pre-emptively reply:

* No one can model the technical arguments in isolation. Basic facts, such as the accuracy of technical papers on AI, or the filtering processes determining what you read and what you don’t, depend on sociopolitical phenomena. This is far more true for people who don’t themselves have AI expertise.
* “When AGI will be developed” isn’t just a technical question. It depends on what people actually choose to do (and what groups of people actually succeed in accomplishing), not just what can be done in theory. And so basic questions like “how good is the epistemology of the AI field about AI timelines?” matter directly.
* The sociopolitical phenomena are actively making technical discussion harder. I’ve had a well-reputed person in the AI risk space discourage me from writing publicly about the technical arguments, on the basis that getting people to think through them might accelerate AI timelines (yes, really).

Which is not to say that modeling such technical arguments is not important for forecasting AGI. I certainly could have written a post evaluating such arguments, and I decided to write this post instead, in part because I don’t have much to say on this issue that Gary Marcus hasn’t [already said](https://arxiv.org/abs/1801.00631). (Of course, I’d have written a substantially different post, or none at all, if I believed the technical arguments that AGI is likely to come soon had merit to them)

What I’m not saying
-------------------

I’m not saying:

1. That deep learning isn’t a major AI advance.
2. That deep learning won’t substantially change the world in the next 20 years (through narrow AI).
3. That I’m certain that AGI isn’t coming in the next 20 years.
4. That AGI isn’t existentially important on long timescales.
5. That it isn’t possible that some AI researchers have asymmetric information indicating that AGI is coming in the next 20 years. (Unlikely, but possible)
6. That people who have technical expertise shouldn’t be evaluating technical arguments on their merits.
7. That most of what’s going on is people consciously lying. (Rather, covert deception hidden from conscious attention (e.g. motivated reasoning) is pervasive; see [The Elephant in the Brain](https://www.amazon.com/dp/B077GZT9Q1/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1))
8. That many people aren’t sincerely confused on the issue.

I’m saying that there are systematic sociopolitical phenomena that cause distortions in AI estimates, especially towards shorter timelines. I’m saying that people are being duped into believing a lie. And at the point where [73% of tech executives say they believe AGI will be developed in the next 10 years](https://twitter.com/psych_of_tech/status/1106302905555042305), it’s a major one.

*This has happened before.* And, in all likelihood, *this will happen again*.

The AI Timelines Scam


This is an account of free choice in a physical universe. It is very much relevant to decision theory and philosophy of science. It is largely metaphysical, in terms of taking certain things to be basically real and examining what can be defined in terms of these things.

The starting point of this account is critical and agential. By agential, I mean that the ontology I am using is from the point of view of an agent: a perspective that can, at the very least, receive observations, have cognitions, and take actions. By critical, I mean that this ontology involves uncertain conjectures subject to criticism, such as criticism of being logically incoherent or incompatible with observations. This is very much in a similar spirit to [critical rationalism](https://en.wikipedia.org/wiki/Critical_rationalism).

Close attention will be paid to falsifiability and refutation, principally for *ontological* purposes, and secondarily for epistemic purposes. Falsification conditions specify the meanings of laws and entities relative to the perspective of some potentially falsifying agent. While the agent may believe in unfalsifiable entities, falsification conditions will serve to precisely pin down that which can be precisely pinned down.

I have only seen “agential” used in the philosophical literature in the context of [agential realism](https://en.wikipedia.org/wiki/Agential_realism), a view I do not understand well enough to comment on. I was tempted to use “subjective”; however, while subjects have observations, they do not necessarily have the ability to take actions. Thus I believe “agential” has a more concordant denotation.

You’ll note that my notion of “agent” already assumes one can take actions. Thus, a kind of free will is taken as metaphysically basic. This presupposition may cause problems later. However, I will try to show that, if careful attention is paid, the obvious problems (such as contradiction with determinism) can be avoided.

The perspective in this post can be seen as starting from agency, defining consequences in terms of agency, and defining physics in terms of consequences. In contrast, the most salient competing decision theory views (including [framings](https://plato.stanford.edu/entries/decision-causal/) of CDT, EDT, and FDT) define agency in terms of consequences (“expected utility maximization”), and consequences in terms of physics (“counterfactuals”). So I am rebasing the ontological stack, turning it upside-down. This is less absurd than it first appears, as will become clear.

(For simplicity, assume observations and actions are both symbols taken from some finite alphabet.)

Naive determinism
-----------------

Let’s first, within a critical agential ontology, disprove some very basic forms of determinism.

Let A be some action. Consider the statement: “I will take action A”. An agent believing this statement may falsify it by taking any action B not equal to A. Therefore, this statement does not hold as a law. It may be falsified at will.

Let *f*() be some computable function returning an action. Consider the statement: “I will take action *f*()”. An agent believing this statement may falsify it by taking an action B not equal to *f*(). Note that, since the agent is assumed to be able to compute things, *f*() may be determined. So, indeed, this statement does not hold as a law, either.

This contradicts a certain strong formulation of naive determinism: the idea that one’s action is necessarily determined by some known, computable function.

Action-consequences
-------------------

But wait, what about physics? To evaluate what physical determinism even means, we need to translate physics into a critical agential ontology. However, before we turn to physics, we will first consider action-consequences, which are easier to reason about.

Consider the statement: “If I take action A, I will immediately there-after observe O.” This statement is falsifiable, which means that if it is false, there is some policy the agent can adopt that will falsify it. Specifically, the agent may adopt the policy of taking action A. If the agent will, in fact, not observe O after taking this action, then the agent will learn this, falsifying the statement. So the statement is falsifiable.

Finite conjunctions of falsifiable statements are themselves falsifiable. Therefore, the conjunction “If I take action A, I will immediately there-after observe O; if I take action B, I will immediately there-after observe P” is, likewise, falsifiable.

Thus, the agent may have falsifiable beliefs about observable consequences of actions. This is a possible starting point for decision theory: actions having consequences is already assumed in the ontology of VNM utility theory.

Falsification and causation
---------------------------

Now, the next step is to account for physics. Luckily, the falsificationist paradigm was designed around demarcating scientific hypotheses, such that it naturally describes physics.

Interestingly, falsificationism takes agency (in terms of observations, computation, and action) as more basic than physics. For a thing to be falsifiable, it must be *able* to be falsified by some agent, seeing some observation. And the word *able* implies freedom.

Let’s start with some basic [Popperian logic](https://en.wikipedia.org/wiki/The_Logic_of_Scientific_Discovery). Let *f* be some testable function (say, connected to a computer terminal) taking in a natural number and returning a Boolean. Consider the hypothesis: “For all *x*, *f*(*x*) is true”. This statement is falsifiable: if it’s false, then there exists some action-sequence an agent can take (typing *x* into the terminal, one digit at a time) that will prove it to be false.

The given hypothesis is a kind of scientific law. It specifies a regularity in the environment.

Note that there is a “bridge condition” at play here. That bridge condition is that the function *f* is, indeed, connected to the terminal, such that the agent’s observations of *f* are trustworthy. In a sense, the bridge condition specifies what *f* is, from the agent’s perspective; it allows the agent to locate *f* as opposed to some other function.

Let us now consider causal hypotheses. We already considered action-consequences. Now let us extend this analysis to reasoning about causation between external entities.

Consider the hypothesis: “If the match is struck, then it will alight immediately”. This hypothesis is falsifiable by an agent who is *able* to strike the match. If the hypothesis is false, then the agent may refute it by choosing to strike the match and then seeing the result. However, an agent who is unable to strike the match cannot falsify it. (Of course, this assumes the agent may see whether the match is alight after striking it)

Thus, we are defining causality in terms of agency. The falsification conditions for a causal hypothesis refer to the agent’s abilities. This seems somewhat wonky at first, but it is quite similar to [Pearlian casuality](https://en.wikipedia.org/wiki/Causality_(book)), which defines causation in terms of metaphysically-real interventions. This order of definition radically reframes the determinism vs. free will apparent paradox, by defining the conditions of determinism (causality) in terms of potential action.

External physics
----------------

Let us now continue, proceeding to more universal physics. Consider the law of gravity, according to which a dropped object will accelerate downward at a near-constant weight. How might we port this law into an agential ontology?

Here is the assumption about how the agent interacts with gravity. The agent will choose some natural number as the height of an object. Thereafter, the object will fall, while a camera will record the height of the object at each natural-number time expressed in milliseconds, to the nearest natural-number millimeter from the ground. The agent may observe a printout of the camera data afterwards.

Logically, constant gravity implies, and is implied by, a particular quadratic formula for the height of the object as a function of the object’s starting height and the amount of time that has passed. This formula implies the content of the printout, as a function of the chosen height. So, the agent may falsify constant gravity (*in the observable domain*) by choosing an object-height, placing an object at that height, letting it fall, and checking the printout, which will show the law of constant gravity to be false, if the law in fact does not hold for objects dropped at that height (to the observed level of precision).

Universal constant gravity is not similarly falsifiable by this agent, because this agent may only observe this given experimental setup. However, a domain-limited law, stating that the law of constant gravity holds for all possible object-heights in this setup, up to the camera’s precision, is falsifiable.

It may seem that I am being incredibly pedantic about what a physical law is and what the falsification conditions are; however, I believe this level of pedantry is necessary for critically examining the notion of physical determinism to a high-enough level of rigor to check interaction with free will.

Internal physics
----------------

We have, so far, considered the case of an agent falsifying a physical law that applies to an external object. To check interaction with free will, we must interpret physical law applied to the agent’s internals, on which the agent’s cognition is, perhaps, running in a manner similar to software.

Let’s consider the notion that the agent itself is “running on” some Turing machine. We will need to specify precisely what such “running on” means.

Let C be the computer that the agent is considering whether it is running on. C has, at each time, a tape-state, a Turing machine state, an input, and an output. The input is attached to a sensor (such as a camera), and the output is attached to an actuator (such as a motor).

For simplicity, let us say that the history of tapes, states, inputs, and outputs is saved, such that it can be queried at a later time.

We may consider the hypothesis that C, indeed, implements the correct dynamics for a given Turing machine specification. These dynamics imply a relation between future states and past states. An agent may falsify these dynamics by checking the history and seeing if the dynamics hold.

Note that, because some states or tapes may be unreachable, it is not possible to falsify the hypothesis that C implements correct dynamics starting from unreachable states. Rather, only behavior following from reachable states may be checked.

Now, let us think on an agent considering whether they “run on” this computer C. The agent may be assumed to be able to query the history of C, such that it may itself falsify the hypothesis that C implements Turing machine specification M, and other C-related hypotheses as well.

Now, we can already name some ways that “I run on C” may be falsified:

* Perhaps there is a policy I may adopt, and a time *t*, such that if I implement this policy, I will observe O at time *t*, but C will observe something other than O at time *t*.
* Perhaps there is a policy I may adopt, and a time *t*, such that if I implement this policy, I will take action A at time *t*, but C will take an action other than A at time *t*.

The agent may prove these falsification conditions by adopting a given policy until some time *t*, and then observing C’s observation/action at time *t*, compared to their own observation/action.

I do not argue that the converse of these conditions exhaust what it means that “I run on C”. However, they at least restrict the possibility space by a very large amount. For the falsification conditions given to not hold, the observations and behavior of C must be identical with the agent’s own observations and behavior, for all possible policies the agent may adopt.

I will name the hypothesis with the above falsification conditions: “I effectively run on C”. This conveys that these conditions may not be exhaustive, while still being quite specific, and relating to effects between the agent and the environment (observations and actions).

Note that the agent can hypothesize itself to effectively run on multiple computers! The conditions for effectively running on one computer do not contradict the conditions for effectively running on another computer. This naturally handles cases of identical physical instantiations of a single agent.

At this point, we have an account of an agent who:

* Believes they have observations and take free actions
* May falsifiably hypothesize physical law
* May falsifiably hypothesize that some computer implements a Turing machine specification
* May falsifiably hypothesize that they themselves effectively run on some computer

I have not yet shown that this account is consistent. There may be paradoxes. However, this at least represents the subject matter covered in a unified critical agential ontology.

Paradoxes sought and evaluated
------------------------------

Let us now seek out paradox. We showed before that the hypothesis “I take action *f*()” may be refuted at will, and therefore does not hold as a necessary law. We may suspect that “I effectively run on C” runs into similar problems.

#### Self-contradiction

Remember that, for the “I effectively run on C” hypothesis to be falsified, it must be falsified at some time, at which the agent’s observation/action comes apart from C’s. In the “I take action *f*()” case, we had the agent simulate *f*() in order to take an opposite action. However, C need not halt, so the agent cannot simulate C until halting. Instead, the agent may select some time *t*, and run C for *t* steps. But, by the time the agent has simulated C for *t* steps, the time is already past *t*, and so the agent may not contradict C’s behavior at time *t*, by taking an opposite action. Rather, the agent only knows what C does at time *t* at some time later than *t*, and only their behavior after this time may depend on this knowledge.

So, this paradox is avoided by the fact that the agent cannot contradict its own action before knowing it, but cannot know it before taking it.

We may also try to create a paradox by assuming an external super-fast computer runs a copy of C in parallel, and feeds this copy’s action on subjective time-step *t* into the original C’s observation before time *t*; this way, the agent may observe its action before it takes it. However, now the agent’s action is dependent on its observation, and so the external super-fast computer must decide which observation to feed into the parallel C. The external computer cannot know what C will do before producing this observation, and so this attempt at a paradox cannot stand without further elaboration.

We see, now, that if free will and determinism are compatible, it is due to limitations on the agent’s knowledge. The agent, knowing it runs on C, cannot thereby determine what action it takes at time *t*, until a later time. And the initial attempt to provide this knowledge externally fails.

#### Downward causation

Let us now consider a general criticism of functionalist views, which is that of downward causation: if a mental entity (such as observation or action) causes a physical entity, doesn’t that either mean that the mental entity is physical, or that physics is not causally closed?

Recall that we have defined causation in terms of the agent’s action possibilities. It is straightforwardly the case, then, that the agent’s action at time *t* causes changes in the environment.

But, what of the physical cause? Perhaps it is also the case that C’s action at time *t* causes changes in the environment. If so, there is a redundancy, in that the change in the environment is caused both by the agent’s action and by C’s action. We will examine this possible redundancy to find potential conflicts.

To consider ways that C’s action may change the environment, we must consider how the agent may intervene on C’s action. Let us say we are concerned with C’s action at time *t*. Then we may consider the agent at some time *u* < *t* taking an action that will cause C’s action at time *t* to be over-written. For example, the agent may consider programming an external circuit that will interact with C’s circuit (“its circuit”).

However, if the agent performs this intervention, then the agent’s action at time *t* has no influence on C’s action at time *t*. This is because C’s action is, necessarily, equal to the value chosen at time *u*. (Note that this lack of influence means that the agent *does not effectively run on C*, for the notion of “effectively run on” considered! However, the agent may be said to effectively run on C with one exception.)

So, there is no apparent way to set up a contradiction between these interventions. If the agent decides early (at time *u*) to determine C’s action at time *t*, then that decision causes C’s action at time *t*; if the agent does not do so, then the agent’s decision at time *t* causes C’s action at time *t*; and these are mutually exclusive. Hence, there is not an apparent problem with redundant causality.

#### Epiphenomenalism

It may be suspected that the agent I take to be real is epiphenomenal. Perhaps all may be explained in a physicalist ontology, with no need to posit that there exists an agent that has observations and takes actions. (This is a criticism levied at some views on consciousness; my notion of metaphysically-real observations is similar enough to consciousness that these criticisms are potentially applicable)

The question in regards to explanatory power is: what is being explained, in terms of what? My answer is: observations are being explained, in terms of hypotheses that may be falsified by action/observations.

An eliminativist perspective denies the agent’s observations, and thus fails to explain what ought to be explained, in my view. However, eliminativists will typically believe that “scientific observation” is possible, and seek to explain scientific observations.

A relevant point to make here is that the notion of scientific observation assumes there is some scientific process happening that has observations. Indeed, the scientific method includes actions, such as testing, which rely on the scientific process taking actions. Thus, scientific processes may be considered as agents in the sense I am using the term.

My view is that erasing the agency of both individual scientists, and of scientific processes, puts the ontological and epistemic status of physics on shaky ground. It is hard to say why one should believe in physics, except in terms of it explaining observations, including experimental observations that require taking actions. And it is hard to say what it means for a physical hypothesis to be true, with no reference to how the hypothesis connects with observation and action.

In any case, the specter of epiphenomenalism presents no immediate paradox, and I believe that it does not succeed as a criticism.

Comparison to Gary Drescher’s view
----------------------------------

I will now compare my account to Gary Drescher’s view. I have found Drescher’s view to be both particularly systematic and compelling, and to be quite similar to the views of other relevant philosophers such as Daniel Dennett and Eliezer Yudkowsky. Therefore, I will compare and contrast my view with Drescher’s. This will dispel the illusion that I am not saying anything new.

Notably, Drescher makes a similar observation to mine on Pearl: “Pearl’s formalism models free will rather than mechanical choice.”

Quoting section 5.3 of *Good and Real*:

> Why did it take that action? In pursuit of what goal was the action selected? Was that goal achieved? Would the goal have been achieved if the machine had taken this other action instead? The system includes the assertion that if the agent were to do X, then Y would (probably) occur; is that assertion true? The system does not include the assertion that if it were to do P, Q would probably occur; is that omitted assertion true? Would the system have taken some other action just now if it had included that assertion? Would it then have better achieved its goals?
>
> Insofar as such questions are meaningful and answerable, the agent makes choices in at least the sense that the correctness of its actions with respect to its designated goals is analyzable. That is to say, there can be means-end connections between its actions and its goals: its taking an action for the sake of a goal can make sense. And this is so despite the fact that everything that will happen-including every action taken and every goal achieved or not-is inalterably determined once the system starts up. Accordingly, I propose to call such an agent a choice machine.
>
>

Drescher is defining conditions of choice and agency in terms of whether the decisions “make sense” with respect to some goal, in terms of means-end connections. This is a “outside” view of agency in contrast with my “inside” view. That is, it says some thing is an agent when its actions connect with some goal, and when the internal logic of that thing takes into account this connection.

This is in contrast to my view, which takes agency to be metaphysically basic, and defines physical outside views (and indeed, physics itself) in terms of agency.

My view would disagree with Drescher’s on the “inalterably determined” assertion. In an earlier chapter, Drescher describes a deterministic block-universe view. This view-from-nowhere implies that future states are determinable from past states. In contrast, the view I present here rejects views-from-nowhere, instead taking the view of some agent in the universe, from whose perspective the future course is not already determined (as already argued in examinations of paradox).

Note that these disagreements are principally about metaphysics and ontology, rather than scientific predictions. I am unlikely to predict the results of scientific experiments differently from Drescher on account of this view, but am likely to account for the scientific process, causation, choice, and so on in different language, and using a different base model.

Conclusion and further research
-------------------------------

I believe the view I have presented to be superior to competing views on multiple fronts, most especially logical/philosophical systematic coherence. I do not make the full case for this in this post, but take the first step, of explicating the basic ontology and how it accounts for phenomena that are critically necessary to account for.

An obvious next step is to tackle decision theory. Both Bayesianism and VNM decision theory are quite concordant with critical agential ontology, in that they propose coherence conditions on agents, which can be taken as criticisms. Naturalistic decision theory involves reconciling choice with physics, and so a view that already includes both is a promising starting point.

Multi-agent systems are quite important as well. The view presented so far is near-solipsistic, in that there is a single agent who conceptualizes the world. It will need to be defined what it means for there to be “other” agents. Additionally, “aggregative” agents, such as organizations, are important to study, including in terms of what it means for a singular agent to participate in an aggregative agent. “Standardized” agents, such as hypothetical skeptical mathematicians or philosophers, are also worthy subjects of study; these standardized agents are relevant in reasoning about argumentation and common knowledge. Also, while the discussion so far has been in terms of [closed individualism](https://opentheory.net/2018/09/a-new-theory-of-open-individualism/), alternative identity views such as empty individualism and open individualism are worth considering from a critical agential perspective.

Other areas of study include naturalized epistemology and philosophy of mathematics. The view so far is primarily ontological, secondarily epistemological. With the ontology in place, epistemology can be more readily explored.

I hope to explore the consequences of this metaphysics further, in multiple directions. Even if I ultimately abandon it, it will have been useful to develop a coherent view leading to an illuminating refutation.

A critical agential account of free will, causation, and physics