---
layout: chapter
title: "Time inconsistency II"
description: Formal model of time-inconsistent agent, Gridworld and Procrastination examples.

---

## Formal Model and Implementation of Hyperbolic Discounting

### Formal Model of Naive and Sophisticated Hyperbolic Discounters

To formalize Naive and Sophisticated hyperbolic discounting, we make a small modificiation to the MDP agent model. The key idea is to add a variable for measuring time, the *delay*, which is distinct from the objective time-index (called `timeLeft` in our implementation). The objective time-index is crucial to planning for finite-horizon MDPs because it determines how far the agent can travel or explore before time is up. By contrast, the delays are *subjective*. They are used by the agent in *evaluating* possible future rewards but they are not an independent feature of the decision problem.

We use delays because discounting agents have preferences over when they receive rewards. When evaluating future prospects, they need to keep track of how far ahead in time that reward occurs. Naive and Sophisticated agents *evaluate* future rewards in the same way. They differ in how they simulate their future actions.

The Naive agent at objective time $$t$$ assumes his future self at objective time $$t+c$$ (where $$c>0$$) shares his time preference. So he simulates the $$(t+c)$$-agent as evaluating a reward at time $$t+c$$ with delay $$=c$$ rather than the true delay $$=0$$. The Sophisticated agent correctly models his $$(t+c)$$-agent future self as evaluating an immediate reward with delay $$=0$$.

To be more concrete, suppose both Naive and Sophisticated have a discount function $$1/(1+kd)$$, where $$d$$ is how much the reward is delayed. When simulating his future self at $$t+c$$, the Naive agent assumes he'll discount immediate gains at rate $$1/(1+kc)$$ and the Sophisticated agent (correctly) assumes a rate of $$1/(1+0)$$.

Adding delays to our model is straightforward. In defining the MDP agent, we presented Bellman-style recursions for the expected utility of state-action pairs. Discounting agents evaluate states and actions differently depending on their *delay* from the present. So we now define expected utilities of state-action-delay triples:

$$
EU[s,a,d] = \delta(d)U(s, a) + \mathbb{E}_{s', a'}(EU[s', a',d+1])
$$

where:

- $$\delta  \colon \mathbb{N} \to \mathbb{R}$$ is the discount function from the delay to the discount factor. In our examples we have (where $$k>0$$ is the discount constant):

$$
\delta(d) = \frac{1}{1+kd}
$$

- $$s' \sim T(s,a)$$ exactly as in the non-discounting case.

- $$a' \sim C(s'; d_P)$$ where $$d_P=0$$ for Sophisticated and $$d_P=d+1$$ for Naive.

The function $$C \colon S \times \mathbb{N} \to A$$ is again the *act* function. For $$C(s'; d+1)$$ we take a softmax over the expected value of each action $$a$$, namely, $$EU[s',a,d+1]$$. The act function now takes a delay argument. We interpret $$C(s';d+1)$$ as "the softmax action the agent would take in state $$s'$$ given that their rewards occur with a delay $$d+1$$".

The Naive agent simulates his future actions by computing $$C(s';d+1)$$; the Sophisticated agent computes the action that will *actually* occur, which is $$C(s';0)$$. So if we want to simulate an environment including a hyperbolic discounter, we can compute the agent's action with $$C(s;0)$$ for every state $$s$$.

### Implementing the hyperbolic discounter

As with the MDP and POMDP agents, our WebPPL implementation directly translates the mathematical formulation of Naive and Sophisticated hyperbolic discounting. The variable names correspond as follows:

- The function $$\delta$$ is named `discountFunction`

- The "perceived delay", which controls how the agent's simulated future self evaluates rewards, is $$d$$ in the math and `perceivedDelay` below.

- $$s'$$, $$a'$$, $$d+1$$ correspond to `nextState`, `nextAction` and `delay+1` respectively.

This codebox simplifies the code for the hyperbolic discounter by omitting definitions of `transition`, `utility` and so on:

~~~~
var makeAgent = function(params, world) {

  var act = dp.cache(
    function(state, delay){
      return Infer({ model() {
        var action = uniformDraw(stateToActions(state));
        var eu = expectedUtility(state, action, delay);
        factor(params.alpha * eu);
        return action;
      }});
    });

  var expectedUtility = dp.cache(
    function(state, action, delay){
      var u = discountFunction(delay) * utility(state, action);
      if (state.terminateAfterAction){
        return u;
      } else {
        return u + expectation(Infer({ model() {
          var nextState = transition(state, action);
          var perceivedDelay = isNaive ? delay + 1 : 0;
          var nextAction = sample(act(nextState, perceivedDelay));
          return expectedUtility(nextState, nextAction, delay+1);
        }}));
      }
    });

  return { params, expectedUtility, act };
};
~~~~

The next codebox shows how the Naive agent can end up at Donut North in the Restaurant Choice problem, despite this being dominated for any possible utility function. The Naive agent first moves in the direction of Veg, which initially looks better than Donut South. When right outside Donut North, discounting makes it look better than Veg. To visualize this, we display the agent's expected utility calculations at different steps along its trajectory. The crucial values are the `expectedValue` of going left at [3,5] when `delay=0` compared with `delay=4`. The function `plannedTrajectories` uses `expectedValue` to access these values. For each timestep, we plot the agent's position and the expected utility of each action they might perform in the future.

~~~~
///fold: makeAgent, mdp, plannedTrajectories
var makeAgent = function(params, world) {
  var defaultParams = {
    alpha: 500,
    discount: 1
  };
  var params = extend(defaultParams, params);
  var stateToActions = world.stateToActions;
  var transition = world.transition;
  var utility = params.utility;
  var paramsDiscountFunction = params.discountFunction;

  var discountFunction = (
    paramsDiscountFunction ?
    paramsDiscountFunction :
    function(delay){ return 1/(1 + params.discount*delay); });

  var isNaive = params.sophisticatedOrNaive === 'naive';

  var act = dp.cache(
    function(state, delay) {
      var delay = delay || 0; // make sure delay is never undefined
      return Infer({ model() {
        var action = uniformDraw(stateToActions(state));
        var eu = expectedUtility(state, action, delay);
        factor(params.alpha * eu);
        return action;
      }});
    });

  var expectedUtility = dp.cache(
    function(state, action, delay) {
      var u = discountFunction(delay) * utility(state, action);
      if (state.terminateAfterAction){
        return u;
      } else {
        return u + expectation(Infer({ model() {
          var nextState = transition(state, action);
          var perceivedDelay = isNaive ? delay + 1 : 0;
          var nextAction = sample(act(nextState, perceivedDelay));
          return expectedUtility(nextState, nextAction, delay+1);
        }}));
      }
    });

  return { params, expectedUtility, act };
};

var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});

var MAPActionPath = function(state, world, agent, actualTotalTime, statesOrActions) {
  var perceivedTotalTime = state.timeLeft;
  assert.ok(perceivedTotalTime  > 1 || state.terminateAfterAction==false,
            'perceivedTime<=1. If=1 then should have state.terminateAfterAction,' +
            ' but then simulate wont work ' + JSON.stringify(state));

  var agentAction = agent.act;
  var expectedUtility = agent.expectedUtility;
  var transition = world.transition;

  var sampleSequence = function (state, actualTimeLeft) {
    var action = agentAction(state, actualTotalTime-actualTimeLeft).MAP().val;
    var nextState = transition(state, action);
    var out = {states:state, actions:action, both:[state,action]}[statesOrActions];
    if (actualTimeLeft==0 || state.terminateAfterAction){
      return [out];
    } else {
      return [ out ].concat( sampleSequence(nextState, actualTimeLeft-1));
    }
  };
  return sampleSequence(state, actualTotalTime);
};

var plannedTrajectory = function(world, agent) {
  var getExpectedUtilities = function(trajectory, agent, actions) {
    var expectedUtility = agent.expectedUtility;
    var v = mapIndexed(function(i, state) {
      return [state, map(function (a) { return  expectedUtility(state, a, i); }, actions)];
    }, trajectory );
    return v;
  };
  return function(state) {
    var currentPlan = MAPActionPath(state, world, agent, state.timeLeft, 'states');
    return getExpectedUtilities(currentPlan, agent, world.actions);
  };
}

var plannedTrajectories = function(trajectory, world, agent) {
  var getTrajectory = plannedTrajectory(world, agent);
  return map(getTrajectory, trajectory);
}
///

var world = mdp.world;
var start = mdp.startState;

var utilityTable = {
  'Donut N': [10, -10],  // [immediate reward, delayed reward]
  'Donut S': [10, -10],
  'Veg': [-10, 20],
  'Noodle': [0, 0],
  'timeCost': -.01  // cost of taking a single action
};

var restaurantUtility = function(state, action) {
  var feature = world.feature;
  var name = feature(state).name;
  if (name) {
    return utilityTable[name][state.timeAtRestaurant]
  } else {
    return utilityTable.timeCost;
  }
};

var runAndGraph = function(agent) {
  var trajectory = simulateMDP(mdp.startState, world, agent);
  var plans = plannedTrajectories(trajectory, world, agent);
  viz.gridworld(world, {
    trajectory,
    dynamicActionExpectedUtilities: plans
  });
};

var agent = makeAgent({
  sophisticatedOrNaive: 'naive',
  utility: restaurantUtility
}, world);

print('Naive agent: \n\n');
runAndGraph(agent);
~~~~

We run the Sophisticated agent with the same parameters and visualization.

~~~~
///fold:
var makeAgent = function(params, world) {
  var defaultParams = {
    alpha: 500,
    discount: 1
  };
  var params = extend(defaultParams, params);
  var stateToActions = world.stateToActions;
  var transition = world.transition;
  var utility = params.utility;
  var paramsDiscountFunction = params.discountFunction;

  var discountFunction = (
    paramsDiscountFunction ?
    paramsDiscountFunction :
    function(delay){ return 1/(1+params.discount*delay); });

  var isNaive = params.sophisticatedOrNaive === 'naive';

  var act = dp.cache(
    function(state, delay) {
      var delay = delay || 0; // make sure delay is never undefined
      return Infer({ model() {
        var action = uniformDraw(stateToActions(state));
        var eu = expectedUtility(state, action, delay);
        factor(params.alpha * eu);
        return action;
      }});
    });

  var expectedUtility = dp.cache(
    function(state, action, delay) {
      var u = discountFunction(delay) * utility(state, action);
      if (state.terminateAfterAction){
        return u;
      } else {
        return u + expectation(Infer({ model() {
          var nextState = transition(state, action);
          var perceivedDelay = isNaive ? delay + 1 : 0;
          var nextAction = sample(act(nextState, perceivedDelay));
          return expectedUtility(nextState, nextAction, delay+1);
        }}));
      }
    });

  return { params, expectedUtility, act };
};

var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});

var world = mdp.world;

var utilityTable = {
  'Donut N': [10, -10],  // [immediate reward, delayed reward]
  'Donut S': [10, -10],
  'Veg': [-10, 20],
  'Noodle': [0, 0],
  'timeCost': -.01  // cost of taking a single action
};

var restaurantUtility = function(state, action) {
  var feature = world.feature;
  var name = feature(state).name;
  if (name) {
    return utilityTable[name][state.timeAtRestaurant]
  } else {
    return utilityTable.timeCost;
  }
};

var MAPActionPath = function(state, world, agent, actualTotalTime, statesOrActions) {
  var perceivedTotalTime = state.timeLeft;
  assert.ok(perceivedTotalTime  > 1 || state.terminateAfterAction==false,
            'perceivedTime<=1. If=1 then should have state.terminateAfterAction,' +
            ' but then simulate wont work ' + JSON.stringify(state));

  var agentAction = agent.act;
  var expectedUtility = agent.expectedUtility;
  var transition = world.transition;

  var sampleSequence = function (state, actualTimeLeft) {
    var action = agentAction(state, actualTotalTime-actualTimeLeft).MAP().val;
    var nextState = transition(state, action);
    var out = {states:state, actions:action, both:[state,action]}[statesOrActions];
    if (actualTimeLeft==0 || state.terminateAfterAction){
      return [out];
    } else {
      return [ out ].concat( sampleSequence(nextState, actualTimeLeft-1));
    }
  };
  return sampleSequence(state, actualTotalTime);
};

var plannedTrajectory = function(world, agent) {
  var getExpectedUtilities = function(trajectory, agent, actions) {
    var expectedUtility = agent.expectedUtility;
    var v = mapIndexed(function(i, state) {
      return [state, map(function (a) { return  expectedUtility(state, a, i); }, actions)];
    }, trajectory );
    return v;
  };
  return function(state) {
    var currentPlan = MAPActionPath(state, world, agent, state.timeLeft, 'states');
    return getExpectedUtilities(currentPlan, agent, world.actions);
  };
};

var plannedTrajectories = function(trajectory, world, agent) {
  var getTrajectory = plannedTrajectory(world, agent);
  return map(getTrajectory, trajectory);
};

var runAndGraph = function(agent) {
  var trajectory = simulateMDP(mdp.startState, world, agent);
  var plans = plannedTrajectories(trajectory, world, agent);
  viz.gridworld(world, {
    trajectory,
    dynamicActionExpectedUtilities: plans
  });
};
///

var agent = makeAgent({
  sophisticatedOrNaive: 'sophisticated',
  utility: restaurantUtility
}, world);

print('Sophisticated agent: \n\n');
runAndGraph(agent);
~~~~

>**Exercise**: What would an exponential discounter with identical preferences to the agents above do on the Restaurant Choice problem? Implement an exponential discounter in the codebox above by adding a `discountFunction` property to the `params` argument to `makeAgent`.

--------

### Example: Procrastinating on a task

Compared to the Restaurant Choice problem, procrastination leads to (systematically biased) behavior that is especially hard to explain on the softmax noise mode.

> **The Procrastination Problem**
>
You have a hard deadline of ten days to complete a task (e.g. write a paper for class, apply for a school or job). Completing the task takes a full day and has a *cost* (it's unpleasant work). After the task is complete you get a *reward* (typically exceeding the cost). There is an incentive to finish early: every day you delay finishing, your reward gets slightly smaller. (Imagine that it's good for your reputation to complete tasks early or that early applicants are considered first).

Note that if the task is worth doing at the last minute, then you should do it immediately (because the reward diminishes over time). Yet people often do this kind of task at the last minute -- the worst possible time to do it!

Hyperbolic discounting provides an elegant model of this behavior. On Day 1, a hyperbolic discounter will prefer that they complete the task tomorrow rather than today. Moreover, a Naive agent wrongly predicts they will complete the task tomorrow and so puts off the task till Day 2. When Day 2 arrives, the Naive agent reasons in the same way -- telling themself that they can avoid the work today by putting it off till tomorrow. This continues until the last possible day, when the Naive agent finally completes the task.

In this problem, the behavior of optimal and time-inconsistent agents with identical preferences (i.e. utility functions) diverges. If the deadline is $$T$$ days from the start, the optimal agent will do the task immediately and the Naive agent will do the task on Day $$T$$. Any problem where a time-inconsistent agent receives exponentially lower reward than an optimal agent contains a close variant of our Procrastination Problem refp:kleinberg2014time [^kleinberg].

[^kleinberg]: Kleinberg and Oren's paper considers a variant problem where the each cost/penalty for waiting is received immediately (rather than being delayed until the time the task is done). In this variant, the agent must eventually complete the task. The authors consider "semi-myopic" time-inconsistent agents, i.e. agents who do not discount their next reward, but discount all future rewards by $$\beta < 1$$. They show that in any problem where a semi-myopic agent receives exponentially lower reward than an optimal agent, the problem must contain a copy of their variant of the Procrastination Problem.

We formalize the Procrastination Problem in terms of a deterministic graph. Suppose the **deadline** is $$T$$ steps from the start. Assume that after $$t$$ < $$T$$ steps the agent has not yet completed the task. Then the agent can take the action `"work"` (which has **work cost** $$-w$$) or the action `"wait"` with zero cost. After the `"work"` action the agent transitions to the `"reward"` state and receives $$+(R - t \epsilon)$$, where $$R$$ is the **reward** for the task and $$\epsilon$$ is how much the reward diminishes for every day of waiting (the **wait cost**). See Figure 3 below.

>**Figure 3:** Transition graph for Procrastination Problem. States are represented by nodes. Edges are state-transitions and are labeled with the action name and the utility of the state-action pair. Terminal nodes have a bold border and their utility is labeled below.

We simulate the behavior of hyperbolic discounters on the Procrastination Problem. We vary the discount rate $$k$$ while holding the other parameters fixed. The agent's behavior can be summarized by its final state (`"wait_state"` or `"reward_state`) and by how much time elapses before termination. When $$k$$ is sufficiently high, the agent will not even complete the task on the last day.

~~~~
///fold: makeProcrastinationMDP, makeProcrastinationUtility
var makeProcrastinationMDP = function(deadlineTime) {
  var stateLocs = ["wait_state", "reward_state"];
  var actions = ["wait", "work", "relax"];

  var stateToActions = function(state) {
    return (state.loc === "wait_state" ?
            ["wait", "work"] :
            ["relax"]);
  };

  var advanceTime = function (state) {
    var newTimeLeft = state.timeLeft - 1;
    var terminateAfterAction = (newTimeLeft === 1 ||
                                state.loc === "reward_state");
    return extend(state, {
      timeLeft: newTimeLeft,
      terminateAfterAction: terminateAfterAction
    });
  };

  var transition = function(state, action) {
    assert.ok(_.includes(stateLocs, state.loc) && _.includes(actions, action),
              'procrastinate transition:' + [state.loc,action]);

    if (state.loc === "reward_state") {
      return advanceTime(state);
    } else if (action === "wait") {
      var waitSteps = state.waitSteps + 1;
      return extend(advanceTime(state), { waitSteps });
    } else {
      var newState = extend(state, { loc: "reward_state" });
      return advanceTime(newState);
    }
  };

  var feature = function(state) {
    return state.loc;
  };

  var startState = {
    loc: "wait_state",
    waitSteps: 0,
    timeLeft: deadlineTime,
    terminateAfterAction: false
  };

  return {
    actions,
    stateToActions,
    transition,
    feature,
    startState
  };
};

var makeProcrastinationUtility = function(utilityTable) {
  assert.ok(hasProperties(utilityTable, ['waitCost', 'workCost', 'reward']),
            'makeProcrastinationUtility args');
  var waitCost = utilityTable.waitCost;
  var workCost = utilityTable.workCost;
  var reward = utilityTable.reward;

  // NB: you receive the *workCost* when you leave the *wait_state*
  // You then receive the reward when leaving the *reward_state* state
  return function(state, action) {
    if (state.loc === "reward_state") {
      return reward + state.waitSteps * waitCost;
    } else if (action === "work") {
      return workCost;
    } else {
      return 0;
    }
  };
};
///

// Construct Procrastinate world
var deadline = 10;
var world = makeProcrastinationMDP(deadline);

// Agent params
var utilityTable = {
  reward: 4.5,
  waitCost: -0.1,
  workCost: -1
};

var params = {
  utility: makeProcrastinationUtility(utilityTable),
  alpha: 1000,
  discount: null,
  sophisticatedOrNaive: 'sophisticated'
};

var getLastState = function(discount){
  var agent = makeMDPAgent(extend(params, { discount: discount }), world);
  var states = simulateMDP(world.startState, world, agent, 'states');
  return [last(states).loc, states.length];
};

map(function(discount) {
  var lastState = getLastState(discount);
  print('Discount: ' + discount + '. Last state: ' + lastState[0] +
        '. Time: ' + lastState[1] + '\n')
}, _.range(8));
~~~~

>**Exercise:**

> 1. Explain how an exponential discounter would behave on this task. Assume their utilities are the same as above and consider different discount rates.
> 2. Run the codebox above with a Sophisticated agent. Explain the results.

Next chapter: [Myopia for rewards and belief updates](/chapters/5c-myopic.html)

### Footnotes


Modeling Agents with Probabilistic Programs

---
layout: chapter
title: Joint inference of biases and preferences II
description: Explaining temptation and pre-commitment using softmax noise and hyperbolic discounting.

---

## Restaurant Choice: Time-inconsistent vs. optimal MDP agents

Returning to the MDP Restaurant Choice problem, we compare a model that assumes an optimal, non-discounting MDP agent to a model that includes both time-inconsistent and optimal agents. We also consider models that expand the set of preferences the agent can have.

### Assume discounting, infer "Naive" or "Sophisticated"

Before making a direct comparison, we demonstrate that we can infer the preferences of time-inconsistent agents from observations of their behavior.

First we condition on the path where the agent moves to Donut North. We call this the Naive path because it is distinctive to the Naive hyperbolic discounter (who is tempted by Donut North on the way to Veg):

~~~~
///fold: restaurant choice MDP, naiveTrajectory
var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});

var naiveTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"u"],
  [{"loc":[3,2],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"u"],
  [{"loc":[3,3],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[3,2]},"u"],
  [{"loc":[3,4],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[3,3]},"u"],
  [{"loc":[3,5],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[3,4]},"l"],
  [{"loc":[2,5],"terminateAfterAction":false,"timeLeft":6,"previousLoc":[3,5],"timeAtRestaurant":0},"l"],
  [{"loc":[2,5],"terminateAfterAction":true,"timeLeft":6,"previousLoc":[2,5],"timeAtRestaurant":1},"l"]
];
///
viz.gridworld(mdp.world, { trajectory: naiveTrajectory });
~~~~

For inference, we specialize the approach in the previous chapter for agents in MDPs that are potentially time inconsistent. So we infer $$\nu$$ and $$k$$ (the hyperbolic discounting parameters) but not the initial belief state $$b_0$$. The function `exampleGetPosterior` is a slightly simplified version of the library function we use below.

~~~~
var exampleGetPosterior = function(mdp, prior, observedStateAction){
  var world = mdp.world;
  var makeUtilityFunction = mdp.makeUtilityFunction;
  return Infer({ model() {

    // Sample parameters from prior
    var priorUtility = prior.priorUtility;
    var utilityTable = priorUtility();
    var priorDiscounting = prior.discounting
    var sophisticatedOrNaive = priorDiscounting().sophisticatedOrNaive;

    var priorAlpha = prior.priorAlpha;

    // Create agent with those parameters
    var agent = makeMDPAgent({
      utility: makeUtilityFunction(utilityTable),
      alpha: priorAlpha(),
      discount: priorDiscounting().discount,
      sophisticatedOrNaive : sophisticatedOrNaive
    }, world);

    var agentAction = agent.act;

    // Condition on observed actions
    map(function(stateAction) {
      var state = stateAction[0];
      var action = stateAction[1];
      observe(agentAction(state, 0), action);
    }, observedStateAction);

    // return parameters and summary statistics
    var vegMinusDonut = sum(utilityTable['Veg']) - sum(utilityTable['Donut N']);

    return {
      utility: utilityTable,
      sophisticatedOrNaive: discounting.sophisticatedOrNaive,
      discount: discounting.discount,
      alpha,
      vegMinusDonut,
    };
  }});
};
~~~~

This inference function allows for inference over the softmax parameter ($$\alpha$$ or `alpha`) and the discount constant ($$k$$ or `discount`). For this example, we fix these values so that the agent has low noise ($$\alpha=1000$$) and so $$k=1$$. We also fix the `timeCost` utility to be small and negative and Noodle's utility to be negative. We infer only the agent's utilities and whether they are Naive or Sophisticated.

~~~~
///fold: Call to hyperbolic library function and helper display function
var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});

var naiveTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"u"],
  [{"loc":[3,2],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"u"],
  [{"loc":[3,3],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[3,2]},"u"],
  [{"loc":[3,4],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[3,3]},"u"],
  [{"loc":[3,5],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[3,4]},"l"],
  [{"loc":[2,5],"terminateAfterAction":false,"timeLeft":6,"previousLoc":[3,5],"timeAtRestaurant":0},"l"],
  [{"loc":[2,5],"terminateAfterAction":true,"timeLeft":6,"previousLoc":[2,5],"timeAtRestaurant":1},"l"]
];

var restaurantHyperbolicInfer = getRestaurantHyperbolicInfer();
var getPosterior = restaurantHyperbolicInfer.getPosterior;

var displayResults = function(priorDist, posteriorDist) {

  var priorUtility = priorDist.MAP().val.utility;
  print('Prior highest-probability utility for Veg: ' + priorUtility['Veg']
        + '. Donut: ' + priorUtility['Donut N'] + ' \n');

  var posteriorUtility = posteriorDist.MAP().val.utility;
  print('Posterior highest-probability utility for Veg: '
        + posteriorUtility['Veg'] + '. Donut: ' + posteriorUtility['Donut N']
        + ' \n');

  var getPriorProb = function(x){
    var label = _.keys(x)[0];
    var dist = getMarginalObject(priorDist, label);
    return Math.exp(dist.score(x));
  };

  var getPosteriorProb = function(x){
    var label = _.keys(x)[0];
    var dist = getMarginalObject(posteriorDist, label);
    return Math.exp(dist.score(x));
  };

  var sophisticationPriorDataTable = map(
    function(x) {
      return {
        sophisticatedOrNaive: x,
        probability: getPriorProb({sophisticatedOrNaive: x}),
        distribution: 'prior'
      };
    },
    ['naive', 'sophisticated']);

  var sophisticationPosteriorDataTable = map(
    function(x){
      return {
        sophisticatedOrNaive: x,
        probability: getPosteriorProb({sophisticatedOrNaive: x}),
        distribution: 'posterior'
      };
    },
    ['naive', 'sophisticated']);

  var sophisticatedOrNaiveDataTable = append(sophisticationPosteriorDataTable,
                                             sophisticationPriorDataTable);

  viz.bar(sophisticatedOrNaiveDataTable, { groupBy: 'distribution' });

  var vegMinusDonutPriorDataTable = map(
    function(x){
      return {
        vegMinusDonut: x,
        probability: getPriorProb({vegMinusDonut: x}),
        distribution: 'prior'
      };
    },
    [-60, -50, -40, -30, -20, -10, 0, 10, 20, 30, 40, 50, 60]);

  var vegMinusDonutPosteriorDataTable = map(
    function(x){
      return {
        vegMinusDonut: x,
        probability: getPosteriorProb({vegMinusDonut: x}),
        distribution: 'posterior'
      };
    },
    [-60, -50, -40, -30, -20, -10, 0, 10, 20, 30, 40, 50, 60]);

  var vegMinusDonutDataTable = append(vegMinusDonutPriorDataTable,
                                      vegMinusDonutPosteriorDataTable);

  viz.bar(vegMinusDonutDataTable, { groupBy: 'distribution' });

  var donutTemptingPriorDataTable = map(
    function(x) {
      return {
        donutTempting: x,
        probability: getPriorProb({donutTempting: x}),
        distribution: 'prior'
      };
    },
    [true, false]);

  var donutTemptingPosteriorDataTable = map(
    function(x) {
      return {
        donutTempting: x,
        probability: getPosteriorProb({donutTempting: x}),
        distribution: 'posterior'
      };
    },
    [true, false]);

  var donutTemptingDataTable = append(donutTemptingPriorDataTable,
                                      donutTemptingPosteriorDataTable);

  viz.bar(donutTemptingDataTable, { groupBy: 'distribution' });
};
///

// Prior on agent's utility function: each restaurant has an
// *immediate* utility and a *delayed* utility (which is received after a
// delay of 1).
var priorUtility = function(){
  var utilityValues =  [-10, 0, 10, 20];
  var donut = [uniformDraw(utilityValues), uniformDraw(utilityValues)];
  var veg = [uniformDraw(utilityValues), uniformDraw(utilityValues)];
  return {
    'Donut N': donut,
    'Donut S': donut,
    'Veg': veg,
    'Noodle': [-10, -10],
    'timeCost': -.01
  };
};

var priorDiscounting = function(){
  return {
    discount: 1,
    sophisticatedOrNaive: uniformDraw(['naive', 'sophisticated'])
  };
};
var priorAlpha = function(){ return 1000; };
var prior = {
  utility: priorUtility,
  discounting: priorDiscounting,
  alpha: priorAlpha
};

// Get world and observations
var posterior = getPosterior(mdp.world, prior, naiveTrajectory);

// To get the prior, we condition on the empty list of observations
displayResults(getPosterior(mdp.world, prior, []), posterior);
~~~~

We display maximum values and marginal distributions for both the prior and the posterior conditioned on the path shown above. To compute the prior, we simply condition on the empty list of observations.

The first graph shows the distribution over whether the agent is Sophisticated or Naive (labeled `sophisticatedOrNaive`). For the other graphs, we compute summary statistics of the agent's parameters and display the distribution over them. The variable `vegMinusDonut` is the difference in *total* utility between Veg and Donut, ignoring the fact that each restaurant has an *immediate* and *delayed* utility. Inference rules out cases where the total utility is equal (which is most likely in the prior), since the agent would simply go to Donut South in that case. Finally, we introduce a variable `donutTempting`, which is true if the agent prefers Veg to Donut North at the start but reverses this preference when adjacent to Donut North. The prior probability of `donutTempting` is less than $$0.1$$, since it depends on relatively delicate balance of utilities and the discounting behavior. The posterior is closer to $$0.9$$, suggesting (along with the posterior on `sophisticatedOrNaive`) that this is the explanation of the data favored by the model.

--------

Using the same prior, we condition on the "Sophisticated" path (i.e. the path distinctive to the Sophisticated agent who avoids the temptation of Donut North and takes the long route to Veg):

~~~~
///fold:
var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});

var sophisticatedTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"u"],
  [{"loc":[3,2],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"u"],
  [{"loc":[3,3],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[3,2]},"r"],
  [{"loc":[4,3],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[3,3]},"r"],
  [{"loc":[5,3],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[4,3]},"u"],
  [{"loc":[5,4],"terminateAfterAction":false,"timeLeft":6,"previousLoc":[5,3]},"u"],
  [{"loc":[5,5],"terminateAfterAction":false,"timeLeft":5,"previousLoc":[5,4]},"u"],
  [{"loc":[5,6],"terminateAfterAction":false,"timeLeft":4,"previousLoc":[5,5]},"l"],
  [{"loc":[4,6],"terminateAfterAction":false,"timeLeft":3,"previousLoc":[5,6]},"u"],
  [{"loc":[4,7],"terminateAfterAction":false,"timeLeft":2,"previousLoc":[4,6],"timeAtRestaurant":0},"l"],
  [{"loc":[4,7],"terminateAfterAction":true,"timeLeft":2,"previousLoc":[4,7],"timeAtRestaurant":1},"l"]
];
///
viz.gridworld(mdp.world, { trajectory: sophisticatedTrajectory });
~~~~

Here are the results of inference:

~~~~
///fold: Definition of world, prior and inference function is same as above codebox
var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});

var restaurantHyperbolicInfer = getRestaurantHyperbolicInfer();
var getPosterior = restaurantHyperbolicInfer.getPosterior;

var displayResults = function(priorDist, posteriorDist) {

  var priorUtility = priorDist.MAP().val.utility;
  print('Prior highest-probability utility for Veg: ' + priorUtility['Veg']
        + '. Donut: ' + priorUtility['Donut N'] + ' \n');

  var posteriorUtility = posteriorDist.MAP().val.utility;
  print('Posterior highest-probability utility for Veg: '
        + posteriorUtility['Veg'] + '. Donut: ' + posteriorUtility['Donut N']
        + ' \n');

  var getPriorProb = function(x) {
    var label = _.keys(x)[0];
    var dist = getMarginalObject(priorDist, label);
    return Math.exp(dist.score(x));
  };

  var getPosteriorProb = function(x) {
    var label = _.keys(x)[0];
    var dist = getMarginalObject(posteriorDist, label);
    return Math.exp(dist.score(x));
  };

  var sophisticationPriorDataTable = map(
    function(x){
      return {
        sophisticatedOrNaive: x,
        probability: getPriorProb({sophisticatedOrNaive: x}),
        distribution: 'prior'
      };
    },
    ['naive', 'sophisticated']);

  var sophisticationPosteriorDataTable = map(
    function(x){
      return {
        sophisticatedOrNaive: x,
        probability: getPosteriorProb({sophisticatedOrNaive: x}),
        distribution: 'posterior'
      };
    },
    ['naive', 'sophisticated']);

  var sophisticatedOrNaiveDataTable = append(sophisticationPriorDataTable,
                                             sophisticationPosteriorDataTable);

  viz.bar(sophisticatedOrNaiveDataTable, { groupBy: 'distribution' });

  var vegMinusDonutPriorDataTable = map(
    function(x) {
      return {
        vegMinusDonut: x,
        probability: getPriorProb({vegMinusDonut: x}),
        distribution: 'prior'
      };
    },
    [-60, -50, -40, -30, -20, -10, 0, 10, 20, 30, 40, 50, 60]);

  var vegMinusDonutPosteriorDataTable = map(
    function(x) {
      return {
        vegMinusDonut: x,
        probability: getPosteriorProb({vegMinusDonut: x}),
        distribution: 'posterior'
      };
    },
    [-60, -50, -40, -30, -20, -10, 0, 10, 20, 30, 40, 50, 60]);

  var vegMinusDonutDataTable = append(vegMinusDonutPriorDataTable,
                                      vegMinusDonutPosteriorDataTable);

  viz.bar(vegMinusDonutDataTable, { groupBy: 'distribution' });

  var donutTemptingPriorDataTable = map(
    function(x) {
      return {
        donutTempting: x,
        probability: getPriorProb({ donutTempting: x }),
        distribution: 'prior'
      };
    },
    [true, false]);

  var donutTemptingPosteriorDataTable = map(
    function(x) {
      return {
        donutTempting: x,
        probability: getPosteriorProb({ donutTempting: x }),
        distribution: 'posterior'
      };
    },
    [true, false]);

  var donutTemptingDataTable = append(donutTemptingPriorDataTable,
                                      donutTemptingPosteriorDataTable);

  viz.bar(donutTemptingDataTable, { groupBy: 'distribution' });
};

// Prior on agent's utility function
var priorUtility = function() {
  var utilityValues =  [-10, 0, 10, 20];
  var donut = [uniformDraw(utilityValues), uniformDraw(utilityValues)];
  var veg = [uniformDraw(utilityValues), uniformDraw(utilityValues)];
  return {
    'Donut N': donut,
    'Donut S': donut,
    'Veg': veg,
    'Noodle': [-10, -10],
    'timeCost': -.01
  };
};

var priorDiscounting = function(){
  return {
    discount: 1,
    sophisticatedOrNaive: uniformDraw(['naive','sophisticated'])
  };
};
var priorAlpha = function(){ return 1000; };
var prior = {
  utility: priorUtility,
  discounting: priorDiscounting,
  alpha: priorAlpha
};

var sophisticatedTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"u"],
  [{"loc":[3,2],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"u"],
  [{"loc":[3,3],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[3,2]},"r"],
  [{"loc":[4,3],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[3,3]},"r"],
  [{"loc":[5,3],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[4,3]},"u"],
  [{"loc":[5,4],"terminateAfterAction":false,"timeLeft":6,"previousLoc":[5,3]},"u"],
  [{"loc":[5,5],"terminateAfterAction":false,"timeLeft":5,"previousLoc":[5,4]},"u"],
  [{"loc":[5,6],"terminateAfterAction":false,"timeLeft":4,"previousLoc":[5,5]},"l"],
  [{"loc":[4,6],"terminateAfterAction":false,"timeLeft":3,"previousLoc":[5,6]},"u"],
  [{"loc":[4,7],"terminateAfterAction":false,"timeLeft":2,"previousLoc":[4,6],"timeAtRestaurant":0},"l"],
  [{"loc":[4,7],"terminateAfterAction":true,"timeLeft":2,"previousLoc":[4,7],"timeAtRestaurant":1},"l"]
];
///

// Get world and observations
var posterior = getPosterior(mdp.world, prior, sophisticatedTrajectory);
displayResults(getPosterior(mdp.world, prior, []), posterior);
~~~~

If the agent goes directly to Veg, then they don't provide information about whether they are Naive or Sophisticated. Using the same prior again, we do inference on this path:

~~~~
///fold:
var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});

var vegDirectTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"u"],
  [{"loc":[3,2],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"u"],
  [{"loc":[3,3],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[3,2]},"u"],
  [{"loc":[3,4],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[3,3]},"u"],
  [{"loc":[3,5],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[3,4]},"u"],
  [{"loc":[3,6],"terminateAfterAction":false,"timeLeft":6,"previousLoc":[3,5]},"r"],
  [{"loc":[4,6],"terminateAfterAction":false,"timeLeft":5,"previousLoc":[3,6]},"u"],
  [{"loc":[4,7],"terminateAfterAction":false,"timeLeft":4,"previousLoc":[4,6],"timeAtRestaurant":0},"l"],
  [{"loc":[4,7],"terminateAfterAction":true,"timeLeft":4,"previousLoc":[4,7],"timeAtRestaurant":1},"l"]
];
///
viz.gridworld(mdp.world, { trajectory: vegDirectTrajectory });
~~~~

Here are the results of inference:

~~~~
// Definition of world, prior and inference function is same as above codebox

///fold:
var restaurantHyperbolicInfer = getRestaurantHyperbolicInfer();
var getPosterior = restaurantHyperbolicInfer.getPosterior;

var displayResults = function(priorDist, posteriorDist) {

  var priorUtility = priorDist.MAP().val.utility;
  print('Prior highest-probability utility for Veg: ' + priorUtility['Veg']
    + '. Donut: ' + priorUtility['Donut N'] + ' \n');

  var posteriorUtility = posteriorDist.MAP().val.utility;
  print('Posterior highest-probability utility for Veg: '
    + posteriorUtility['Veg'] + '. Donut: ' + posteriorUtility['Donut N']
    + ' \n');

  var getPriorProb = function(x) {
    var label = _.keys(x)[0];
    var dist = getMarginalObject(priorDist, label);
    return Math.exp(dist.score(x));
  };

  var getPosteriorProb = function(x) {
    var label = _.keys(x)[0];
    var dist = getMarginalObject(posteriorDist, label);
    return Math.exp(dist.score(x));
  };

  var sophisticationPriorDataTable = map(function(x) {
    return {sophisticatedOrNaive: x,
            probability: getPriorProb({sophisticatedOrNaive: x}),
            distribution: 'prior'};
  }, ['naive', 'sophisticated']);

  var sophisticationPosteriorDataTable = map(function(x) {
    return {sophisticatedOrNaive: x,
            probability: getPosteriorProb({sophisticatedOrNaive: x}),
            distribution: 'posterior'};
  }, ['naive', 'sophisticated']);

  var sophisticatedOrNaiveDataTable = append(sophisticationPriorDataTable,
                                             sophisticationPosteriorDataTable);

  viz.bar(sophisticatedOrNaiveDataTable, { groupBy: 'distribution' });

  var vegMinusDonutPriorDataTable = map(function(x){
    return {
      vegMinusDonut: x,
      probability: getPriorProb({vegMinusDonut: x}),
      distribution: 'prior'
    };
  }, [-60, -50, -40, -30, -20, -10, 0, 10, 20, 30, 40, 50, 60]);

  var vegMinusDonutPosteriorDataTable = map(function(x){
    return {vegMinusDonut: x,
            probability: getPosteriorProb({vegMinusDonut: x}),
            distribution: 'posterior'};
  }, [-60, -50, -40, -30, -20, -10, 0, 10, 20, 30, 40, 50, 60]);

  var vegMinusDonutDataTable = append(vegMinusDonutPriorDataTable,
                                      vegMinusDonutPosteriorDataTable);

  viz.bar(vegMinusDonutDataTable, {groupBy: 'distribution'});

  var donutTemptingPriorDataTable = map(function(x){
    return {
      donutTempting: x,
      probability: getPriorProb({donutTempting: x}),
      distribution: 'prior'
    };
  }, [true, false]);

  var donutTemptingPosteriorDataTable = map(function(x){
    return {
      donutTempting: x,
      probability: getPosteriorProb({donutTempting: x}),
      distribution: 'posterior'
    };
  }, [true, false]);

  var donutTemptingDataTable = append(donutTemptingPriorDataTable,
                                      donutTemptingPosteriorDataTable);

  viz.bar(donutTemptingDataTable, { groupBy: 'distribution' });
};

// Prior on agent's utility function
var priorUtility = function() {
  var utilityValues =  [-10, 0, 10, 20];
  var donut = [uniformDraw(utilityValues), uniformDraw(utilityValues)];
  var veg = [uniformDraw(utilityValues), uniformDraw(utilityValues)];
  return {
    'Donut N': donut,
    'Donut S': donut,
    'Veg': veg,
    'Noodle': [-10, -10],
    'timeCost': -.01
  };
};

var priorDiscounting = function() {
  return {
    discount: 1,
    sophisticatedOrNaive: uniformDraw(['naive','sophisticated'])
  };
};
var priorAlpha = function(){return 1000;};
var prior = {
  utility: priorUtility,
  discounting: priorDiscounting,
  alpha: priorAlpha
};

var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});

var vegDirectTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"u"],
  [{"loc":[3,2],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"u"],
  [{"loc":[3,3],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[3,2]},"u"],
  [{"loc":[3,4],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[3,3]},"u"],
  [{"loc":[3,5],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[3,4]},"u"],
  [{"loc":[3,6],"terminateAfterAction":false,"timeLeft":6,"previousLoc":[3,5]},"r"],
  [{"loc":[4,6],"terminateAfterAction":false,"timeLeft":5,"previousLoc":[3,6]},"u"],
  [{"loc":[4,7],"terminateAfterAction":false,"timeLeft":4,"previousLoc":[4,6],"timeAtRestaurant":0},"l"],
  [{"loc":[4,7],"terminateAfterAction":true,"timeLeft":4,"previousLoc":[4,7],"timeAtRestaurant":1},"l"]
];
///

var posterior = getPosterior(mdp.world, prior, vegDirectTrajectory);
displayResults(getPosterior(mdp.world, prior, []), posterior);
~~~~

---------

### Assume non-discounting, infer preferences and softmax

We want to compare a model that assumes an optimal MDP agent with one that allows for time-inconsistency. We first show the inferences by the model that assumes optimality. This model can only explain the anomalous Naive and Sophisticated paths in terms of softmax noise (lower values for $$\alpha$$). We display the prior and posteriors for both the Naive and Sophisticated paths.

~~~~
///fold:
var restaurantHyperbolicInfer = getRestaurantHyperbolicInfer();
var getPosterior = restaurantHyperbolicInfer.getPosterior;

var displayResults = function(priorDist, posteriorDist) {

  var priorUtility = priorDist.MAP().val.utility;
  print('Prior highest-probability utility for Veg: ' + priorUtility['Veg']
        + '. Donut: ' + priorUtility['Donut N'] + ' \n');

  var posteriorUtility = posteriorDist.MAP().val.utility;
  print('Posterior highest-probability utility for Veg: '
        + posteriorUtility['Veg'] + '. Donut: ' + posteriorUtility['Donut N']
        + ' \n');

  var getPriorProb = function(x) {
    var label = _.keys(x)[0];
    var dist = getMarginalObject(priorDist, label);
    return Math.exp(dist.score(x));
  };

  var getPosteriorProb = function(x) {
    var label = _.keys(x)[0];
    var dist = getMarginalObject(posteriorDist, label);
    return Math.exp(dist.score(x));
  };

  var vegMinusDonutPriorDataTable = map(
    function(x){
      return {
        vegMinusDonut: x,
        probability: getPriorProb({ vegMinusDonut: x }),
        distribution: 'prior'
      };
    },
    [-50, -40, -30, -20, -10, 0, 10, 20, 30, 40, 50]);

  var vegMinusDonutPosteriorDataTable = map(
    function(x){
      return {
        vegMinusDonut: x,
        probability: getPosteriorProb({ vegMinusDonut: x }),
        distribution: 'posterior'
      };
    },
    [-50, -40, -30, -20, -10, 0, 10, 20, 30, 40, 50]);

  var vegMinusDonutDataTable = append(vegMinusDonutPriorDataTable,
                                      vegMinusDonutPosteriorDataTable);

  viz.bar(vegMinusDonutDataTable, { groupBy: 'distribution' });

  var alphaPriorDataTable = map(
    function(x){
      return {
        alpha: x,
        probability: getPriorProb({ alpha: x }),
        distribution: 'prior'
      };
    },
    [0.1, 10, 100, 1000]);

  var alphaPosteriorDataTable = map(
    function(x){
      return {
        alpha: x,
        probability: getPosteriorProb({ alpha: x }),
        distribution: 'posterior'
      };
    },
    [0.1, 10, 100, 1000]);

  var alphaDataTable = append(alphaPriorDataTable,
                              alphaPosteriorDataTable);

  viz.bar(alphaDataTable, { groupBy: 'distribution' });
};

var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});

var naiveTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"u"],
  [{"loc":[3,2],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"u"],
  [{"loc":[3,3],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[3,2]},"u"],
  [{"loc":[3,4],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[3,3]},"u"],
  [{"loc":[3,5],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[3,4]},"l"],
  [{"loc":[2,5],"terminateAfterAction":false,"timeLeft":6,"previousLoc":[3,5],"timeAtRestaurant":0},"l"],
  [{"loc":[2,5],"terminateAfterAction":true,"timeLeft":6,"previousLoc":[2,5],"timeAtRestaurant":1},"l"]
];

var sophisticatedTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"u"],
  [{"loc":[3,2],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"u"],
  [{"loc":[3,3],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[3,2]},"r"],
  [{"loc":[4,3],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[3,3]},"r"],
  [{"loc":[5,3],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[4,3]},"u"],
  [{"loc":[5,4],"terminateAfterAction":false,"timeLeft":6,"previousLoc":[5,3]},"u"],
  [{"loc":[5,5],"terminateAfterAction":false,"timeLeft":5,"previousLoc":[5,4]},"u"],
  [{"loc":[5,6],"terminateAfterAction":false,"timeLeft":4,"previousLoc":[5,5]},"l"],
  [{"loc":[4,6],"terminateAfterAction":false,"timeLeft":3,"previousLoc":[5,6]},"u"],
  [{"loc":[4,7],"terminateAfterAction":false,"timeLeft":2,"previousLoc":[4,6],"timeAtRestaurant":0},"l"],
  [{"loc":[4,7],"terminateAfterAction":true,"timeLeft":2,"previousLoc":[4,7],"timeAtRestaurant":1},"l"]
];
///

// Prior on agent's utility function
var priorUtility = function() {
  var utilityValues = [-10, 0, 10, 20, 30, 40];
  // with no discounting, delayed utilities are ommitted
  var donut = [uniformDraw(utilityValues), 0];
  var veg = [uniformDraw(utilityValues), 0];
  return {
    'Donut N': donut,
    'Donut S': donut,
    'Veg': veg,
    'Noodle': [-10, -10],
    'timeCost': -.01
  };
};

// We assume no discounting (so *sophisticated* has no effect here)
var priorDiscounting = function() {
  return {
    discount: 0,
    sophisticatedOrNaive: 'sophisticated'
  };
};

var priorAlpha = function(){ return uniformDraw([0.1, 10, 100, 1000]); };
var prior = {
  utility: priorUtility,
  discounting: priorDiscounting,
  alpha: priorAlpha
};

// Get world and observations
var world = mdp.world;

print('Prior and posterior after observing Naive path');
var posteriorNaive = getPosterior(world, prior, naiveTrajectory);
displayResults(getPosterior(world, prior, []), posteriorNaive);

print('Prior and posterior after observing Sophisticated path');
var posteriorSophisticated = getPosterior(world, prior, sophisticatedTrajectory);
displayResults(getPosterior(world, prior, []), posteriorSophisticated);
~~~~

The graphs show two important results:

1. For the Naive path, the agent is inferred to prefer Donut, while for the Sophisticated path, Veg is inferred. In both cases, the inference fits with where the agent ends up.

2. High values for $$\alpha$$ are ruled out in each case, showing that the model explains the behavior in terms of noise.

What happens if we observe the agent taking the Naive path *repeatedly*? While noise is needed to explain the agent's path, too much noise is inconsistent with taking an identical path repeatedly. This is confirmed in the results below:

~~~~
///fold: Prior is same as above
var restaurantHyperbolicInfer = getRestaurantHyperbolicInfer();
var getPosterior = restaurantHyperbolicInfer.getPosterior;

var displayResults = function(priorDist, posteriorDist) {

  var priorUtility = priorDist.MAP().val.utility;
  print('Prior highest-probability utility for Veg: ' + priorUtility['Veg']
        + '. Donut: ' + priorUtility['Donut N'] + ' \n');

  var posteriorUtility = posteriorDist.MAP().val.utility;
  print('Posterior highest-probability utility for Veg: '
        + posteriorUtility['Veg'] + '. Donut: ' + posteriorUtility['Donut N']
        + ' \n');

  var getPriorProb = function(x) {
    var label = _.keys(x)[0];
    var dist = getMarginalObject(priorDist, label);
    return Math.exp(dist.score(x));
  };

  var getPosteriorProb = function(x) {
    var label = _.keys(x)[0];
    var dist = getMarginalObject(posteriorDist, label);
    return Math.exp(dist.score(x));
  };

  var vegMinusDonutPriorDataTable = map(
    function(x) {
      return {
        vegMinusDonut: x,
        probability: getPriorProb({ vegMinusDonut: x }),
        distribution: 'prior'
      };
    },
    [-50, -40, -30, -20, -10, 0, 10, 20, 30, 40, 50]);

  var vegMinusDonutPosteriorDataTable = map(
    function(x){
      return {
        vegMinusDonut: x,
        probability: getPosteriorProb({ vegMinusDonut: x }),
        distribution: 'posterior'
      };
    },
    [-50, -40, -30, -20, -10, 0, 10, 20, 30, 40, 50]);

  var vegMinusDonutDataTable = append(vegMinusDonutPriorDataTable,
                                      vegMinusDonutPosteriorDataTable);

  viz.bar(vegMinusDonutDataTable, { groupBy: 'distribution' });

  var alphaPriorDataTable = map(
    function(x){
      return {
        alpha: x,
        probability: getPriorProb({alpha: x}),
        distribution: 'prior'
      };
    },
    [0.1, 10, 100, 1000]);

  var alphaPosteriorDataTable = map(
    function(x){
      return {
        alpha: x,
        probability: getPosteriorProb({alpha: x}),
        distribution: 'posterior'
      };
    },
    [0.1, 10, 100, 1000]);

  var alphaDataTable = append(alphaPriorDataTable,
                              alphaPosteriorDataTable);

  viz.bar(alphaDataTable, { groupBy: 'distribution' });
};

// Prior on agent's utility function
var priorUtility = function() {
  var utilityValues = [-10, 0, 10, 20, 30, 40];
  // with no discounting, delayed utilities are ommitted
  var donut = [uniformDraw(utilityValues), 0];
  var veg = [uniformDraw(utilityValues), 0];
  return {
    'Donut N': donut,
    'Donut S': donut,
    'Veg': veg,
    'Noodle': [-10, -10],
    'timeCost': -.01
  };
};

// We assume no discounting (so *sophisticated* has no effect here)
var priorDiscounting = function(){
  return {
    discount: 0,
    sophisticatedOrNaive: 'sophisticated'
  };
};

var priorAlpha = function(){
  return uniformDraw([0.1, 10, 100, 1000]);
};
var prior = {
  utility: priorUtility,
  discounting: priorDiscounting,
  alpha: priorAlpha
};

var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});

var naiveTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"u"],
  [{"loc":[3,2],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"u"],
  [{"loc":[3,3],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[3,2]},"u"],
  [{"loc":[3,4],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[3,3]},"u"],
  [{"loc":[3,5],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[3,4]},"l"],
  [{"loc":[2,5],"terminateAfterAction":false,"timeLeft":6,"previousLoc":[3,5],"timeAtRestaurant":0},"l"],
  [{"loc":[2,5],"terminateAfterAction":true,"timeLeft":6,"previousLoc":[2,5],"timeAtRestaurant":1},"l"]
];
///

var numberRepeats = 2; // with 2 repeats, we condition a total of 3 times
var posteriorNaive = getPosterior(mdp.world, prior, naiveTrajectory, numberRepeats);
print('Prior and posterior after conditioning 3 times on Naive path');
displayResults(getPosterior(mdp.world, prior, []), posteriorNaive);
~~~~

--------

### Model that includes discounting: jointly infer discounting, preferences, softmax noise

Our inference model now has the optimal agent as a special case but also includes time-inconsistent agents. This model jointly infers the discounting behavior, the agent's utilities and the softmax noise.

We show two different posteriors. The first is after conditioning on the Naive path (as above). In the second, we imagine that we have observed the agent taking the same path on multiple occasions (three times) and we condition on this.

~~~~
///fold:
var restaurantHyperbolicInfer = getRestaurantHyperbolicInfer();
var getPosterior = restaurantHyperbolicInfer.getPosterior;

var displayResults = function(priorDist, posteriorDist) {

  var priorUtility = priorDist.MAP().val.utility;
  print('Prior highest-probability utility for Veg: ' + priorUtility['Veg']
        + '. Donut: ' + priorUtility['Donut N'] + ' \n');

  var posteriorUtility = posteriorDist.MAP().val.utility;
  print('Posterior highest-probability utility for Veg: '
        + posteriorUtility['Veg'] + '. Donut: ' + posteriorUtility['Donut N']
        + ' \n');

  var getPriorProb = function(x) {
    var label = _.keys(x)[0];
    var dist = getMarginalObject(priorDist, label);
    return Math.exp(dist.score(x));
  };

  var getPosteriorProb = function(x) {
    var label = _.keys(x)[0];
    var dist = getMarginalObject(posteriorDist, label);
    return Math.exp(dist.score(x));
  };

  var sophisticationPriorDataTable = map(
    function(x) {
      return {
        sophisticatedOrNaive: x,
        probability: getPriorProb({ sophisticatedOrNaive: x }),
        distribution: 'prior'
      };
    },
    ['naive', 'sophisticated']);

  var sophisticationPosteriorDataTable = map(
    function(x) {
      return {
        sophisticatedOrNaive: x,
        probability: getPosteriorProb({ sophisticatedOrNaive: x }),
        distribution: 'posterior'
      };
    },
    ['naive', 'sophisticated']);

  var sophisticatedOrNaiveDataTable = append(sophisticationPosteriorDataTable,
                                             sophisticationPriorDataTable);

  viz.bar(sophisticatedOrNaiveDataTable, { groupBy: 'distribution' });

  var vegMinusDonutPriorDataTable = map(
    function(x) {
      return {
        vegMinusDonut: x,
        probability: getPriorProb({ vegMinusDonut: x }),
        distribution: 'prior'
      };
    },
    [-10, 0, 10, 20, 30, 40, 50, 60, 70]);

  var vegMinusDonutPosteriorDataTable = map(
    function(x) {
      return {
        vegMinusDonut: x,
        probability: getPosteriorProb({ vegMinusDonut: x }),
        distribution: 'posterior'
      };
    },
    [-10, 0, 10, 20, 30, 40, 50, 60, 70]);

  var vegMinusDonutDataTable = append(vegMinusDonutPriorDataTable,
                                      vegMinusDonutPosteriorDataTable);

  viz.bar(vegMinusDonutDataTable, { groupBy: 'distribution' });

  var donutTemptingPriorDataTable = map(
    function(x) {
      return {
        donutTempting: x,
        probability: getPriorProb({donutTempting: x}),
        distribution: 'prior'
      };
    },
    [true, false]);

  var donutTemptingPosteriorDataTable = map(
    function(x){
      return {
        donutTempting: x,
        probability: getPosteriorProb({ donutTempting: x }),
        distribution: 'posterior'
      };
    },
    [true, false]);

  var donutTemptingDataTable = append(donutTemptingPriorDataTable,
                                      donutTemptingPosteriorDataTable);

  viz.bar(donutTemptingDataTable, { groupBy: 'distribution' });

  var alphaPriorDataTable = map(
    function(x){
      return {
        alpha: x,
        probability: getPriorProb({alpha: x}),
        distribution: 'prior'
      };
    },
    [0.1, 10, 1000]);

  var alphaPosteriorDataTable = map(
    function(x){
      return {
        alpha: x,
        probability: getPosteriorProb({ alpha: x }),
        distribution: 'posterior'
      };
    },
    [0.1, 10, 1000]);

  var alphaDataTable = append(alphaPriorDataTable,
                              alphaPosteriorDataTable);

  viz.bar(alphaDataTable, { groupBy: 'distribution' });
};

var naiveTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"u"],
  [{"loc":[3,2],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"u"],
  [{"loc":[3,3],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[3,2]},"u"],
  [{"loc":[3,4],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[3,3]},"u"],
  [{"loc":[3,5],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[3,4]},"l"],
  [{"loc":[2,5],"terminateAfterAction":false,"timeLeft":6,"previousLoc":[3,5],"timeAtRestaurant":0},"l"],
  [{"loc":[2,5],"terminateAfterAction":true,"timeLeft":6,"previousLoc":[2,5],"timeAtRestaurant":1},"l"]
];

var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});
///

// Prior on agent's utility function. We fix the delayed utilities
// to make inference faster
var priorUtility = function() {
  var utilityValues =  [-10, 0, 10, 20, 30];
  var donut = [uniformDraw(utilityValues), -10];
  var veg = [uniformDraw(utilityValues), 20];
  return {
    'Donut N': donut,
    'Donut S': donut,
    'Veg': veg,
    'Noodle': [-10, -10],
    'timeCost': -.01
  };
};

var priorDiscounting = function() {
  return {
    discount: uniformDraw([0, 1]),
    sophisticatedOrNaive: uniformDraw(['naive','sophisticated'])
  };
};
var priorAlpha = function(){
  return uniformDraw([.1, 10, 1000]);
};
var prior = {
  utility: priorUtility,
  discounting: priorDiscounting,
  alpha: priorAlpha
};

// Get world and observations
var world = mdp.world;

var posterior = getPosterior(world, prior, naiveTrajectory);
print('Prior and posterior after observing Naive path');
displayResults(getPosterior(world, prior, []), posterior);

print('Prior and posterior after observing Naive path three times');
var numberRepeats = 2;
displayResults(getPosterior(world, prior, []),
               getPosterior(world, prior, naiveTrajectory, numberRepeats));
~~~~

Conditioning on the Naive path once, the probabilities of the agent being Naive and of `donutTempting` both go up. However, the probability of high softmax noise also goes up. In terms of preferences, we rule out a strong preference for Veg and slightly reduce a preference for Donut. So if the agent were Naive, tempted by Donut and with very low noise, our inference would not place most of the posterior on this explanation. There are two reasons for this. First, this agent is unlikely in the prior. Second, the explanation of the behavior in terms of noise is plausible. (In our Gridworld setup, we don't allow the agent to backtrack to the previous state. This means there are few cases where a softmax noisy agent would behavior differently than a low noise one.). Conditioning on the same Naive path three times makes the explanation in terms of noise much less plausible: the agent would makes the same "mistake" three times and makes no other mistakes. (The results for the Sophisticated path are similar.)

In summary, if we observe the agent repeatedly take the Naive path, the "Optimal Model" explains this in terms of a preference for Donut and significant softmax noise (explaining why the agent takes Donut North over Donut South). The "Discounting Model" is similar to the Optimal Model when it observes the Naive path *once*. However, observing it multiple times, it infers that the agent has low noise and an overall preference for Veg.

------

### Preferences for the two Donut Store branches can vary

Another explanation of the Naive path is that the agent has a preference for the "Donut N" branch of the Donut Store over the "Donut S" branch. Maybe this branch is better run or has more space. If we add this to our set of possible preferences, inference changes significantly.

To speed up inference, we use a fixed assumption that the agent is Naive. There are three explanations of the agent's path:

1. Softmax noise: measured by $$\alpha$$
2. The agent is Naive and tempted by Donut: measured by `discount` and `donutTempting`
3. The agent prefers Donut N to Donut S: measured by `donutNGreaterDonutS` (i.e. Donut N's utility is greater than Donut S's).

These three can also be combined to explain the behavior.

~~~~
///fold:
var restaurantHyperbolicInfer = getRestaurantHyperbolicInfer();
var getPosterior = restaurantHyperbolicInfer.getPosterior;

var naiveTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"u"],
  [{"loc":[3,2],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"u"],
  [{"loc":[3,3],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[3,2]},"u"],
  [{"loc":[3,4],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[3,3]},"u"],
  [{"loc":[3,5],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[3,4]},"l"],
  [{"loc":[2,5],"terminateAfterAction":false,"timeLeft":6,"previousLoc":[3,5],"timeAtRestaurant":0},"l"],
  [{"loc":[2,5],"terminateAfterAction":true,"timeLeft":6,"previousLoc":[2,5],"timeAtRestaurant":1},"l"]
];

var displayResults = function(priorDist, posteriorDist) {

  var priorUtility = priorDist.MAP().val.utility;
  print('Prior highest-probability utility for Veg: ' + priorUtility['Veg']
        + '. Donut: ' + priorUtility['Donut N'] + ' \n');

  var posteriorUtility = posteriorDist.MAP().val.utility;
  print('Posterior highest-probability utility for Veg: '
        + posteriorUtility['Veg'] + '. Donut: ' + posteriorUtility['Donut N']
        + ' \n');

  var getPriorProb = function(x) {
    var label = _.keys(x)[0];
    var dist = getMarginalObject(priorDist, label);
    return Math.exp(dist.score(x));
  };

  var getPosteriorProb = function(x) {
    var label = _.keys(x)[0];
    var dist = getMarginalObject(posteriorDist, label);
    return Math.exp(dist.score(x));
  };

  var alphaPriorDataTable = map(
    function(x) {
      return {
        alpha: x,
        probability: getPriorProb({alpha: x}),
        distribution: 'prior'
      };
    },
    [0.1, 100, 1000]);

  var alphaPosteriorDataTable = map(
    function(x) {
      return {
        alpha: x,
        probability: getPosteriorProb({alpha: x}),
        distribution: 'posterior'
      };
    },
    [0.1, 100, 1000]);

  var alphaDataTable = append(alphaPriorDataTable,
                              alphaPosteriorDataTable);

  viz.bar(alphaDataTable, { groupBy: 'distribution' });

  var donutTemptingPriorDataTable = map(
    function(x) {
      return {
        donutTempting: x,
        probability: getPriorProb({ donutTempting: x }),
        distribution: 'prior'
      };
    },
    [true, false]);

  var donutTemptingPosteriorDataTable = map(
    function(x) {
      return {
        donutTempting: x,
        probability: getPosteriorProb({ donutTempting: x }),
        distribution: 'posterior'
      };
    },
    [true, false]);

  var donutTemptingDataTable = append(donutTemptingPriorDataTable,
                                      donutTemptingPosteriorDataTable);

  viz.bar(donutTemptingDataTable, { groupBy: 'distribution' });

  var discountPriorDataTable = map(
    function(x) {
      return {
        discount: x,
        probability: getPriorProb({ discount: x }),
        distribution: 'prior'
      };
    },
    [0, 1]);

  var discountPosteriorDataTable = map(
    function(x) {
      return {
        discount: x,
        probability: getPosteriorProb({ discount: x }),
        distribution: 'posterior'
      };
    },
    [0, 1]);

  var discountDataTable = append(discountPriorDataTable,
                                 discountPosteriorDataTable);

  viz.bar(discountDataTable, { groupBy: 'distribution' });

  var donutNvsSPriorDataTable = map(
    function(x) {
      return {
        donutNGreaterDonutS: x,
        probability: getPriorProb({ donutNGreaterDonutS: x }),
        distribution: 'prior'
      };
    },
    [false, true]);

  var donutNvsSPosteriorDataTable = map(
    function(x) {
      return {
        donutNGreaterDonutS: x,
        probability: getPosteriorProb({ donutNGreaterDonutS: x }),
        distribution: 'posterior'
      };
    },
    [false, true]);

  var donutNvsSDataTable = append(donutNvsSPriorDataTable,
                                  donutNvsSPosteriorDataTable);

  viz.bar(donutNvsSDataTable, { groupBy: 'distribution' });
};

var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});
///

// Prior on agent's utility function
var priorUtility = function() {
  var utilityValues = [-10, 0, 10, 20];
  return {
    'Donut N': [uniformDraw(utilityValues), -10],
    'Donut S': [uniformDraw(utilityValues), -10],
    'Veg': [20, uniformDraw(utilityValues)],
    'Noodle': [-10, -10],
    'timeCost': -.01
  };
};

var priorDiscounting = function() {
  return {
    discount: uniformDraw([0, 1]),
    sophisticatedOrNaive: 'naive'
  };
};
var priorAlpha = function(){
  return uniformDraw([.1, 100, 1000]);
};
var prior = {
  utility: priorUtility,
  discounting: priorDiscounting,
  alpha: priorAlpha
};

// Get world and observations
var posterior = getPosterior(mdp.world, prior, naiveTrajectory);
displayResults(getPosterior(mdp.world, prior, []), posterior);
~~~~

The explanation in terms of Donut North being preferred does well in the posterior. This is because the discounting explanation (even assuming the agent is Naive) is unlikely a priori (due to our simple uniform priors on utilities and discounting). While high noise is more plausible a priori, the noise explanation still needs to posit a low probability series of events.

We see a similar result if we enrich the set of possible utilities for the Sophisticated path. This time, we allow the `timeCost`, i.e. the cost for taking a single timestep, to be positive. This means the agent prefers to spend as much time as possible moving around before reaching a restaurant. Here are the results:

Observe the sophisticated path with possibly positive timeCost:

~~~~
///fold:
var restaurantHyperbolicInfer = getRestaurantHyperbolicInfer();
var getPosterior = restaurantHyperbolicInfer.getPosterior;

var displayResults = function(priorDist, posteriorDist) {

  var priorUtility = priorDist.MAP().val.utility;
  print('Prior highest-probability utility for Veg: ' + priorUtility['Veg']
        + '. Donut: ' + priorUtility['Donut N'] + ' \n');

  var posteriorUtility = posteriorDist.MAP().val.utility;
  print('Posterior highest-probability utility for Veg: '
        + posteriorUtility['Veg'] + '. Donut: ' + posteriorUtility['Donut N']
        + ' \n');

  var getPriorProb = function(x) {
    var label = _.keys(x)[0];
    var dist = getMarginalObject(priorDist, label);
    return Math.exp(dist.score(x));
  };

  var getPosteriorProb = function(x) {
    var label = _.keys(x)[0];
    var dist = getMarginalObject(posteriorDist, label);
    return Math.exp(dist.score(x));
  };

  var alphaPriorDataTable = map(
    function(x) {
      return {
        alpha: x,
        probability: getPriorProb({alpha: x}),
        distribution: 'prior'
      };
    },
    [0.1, 100, 1000]);

  var alphaPosteriorDataTable = map(
    function(x) {
      return {
        alpha: x,
        probability: getPosteriorProb({alpha: x}),
        distribution: 'posterior'
      };
    },
    [0.1, 100, 1000]);

  var alphaDataTable = append(alphaPriorDataTable,
                              alphaPosteriorDataTable);

  viz.bar(alphaDataTable, { groupBy: 'distribution' });

  var donutTemptingPriorDataTable = map(
    function(x) {
      return {
        donutTempting: x,
        probability: getPriorProb({ donutTempting: x }),
        distribution: 'prior'
      };
    },
    [true, false]);

  var donutTemptingPosteriorDataTable = map(
    function(x) {
      return {
        donutTempting: x,
        probability: getPosteriorProb({ donutTempting: x }),
        distribution: 'posterior'
      };
    },
    [true, false]);

  var donutTemptingDataTable = append(donutTemptingPriorDataTable,
                                      donutTemptingPosteriorDataTable);

  viz.bar(donutTemptingDataTable, { groupBy: 'distribution' });

  var discountPriorDataTable = map(
    function(x){
      return {
        discount: x,
        probability: getPriorProb({ discount: x }),
        distribution: 'prior'
      };
    },
    [0, 1]);

  var discountPosteriorDataTable = map(
    function(x){
      return {
        discount: x,
        probability: getPosteriorProb({ discount: x }),
        distribution: 'posterior'
      };
    },
    [0, 1]);

  var discountDataTable = append(discountPriorDataTable,
                                 discountPosteriorDataTable);

  viz.bar(discountDataTable, { groupBy: 'distribution' });

  var timeCostPriorDataTable = map(
    function(x) {
      return {
        timeCost: x,
        probability: getPriorProb({ timeCost: x }),
        distribution: 'prior'
      };
    },
    [-0.01, 0.1, 1]);

  var timeCostPosteriorDataTable = map(
    function(x) {
      return {
        timeCost: x,
        probability: getPosteriorProb({ timeCost: x }),
        distribution: 'posterior'
      };
    },
    [-0.01, 0.1, 1]);

  var timeCostDataTable = append(timeCostPriorDataTable,
                                 timeCostPosteriorDataTable);

  viz.bar(timeCostDataTable, { groupBy: 'distribution' });
};

var sophisticatedTrajectory = [
  [{"loc":[3,1],"terminateAfterAction":false,"timeLeft":11},"u"],
  [{"loc":[3,2],"terminateAfterAction":false,"timeLeft":10,"previousLoc":[3,1]},"u"],
  [{"loc":[3,3],"terminateAfterAction":false,"timeLeft":9,"previousLoc":[3,2]},"r"],
  [{"loc":[4,3],"terminateAfterAction":false,"timeLeft":8,"previousLoc":[3,3]},"r"],
  [{"loc":[5,3],"terminateAfterAction":false,"timeLeft":7,"previousLoc":[4,3]},"u"],
  [{"loc":[5,4],"terminateAfterAction":false,"timeLeft":6,"previousLoc":[5,3]},"u"],
  [{"loc":[5,5],"terminateAfterAction":false,"timeLeft":5,"previousLoc":[5,4]},"u"],
  [{"loc":[5,6],"terminateAfterAction":false,"timeLeft":4,"previousLoc":[5,5]},"l"],
  [{"loc":[4,6],"terminateAfterAction":false,"timeLeft":3,"previousLoc":[5,6]},"u"],
  [{"loc":[4,7],"terminateAfterAction":false,"timeLeft":2,"previousLoc":[4,6],"timeAtRestaurant":0},"l"],
  [{"loc":[4,7],"terminateAfterAction":true,"timeLeft":2,"previousLoc":[4,7],"timeAtRestaurant":1},"l"]
];

var ___ = ' ';
var DN = { name : 'Donut N' };
var DS = { name : 'Donut S' };
var V = { name : 'Veg' };
var N = { name : 'Noodle' };

var grid = [
  ['#', '#', '#', '#',  V , '#'],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', DN , ___, '#', ___],
  ['#', '#', '#', ___, '#', ___],
  ['#', '#', '#', ___, ___, ___],
  ['#', '#', '#', ___, '#',  N ],
  [___, ___, ___, ___, '#', '#'],
  [DS , '#', '#', ___, '#', '#']
];

var mdp = makeGridWorldMDP({
  grid,
  noReverse: true,
  maxTimeAtRestaurant: 2,
  start: [3, 1],
  totalTime: 11
});
///

// Prior on agent's utility function
var priorUtility = function() {
  var utilityValues =  [-10, 0, 10, 20, 30];
  var donut = [uniformDraw(utilityValues), -10]
  var veg = [uniformDraw(utilityValues), 20];
  return {
    'Donut N': donut,
    'Donut S': donut,
    'Veg': veg,
    'Noodle': [-10, -10],
    'timeCost': uniformDraw([-0.01, 0.1, 1])
  };
};

var priorDiscounting = function() {
  return {
    discount: uniformDraw([0, 1]),
    sophisticatedOrNaive: 'sophisticated'
  };
};
var priorAlpha = function(){
  return uniformDraw([0.1, 100, 1000]);
};
var prior = {
  utility: priorUtility,
  discounting: priorDiscounting,
  alpha: priorAlpha
};

var posterior = getPosterior(mdp.world, prior, sophisticatedTrajectory);
displayResults(getPosterior(mdp.world, prior, []), posterior);
~~~~

Next chapter: [Multi-agent models](/chapters/7-multi-agent.html)

Owain Evans
Andreas Stuhlmüller
John Salvatier
Daniel Filan


---
layout: chapter
title: Modeling Agents & Reinforcement Learning with Probabilistic Programming
hidden: true
---

## Intro

### Motivation

Why probabilistic programming?
- **ML:** predictions based on prior assumptions and data
- **Deep Learning:** lots of data + very weak assumptions
- **Rule-based systems:** strong assumptions + little data
- **Probabilistic programming:** a flexible middle ground

Why model agents?
- Build **artificial agents** to automate decision-making
    - Example: stock trading
- **Model humans** to build helpful ML systems
    - Examples: recommendation systems, dialog systems

### Preview

What to get out of this talk:
- Intuition for programming in a PPL
- Core PPL concepts
- Why are PPLs uniquely suited for modeling agents?
- Idioms for writing agents as PPs
- How do RL and PP relate?

What not to expect:
- Lots of applications
- Production-ready systems

## Probabilistic programming basics

### Our language: WebPPL

Try it at [webppl.org](http://webppl.org)

### A functional subset of JavaScript

Why JS?
- Fast
- Rich ecosystem
- Actually a nice language underneath all the cruft
- Runs locally via node.js, but also in browser:
    - [SmartPages](https://stuhlmueller.org/smartpages/)
    - [Image inference viz](http://dippl.org/examples/vision.html)
    - [Spaceships](http://dritchie.github.io/web-procmod/)
    - [Agent viz](http://agentmodels.org/chapters/3b-mdp-gridworld.html#hiking-in-gridworld)

~~~~
var xs = [1, 2, 3, 4];

var square = function(x) {
  return x * x;
};

map(square, xs);
~~~~

### Distributions and sampling

Docs: [distributions](http://docs.webppl.org/en/dev/distributions.html)

#### Discrete distributions

Examples: `Bernoulli`, `Categorical`

Sampling helpers: `flip`, `categorical`

~~~~
var dist = Bernoulli({ p: 0.3 });

var flip = function(p) {
  return sample(Bernoulli({ p }));
}

flip(.3)
~~~~

#### Continuous distributions

Examples: `Gaussian`, `Beta`

~~~~
var dist = Gaussian({
  mu: 1,
  sigma: 0.5
});

viz(repeat(1000, function() { return sample(dist); }));
~~~~

#### Building complex distributions out of simple parts

Example: geometric distribution

~~~~
var geometric = function(p) {
  if (flip(p)) {
    return 0;
  } else {
    return 1 + geometric(p);
  }
};

viz(repeat(100, function() { return geometric(.5); }));
~~~~

### Inference

#### Reifying distributions

`Infer` reifies the geometric distribution so that we can compute probabilities:

~~~~
var geometric = function(p) {
  if (flip(p)) {
    return 0;
  } else {
    return 1 + geometric(p);
  }
};

var model = function() {
  return geometric(.5);
};

var dist = Infer({
  model,
  maxExecutions: 100
});

viz(dist);

Math.exp(dist.score(3))
~~~~

#### Computing conditional distributions

Example: inferring the weight of a geometric distribution

~~~~
var geometric = function(p) {
  if (flip(p)) {
    return 0;
  } else {
    return 1 + geometric(p);
  }
}

var model = function() {
  var u = uniform(0, 1);
  var x = geometric(u);
  condition(x < 4);
  return u;
}

var dist = Infer({
  model,
  method: 'rejection',
  samples: 1000
})

dist
~~~~

#### Technical note: three ways to condition

~~~~
var model = function() {
  var p = flip(.5) ? 0.5 : 1;
  var coin = Bernoulli({ p });

  var x = sample(coin);
  condition(x === true);

//  observe(coin, true);

//  factor(coin.score(true));

  return { p };
}

viz.table(Infer({ model }));
~~~~

#### A slightly less toy example: regression

Docs: [inference algorithms](http://docs.webppl.org/en/master/inference/methods.html)

~~~~
var xs = [1, 2, 3, 4, 5];
var ys = [2, 4, 6, 8, 10];

var model = function() {
  var slope = gaussian(0, 10);
  var offset = gaussian(0, 10);
  var f = function(x) {
    var y = slope * x + offset;
    return Gaussian({ mu: y, sigma: .1 })
  };
  map2(function(x, y){
    observe(f(x), y)
  }, xs, ys)
  return { slope, offset };
}

Infer({
  model,
  method: 'MCMC',
  kernel: {HMC: {steps: 10, stepSize: .01}},
  samples: 2000,
})
~~~~

## Agents as probabilistic programs

### Deterministic choices

~~~~
var actions = ['italian', 'french'];

var outcome = function(action) {
  if (action === 'italian') {
    return 'pizza';
  } else {
    return 'steak frites';
  }
};

var actionDist = Infer({
  model() {
    var action = uniformDraw(actions);
    condition(outcome(action) === 'pizza');
    return action;
  }
});

actionDist
~~~~

### Expected utility

~~~~
var actions = ['italian', 'french'];

var transition = function(state, action) {
  var nextStates = ['bad', 'good', 'spectacular'];
  var nextProbs = ((action === 'italian') ?
                   [0.2, 0.6, 0.2] :
                   [0.05, 0.9, 0.05]);
  return categorical(nextProbs, nextStates);
};

var utility = function(state) {
  var table = {
    bad: -10,
    good: 6,
    spectacular: 8
  };
  return table[state];
};

var expectedUtility = function(action) {
  var utilityDist = Infer({
    model: function() {
      var nextState = transition('initialState', action);
      var u = utility(nextState);
      return u;
    }
  });
  return expectation(utilityDist);
};

map(expectedUtility, actions);
~~~~

### Softmax-optimal decision-making

~~~~
var actions = ['italian', 'french'];

var transition = function(state, action) {
  var nextStates = ['bad', 'good', 'spectacular'];
  var nextProbs = ((action === 'italian') ?
                   [0.2, 0.6, 0.2] :
                   [0.05, 0.9, 0.05]);
  return categorical(nextProbs, nextStates);
};

var utility = function(state) {
  var table = {
    bad: -10,
    good: 6,
    spectacular: 8
  };
  return table[state];
};

var alpha = 1;

var agent = function(state) {
  return Infer({
    model() {

      var action = uniformDraw(actions);

      var expectedUtility = function(action) {
        var utilityDist = Infer({
          model: function() {
            var nextState = transition('initialState', action);
            var u = utility(nextState);
            return u;
          }
        });
        return expectation(utilityDist);
      };

      var eu = expectedUtility(action);

      factor(eu);

      return action;

    }
  });
};

agent('initialState');
~~~~

## Sequential decision problems

- [Restaurant Gridworld](http://agentmodels.org/chapters/3a-mdp.html) (1, last)
- Structure of expected utility recursion
- Dynamic programming

~~~~
var act = function(state) {
  return Infer({ model() {
    var action = uniformDraw(stateToActions(state));
    var eu = expectedUtility(state, action);
    factor(eu);
    return action;
  }});
};

var expectedUtility = function(state, action){
  var u = utility(state, action);
  if (isTerminal(state)){
    return u;
  } else {
    return u + expectation(Infer({ model() {
      var nextState = transition(state, action);
      var nextAction = sample(act(nextState));
      return expectedUtility(nextState, nextAction);
    }}));
  }
};
~~~~

- [Hiking Gridworld](http://agentmodels.org/chapters/3b-mdp-gridworld.html) (1, 2, 3, last)
- Expected state-action utilities (Q values)
- [Temporal inconsistency](http://agentmodels.org/chapters/5b-time-inconsistency.html) in Restaurant Gridworld

## Reasoning about agents

- [Learning about preferences from observations](http://agentmodels.org/chapters/4-reasoning-about-agents.html) (1 & 2)

## Multi-agent models

### A simple example: Coordination games

~~~~
var locationPrior = function() {
  if (flip(.55)) {
    return 'popular-bar';
  } else {
    return 'unpopular-bar';
  }
}

var alice = dp.cache(function(depth) {
  return Infer({ model() {
    var myLocation = locationPrior();
    var bobLocation = sample(bob(depth - 1));
    condition(myLocation === bobLocation);
    return myLocation;
  }});
});

var bob = dp.cache(function(depth) {
  return Infer({ model() {
    var myLocation = locationPrior();
    if (depth === 0) {
      return myLocation;
    } else {
      var aliceLocation = sample(alice(depth));
      condition(myLocation === aliceLocation);
      return myLocation;
    }
  }});
});

alice(5)
~~~~

### Other examples

- [Game playing: tic-tac-toe](http://agentmodels.org/chapters/7-multi-agent.html)
- [Language understanding](http://agentmodels.org/chapters/7-multi-agent.html)

## Reinforcement learning

### Algorithms vs Models

- Models: encode world knowledge
    - PPLs suited for expressing models
- Algorithms: encode mechanisms (for inference, optimization)
    - RL is mostly about algorithms
- But some algorithms can be expressed using PPL components

### Inference vs. Optimization

~~~~
var k = 3;  // number of heads
var n = 10; // number of coin flips

var model = function() {
  var p = sample(Uniform({ a: 0, b: 1}));
  var dist = Binomial({ p, n });
  observe(dist, k);
  return p;
};

var dist = Infer({
  model,
  method: ‘MCMC',
  samples: 100000,
  burn: 1000
});

expectation(dist);
~~~~

~~~~
var k = 3;  // number of heads
var n = 10; // number of coin flips

var model = function() {
  var p = Math.sigmoid(modelParam({ name: 'p' }));
  var dist = Binomial({ p, n });
  observe(dist, k);
  return p;
};

Optimize({
  model,
  steps: 1000,
  optMethod: { sgd: { stepSize: 0.01 }}
});

Math.sigmoid(getParams().p);
~~~~

### Policy Gradient

~~~~
///fold:
var numArms = 10;

var meanRewards = map(
  function(i) {
    if ((i === 7) || (i === 3)) {
      return 5;
    } else {
      return 0;
    }
  },
  _.range(numArms));

var blackBox = function(action) {
  var mu = meanRewards[action];
  var u = Gaussian({ mu, sigma: 0.01 }).sample();
  return u;
};
///

// actions: [0, 1, 2, ..., 9]

// blackBox: action -> utility

var agent = function() {
  var ps = softmax(modelParam({ dims: [numArms, 1], name: 'ps' }));
  var action = sample(Discrete({ ps }));
  var utility = blackBox(action);
  factor(utility);
  return action;
};

Optimize({ model: agent, steps: 10000 });

var params = getParams();
viz.bar(
  _.range(10),
  _.flatten(softmax(params.ps[0]).toArray()));
~~~~

## Conclusion

What to get out of this talk, revisited:

- **Intuition for programming in a PPL**
- **Core PPL concepts**
    - Distributions & samplers
    - Inference turns samplers into distributions
    - `sample` turns distributions into samples
    - Optimization fits free parameters
- **Idioms for writing agents as probabilistic programs**
    - Planning as inference
    - Sequential planning via recursion into the future
    - Multi-agent planning via recursion into other agents' minds
- **Why are PPLs uniquely suited for modeling agents?**
    - Agents are structured programs
    - Planning via nested conditional distributions
- **How do RL and PP relate?**
    - Algorithms vs models
    - Policy gradient as a PP

Where to go from here:
- [WebPPL](http://webppl.org) (webppl.org)
- [AgentModels](http://agentmodels.org) (agentmodels.org)
- andreas@ought.com


---
layout: chapter
title: Efficient inference
description: Difficulty of inference, in particular for POMDPs and inverse planning. Outline of inference strategies.
status: stub
is_section: true
hidden: true
---