Probability notation for Bayes' rule

[Bayes' rule](https://arbital.com/p/1lz) relates [prior](https://arbital.com/p/1rm) belief and the [likelihood](https://arbital.com/p/1rq) of evidence to [posterior](https://arbital.com/p/1rp) belief.

These quantities are often written using [conditional probabilities](https://arbital.com/p/1rj):

- [Prior](https://arbital.com/p/1rm) belief in the hypothesis:  $\mathbb P(H).$
- [Likelihood](https://arbital.com/p/1rq) of evidence, conditional on the hypothesis:  $\mathbb P(e \mid H).$
- [Posterior](https://arbital.com/p/1rp) belief in hypothesis, after seeing evidence:  $\mathbb P(H \mid e).$

For example, Bayes' rule in the [odds form](https://arbital.com/p/1x5) describes the relative belief in a hypothesis $H_1$ vs an alternative $H_2,$ given a piece of evidence $e,$ as follows:

$$\dfrac{\mathbb P(H_1)}{\mathbb P(H_2)} \times \dfrac{\mathbb P(e \mid H_1)}{\mathbb P(e \mid H_2)} = \dfrac{\mathbb P(H_1\mid e)}{\mathbb P(H_2\mid e)}.$$

Alonzo Church
Nate Soares
Alexei Andreev
Eliezer Yudkowsky

Summary:
[Bayes' rule](https://arbital.com/p/1lz) relates [prior](https://arbital.com/p/1rm) belief and the [likelihood](https://arbital.com/p/1rq) of evidence to [posterior](https://arbital.com/p/1rp) belief.

These quantities are often denoted using [conditional probabilities](https://arbital.com/p/1rj):

- [Prior](https://arbital.com/p/1rm) belief in hypothesis:  $\mathbb P(H).$
- [Likelihood](https://arbital.com/p/1rq) of evidence, conditional on hypothesis:  $\mathbb P(e \mid H).$
- [Posterior](https://arbital.com/p/1rp) belief:  $\mathbb P(H \mid e).$


Probability notation for Bayes' rule: Intro (Math 1)

To denote some of the quantities used in Bayes' rule, we'll need [conditional probabilities](https://arbital.com/p/1rj).  The conditional probability $\mathbb{P}(X\mid Y)$ means "The [probability](https://arbital.com/p/1rf) of $X$ given $Y$."  That is, $\mathbb P(\mathrm{left}\mid \mathrm{right})$ means "The probability that $\mathrm{left}$ is true, assuming that $\mathrm{right}$ is true."

$\mathbb P(\mathrm{yellow}\mid \mathrm{banana})$ is the probability that a banana is yellow - if we know something to be a banana, what is the probability that it is yellow?  $\mathbb P(\mathrm{banana}\mid \mathrm{yellow})$ is the probability that a yellow thing is a banana - if the right, known side is yellowness, then, we ask the question on the left, what is the probability that this is a banana?

In probability theory, the definition of "conditional probability" is that the conditional probability of $L,$ given $R,$ is found by looking at the probability of possibilities with both $L$ *and* $R$ *within* all possibilities with $R.$  Using $L \wedge R$ to denote the logical proposition "L and R both true":

$\mathbb P(L\mid R) = \frac{\mathbb P(L \wedge R)}{\mathbb P(R)}$

Suppose you have a bag containing objects that are either red or blue, and either square or round:

$$\begin{array}{l|r|r}
& Red & Blue \\
\hline
Square & 1 & 2 \\
\hline
Round & 3 & 4
\end{array}$$

If you reach in and feel a round object, the conditional probability that it is red is:

$\mathbb P(\mathrm{red} \mid  \mathrm{round}) = \dfrac{\mathbb P(\mathrm{red} \wedge \mathrm{round})}{\mathbb P(\mathrm{round})} \propto \dfrac{3}{3 + 4} = \frac{3}{7}$

If you look at the object nearest the top, and can see that it's blue, but not see the shape, then the conditional probability that it's a square is:

$\mathbb P(\mathrm{square} \mid  \mathrm{blue}) = \dfrac{\mathbb P(\mathrm{square} \wedge \mathrm{blue})}{\mathbb P(\mathrm{blue})} \propto \dfrac{2}{2 + 4} = \frac{1}{3}$

![conditional probabilities bag](https://i.imgur.com/zscEdLj.png?0)

# Updating as conditioning

Bayes' rule is useful because the process of *observing new evidence* can be interpreted as *conditioning a probability distribution.*

Again, the Diseasitis problem:

> 20% of the patients in the screening population start out with Diseasitis. Among patients with Diseasitis, 90% turn the tongue depressor black. 30% of the patients without Diseasitis will also turn the tongue depressor black. Among all the patients with black tongue depressors, how many have Diseasitis?

Consider a single patient, before observing any evidence.  There are four possible worlds we could be in, the product of (sick vs. healthy) times (positive vs. negative result):

$$\begin{array}{l|r|r}
& Sick & Healthy \\
\hline
Test + & 18\% & 24\% \\
\hline
Test - & 2\% & 56\%
\end{array}$$

To actually *observe* that the patient gets a negative result, is to eliminate from further consideration the possible worlds where the patient gets a positive result:

![bayes elimination](https://i.imgur.com/LGeGIzW.png?0)

Once we observe the result $\mathrm{positive}$, all of our future reasoning should take place, not in our old $\mathbb P(\cdot),$ but in our new $\mathbb P(\cdot \mid  \mathrm{positive}).$  This is why, after observing "$\mathrm{positive}$" and revising our probability distribution, when we ask about the probability the patient is sick, we are interested in the new probability $\mathbb P(\mathrm{sick}\mid \mathrm{positive})$ and not the old probability $\mathbb P(\mathrm{sick}).$

## Example: Socks-dresser problem

Realizing that *observing evidence* corresponds to [eliminating probability mass](https://arbital.com/p/1y6) and concerning ourselves only with the probability mass that remains, is the key to solving the [sock-dresser search](https://arbital.com/p/55b) problem:

> You left your socks somewhere in your room.  You think there's a 4/5 chance that they're in your dresser, so you start looking through your dresser's 8 drawers.  After checking 6 drawers at random, you haven't found your socks yet.  What is the probability you will find your socks in the next drawer you check?

We initially have 20% of the probability mass in "Socks outside the dresser", and 80% of the probability mass for "Socks inside the dresser".  This corresponds to 10% probability mass for each of the 8 drawers.

After eliminating the probability mass in 6 of the drawers, we have 40% of the original mass remaining, 20% for "Socks outside the dresser" and 10% each for the remaining 2 drawers.

Since this remaining 40% probability mass is now our whole world, the effect on our probability distribution is like amplifying the 40% until it expands back up to 100%, aka [renormalizing the probability distribution](https://arbital.com/p/1rk).  This is why we divide $\mathbb P(L \wedge R)$ by $\mathbb P(R)$ to get the new probabilities.

In this case, we divide "20% probability of being outside the dresser" by 40%, and then divide the 10% probability mass in each of the two drawers by 40%.  So the new probabilities are 1/2 for outside the dresser, and 1/4 each for the 2 drawers.  Or more simply, we could observe that, among the remaining probability mass of 40%, the "outside the dresser" hypothesis has half of it, and the two drawers have a quarter each.

So the probability of finding our socks in the next drawer is 25%.

Note that as we open successive drawers, we both become more confident that the socks are not in the dresser at all (since we eliminated several drawers they could have been in), *and* also expect more that we might find the socks in the next drawer we open (since there are so few remaining).

# Priors, likelihoods, and posteriors

Bayes' theorem is generally inquiring about some question of the form $\mathbb P(\mathrm{hypothesis}\mid \mathrm{evidence})$ - the $\mathrm{evidence}$ is known or assumed, so that we are now mentally living in the revised probability distribution $\mathbb P(\cdot\mid \mathrm{evidence}),$ and we are asking what we infer or guess about the $hypothesis.$  This quantity is the **[posterior probability](https://arbital.com/p/1rp)** of the $\mathrm{hypothesis}.$

To carry out a Bayesian revision, we also need to know what our beliefs were before we saw the evidence.  (E.g., in the Diseasitis problem, the chance that a patient who hasn't been tested yet is sick.)  This is often written $\mathbb P(\mathrm{hypothesis}),$ and the hypothesis's probability isn't being conditioned on anything because it is our **[prior](https://arbital.com/p/1rm)** belief.

The remaining pieces of key information are the **[likelihoods](https://arbital.com/p/1rq)** of the evidence, given each hypothesis.  To interpret the meaning of the positive test result as evidence, we need to imagine ourselves in the world where the patient is sick - *assume* the patient to be sick, as if that were known - and then ask, just as if we hadn't seen any test result yet, what we think the probability of the evidence would be in that world.  And then we have to do a similar operation again, this time mentally inhabiting the world where the patient is healthy.  And unfortunately, it so happens that the standard notation are such as to make this idea be denoted $\mathbb P(\mathrm{evidence}\mid \mathrm{hypothesis})$ - looking deceptively like the notation for the posterior probability, but written in the reverse order.  Not surprisingly, this trips people up a bunch until they get used to it.  (You would at least hope that the standard symbol $\mathbb P(\cdot \mid \cdot)$ wouldn't be *symmetrical,* but it is.  Alas.)

## Example

Suppose you're Sherlock Holmes investigating a case in which a red hair was left at the scene of the crime.

The Scotland Yard detective says, "Aha!  Then it's Miss Scarlet.  She has red hair, so if she was the murderer she almost certainly would have left a red hair there.  $\mathbb P(\mathrm{redhair}\mid \mathrm{Scarlet}) = 99\%,$ let's say, which is a near-certain conviction, so we're done."

"But no," replies Sherlock Holmes.  "You see, but you do not correctly track the meaning of the [conditional probabilities](https://arbital.com/p/1rj), detective.  The knowledge we require for a conviction is not $\mathbb P(\mathrm{redhair}\mid \mathrm{Scarlet}),$ the chance that Miss Scarlet would leave a red hair, but rather $\mathbb P(\mathrm{Scarlet}\mid \mathrm{redhair}),$ the chance that this red hair was left by Scarlet.  There are other people in this city who have red hair."

"So you're saying..." the detective said slowly, "that $\mathbb P(\mathrm{redhair}\mid \mathrm{Scarlet})$ is actually much lower than $1$?"

"No, detective.  I am saying that just because $\mathbb P(\mathrm{redhair}\mid \mathrm{Scarlet})$ is high does not imply that $\mathbb P(\mathrm{Scarlet}\mid \mathrm{redhair})$ is high.  It is the latter probability in which we are interested - the degree to which, *knowing* that a red hair was left at the scene, we *infer* that Miss Scarlet was the murderer.  The posterior, as the Bayesians say.  This is not the same quantity as the degree to which, *assuming* Miss Scarlet was the murderer, we would *guess* that she might leave a red hair.  That is merely the likelihood of the evidence, conditional on Miss Scarlet having done it."

## Visualization

Using the [waterfall](https://arbital.com/p/1wy) for the [Diseasitis problem](https://arbital.com/p/22s):

![waterfall labeled probabilities](https://i.imgur.com/f9E0ltp.png?0)

Eric Bruylant
Nate Soares
Alexei Andreev
Eliezer Yudkowsky


Eric Bruylant

summary: Eric Bruylant has been playing with wikis for almost a decade and thinks Arbital has great potential.

Eric Bruylant has been playing with wikis for almost a decade and thinks Arbital has great potential.


Bayes' rule: Vector form

%todo: This page conflates two concepts: (1) You can perform a Bayesian update on multiple hypotheses at once, by representing hypotheses via vectors; and (2) you can perform multiple Bayesian updates by multiplying by all the likelihood functions (and only normalizing once at the end). We should probably have one page for each concept, and we should possibly split this page in order to make them. (It's not yet clear whether we want one unified page for both ideas, as this one currently is.)%
%comment:  Comment from ESY: it seems to me that these two concepts are sufficiently closely related, and sufficiently combined in their demonstration, that we want to explain them on the same page.  They could arguably have different concept pages, though.%

[Bayes' rule](https://arbital.com/p/1lz) in the [odds form](https://arbital.com/p/1x5) says that for every pair of hypotheses, their relative prior odds, times the relative likelihood of the evidence, equals the relative posterior odds.

Let $\mathbf H$ be a vector of hypotheses $H_1, H_2, \ldots$ Because Bayes' rule holds between every pair of hypotheses in $\mathbf H,$ we can simply multiply an odds vector by a likelihood vector in order to get the correct posterior vector:

$$\mathbb O(\mathbf H) \times \mathcal L_e(\mathbf H) = \mathbb O(\mathbf H \mid e)$$

%comment: Comment from EN: It seems to me that the dot product would be more appropriate.%

where $\mathbb O(\mathbf H)$ is the vector of relative prior odds between all the $H_i$, $\mathcal L_e(\mathbf H)$ is the vector of relative likelihoods with which each $H_i$ predicted $e,$ and $\mathbb O(\mathbf H \mid e)$ is the relative posterior odds between all the $H_i.$

In fact, we can keep multiplying by likelihood vectors to perform multiple updates at once:

$$\begin{array}{r}
\mathbb O(\mathbf H) \\
\times\ \mathcal L_{e_1}(\mathbf H) \\
\times\ \mathcal L_{e_2}(\mathbf H \wedge e_1) \\
\times\ \mathcal L_{e_3}(\mathbf H \wedge e_1 \wedge e_2) \\
= \mathbb O(\mathbf H \mid e_1 \wedge e_2 \wedge e_3)
\end{array}$$

For example, suppose there's a bathtub full of coins.  Half of the coins are "fair" and have a 50% probability of producing heads on each coinflip. A third of the coins is biased towards heads and produces heads 75% of the time.   The remaining coins are biased against heads, which they produce only 25% of the time.  You pull out a coin at random, flip it 3 times, and get the result THT.  What's the chance that this was a fair coin?

We have three hypotheses, which we'll call $H_{fair},$  $H_{heads}$, and $H_{tails}$ and respectively, with relative odds of $(1/2 : 1/3 : 1/6).$ The relative likelihoods that these three hypotheses assign to a coin landing heads is $(2 : 3 : 1)$; the relative likelihoods that they assign to a coin landing tails is $(2 : 1 : 3).$ Thus, the posterior odds for all three hypotheses are:

$$\begin{array}{rll}
(1/2 : 1/3 : 1/6) = & (3 : 2 : 1) & \\
\times & (2 : 1 : 3) & \\
\times & (2 : 3 : 1) & \\
\times & (2 : 1 : 3) & \\
= & (24 : 6 : 9) & = (8 : 2 : 3) = (8/13 : 2/13 : 3/13)
\end{array}$$

...so there is an 8/13 or ~62% probability that the coin is fair.

If you were only familiar with the [probability form](https://arbital.com/p/554) of Bayes' rule, which only works for one hypothesis at a time and which only uses probabilities (and so [normalizes](https://arbital.com/p/1rk) the odds into probabilities at every step)...

$$\mathbb P(H_i\mid e) = \dfrac{\mathbb P(e\mid H_i)P(H_i)}{\sum_k \mathbb P(e\mid H_k)P(H_k)}$$

...then you might have had some gratuitous difficulty solving this problem.

Also, if you hear the idiom of "convert to odds, multiply lots and lots of things, convert back to probabilities" and think "hmm, this sounds like a place where [transforming into log-space](https://arbital.com/p/4h0) (where all multiplications become additions) might yield efficiency gains," then congratulations, you just invented the [log-odds from of Bayes' rule](https://arbital.com/p/1zh). Not only is it efficient, it also gives rise to a natural unit of measure for "strength of evidence" and "strength of belief".

# Naive Bayes

Multiplying an array of odds by an array of likelihoods is the idiom used in Bayesian spam filters.  Suppose that there are three categories of email, "Business", "Personal", and "Spam", and that the user hand-labeling the last 100 emails has labeled 50 as Business, 30 as Personal, and 20 as spam.  The word "buy" has appeared in 10 Business emails, 3 Personal emails, and 10 spam emails.  The word "rationality" has appeared in 30 Business emails, 15 Personal emails, and 1 spam email.

First, we assume that the frequencies in our data are representative of the 'true' frequencies.  (Taken literally, if we see a word we've never seen before, we'll be multiplying by a zero probability.  [Good-Turing frequency estimation](https://en.wikipedia.org/wiki/Good%E2%80%93Turing_frequency_estimation) would do better.)

Second, we make the *naive Bayes* assumption that a spam email which contains the word "buy" is no more or less likely than any other spam email to contain the word "rationality", and so on with the other categories.

Then we'd filter a message containing the phrase "buy rationality" as follows:

[Prior](https://arbital.com/p/1rm) odds: $(5 : 3 : 2)$

[Likelihood ratio](https://arbital.com/p/1rq) for "buy":

$$\left(\frac{10}{50} : \frac{3}{30} : \frac{10}{20}\right) = \left(\frac{1}{5} : \frac{1}{10} : \frac{1}{2}\right) = (2 : 1 : 5)$$

Likelihood ratio for "rationality":

$$\left(\frac{30}{50} : \frac{15}{30} : \frac{1}{20}\right) = \left(\frac{3}{5} : \frac{1}{2} : \frac{1}{20}\right) = (12 : 10 : 1)$$

Posterior odds:

$$(5 : 3 : 2) \times (2 : 1 : 5) \times (12 : 10 : 1) = (120 : 30 : 10) = \left(\frac{12}{16} : \frac{3}{16} : \frac{1}{16}\right)$$

%%comment:  12/16 is intentionally not in lowest form so that the 12 : 3 : 1 ratio can be clear.%%

This email would be 75% likely to be a business email, if the Naive Bayes assumptions are true.  They're almost certainly *not* true, for reasons discussed in more detail below.  But while Naive Bayes calculations are usually quantitatively wrong, they often point in the right qualitative direction - this email may indeed be more likely than not to be a business email.

(An actual implementation should add [log-likelihoods](https://arbital.com/p/1zh) rather than multiplying by ratios, so as not to risk floating-point overflow or underflow.)

# Non-naive multiple updates

To do a multiple update less naively, we must do the equivalent of asking about the probability that a Business email contains the word "rationality", *given* that it contained the word "buy".

As a real-life example, in a certain [rationality workshop](http://rationality.org/), one participant was observed to have taken another participant to a museum, and also, on a different day, to see their workplace.  A betting market soon developed on whether the two were romantically involved.  One participant argued that, as an eyeball estimate, someone was 12 times as likely to take a fellow participant to a museum, or to their workplace, if they were romantically involved, vs. just being strangers.  They then multiplied their prior odds by a 12 : 1 likelihood ratio for the museum trip and another 12 : 1 likelihood ratio for the workplace trip, and concluded that these two were almost certainly romantically attracted.

It later turned out that the two were childhood acquaintances who were not romantically involved.  What went wrong?

If we want to update hypotheses on multiple pieces of evidence, we need to mentally stay inside the world of each hypothesis, and condition the likelihood of future evidence on the evidence already observed.  Suppose the two are *not* romantically attracted.  We observe them visit a museum.  Arguendo, we might indeed suppose that this has a probability of, say, 1% (we don't usually expect strangers to visit museums together) which might be about 1/12 the probability of making that observation if the two were romantically involved.

But after this, when we observe the workplace visit, we need to ask about the probability of the workplace visit, *given* that the two were romantically attracted *and* that they visited a museum.  This might suggest that if two non-attracted people visit a museum together *for whatever reason*, they don't just have the default probability of a non-attracted couple of making a workplace visit.  In other words:

$$\mathbb P({workplace}\mid \neg {romance} \wedge {museum}) \neq \mathbb P({workplace}\mid \neg {romance})$$

Naive Bayes, in contrast, would try to approximate the quantity $\mathbb P({museum} \wedge {workplace} \mid \neg {romance})$ as the product of $\mathbb P({museum}\mid \neg {romance}) \cdot \mathbb P({workplace}\mid \neg {romance}).$  This is what the participants did when they multiplied by a 1/12 likelihood ratio twice.

The result was a kind of double-counting of the evidence — they took into account the prior improbability of a random non-romantic couple "going places together" twice in a row, for the two pieces of evidence, and ended up performing a total update that was much too strong.

Naive Bayes spam filters often end up assigning ludicrously extreme odds, on the order of googols to one, that an email is spam or personal; and then they're sometimes wrong anyways.  If an email contains the phrase "pharmaceutical" and "pharmacy", a spam filter will double-count the improbability of a personal email talking about pharmacies, rather than considering that if I actually do get a personal email talking about a pharmacy, it is much more likely to contain the word "pharmaceutical" as well. So because of the Naive Bayes assumption, naive Bayesian spam filters are not anything remotely like well-calibrated, and they update much too extremely on the evidence.  On the other hand, they're often extreme in the correct qualitative direction — something assigned googol-to-one odds of being spam isn't *always* spam but it might be spam, say, 99.999% of the time.

To do non-naive Bayesian updates on multiple pieces of evidence, just remember to mentally inhabit the world *where the hypothesis is true*, and then ask about the likelihood of each successive piece of evidence, *in* the world where the hypothesis is true *and* the previous pieces of evidence were observed.  Don't ask, "What is the likelihood that a non-romantic couple would visit one person's workplace?" but "What is the likelihood that a non-romantic couple which previously visited a museum for some unknown reason would also visit the workplace?"

In our example with the coins in the bathtub, the likelihoods of the evidence were independent on each step - *assuming* a coin to be fair, it's no more or less likely to produce heads on the second flip after producing heads on the first flip.  So in our bathtub-coins example, the Naive Bayes assumption was actually true.

Francis Marineau
Eliezer Yudkowsky
gia dang
Alexei Andreev
Connor Flexman
Erik Nash
Eric Bruylant
Nate Soares
Patrick LaVictoire
Ryan White
Nick Jordan
Nate Windwood
Fedor Belolutskiy
Adom Hartell

Summary:
The [odds form of Bayes' rule](https://arbital.com/p/1x5) works for odds ratios between more than two hypotheses, and applying multiple pieces of evidence.  Suppose there's a bathtub full of coins.  1/2 of the coins are fair and have a 50% probability of producing heads on each coinflip; 1/3 of the coins produce 25% heads; and 1/6 produce 75% heads.  You pull out a coin at random, flip it 3 times, and get the result HTH.  You may calculate:

$$\begin{array}{rll}
(1/2 : 1/3 : 1/6) = & (3 : 2 : 1) & \\
\times & (2 : 1 : 3) & \\
\times & (2 : 3 : 1) & \\
\times & (2 : 1 : 3) & \\
= & (24 : 6 : 9) & = (8 : 2 : 3)
\end{array}$$


Bayes' rule: Log-odds form

[The odds form](https://arbital.com/p/1x5) of [Bayes's Rule](https://arbital.com/p/1lz) states that the [prior](https://arbital.com/p/1rm) [odds](https://arbital.com/p/odds_ratio) times the [likelihood ratio](https://arbital.com/p/56t) equals the [posterior odds](https://arbital.com/p/1rp). We can take the log of both sides of this equation, yielding an equivalent equation which uses addition instead of multiplication.

Letting $H_i$ and $H_j$ denote hypotheses and $e$ denote evidence, the log-odds form of Bayes' rule states:

$$
\log \left ( \dfrac
   {\mathbb P(H_i\mid e)}
   {\mathbb P(H_j\mid e)}
\right )
=
\log \left ( \dfrac
   {\mathbb P(H_i)}
   {\mathbb P(H_j)}
\right )
+
\log \left ( \dfrac
   {\mathbb P(e\mid H_i)}
   {\mathbb P(e\mid H_j)}
\right ).
$$

This can be numerically efficient for when you're carrying out [lots of updates one after another](https://arbital.com/p/1zg).  But a more important reason to think in log odds is to get a better grasp on the notion of 'strength of evidence'.

# Logarithms of likelihood ratios

Suppose you're visiting your friends Andrew and Betty, who are a couple. They promised that one of them would pick you up from the airport when you arrive. You're not sure which one is in fact going to pick you up ([prior odds](https://arbital.com/p/1rm) of 50:50), but you do know three things:

1. They have both a blue car and a red car. Andrew prefers to drive the blue car, Betty prefers to drive the red car, but the correlation is relatively weak. (Sometimes, which car they drive depends on which one their child is using.) Andrew is 2x as likely to drive the blue car as Betty.
2. Betty tends to honk the horn at you to get your attention. Andrew does this too, but less often. Betty is 4x as likely to honk as Andrew.
3. Andrew tends to run a little late (more often than Betty). Betty is 2x as likely to have the car already at the airport when you arrive.

All three observations are independent as far as you know (that is, you don't think Betty's any more or less likely to be late if she's driving the blue car, and so on).

Let's say we see a blue car, already at the airport, which honks.

The odds form of this calculation would be a $(1 : 1)$ prior for Betty vs. Andrew, times likelihood ratios of $(1 : 2) \times (4 : 1) \times (2 : 1),$ yielding posterior odds of $(1 \times 4 \times 2 : 2 \times 1 \times 1) = (8 : 2) = (4 : 1)$, so it's 4/5 = 80% likely to be Betty.

Here's the log odds form of the same calculation, using 1 **bit** to denote each factor of $2$ in belief or evidence:

- Prior belief in Betty of $\log_2 (\frac{1}{1}) = 0$ bits.
- Evidence of $\log_2 (\frac{1}{2}) = {-1}$ bits against Betty.
- Evidence of $\log_2 (\frac{4}{1}) = {+2}$ bits for Betty.
- Evidence of $\log_2 (\frac{2}{1}) = {+1}$ bit for Betty.
- Posterior belief of $0 + {^-1} + {^+2} + {^+1} = {^+2}$ bits that Betty is picking us up.

If your posterior belief is +2 bits, then your posterior odds are $(2^{+2} : 1) = (4 : 1),$ yielding a posterior probability of 80% that Betty is picking you up.

Evidence and belief represented this way is *additive,* which can make it an easier fit for intuitions about "strength of credence" and "strength of evidence"; we'll soon develop this point in further depth.

# The log-odds line

Imagine you start out thinking that the hypothesis $H$ is just as likely as $\lnot H,$ its negation. Then you get five separate independent $2 : 1$ updates in favor of $H.$ What happens to your probabilities?

Your odds (for $H$) go from $(1 : 1)$ to $(2 : 1)$ to $(4 : 1)$ to $(8 : 1)$ to $(16 : 1)$ to $(32 : 1).$

Thus, your probabilities go from $\frac{1}{2} = 50\%$ to $\frac{2}{3} \approx 67\%$ to $\frac{4}{5} = 80\%$ to $\frac{8}{9} \approx 89\%$ to $\frac{16}{17} \approx 94\%$ to $\frac{32}{33} \approx 97\%.$

Graphically representing these changing probabilities on a line that goes from 0 to 1:

![0 updates](https://i.imgur.com/pJOVvl1.png)
![3 updates](https://i.imgur.com/9AxEpQA.png)
![5 updates](https://i.imgur.com/HHeyAzO.png)

We observe that the probabilities approach 1 but never get there — they just keep stepping across a fraction of the remaining distance, eventually getting all scrunched up near the right end.

If we instead convert the probabilities into log-odds, the story is much nicer. 50% probability becomes 0 bits of credence, and every independent $(2 : 1)$ observation in favor of $H$ shifts belief by one unit along the line.

![the log-odds line](https://i.imgur.com/TquetiU.png)

(As for what happens when we approach the end of the line, there isn't one!  0% probability becomes $-\infty$ bits of credence and 100% probability becomes $+\infty$ bits of credence.%%%knows-requisite([https://arbital.com/p/1r6](https://arbital.com/p/1r6)):%%note: This un-scrunching of the [https://arbital.com/p/-interval](https://arbital.com/p/-interval) $(0,1)$ into the entire [https://arbital.com/p/-real_line](https://arbital.com/p/-real_line) is done by an application of the [inverse logistic](https://arbital.com/p/inverse_logistic_function) function.%% %%%)

# Intuitions about the log-odds line

There are a number of ways in which this infinite log-odds line is a better place to anchor your intuitions about "belief" than the usual [1](https://arbital.com/p/0,) probability interval. For example:

- Evidence you are twice as likely to see if the hypothesis is true than if it is false is ${+1}$ bits of evidence and a ${^+1}$-bit update, regardless of how confident or unconfident you were to start with--the strength of new evidence, and the distance we update, shouldn't *depend on* our prior belief.
- If your credence in something is 0 bits--neither positive or negative belief--then you think the odds are 1:1.
- The distance between $0.01$ and $0.000001$ is much greater than the distance between $0.11$ and $0.100001.$

To expand on the final point: on the 0-1 probability line, the difference between 0.01 (a 1% chance) and 0.000001 (a 1 in a million chance) is roughly the same as the distance between 11% and 10%.  This doesn't match our sense for the intuitive strength of a claim:  The difference between "1 in 100!" and "1 in a million!" feels like a far bigger jump than the difference between "11% probability" and "a hair over 10% probabiility."

On the log-odds line, a 1 in 100 credibility is ${^-2}$ orders of magnitude, and a "1 in a million" credibility is ${^-6}$ orders of magnitude.  The distance between them is minus 4 orders of magnitude, that is, $\log_{10}(10^{-6}) - \log_{10}(10^{-2})$ yields ${^-4}$ magnitudes, or roughly ${^-13.3}$ bits.  On the other hand, 11% to 10% is $\log_{10}(\frac{0.10}{0.90}) - \log_{10}(\frac{0.11}{0.89}) \approx {^-0.954}-{^-0.907} \approx {^-0.046}$ magnitudes, or ${^-0.153}$ bits.

The log-odds line doesn't compress the vast differences available near the ends of the probability spectrum. Instead, it exhibits a "belief bar" carrying on indefinitely in both directions--every time you see evidence with a likelihood ratio of $2 : 1,$ it adds one more bit of credibility.

The [Weber-Fechner law](https://en.wikipedia.org/wiki/Weber%E2%80%93Fechner_law) says that most human sensory perceptions are logarithmic, in the sense that a factor-of-2 intensity change *feels* like around the same amount of increase no matter where you are on the scale.  Doubling the physical intensity of a sound feels to a human like around the same amount of change in that sound whether the initial sound was 40 decibels or 60 decibels.  That's why there's an exponential decibel scale of sound intensities in the first place!

Thus the log-odds form should be, in a certain sense, the *most* intuitive variant of Bayes' rule to use: Just add the evidence-strength to the belief-strength!  If you can make your feelings of evidence-strength and belief-strength be proportional to the logarithms of ratios, that is.

Finally, the log-odds representation gives us an even easier way to see how [extraordinary claims require extraordinary evidence](https://arbital.com/p/21v): If your prior belief in $H$ is -30 bits, and you see evidence on the order of +5 bits for $H$, then you're going to wind up with -25 bits of belief in $H$, which means you still think it's far less likely than the alternatives.

# Example: Blue oysters

Consider the [blue oyster](https://arbital.com/p/54v) example problem:

You're collecting exotic oysters in Nantucket, and there are two different bays from which you could harvest oysters.

- In both bays, 11% of the oysters contain valuable pearls and 89% are empty.
- In the first bay, 4% of the pearl-containing oysters are blue, and 8% of the non-pearl-containing oysters are blue.
- In the second bay, 13% of the pearl-containing oysters are blue, and 26% of the non-pearl-containing oysters are blue.

Would you rather have a blue oyster from the first bay or the second bay?  Well, we first note that the likelihood ratio from "blue oyster" to "full vs. empty" is $1 : 2$ in either case, so both kinds of blue oyster are equally valuable.  (Take a moment to reflect on how obvious this would *not* seem before learning about Bayes' rule!)

But what's the chance of (either kind of) a blue oyster containing a pearl?  Hint: this would be a good time to convert your credences into bits (factors of 2).

%%hidden(Answer):
89% is around 8 times as much as 11%, so we start out with ${^-3}$ bits of belief that a random oyster contains a pearl.

Full oysters are 1/2 as likely to be blue as empty oysters, so seeing that an oyster is blue is ${^-1}$ bits of evidence against it containing a pearl.

Posterior belief should be around ${^-4}$ bits or $(1 : 16)$ against, or a probability of 1/17... so a bit more than 5% (1/20) maybe?  (Actually 5.88%.)
%%

# Real-life example: HIV test

A study of Chinese blood donors %note: Citation needed% found that roughly 1 in 100,000 of them had HIV (as determined by a very reliable gold-standard test).  The non-gold-standard test used for initial screening had a *sensitivity* of 99.7% and a *specificity* of 99.8%, meaning respectively that $\mathbb P({positive}\mid {HIV}) = .997$ and $\mathbb P({negative}\mid \neg {HIV}) = .998$, i.e., $\mathbb P({positive} \mid  \neg {HIV}) = .002.$

That is: the prior odds are $1 : 100,000$ against HIV, and a positive result in an initial screening favors HIV with a likelihood ratio of $500 : 1.$

Using log base 10 (because those are [easier to do in your head](https://arbital.com/p/416)):

- The prior belief in HIV was about -5 magnitudes.
- The evidence was a tad less than +3 magnitudes strong, since 500 is less than 1,000.  ($\log_{10}(500) \approx 2.7$).

So the posterior belief in HIV is a tad underneath -2 magnitudes, i.e., less than a 1 in 100 chance of HIV.

Even though the screening had a $500 : 1$ likelihood ratio in favor of HIV, someone with a positive screening result _really_ should not panic!

Admittedly, this setup had people being screened randomly, in a relatively non-AIDS-stricken country.  You'd need separate statistics for people who are getting tested for HIV because of specific worries or concerns, or in countries where HIV is highly prevalent.  *Nonetheless,* the points that "only a tiny fraction of people have illness X" and that "preliminary observations Y may not have *correspondingly tiny* false positive rates" are worth remembering for many illnesses X and observations Y.

# Exposing infinite credences

The log-odds representation exposes the degree to which $0$ and $1$ are very unusual among the classical probabilities. For example, [if you ever assign probability absolutely 0 or 1 to a hypothesis, then no amount of evidence can change your mind about it, ever](https://arbital.com/p/4mq).

On the log-odds line, credences range from $-\infty$ to $+\infty,$ with the infinite extremes corresponding to probability $0$ and $1$ which can thereby be seen as "infinite credences". That's not to say that $0$ and $1$ probabilities should never be used. For an ideal reasoner, the probability $\mathbb P(X) + \mathbb P(\lnot X)$ should be 1 (where $\lnot X$ is the logical negation of $X$).%note:For us mere mortals, [consider avoiding extreme probabilities even then](https://arbital.com/p/http://lesswrong.com/lw/jr/how_to_convince_me_that_2_2_3/).% Nevertheless, these infinite credences of 0 and 1 behave like 'special objects' with a qualitatively different behavior from the ordinary credence spectrum. Statements like "After seeing a piece of strong evidence, my belief should never be exactly what it was previously" are false for extreme credences, just as statements like "subtracting 1 from a number produces a lower number" are false if you insist on regarding %%knows-requisite([https://arbital.com/p/1r6](https://arbital.com/p/1r6)): [$\aleph_0$](https://arbital.com/p/aleph_0) %% %%!knows-requisite([https://arbital.com/p/1r6](https://arbital.com/p/1r6)): infinity %% as a number.

# Evidence in decibels

E.T. Jaynes, in *Probability Theory: The Logic of Science* (section 4.2), reports that using decibels of evidence makes them easier to grasp and use by humans.

If an hypothesis has a likelihood ratio of $o$, then its evidence in decibels is given by the formula $e = 10\log_{10}(o)$.

In this scheme, multiplying the likelihood ratio by 2 means approximately adding 3dB. Multiplying by 10 means adding 10dB.

Jayne reports having used decimal logarithm first, for their  ease of calculation and having tried to switch to natural logarithms with the advent of pocket calculators. But decimal logarithms were found to be easier to grasp.

Eliezer Yudkowsky
Alexei Andreev
Eric Rogstad
Szymon Slawinski
Eric Bruylant
Nate Soares
Viktor Gregor
John Schmitt
Lars Øvlisen
Pierre Thierry
Joe Zeng

Summary:
[Bayes's Rule](https://arbital.com/p/1x5) states that the [prior odds](https://arbital.com/p/1rm) times the [likelihood ratio](https://arbital.com/p/56t) equals the [posterior odds](https://arbital.com/p/1rp).  We can take the logarithms of both sides of the equation to get an equivalent rule which uses addition instead of multiplication.  Letting $H_i$ and $H_j$ denote hypotheses and $e$ denote evidence:

$$
\log \left ( \dfrac
   {\mathbb P(H_i\mid e)}
   {\mathbb P(H_j\mid e)}
\right )
=
\log \left ( \dfrac
   {\mathbb P(H_i)}
   {\mathbb P(H_j)}
\right )
+
\log \left ( \dfrac
   {\mathbb P(e\mid H_i)}
   {\mathbb P(e\mid H_j)}
\right ).
$$

To use a real-life example, a study of Chinese blood donors found that roughly 1 in 100,000 of them had HIV (as determined by a very reliable gold-standard test). The non-gold-standard test used for initial screening had a sensitivity of 99.7% and a specificity of 99.8%, meaning that it was 500 times as likely to return positive for infected as non-infected patients.  Then our prior belief is -5 orders of magnitude against HIV, and if we then observe a positive test result, this is evidence of strength +2.7 orders of magnitude for HIV.  Our posterior belief is -2.3 orders of magnitude, or odds of less than 1 to a 100, against HIV.


Bayes' rule: Functional form

[Bayes' rule](https://arbital.com/p/1lz) generalizes to continuous [functions](https://arbital.com/p/3jy), and states, "The [posterior](https://arbital.com/p/1rp) probability [density](https://en.wikipedia.org/wiki/Probability_density_function) is *proportional* to the [likelihood](https://arbital.com/p/1rq) function times the [prior](https://arbital.com/p/1rm) probability [density](https://en.wikipedia.org/wiki/Probability_density_function)."

$$\mathbb P(H_x\mid e) \propto \mathcal L_e(H_x) \cdot \mathbb P(H_x)$$

## Example

Suppose we have a biased coin with an unknown bias $b$ between 0 and 1 of coming up heads on each individual coinflip.  Since the bias $b$ is a continuous variable, we express our beliefs about the coin's bias using a [probability density function](https://en.wikipedia.org/wiki/Probability_density_function) $\mathbb P(b),$ where $\mathbb P(b)\cdot \mathrm{d}b$ is the probability that $b$ is in the interval $[+ \mathrm{d}b](https://arbital.com/p/b)$ for $\mathrm db$ small. (Specifically, the probability that $b$ is in the interval $[b](https://arbital.com/p/a,)$ is $\int_a^b \mathbb P(b) \, \mathrm db.$)

By hypothesis, we start out completely ignorant of the bias $b,$ meaning that all initial values for $b$ are equally likely.  Thus, $\mathbb P(b) = 1$ for all values of $b,$ which means that $\mathbb P(b)\, \mathrm db = \mathrm db$ (e.g., the chance of $b$ being found in the interval from 0.72 to 0.76 is 0.04).

![plot y = 1 + x * 0, x = 0 to 1](https://i.imgur.com/jUHn9pq.png?0)

We then flip the coin, and observe it to come up tails. This is our first piece of evidence. The likelihood $\mathcal L_{t_1}(b)$ of observation $t_1$ given bias $b$ is a continuous function of $b$, equal to 0.4 if $b = 0.6,$ 0.67 if $b = 0.33,$ and so on (because $b$ is the probability of heads and the observation was tails).

Graphing the likelihood function $\mathcal L_{t_1}(b)$ as it takes in the fixed evidence $t_1$ and ranges over variable $b,$ we obtain the straightforward graph $\mathcal L_{t_1}(b) = 1 - b.$

![plot y = 1 - x, x = 0 to 1](https://i.imgur.com/piyKfWe.png?0)

If we multiply the likelihood function by the prior probability function as it ranges over $b$, we obtain a *relative probability* function on the posterior, $\mathbb O(b\mid t_1) = \mathcal L_{t_1}(b) \cdot \mathbb P(b) = 1 - b,$ which gives us the same graph again:

![plot y = 1 - x, x = 0 to 1](https://i.imgur.com/piyKfWe.png?0)

But this can't be our posterior *probability* function because it doesn't integrate to 1.  $\int_0^1 (1 - b) \, \mathrm db = \frac{1}{2}.$  (The area under a triangle is half the base times the height.)  [Normalizing](https://arbital.com/p/1rk) this relative probability function will give us the posterior probability function:

$\mathbb P(b \mid t_1) = \dfrac{\mathbb O(b \mid t_1)}{\int_0^1 \mathbb O(b \mid t_1) \, \mathrm db} = 2 \cdot (1 - f)$

![plot y = 2(1 - x), x = 0 to 1](https://i.imgur.com/PtBSP6M.png?0)

The shapes are the same, and only the *y*-axis labels have changed to reflect the different heights of the pre-normalized and normalized function.

Suppose we now flip the coin another two times, and it comes up heads then tails. We'll denote this piece of evidence $h_2t_3.$  Although these two coin tosses pull our beliefs about $b$ in opposite directions, they don't cancel out — far from it! In fact, one value of $b$ ("the coin is always tails") is completely eliminated by this evidence, and many extreme values of $b$ ("almost always heads" and "almost always tails") are hit badly. That is, while the heads and the coins tails pull our beliefs in opposite directions, they don't pull with the same strength on all possible values of $b.$

We multiply the old belief

![plot y = 2(1 - x), x = 0 to 1](https://i.imgur.com/PtBSP6M.png?0)

by the additional pieces of evidence

![](https://i.imgur.com/aOx1avR.png?0)

and

![](https://i.imgur.com/piyKfWe.png?0)

and obtain the posterior *relative* density

![plot y = 2(1 - x)x(1 - x), x = 0 to 1](https://i.imgur.com/Gi7VqGo.png?0)

which is proportional to the [normalized](https://arbital.com/p/1rk) posterior probability

![plot y = 12(1 - x)x(1 - x), x = 0 to 1](https://i.imgur.com/tQueclr.png?0)

Writing out the whole operation from scratch:

$$\mathbb P(b \mid t_1h_2t_3) = \frac{\mathcal L_{t_1h_2t_3}(b) \cdot \mathbb P(b)}{\mathbb P(t_1h_2t_3)} = \frac{(1 - b) \cdot b \cdot (1 - b) \cdot 1}{\int_0^1 (1 - b) \cdot b \cdot (1 - b) \cdot 1 \, \mathrm{d}b} = {12\cdot b(1 - b)^2}$$

Note that it's okay for a posterior probability density to be greater than 1, so long as the total probability *mass* isn't greater than 1.  If there's probability density 1.2 over an interval of 0.1, that's only a probability of 0.12 for the true value to be found in that interval.

Thus, intuitively, Bayes' rule "just works" when calculating the posterior probability density from the prior probability density function and the (continuous) likelihood ratio function. A proof is beyond the scope of this guide; refer to [Proof of Bayes' rule in the continuous case](https://arbital.com/p/).

Eliezer Yudkowsky
Alexei Andreev
Eric Rogstad
Eric Bruylant
Nate Soares

Summary:
[Bayes' rule](https://arbital.com/p/1lz) generalizes to continuous [functions](https://arbital.com/p/3jy), and says, "The [posterior](https://arbital.com/p/1rp) probability [density](https://en.wikipedia.org/wiki/Probability_density_function) is *proportional* to the [likelihood](https://arbital.com/p/1rq) function times the [prior](https://arbital.com/p/1rm) probability [density](https://en.wikipedia.org/wiki/Probability_density_function)."

$$\mathbb P(H_x\mid e) \propto \mathcal L_e(H_x) \cdot \mathbb P(H_x)$$


Bayes' rule: Proportional form

If $H_i$ and $H_j$ are hypotheses and $e$ is a piece of evidence, [Bayes' rule](https://arbital.com/p/1lz) states:

$$\dfrac{\mathbb P(H_i)}{\mathbb P(H_j)} \times  \dfrac{\mathbb P(e\mid H_i)}{\mathbb P(e\mid H_j)} = \dfrac{\mathbb P(H_i\mid e)}{\mathbb P(H_j\mid e)}$$

%%if-after([https://arbital.com/p/55z](https://arbital.com/p/55z)):
In the [Diseasitis problem](https://arbital.com/p/22s), we use this form of Bayes' rule to justify calculating the posterior odds of sickness via the calculation $(1 : 4) \times (3 : 1) = (3 : 4).$
%%

%%!if-after([https://arbital.com/p/55z](https://arbital.com/p/55z)):
In the [Diseasitis problem](https://arbital.com/p/22s), 20% of the patients in a screening population have Diseasitis, 90% of sick patients will turn a chemical strip black, and 30% of healthy patients will turn a chemical strip black.  We can use the form of Bayes' rule above to justify solving this problem via the calculation $(1 : 4) \times (3 : 1) = (3 : 4).$
%%

If instead of treating the ratios as odds, we actually calculate out the numbers for each term of the equation, we instead get the calculation $\frac{1}{4} \times \frac{3}{1} = \frac{3}{4},$ or $0.25 \times 3 = 0.75.$

If we try to directly interpret this, it says: "If a patient starts out 0.25 times as likely to be sick as healthy, and we see a test result that is 3 times as likely to occur if the patient is sick as if the patient is healthy, we conclude the patient is 0.75 times as likely to be sick as healthy."

This is valid reasoning, and we call it the *proportional* form of Bayes' rule.  To get the probability back out, we reason that if there's 0.75 sick patients to every 1 healthy patient in a bag, the bag comprises 0.75/(0.75 + 1) = 3/7 = 43% sick patients.

# Spotlight visualization

One way of looking at this result is that, since odds ratios are equivalent under multiplication by a positive constant, we can fix the right side of the odds ratio as equaling 1 and ask about what's on the left side.  This is what we do when seeing the calculation as $(0.25 : 1) \cdot (3 : 1) = (0.75 : 1),$ the form suggested by the theorem proved above.

We could visualize Bayes' rule as a pair of spotlights with different starting intensities, that go through lenses that amplify or reduce each incoming unit of light by a fixed multiplier.  In the [Diseasitis](https://arbital.com/p/22s) case, if we fix the right-side blue beam as having a starting intensity of 1 and a multiplying lens of 1, and we fix the left-side beam of having a starting intensity of 0.25 and a multiplying lens of 3x, then the result gives us a visualization of the calculation prescribed by Bayes' rule:

![bayes lights](https://i.imgur.com/BIgoE87.png?0)

Note the similarity to a [waterfall diagram](https://arbital.com/p/1wy).  The main thing the spotlight visualization adds is that we can imagine varying the absolute intensities of the lights and lenses, while preserving their relative intensities, in such a way as to make the right-side beams and lenses equal 1.

[draw the pre-proportional, odds form of the spotlight visualization.](https://arbital.com/p/fixme:)

%todo: add example problem in proportional/spotlight form%

# Usefulness in informal argument

The proportional form of Bayes' rule is perhaps the fastest way of describing Bayesian reasoning that sounds like it ought to be true.  If you were having a fictional character suddenly give a Bayesian argument in the middle of a story being read by many people who'd never heard of Bayes' rule, you might have them [say](http://hpmor.com/chapter/86):

> "Suppose the Dark Mark is certain to continue while the Dark Lord's sentience lives on, but a priori we'd only have guessed a twenty percent chance of the Dark Mark continuing to exist after the Dark Lord dies. Then the observation, "The Dark Mark has not faded" is five times as likely to occur in worlds where the Dark Lord is alive as in worlds where the Dark Lord is dead. Is that really commensurate with the prior improbability of immortality? Let's say the prior odds were a hundred-to-one against the Dark Lord surviving. If a hypothesis is a hundred times as likely to be false versus true, and then you see evidence five times more likely if the hypothesis is true versus false, you should update to believing the hypothesis is twenty times as likely to be false as true."

Similarly, if you were a doctor trying to explain the meaning of a positive test result to a patient, you might say:  "If we haven't seen any test results, patients like you are a thousand times as likely to be healthy as sick.  This test is only a hundred times as likely to be positive for sick as for healthy patients.  So now we think you're ten times as likely to be healthy as sick, which is still a pretty good chance!"

[Visual diagrams](https://arbital.com/p/1wy) and special notation for [odds](https://arbital.com/p/1x5) and [relative likelihoods](https://arbital.com/p/1rq) might make Bayes' rule more intuitive, but the proportional form is probably the most valid-*sounding* thing that *is* quantitatively correct that you can say in three sentences.

[write a from-scratch Standalone Intro of the proportional form of Bayes' rule in particular, using the Diseasitis example and going from frequency diagram to waterfall to spotlight, with no proofs, just to justify the proportional form.  add to Main a statement that if you can phrase things in proportional form, there exists a Standalone Intro that justifies it quickly.](https://arbital.com/p/fixme:)

Eliezer Yudkowsky
Alexei Andreev
Eric Rogstad
Eric Bruylant
Nate Soares
V N
Himanshu Chaturvedi
Viktor Riabtsev

Summary: Suppose that Professor Plum and Miss Scarlet are two suspects in a murder, and that we start out thinking that Professor Plum is twice as likely to have committed the murder as Miss Scarlet.  We then discover that the victim was poisoned.  We think that Professor Plum is around one-fourth as likely to use poison as Miss Scarlet.  Then after observing the victim was poisoned, we should think Plum is around half as likely to have committed the murder as Scarlet: $2 \times \dfrac{1}{4} = \dfrac{1}{2}.$  This reasoning is valid by [Bayes' rule](https://arbital.com/p/1lz).


Bayes' rule: Guide

Bayes' rule or Bayes' theorem is the law of probability governing *the strength of evidence* - the rule saying *how much* to revise our probabilities (change our minds) when we learn a new fact or observe new evidence.

You may want to learn about Bayes' rule if you are:

- A professional who uses statistics, such as a scientist or doctor;
- A computer programmer working in machine learning;
- A human being.

As Philip Tetlock found when studying "superforecasters", people who were especially good at predicting future events:

> The superforecasters are a numerate bunch: many know about Bayes' theorem and could deploy it if they felt it was worth the trouble. But they rarely crunch the numbers so explicitly. What matters far more to the superforecasters than Bayes' theorem is Bayes' core insight of gradually getting closer to the truth by constantly updating in proportion to the weight of the evidence.

— Philip Tetlock and Dan Gardner, [_Superforecasting_](https://arbital.com/p/https://en.wikipedia.org/wiki/Superforecasting)

## Learning Bayes' rule

This guide to Bayes' rule uses Arbital's technology to allow for multiple flavors of introduction. They vary by technical level, speed, and topics covered. After you pick your path, remember that you can still switch between pages, in particular by using the "Say what?" and "Go faster" buttons.

[https://arbital.com/p/multiple-choice](https://arbital.com/p/multiple-choice)

%%%wants-requisite([https://arbital.com/p/62d](https://arbital.com/p/62d)):
%%wants-requisite([https://arbital.com/p/62f](https://arbital.com/p/62f)):
%box:
*Your path will go over all forms of Baye's Rule, along with developing deep appreciation for its scientific usefulness. Your path will contain 12 pages:*

 * *Frequency diagrams: A first look at Bayes*
 * *Waterfall diagrams and relative odds*
 * *Introduction to Bayes' rule: Odds form*
 * *Bayes' rule: Proportional form*
 * *Extraordinary claims require extraordinary evidence*
 * *Ordinary claims require ordinary evidence*
 * *Bayes' rule: Log-odds form*
 * *Shift towards the hypothesis of least surprise*
 * *Bayes' rule: Vector form*
 * *Belief revision as probability elimination*
 * *Bayes' rule: Probability form*
 * *Bayesian view of scientific virtues*
%

%start-path([https://arbital.com/p/61b](https://arbital.com/p/61b))%
%%
%%!wants-requisite([https://arbital.com/p/62f](https://arbital.com/p/62f)):
%box:
No time to waste! Let's plunge directly into [a single-page abbreviated introduction to Bayes' rule](https://arbital.com/p/693).
%
%%
%%%

%%%!wants-requisite([https://arbital.com/p/62d](https://arbital.com/p/62d)):
%%wants-requisite([https://arbital.com/p/62f](https://arbital.com/p/62f)):
%box:
*Your path will teach you the basic odds form of Bayes' rule at a reasonable pace and then delve into the deep mysteries of the Bayes' Rule! Your path will contain 8 pages:*

* *Frequency diagrams: A first look at Bayes*
* *Waterfall diagrams and relative odds*
* *Introduction to Bayes' rule: Odds form*
* *Belief revision as probability elimination*
* *Extraordinary claims require extraordinary evidence*
* *Ordinary claims require ordinary evidence*
* *Shift towards the hypothesis of least surprise*
* *Bayesian view of scientific virtues*
%

%start-path([https://arbital.com/p/62f](https://arbital.com/p/62f))%
%%

%%!wants-requisite([https://arbital.com/p/62f](https://arbital.com/p/62f)):
%box:
*Your path will teach you the basic odds form of Bayes' rule at a reasonable pace. It will contain 3 pages:*

* *Frequency diagrams: A first look at Bayes*
* *Waterfall diagrams and relative odds*
* *Introduction to Bayes' rule: Odds form*
%

%start-path([https://arbital.com/p/62c](https://arbital.com/p/62c))%
%%
%%%

John John
Eliezer Yudkowsky
Alexei Andreev
Eric Rogstad
Tim Huegerich
Kira Kutscher
Eric Bruylant
Nate Soares
Joe Zeng
Chris Cooper


Path: Multiple angles on Bayes's Rule

Congratulations!  You've now learned multiple angles and approaches to Bayes' rule.  Please continue on to the next step of this path!

Summary: A page with all the details of Bayes' rule as requisites, to help autogenerate a sequence that includes all the requisites.


Ignorance prior

An ignorance prior is a a [prior](https://arbital.com/p/1rm) [probability function](https://arbital.com/p/1zj) on some problem of interest, usually with the intended properties of being simple to describe and facilitating good learning from the evidence.  A classic example would be the [inductive prior](https://arbital.com/p/21b) for [Laplace's Rule of Succession](https://arbital.com/p/21c).


Inductive prior

An "inductive prior" is a state of belief, before seeing any evidence, which is conducive to learning when the evidence finally appears.  A classic example would be observing a coin come up heads or tails many times.  If the coin is biased to come up heads 1/4 of the time, the inductive prior from [Laplace's Rule of Succession](https://arbital.com/p/21c) will start predicting future flips to come up tails with 3/4 probability.  The [maximum entropy prior](https://arbital.com/p/) for the coin, which says that every coinflip has a 50% chance of coming up heads and that all sequences of heads and tails are equally probable, will never start to predict that the next flip will be heads, even after observing the coin come up heads thirty times in a row.

The prior in [https://arbital.com/p/11w](https://arbital.com/p/11w) is another example of an inductive prior - far more powerful, far more complicated, and [entirely unimplementable on physically possible hardware](https://arbital.com/p/1mk).

Eliezer Yudkowsky
Nate Soares
Alexei Andreev


Laplace's Rule of Succession

# Theorem and proof

Suppose a sequence $X_1, \dots, X_n$ of binary values (0 or 1), e.g., a potentially non-fair coin which comes up heads or tails on each flip.

Laplace's Rule of succession says that if:

- 1.  We think each coinflip $X_i$ has an independent, unknown likelihood $f$ of coming up heads, and
- 2.  We think that all values of $f$ between 0 and 1 have equal probability density *a priori*...

Then, after observing $M$ heads and $N$ tails, the expected probability of heads on the next coinflip is:

$\dfrac{M + 1}{M + N + 2}$

Proof:

For a hypothetical value of $f$, each coinflip observed has a [likelihood](https://arbital.com/p/1rq) of $f$ if heads or $1 - f$ if tails.

The [prior](https://arbital.com/p/1rm) is uniform between 0 and 1, so a prior density of 1 everywhere.

By [Bayes's Rule](https://arbital.com/p/1zj), after seeing M heads and N tails, the [posterior](https://arbital.com/p/1rp) probability density over $f$ is *proportional* to $1 \cdot f^M(1 - f)^N.$

Then the [normalizing constant](https://arbital.com/p/1rk) is:  $\int_0^1 f^M(1 - f)^N \operatorname{d}\!f = \frac{M!N!}{(M + N + 1)!}.$

So the posterior probability density function is $f^M(1 - f)^N \frac{(M + N + 1)!}{M!N!}.$

Integrating this function, times $f,$ from 0 to 1, will yield the marginal probability of getting heads on the next flip.

The answer is thus:

$\dfrac{(M+1)!N!}{(M + N + 2)!} \cdot \dfrac{(M + N + 1)!}{M!N!} = \dfrac{M + 1}{M + N + 2}.$

# Simpler proof by combinatorics

Although Laplace's Rule of Succession was originally proved (by Thomas Bayes) by finding the posterior probability density and integrating, and the proof of Laplace's Rule illustrates the core idea of an [inductive prior](https://arbital.com/p/21b) in Bayesianism, a simpler intuition for the proof also exists.

Consider the problem originally posed by Thomas Bayes:  An initial billiard is rolled back and forth between the left and right edges of an ideal billiards table until friction brings it to a halt.  We then roll M + N additional billiard balls, and observe that M halt to the left of the initial billiard, and N halt to the right of it.  If this is all we know, what is the probability the next ball halts on the left, or right?

Suppose that we rolled a total of 5 additional billiards, and 2 halted to the left of the original, and 3 halted to the right.  Then, using **|** to symbolize the initial billiard, the billiards would have come to rest in the order:

- **LL|RRR**

Suppose we now roll a new billiard, symbolized by **+**, until it comes to a halt.  It's equally likely to appear at:

- **+LL|RRR**
- **L+L|RRR**
- **LL+|RRR**
- **LL|+RRR**
- **LL|R+RR**
- **LL|RR+R**
- **LL|RRR+**

This means there are 3 ways the ball could be ordered on the left of the **|**, and 4 ways it could be ordered on the right.  Since all left-to-right orderings of 7 randomly rolled billiard balls are equally likely a priori, we assign 3/7 probability that the ball comes to a rest on the left of the original ball's position.

See also the [Wikipedia page](https://en.wikipedia.org/wiki/Rule_of_succession).

# Use and abuse

Laplace's Rule of Succession assumes that all prior values of the frequency $f$ are undistinguished *a priori* in our subjective knowledge.

For example, Laplace used the rule to estimate a probability of the sun rising tomorrow, given that it had risen every day for the past 5000 years, and arrived at odds of around 1826251:1.  But today when we have physical knowledge of the Sun's operation, not every possible 'rate at which the Sun rises each day' is undistinguished.  Furthermore, even in Laplace's time, he should have perhaps thought it especially likely a priori that "the Sun always rises" and "the Sun never rises" were distinguished as unusually likely frequencies of the Sun rising, a priori.

The Rule of Succession follows from assuming approximate ignorance about prior frequencies.  It does not, of itself, justify this assumption. Variations of the rule of succession are obtainable by taking different priors, corresponding to different views of what should count as uninformative. See discussion on the Wikipedia page on [non-informative priors](https://www.wikipedia.com/en/Prior_probability#/Uninformative_priors). For example if starting with the Jeffreys' prior, then after observing $M$ heads and $N$ tails, the expected probability of heads on the next coinflip is:

$\dfrac{M + \dfrac{1}{2}}{M + N + 1}$

# Nomenclature

Laplace's Rule of Succession was the famous problem proved by Thomas Bayes in "An Essay towards solving a Problem in the Doctrine of Chances", read to the Royal Society in 1763, after Bayes's death.  Pierre-Simon Laplace, the first systematizer of what we now know as Bayesian reasoning, was so impressed by this theorem that he named the central theorem of his new discipline after Thomas Bayes.  The original theorem proven by Bayes was popularized by Laplace in arguments about the problem of induction, and so became known as Laplace's Rule of Succession.

Eliezer Yudkowsky
Alexei Andreev
Eric Bruylant
Jakub Łopuszański
Owen Cotton-Barratt

Summary: Suppose you roll an initial billiards ball on a billiards table, letting it bounce off the left and right sides until it comes to a halt.  You mark down where the billiard ball halted.  You then roll additional billiards, and [observe](https://arbital.com/p/1ly) that M come to rest to the left of the original billiard, and N halt to the right of the original billiard.  If this is all someone knows, what probability should they assign that the next billiard comes to halt on the left, or the right?  Laplace's Rule of succession says that the [odds](https://arbital.com/p/1rb) are (M + 1) : (N + 1) for left vs. right.  If we flip a coin with an unknown bias and observe 2 heads and 7 tails, our probability of getting heads next time is (2 + 1)/(2 + 7 + 2) = 3/11.  On the very first flip, it's 1 : 1 or 1/2.


Extraordinary claims require extraordinary evidence

[Bayes' rule](https://arbital.com/p/1lz) tells us how strong a piece of evidence has to be in order to support a given hypothesis.  This lets us see whether a piece of evidence is sufficient, or insufficient, to drive the probability of a hypothesis to over 50%.

For example, consider the [sparking widgets](https://arbital.com/p/559) problem:

> 10% of widgets are bad and 90% are good. 4% of good widgets emit sparks, and 12% of bad widgets emit sparks. Can you calculate in your head what percentage of sparking widgets are bad?

The prior odds are 1 : 9 for bad widgets vs. good widgets.

12% of bad widgets and 4% of good widgets emit sparks, so that's a likelihood ratio of 3 : 1 for sparking (bad widgets are three times as likely to emit sparks).

$(1 : 9 ) \times (3 : 1)  \ = \ (3 : 9) \ \cong \ (1 : 3)$ posterior odds for bad vs. good sparking widgets.  So 1/4 of sparking widgets are bad.

Bad widgets started out relatively rare:  1 in 10.  We applied a test — looking for sparks — that was only 3 times more likely to identify bad widgets as opposed to good ones. *The evidence was weaker than the prior improbability of the claim.*

This doesn't mean we toss out the evidence and ignore it.  It does mean that, after updating on the observation of sparkiness, we only gave 25% posterior probability to the widget being bad — the probability didn't go over 50%.

What would need to change to drive the probability of widget badness to over 50%?  We would need evidence with a more extreme likelihood ratio, more extreme than the (9 : 1) prior odds.  For example, if instead bad widgets were 50% likely to spark and good widgets were 5% likely to spark, the posterior odds would go to (10 : 9) or 53%.

In other words:  For a previously implausible proposition $X$ to end up with a high posterior probability, the likelihood ratio for the new evidence favoring $X$ over its alternatives, needs to be *more extreme* than the prior odds against $X.$

This is the quantitative argument behind the qualitative statement that "extraordinary claims require extraordinary evidence" (a claim popularized by Carl Sagan, which dates back to at least Pierre-Simon Laplace).

That is: an "[extraordinary claim](https://arbital.com/p/26y)" is one with a low [prior probability](https://arbital.com/p/1rm) in advance of considering the evidence, and "extraordinary evidence" is evidence with an extreme [likelihood ratio](https://arbital.com/p/1rq) favoring the claim over its alternatives.

# What makes evidence extraordinary?

The likelihood ratio is defined as:

$$\text{Likelihood ratio} = \dfrac{\text{Probability of seeing the evidence, assuming the claim is true}}{\text{Probability of seeing the evidence, assuming the claim is false}}$$

To obtain an extreme likelihood ratio, the bottom of the fraction has to be very low.  The top of the fraction being very high doesn't help much.  If the top of the fraction is 99% and the bottom is 70%, that's still not a very extreme ratio, and it doesn't help much if the top is 99.9999% instead.

So to get extremely strong evidence, we need to see an observation which is very _improbable,_ given "business as usual," but fairly likely according to the extraordinary claim. This observation would be deserving of the title, "extraordinary evidence".

# Example of an extraordinary claim and ordinary evidence: Bookcase aliens.

Consider the following hypothesis:  What if there are Bookcase Aliens who teleport into our houses at night and drop off bookcases?

Bob offers the following evidence for this claim:  "Last week, I visited my friend's house, and there was a new bookcase there. If there were no bookcase aliens, I wouldn't have expected that my friend would get a new bookcase. But if there are Bookcase Aliens, then the probability of my finding a new bookcase there was much higher. Therefore, my observation, 'There is a new bookcase in my friend's house,' is strong evidence supporting the existence of Bookcase Aliens."

In an intuitive sense, we have a notion that Bob's evidence "There is a new bookcase in my friend's house" is not as extraordinary as the claim "There are bookcase aliens" - that the evidence fails to lift the claim.  Bayes's Rule makes this statement precise.

Bob is, in fact, correct that his observation, "There's a new bookcase in my friend's house", is indeed evidence favoring the Bookcase Aliens.  Depending on how long it's been since Bob last visited that house, there might ceteris paribus be, say, a 1% chance that there would be a new bookcase there.  On the other hand, the Bookcase Aliens hypothesis might assign, say, 50% probability that the Bookcase Aliens would target this particular house among others.  If so, that's a [likelihood ratio](https://arbital.com/p/1rq) of 50:1 favoring the Bookcase Aliens hypothesis.

However, a reasonable prior on Bookcase Aliens would assign this a very *low* [prior probability](https://arbital.com/p/1rm) given our other, previous observations of the world.  Let's be conservative and assign odds of just 1 : 1,000,000,000 against Bookcase Aliens.  Then to raise our [posterior belief](https://arbital.com/p/1rp) in Bookcase Aliens to somewhere in the "pragmatically noticeable" range of 1 : 100, we'd need to see evidence with a cumulative likelihood ratio of 10,000,000 : 1 favoring the Bookcase Aliens.  50 : 1 won't cut it.

What would need to change for the observation "There's a new bookcase in my friend's house" to be convincing evidence of Bookcase Aliens, compared to the alternative hypothesis of "business as usual"?

As suggested by the Bayesian interpretation of [strength of evidence](https://arbital.com/p/22x), what we need to see is an observation which is nigh-impossible if there are *not* bookcase aliens.  We would have to believe that, [conditional](https://arbital.com/p/1rj) on "business as usual" being true, the [likelihood](https://arbital.com/p/1rq) of seeing a bookcase was on the order of 0.00000001%.  That would then take the [likelihood ratio](https://arbital.com/p/1rq), aka strength of evidence, into the rough vicinity of a billion to one favoring Bookcase Aliens over "business as usual".

We would still need to consider whether there might be *other* alternative hypotheses besides Bookcase Aliens and "business as usual", such as a human-operated Bookcase Conspiracy.  But at least we wouldn't be dealing with an observation that was so unsurprising (conditional on business as usual) as to be unable to support *any* kind of extraordinary claim.

However, if instead we suppose that Bookcase Aliens are allegedly 99.999999% probable to add a bookcase to Bob's friend's house, very little changes - the likelihood ratio is 99.99999% : 1% or 100 : 1 instead of 50 : 1.  To obtain an extreme likelihood ratio, we mainly need a tiny denominator rather than a big numerator.  In other words, "extraordinary evidence".

# What makes claims extraordinary?

An obvious next question is what makes a claim 'extraordinary' or 'ordinary'.  This is a [deep separate topic](https://arbital.com/p/26y), but as an example, consider the claim that the Earth is becoming warmer due to carbon dioxide being added to its atmosphere.

To evaluate the ordinariness or extraordinariness of this claim:

- We _don't_ ask whether the future consequences of this claim seem extreme or important.
- We _don't_ ask whether the policies that would be required to address the claim are very costly.
- We ask whether "carbon dioxide warms the atmosphere" or "carbon dioxide fails to warm the atmosphere" seems to conform better to the *deep, causal* generalizations we already have about carbon dioxide and heat.
- *If* we've already considered the deep causal generalizations like those, we don't ask about generalizations *causally downstream* of the deep causal ones we've already considered.  (E.g., we don't say, "But on every observed day for the last 200 years, the global temperature has stayed inside the following range; it would be 'extraordinary' to leave that range.")

These tests suggest that "Large amounts of added carbon dioxide will incrementally warm Earth's atmosphere" would have been an 'ordinary' claim in advance of trying to find any evidence for or against it - it's just how you would expect a greenhouse gas to work, more or less.  Thus, one is not entitled to demand a prediction made by this hypothesis that is wildly unlikely under any other hypothesis before believing it.

# Incremental updating

A key feature of the Bookcase Aliens example is that the followers of Bayes' rule acknowledges the observation of a new bookcase as being, locally, a single piece of evidence with a 50 : 1 likelihood ratio favoring Bookcase Aliens.  The Bayesian doesn't toss the observation out the window because it's *insufficient* evidence; it just gets accumulated into the pool.  If you visit house after house, and see new bookcase after new bookcase, the Bayesian slowly, incrementally, begins to wonder if something strange is going on, rather than dismissing each observation as 'insufficient evidence' and then forgetting it.

This stands in contrast to the instinctive way humans often behave, where, having concluded that they should not believe in Bookcase Aliens on the basis of the evidence in front of them, they discard that evidence entirely, denounce it, and say that it was never any evidence at all. (This is "[treating arguments like soldiers](https://arbital.com/p/https://wiki.lesswrong.com/wiki/Arguments_as_soldiers)" and acting like any evidence in favor of a proposition has to be "defeated.")

The Bayesian just says "yes, that is evidence in favor of the claim, but it's not quantitatively enough evidence." This idiom also stands in contrast to the practice of treating any concession an opponent makes as a victory.  If true claims are supposed to have all their arguments upheld and false claims are supposed to have all their enemy arguments defeated, then a single undefeated claim of *support* stands as a proof of victory, no matter how strong or weak the evidence that it provides. Not so with Bayesians — a Bayesian considers the bookcase observation to be locally a piece of evidence favoring Bookcase Aliens, just massively insufficient evidence.

# Overriding evidence

If you think that a proposition has prior odds of 1 to a $10^{100}$, and then somebody presents evidence with a likelihood ratio of $10^{94}$ to one favoring the proposition, you shouldn't say, "Oh, I guess the posterior odds are 1 to a million." You should instead question whether either (a) you were wrong about the prior odds or (b) the evidence isn't as strong as you assessed.

It's not that hard to end up believing a hypothesis that had very low prior odds. For example, whenever you look at the exact pattern of 10 digits generated by a random number generator, you're coming to believe a hypothesis that had prior odds on the order of ten billion to 1 against it.

But this should only happen with _true_ hypotheses. It's much rarer to find strong support for false hypotheses. Indeed, "strong evidence" is precisely "that sort of evidence we almost *never* see, when the proposition turns out to be false".

Imagine tossing a fair coin at most 300 times, and asking how often the sequence of heads and tails that it generates along the way, ever supports the false hypothesis "this coin comes up heads 3/4ths of the time" strongly over the true hypothesis "this coin is fair". As you can verify using [this code](https://arbital.com/p/https://gist.github.com/Soares/941bdb13233fd0838f1882d148c9ac14), the sequence of coinflips will at some point support the false hypothesis at the 10 : 1 level on about 8% of runs; it will at some point support the false hypothesis at the 100 : 1 level on about 0.8% of runs, and it will at some point support the false hypothesis at the 1000 : 1 level on about 0.08% of runs. (Note that we are less and less likely to be more and more deceived.)

Seeing evidence with a strength of $(10^{94} : 1)$ / 94 orders of magnitude / 312 [bits of evidence](https://arbital.com/p/1zh) supporting a false hypothesis should only happen to you, on average, once every IT DIDN'T HAPPEN.

Witnessing an observation that truly has a $10^{-94}$ probability of occurring if the hypothesis is false, in a case where the hypothesis *is in fact false,* is something that will not happen to anyone even once over the expected lifetime of this universe.

So if you think that the prior odds for a coin being unfair are $(1 : 10^{100})$ against, and then you see the coin flipped 312 times and coming up heads each time... you do not say, "Well, my new posterior odds are $(1 : 10^6)$ against the coin being unfair."  You say, "I guess I was wrong about the prior odds being that low."

Grady Simon
sdfsdf sdfsdf
Eliezer Yudkowsky
Alexei Andreev
Eric Bruylant
Nate Soares