Conditional Probability, Independence, and Bayes' Formula

Motivation: updating on partial information

A second limitation of the event-level view of probability-foundations motivates this note: information arrives partially. Once we learn the first die shows 3, probabilities should change. Sometimes, though, conditioning on a cleverly chosen event makes a hard unconditional probability easy — and sometimes we want to run this backwards, updating a hypothesis’s probability given evidence rather than an event’s probability given a cause.

IntuitionThe central questions

How should probabilities change when we learn that some event occurred, when do they not change, and how do we reverse a conditional probability to reason from evidence back to causes?

Tool	Packages	Summary statistics / rules
conditional probability	an update given $A$	chain rule; $P(\cdot\mid A)$ is itself a probability
independence	”the update changes nothing”	product rules for events and pmfs
total probability + Bayes	reasoning by cases; inverting conditionals	odds × likelihood ratio

NoteRandom variables and their distributions live in a companion note

This note assumes only the sample-space/event machinery of probability-foundations. Random variables, pmfs, expectation, and the named discrete distributions (Bernoulli, binomial, geometric, …) are covered in discrete-random-variables-and-distributions; the “independent random variables” section below borrows that note’s pmf notation.

Conditional probabilities

DefinitionConditional probability

P(B\mid A) = \begin{cases}\dfrac{P(AB)}{P(A)} & P(A) > 0,\\[4pt] \text{undefined} & P(A) = 0.\end{cases}

IntuitionReduced sample space

Knowing $A$ occurred means $\omega\in A$ . Whether $B$ holds now depends only on $AB$ , renormalized so that $A$ itself has probability 1. When outcomes are equally likely, conditioning on $A$ makes the outcomes of $A$ equally likely, so you can just count inside $A$ .

WarningUndefined means undefined

Hajek: a routine computing $P(B\mid A)$ from $P(AB) = 0$ and $P(A) = 0$ should raise an error (“conditioning on an event of probability zero”), not return 0. The error helps catch bugs upstream.

ExampleConditioning can lower, raise, or leave unchanged (Hajek)

Two dice. $A$ = “sum is 6” $= \{15,24,33,42,51\}$ , so $P(A) = \frac5{36}$ .

Event $B$	$P(B)$	$P(B\mid A)$	Effect
numbers differ	$\frac56$	$\frac{4/36}{5/36} = \frac45$	lowered
numbers equal	$\frac16$	$\frac{1/36}{5/36} = \frac15$	raised

With $E$ = “first die even” and $F$ = “sum is 7”: $P(F\mid E) = \frac{3/36}{1/2} = \frac16 = P(F)$ . Unchanged.

ExampleClassic warm-ups (Ross)

Key in a pocket: 80% sure it’s in one of two pockets, 40% each. The left pocket is empty, so $P(R\mid L^c) = \frac{.4}{.6} = \frac23$ .
First die is 3, sum is 8: in the reduced space $\{(3,1),\dots,(3,6)\}$ , the probability is $\frac16$ .
Bridge: North–South hold 8 spades, so East–West share 26 cards with 5 spades. $P(\text{East has 3}) = \binom53\binom{21}{10}/\binom{26}{13} \approx.339$ .

Warning“At least one” is not “the first”

Two coin flips: $P(\text{both heads}\mid\text{first heads}) = \frac12$ , but $P(\text{both heads}\mid\text{at least one head}) = \frac13$ . “At least one head” leaves three equally likely outcomes (HH, HT, TH), not two equally likely cases (“both” vs “one”).

Theorem $P(\cdot\mid A)$ is a probability measure (Ross Prop. 5.1)

For fixed $A$ with $P(A) > 0$ :

$0\le P(B\mid A)\le1$ ;
$P(\Omega\mid A) = 1$ ;
for disjoint $E_1, E_2, \dots$ : $P\big(\bigcup_iE_i\mid A\big) = \sum_iP(E_i\mid A)$ .

So every theorem about probabilities holds for conditional probabilities: complements, inclusion–exclusion, the total-probability rule below. In particular, $Q(E_1\mid E_2) = P(E_1\mid E_2A)$ when $Q = P(\cdot\mid A)$ .

ProofCheck the axioms

(1) $0\le P(BA)\le P(A)$ since $BA\subseteq A$ . (2) $P(\Omega A)/P(A) = 1$ . (3) $\big(\bigcup_iE_i\big)A = \bigcup_iE_iA$ , a disjoint union, so countable additivity of $P$ passes through the division by $P(A)$ .

∎

IntuitionA new view of the experiment

Learning $A$ replaces the probability measure $P$ with $P(\cdot\mid A)$ , a genuine probability measure on the same space.

NoteFrequency interpretation

In $n$ repetitions, $A$ occurs about $nP(A)$ times and $AB$ about $nP(AB)$ times. Among the runs where $A$ occurs, the fraction with $B$ is about $P(AB)/P(A)$ . So the definition matches long-run frequencies.

The multiplication (chain) rule

FormulaChain rule

P(AB) = P(A)P(B\mid A),\qquad P(E_1E_2\cdots E_n) = P(E_1)P(E_2\mid E_1) P(E_3\mid E_1E_2)\cdots P(E_n\mid E_1\cdots E_{n-1}).

ProofTelescoping

Apply the definition to each factor: $P(E_1)\frac{P(E_1E_2)}{P(E_1)}\frac{P(E_1E_2E_3)}{P(E_1E_2)}\cdots$ . All the intermediate terms cancel.

∎

IntuitionBuild the event one stage at a time

The chain rule is the probabilistic counting principle: multiply the chance of each stage, given everything before it.

ExampleChain-rule computations (Ross)

Celine takes chemistry with probability $\frac12$ and then gets an A with probability $\frac23$ : $P(CA) = \frac13$ .
Two reds from 8 red + 4 white: $\frac8{12}\cdot\frac7{11} = \frac{14}{33}$ . If balls are drawn with probability proportional to weight ( $r$ for red, $w$ for white), the answer is $\frac{8r}{8r+4w}\cdot\frac{7r}{7r+4w}$ .
Each of 4 piles of 13 gets an ace: place the aces one by one. The ace of hearts avoids the ace of spades’ pile with probability $\frac{39}{51}$ , the next ace with $\frac{26}{50}$ , the last with $\frac{13}{49}$ : $P = \frac{39\cdot26\cdot13}{51\cdot50\cdot49}\approx.105.$
No two of the 4 strong teams meet among 8 random quarterfinalists: $\frac47\cdot\frac35\cdot\frac23\cdot1 = \frac8{35}$ .

ExampleExactly $k$ matches in the hat problem (Ross)

Let $P_N = \sum_{i=0}^N(-1)^i/i!$ be the no-match probability (probability-foundations). For a specified set of $k$ people, by the chain rule, $P(\text{all }k\text{ match}) = \frac1N\frac1{N-1}\cdots\frac1{N-k+1} = \frac{(N-k)!}{N!}$ . Given that, the rest form an $(N-k)$ -person problem. So

P(\text{exactly }k\text{ matches}) = \binom Nk\frac{(N-k)!}{N!}P_{N-k} = \frac{P_{N-k}}{k!}\;\approx\;\frac{e^{-1}}{k!}.

Mutually independent events

Motivation. If $P(B\mid A) = P(B)$ , learning $A$ tells us nothing about $B$ . That is equivalent to $P(AB) = P(A)P(B)$ . What about $P(A) = 0$ ? Then $P(B\mid A^c) = P(A^cB)/1 = P(B) - P(AB) = P(B)$ . Learning “not $A$ ” changes nothing, so $A$ should count as independent of $B$ .

DefinitionIndependence of two events

$A$ and $B$ are (mutually) independent if $P(AB) = P(A)P(B).$ This definition is symmetric and works even when $P(A) = 0$ . Independence is a property of a pair of events; a single event isn’t “independent” by itself.

ImportantPhysical vs probabilistic independence

Physically separate parts of an experiment (two dice, a coin and a die) are modeled as independent. But independence is just one equation, and it can hold without physical separation. On one die, $A = \{2,4,6\}$ and $B = \{3,6\}$ have $P(AB) = \frac16 = \frac12\cdot \frac13$ , so they are independent.

ExampleIndependent or not? (Ross)

Experiment	Events	Independent?
one card	ace, spade	yes: $\frac1{52} = \frac4{52}\cdot\frac{13}{52}$
two coins	first heads, second tails	yes: $\frac14 = \frac12\cdot\frac12$
two dice	sum 6, first die 4	no: $\frac1{36}\ne\frac5{36}\cdot\frac16$
two dice	sum 7, first die 4	yes: $\frac1{36} = \frac16\cdot\frac16$

IntuitionWhy “sum 7” is special

Whatever the first die shows, exactly one face of the second die completes a 7. So the first die gives no information. For “sum 6,” a first die of 6 kills the chance entirely.

PropositionComplements preserve independence

If $A$ and $B$ are independent, so are $A^c$ and $B$ , $A$ and $B^c$ , and $A^c$ and $B^c$ . All four conditions are equivalent.

ProofSubtract off the intersection

$P(A^cB) = P(B) - P(AB) = (1 - P(A))P(B) = P(A^c)P(B)$ . Apply the same argument again for the other two.

∎

Three or more events

DefinitionPairwise independence

$A, B, C$ are pairwise independent if each pair satisfies the product rule.

WarningPairwise does not imply independent

Two coins (Hajek): $A$ = first heads, $B$ = second heads, $C$ = both coins match. Each has probability .5, and each pairwise intersection has probability .25, so they are pairwise independent. But $P(A\mid BC) = 1\ne P(A)$ .

Two dice (Ross): $E$ = sum 7, $F$ = first die 4, $G$ = second die 3. Again pairwise independent, but $P(E\mid FG) = 1$ .

DefinitionIndependence of $n$ events

$A_1,\dots,A_n$ are independent if for every subcollection $1\le i_1 < \cdots < i_k\le n$ with $k\ge2$ , $P(A_{i_1}\cdots A_{i_k}) = P(A_{i_1})\cdots P(A_{i_k}).$ For three events this is four equations: three pairs plus the triple. An infinite family is independent if every finite subfamily is.

TheoremGrouping

If $A_1,\dots,A_n$ are independent, then events built by Boolean operations from disjoint blocks of them are again independent. For example, $A$ is independent of $BC$ and of $B\cup C$ .

Proof

A

independent of

B\cup C

P(A(B\cup C)) = P(AB) + P(AC) - P(ABC) = P(A)\big[P(B) + P(C) - P(B)P(C)\big] = P(A)P(B\cup C).

∎

Independent trials

DefinitionIndependent subexperiments and trials

Subexperiments are independent if events $E_i$ , each determined only by the $i$ -th subexperiment, are always independent. When every subexperiment has the same outcome set, they are called trials.

FormulaSuccess probabilities in independent trials

With success probability $p$ per trial:

$P(\text{at least one success in }n) = 1 - (1-p)^n$ ;
$P(\text{exactly }k\text{ successes in }n) = \binom nkp^k(1-p)^{n-k}$ (the binomial probability);
$P(\text{all trials succeed}) = \lim_np^n = 0$ if $p < 1$ , by probability-foundations.

DerivationThe binomial probability

Any particular sequence with $k$ successes and $n-k$ failures has probability $p^k(1-p)^{n-k}$ by independence. There are $\binom nk$ such sequences, and they are disjoint.

ExampleParallel system (Ross)

The system works if any of $n$ independent components works (probabilities $p_i$ ): $P(\text{works}) = 1 - P\Big(\bigcap_iA_i^c\Big) = 1 - \prod_i(1 - p_i).$

ExampleThe probabilistic method (Ross)

Color each edge of the complete graph $K_n$ red or blue independently with probability $\frac12$ . A fixed $k$ -set of vertices is monochromatic with probability $2\cdot2^{-\binom k2}$ . By Boole’s inequality, $P(\text{some monochromatic }k\text{-set})\le\binom nk2^{1-k(k-1)/2} < 1\quad\text{if}\quad\binom nk < 2^{k(k-1)/2-1}.$ Then a coloring with no monochromatic $k$ -set exists. The argument proves existence without constructing one. (The same method gives the lower bound in cost-sharing.)

Independent random variables (of discrete type)

DefinitionIndependence of random variables (Hajek Def. 2.4.8)

$X$ and $Y$ are independent if every event $\{X\in A\}$ is independent of every event $\{Y\in B\}$ . More generally, $X_1,\dots,X_n$ are mutually independent if $\{X_1\in A_1\},\dots,\{X_n\in A_n\}$ are always independent events.

TheoremFactorization of the joint pmf

Discrete $X, Y$ are independent iff $P\{X = i, Y = j\} = p_X(i)\,p_Y(j)\quad\text{for all }i,j.$

ProofPoint masses suffice

(⇒) Take $A = \{i\}$ and $B = \{j\}$ .

(⇐) For any $A, B$ :

P\{X\in A, Y\in B\} = \sum_{i\in A}\sum_{j\in B}p_X(i)p_Y(j) = \Big(\sum_{i\in A}p_X(i)\Big)\Big(\sum_{j\in B}p_Y(j)\Big) = P\{X\in A\} P\{Y\in B\}.

The key step: a double sum of products factors into a product of sums.

∎

ExampleCoin and die (Hajek)

A flip $N$ and a roll $X$ , with all 12 outcomes equally likely: $P\{N = H, X = 6\} = \frac1{12} = \frac12\cdot\frac16$ . Every $(N,X)$ pair factors, so any event about $N$ is independent of any event about $X$ .

RemarkWhy $E[X_1X_2] = E[X_1]E[X_2]$ for dice

The factorization $\frac1{36} = \frac16\cdot\frac16$ is what let $\sum_{i,j}ij$ split into $\big(\sum i\big)\big(\sum j\big)$ in the product example. The general statement ( $E[XY] = E[X]E[Y]$ for independent $X,Y$ ) comes later in Hajek, Section 4.4.

The law of total probability, and Bayes’ formula

Total probability

TheoremLaw of total probability

For any event $F$ with $0 < P(F) < 1$ : $P(E) = P(E\mid F)P(F) + P(E\mid F^c)\big[1 - P(F)\big]. \tag{3.1}$ More generally, for a partition $F_1,\dots,F_n$ of $\Omega$ : $P(E) = \sum_{i=1}^nP(E\mid F_i)P(F_i). \tag{3.4}$

ProofSplit

E

along the partition

$E = \bigcup_iEF_i$ with the pieces disjoint, so $P(E) = \sum_iP(EF_i) = \sum_iP(E\mid F_i)P(F_i)$ by the chain rule.

∎

IntuitionReason by cases

$P(E)$ is a weighted average of the conditional probabilities, each weighted by the probability of its case. Choose the cases so that each conditional is easy.

ExampleReasoning by cases (Ross)

Accident-prone policyholders: 30% are accident prone (accident rate .4), the rest have rate .2. So $P(\text{accident}) = .4\cdot.3 + .2\cdot.7 = .26$ .
Flashlights of 3 types: 20%, 30%, 50% of stock with lifetime probabilities .7, .4, .3: $P(\text{lasts}) = .14 + .12 + .15 = .41$ .
Twins: identical twins are always the same sex, fraternal twins are the same sex with probability $\frac12$ . So $P(SS) = P(I) + \frac12(1 - P(I))$ . Observing $P(SS)\approx.64$ gives $P(I) = 2P(SS) - 1\approx.28$ . This estimates something unobservable (DNA identity) from something cheap (sex).
Card after the first ace: condition on the ordering $\mathbf O$ of the other 51 cards. Given $\mathbf O$ , the specified card is equally likely to be in any of 52 slots, and exactly one follows the first ace. So $P = 1/52$ , matching the counting argument.

ExampleEvery strategy wins with probability $1/n$ (Ross)

Cards are turned over one at a time, and you may once predict “the next card is the ace of spades.” If you never guess, you win if the ace is last. Claim: for an $n$ -card deck, every strategy wins with probability $1/n$ .

Induction: let $p$ be the probability your strategy guesses on card 1. Guessing wins with probability $1/n$ . Not guessing requires the first card not to be the ace, $\frac{n-1}n$ , and then you face an $(n-1)$ -card deck, $\frac1{n-1}$ . So $P(\text{win}) = p\cdot\frac1n + (1-p)\frac{n-1}n\frac1{n-1} = \frac1n.$

Conditional expectation and total expectation

DefinitionConditional mean given an event (Hajek §2.10)

For an event $A$ with $P(A) > 0$ ,

E[X\mid A] = \sum_iu_i\,P(X = u_i\mid A),\qquad E[g(X)\mid A] = \sum_i g(u_i)\,P(X = u_i\mid A).

This is the ordinary mean computed with the conditional pmf. Conditional LOTUS holds because $P(\cdot\mid A)$ is a probability measure.

TheoremLaw of total expectation

For a partition $E_1,\dots,E_J$ of $\Omega$ with each $P(E_j) > 0$ , $E[X] = \sum_{j=1}^JE[X\mid E_j]\,P(E_j).$

ProofTotal probability inside the mean

Substitute $P(X = u_i) = \sum_jP(X = u_i\mid E_j)P(E_j)$ into $E[X] = \sum_iu_iP(X = u_i)$ and swap the order of summation.

∎

NoteAlready in use

First-step analysis of the geometric mean and variance (discrete-random-variables-and-distributions) is exactly this rule, with the partition {first trial succeeds, fails}.

ExampleDice with red faces (Hajek)

A bag holds three dice with 1, 2 and 3 red faces. One is drawn at random and rolled repeatedly. Let $E_i$ = the die with $i$ red faces, and $R_j$ = red on roll $j$ .

$P(R_1) = \frac13\big(\frac16 + \frac26 + \frac36\big) = \frac13$ .
$P(R_1R_2) = \frac13\big(\frac1{36} + \frac4{36} + \frac9{36}\big) = \frac7{54}$ , so $P(R_2\mid R_1) = \frac7{18} > \frac13$ .
$P(R_1R_2R_3) = \frac13\cdot\frac{1 + 8 + 27}{216} = \frac1{18}$ and $P(E_3R_1R_2R_3) = \frac13\big(\frac12\big)^3 = \frac1{24}$ , so $P(E_3\mid R_1R_2R_3) = \frac34$ .

The rolls are conditionally independent given the die, but not independent: red makes the redder die, and hence more red, likelier.

ExampleA die, then that many coins (Hajek)

Roll a die ( $X$ ), then flip $X$ fair coins, and let $Y$ = the number of heads. By total probability,

P\{Y = 3\} = \frac16\Big[0 + 0 + \frac18 + \frac4{16} + \frac{10}{32} + \frac{20}{64}\Big] = \frac16,

and $P(X = 3\mid Y = 3) = \frac{1/48}{1/6} = \frac18$ .

ExampleSmoking and lung cancer (Hajek, CDC figures)

Women who smoke are about 13 times likelier to develop lung cancer, and about 15% of women smoke. If nonsmokers’ rate is $c$ ,

P(\text{smoker}\mid\text{cancer}) = \frac{.15\cdot13c}{.15\cdot13c + .85c} = \frac{1.95}{2.80}\approx70\%.

The unknown baseline $c$ cancels. Also, if 18% of adults and 15% of women smoke, and half of adults are women, then 21% of men smoke, and the fraction of smokers who are women is $\frac{15}{15+21} = \frac5{12}$ .

ExampleSocks and boxes (Hajek)

Socks: 4 black, 6 red and 8 yellow, draw 2. $P(\text{same color}) = \frac{6 + 15 + 28}{153} = \frac{49}{153}$ and $P(\text{yellow}\mid\text{same}) = \frac{28}{49} = \frac47$ .
Two boxes: box 1 has 3 black and 2 white balls, box 2 has 2 and 2. Move a random ball from box 1 to box 2 ( $W$ = it was white), then draw from box 2 ( $B$ = black). $P(B) = \frac25\cdot\frac25 + \frac35\cdot\frac35 = \frac{13}{25}$ and $P(W\mid B) = \frac{4/25}{13/25} = \frac4{13} < \frac25$ . Transferring a white ball makes a black draw less likely, so seeing black makes $W$ less likely.

ExampleA mixture of two coins (Hajek)

Pick one of two coins with heads probabilities $p$ and $q = 1-p$ ( $p < \frac12$ ), each with probability $\frac12$ , then flip it $n$ times. Let $X$ be the number of heads.

p_X(k) = \frac12\binom nk\big(p^kq^{n-k} + q^kp^{n-k}\big),\qquad E[X] = \frac{np}2 + \frac{nq}2 = \frac n2.

Using $E[X^2\mid\text{coin}] = (\text{its mean})^2 + npq$ for each coin,

\mathrm{Var}(X) = \Big(\frac{n(1-2p)}2\Big)^2 + np(1-p),\qquad \sigma_X\ge \frac{(1-2p)n}2.

ImportantUncertainty about the model does not average out

A single binomial has $\sigma\propto\sqrt n$ . The mixture has $\sigma\propto n$ , because not knowing which coin is being flipped is a fixed uncertainty that more flips of the same coin cannot remove.

First-step analysis

Condition on the first trial. If the process restarts afresh afterward, you get an equation for the unknown probability.

ExampleA sum of 5 before a sum of 7 (Ross)

$P(5) = \frac4{36}$ , $P(7) = \frac6{36}$ , and neither with probability $\frac{26}{36}$ . Conditioning on the first roll:

P(E) = \frac4{36}\cdot1 + \frac6{36}\cdot0 + \frac{26}{36}P(E) \;\Rightarrow\; P(E) = \frac25.

In general, for mutually exclusive $E, F$ in repeated independent trials, $P(E\text{ before }F) = \frac{P(E)}{P(E) + P(F)}.$

ExampleThe problem of the points (Pascal–Fermat, 1654)

Trials succeed with probability $p$ . What is the probability $P_{n,m}$ of $n$ successes before $m$ failures? This is the fair split of stakes in an interrupted game.

Pascal (first step): $P_{n,m} = pP_{n-1,m} + (1-p)P_{n,m-1}$ , with $P_{n,0} = 0$ and $P_{0,m} = 1$ .
Fermat (imagine playing on): $n$ successes precede $m$ failures iff the first $m+n-1$ trials contain at least $n$ successes: $P_{n,m} = \sum_{k=n}^{m+n-1}\binom{m+n-1}kp^k(1-p)^{m+n-1-k}.$

IntuitionFermat’s trick: play past the end

Pretend the game always runs a fixed number of trials. Extra trials after the decision can’t change the outcome, and a fixed-length game has a binomial count. The same trick shows that “winner serves” and “alternating serve” give the same match-win probability in serve-and- rally games (Ross): with $2n-1$ rallies played, the first server always serves $n$ of them.

ExampleGambler’s ruin (Ross)

$A$ starts with $i$ units and $B$ with $N - i$ . Each flip moves one unit to $A$ with probability $p$ (let $q = 1-p$ ). Conditioning on the first flip gives $P_i = pP_{i+1} + qP_{i-1}$ , with $P_0 = 0$ and $P_N = 1$ . Therefore

P_i = \begin{cases}\dfrac{1 - (q/p)^i}{1 - (q/p)^N} & p\ne\frac12,\\[6pt] \dfrac iN & p = \frac12.\end{cases} \tag{4.5}

DerivationSolving the recurrence

Step 1: Increments are geometric. From $pP_i + qP_i = pP_{i+1} + qP_{i-1}$ we get $P_{i+1} - P_i = \frac qp(P_i - P_{i-1}) = \Big(\frac qp\Big)^iP_1.$

Step 2: Sum the increments. $P_i = P_1\big[1 + \frac qp + \cdots + (\frac qp)^{i-1}\big]$ , a geometric series (or $iP_1$ if $q = p$ ).

Step 3: Use $P_N = 1$ to find $P_1$ .

Step 4: Nobody plays forever. $B$ ’s win probability $Q_i$ comes from the same formula with $p\leftrightarrow q$ and $i\leftrightarrow N-i$ . Algebra shows $P_i + Q_i = 1$ . So the third possibility, the game never ending, has probability 0.

ExampleNumbers and applications

Starting at 5 of 15 total units: $A$ wins with probability $\frac13$ at $p = \frac12$ , but about $0.87$ at $p = .6$ . A small edge compounds.
Huygens’ dice duel (12 coins each; 11 on three dice pays $A$ , 14 pays $B$ ) is gambler’s ruin with $p = \frac{15}{42}$ , $i = 12$ , $N = 24$ .
Sequential drug trial: stop when one drug’s cumulative cures lead the other’s by $M$ . The probability of wrongly declaring the worse drug better is $\frac1{1 + \gamma^M},\qquad\gamma = \frac{p_1(1-p_2)}{p_2(1-p_1)}.$ With $p_1 = .6$ and $p_2 = .4$ : .017 at $M = 5$ and .0003 at $M = 10$ .

ExampleEqual-fortune elimination (Ross)

Player $i$ holds $n_i$ units, $n = \sum n_i$ . Random pairs play fair one-unit games until one player holds everything. $P(i\text{ wins}) = n_i/n$ . Split each player into $n_i$ one-unit “sub-players.” By symmetry each of the $n$ sub-players is equally likely to end up with everything, and team $i$ wins exactly when one of its $n_i$ sub-players does. This holds however the pairs are chosen.

ExampleRuns before runs (Ross)

What is the probability that a run of $n$ successes comes before a run of $m$ failures? Condition on the first trial, then on whether the next $n-1$ (or $m-1$ ) trials continue the run. A broken run resets the problem:

P(E) = \frac{p^{n-1}(1 - q^m)}{p^{n-1} + q^{m-1} - p^{n-1}q^{m-1}}. \tag{5.7}

For a fair coin, 2 heads before 3 tails has probability $\frac7{10}$ , and 2 heads before 4 tails has probability $\frac56$ . The two symmetric probabilities sum to 1, so one of the runs occurs eventually.

ExampleMatching problem by recursion (Ross)

Condition on whether person 1 takes their own hat. With $P_n$ the no-match probability, $P_n = \frac{n-1}nP_{n-1} + \frac1nP_{n-2}$ , i.e. $P_n - P_{n-1} = -\frac1n(P_{n-1} - P_{n-2})$ . With $P_1 = 0$ and $P_2 = \frac12$ , this telescopes to $P_n = \frac1{2!} - \frac1{3!} + \cdots + \frac{(-1)^n}{n!}$ , matching the inclusion–exclusion answer.

Bayes’ formula

TheoremBayes’ formula (Ross Prop. 3.1)

For a partition $F_1,\dots,F_n$ and an event $E$ with $P(E) > 0$ :

P(F_j\mid E) = \frac{P(E\mid F_j)P(F_j)}{\sum_{i=1}^nP(E\mid F_i)P(F_i)}. \tag{3.5}

ProofDefinition over total probability

$P(F_j\mid E) = P(EF_j)/P(E)$ . Write the numerator with the chain rule and the denominator with total probability.

∎

IntuitionPriors, likelihoods, posteriors

Think of the $F_j$ as hypotheses, $P(F_j)$ as priors, $P(E\mid F_j)$ as likelihoods of the evidence, and $P(F_j\mid E)$ as posteriors. Bayes’ formula says how evidence should revise belief: posterior ∝ likelihood × prior.

DefinitionOdds

$\text{odds}(A) = \frac{P(A)}{P(A^c)} = \frac{P(A)}{1 - P(A)}.$ Odds of $\alpha$ are said ” $\alpha$ to 1” in favour. For example, $P(A) = \frac23$ gives odds 2.

FormulaBayes in odds form

\frac{P(H\mid E)}{P(H^c\mid E)} = \frac{P(H)}{P(H^c)}\cdot\frac{P(E\mid H)}{P(E\mid H^c)}. \tag{3.3}

New odds = old odds × likelihood ratio.

ImportantWhat counts as evidence for a hypothesis

$E$ raises the probability of $H$ iff $P(E\mid H) > P(E\mid H^c)$ : the evidence must be more likely under $H$ than under its negation. “Consistent with $H$ ” is not enough.

Reese–Schapiro bridge hearing (1965): the prosecution argued that play consistent with cheating counts toward guilt. But the same play was equally consistent with their standard line, a likelihood ratio of about 1, so it was no evidence at all.

ExampleThe base-rate fallacy — a good test for a rare disease (Ross)

The test detects the disease 95% of the time, gives false positives 1% of the time, and prevalence is 0.5%.

P(D\mid +) = \frac{.95\cdot.005}{.95\cdot.005 + .01\cdot.995} = \frac{95}{294}\approx.323.

Only about a third of positives are sick.

graph LR
  P["200 people tested"] -->|"prevalence 0.005"| D["1 has disease"]
  P -->|"0.995"| H["199 healthy"]
  D -->|"detects 0.95"| TP["0.95 true positive"]
  H -->|"false positive 0.01"| FP["1.99 false positives"]

IntuitionCount people, not probabilities

Per 200 people tested, about 0.95 true positives compete with about 1.99 false positives. The false positives win because the healthy group is about 200 times larger. A rare condition makes even a small false-positive rate dominate.

ExampleBelief updates (Ross)

Situation	Prior	Likelihoods (under $H$ / not $H$ )	Posterior
accident prone, given an accident	.3	.4 / .2	$\frac{.12}{.26} = \frac6{13}$
knew the answer, given correct ( $m$ choices)	$p$	1 / $\frac1m$	$\frac{mp}{1 + (m-1)p}$ ; $\frac56$ at $m = 5$ , $p = \frac12$
disease, positive test, diabetic patient	.6	1 / .3	$\frac{.6}{.72}\approx.833$ (≥ .8, so operate)
guilty, suspect has the criminal’s trait (20% of population)	.6	1 / .2	$\approx.882$
type-A coin (two A’s, one B), heads	$\frac23$	$\frac14$ / $\frac34$	odds $\frac23$ , probability $\frac25$
flashlight type 1, 2, 3 given it lasted	.2, .3, .5	.7, .4, .3	$\frac{14}{41},\frac{12}{41},\frac{15}{41}$
chimp father is male 1 (baby Aa; mother AA; males aa, Aa)	$p$	1 / $\frac12$	$\frac{2p}{1+p} > p$

ExampleSearching for a lost plane (Ross)

The plane is equally likely to be in one of 3 regions. A search of region $i$ misses the plane there with overlook probability $\beta_i$ . After an unsuccessful search of region 1:

P(R_1\mid E) = \frac{\beta_1}{\beta_1 + 2},\qquad P(R_j\mid E) = \frac1{\beta_1 + 2}\ (j = 2,3).

Region 1’s probability drops and the others’ rise. The drop is smaller when $\beta_1$ is large, since a miss is then more plausibly bad luck.

WarningThe three-card swindle (Ross)

The cards are red/red, black/black and red/black. A random card shows red on top. What is $P(\text{other side black})$ ? It is $\frac13$ , not $\frac12$ . Count sides, not cards: of the 3 equally likely red faces, only one belongs to the red/black card.

P(RB\mid R) = \frac{\frac12\cdot\frac13}{1\cdot\frac13 + \frac12\cdot \frac13 + 0} = \frac13.

WarningThe two-child puzzle is underspecified (Ross)

A mother of two is seen with a daughter. What is $P(\text{two girls})$ ? It is $P(G_1G_2\mid G) = \frac1{1 + P(G\mid G_1B_2) + P(G\mid B_1G_2)},$ which depends on how the child walking with her was chosen.

The companion is the elder child with probability $p$ regardless of sex: answer $\frac12$ .
With mixed sexes, she walks with the girl with probability $q$ : answer $\frac1{1+2q}$ . With $q = 1$ this is $\frac13$ , the “at least one girl” answer.

The sample space must include which child is seen, not just the children’s sexes.

ExampleDNA database match (Ross)

One culprit among 1,000,000 townspeople. Each of 10,000 ex-convicts has prior probability $\alpha = c\beta$ and everyone else $\beta$ , so $\alpha = \frac c{10{,}000c + 990{,}000}$ . An innocent person matches with probability $10^{-5}$ . A. J. Jones is the only match in the ex-convict database. Working it through,

P(G\mid M) = \frac{\alpha}{\alpha + 10^{-5}(1 - 10{,}000\alpha)} = \frac1{0.9 + 10^{-5}/\alpha}.

Prior ratio $c$	$P(\text{guilty}\mid\text{match})$
100	.9099
10	.5025
1	.0917

The prior drives the conclusion. The same DNA evidence yields anywhere from 9% to 91%.

Conditional independence and sequential updating

DefinitionConditional independence

$E_1$ and $E_2$ are conditionally independent given $F$ if

P(E_1\mid E_2F) = P(E_1\mid F),\quad\text{equivalently}\quad P(E_1E_2\mid F) = P(E_1\mid F)P(E_2\mid F).

WarningConditionally independent ≠ independent

Accidents in successive years are conditionally independent given whether the policyholder is accident prone. Unconditionally, a first-year accident raises the posterior that the person is accident prone, which raises next year’s risk:

P(A_2\mid A_1) = .4\cdot\tfrac6{13} + .2\cdot\tfrac7{13}\approx.29 > .26 = P(A_2).

ExampleLaplace’s rule of succession (Ross)

The box holds coins $i = 0..k$ , where coin $i$ lands heads with probability $i/k$ . One is picked at random and lands heads $n$ times in a row. Flips are conditionally independent given the coin, so

P(H_{n+1}\mid H_n) = \frac{P(H_{n+1})}{P(H_n)} = \frac{\sum_i(i/k)^{n+1}} {\sum_i(i/k)^n}\;\approx\;\frac{\int_0^1x^{n+1}dx}{\int_0^1x^ndx} = \frac{n+1}{n+2}\quad(k\text{ large}).

TheoremSequential Bayesian updating

If $E_1$ and $E_2$ are conditionally independent given each hypothesis $H_j$ , then the posterior after $E_1$ can serve as the prior for $E_2$ :

P(H_i\mid E_1E_2) = \frac{P(E_2\mid H_i)P(H_i\mid E_1)}{\sum_jP(E_2\mid H_j)P(H_j\mid E_1)}.

For example, tracking which of two coins is being flipped requires storing only the current posterior, not the whole history.

ProofFactor the joint likelihood

By conditional independence, $P(E_1E_2\mid H_i) = P(E_2\mid H_i)P(E_1\mid H_i)$ . So

P(H_i\mid E_1E_2) = \frac{P(E_2\mid H_i)P(E_1\mid H_i)P(H_i)}{P(E_1E_2)} = \frac{P(E_2\mid H_i)P(H_i\mid E_1)}{Q},

with $Q = P(E_1E_2)/P(E_1)$ . Summing over $i$ (the posteriors sum to 1) identifies $Q = \sum_jP(E_2\mid H_j)P(H_j\mid E_1)$ .

∎

Quick reference

Object	Statement	Note
Conditional probability	$P(B\mid A) = P(AB)/P(A)$	undefined if $P(A) = 0$
$P(\cdot\mid A)$	a probability measure	all rules carry over
Chain rule	$P(E_1\cdots E_n) = \prod_kP(E_k\mid E_1\cdots E_{k-1})$	aces in piles $\approx.105$
Exactly $k$ matches	$P_{N-k}/k!\to e^{-1}/k!$
Independence	$P(AB) = P(A)P(B)$	complements preserve it
$n$ events	product rule for every subfamily	pairwise is not enough
Binomial	$\binom nkp^k(1-p)^{n-k}$	independent trials
Parallel system	$1 - \prod(1 - p_i)$
Independent RVs	joint pmf factorizes	⇒ $E[X_1X_2] = E[X_1]E[X_2]$ for dice
Total probability	$P(E) = \sum_iP(E\mid F_i)P(F_i)$	reason by cases
$E$ before $F$	$P(E)/(P(E) + P(F))$	5 before 7: $2/5$
Problem of points	$\sum_{k\ge n}\binom{m+n-1}kp^k(1-p)^{m+n-1-k}$	play past the end
Gambler’s ruin	$\frac{1 - (q/p)^i}{1 - (q/p)^N}$ or $i/N$	ends with probability 1
Bayes	$P(F_j\mid E)\propto P(E\mid F_j)P(F_j)$
Odds form	posterior odds = prior odds × likelihood ratio	evidence iff ratio > 1
Base rates	$P(D\mid +)\approx.32$ for a 95%/1% test at 0.5% prevalence
Three cards	$1/3$	count sides
Conditional independence	$P(E_1E_2\mid F) = P(E_1\mid F)P(E_2\mid F)$	not unconditional
Rule of succession	$(n+1)/(n+2)$
Conditional mean	$E[X\mid A] = \sum u\,P(X = u\mid A)$	conditional LOTUS
Total expectation	$E[X] = \sum_jE[X\mid E_j]P(E_j)$	behind first-step analysis
Coin mixture	$\mathrm{Var} = (n(1-2p)/2)^2 + np(1-p)$	$\sigma\propto n$ , not $\sqrt n$

Facts to keep to hand

Conditioning replaces $P$ with a new probability measure. Every rule survives. Most apparent paradoxes (at least one head, three cards, two children) come from a misspecified reduced sample space. Fix them by returning to the definition and counting outcomes, sides, or which child was seen.
Independence is algebra, not physics. Physical separation justifies assuming it, but it can hold by coincidence (even and multiple of 3 on one die). Pairwise independence does not imply mutual independence. And conditional independence given a hidden cause coexists with unconditional dependence (accident proneness).
Total probability plus a restart gives an equation. 5 before 7, gambler’s ruin, runs, matching, points, and the ace-guessing game all condition on the first step and exploit the fact that the process begins afresh. Fermat’s alternative is to extend the game to a fixed length so a binomial count applies.
Bayes is “posterior odds = prior odds × likelihood ratio.” Evidence supports $H$ only if it is likelier under $H$ than under not- $H$ . Rare hypotheses need very large likelihood ratios, which explains the base-rate fallacy, the DNA-database sensitivity to priors, and why “consistent with guilt” is not evidence of guilt.
Total expectation is total probability for means. Condition on a partition and average the conditional means. It underlies first-step analysis, and it shows that model uncertainty (which coin?) adds spread that grows like $n$ rather than $\sqrt n$ .

References

Hajek, Probability with Engineering Applications (ECE 313 notes) — chapter 2, §§2.3–2.4.2, 2.10
Ross, A First Course in Probability — chapter 3

Related

Conditional Probability, Independence, and Bayes' Formula

Motivation: updating on partial information

Conditional probabilities

The multiplication (chain) rule

Mutually independent events

Three or more events

Independent trials

Independent random variables (of discrete type)

The law of total probability, and Bayes’ formula

Total probability

Conditional expectation and total expectation

First-step analysis

Bayes’ formula

Conditional independence and sequential updating

Quick reference

References

Mentioned in