Parameter Estimation and Binary Hypothesis Testing

Two faces of the same likelihood machinery. Estimation picks the parameter value that makes the observed data most probable — once where the likelihood is a pmf (discrete-type observations) and once where it’s a pdf (continuous-type observations, e.g. a Gaussian’s mean/variance or an exponential’s rate). Testing asks a discrete question instead of a continuous one: given an observation generated under one of two competing hypotheses, decide which — via the ML rule (likelihood ratio vs. 1) or the MAP rule (likelihood ratio vs. a prior-odds threshold), again for both discrete- and continuous-type observations.

Maximum likelihood parameter estimation

DefinitionLikelihood and the ML estimate (Hajek §2.8)

Suppose $X$ has pmf $p_\theta$ with an unknown parameter $\theta$ , and we observe $X = k$ . The likelihood of $\theta$ is $p_\theta(k)$ , viewed as a function of $\theta$ with $k$ fixed. The maximum likelihood estimate is $\hat\theta_{ML}(k) = \arg\max_\theta p_\theta(k),$ the parameter value under which the observation is least surprising.

Component	Detail
Fixed	the observation $k$
Varied	the parameter $\theta$
Maximized	$p_\theta(k)$ , or equivalently $\ln p_\theta(k)$
Not	a probability distribution over $\theta$ (no prior is involved)

WarningMaximize over $\theta$ , not over $k$

Maximizing $p_\theta(k)$ over $k$ with $\theta$ known answers a different question: the best guess of the next observation. Estimation fixes the data and varies the parameter.

ExampleFour ML estimates (Hajek)

Model	Observation	Likelihood	$\hat\theta_{ML}$
binomial $(n,p)$ (a bent coin)	$k$ heads in $n$ flips	$\binom nkp^k(1-p)^{n-k}$	$\hat p = k/n$
uniform on $\{1,\dots,n\}$	$X = k$	$\frac1n\mathbf 1\{n\ge k\}$	$\hat n = k$
geometric $(p)$	$L = k$	$(1-p)^{k-1}p$	$\hat p = 1/k$
Poisson $(\lambda)$	$X = k$	$\lambda^ke^{-\lambda}/k!$	$\hat\lambda = k$

DerivationWhere each maximum is

Binomial: $\frac d{dp}p^k(1-p)^{n-k} = (k - np)p^{k-1}(1-p)^{n-k-1}$ , positive for $p < k/n$ and negative after. The endpoint cases $k = 0$ and $k = n$ give $\hat p = 0$ and $1$ , also equal to $k/n$ .

Uniform: the likelihood is 0 for $n < k$ , jumps to $\frac1k$ at $n = k$ , and decreases after, so $\hat n = k$ .

Geometric: $\frac d{dp}(1-p)^{k-1}p = (1 - kp)(1-p)^{k-2}$ , which changes sign at $p = 1/k$ . For $k = 1$ the likelihood $p$ peaks at $p = 1 = 1/k$ .

Poisson: $\frac d{d\lambda}\lambda^ke^{-\lambda} = (k - \lambda) \lambda^{k-1}e^{-\lambda}$ , which changes sign at $\lambda = k$ . For $k = 0$ the likelihood $e^{-\lambda}$ peaks at $\lambda = 0$ .

ExampleCapture–recapture

The hypergeometric ML estimate of a population size, $\hat N = \lfloor mn/i\rfloor$ , is worked out in discrete-random-variables-and-distributions.

Binary hypothesis testing (discrete-type observations)

DefinitionThe testing problem (Hajek §2.11)

Exactly one of two hypotheses is true, $H_1$ or $H_0$ . The observation $X$ has pmf $p_1$ under $H_1$ and $p_0$ under $H_0$ . A decision rule declares $H_1$ or $H_0$ for each possible value of $X$ .

graph LR
  H["Switch: H1 or H0 is true"] --> S["System generates X with pmf p1 or p0"]
  S --> D["Decision rule"]
  D --> O["Declare H1 or H0"]

DefinitionLikelihood matrix, false alarm, miss

The likelihood matrix has one row per hypothesis and one column per value of $X$ , with entries $p_i(k)$ . Mark one entry per column to specify a rule. Then:

$p_{\text{false alarm}} = P(\text{declare }H_1\mid H_0)$ is the sum of the unmarked $H_0$ entries (statistics: type I error);
$p_{\text{miss}} = P(\text{declare }H_0\mid H_1)$ is the sum of the unmarked $H_1$ entries (type II error).

Both are conditional probabilities.

ExampleA running likelihood matrix (Hajek)

	$X = 0$	$X = 1$	$X = 2$	$X = 3$
$H_1$	0.0	0.1	0.3	0.6
$H_0$	0.4	0.3	0.2	0.1

Rule “declare $H_1$ iff $X\ge1$ ”: $p_{\text{fa}} = .3 + .2 + .1 = .6$ and $p_{\text{miss}} = 0$ . Switching the $X = 1$ column to $H_0$ gives $p_{\text{fa}} = .3$ and $p_{\text{miss}} = .1$ .

ImportantEvery change trades one error for the other

Moving a column’s mark from one row to the other unmarks one entry and marks another, so one error probability rises as the other falls. In general no rule makes both small. Hajek’s advice: compute the pair $(p_{\text{fa}}, p_{\text{miss}})$ for several rules, then choose.

Maximum likelihood (ML) decision rule

DefinitionML rule and the likelihood ratio test

Declare the hypothesis under which the observation is more likely: mark the larger entry in each column. Equivalently, with the likelihood ratio

\Lambda(k) = \frac{p_1(k)}{p_0(k)},\qquad\text{declare }H_1\text{ iff } \Lambda(X) > 1.

A likelihood ratio test (LRT) with threshold $\tau$ declares $H_1$ iff $\Lambda(X) > \tau$ . Raising $\tau$ lowers $p_{\text{fa}}$ and raises $p_{\text{miss}}$ . The ML rule is the LRT with $\tau = 1$ .

ExampleML rule for the running matrix

Declare $H_1$ for $X\in\{2,3\}$ and $H_0$ for $X\in\{0,1\}$ : $p_{\text{fa}} = .2 + .1 = .3$ and $p_{\text{miss}} = .0 + .1 = .1$ .

Maximum a posteriori (MAP) decision rule

DefinitionPriors and the joint probability matrix

Assume priors $\pi_i = P(H_i)$ . The joint probability matrix has entries $P(H_i, X = k) = \pi_ip_i(k)$ : each likelihood row scaled by its prior. All its entries sum to 1.

DefinitionMAP rule

Declare the hypothesis with the larger posterior $P(H_i\mid X = k)$ . Both posteriors share the denominator $P\{X = k\}$ (Bayes’ formula), so this means marking the larger entry in each column of the joint matrix. Equivalently,

\text{declare }H_1\iff\pi_1p_1(X) > \pi_0p_0(X)\iff\Lambda(X) > \frac{\pi_0}{\pi_1}.

MAP is the LRT with $\tau = \pi_0/\pi_1$ . With a uniform prior it reduces to ML.

TheoremMAP minimizes the average error probability

For any rule, $p_e = \pi_0p_{\text{fa}} + \pi_1p_{\text{miss}}$ equals the sum of the unmarked entries of the joint matrix. MAP marks the larger entry in every column, so it leaves the smallest possible sum unmarked. Hajek calls it the minimum-probability-of-error rule.

IntuitionWhy a larger $\pi_0$ raises the threshold

If $H_0$ is a priori much likelier, it takes stronger evidence (a larger likelihood ratio) to declare $H_1$ .

ExampleMAP for the running matrix, $\pi_0 = .8$

	$X = 0$	$X = 1$	$X = 2$	$X = 3$
$H_1$	.00	.02	.06	.12
$H_0$	.32	.24	.16	.08

MAP declares $H_1$ only for $X = 3$ . That gives $p_{\text{fa}} = .1$ , $p_{\text{miss}} = .4$ and $p_e = .16$ , versus $p_e = .8(.3) + .2(.1) = .26$ for the ML rule. (These three numbers are computed here, not in Hajek.)

ExampleFair or biased coin, 5 flips (Hajek)

$H_1$ : heads with probability $\frac23$ ; $H_0$ : fair. $X$ = heads in 5 flips. The binomial coefficients cancel in the ratio:

\Lambda(k) = \frac{(2/3)^k(1/3)^{5-k}}{(1/2)^5} = 2^k\frac{32}{243}\approx \frac{2^k}{7.6}.

Rule	Declares $H_1$ iff	$p_{\text{fa}}$	$p_{\text{miss}}$	$p_e$ with $(\pi_0,\pi_1) = (.2,.8)$
ML ( $\tau = 1$ )	$X\ge3$	.5	$51/243\approx.210$	.268
MAP ( $\tau = .25$ )	$X\ge1$	$1 - 2^{-5}\approx.97$	$1/243\approx.0041$	.197

ExamplePhoton counting (Hajek)

A laser sends a bit. The photon count is Poisson(6) for a one and Poisson(2) for a zero:

\Lambda(k) = \Big(\frac62\Big)^ke^{-(6-2)} = 3^ke^{-4}\approx\frac{3^k} {54.6}.

ML: declare one iff $3^X\ge54.6$ , i.e. $X\ge4$ .
MAP with zeros 5 times likelier ( $\pi_0/\pi_1 = 5$ ): declare one iff $3^X\ge273$ , i.e. $X\ge6$ .

On $X = 4$ or $5$ the rules disagree: the evidence favors a one, but the prior favors a zero more.

ExampleSensor fusion (Hajek)

Two motion sensors report $X, Y\in\{0,1,2\}$ , and $H_1$ = a person is present. The readings are conditionally independent given the hypothesis, so the joint likelihoods multiply:

	$X$ =0	1	2		$Y$ =0	1	2
$H_1$	.1	.3	.6		.1	.1	.8
$H_0$	.8	.1	.1		.7	.2	.1

$(X,Y)$	00	01	02	10	11	12	20	21	22
$H_1$	.01	.01	.08	.03	.03	.24	.06	.06	.48
$H_0$	.56	.16	.08	.07	.02	.01	.07	.02	.01

ML (tie at (0,2) broken toward $H_1$ ): declare $H_1$ on $\{02, 11, 12, 21, 22\}$ . $p_{\text{fa}} = .14$ and $p_{\text{miss}} = .11$ .
MAP with $\pi_1 = .2$ : declare $H_1$ iff $p_1 > 4p_0$ , only on $\{12, 22\}$ . $p_{\text{fa}} = .02$ , $p_{\text{miss}} = .28$ , and $p_e = .8(.02) + .2(.28) = .072$ .
ML’s average error under the same priors: $.8(.14) + .2(.11) = .134$ , larger, as MAP optimality guarantees.

Still to write

NoteTODO — ML parameter estimation for continuous-type variables

Needs Hajek’s continuous-observation version (maximizing a pdf). Source text not yet in sources/.

NoteTODO — Binary hypothesis testing with continuous-type observations

Needs Hajek’s continuous version (LRTs with pdfs). Source text not yet in sources/.

Quick reference

Object	Statement	Note
ML estimate	$\hat\theta = \arg\max_\theta p_\theta(k)$	data fixed, parameter varied
Standard ML estimates	binomial $k/n$ ; geometric $1/k$ ; Poisson $k$ ; uniform $k$
False alarm / miss	$P(\text{declare }H_1\mid H_0)$ / $P(\text{declare }H_0\mid H_1)$	type I / type II
Likelihood ratio	$\Lambda(k) = p_1(k)/p_0(k)$	LRT: declare $H_1$ iff $\Lambda > \tau$
ML rule	LRT with $\tau = 1$	larger entry per column
MAP rule	LRT with $\tau = \pi_0/\pi_1$	minimizes $p_e = \pi_0p_{\text{fa}} + \pi_1p_{\text{miss}}$

Facts to keep to hand

One likelihood, two uses. Estimation maximizes $p_\theta(k)$ over a range of parameters. Testing compares it across two hypotheses.
Every rule here is a likelihood ratio test. ML and MAP differ only in the threshold, and the threshold is the prior odds. Moving it trades false alarms for misses.
MAP is optimal for the prior you assumed. It minimizes average error under that prior. Reporting both conditional error probabilities, as Hajek recommends, shows what the choice of rule costs.

References

Hajek, Probability with Engineering Applications (ECE 313 notes) — chapter 2, §§2.8, 2.11 (continuous-type sections pending)

Related

Parameter Estimation and Binary Hypothesis Testing

Maximum likelihood parameter estimation

Binary hypothesis testing (discrete-type observations)

Maximum likelihood (ML) decision rule

Maximum a posteriori (MAP) decision rule

Still to write

Quick reference

References

Mentioned in