Differential Entropy

Motivation: the wire carries voltages, not letters

Every quantity so far has assumed a finite alphabet: 26 keys, two symbols, an erasure. But the actual copper wire carries a voltage — a real number — and so does every other physical channel: the amplitude of a radio signal, the position of a needle, the temperature of a sensor. If information theory cannot price a continuous source, it cannot price most of the systems it was invented for.

The obvious move is to replace the sum with an integral and the pmf with a density. That move mostly works, and the resulting quantity — differential entropy — is related to shortest description length and is similar in many ways to discrete entropy. But there are important differences, and there is need for some care in using the concept.

WarningRead this before the definition

The asymptotic-equipartition-property note already flagged the trap: for a continuous $X$ , plugging into the discrete formula gives $H(X) = +\infty$ identically, because every single point has probability zero. Telling someone a real number to infinite precision costs infinite bits, and that is not an artifact — it is true. Differential entropy is therefore not “the entropy of a continuous variable”; it is a different functional that happens to obey many of the same identities. The whole of exists to say exactly how the two are related and where the missing infinity went.

Definitions

DefinitionContinuous random variable and density

Let $X$ have cumulative distribution function $F(x) = \Pr(X \le x)$ . If $F(x)$ is continuous, $X$ is said to be continuous. Let $f(x) = F'(x)$ where the derivative is defined. If $\int_{-\infty}^{\infty} f(x)\,dx = 1$ , then $f(x)$ is the probability density function for $X$ . The set where $f(x) > 0$ is the support set of $X$ .

DefinitionDifferential entropy

The differential entropy $h(X)$ of a continuous random variable $X$ with density $f(x)$ is

$h(X) = - \int_S f(x) \log f(x) \, dx$

where $S$ is the support set of the random variable.

Component	Detail
Signature	$h : \{\text{densities on } \mathbb{R}^n\} \rightarrow [-\infty, \infty]$ — note the negative values are allowed, unlike $H$ .
Domain	The density $f$ , not the variable. As in the discrete case it depends only on the probability law, so it is sometimes written $h(f)$ rather than $h(X)$ .
Codomain	An extended real number. Interpreting it takes work — see ; it is not “average bits of surprise”.

Caution“If it exists”

As in every statement involving an integral, or even a density, we should append if it exists. It is easy to construct random variables for which no density exists, or for which the above integral does not exist. Every theorem below carries this silent hypothesis.

WarningNotation collision

Cover uses lowercase $h$ for differential entropy, and we used lowercase $h$ in the first note for the self-information $h(p) = -\log_2 p$ . They are unrelated. From here on $h$ means differential entropy; where surprise is meant it is written $-\log p(x)$ explicitly.

Two worked densities

ExampleUniform distribution on $[0, a]$

The density is $1/a$ on $[0,a]$ and 0 elsewhere, so

$h(X) = - \int_0^a \frac{1}{a} \log \frac{1}{a} \, dx = \log a$

ImportantDifferential entropy can be negative

For $a < 1$ we get $\log a < 0$ . Unlike discrete entropy, differential entropy can be negative — which immediately kills any reading of it as “a number of bits”. What is always sensible is the exponential: $2^{h(X)} = 2^{\log a} = a$ is the volume of the support set, which is non-negative as we expect. Keep this in mind: throughout this chapter the meaningful object is often $2^h$ rather than $h$ .

ExampleNormal distribution

Let $X \sim \phi(x) = \frac{1}{\sqrt{2\pi\sigma^2}} e^{-x^2/2\sigma^2}$ . Computing in nats:

$h(\phi) = - \int \phi \ln \phi = - \int \phi(x) \left[ -\frac{x^2}{2\sigma^2} - \ln\sqrt{2\pi\sigma^2} \right] dx$

Split the two terms. The first is an expectation of $x^2$ , the second a constant pulled out of a density that integrates to 1:

$= \frac{\mathbb{E}X^2}{2\sigma^2} + \frac{1}{2}\ln 2\pi\sigma^2 = \frac{1}{2} + \frac{1}{2}\ln 2\pi\sigma^2$

using $\mathbb{E}X^2 = \sigma^2$ . Now absorb the $\tfrac12$ using $\tfrac12 = \tfrac12 \ln e$ :

$= \frac{1}{2}\ln e + \frac{1}{2}\ln 2\pi\sigma^2 = \frac{1}{2} \ln 2\pi e \sigma^2 \text{ nats}$

FormulaDifferential entropy of a Gaussian

$h(\phi) = \frac{1}{2} \log 2\pi e \sigma^2 \text{ bits}$

IntuitionReading the Gaussian formula

Entropy depends on $\sigma$ but not on the mean — sliding the bell curve along the axis does not change how spread out it is (formalized later as ). And it grows like $\log \sigma$ : doubling the spread buys exactly one more bit, matching the intuition that you need one extra bit of precision to locate a point in a range twice as wide.

AEP for continuous random variables

One of the important roles of discrete entropy is in the AEP: $p(X_1, \ldots, X_n)$ is close to $2^{-nH}$ with high probability, which lets us define the typical set. We can do exactly the same for continuous variables — with volume replacing cardinality.

TheoremContinuous AEP

Let $X_1, X_2, \ldots, X_n$ be drawn i.i.d. according to the density $f(x)$ . Then

$-\frac{1}{n} \log f(X_1, X_2, \ldots, X_n) \longrightarrow \mathbb{E}[-\log f(X)] = h(X) \quad \text{in probability}$

ProofIdentical to the discrete case

Independence makes $-\frac1n \log f(X^n) = -\frac1n \sum_i \log f(X_i)$ a sample mean of i.i.d. terms; the weak law of large numbers sends it to $\mathbb{E}[-\log f(X)] = h(X)$ . The argument is the same three lines as in the discrete AEP — nothing about the proof cared whether the alphabet was finite.

∎

DefinitionTypical set (continuous)

For $\epsilon > 0$ and any $n$ ,

$A_\epsilon^{(n)} = \left\{ (x_1, \ldots, x_n) \in S^n : \left\lvert -\frac{1}{n} \log f(x_1, \ldots, x_n) - h(X) \right\rvert \le \epsilon \right\}$

where $f(x_1, \ldots, x_n) = \prod_{i=1}^n f(x_i)$ .

DefinitionVolume

The volume of a set $A \subset \mathbb{R}^n$ is

$\text{Vol}(A) = \int_A dx_1\, dx_2 \cdots dx_n$

IntuitionWhat replaces “counting the tapes”

In the discrete case we counted how many typical tapes there were. Here there are uncountably many, so counting is meaningless — but volume is the natural analogue, and every discrete statement about $\lvert A_\epsilon^{(n)} \rvert$ has a continuous counterpart about $\text{Vol}(A_\epsilon^{(n)})$ with the same exponent.

TheoremProperties of the continuous typical set

$\Pr\big(A_\epsilon^{(n)}\big) > 1 - \epsilon$ for $n$ sufficiently large.
$\text{Vol}\big(A_\epsilon^{(n)}\big) \le 2^{n(h(X) + \epsilon)}$ for all $n$ .
$\text{Vol}\big(A_\epsilon^{(n)}\big) \ge (1-\epsilon) 2^{n(h(X) - \epsilon)}$ for $n$ sufficiently large.

ProofIntegrate instead of sum; otherwise line-for-line the discrete proof

Property 1 is the continuous AEP restated as a probability, exactly as in the discrete case.

Property 2 — upper bound. Start from total probability and restrict to the typical set:

$1 = \int_{S^n} f(x_1, \ldots, x_n)\, dx^n \ge \int_{A_\epsilon^{(n)}} f(x_1, \ldots, x_n)\, dx^n$

Every typical point has density at least $2^{-n(h(X)+\epsilon)}$ , so

$\ge \int_{A_\epsilon^{(n)}} 2^{-n(h(X)+\epsilon)}\, dx^n = 2^{-n(h(X)+\epsilon)} \int_{A_\epsilon^{(n)}} dx^n = 2^{-n(h(X)+\epsilon)} \text{Vol}\big(A_\epsilon^{(n)}\big)$

The one substantive change from the discrete proof: pulling the constant out of the integral leaves $\int dx^n$ , which is the volume, where in the discrete case it left a bare sum of 1’s, which was the count. Rearranging gives property 2.

Property 3 — lower bound. For $n$ large enough that property 1 holds,

$1 - \epsilon \le \int_{A_\epsilon^{(n)}} f(x^n)\, dx^n \le \int_{A_\epsilon^{(n)}} 2^{-n(h(X)-\epsilon)} dx^n = 2^{-n(h(X)-\epsilon)} \text{Vol}\big(A_\epsilon^{(n)}\big)$

using the density ceiling this time. Rearranging gives property 3.

Together, for $n$ sufficiently large:

$(1-\epsilon) 2^{n(h(X)-\epsilon)} \le \text{Vol}\big(A_\epsilon^{(n)}\big) \le 2^{n(h(X)+\epsilon)}$

∎

TheoremMinimality

The set $A_\epsilon^{(n)}$ is the smallest volume set with probability $\ge 1 - \epsilon$ , to first order in the exponent. (Proof: same as the discrete case.)

ImportantThe interpretation of differential entropy

The smallest set containing most of the probability has volume about $2^{nh}$ . That is an $n$ -dimensional volume, so the corresponding side length is

$\big(2^{nh}\big)^{1/n} = 2^h$

So: differential entropy is the logarithm of the equivalent side length of the smallest set containing most of the probability. Low entropy means the variable is confined to a small effective volume; high entropy means it is widely dispersed. This is the reading that makes negative values unsurprising — a side length below 1 has a negative logarithm, and nothing is wrong.

NoteA related quantity

Just as entropy is related to the volume of the typical set, there is a quantity called Fisher information related to its surface area.

Relation to discrete entropy

The problem. We now have two objects — $H$ for discrete variables, $h$ for continuous — that share almost all their algebra but are not the same kind of thing. What exactly is the relationship? The answer comes from quantization: chop the real line into bins and watch what happens as the bins shrink.

Setup. Divide the range of $X$ into bins of length $\Delta$ , and assume the density is continuous within bins. By the mean value theorem there exists a value $x_i$ within each bin such that

$f(x_i)\Delta = \int_{i\Delta}^{(i+1)\Delta} f(x)\, dx$

Define the quantized random variable

$X^\Delta = x_i \quad \text{if } i\Delta \le X < (i+1)\Delta$

so that $p_i = \Pr(X^\Delta = x_i) = f(x_i)\Delta$ . This is a genuinely discrete variable, so its $H$ is well defined.

DerivationEntropy of the quantized variable

Step 1: Write the discrete entropy.

$H(X^\Delta) = - \sum_{-\infty}^{\infty} p_i \log p_i = - \sum_{-\infty}^{\infty} f(x_i)\Delta \log\big(f(x_i)\Delta\big)$

Step 2: Split the logarithm of the product.

$= - \sum \Delta f(x_i) \log f(x_i) - \sum \Delta f(x_i) \log \Delta$

Step 3: Collapse the second sum. Since $\sum \Delta f(x_i) = \int f(x)\,dx = 1$ , the second term is just $\log \Delta$ :

$H(X^\Delta) = - \sum \Delta f(x_i) \log f(x_i) - \log \Delta$

Step 4: Take $\Delta \to 0$ . If $f(x)\log f(x)$ is Riemann integrable — a condition needed to ensure the limit is well defined — the first term approaches $-\int f(x)\log f(x)\,dx = h(f)$ by the definition of Riemann integrability.

TheoremQuantization relation

If the density $f(x)$ of $X$ is Riemann integrable, then

$H(X^\Delta) + \log \Delta \longrightarrow h(f) = h(X) \qquad \text{as } \Delta \rightarrow 0$

ImportantWhere the infinity went

Rearranged, $H(X^\Delta) \approx h(X) - \log\Delta = h(X) + \log\frac{1}{\Delta}$ . As $\Delta \to 0$ the term $\log\frac1\Delta$ diverges — that is the infinite cost of specifying a real number exactly, exactly as warned at the top. Differential entropy is what is left over after that divergence is subtracted off. It is a relative quantity, measured against the precision scale, which is why it carries no absolute meaning and can be negative.

Concretely: setting $\Delta = 2^{-n}$ , the entropy of an $n$ -bit quantization of $X$ is approximately $h(X) + n$ .

ExampleThree quantizations

$X$ uniform on $[0,1]$ , $\Delta = 2^{-n}$ : here $h = 0$ and $H(X^\Delta) = n$ , so $n$ bits suffice to describe $X$ to $n$ -bit accuracy — the baseline.
$X$ uniform on $[0, \tfrac18]$ : the first 3 bits after the point must be 0, so describing $X$ to $n$ -bit accuracy requires only $n - 3$ bits. This agrees with $h(X) = \log \tfrac18 = -3$ — and gives the negative entropy a perfectly concrete meaning: a 3-bit discount.
$X \sim N(0, \sigma^2)$ with $\sigma^2 = 100$ : describing $X$ to $n$ -bit accuracy costs on average $n + \tfrac12 \log(2\pi e \sigma^2) = n + 5.37$ bits — a surcharge, because the distribution is spread over far more than a unit interval.

In general, $h(X) + n$ is the average number of bits required to describe $X$ to $n$ -bit accuracy.

RemarkA discrete variable has $h = -\infty$

Consistently, the differential entropy of a discrete random variable can be considered $-\infty$ : then $2^{-\infty} = 0$ , agreeing with the fact that the support set of a discrete variable has volume zero.

Joint and conditional differential entropy

The extensions are the obvious ones, with integrals in place of sums.

DefinitionJoint differential entropy

$h(X_1, X_2, \ldots, X_n) = - \int f(x^n) \log f(x^n) \, dx^n$

DefinitionConditional differential entropy

If $X, Y$ have joint density $f(x,y)$ ,

$h(X \mid Y) = - \int f(x,y) \log f(x \mid y) \, dx\, dy$

Since $f(x \mid y) = f(x,y)/f(y)$ in general, we can also write

$h(X \mid Y) = h(X, Y) - h(Y)$

CautionSubtracting infinities

The identity $h(X \mid Y) = h(X,Y) - h(Y)$ requires care if any of the differential entropies are infinite — the right-hand side can be $\infty - \infty$ . This is a real hazard here in a way it never was for discrete entropy on a finite alphabet.

IntuitionNote the shape is the same as before

This is the [[entropy-mutual-information#Conditional entropy $H(Y \mid X)$ |conditional entropy]] pattern verbatim: joint density as the outside weight, conditional density inside the logarithm. The justification is identical — we average over how often both events co-occur, but measure the surprise of $X$ given $Y$ already happened.

TheoremEntropy of a multivariate normal

Let $X_1, \ldots, X_n$ have a multivariate normal distribution with mean $\mu$ and covariance matrix $K$ . Then

$h(X_1, \ldots, X_n) = h\big(\mathcal{N}_n(\mu, K)\big) = \frac{1}{2} \log (2\pi e)^n \lvert K \rvert \text{ bits}$

where $\lvert K \rvert$ is the determinant of $K$ .

ProofThe quadratic form in the exponent has a trace that collapses

Step 1: Write the density and take $-\mathbb{E}\log$ . The pdf is

$f(x) = \frac{1}{\sqrt{2\pi}^{\,n} \lvert K \rvert^{1/2}} e^{-\frac{1}{2}(x-\mu)^T K^{-1} (x-\mu)}$

so, in nats,

$h(f) = - \int f(x)\left[ -\frac{1}{2}(x-\mu)^T K^{-1}(x-\mu) - \ln \sqrt{2\pi}^{\,n}\lvert K\rvert^{1/2} \right] dx$

$= \frac{1}{2}\mathbb{E}\left[ \sum_{i,j} (X_i - \mu_i)\big(K^{-1}\big)_{ij} (X_j - \mu_j) \right] + \frac{1}{2}\ln(2\pi)^n \lvert K \rvert$

Step 2: Push the expectation inside — this is the key move. Expectation is linear and $\big(K^{-1}\big)_{ij}$ is a constant, so

$= \frac{1}{2}\sum_{i,j} \mathbb{E}\big[(X_j - \mu_j)(X_i - \mu_i)\big] \big(K^{-1}\big)_{ij} + \frac{1}{2}\ln(2\pi)^n \lvert K \rvert$

Step 3: Recognize the covariance and contract. The expectation is exactly $K_{ji}$ , so the double sum is a matrix product traced out:

$= \frac{1}{2}\sum_{i,j} K_{ji}\big(K^{-1}\big)_{ij} + \cdots = \frac{1}{2}\sum_j \big(KK^{-1}\big)_{jj} + \cdots = \frac{1}{2}\sum_j I_{jj} + \cdots = \frac{n}{2} + \frac{1}{2}\ln(2\pi)^n\lvert K \rvert$

Step 4: Absorb $n/2$ using $\tfrac{n}{2} = \tfrac12 \ln e^n$ .

$= \frac{1}{2}\ln (2\pi e)^n \lvert K \rvert \text{ nats} = \frac{1}{2}\log(2\pi e)^n\lvert K\rvert \text{ bits}$

∎

IntuitionThe determinant is the volume

$\lvert K \rvert$ is the squared volume of the ellipsoid the Gaussian occupies, so this formula is the scalar case $\tfrac12\log 2\pi e\sigma^2$ with $\sigma^2 \to \lvert K \rvert$ — again entropy is log-volume, consistent with the typical-set reading above.

Relative entropy and mutual information

DefinitionRelative entropy between densities

$D(f \Vert g) = \int f \log \frac{f}{g}$

$D(f \Vert g)$ is finite only if the support set of $f$ is contained in the support set of $g$ . Motivated by continuity, we set $0\log\frac00 = 0$ .

IntuitionThe same infinite penalty as before

The support condition is the continuous form of the warning from the discrete case: if your codebook $g$ assigns zero probability to something reality $f$ actually produces, the penalty is infinite. Nothing has changed except sums becoming integrals.

DefinitionMutual information between densities

$I(X;Y) = \int f(x,y) \log \frac{f(x,y)}{f(x)f(y)}\, dx\, dy$

FormulaThe familiar identities survive intact

$I(X;Y) = h(X) - h(X \mid Y) = h(Y) - h(Y \mid X) = h(X) + h(Y) - h(X,Y)$

$I(X;Y) = D\big(f(x,y) \,\Vert\, f(x)f(y)\big)$

ImportantMutual information is the well-behaved one

This is the crucial asymmetry of the chapter. Differential entropy is scale-dependent and can be negative; mutual information is neither. It is the limit of the mutual information between quantized versions:

$I(X^\Delta; Y^\Delta) = H(X^\Delta) - H(X^\Delta \mid Y^\Delta) \approx \big(h(X) - \log\Delta\big) - \big(h(X\mid Y) - \log \Delta\big) = I(X;Y)$

The $\log\Delta$ terms cancel. The divergence that made $h$ a relative quantity affects both terms equally and drops out of the difference. This is exactly why the asymptotic-equipartition-property note said differential entropy “is best treated as a term inside invariant expressions such as $D_{\mathrm{KL}}$ or mutual information” — here is the cancellation that makes it so.

The master definition

DefinitionMutual information via partitions

Let $\mathcal{X}$ be the range of $X$ . A partition $P$ of $\mathcal{X}$ is a finite collection of disjoint sets $P_i$ with $\cup_i P_i = \mathcal{X}$ . The quantization of $X$ by $P$ , denoted $[X]_P$ , is the discrete random variable with $\Pr([X]_P = i) = \Pr(X \in P_i) = \int_{P_i} dF(x)$ . Then

$I(X;Y) = \sup_{P, Q} I\big([X]_P ; [Y]_Q\big)$

where the supremum is over all finite partitions $P$ and $Q$ .

ImportantWhy this is the definition that always works

This is the master definition of mutual information: it applies to any joint distribution — ones with atoms, with densities, and with singular parts — and needs no density to exist. Refining the partitions gives a monotonically increasing sequence $I([X]_P; [Y]_Q) \nearrow I$ , so the supremum is a genuine limit rather than an abstract bound. It reduces to the integral formula for variables with a density, and to the discrete definition for discrete variables.

Note that no such master definition exists for $h$ — there is no partition-based quantity converging to differential entropy, because the corresponding sequence diverges. Another way of seeing that $I$ is the fundamental object and $h$ the derived, fragile one.

ExampleMutual information between correlated Gaussians

Let $(X, Y) \sim \mathcal{N}(0, K)$ with

$K = \begin{bmatrix} \sigma^2 & \rho\sigma^2 \\ \rho\sigma^2 & \sigma^2 \end{bmatrix}$

Then $h(X) = h(Y) = \tfrac12\log(2\pi e)\sigma^2$ , and since $\lvert K \rvert = \sigma^4(1-\rho^2)$ , $h(X,Y) = \tfrac12 \log(2\pi e)^2 \sigma^4 (1-\rho^2)$ . So

$I(X;Y) = h(X) + h(Y) - h(X,Y) = -\frac{1}{2}\log(1 - \rho^2)$

Sanity checks: if $\rho = 0$ , $X$ and $Y$ are independent and $I = -\tfrac12\log 1 = 0$ . If $\rho = \pm 1$ they are perfectly correlated and $I = \infty$ — knowing one pins the other to infinite precision, which is infinitely many bits. Note the individual $h$ ‘s depend on $\sigma$ but $I$ does not: rescaling both variables cannot change how much one says about the other.

Properties

TheoremInformation inequality

$D(f \Vert g) \ge 0$ with equality iff $f = g$ almost everywhere.

ProofJensen, exactly as in the discrete case

Let $S$ be the support set of $f$ :

$-D(f\Vert g) = \int_S f \log \frac{g}{f} \;\overset{\text{Jensen}}{\le}\; \log \int_S f \frac{g}{f} = \log \int_S g \le \log 1 = 0$

The middle step is Jensen’s inequality applied to the concave $\log$ (pulling it outside the integral), and the last holds because $\int_S g \le 1$ when $S$ is only part of $g$ ‘s support. Equality requires equality in Jensen, which occurs iff $f = g$ a.e.

∎

CorollaryNon-negativity and conditioning

$I(X;Y) \ge 0$ , with equality iff $X$ and $Y$ are independent.
$h(X \mid Y) \le h(X)$ , with equality iff $X$ and $Y$ are independent.

ImportantWhat did and did not survive the transition

Note carefully: $I \ge 0$ survives, but $h \ge 0$ does not. Conditioning still reduces entropy, and relative entropy is still non-negative — every inequality between entropies carries over, because the divergent $\log\frac1\Delta$ appears on both sides and cancels. Only the statements about the absolute level of $h$ fail.

TheoremChain rule for differential entropy

$h(X_1, X_2, \ldots, X_n) = \sum_{i=1}^n h(X_i \mid X_1, \ldots, X_{i-1})$

(Follows directly from the definitions, as in the [[entropy-mutual-information#Chain rule for entropy ( $n$ variables)|discrete case]].)

CorollaryIndependence bound

$h(X_1, X_2, \ldots, X_n) \le \sum_{i=1}^n h(X_i)$ with equality iff the $X_i$ are independent. (Chain rule plus “conditioning reduces entropy”.)

ExampleApplication — Hadamard’s inequality

Let $X \sim \mathcal{N}(0, K)$ and apply the independence bound. The left side is $\tfrac12\log(2\pi e)^n\lvert K\rvert$ and each right-side term is $\tfrac12\log 2\pi e K_{ii}$ , so

$\lvert K \rvert \le \prod_{i=1}^n K_{ii}$

which is Hadamard’s inequality — a purely linear-algebraic fact, obtained for free from an information-theoretic one. A number of determinant inequalities can be derived this way.

Behaviour under transformations

TheoremTranslation invariance

$h(X + c) = h(X)$ (Immediate from the definition — shifting the density along the axis changes nothing inside the integral.)

TheoremScaling

$h(aX) = h(X) + \log \lvert a \rvert$

ProofChange of variables

Let $Y = aX$ , so $f_Y(y) = \frac{1}{\lvert a\rvert} f_X\!\left(\frac{y}{a}\right)$ . Then

$h(aX) = - \int \frac{1}{\lvert a\rvert} f_X\!\left(\frac{y}{a}\right) \log \left[ \frac{1}{\lvert a\rvert} f_X\!\left(\frac{y}{a}\right) \right] dy$

Substituting $x = y/a$ (so $dy = \lvert a \rvert dx$ ) and splitting the logarithm:

$= - \int f_X(x) \log f_X(x)\, dx + \log\lvert a\rvert = h(X) + \log \lvert a \rvert$

∎

CorollaryLinear maps

$h(AX) = h(X) + \log \lvert \det(A) \rvert$

WarningThis is the non-invariance the AEP note warned about

Discrete entropy is invariant under any relabelling; differential entropy is not. Stretching the variable by $a$ adds $\log\lvert a\rvert$ bits, because it stretches the typical volume by the same factor. Switching from volts to millivolts changes $h$ by $\log 1000$ without changing a single physical fact — which is the sharpest possible statement that $h$ is not an intrinsic property of the random variable, only of the variable together with a choice of scale. Once again, $I(X;Y)$ is immune: the $\log\lvert a \rvert$ cancels in the difference.

The Gaussian maximizes entropy

TheoremMaximum entropy under a covariance constraint

Let $X \in \mathbb{R}^n$ have zero mean and covariance $K = \mathbb{E}XX^t$ . Then

$h(X) \le \frac{1}{2}\log(2\pi e)^n \lvert K \rvert$

with equality iff $X \sim \mathcal{N}(0, K)$ .

ProofA relative-entropy trick: the cross term is blind to which density it sees

Let $g$ be any density with $\int g(x) x_i x_j\, dx = K_{ij}$ , and let $\phi_K$ be the $\mathcal{N}(0,K)$ density. Start from non-negativity:

$0 \le D(g \Vert \phi_K) = \int g \log\frac{g}{\phi_K} = -h(g) - \int g \log \phi_K$

The key step: $\log \phi_K(x)$ is a quadratic form in $x$ , so $\int g \log\phi_K$ depends on $g$ only through its second moments — which by assumption match those of $\phi_K$ . Hence we may substitute:

$\int g \log \phi_K = \int \phi_K \log \phi_K$

and the right side is $-h(\phi_K)$ . Therefore

$0 \le -h(g) + h(\phi_K) \quad \Longrightarrow \quad h(g) \le h(\phi_K)$

∎

IntuitionWhy the Gaussian, and why this proof is slick

The constraint fixes the covariance — the “energy budget” or the spread. Among all densities with that spread, the Gaussian is the most dispersed, hence the most uncertain. The proof works because the Gaussian’s log-density is exactly a quadratic form, so the constraint we imposed is precisely the statistic the comparison term depends on. This pairing of a constraint with an exponential family is the general recipe for maximum-entropy results.

Compare the discrete maximum entropy bound $H(X) \le \log\lvert\mathcal{X}\rvert$ , proved by the same $D(p\Vert u) \ge 0$ trick against the uniform. Same argument, different reference distribution: uniform when the constraint is a bounded alphabet, Gaussian when the constraint is a covariance.

Estimation error and differential entropy

The problem. Fano’s inequality bounded the probability of error when guessing a discrete symbol. For a continuous variable, “wrong” is the wrong question — any estimate is wrong with probability 1. The right question is how far off you are, and the natural measure is squared error. Fano’s continuous counterpart therefore bounds variance rather than error probability.

TheoremEstimation counterpart to Fano’s inequality

For any random variable $X$ and estimator $\hat{X}$ , with $h(X)$ in nats,

$\mathbb{E}(X - \hat{X})^2 \ge \frac{1}{2\pi e} e^{2h(X)}$

with equality if and only if $X$ is Gaussian and $\hat{X}$ is the mean of $X$ .

ProofBest estimator, then maximum entropy in reverse

$\mathbb{E}(X-\hat{X})^2 \;\overset{(a)}{\ge}\; \min_{\hat{X}} \mathbb{E}(X-\hat{X})^2 = \mathbb{E}\big(X - \mathbb{E}(X)\big)^2 = \text{var}(X) \;\overset{(b)}{\ge}\; \frac{1}{2\pi e} e^{2h(X)}$

where (a) uses that the mean is the best mean-squared estimator, and (b) is the maximum-entropy theorem read backwards: since $h(X) \le \tfrac12\ln 2\pi e\,\text{var}(X)$ , solving for the variance gives $\text{var}(X) \ge \frac{1}{2\pi e}e^{2h(X)}$ . Equality in (a) needs $\hat{X}$ to be the mean; equality in (b) needs $X$ Gaussian.

∎

CorollaryWith side information

Given side information $Y$ and estimator $\hat{X}(Y)$ ,

$\mathbb{E}\big(X - \hat{X}(Y)\big)^2 \ge \frac{1}{2\pi e} e^{2h(X \mid Y)}$

IntuitionWhat this says physically

You cannot estimate a variable more accurately than its differential entropy permits. A source with high $h$ occupies a large effective volume, and no amount of cleverness shrinks your expected squared error below that volume’s scale. The corollary is the useful engineering form: observing $Y$ helps exactly insofar as it reduces $h(X \mid Y)$ — which, by the identity $I(X;Y) = h(X) - h(X\mid Y)$ , means the estimation gain is governed by the mutual information the observation carries. This is the bridge from information theory to estimation theory.

Quick reference

Object	Formula	Note
Differential entropy	$h(X) = -\int_S f\log f$	Can be negative; not “bits of surprise”.
Uniform on $[0,a]$	$h = \log a$	$2^h = a$ , the support volume.
Gaussian	$h = \tfrac12\log 2\pi e\sigma^2$	Mean-independent.
Multivariate normal	$h = \tfrac12\log(2\pi e)^n\lvert K\rvert$	Determinant $=$ volume.
Continuous AEP	$-\tfrac1n\log f(X^n) \to h(X)$ in prob.	Same weak-law proof.
Typical volume	$(1-\epsilon)2^{n(h-\epsilon)} \le \text{Vol} \le 2^{n(h+\epsilon)}$	Volume replaces cardinality.
Interpretation	side length $= 2^{h}$	Log of the effective side length.
Quantization	$H(X^\Delta) + \log\Delta \to h(X)$	$n$ -bit description costs $\approx h(X) + n$ .
Relative entropy	$D(f\Vert g) = \int f\log\frac fg$	$\ge 0$ ; needs $\text{supp}(f) \subseteq \text{supp}(g)$ .
Mutual information	$I = h(X) - h(X\mid Y) = D(f_{XY}\Vert f_Xf_Y)$	$\ge 0$ , scale-invariant, has a master definition.
Correlated Gaussians	$I = -\tfrac12\log(1-\rho^2)$	$0$ at $\rho=0$ , $\infty$ at $\rho=\pm1$ .
Translation	$h(X+c) = h(X)$	—
Scaling	$h(aX) = h(X) + \log\lvert a\rvert$ ; $h(AX) = h(X) + \log\lvert\det A\rvert$	Not relabelling-invariant.
Max entropy	$h(X) \le \tfrac12\log(2\pi e)^n\lvert K\rvert$	Gaussian is extremal at fixed covariance.
Estimation bound	$\mathbb{E}(X-\hat X)^2 \ge \frac{1}{2\pi e}e^{2h(X)}$	Fano’s continuous counterpart.

What carries over from the discrete theory, and what does not:

Survives	Fails
$D \ge 0$ , $I \ge 0$	$h \ge 0$ — differential entropy can be negative
Chain rules for $h$ , $I$ , $D$	Relabelling invariance — $h$ shifts by $\log\lvert\det A\rvert$
Conditioning reduces entropy	Absolute interpretation as “bits” — only $h$ differences mean anything
Independence bound, max-entropy arguments	A partition-based master definition of $h$ (exists for $I$ only)
The AEP and typical-set machinery, with volume for cardinality	Finiteness — $\infty - \infty$ is a real hazard in $h(X\mid Y) = h(X,Y) - h(Y)$

The one-line summary: $h$ is discrete entropy minus the divergent $\log\frac{1}{\Delta}$ cost of infinite precision, so it is a quantity relative to a scale. Every identity in which that divergence cancels — every difference of entropies — survives intact; every statement about the absolute level of $h$ does not.

References

[[lit/cover2006-elements|Cover & Thomas, Elements of Information Theory]] — Chapter 8

Related