Information Theory
Entropy, coding, and the limits of compression and communication.
Entropy, mutual information, source/channel coding, and the theoretical limits on compression and reliable communication.
Notes
- Maximum Entropy The maximum entropy distribution under moment constraints, worked examples, the anomalous ε-achievable case, spectrum estimation, and Burg's maximum entropy theorem
- Information Theory and Statistics The method of types, universal source coding, Sanov's theorem and large deviations, the conditional limit theorem, hypothesis testing, Chernoff–Stein and Chernoff information, and Fisher information
- Rate Distortion Theory Lossy compression — quantization, the rate distortion function for binary and Gaussian sources, reverse water-filling, the converse and achievability proofs, and the Blahut–Arimoto algorithm
- The Gaussian Channel Capacity of the additive white Gaussian noise channel, sphere packing, bandlimited channels and the Shannon–Hartley formula, water-filling over parallel and colored channels, and how little feedback buys
- Differential Entropy Entropy of continuous random variables, the continuous AEP and volume of the typical set, the quantization relation, the Gaussian as maximum-entropy distribution, and the estimation counterpart to Fano
- Channel Capacity The information capacity of a discrete memoryless channel, worked examples, jointly typical sequences, Shannon's second theorem and its converse, Hamming codes, feedback, and source–channel separation
- Entropy Rates of a Stochastic Process Stationarity, Markov chains, the entropy rate and its two definitions, random walks on graphs, the second law, and functions of Markov chains
- The Asymptotic Equipartition Property The information-theoretic law of large numbers, the typical set, the source code it yields, and its optimality among high-probability sets
- Information Theory Inequalities Jensen's Inequality, Log Sum Inequality, Data Processing Inequality and Sufficient Statistics, and Fano's Inequality
- Elements of Information Theory My annotations of the book "Elements of Information Theory" (Cover & Thomas).
- Data Compression and Source Coding Kraft inequality, Huffman codes, and Shannon-Fano-Elias coding bounds.
- Information, Entropy, Relative Entropy, and Mutual Information Fundamental definitions, Relationships between them, and Chain rules
- Kolmogorov Complexity Incompressible sequences, Occam's Razor, and the Minimum Description Length principle.
- Network Information Theory Multiple-access channels, Slepian-Wolf encoding, and broadcast/relay channels.
- Information Theory in Gambling and Portfolios Side information, log-optimal portfolios, Kelly criterion, and universal portfolios.
- Universal Source Coding Arithmetic coding, Lempel-Ziv algorithms, and optimality proofs.