Variational Autoencoders

Turns the plain autoencoder (a deterministic bottleneck trained to reconstruct its input) into a generative model by making the latent code probabilistic: an encoder distribution $q(z\mid x)$, a decoder/likelihood $p(x\mid z)$, and a prior $p(z)$ you can sample from at generation time. Training maximizes the ELBO — a lower bound on the log-likelihood that decomposes into a reconstruction term and a KL term pulling the encoder toward the prior. Visualizing what this actually does to the latent space exposes the VAE’s characteristic failure modes: the reconstruction/KL tradeoff behaving like an entropy push-pull, regions of the prior the encoder never populates (the “prior hole”), and the decoder learning to ignore the latent code entirely (posterior collapse) — problems diffusion models are partly a response to.

Outline (TODO — flesh out each)

• The autoencoder (AE) — deterministic bottleneck, reconstruction loss
• Giving the latent code semantics
• The VAE’s probabilistic story; the evidence lower bound (ELBO)
• Visualizing the VAE — the “moon eclipse” picture
• Failure mode: entropy push-pull between reconstruction and KL terms
• Failure mode: the prior hole problem
• Failure mode: posterior collapse