Diffusion Models

The course’s central object, built up over five lectures. Start from a hierarchical VAE — many latent layers instead of one — and specialize it to the variational diffusion model (VDM), where the “encoder” at each layer is a fixed, simple Gaussian noising step rather than a learned network; this fixes the ELBO into a sum of per-step denoising terms. The resulting training objective admits three equivalent parameterizations — predict the clean image, predict the added noise, or predict the score $\nabla_x \log p(x)$ — related to each other by Tweedie’s formula, which is also the bridge to score matching as an independent derivation of the same model. The score’s role as an MMSE-optimal denoiser motivates Langevin dynamics as a sampler, noise annealing to make Langevin dynamics work across scales, and the final synthesis: annealed Langevin dynamics, score matching, and DDPM sampling are all the same time-varying force field, which is exactly a discretized differential equation — the bridge to the SDE/flow formulation that follows.

Outline (TODO — flesh out each)

• Hierarchical VAEs
• Variational diffusion models (VDM)
• Gaussian encoders; the updated ELBO for VDM
• Three interpretations: image prediction, noise prediction, score prediction
• Tweedie’s formula → score matching
• The score as an MMSE estimator (optimal denoiser)
• Langevin dynamics; noise annealing
• Zoom out: annealed Langevin dynamics, score matching, and DDPM as one idea
• Time-varying force field = a differential equation