Guidance, CLIP, and Text-to-Image Models
Stub. Steering a diffusion model's generation toward a class or condition — classifier-based guidance (the derivative of classifier logits), classifier-free guidance and its scaling factor; and how this enables text-to-image generation, via CLIP as the text/image alignment model and latent diffusion models (LDM) for efficiency.
An unconditional diffusion model samples from ; guidance is how you turn it into a conditional sampler for without necessarily retraining from scratch. Classifier-based guidance adds the gradient of an auxiliary classifier’s log-probability to the score; classifier-free guidance instead trains a single model on both conditional and unconditional objectives and extrapolates between them at sampling time, controlled by a scaling factor that trades diversity for fidelity to the condition. Text-to-image generation is the flagship application: CLIP provides a shared text/image embedding space to condition on, and latent diffusion models (LDM) run the diffusion process in a compressed latent space (via a VAE) rather than pixel space, for tractability at scale.
Outline (TODO — flesh out each)
- Classifier-based guidance — the derivative of classifier logits
- Classifier-free guidance
- The guidance scaling factor
- The text-to-image (T2I) problem statement
- CLIP — joint text/image embeddings
- Latent diffusion models (LDM)