Editing Diffusion Models — Low-Rank Structure and Surgery
Stub. Inpainting/outpainting as a form of "surgery" on a pretrained diffusion model, why diffusion models generalize rather than memorize (low-dimensional subspaces in the denoiser), and precise, training-free editing via the Jacobian of the denoiser — the LOCO Edit method.
Treats a pretrained diffusion model as an object to be edited rather than just sampled from. Inpainting and outpainting are the simplest form of surgery — constrain part of the generation to match given content. The deeper question is why diffusion models generalize (produce novel, coherent samples) instead of memorizing the training set, which recent work traces to the denoiser’s Jacobian having a low-dimensional range — the model effectively projects onto a low-dimensional data manifold at each step. That same Jacobian structure enables precise, training-free semantic editing: LOCO Edit finds low-rank, localized directions in the denoiser’s Jacobian that correspond to interpretable edits.
Outline (TODO — flesh out each)
- Inpainting and outpainting as model surgery
- Generalization and reproducibility in diffusion models
- Low-dimensional subspaces in the denoiser
- Precise editing of diffusion models
- The Jacobian of the denoiser
- LOCO Edit