Differentiation, Adjoints, and Automatic Differentiation

Corresponds to Chapter 11. A deep network (or any long nested composition $f_n\circ\cdots\circ f_1$) has a gradient that, by the chain rule, is a product of many Jacobians — computing this product left-to-right (forward) is possible but wasteful for a scalar loss with many parameters; the method of adjoints computes it right-to-left (backward) instead, reusing each intermediate Jacobian-vector product exactly once, which is the mathematical content of backpropagation in deep learning. Automatic differentiation is the general software technique implementing this (and its forward-mode dual) for arbitrary code, not just explicit compositions. The chapter closes with an alternative route to the same adjoint equations via the Lagrangian and the implicit function theorem — reformulating a constrained (“progressive”/sequential) function evaluation as a constrained optimization problem — which extends directly to optimal control problems like LQR, where the adjoint variables are exactly the costate trajectory from Pontryagin’s maximum principle.

Outline (TODO — flesh out each)

• The chain rule for a nested composition of vector functions
• The method of adjoints
• Adjoints in deep learning (backpropagation)
• Automatic differentiation
• Derivations via the Lagrangian and the implicit function theorem
• A constrained optimization formulation of the progressive function
• Unconstrained vs. constrained formulations, a general perspective
• Extension: control