Learning as a Dynamical System

Corresponds to lectures: Learning as a dynamical system (9/24); SGD as a random dynamical system (9/29); Learning can converge without forgetting (10/1); Where can the past live? (10/6) — the syllabus’s “Learning as a dynamical system” unit (weeks 5–6; vanishing/exploding gradients and gradient stabilization are expected in later lectures in this unit). The central move is to stop thinking of training as “an optimizer finds a minimum” and instead treat the sequence of parameter updates as a trajectory of a dynamical system: gradient descent is the discrete-time map version of the continuous gradient-flow ODE $\dot\theta = -\nabla L(\theta)$ (the same object Langevin dynamics perturbs with noise for sampling, run here instead to find a minimum), and SGD is that same map driven by random minibatch noise — a random dynamical system (stochastic-bandits-and-regret covers the measure-theoretic machinery for reasoning rigorously about randomness unfolding over time). Learning “without forgetting” and “where the past can live” are both questions about a recurrent/continually-trained system’s memory: which directions in parameter or activity space preserve information about earlier inputs or earlier training data without it decaying or being overwritten — directly governed by the stability/Lyapunov analysis of the previous two notes.

Outline (TODO — flesh out each)

• Gradient flow and gradient descent as a dynamical system
• SGD as a random dynamical system
• Learning can converge without forgetting
• Where can the past live? — memory in recurrent dynamics
• Vanishing / exploding gradients (pending — later lecture in this unit)
• Gradient stabilization (pending — later lecture in this unit)