Understanding multi-pass stochastic gradient descent via dynamical mean-field theory

Artificial neural networks trained via stochastic gradient descent (SGD) have achieved impressive performances. A general consensus has arisen that understanding SGD successful optimization requires a detailed description of the dynamical trajectory traversed during training. Yet this task is highly nontrivial given that SGD follows a nonequilibrium dynamics in a high-dimensional non-convex loss landscape. Thus, the practical success of SGD is still largely unexplained. We have applied dynamical mean-field theory to derive a full description of the learning curves of SGD and of its performances in prototypical models. This is the first work tracking the high-dimensional dynamics of SGD in the realistic case where the network reuses the available examples multiple times. We have also investigated how different sources of algorithmic noise affect the performance. Comparing SGD to gradient descent in an intrinsically hard problem (phase retrieval) we have shown that SGD noise is key to find good solutions. We have found that an effective fluctuation-dissipation theorem characterizes the stationary dynamics of SGD, extracting the related effective temperature as a function of the hyperparameters. These results point out a novel analogy between SGD and active and driven systems.