A mathematical analysis of learning in both biological and artificial neural networks

ORAL · Invited

Abstract

An exciting area of intellectual activity in this century may well revolve around a synthesis of machine learning, theoretical physics, and neuroscience.  The unification of these fields will likely enable us to exploit the power of complex systems analysis, developed in theoretical physics and applied mathematics, to elucidate the design principles governing neural systems, both biological and artificial, and deploy these principles to develop better algorithms in machine learning.  We will give several vignettes in this direction, including:  (1) determining the best optimization problem to solve in order to learn regression in high dimensions;  (2) finding exact solutions to the dynamics of generalization error in deep linear networks; (3) analyzing how neural networks can learn semantic concepts like infants. 

–

Publication: M. Advani and S. Ganguli, Statistical mechanics of optimal convex inference in high dimensions, Physical Review X, 2016.

A.K. Lampinen and S. Ganguli, An analytic theory of generalization dynamics and transfer learning in deep linear networks, International Conference on Learning Representations (ICLR), 2019.

A. Saxe, J. McClelland, S. Ganguli, A. Saxe, J. McClelland, S. Ganguli, Exact solutions to the nonlinear dynamics of learning in deep neural networks, PNAS 2019.

Y. Bahri, J. Kadmon, J. Pennington, S. Schoenholz, J. Sohl-Dickstein, and S. Ganguli, Statistical mechanics of deep learning, Annual Reviews of Condensed Matter Physics, 2020.

Presenters

  • Surya Ganguli

    Stanford, Stanford University

Authors

  • Surya Ganguli

    Stanford, Stanford University