The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold

Deep neural networks are capable of exploring arbitrary hypotheses. However, we demonstrate that when paired with a concrete task and algorithm, the training of deep networks explores only a tiny fraction of the space of available hypotheses. To visualize and identify this space, we developed an information-geometric lens. Focusing this lens on our experimental data uncovers that networks with many different architectures, trained with different optimization procedures and regularization techniques, traverse the same manifold. Moreover, networks trained on different tasks also lie on a low-dimensional manifold.

We study the details of this manifold and find that networks with different architectures follow distinguishable trajectories, while other factors have minimal influence: larger networks train along a similar manifold as smaller networks, just faster; and networks initialized at very different points in the prediction space converge to solutions along a similar manifold. We analytically predict this phenomenon for linear networks, showing that it critically depends on the structure of the task.

This work was conducted in collaboration with Pratik Chaudhari (University of Pennsylvania), Jialin Mao (University of Pennsylvania), Rahul Ramesh (University of Pennsylvania), Rubing Yang (University of Pennsylvania), Mark Transtrum (Brigham Young University), Han Kheng Teoh (Cornell University), and James P. Sethna (Cornell University).