Entropic Confinement and Mode Connectivity in Overparameterized Neural Networks
By: Luca Di Carlo, Chase Goddard, David J. Schwab
Potential Business Impact:
Keeps AI focused on one good answer.
Modern neural networks exhibit a striking property: basins of attraction in the loss landscape are often connected by low-loss paths, yet optimization dynamics generally remain confined to a single convex basin and rarely explore intermediate points. We resolve this paradox by identifying entropic barriers arising from the interplay between curvature variations along these paths and noise in optimization dynamics. Empirically, we find that curvature systematically rises away from minima, producing effective forces that bias noisy dynamics back toward the endpoints - even when the loss remains nearly flat. These barriers persist longer than energetic barriers, shaping the late-time localization of solutions in parameter space. Our results highlight the role of curvature-induced entropic forces in governing both connectivity and confinement in deep learning landscapes.
Similar Papers
Neural Thermodynamics I: Entropic Forces in Deep and Universal Representation Learning
Machine Learning (CS)
Explains how AI learns by using "entropic forces."
On the Convergence of Overparameterized Problems: Inherent Properties of the Compositional Structure of Neural Networks
Machine Learning (CS)
Makes AI learn faster by understanding its own structure.
Overparametrization bends the landscape: BBP transitions at initialization in simple Neural Networks
Disordered Systems and Neural Networks
Helps computers learn better with more data.