Regularization Implies balancedness in the deep linear network
By: Kathryn Lindsey, Govind Menon
Potential Business Impact:
Makes computer learning faster and simpler.
We use geometric invariant theory (GIT) to study the deep linear network (DLN). The Kempf-Ness theorem is used to establish that the $L^2$ regularizer is minimized on the balanced manifold. This allows us to decompose the training dynamics into two distinct gradient flows: a regularizing flow on fibers and a learning flow on the balanced manifold. We show that the regularizing flow is exactly solvable using the moment map. This approach provides a common mathematical framework for balancedness in deep learning and linear systems theory. We use this framework to interpret balancedness in terms of model reduction and Bayesian principles.
Similar Papers
Entropic Regularization in the Deep Linear Network
Neural and Evolutionary Computing
Makes computer learning faster and more accurate.
An entropy formula for the Deep Linear Network
Machine Learning (CS)
Teaches computers how to learn like a brain.
Manifold Learning with Normalizing Flows: Towards Regularity, Expressivity and Iso-Riemannian Geometry
Machine Learning (CS)
Makes computers understand messy, mixed-up data better.