Entropic Regularization in the Deep Linear Network
By: Alan Chen, Tejas Kotwal, Govind Menon
Potential Business Impact:
Makes computer learning faster and more accurate.
We study regularization for the deep linear network (DLN) using the entropy formula introduced in arXiv:2509.09088. The equilibria and gradient flow of the free energy on the Riemannian manifold of end-to-end maps of the DLN are characterized for energies that depend symmetrically on the singular values of the end-to-end matrix. The only equilibria are minimizers and the set of minimizers is an orbit of the orthogonal group. In contrast with random matrix theory there is no singular value repulsion. The corresponding gradient flow reduces to a one-dimensional ordinary differential equation whose solution gives explicit relaxation rates toward the minimizers. We also study the concavity of the entropy in the chamber of singular values. The entropy is shown to be strictly concave in the Euclidean geometry on the chamber but not in the Riemannian geometry defined by the DLN metric.
Similar Papers
An entropy formula for the Deep Linear Network
Machine Learning (CS)
Teaches computers how to learn like a brain.
Regularization Implies balancedness in the deep linear network
Machine Learning (CS)
Makes computer learning faster and simpler.
Gradient Flow Equations for Deep Linear Neural Networks: A Survey from a Network Perspective
Machine Learning (CS)
Helps computers learn by simplifying math.