Mathematical Foundations of Neural Tangents and Infinite-Width Networks
By: Rachana Mysore , Preksha Girish , Kavitha Jayaram and more
Potential Business Impact:
Makes AI learn better and faster.
We investigate the mathematical foundations of neural networks in the infinite-width regime through the Neural Tangent Kernel (NTK). We propose the NTK-Eigenvalue-Controlled Residual Network (NTK-ECRN), an architecture integrating Fourier feature embeddings, residual connections with layerwise scaling, and stochastic depth to enable rigorous analysis of kernel evolution during training. Our theoretical contributions include deriving bounds on NTK dynamics, characterizing eigenvalue evolution, and linking spectral properties to generalization and optimization stability. Empirical results on synthetic and benchmark datasets validate the predicted kernel behavior and demonstrate improved training stability and generalization. This work provides a comprehensive framework bridging infinite-width theory and practical deep-learning architectures.
Similar Papers
Finite-Width Neural Tangent Kernels from Feynman Diagrams
Machine Learning (CS)
Helps computers learn better by understanding tiny changes.
Finite-Width Neural Tangent Kernels from Feynman Diagrams
Machine Learning (CS)
Explains how computer brains learn better.
The Spectral Dimension of NTKs is Constant: A Theory of Implicit Regularization, Finite-Width Stability, and Scalable Estimation
Machine Learning (CS)
Helps computers learn better with fewer mistakes.