Non-Vacuous Generalization Bounds: Can Rescaling Invariances Help?
By: Damien Rouchouse , Antoine Gonon , Rémi Gribonval and more
Potential Business Impact:
Makes AI learn better by fixing how it measures learning.
A central challenge in understanding generalization is to obtain non-vacuous guarantees that go beyond worst-case complexity over data or weight space. Among existing approaches, PAC-Bayes bounds stand out as they can provide tight, data-dependent guarantees even for large networks. However, in ReLU networks, rescaling invariances mean that different weight distributions can represent the same function while leading to arbitrarily different PAC-Bayes complexities. We propose to study PAC-Bayes bounds in an invariant, lifted representation that resolves this discrepancy. This paper explores both the guarantees provided by this approach (invariance, tighter bounds via data processing) and the algorithmic aspects of KL-based rescaling-invariant PAC-Bayes bounds.
Similar Papers
PAC-Bayesian Generalization Bounds for Graph Convolutional Networks on Inductive Node Classification
Machine Learning (CS)
Helps computers learn from changing online connections.
Some theoretical improvements on the tightness of PAC-Bayes risk certificates for neural networks
Machine Learning (CS)
Makes AI more trustworthy and reliable.
PAC-Bayesian Reinforcement Learning Trains Generalizable Policies
Machine Learning (CS)
Helps robots learn faster and safer.