Why Smooth Stability Assumptions Fail for ReLU Learning
By: Ronald Katende
Stability analyses of modern learning systems are frequently derived under smoothness assumptions that are violated by ReLU-type nonlinearities. In this note, we isolate a minimal obstruction by showing that no uniform smoothness-based stability proxy such as gradient Lipschitzness or Hessian control can hold globally for ReLU networks, even in simple settings where training trajectories appear empirically stable. We give a concrete counterexample demonstrating the failure of classical stability bounds and identify a minimal generalized derivative condition under which stability statements can be meaningfully restored. The result clarifies why smooth approximations of ReLU can be misleading and motivates nonsmooth-aware stability frameworks.
Similar Papers
On the Stability of Neural Networks in Deep Learning
Machine Learning (CS)
Makes computer brains more stable and trustworthy.
Analytic and Variational Stability of Deep Learning Systems
Machine Learning (CS)
Makes AI learn better and stay stable.
On the Complexity-Faithfulness Trade-off of Gradient-Based Explanations
Machine Learning (CS)
Makes AI explanations clearer and more trustworthy.