Tight analyses of first-order methods with error feedback
By: Daniel Berg Thomsen, Adrien Taylor, Aymeric Dieuleveut
Potential Business Impact:
Makes computers learn faster by talking less.
Communication between agents often constitutes a major computational bottleneck in distributed learning. One of the most common mitigation strategies is to compress the information exchanged, thereby reducing communication overhead. To counteract the degradation in convergence associated with compressed communication, error feedback schemes -- most notably $\mathrm{EF}$ and $\mathrm{EF}^{21}$ -- were introduced. In this work, we provide a tight analysis of both of these methods. Specifically, we find the Lyapunov function that yields the best possible convergence rate for each method -- with matching lower bounds. This principled approach yields sharp performance guarantees and enables a rigorous, apples-to-apples comparison between $\mathrm{EF}$, $\mathrm{EF}^{21}$, and compressed gradient descent. Our analysis is carried out in a simplified yet representative setting, which allows for clean theoretical insights and fair comparison of the underlying mechanisms.
Similar Papers
Improved Convergence in Parameter-Agnostic Error Feedback through Momentum
Optimization and Control
Makes AI learn faster without needing expert settings.
Safe-EF: Error Feedback for Nonsmooth Constrained Optimization
Machine Learning (CS)
Robots learn faster, safer, with less data sent.
Composite Optimization with Error Feedback: the Dual Averaging Approach
Optimization and Control
Makes computers learn faster with less data.