Rethinking PCA Through Duality
By: Jan Quan, Johan Suykens, Panagiotis Patrinos
Potential Business Impact:
Finds hidden patterns in data more easily.
Motivated by the recently shown connection between self-attention and (kernel) principal component analysis (PCA), we revisit the fundamentals of PCA. Using the difference-of-convex (DC) framework, we present several novel formulations and provide new theoretical insights. In particular, we show the kernelizability and out-of-sample applicability for a PCA-like family of problems. Moreover, we uncover that simultaneous iteration, which is connected to the classical QR algorithm, is an instance of the difference-of-convex algorithm (DCA), offering an optimization perspective on this longstanding method. Further, we describe new algorithms for PCA and empirically compare them with state-of-the-art methods. Lastly, we introduce a kernelizable dual formulation for a robust variant of PCA that minimizes the $l_1$ deviation of the reconstruction errors.
Similar Papers
A Reproduction Study: The Kernel PCA Interpretation of Self-Attention Fails Under Scrutiny
Machine Learning (CS)
Shows self-attention doesn't work like a math trick.
Axes that matter: PCA with a difference
Computational Finance
Makes complex math problems simpler for computers.
Optimization over Trained Neural Networks: Difference-of-Convex Algorithm and Application to Data Center Scheduling
Optimization and Control
Solves tough computer problems for cheaper energy.