PARQ: Piecewise-Affine Regularized Quantization
By: Lisa Jin , Jianhao Ma , Zechun Liu and more
Potential Business Impact:
Makes computer models smaller and faster.
We develop a principled method for quantization-aware training (QAT) of large-scale machine learning models. Specifically, we show that convex, piecewise-affine regularization (PAR) can effectively induce the model parameters to cluster towards discrete values. We minimize PAR-regularized loss functions using an aggregate proximal stochastic gradient method (AProx) and prove that it has last-iterate convergence. Our approach provides an interpretation of the straight-through estimator (STE), a widely used heuristic for QAT, as the asymptotic form of PARQ. We conduct experiments to demonstrate that PARQ obtains competitive performance on convolution- and transformer-based vision tasks.
Similar Papers
Quantization through Piecewise-Affine Regularization: Optimization and Statistical Guarantees
Machine Learning (CS)
Makes computers learn better with less data.
ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference
Computation and Language
Makes big AI models smaller and faster.
Stabilizing Quantization-Aware Training by Implicit-Regularization on Hessian Matrix
CV and Pattern Recognition
Makes AI models smaller and faster.