SignRoundV2: Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs
By: Wenhua Cheng , Weiwei Zhang , Heng Guo and more
Potential Business Impact:
Makes AI smarter using less computer power.
Extreme low-bit quantization is critical for efficiently deploying Large Language Models (LLMs), yet it often leads to severe performance degradation at 2-bits and even 4-bits (e.g., MXFP4). We present SignRoundV2, a post-training quantization framework that is highly effective even without mixed-precision. SignRoundV2 introduces (1) a fast sensitivity metric that combines gradient information with quantization-induced deviations to guide layer-wise bit allocation, and (2) a lightweight pre-tuning search for quantization scales to improve extremely low-bit quantization. These components allow SignRoundV2 to close the gap with full-precision models. Extensive experiments indicate that our method sustains competitive accuracy for LLMs, achieving production-grade performance with about 1 percent variance at 4-5 bits and strong results even at 2 bits. The implementation is available at https://github.com/intel/auto-round.
Similar Papers
SplitQuantV2: Enhancing Low-Bit Quantization of LLMs Without GPUs
Machine Learning (CS)
Makes smart computer programs run on small devices.
LittleBit: Ultra Low-Bit Quantization via Latent Factorization
Machine Learning (CS)
Makes big AI models fit on small devices.
Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Models
Computation and Language
Makes big AI models smaller, faster, cheaper.