TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control
By: Yuxiang Chen , Xiaoming Xu , Pengle Zhang and more
Potential Business Impact:
Makes big computer brains train much faster and cheaper.
Large Language Models (LLMs) training is prohibitively expensive, driving interest in low-precision fully-quantized training (FQT). While novel 4-bit formats like NVFP4 offer substantial efficiency gains, achieving near-lossless training at such low precision remains challenging. We introduce TetraJet-v2, an end-to-end 4-bit FQT method that leverages NVFP4 for activations, weights, and gradients in all linear layers. We identify two critical issues hindering low-precision LLM training: weight oscillation and outliers. To address these, we propose: 1) an unbiased double-block quantization method for NVFP4 linear layers, 2) OsciReset, an algorithm to suppress weight oscillation, and 3) OutControl, an algorithm to retain outlier accuracy. TetraJet-v2 consistently outperforms prior FP4 training methods on pre-training LLMs across varying model sizes up to 370M and data sizes up to 200B tokens, reducing the performance gap to full-precision training by an average of 51.3%.
Similar Papers
Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling
Computation and Language
Makes AI models train faster and better.
Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling
Computation and Language
Trains AI faster and better using less computer memory.
SignRoundV2: Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs
Computation and Language
Makes AI smarter using less computer power.