GIFT: Unlocking Global Optimality in Post-Training via Finite-Temperature Gibbs Initialization
By: Zhengyang Zhao , Lu Ma , Yizhen Jiang and more
The prevailing post-training paradigm for Large Reasoning Models (LRMs)--Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL)--suffers from an intrinsic optimization mismatch: the rigid supervision inherent in SFT induces distributional collapse, thereby exhausting the exploration space necessary for subsequent RL. In this paper, we reformulate SFT within a unified post-training framework and propose Gibbs Initialization with Finite Temperature (GIFT). We characterize standard SFT as a degenerate zero-temperature limit that suppresses base priors. Conversely, GIFT incorporates supervision as a finite-temperature energy potential, establishing a distributional bridge that ensures objective consistency throughout the post-training pipeline. Our experiments demonstrate that GIFT significantly outperforms standard SFT and other competitive baselines when utilized for RL initialization, providing a mathematically principled pathway toward achieving global optimality in post-training. Our code is available at https://github.com/zzy1127/GIFT.
Similar Papers
Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting
Machine Learning (CS)
Keeps AI smart while teaching it new things.
On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
Machine Learning (CS)
Makes AI learn better from examples.
GiFT: Gibbs Fine-Tuning for Code Generation
Machine Learning (CS)
Teaches computers to write better code.