Relative Trajectory Balance is equivalent to Trust-PCL
By: Tristan Deleu , Padideh Nouri , Yoshua Bengio and more
Potential Business Impact:
Makes AI models create better, more focused results.
Recent progress in generative modeling has highlighted the importance of Reinforcement Learning (RL) for fine-tuning, with KL-regularized methods in particular proving to be highly effective for both autoregressive and diffusion models. Complementing this line of work, the Relative Trajectory Balance (RTB) objective was recently introduced in the context of Generative Flow Networks (GFlowNets) to serve the same role of improving fine-tuning in sequential generative models. Building on prior work linking GFlowNets and maximum-entropy RL, we establish in this paper an equivalence between RTB and Trust-PCL, an off-policy RL method with KL regularization. This equivalence situates RTB within the broader theoretical landscape of KL-regularized RL, and clarifies its relationship to earlier methods. Leveraging this insight, we revisit an illustrative example from the RTB paper and show that KL-regularized RL methods achieve comparable performance, offering an alternative perspective to what was previously reported.
Similar Papers
Solving Bayesian inverse problems with diffusion priors and off-policy RL
Machine Learning (CS)
Makes AI better at solving hard science puzzles.
Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training
Machine Learning (CS)
Teaches AI to learn faster and better.
Best Policy Learning from Trajectory Preference Feedback
Machine Learning (CS)
Teaches AI to learn better from people's choices.