ToMPO: Training LLM Strategic Decision Making from a Multi-Agent Perspective
By: Yiwen Zhang , Ziang Chen , Fanqi Kong and more
Potential Business Impact:
Helps computers make smarter choices by guessing others' plans.
Large Language Models (LLMs) have been used to make decisions in complex scenarios, where they need models to think deeply, reason logically, and decide wisely. Many existing studies focus solely on multi-round conversations in social tasks or simulated environments, neglecting the various types of decisions and their interdependence. Current reinforcement learning methods struggle to consider the strategies of others during training. To address these issues, we first define a strategic decision-making problem that includes two types of decisions and their temporal dependencies. Furthermore, we propose **T**heory **o**f **M**ind **P**olicy **O**ptimization **(ToMPO)** algorithm to optimize the perception of other individual strategies and the game situation trends. Compared to the Group Relative Policy Optimization (GRPO) algorithm, ToMPO enhances the LLM's strategic decision-making mainly by: 1) generating rollouts based on reasoning the strategies of other individuals, 2) estimating advantages at both the graph-level and sample-level, and 3) balancing global and partial rewards. The ToMPO algorithm outperforms the GRPO method by 35% in terms of model output compliance and cooperative outcomes. Additionally, when compared to models with parameter sizes 100 times larger, it shows an 18% improvement. This demonstrates the effectiveness of the ToMPO algorithm in enhancing the model's strategic decision-making capabilities.
Similar Papers
Multi-Agent Tool-Integrated Policy Optimization
Computation and Language
Helps AI agents work together to solve harder problems.
MPO: Boosting LLM Agents with Meta Plan Optimization
Computation and Language
Teaches computers to plan better and learn faster.
Empowering Multi-Turn Tool-Integrated Reasoning with Group Turn Policy Optimization
Machine Learning (CS)
Teaches AI to solve math problems step-by-step.