MindPower: Enabling Theory-of-Mind Reasoning in VLM-based Embodied Agents
By: Ruoxuan Zhang , Qiyun Zheng , Zhiyu Zhou and more
Potential Business Impact:
Robots understand what people think and do.
Theory of Mind (ToM) refers to the ability to infer others' mental states, such as beliefs, desires, and intentions. Current vision-language embodied agents lack ToM-based decision-making, and existing benchmarks focus solely on human mental states while ignoring the agent's own perspective, hindering coherent decision and action generation. To address this, we propose MindPower, a Robot-Centric framework integrating Perception, Mental Reasoning, Decision Making and Action. Given multimodal inputs, MindPower first perceives the environment and human states, then performs ToM Reasoning to model both self and others, and finally generates decisions and actions guided by inferred mental states. Furthermore, we introduce Mind-Reward, a novel optimization objective that encourages VLMs to produce consistent ToM Reasoning and behavior. Our model outperforms GPT-4o by 12.77% in decision making and 12.49% in action generation.
Similar Papers
Large Language Models as Theory of Mind Aware Generative Agents with Counterfactual Reflection
Computation and Language
Lets computers understand what people think and feel.
Theory of Mind in Large Language Models: Assessment and Enhancement
Computation and Language
Helps computers understand what people are thinking.
Theory of Mind Using Active Inference: A Framework for Multi-Agent Cooperation
Artificial Intelligence
Lets robots guess friends' goals to team up better