Score: 3

Are Vision Language Models Cross-Cultural Theory of Mind Reasoners?

Published: December 19, 2025 | arXiv ID: 2512.17394v1

By: Zabir Al Nazi , G M Shahariar , Abrar Hossain and more

BigTech Affiliations: Stanford University

Potential Business Impact:

Lets computers understand people from different cultures.

Business Areas:
Natural Language Processing Artificial Intelligence, Data and Analytics, Software

Theory of Mind (ToM) -- the ability to attribute beliefs, desires, and emotions to others -- is fundamental for human social intelligence, yet remains a major challenge for artificial agents. Existing Vision-Language Models (VLMs) are increasingly applied in socially grounded tasks, but their capacity for cross-cultural ToM reasoning is largely unexplored. In this work, we introduce CulturalToM-VQA, a new evaluation benchmark containing 5095 questions designed to probe ToM reasoning across diverse cultural contexts through visual question answering. The dataset captures culturally grounded cues such as rituals, attire, gestures, and interpersonal dynamics, enabling systematic evaluation of ToM reasoning beyond Western-centric benchmarks. Our dataset is built through a VLM-assisted human-in-the-loop pipeline, where human experts first curate culturally rich images across traditions, rituals, and social interactions; a VLM then assist in generating structured ToM-focused scene descriptions, which are refined into question-answer pairs spanning a taxonomy of six ToM tasks and four graded complexity levels. The resulting dataset covers diverse theory of mind facets such as mental state attribution, false belief reasoning, non-literal communication, social norm violations, perspective coordination, and multi-agent reasoning.

Country of Origin
πŸ‡ΊπŸ‡Έ πŸ‡§πŸ‡© United States, Bangladesh

Repos / Data Links

Page Count
25 pages

Category
Computer Science:
Computation and Language