VNU-Bench: A Benchmarking Dataset for Multi-Source Multimodal News Video Understanding
By: Zibo Liu , Muyang Li , Zhe Jiang and more
Potential Business Impact:
Helps computers compare news from different TV channels.
News videos are carefully edited multimodal narratives that combine narration, visuals, and external quotations into coherent storylines. In recent years, there have been significant advances in evaluating multimodal large language models (MLLMs) for news video understanding. However, existing benchmarks largely focus on single-source, intra-video reasoning, where each report is processed in isolation. In contrast, real-world news consumption is inherently multi-sourced: the same event is reported by different outlets with complementary details, distinct narrative choices, and sometimes conflicting claims that unfold over time. Robust news understanding, therefore, requires models to compare perspectives from different sources, align multimodal evidence across sources, and synthesize multi-source information. To fill this gap, we introduce VNU-Bench, the first benchmark for multi-source, cross-video understanding in the news domain. We design a set of new question types that are unique in testing models' ability of understanding multi-source multimodal news from a variety of different angles. We design a novel hybrid human-model QA generation process that addresses the issues of scalability and quality control in building a large dataset for cross-source news understanding. The dataset comprises 429 news groups, 1,405 videos, and 2,501 high-quality questions. Comprehensive evaluation of both closed- and open-source multimodal models shows that VNU-Bench poses substantial challenges for current MLLMs.
Similar Papers
MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs
CV and Pattern Recognition
Tests AI's skill with many videos at once.
MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues
CV and Pattern Recognition
Tests AI's ability to talk about videos.
Perception, Understanding and Reasoning, A Multimodal Benchmark for Video Fake News Detection
CV and Pattern Recognition
Helps computers spot fake videos better.