Needle in the Web: A Benchmark for Retrieving Targeted Web Pages in the Wild
By: Yumeng Wang , Tianyu Fan , Lingrui Xu and more
Potential Business Impact:
Helps computers find answers to tricky, unclear questions.
Large Language Models (LLMs) have evolved from simple chatbots into sophisticated agents capable of automating complex real-world tasks, where browsing and reasoning over live web content is key to assessing retrieval and cognitive skills. Existing benchmarks like BrowseComp and xBench-DeepSearch emphasize complex reasoning searches requiring multi-hop synthesis but neglect Fuzzy Exploratory Search, namely queries that are vague and multifaceted, where users seek the most relevant webpage rather than a single factual answer. To address this gap, we introduce Needle in the Web, a novel benchmark specifically designed to evaluate modern search agents and LLM-based systems on their ability to retrieve and reason over real-world web content in response to ambiguous, exploratory queries under varying levels of difficulty. Needle in the Web comprises 663 questions spanning seven distinct domains. To ensure high query quality and answer uniqueness, we employ a flexible methodology that reliably generates queries of controllable difficulty based on factual claims of web contents. We benchmark three leading LLMs and three agent-based search systems on Needle in the Web, finding that most models struggle: many achieve below 35% accuracy, and none consistently excel across domains or difficulty levels. These findings reveal that Needle in the Web presents a significant challenge for current search systems and highlights the open problem of effective fuzzy retrieval under semantic ambiguity.
Similar Papers
WideSearch: Benchmarking Agentic Broad Info-Seeking
Computation and Language
Tests if computer helpers can find lots of facts.
WideSearch: Benchmarking Agentic Broad Info-Seeking
Computation and Language
Tests if computers can find lots of information.
Look It Up: Analysing Internal Web Search Capabilities of Modern LLMs
Computation and Language
Helps AI know when to search for answers.