When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search
Abstract
DiscoBench evaluates search agents' ability to handle ambiguous queries through clarification questioning and recovery in multi-step information-seeking tasks across diverse real-world domains.
Search agents powered by large language models (LLMs) are increasingly used to solve complex information-seeking tasks, requiring multi-step retrieval and reasoning to fulfill user goals. However, existing benchmarks often assume that user queries are complete and explicit, overlooking the fact that real-world search requests are frequently vague, underspecified, or even factually incorrect. In deep search scenarios, such ambiguity can propagate along multi-step reasoning chains and lead agents toward incorrect search trajectories. To address this gap, we introduce DiscoBench, a benchmark for clarification-aware deep search, designed to evaluate whether search agents can proactively identify ambiguity, ask effective clarification questions, and recover correct reasoning paths through user interaction. DiscoBench contains 211 samples and 463 ambiguity instances across 11 real-world domains, covering four ambiguity types. We further design a user simulator for multi-turn interaction and evaluate model performance from four perspectives: task utility, ambiguity detection, interaction strategy, and cost efficiency. Experiments on representative LLMs show that ambiguity detection and effective clarification are distinct capabilities, and that repeatedly searching instead of asking for clarification often performs worse than direct guessing, highlighting a critical gap between retrieval ability and interactive problem-solving in current search agents.
Community
We introduce DiscoBench: a benchmark for evaluating when an LLM search agent should stop searching and ask for clarification.
Many search-agent benchmarks assume that user queries are complete and well-specified. But in real-world deep search, requests are often vague, underspecified, or even factually incorrect. In these cases, the answer may still exist, but it is not reachable through search alone until the agent resolves the ambiguity with the user.
DiscoBench evaluates this capability with 211 samples and 463 ambiguity instances across 11 real-world domains. It tests whether agents can detect ambiguous checkpoints, ask effective clarification questions, and recover the correct reasoning path through interaction.
A key finding is that failure is not simply caused by missing knowledge or weak retrieval. Repeatedly searching can even be worse than directly guessing: SearchHeavyGuess achieves lower pass rates than DirectGuess, while SearchThenAsk reaches the highest pass rate. This suggests that current agents may notice uncertainty in retrieval results, but still fail to convert that uncertainty into the right external action.
This points to a broader question for agent evaluation: beyond measuring whether an agent eventually completes a task, should we also evaluate whether it chooses the right next action when the current path becomes unreliable?
DiscoBench highlights this missing capability: transforming uncertainty into the right interaction decision.
Happy to hear thoughts from the community!
Get this paper in your agent:
hf papers read 2606.27669 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper