dorsal/arxiv
View SchemaIDRBench: Interactive Deep Research Benchmark
| Authors | Yingchaojie Feng, Qiang Huang, Xiaoya Xie, Zhaorui Yang, Jun Yu, Wei Chen, Anthony K. H. Tung |
|---|---|
| Categories | |
| ArXiv ID | 2601.06676vv1 |
| URL | https://arxiv.org/abs/2601.06676 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Deep research agents powered by Large Language Models (LLMs) can perform multi-step reasoning, web exploration, and long-form report generation. However, most existing systems operate in an autonomous manner, assuming fully specified user intent and evaluating only final outputs. In practice, research goals are often underspecified and evolve during exploration, making sustained interaction essential for robust alignment. Despite its importance, interaction remains largely invisible to existing deep research benchmarks, which neither model dynamic user feedback nor quantify its costs. We introduce IDRBench, the first benchmark for systematically evaluating interactive deep research. IDRBench combines a modular multi-agent research framework with on-demand interaction, a scalable reference-grounded user simulator, and an interaction-aware evaluation suite that jointly measures interaction benefits (quality and alignment) and costs (turns and tokens). Experiments across seven state-of-the-art LLMs show that interaction consistently improves research quality and robustness, often outweighing differences in model capacity, while revealing substantial trade-offs in interaction efficiency.
{
"annotation_id": "2dd1824f-61a6-4a6b-820b-2b1a9c11d5c5",
"date_created": "2026-02-17T05:53:08.617000Z",
"date_modified": "2026-02-17T05:53:08.617000Z",
"file_hash": "c4ea8f040bac0a98f5407a41449f9cb79c7d3708531a2e764506d890398025e8",
"private": false,
"record": {
"abstract": "Deep research agents powered by Large Language Models (LLMs) can perform multi-step reasoning, web exploration, and long-form report generation. However, most existing systems operate in an autonomous manner, assuming fully specified user intent and evaluating only final outputs. In practice, research goals are often underspecified and evolve during exploration, making sustained interaction essential for robust alignment. Despite its importance, interaction remains largely invisible to existing deep research benchmarks, which neither model dynamic user feedback nor quantify its costs. We introduce IDRBench, the first benchmark for systematically evaluating interactive deep research. IDRBench combines a modular multi-agent research framework with on-demand interaction, a scalable reference-grounded user simulator, and an interaction-aware evaluation suite that jointly measures interaction benefits (quality and alignment) and costs (turns and tokens). Experiments across seven state-of-the-art LLMs show that interaction consistently improves research quality and robustness, often outweighing differences in model capacity, while revealing substantial trade-offs in interaction efficiency.",
"arxiv_id": "2601.06676",
"authors": [
"Yingchaojie Feng",
"Qiang Huang",
"Xiaoya Xie",
"Zhaorui Yang",
"Jun Yu",
"Wei Chen",
"Anthony K. H. Tung"
],
"categories": [
"cs.CL",
"cs.AI",
"cs.HC"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "IDRBench: Interactive Deep Research Benchmark",
"url": "https://arxiv.org/abs/2601.06676",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "a5d922e5-bd6e-40aa-827b-87f2a1c64c49",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}