Elicit: Benchmarks for Scientific Literature Review Performance (public)
Benchmarks for Scientific Literature Review Performance (public)
Research reportView only
Create alertChat
September 12, 2025
Which benchmarks measure performance on scientific literature review tasks?
Benchmarks for scientific literature review tasks include datasets like LitSearch, SciReviewGen, SWIFT‐Review, and RoBBR that measure performance across retrieval, generation, screening, extraction, and meta-evaluation areas.
Abstract
Benchmarks for scientific literature review tasks address five primary areas: retrieval, generation, screening, extraction, and meta-evaluation. Seventeen studies target retrieval using datasets such as LitSearch, CLEF TAR, CSMeD, SIGIR2017-PICO-Collection, FASS‐BSLR, RELISH, and ResearchArena. These studies report performance using metrics such as recall, precision, Mean Average Precision, normalized Discounted Cumulative Gain, and Work Saved over Sampling. Eleven studies focus on generation tasks—including review, table, or abstract writing—with benchmarks like SciReviewGen, ARXIV2TABLE, MSLR2022, ScholarQABench, and Manalyzer; performance is measured by ROUGE scores, hallucination rates, semantic coverage, and human ratings. Nine studies on screening tasks (using tools such as SWIFT‐Review and LGAR) report measures such as precision at high recall levels and WSS@95, while five studies on extraction or structured analysis (using RoBBR, EvidenceBench, and ECACT) employ metrics such as macro‐F1, accuracy, and Cohen’s kappa. One study adopts a meta‐evaluation benchmark using Elo ratings and inter‐annotator agreement. Overall, papers combine automated retrieval and overlap metrics with human evaluation methods to capture the multifaceted nature of performance in scientific literature review tasks.
Methods
We analyzed 40 sources from an initial pool of 1000, using 5 screening criteria. Each paper was reviewed for 4 key aspects that mattered most to the research question. More on methods
Papers identified with Elicit search
n = 1000
Papers screened using: Benchmark Focus, Computational Literature Review Systems, Beyond Methods Description Only, Literature Review Specific Application, Computational Component
n = 1000
Papers screened out
n = 960
Papers included for extraction
n = 40
Press enter or space to select a node.You can then use the arrow keys to move the node around. Press delete to remove it and escape to cancel.
Press enter or space to select an edge. You can then press delete to remove it or escape to cancel.
Paper search
Using your research question “Which benchmarks measure performance on scientific literature review tasks?”, we searched across over 126 million academic papers from the Semantic Scholar corpus. We retrieved the 1000 papers most relevant to the query.
Screening
We screened in sources based on their abstracts that met these criteria:
- Benchmark Focus: Does this study describe, develop, evaluate, present datasets for, or validate benchmarks/evaluation frameworks specifically designed to measure performance on scientific literature review tasks?
- Computational Literature Review Systems: Does this study focus on computational systems that perform literature search, screening, extraction, synthesis, or other core literature review functions, OR does it synthesize knowledge about literature review benchmarks through systematic/narrative reviews?
- Beyond Methods Description Only: Does this study go beyond only describing literature review methods to actually present, evaluate, or analyze benchmarks or evaluation metrics?
- Literature Review Specific Application: Are the benchmarks or evaluation approaches specifically applied to literature review tasks (rather than focusing solely on general information retrieval or text mining without literature review application)?
- Computational Component: Does this study include computational components rather than focusing exclusively on manual literature review processes?
We considered all screening questions together and made a holistic judgement about whether to screen in each paper.
Data extraction
We asked a large language model to extract each data column below from each paper. We gave the model the extraction instructions shown below for each column.
- Benchmark Type:
Identify and describe the specific type of benchmark created in the study. Look for explicit statements about the benchmark’s purpose, focus, and unique characteristics.
Extraction guidelines:
- Locate the benchmark description in the introduction or methods section
- Capture the specific scientific literature review task the benchmark addresses
- If multiple benchmark types are described, list all of them
- Be precise about the benchmark’s scope (e.g., retrieval, generation, search)
Examples might include:
- Hierarchical catalogue generation benchmark
- Literature search retrieval benchmark
- Information distillation benchmark
If no clear benchmark is described, write “Not applicable” or “No benchmark created”
- Benchmark Dataset Characteristics:
Extract detailed information about the dataset used to create or evaluate the benchmark.
Look for:
- Total number of items in the dataset
- Types of items (e.g., research papers, citations, queries)
- Domain or field of research covered
- Data collection method
- Source of data items
Extraction guidelines:
- Prioritize quantitative details (e.g., “7.6k literature review catalogues and 389k reference papers”)
- Note the specific research domains if mentioned
- If multiple datasets are used, list all with their characteristics
- If dataset details are incomplete, note “Insufficient information”
Format examples:
“7,600 literature review catalogues from scientific papers”
“597 literature search queries across ML and NLP domains”
Evaluation Metrics:
Identify and describe the specific metrics used to assess benchmark performance.
Extraction guidelines:
- Locate metrics in results, methods, or discussion sections
- Capture both quantitative metrics and qualitative assessment approaches
- Note the specific performance dimensions being measured
- If multiple metrics are used, list all of them
Look for metrics such as:
- Recall@5
- Semantic similarity scores
- Informativeness ratings
- Comparative performance against baseline models
Format examples:
“24.8% difference in absolute recall@5”
“Two evaluation metrics for informativeness and structural similarity”
Benchmark Models and Comparisons:
Document the models or systems tested against the benchmark.
Extraction guidelines:
- List all models/systems evaluated
- Note their performance relative to the benchmark
- Capture any comparative analysis between different approaches
- Include both state-of-the-art and baseline models
Look for:
- Specific model names (e.g., BART, ChatGPT)
- Performance rankings
- Comparative performance percentages
Format examples:
- “BART and ChatGPT tested, with performance variations noted”
- “Commercial search engines performed 32 points below best dense retriever”
Results
Characteristics of Included Studies
Study
Study Focus
Benchmark/Dataset Name
Literature Review Task Type
Evaluation Approach
Full text retrieved
Zhu et al., 2023
Hierarchical catalogue generation for literature reviews
HiCaD
Hierarchical catalogue generation
CEDS (semantic/structural similarity), CQE (informativeness)
Yes
Ajith et al., 2024
Literature search retrieval
LitSearch
Literature search/retrieval
Recall at 5, recall at 20, normalized Discounted Cumulative Gain at 10 (nDCG@10)
Yes
Kanoulas et al., 2019
Systematic review retrieval and ranking
CLEF 2019 e-Health Technology-Assisted Review (TAR)
Retrieval, ranking for systematic reviews
No mention found in abstract
No
Kasanishi et al., 2023
Automatic literature review generation
SciReviewGen
Literature review generation (summarization)
Recall-Oriented Understudy for Gisting Evaluation (ROUGE), human evaluation (relevance, coherence, informativeness, factuality)
Yes
Howard et al., 2016
Automated citation screening
SWIFT-Review datasets
Citation screening
Work Saved over Sampling at 95% (WSS@95%), precision at 95% recall
Yes
Wang et al., 2025a
Literature review table generation
ARXIV2TABLE
Table generation for literature reviews
Recall, precision, F1 for schema/cell/pairwise overlap
Yes
Kusa et al., 2023a
Outcome-based evaluation of systematic review automation
CLEF TAR 2019
Systematic review automation (retrieval)
Mean Average Precision (MAP), Recall at k%, WSS, Area Under the Curve (AUC), outcome difference
Yes
Kusa et al., 2023b
Automated citation screening
CSMeD, CSMeD-FT
Citation/full-text screening
True Negative Rate at 95% (TNR@95%), normalized Precision at 95% (nP@95%), nDCG@10, MAP, macro-precision/recall/F1
Yes
Scells et al., 2017
Retrieval for systematic reviews
SIGIR2017-PICO-Collection
Retrieval, screening prioritization
Precision-recall, F-beta, WSS, Average Precision (AP), nDCG, MAP
Yes
Budau and Ensan, 2024
Automated study search for biomedical systematic literature reviews
FASS-BSLR
Study search (retrieval, Boolean query generation)
Precision, Recall, NDCG, MAP, Recall at 1000
No
toof
Pageof
View 30 more rows
Note:All abbreviations and acronyms are defined at first use in the table or in this note. “No mention found” indicates that we did not find mention of the information in the available abstract or full text.
Distribution of Literature Review Automation Tasks
- Retrieval, search, or ranking tasks:Addressed in 17 studies. These focus on identifying and prioritizing relevant literature for systematic reviews or evidence synthesis.
- Generation tasks:Addressed in 11 studies. These include review, table, summary, reference, or abstract generation.
- Screening tasks:Addressed in 9 studies. These include citation, abstract, or full-text screening.
- Extraction tasks:Addressed in 5 studies. These include data, evidence, risk of bias, or meta-analysis.
- Meta-evaluation or meta-analysis:Addressed in 1 study.
- Other tasks:Addressed in 10 studies. These include query generation, compliance, information distillation, academic survey, trial design, systematic review-style reasoning, or multiple systematic literature review tasks.
Distribution of Evaluation Approaches
- Information retrieval metrics:Used in 27 studies. Metrics include precision, recall, F1, Mean Average Precision (MAP), normalized Discounted Cumulative Gain (nDCG), Mean Reciprocal Rank (MRR), Average Precision (AP), and others.
- Human evaluation:Used in 6 studies. Includes inter-annotator agreement, Cohen’s kappa, or explicit human ratings.
- Overlap, coverage, or consistency metrics:Used in 6 studies. Includes schema/cell overlap, coverage, factual consistency, or hallucination rate.
- Specialized or composite metrics:Used in 7 studies. Includes Elo ratings, CEDS, CQE, ECACT, or outcome difference.
- No mention found of evaluation approach:In 6 studies.
Some studies addressed multiple task types or used multiple evaluation approaches. We did not find mention of evaluation approach for 6 studies, and task type information was sometimes ambiguous due to overlapping or unclear descriptions.
Thematic Analysis
Benchmarks for Literature Retrieval and Screening Tasks
Benchmarks in this category focus on identifying, retrieving, and prioritizing relevant literature for systematic reviews or evidence synthesis. Key benchmarks include:
- LitSearch, CLEF Technology-Assisted Review (TAR) (2017–2019), CSMeD, SIGIR2017-PICO-Collection, FASS-BSLR, RELISH, and ResearchArena:These benchmarks use large-scale datasets such as PubMed, MEDLINE, and S2ORC, and evaluate models using recall, precision, Mean Average Precision, normalized Discounted Cumulative Gain, Work Saved over Sampling, and related metrics.
- Model types:Both classical information retrieval models (such as BM25 and term frequency-inverse document frequency) and modern transformer-based or large language model approaches are tested.
- Findings:Some studies (e.g., Kang and Xiong, 2024) report that keyword-based retrieval still outperforms large language model-based methods in certain settings, while others (e.g., Skarlinski et al., 2024) demonstrate large language models matching or exceeding human performance in specific retrieval and synthesis tasks.
Benchmarks for Literature Summarization and Synthesis Tasks
This theme covers benchmarks designed to evaluate the ability of models to generate summaries, reviews, or structured tables from collections of scientific papers. Notable benchmarks include:
- SciReviewGen, ARXIV2TABLE, MSLR2022, ScholarQABench, and Manalyzer:Evaluation metrics include ROUGE, human preference ratings, semantic coverage, factual consistency, and hallucination rates.
- Findings:Some large language models and transformer-based models achieve promising results, but challenges remain in factuality, citation accuracy, and domain adaptation. Human evaluation is often used to supplement automated metrics, reflecting the complexity of assessing synthesis quality.
Benchmarks for Structured Literature Analysis Tasks
Structured analysis tasks include risk of bias assessment, data element extraction, compliance and traceability, and meta-evaluation. Key benchmarks:
- RoBBR, EvidenceBench, ECACT, SciArena-Eval, and internal datasets:These benchmarks use composite or task-specific metrics such as ECACT score, macro-F1, accuracy, and Cohen’s kappa.
- Findings:Automation is advancing, but human oversight remains essential, especially in high-stakes or regulatory contexts.
Evaluation Methodologies Across Benchmarks
Benchmark Category
Primary Metrics
Evaluation Method
Performance Measurement Focus
Retrieval/Screening
Recall, precision, Mean Average Precision, normalized Discounted Cumulative Gain, Work Saved over Sampling, Average Precision
Automated scoring, comparison to human or baseline models
Relevance, workload reduction, ranking effectiveness
Summarization/Synthesis
ROUGE, human ratings, semantic coverage, hallucination rate
Automated metrics, human evaluation
Informativeness, factuality, coherence, citation accuracy
Structured Analysis
Macro-F1, accuracy, Cohen’s kappa, composite scores (ECACT)
Automated scoring, statistical analysis, human validation
Bias, compliance, extraction accuracy, traceability
Meta-evaluation
Elo ratings, accuracy, agreement
Human preference voting, model-based evaluator assessment
Alignment with human judgment, evaluator reliability
toof
Pageof
Summary of Evaluation Approaches:
- Primary metrics:Accuracy was the only metric used in more than one category (2 out of 4 categories). All other metrics were unique to a single category, including recall, precision, Mean Average Precision, normalized Discounted Cumulative Gain, Work Saved over Sampling, Average Precision, ROUGE, human ratings, semantic coverage, hallucination rate, macro-F1, Cohen’s kappa, composite scores (ECACT), Elo ratings, and agreement.
- Evaluation methods:Automated evaluation methods were used in 3 out of 4 categories. Human-based evaluation methods were used in all 4 categories. Comparison-based methods, statistical analysis, and model-based evaluator assessment were each used in 1 out of 4 categories.
- Coverage:We did not find any categories without at least one automated and one human-based evaluation method. No information was missing for any category.
References
Download BIBDownload RISDownload TXT
Report
Status
Gather sources
1000 sources found
Details
Screen sources
40 sources included
Details
Extract data
160 data points extracted
Details
Generate report
Save PDF
BIBLaTeX, ZoteroRISZotero, MendeleyTXTAPA BibliographyPDFPDFDOCXMicrosoft Word
Chat
Got some follow-up questions?
Sign up or sign in to chat with this report.
Back
DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents
Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, Zhendong Mao
arXiv.org·
2025·
3 citations
SourceDOI
Plain textPDFFigures
Searching for PDF
Unable to find PDF from source
Back
Benchmark Type
DeepResearch Bench: A comprehensive benchmark for evaluating the report generation and web retrieval capabilities of Deep Research Agents
Benchmark Dataset Characteristics
- Total number of items in the dataset: 100 PhD-level research tasks - Types of items: Research tasks - Domain or field of research covered: 22 distinct fields - Data collection method: Collected from interactions with web search-enabled LLM Chatbot - Source of data items: In-house dataset of 96,147 raw user queries
Evaluation Metrics
- Pairwise Agreement Rate - Overall Pearson Correlation - Filtered Average Pearson Correlation - Filtered Average Spearman Correlation - Citation Accuracy (C. Acc.) - Average Effective Citations per Task (E. Cit.) - Overall Score - Comprehensiveness - Depth - Instruction Following - Readability
Benchmark Models and Comparisons
- DeepResearch Bench is the benchmark used for evaluations. - Models tested include: - LLM with Search Tools: Claude-3.7-Sonnet w/Search, Claude-3.5-Sonnet w/Search, Perplexity-Sonar-Reasoning-Pro(high), Perplexity-Sonar-Reasoning(high), Perplexity-Sonar-Pro(high), Perplexity-Sonar(high), Gemini-2.5-Pro-Grounding, Gemini-2.5-Flash-Grounding, GPT-4o-Search-Preview (high), GPT-4o-Mini-Search-Preview(high), GPT-4.1 w/Search(high), GPT-4.1-mini w/Search(high) - Deep Research Agents: Grok Deeper Search, Perplexity Deep Research, Gemini-2.5-Pro Deep Research, OpenAI Deep Research - Performance rankings and comparative analysis are detailed in the tables on pages 6, 8, and 9.
Deep Research Agents are a prominent category of LLM-based agents. By autonomously orchestrating multistep web exploration, targeted retrieval, and higher-order synthesis, they transform vast amounts of online information into analyst-grade, citation-rich reports--compressing hours of manual desk research into minutes. However, a comprehensive benchmark for systematically evaluating the capabilities of these agents remains absent. To bridge this gap, we present DeepResearch Bench, a benchmark consisting of 100 PhD-level research tasks, each meticulously crafted by domain experts across 22 distinct fields. Evaluating DRAs is inherently complex and labor-intensive. We therefore propose two novel methodologies that achieve strong alignment with human judgment. The first is a reference-based method with adaptive criteria to assess the quality of generated research reports. The other framework is introduced to evaluate DRA's information retrieval and collection capabilities by assessing its effective citation count and overall citation accuracy. We have open-sourced DeepResearch Bench and key components of these frameworks at https://github.com/Ayanami0730/deep\_research\_bench to accelerate the development of practical LLM-based agents.
1Introduction
We now enter a new phase of AI [35], a period marked by comprehensive advances in the capabilities of the large language model (LLM) [5,20]. These advancements enable the construction of LLM-based Agent systems designed to tackle increasingly complex tasks [15,8,34]. In this evolving landscape, defining tasks that genuinely reflect real-world demands and designing robust evaluation methodologies to measure the progress of these Agent systems are becoming critically important. Deep research represents one such well-defined task domain, with Deep Research Agents (DRAs) [13,37,24] emerging as the most widely utilized LLM-based agents today. Users leverage these agents to enhance their productivity significantly.
However, comprehensively evaluating the capabilities of these DRAs presents substantial challenges. Because their internal reasoning and information retrieval processes are not transparent, the final generated report becomes the primary interface through which their overall performance can be assessed. Moreover, evaluating the quality of these extensive research reports remains an open-ended problem, as establishing a definitive 'ground truth' or standard answer for complex research queries is almost impossible.
These demanding evaluation requirements for DRAs pose a significant challenge for existing evaluation frameworks, which often fall short of offering a dedicated assessment of the multifaceted capabilities of such agents [14]. Current benchmarks typically focus on assessing isolated capabilities-such as web browsing and information retrieval [28,38,39], or generative abilities disconnected from real-time information acquisition [23,1,31].
To fill this gap in evaluating end-to-end research agents, we introduce DeepResearch Bench, a 100-task benchmark spanning 22 domains, with each task meticulously crafted and iteratively refined by domain experts. To ensure this benchmark authentically reflects real research needs, we carefully determine the specific number of tasks from each topic domain. This is guided by a statistical analysis of over 96,000 real-world user queries, adhering to the pipeline illustrated in Figure 2(a) .
Building on this dataset, we introduce two novel, highly human-aligned evaluation methodologies to comprehensively assess both the quality of final reports generated by DRAs and their ability to gather web information effectively. The first one is a Reference-based and Adaptive Criteria-driven Evaluation framework with Dynamic Weighting (denoted as RACE for ease of subsequent reference), which targets the assessment of report generation quality. And the other one is a framework for Factual Abundance and Citation Trustworthiness (denoted as FACT), which focuses on evaluating information retrieval and citation accuracy. Overview results are shown in Figure 1 . Furthermore, we believe these methodologies are not confined to deep research scenarios. RACE can serve as a general framework for evaluating text generation capabilities, while FACT offers a robust approach for assessing the coverage and reliability of information retrieved by LLMs.
Our primary contributions are as follows:
• We present DeepResearch Bench, the first specialized benchmark for evaluating Deep Research Agents, constructed through a systematic process that reflects authentic user needs.
• We further propose RACE and FACT, two novel evaluation frameworks that respectively assess the report generation quality and the information retrieval abilities of Deep Research Agents.
• We conduct comprehensive human studies to validate the reliability of our frameworks and publicly release the benchmark and evaluation protocols to foster future research.
2 DeepResearch Bench: Topic Distribution Analysis & Data Collection
2.1Topic Distribution Analysis
Deep Research Agents (DRAs) are intended to serve actual human research needs. Therefore, to effectively test their capabilities, the design of DeepResearch Bench should be grounded in the real-world distribution of human research task demands. To obtain this distribution, we collect an in-house dataset of 96,147 raw user queries from interactions with web search-enabled LLM Chatbot. To ensure user privacy, all raw query logs underwent rigorous anonymization, removing personally identifiable information such as user IDs, IP addresses, and session metadata.
We then define the concept of Deep Research tasks as problems requiring agents to conduct multiple rounds of web searches, gather information, perform analysis, and produce high-quality reports. Guided by this definition, we employ DeepSeek-V3-0324 [4] to conduct filtering, identifying queries that aligned with our deep research requirements. This process ultimately yields a dataset of 44,019 queries conforming to our deep research task definition.
To categorize the deep research queries, we adopt the topic taxonomy proposed by WebOrganizer [29], selecting 22 distinct topic domains for this classification. We then employ DeepSeek-V3-0324 to classify these 44,019 queries into the 22 selected topic domains. By statistically aggregating the LLM's classification results, we obtain the distribution of these queries across the various topics. This distribution, visualized in Figure 3 , indicates the real-world user demand for deep research within these domains.
2.2Benchmark Task Collection
Guided by the real-world user demand distribution, we first determine the target number of tasks for each topic domain within the DeepResearch Bench dataset. Subsequently, considering the significant resources required to run deep research agents and evaluate their results, we proportionally compress this determined distribution to form a final dataset of 100 tasks, 50 in Chinese and 50 in English. This compression process meticulously ensures that the benchmark maintains the same topical balance as the observed real-world distribution.
Once the target task count for the topic domain is determined, our focus shifts to constructing research tasks that are highly challenging and firmly grounded in authentic research demands. This process specifically aims to test the upper limits of the Deep Research Agents' capabilities. We invite Ph.D. or senior practitioners with more than five years of relevant domain experience to propose candidate tasks, as shown in Figure 4 . All submissions undergo a manual screening by our team to verify their quality, clarity, complexity, and alignment with our definition of deep research. This rigorous vetting process yields the 100 high-quality benchmark tasks that comprise the DeepResearch Bench.
3Evaluation Methodology
Our evaluation methodology focuses on two critical aspects: their capabilities in information retrieval and collection, as well as the quality of the final reports they generate. To assess these respective dimensions, we developed two complementary frameworks, RACE and FACT.
3.1RACE: A Framework for Report Quality Evaluation
Evaluating long-form research reports presents significant challenges. Existing approaches using fixed checklists [23] or static rubrics [25,1] struggle to adapt to diverse tasks, specialized domains, and nuanced quality aspects of deep research tasks. To address this, we introduce our Reference-based Adaptive Criteria-driven Evaluation framework with Dynamic Weighting (RACE), leveraging the LLM-as-a-Judge method [36]. RACE offers a more adaptive and robust evaluation by first dynamically generating task-specific weights and criteria. It then employs a reference-based scoring approach, comparing the target report with a high-quality reference. Finally, a relative score is computed to assess the quality of the target report.
These final weights, W d , ensure that evaluation aligns with the task's intent. Subsequently, for each dimension d, the Judge LLM generates a set of K d tailored criteria {c d,k } and their corresponding weights {w d,k } (where
, which are clear and actionable for evaluating the report within that dimension.
Reference-Based Scoring. Preliminary experiments suggest that scoring reports in isolation often yields insufficiently discriminative results; models tend to assign uniformly high scores, thereby masking genuine quality variations. To mitigate this, RACE adopts a reference-based scoring strategy. For each task t, a high-quality report R ref is selected as a reference. All generated criteria {c d,k } across all dimensions are aggregated into a comprehensive list C t . The Judge LLM then analyzes the target report R tgt and the reference report R ref against each criterion c ∈ C t . This yields lists of scores for both reports for each criterion, which are then used for final score calculation:
Overall Score Calculation. Finally, we compute the overall quality score of the target report.
3.2FACT: A Framework for Web Retrieval Evaluation
To assess the factual grounding of report content and the agent's effectiveness in retrieving and utilizing web-based information, we introduce the framework for Factual Abundance and Citation Trustworthiness(FACT). This framework evaluates DRAs through the following automated steps:
Statement-URL Pair Extraction and Deduplication. We employ a Judge LLM to extract discrete statements from reports generated by DRAs with their corresponding cited source URLs. Then, the Judge LLM examines the pairs to identify cases where multiple statements associated with the same URL describe exactly the same fact. In such cases, only one representative Statement-URL pair is retained, ensuring each unique factual claim is represented only once.
Support Judgment. Each unique Statement-URL pair undergoes a support evaluation. We retrieve the textual content of the webpage using the Jina Reader API. Then, the Judge LLM assesses whether this content provides sufficient evidence to support the statement. This results in a binary judgment ('support' or 'not support') for each pair, determining whether the citation accurately grounds the claim. Among the LLMs with Search Tools, Claude-3.7-Sonnet delivers an impressive performance, with its overall score exceeding Grok Deeper Search, which is perhaps attributable to Claude being allowed to perform multi-turn web searches. Perplexity-Sonar-Reasoning-Pro(high) also demonstrated similar performance. When viewing scores of different models across topics in Figure 5 , we can observe that individual models maintain relatively stable performance across different topic tasks, demonstrating the robustness of our RACE evaluation framework. Interestingly, we discover that for the Chinese task within the transportation topic, all models perform below their respective average levels, indicating that this particular question set presents a higher difficulty level. The generated articles and corresponding scores for all models can be viewed on the leaderboard hosted on Hugging Face Spaces.
4.2.2Evaluation on FACT Framework
Viewing evaluation results by FACT in
4.3Human Consistency
Evaluating the quality of deep research reports remains an open-ended task. Therefore, to validate the effectiveness of our proposed RACE framework, we must assess its human consistency. We conduct experiments using 50 Chinese tasks from DeepResearch Bench, with reports generated by four distinct agents. For each task, three domain-expert annotators score these reports. Further details are provided in Appendix G.1.
4.3.1Human Data Collection
To gather human judgments, we recruit 70+ annotators with Master's degrees and relevant domain expertise. Using a custom interface, they evaluate reports across four dimensions and overall performance, guided only by basic scoring criteria to minimize bias. Each annotator is limited to a maximum of three queries to ensure diverse perspectives.
4.3.2Evaluation Metrics
To validate the consistency between evaluation methods and human judgment, we design four metrics that quantify different aspects of alignment with human evaluations. The detailed calculation processes for all metrics are provided in Appendix F.
Pairwise Agreement Rate This metric measures how often our evaluation method's preferences match human experts' preferences when comparing pairs of reports. It reflects the reliability of our framework in replicating human comparative judgments across all tasks.
Overall Pearson Correlation This metric quantifies the linear relationship between average model scores from our evaluation method and those from human experts. It demonstrates how well our framework's absolute scoring aligns with human evaluation across all deep research assistant models.
Filtered Average Pearson & Spearman Correlation
When computing per-task average correlation coefficients, individual inconsistencies can have a more pronounced impact on the results compared to global correlations. To address this issue, we first filter out tasks where expert judgments show low agreement by removing tasks with negative Intraclass Correlation Coefficients (ICC < 0). The ICC is a statistical measure of rater consistency, and negative values indicate poor inter-rater reliability. After applying this filtering criterion, 37 tasks (out of the original set) remained in our experiment, forming a subset with demonstrably higher expert consensus. We then compute two complementary metrics: the Filtered Average Pearson Correlation (FAP) and the Filtered Average Spearman Correlation (FAS). Together, these filtered metrics provide a more robust assessment of how well automated evaluation aligns with consistent human judgment across different tasks.
4.3.4Comparison of Different Judge LLM
Leveraging the RACE framework, we further compare the performance and cost of several leading proprietary LLMs when used as the Judge LLM. As detailed in Table 3 , Gemini 2.5 Pro Preview not only achieves the best overall performance but also maintains a competitive average cost ($0.13 per query), only higher than that of o4-mini. Therefore, to effectively balance performance and cost, we select Gemini 2.5 Pro Preview as the Judge LLM in our final framework. [18,27], promising to significantly alter many aspects of daily life and professional work. Yao's blog [35] emphasizes that defining more realistic problems and designing novel evaluation methods are crucial for developing more practical AI Agent systems. Numerous evaluations have already been designed specifically for Agents, targeting diverse capabilities. These include evaluations for agents in scientific domains [2,12,17,3], creative writing [31,1,23], code generation and software engineering [10,40,22,9,32], and in their roles as human assistants, often enhanced by capabilities such as web Browse and tool-use [28,38,33,6,26]. This perspective underscores our belief that constructing benchmarks specifically designed for Deep Research Agent, grounded in real-world scenarios, alongside developing human-aligned evaluation methods, is urgently needed to guide the development of AI agent systems.
Deep Research Agent Following the release of Deep Research Agents (DRAs) by OpenAI [19] and Google's Gemini [7], these agents gained significant attention and became one of the most widely deployed categories of LLM-based agents. Subsequently, related works [11, 13, 37] quickly followed, introducing their own designed DRA frameworks. However, the field still lacks a standardized evaluation methodology for these DRAs, preventing meaningful comparative analysis of their capabilities. Among these works, some utilize QA datasets [21, 16,30] as evaluation metrics; however, this approach neither aligns with real-world DRA applications nor comprehensively assesses their broader capabilities.
Others employ the LLM-as-a-judge methodology [36], yet these efforts lack both a comprehensive framework design and verification of human consistency. In contrast, our DeepResearch Bench addresses this gap by providing a systematic and unified evaluation method with strong human consistency, thereby supporting subsequent DRA development and assessment.
6Conclusion
In this work, we introduce DeepResearch Bench, the first comprehensive benchmark for evaluating the report generation and web retrieval capabilities of Deep Research Agents. Comprising 100 high-quality research tasks across 22 distinct domains, this benchmark is meticulously curated to reflect authentic user needs. Our key evaluation frameworks, RACE and FACT, have demonstrated high consistency with human judgments, affirming their reliability. We hope DeepResearch Bench will guide developers and researchers in constructing more powerful and human-centric AI agent systems that truly address genuine user requirements.
A Limitations
While DeepResearch Bench and the RACE & FACT evaluation framework offer the first comprehensive assessment of Deep Research Agents, several limitations remain.
Benchmark scale. Creating tasks that authentically reflect real research challenges requires substantial expertise and effort. Each of our 100 tasks was developed by a verified PhD-level expert who underwent orientation on our task specifications and participated in multiple rounds of refinement. Following rigorous internal quality screening, only superior tasks were included. This meticulous process ensures high quality but inevitably constrains the dataset size. Expanding the benchmark while maintaining our quality standards would enhance statistical robustness and topic coverage, representing a key priority for future development.
Domain coverage bias. Despite our team's breadth of expertise, inadvertent biases may have been introduced during the curation and selection process. We implemented multiple review layers and explicit acceptance criteria to mitigate this risk, yet we cannot guarantee complete impartiality. Future iterations will incorporate additional external reviewers with specialized domain knowledge to achieve a more balanced representation across fields.
Human evaluation throughput. The assessment of research reports presents inherent challenges: a single report may contain dozens of pages with dense technical content and sophisticated analysis. Even for PhD-level evaluators, a thorough review typically requires 30-60 minutes, with the evaluation of multiple reports multiplying this burden.
For the current study, we limited human evaluation to three experts per task, covering 50 tasks across four agents (totaling 600 reports). While sufficient to validate the RACE framework, this sample remains modest and susceptible to individual variability. We intend to conduct more extensive annotation campaigns to strengthen statistical confidence and refine our automated metrics, thereby maintaining alignment with human judgment.
Broader Impact Our work on DeepResearch Bench aims to significantly accelerate research and innovation by providing a robust framework for developing more capable Deep Research Agents (DRAs). This can democratize access to advanced research methodologies, empowering a wider range of users across various domains. However, the advancement of powerful DRAs also presents societal challenges that warrant careful consideration. These include the potential for generating sophisticated misinformation if not rigorously validated, the risk of over-reliance, which could lead to a decline in critical thinking and manual research skills, and the possibility of amplifying biases inherent in the underlying models and the vast web data they process. DeepResearch Bench, with its multifaceted evaluation that includes factual grounding (FACT) and quality assessment (RACE), is designed as a step towards mitigating such risks by fostering the development of more reliable, transparent, and responsible AI agents. Continuous vigilance, ongoing research into AI safety and ethics, and multi-stakeholder engagement will be crucial to ensure these powerful tools are developed and deployed in a manner that benefits society while minimizing potential harms.
B Definitions of Evaluation Dimensions in RACE
The RACE framework evaluates research reports based on four top-level dimensions. Their definitions are provided in Table 4 . Article has a clear structure, fluent language, and is easy to understand.
C Judge LLM Selection for the FACT Framework
The FACT framework employs a Judge LLM for crucial automated steps: Statement-URL Pair Extraction and Deduplication, followed by Support Judgment. The selection of Judge LLM is pivotal, aiming to balance evaluation accuracy with operational costs, especially given the significant token consumption inherent in these processes. To determine an optimal model for these tasks, we specifically evaluated Gemini-2.5-Flash. Its judgments were compared against human evaluations on a randomly sampled set of 100 statement-URL pairs derived from our benchmark tasks. This comparison demonstrated strong agreement with human annotators: Gemini-2.5-Flash's judgment aligned with human 'support' determinations in 96% of cases and with 'not support' determinations in 92% of cases.
We further find out that the accuracy of Gemini-2.5-Flash in these FACT-specific evaluation steps is very close to that of Gemini-2.5-Pro. The operations within the FACT framework (such as extracting statements from full reports and analyzing webpage content for support) are known to be token-intensive, making cost-effectiveness a critical consideration. Since Gemini-2.5-Flash demonstrated comparable accuracy to Gemini-2.5-Pro for these specific tasks but at a more advantageous cost, we select Gemini-2.5-Flash as the Judge LLM for the FACT framework. This choice enables us to maintain high evaluation reliability while managing operational costs effectively.
D Data Collection Timeframes for Evaluated Models
The data for the commercial models evaluated in this paper were collected during specific timeframes in 2025, as detailed in Table 5 . These dates indicate when the model outputs used in our experiments were generated.
E Detailed Calculation of Citation Metrics
This appendix provides the detailed definitions and calculation methods for the Citation Accuracy (C. Acc.) and Average Effective Citations per Task (E. Cit.) metrics used within the FACT framework.
Let T denote the set of all tasks in the benchmark, and |T | be the total number of tasks. For each task t ∈ T :
• Let U t be the set of unique statement-URL pairs extracted for task t after the deduplication process. • Let N u,t = |U t | be the total number of unique statement-URL pairs for task t that undergo support judgment.
• For each statement-URL pair in U t , a support judgment is rendered, which can be either 'support' or 'not support'. • Let N s,t be the number of statement-URL pairs that are judged as 'support' for task t.
E.1 Citation Accuracy (C. Acc.)
Citation Accuracy (C. Acc.) assesses the Deep Research Agent's (DRA) ability to accurately apply retrieved information for precise statements. It is calculated by first determining the proportion of 'support' statement-URL pairs for each individual task, and then averaging these per-task accuracies across all tasks in the benchmark.
The accuracy for a single task t, denoted as Acc t , is defined as:
This definition ensures that tasks for which the agent produces no citable statements (i.e., N u,t = 0) contribute an accuracy of 0 to the overall average, reflecting a failure to provide supported information for that task.
The overall Citation Accuracy (C. Acc.) is then computed as the average of these per-task accuracies over all tasks in the benchmark:
E.2 Average Effective Citations per Task (E. Cit.)
Average Effective Citations per Task (E. Cit.) evaluates, on average, how much useful and relevant information the agent retrieves and correctly supports with evidence for each task. It is computed by summing the total number of 'support' statement-URL pairs across all tasks and then dividing by the total number of tasks in the benchmark.
The Average Effective Citations per Task (E. Cit.) is calculated as:
This metric provides a direct measure of the average quantity of verifiably supported statements an agent generates per task.
F Detailed Calculation of Human Consistency Metrics
This appendix provides the detailed calculation methods for the four metrics used to validate the consistency between our RACE framework and human judgment, as introduced in Section 4.3.2.
F.1 Pairwise Agreement Rate
The Pairwise Agreement Rate measures the proportion of report pairs (across all tasks) where the evaluation method's preference matches the human preference.
For each of the N t = 50 tasks in our study, four generated deep research reports result in N p = 4 2 = 6 unique pairs per task. Human preference for each pair (e.g., Report A is better than Report B, or they are tied) is established from the average overall scores assigned to each report by three domain experts.
Let I(t, p) be an indicator function for task t and pair p:
1 if the method's preference matches the human preference for pair p of task t 0 otherwise.
The Pairwise Agreement Rate is then calculated as:
This metric reflects the evaluation method's reliability in replicating human comparative judgments.
F.2 Overall Pearson Correlation
This metric quantifies the linear relationship between average model scores from the evaluation method and those from human experts, aggregated across all N t = 50 tasks.
Let X be a vector of average scores per model (e.g., for the different DRAs evaluated) obtained from our method, aggregated across all tasks. Let Y be the corresponding vector of average scores per model, obtained from human experts and aggregated across all tasks. The Overall Pearson Correlation is the standard Pearson correlation coefficient r(X, Y ) calculated between these two vectors. This reflects the overall score correlation between the method and human experts for the evaluated DRAs.
F.3 Filtered Average Pearson Correlation
This metric calculates the average of per-task Pearson correlations (r t ) between the method's scores and mean human scores, specifically on tasks where human judgment is more consistent.
Given that human scores are from a limited number of experts (k = 3) for the n = 4 reports per task, expert inconsistencies can affect task-level metric stability. To mitigate this, tasks are filtered based on inter-rater reliability using the Intraclass Correlation Coefficient (ICC). For each task, ICC(1,1) (a one-way random effects model) is computed from the k = 3 human experts' scores for the n = 4 reports:
where MSB is the mean square between reports and MSW is the mean square within reports. Tasks with poor inter-rater reliability (e.g., ICC(1, 1) < 0) are excluded. This yields a filtered subset of N filtered tasks, denoted as T filtered (37 in our experiments).
The Filtered Average Pearson correlation is then the average of per-task Pearson correlations (r t ) between the method's scores and mean human scores over T filtered :
This procedure provides a more robust assessment of absolute-score correlation.
F.4 Filtered Average Spearman Correlation
Using the same filtering method and the subset T filtered , this metric evaluates model ranking consistency.
For each task t ∈ T filtered , the Spearman rank correlation coefficient ρ t is calculated between model rankings derived from our evaluation method and those from average human scores. The Filtered Average Spearman Correlation is then the average of these ρ t values:
This reflects how well the method preserves relative model ordering compared to humans, specifically on tasks with more consistent human judgments.
G.1 Human Evaluation Effort
The human evaluation process required a considerable investment of time to ensure a thorough assessment. On average, each expert annotator spent approximately 1.5 hours per query to evaluate the reports from the four agents. This meticulous process culminated in a total of 225 person-hours of human evaluation across all tasks and annotators. This substantial effort yielded a robust and reliable dataset of human judgments, serving as the basis for our analysis of human consistency.
G.2 Configuration of LLMs with Web Search Tools
To ensure a standardized and comparable evaluation environment for Large Language Models (LLMs) equipped with built-in web search tools, the following configurations were uniformly applied:
• Thinking Budget/Computational Resources: For models that support a configurable "thinking budget" or a similar computational resource limit for generation, this was uniformly set to a high value, equivalent to 16,000 tokens where applicable. This allowed models ample processing capacity for complex queries. • Search Context Size: In cases where models offered a parameter to control the amount of information retrieved and utilized from web searches (e.g., the search_context_size option as found in the Perplexity AI API), this was consistently set to "high". This configuration aimed to maximize the contextual information available to the LLM from its search activities.
• Maximum Search Iterations: The maximum number of web search queries, or "search turns," permitted during the generation process was standardized to five for all LLMs that provided such a configurable limit. This ensured a comparable depth of web exploration across these models.
• Output Length: To accommodate potentially comprehensive responses while maintaining consistency, the maximum output token limit for all LLMs was set to 36,000 tokens. If a model's inherent maximum output capacity was less than 36,000 tokens, its specific native maximum limit was adhered to.
• Citation Formatting and Standardization: A critical aspect of our methodology was the standardization of citation presentation to facilitate consistent downstream evaluation, particularly when employing frameworks like FACT for factual assessment. Citations as provided by each LLM were parsed in accordance with their respective official API documentation. Subsequently, the generated reports were systematically restructured: citation markers were inserted in the format ' [1][2]' at the end of the relevant sentences, and a consolidated 'References' list, compiling all unique cited sources, was appended to the conclusion of each article. This uniform approach to citation structure was essential for equitable and rigorous factual verification.
These standardized settings were implemented to minimize variability arising from differing default configurations and to enable a more direct comparison of the models' capabilities in the context of deep research tasks. The research team will conduct in-depth and comprehensive research based on the '
H Prompt Templates
Your Task: As an evaluation expert, you need to set the evaluation criteria weights for this specific '
3. Instruction Following: Whether the report accurately and completely responds to all requirements and constraints of the task. 4. Readability: Clarity of structure, fluency of language, effectiveness of data presentation, and overall ease of understanding. Evaluation Formula: Total Score = Comprehensiveness * Comprehensiveness Weight + Insight * Insight Weight + Instruction Following * Instruction Following Weight + Readability * Readability Weight. (Note:
The sum of all weights must be exactly 1.0) Core Requirements:
1. In-depth Task Analysis: Carefully study the specific content of the '
3.... <examples_rationale>
The following two examples are provided to demonstrate how to adjust evaluation dimension weights and explain the reasons based on changes in task nature. Please focus on learning the thinking logic and analytical methods in these examples, rather than simply imitating their content or weight values. </examples_rationale> <example_1>
• Insight (0.35): The task requires a deep analysis of feasibility, including market demand, costs, competition, and regulatory landscape. The quality of the strategic recommendations derived from this analysis is key. • ... <json_output> {{ "comprehensiveness": 0.30, "insight": 0.35, "instruction_following": 0.20, "readability": 0.15 }} </json_output> </example_1> Please strictly follow the above instructions and methods. Now, begin your work on the following specific task:
1. Comprehensiveness: The breadth, depth, and relevance of information coverage.
2. Insight: The depth, originality, logic, and value of the analysis and conclusions.
3. Instruction Following: Whether the report accurately and completely responds to all requirements and constraints of the task. 4. Readability: Clarity of structure, fluency of language, effectiveness of data presentation, and overall ease of understanding.
1. Analyze Task: Deeply analyze the '
1. Task-Centric: Analysis, criteria, explanations, and weights must directly relate to the core requirements and characteristics of the '
1. Remote Work Trends & Adoption Data: Coverage of current and projected remote/hybrid work models, adoption rates across industries and demographics. 2. Impact on Commercial Real Estate Sectors: Analysis of effects on office, retail, and industrial spaces, including vacancy rates, leasing trends, and property valuations in major US cities. 3. Geographical Variations: Examination of how impacts differ across various major US cities (e.g., tech hubs vs. financial centers, downtown vs. suburban). 4. ... Weight allocation should be balanced between the impact analysis (remote work trends, sector impacts, geographical variations) and the investment strategy section, as both are critical to fulfilling the task. Within impact analysis, specific sector impacts and geographical variations are key to actionable insights. <json_output> [ {{ "criterion": "Analysis of Remote Work Trends and Adoption", "explanation": "Assesses if the article thoroughly examines current and projected remote/hybrid work models, adoption statistics across different industries, and demographic factors influencing these trends. This forms the basis of the impact analysis.", "weight": 0.15 }}, {{ "criterion": "Comprehensive Coverage of CRE Sector Impacts", "explanation": "Evaluates if the article details the impact on various commercial real estate (CRE) sectors (office, retail, industrial, etc.), including data on vacancy rates, rental trends, and property valuations in major US cities.", "weight": 0.20 }}, {{ "criterion": "Examination of Geographical Variations and Nuances", "explanation": "Checks if the article analyzes how the impact of remote work on CRE differs across various major US cities, considering their unique economic structures and demographic profiles.", "weight": 0.15 }}, {{ "criterion": "Discussion of Broader Economic and Social Consequences", "explanation": "Assesses coverage of wider implications, such as effects on urban planning, transportation, local economies dependent on office workers, and ancillary businesses.", "weight": 0.10 }}, ... ] </json_output> Please strictly follow the above instructions and methods. Now, begin your work on the following specific task:
1. Analyze Each Criterion: Consider how each article fulfills the requirements of each criterion. 2. Comparative Evaluation: Analyze how the two articles perform on each criterion, referencing the content and criterion explanation. 3. Score Separately: Based on your comparative analysis, score each article on each criterion (0-10 points).
Scoring Rules
For each criterion, score both articles on a scale of 0-10 (continuous values). The score should reflect the quality of performance on that criterion:
• 0-2 points: Very poor performance. Almost completely fails to meet the criterion requirements. Evaluation Criteria Now, you need to evaluate and compare these two articles based on the following fixed evaluation criteria list, providing comparative analysis and scoring each on a scale of 0-10. Each criterion includes an explanation, please understand carefully. <criteria_list> Comprehensiveness [ {{ "criterion": "Information Coverage Breadth", "explanation": "Evaluates whether the article covers all key areas and aspects related to the topic without omitting important information.", "weight": 0.25 }}, {{ "criterion": "Information Depth and Detail", "explanation": "Evaluates whether the article provides sufficiently detailed information rather than just surfacelevel overviews.", "weight": 0. ... }} </output_format> Now, please evaluate the two articles based on the research task and criteria, providing detailed comparative analysis and scores according to the requirements above. Ensure your output follows the specified '<output_format>' and that the JSON format is parsable, with all characters that might cause JSON parsing errors properly escaped. </user_prompt> Point-wise Score Prompt <system_role> You are a strict, meticulous, and objective research article evaluation expert. You excel at using specific assessment criteria to thoroughly evaluate research articles, providing precise scores and clear justifications. </system_role> <user_prompt> Task Background There is a deep research task, and you need to evaluate a research article written for this task. We will assess the article across four dimensions: Comprehensiveness, Insight, Instruction Following, and Readability. The content is as follows:
1. Analyze Each Criterion: Consider how the article fulfills the requirements of each criterion.
2.Analysis and Evaluation:
Analyze the article's performance on each criterion, referencing the content and criterion explanation, noting strengths and weaknesses. 3. Score: Based on your analysis, score the article on each criterion (0-10 points).
Scoring Rules
For each criterion, score the article on a scale of 0-10 (continuous values). The score should reflect the quality of performance on that criterion:
• 0-2 points: Very poor performance. Almost completely fails to meet the criterion requirements.
• 2-4 points: Poor performance. Minimally meets the criterion requirements with significant deficiencies.
• 4-6 points: Average performance. Basically meets the criterion requirements, neither good nor bad.
• 6-8 points: Good performance. Largely meets the criterion requirements with notable strengths.
• 8-10 points: Excellent/outstanding performance. Fully meets or exceeds the criterion requirements. Output Format Requirements Please strictly follow the '<output_format>' below for each criterion evaluation. Do not include any other unrelated content, introduction, or summary. ... }} </output_format> Now, please evaluate the article based on the research task and criteria, providing detailed analysis and scores according to the requirements above. Ensure your output follows the specified '<output_format>' and that the JSON format is parsable, with all characters that might cause JSON parsing errors properly escaped. </user_prompt> Vanilla Prompt <system_role> You are a strict, meticulous, and objective research article evaluation expert. You excel at using specific assessment criteria to thoroughly evaluate research articles, providing precise scores and clear justifications. </system_role> <user_prompt> Task Background There is a deep research task, and you need to evaluate a research article written for this task.
annex
Miserendino, Sandhini Agarwal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay, Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang, Taylor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng
Failed to load PDF:
StripeM-Inner
Paper sources
Abstract screening pilot
Abstract screening results
Extraction pilot
Extraction results
Research report
\
- *
Modify setup
- *