Which benchmarks measure performance on scientific literature review tasks?

Benchmarks

Benchmarks for scientific literature review tasks include datasets like LitSearch, SciReviewGen, SWIFT‐Review, and RoBBR that measure performance across retrieval, generation, screening, extraction, and meta-evaluation areas.

Abstract

Benchmarks for scientific literature review tasks address five primary areas: retrieval, generation, screening, extraction, and meta-evaluation. Seventeen studies target retrieval using datasets such as LitSearch, CLEF TAR, CSMeD, SIGIR2017-PICO-Collection, FASS‐BSLR, RELISH, and ResearchArena. These studies report performance using metrics such as recall, precision, Mean Average Precision, normalized Discounted Cumulative Gain, and Work Saved over Sampling. Eleven studies focus on generation tasks—including review, table, or abstract writing—with benchmarks like SciReviewGen, ARXIV2TABLE, MSLR2022, ScholarQABench, and Manalyzer; performance is measured by ROUGE scores, hallucination rates, semantic coverage, and human ratings. Nine studies on screening tasks (using tools such as SWIFT‐Review and LGAR) report measures such as precision at high recall levels and WSS@95, while five studies on extraction or structured analysis (using RoBBR, EvidenceBench, and ECACT) employ metrics such as macro‐F1, accuracy, and Cohen’s kappa. One study adopts a meta‐evaluation benchmark using Elo ratings and inter‐annotator agreement. Overall, papers combine automated retrieval and overlap metrics with human evaluation methods to capture the multifaceted nature of performance in scientific literature review tasks.

Methods

We analyzed 40 sources from an initial pool of 1000, using 5 screening criteria. Each paper was reviewed for 4 key aspects that mattered most to the research question.

Papers identified with Elicit search

n = 1000
Papers screened using: Benchmark Focus, Computational Literature Review Systems, Beyond Methods Description Only, Literature Review Specific Application, Computational Component
n = 1000
Papers screened out
n = 960
Papers included for extraction
n = 40

Paper search

Using your research question “Which benchmarks measure performance on scientific literature review tasks?”, we searched across over 126 million academic papers from the Semantic Scholar corpus. We retrieved the 1000 papers most relevant to the query.

Screening

We screened in sources based on their abstracts that met these criteria:

• Benchmark Focus: Does this study describe, develop, evaluate, present datasets for, or validate benchmarks/evaluation frameworks specifically designed to measure performance on scientific literature review tasks?
• Computational Literature Review Systems: Does this study focus on computational systems that perform literature search, screening, extraction, synthesis, or other core literature review functions, OR does it synthesize knowledge about literature review benchmarks through systematic/narrative reviews?
• Beyond Methods Description Only: Does this study go beyond only describing literature review methods to actually present, evaluate, or analyze benchmarks or evaluation metrics?
• Literature Review Specific Application: Are the benchmarks or evaluation approaches specifically applied to literature review tasks (rather than focusing solely on general information retrieval or text mining without literature review application)?
• Computational Component: Does this study include computational components rather than focusing exclusively on manual literature review processes?

Data extraction

We asked a large language model to extract each data column below from each paper.

• Benchmark Type: Identify and describe the specific type of benchmark created in the study. Look for explicit statements about the benchmark’s purpose, focus, and unique characteristics.

Examples:

• Hierarchical catalogue generation benchmark
• Literature search retrieval benchmark
• Information distillation benchmark

Benchmark Dataset Characteristics:

• Total number of items in the dataset
• Types of items (e.g., research papers, citations, queries)
• Domain or field of research covered
• Data collection method
• Source of data items

Evaluation Metrics:

Identify and describe the specific metrics used to assess benchmark performance. Look for metrics such as:

• Recall@5
• Semantic similarity scores
• Informativeness ratings
• Comparative performance against baseline models

Benchmark Models and Comparisons:

Document the models or systems tested against the benchmark.

• List all models/systems evaluated
• Note their performance relative to the benchmark
• Capture any comparative analysis between different approaches

Results

Characteristics of Included Studies

Distribution of Literature Review Automation Tasks

• Retrieval, search, or ranking tasks: Addressed in 17 studies.
• Generation tasks: Addressed in 11 studies.
• Screening tasks: Addressed in 9 studies.
• Extraction tasks: Addressed in 5 studies.
• Meta-evaluation or meta-analysis: Addressed in 1 study.
• Other tasks: Addressed in 10 studies.

Distribution of Evaluation Approaches

• Information retrieval metrics: Used in 27 studies.
• Human evaluation: Used in 6 studies.
• Overlap, coverage, or consistency metrics: Used in 6 studies.
• Specialized or composite metrics: Used in 7 studies.

Thematic Analysis

Benchmarks for Literature Retrieval and Screening Tasks

Benchmarks in this category focus on identifying, retrieving, and prioritizing relevant literature for systematic reviews or evidence synthesis.

Benchmarks for Literature Summarization and Synthesis Tasks

This theme covers benchmarks designed to evaluate the ability of models to generate summaries, reviews, or structured tables from collections of scientific papers.

Benchmarks for Structured Literature Analysis Tasks

Structured analysis tasks include risk of bias assessment, data element extraction, compliance and traceability, and meta-evaluation.

Evaluation Methodologies Across Benchmarks

Conclusion

This paper puts forward a novel, outcome-based evaluation framework for assessing the effectiveness of automatic search strategies and citation screening methods in the context of systematic literature reviews. Our proposed framework evaluates the quality of these methods based on how closely the outcomes of their included publications match the actual review outcomes. We believe that this approach offers a more accurate reflection of real-world scenarios where not all included publications have the same impact on the final review outcome.