Elicit: Benchmarks for Scientific Literature Review Performance (public)
Benchmarks for Scientific Literature Review Performance (public)
Research reportView only
Create alertChat
September 12, 2025
Which benchmarks measure performance on scientific literature review tasks?
Benchmarks for scientific literature review tasks include datasets like LitSearch, SciReviewGen, SWIFT‐Review, and RoBBR that measure performance across retrieval, generation, screening, extraction, and meta-evaluation areas.
Abstract
Benchmarks for scientific literature review tasks address five primary areas: retrieval, generation, screening, extraction, and meta-evaluation. Seventeen studies target retrieval using datasets such as LitSearch, CLEF TAR, CSMeD, SIGIR2017-PICO-Collection, FASS‐BSLR, RELISH, and ResearchArena. These studies report performance using metrics such as recall, precision, Mean Average Precision, normalized Discounted Cumulative Gain, and Work Saved over Sampling. Eleven studies focus on generation tasks—including review, table, or abstract writing—with benchmarks like SciReviewGen, ARXIV2TABLE, MSLR2022, ScholarQABench, and Manalyzer; performance is measured by ROUGE scores, hallucination rates, semantic coverage, and human ratings. Nine studies on screening tasks (using tools such as SWIFT‐Review and LGAR) report measures such as precision at high recall levels and WSS@95, while five studies on extraction or structured analysis (using RoBBR, EvidenceBench, and ECACT) employ metrics such as macro‐F1, accuracy, and Cohen’s kappa. One study adopts a meta‐evaluation benchmark using Elo ratings and inter‐annotator agreement. Overall, papers combine automated retrieval and overlap metrics with human evaluation methods to capture the multifaceted nature of performance in scientific literature review tasks.
Methods
We analyzed 40 sources from an initial pool of 1000, using 5 screening criteria. Each paper was reviewed for 4 key aspects that mattered most to the research question. More on methods
Papers identified with Elicit search
n = 1000
Papers screened using: Benchmark Focus, Computational Literature Review Systems, Beyond Methods Description Only, Literature Review Specific Application, Computational Component
n = 1000
Papers screened out
n = 960
Papers included for extraction
n = 40
Press enter or space to select a node.You can then use the arrow keys to move the node around. Press delete to remove it and escape to cancel.
Press enter or space to select an edge. You can then press delete to remove it or escape to cancel.
Paper search
Using your research question “Which benchmarks measure performance on scientific literature review tasks?”, we searched across over 126 million academic papers from the Semantic Scholar corpus. We retrieved the 1000 papers most relevant to the query.
Screening
We screened in sources based on their abstracts that met these criteria:
- Benchmark Focus: Does this study describe, develop, evaluate, present datasets for, or validate benchmarks/evaluation frameworks specifically designed to measure performance on scientific literature review tasks?
- Computational Literature Review Systems: Does this study focus on computational systems that perform literature search, screening, extraction, synthesis, or other core literature review functions, OR does it synthesize knowledge about literature review benchmarks through systematic/narrative reviews?
- Beyond Methods Description Only: Does this study go beyond only describing literature review methods to actually present, evaluate, or analyze benchmarks or evaluation metrics?
- Literature Review Specific Application: Are the benchmarks or evaluation approaches specifically applied to literature review tasks (rather than focusing solely on general information retrieval or text mining without literature review application)?
- Computational Component: Does this study include computational components rather than focusing exclusively on manual literature review processes?
We considered all screening questions together and made a holistic judgement about whether to screen in each paper.
Data extraction
We asked a large language model to extract each data column below from each paper. We gave the model the extraction instructions shown below for each column.
- Benchmark Type:
Identify and describe the specific type of benchmark created in the study. Look for explicit statements about the benchmark’s purpose, focus, and unique characteristics.
Extraction guidelines:
- Locate the benchmark description in the introduction or methods section
- Capture the specific scientific literature review task the benchmark addresses
- If multiple benchmark types are described, list all of them
- Be precise about the benchmark’s scope (e.g., retrieval, generation, search)
Examples might include:
- Hierarchical catalogue generation benchmark
- Literature search retrieval benchmark
- Information distillation benchmark
If no clear benchmark is described, write “Not applicable” or “No benchmark created”
- Benchmark Dataset Characteristics:
Extract detailed information about the dataset used to create or evaluate the benchmark.
Look for:
- Total number of items in the dataset
- Types of items (e.g., research papers, citations, queries)
- Domain or field of research covered
- Data collection method
- Source of data items
Extraction guidelines:
- Prioritize quantitative details (e.g., “7.6k literature review catalogues and 389k reference papers”)
- Note the specific research domains if mentioned
- If multiple datasets are used, list all with their characteristics
- If dataset details are incomplete, note “Insufficient information”
Format examples:
“7,600 literature review catalogues from scientific papers”
“597 literature search queries across ML and NLP domains”
Evaluation Metrics:
Identify and describe the specific metrics used to assess benchmark performance.
Extraction guidelines:
- Locate metrics in results, methods, or discussion sections
- Capture both quantitative metrics and qualitative assessment approaches
- Note the specific performance dimensions being measured
- If multiple metrics are used, list all of them
Look for metrics such as:
- Recall@5
- Semantic similarity scores
- Informativeness ratings
- Comparative performance against baseline models
Format examples:
“24.8% difference in absolute recall@5”
“Two evaluation metrics for informativeness and structural similarity”
Benchmark Models and Comparisons:
Document the models or systems tested against the benchmark.
Extraction guidelines:
- List all models/systems evaluated
- Note their performance relative to the benchmark
- Capture any comparative analysis between different approaches
- Include both state-of-the-art and baseline models
Look for:
- Specific model names (e.g., BART, ChatGPT)
- Performance rankings
- Comparative performance percentages
Format examples:
- “BART and ChatGPT tested, with performance variations noted”
- “Commercial search engines performed 32 points below best dense retriever”
Results
Characteristics of Included Studies
Study
Study Focus
Benchmark/Dataset Name
Literature Review Task Type
Evaluation Approach
Full text retrieved
Zhu et al., 2023
Hierarchical catalogue generation for literature reviews
HiCaD
Hierarchical catalogue generation
CEDS (semantic/structural similarity), CQE (informativeness)
Yes
Ajith et al., 2024
Literature search retrieval
LitSearch
Literature search/retrieval
Recall at 5, recall at 20, normalized Discounted Cumulative Gain at 10 (nDCG@10)
Yes
Kanoulas et al., 2019
Systematic review retrieval and ranking
CLEF 2019 e-Health Technology-Assisted Review (TAR)
Retrieval, ranking for systematic reviews
No mention found in abstract
No
Kasanishi et al., 2023
Automatic literature review generation
SciReviewGen
Literature review generation (summarization)
Recall-Oriented Understudy for Gisting Evaluation (ROUGE), human evaluation (relevance, coherence, informativeness, factuality)
Yes
Howard et al., 2016
Automated citation screening
SWIFT-Review datasets
Citation screening
Work Saved over Sampling at 95% (WSS@95%), precision at 95% recall
Yes
Wang et al., 2025a
Literature review table generation
ARXIV2TABLE
Table generation for literature reviews
Recall, precision, F1 for schema/cell/pairwise overlap
Yes
Kusa et al., 2023a
Outcome-based evaluation of systematic review automation
CLEF TAR 2019
Systematic review automation (retrieval)
Mean Average Precision (MAP), Recall at k%, WSS, Area Under the Curve (AUC), outcome difference
Yes
Kusa et al., 2023b
Automated citation screening
CSMeD, CSMeD-FT
Citation/full-text screening
True Negative Rate at 95% (TNR@95%), normalized Precision at 95% (nP@95%), nDCG@10, MAP, macro-precision/recall/F1
Yes
Scells et al., 2017
Retrieval for systematic reviews
SIGIR2017-PICO-Collection
Retrieval, screening prioritization
Precision-recall, F-beta, WSS, Average Precision (AP), nDCG, MAP
Yes
Budau and Ensan, 2024
Automated study search for biomedical systematic literature reviews
FASS-BSLR
Study search (retrieval, Boolean query generation)
Precision, Recall, NDCG, MAP, Recall at 1000
No
toof
Pageof
View 30 more rows
Note:All abbreviations and acronyms are defined at first use in the table or in this note. “No mention found” indicates that we did not find mention of the information in the available abstract or full text.
Distribution of Literature Review Automation Tasks
- Retrieval, search, or ranking tasks:Addressed in 17 studies. These focus on identifying and prioritizing relevant literature for systematic reviews or evidence synthesis.
- Generation tasks:Addressed in 11 studies. These include review, table, summary, reference, or abstract generation.
- Screening tasks:Addressed in 9 studies. These include citation, abstract, or full-text screening.
- Extraction tasks:Addressed in 5 studies. These include data, evidence, risk of bias, or meta-analysis.
- Meta-evaluation or meta-analysis:Addressed in 1 study.
- Other tasks:Addressed in 10 studies. These include query generation, compliance, information distillation, academic survey, trial design, systematic review-style reasoning, or multiple systematic literature review tasks.
Distribution of Evaluation Approaches
- Information retrieval metrics:Used in 27 studies. Metrics include precision, recall, F1, Mean Average Precision (MAP), normalized Discounted Cumulative Gain (nDCG), Mean Reciprocal Rank (MRR), Average Precision (AP), and others.
- Human evaluation:Used in 6 studies. Includes inter-annotator agreement, Cohen’s kappa, or explicit human ratings.
- Overlap, coverage, or consistency metrics:Used in 6 studies. Includes schema/cell overlap, coverage, factual consistency, or hallucination rate.
- Specialized or composite metrics:Used in 7 studies. Includes Elo ratings, CEDS, CQE, ECACT, or outcome difference.
- No mention found of evaluation approach:In 6 studies.
Some studies addressed multiple task types or used multiple evaluation approaches. We did not find mention of evaluation approach for 6 studies, and task type information was sometimes ambiguous due to overlapping or unclear descriptions.
Thematic Analysis
Benchmarks for Literature Retrieval and Screening Tasks
Benchmarks in this category focus on identifying, retrieving, and prioritizing relevant literature for systematic reviews or evidence synthesis. Key benchmarks include:
- LitSearch, CLEF Technology-Assisted Review (TAR) (2017–2019), CSMeD, SIGIR2017-PICO-Collection, FASS-BSLR, RELISH, and ResearchArena:These benchmarks use large-scale datasets such as PubMed, MEDLINE, and S2ORC, and evaluate models using recall, precision, Mean Average Precision, normalized Discounted Cumulative Gain, Work Saved over Sampling, and related metrics.
- Model types:Both classical information retrieval models (such as BM25 and term frequency-inverse document frequency) and modern transformer-based or large language model approaches are tested.
- Findings:Some studies (e.g., Kang and Xiong, 2024) report that keyword-based retrieval still outperforms large language model-based methods in certain settings, while others (e.g., Skarlinski et al., 2024) demonstrate large language models matching or exceeding human performance in specific retrieval and synthesis tasks.
Benchmarks for Literature Summarization and Synthesis Tasks
This theme covers benchmarks designed to evaluate the ability of models to generate summaries, reviews, or structured tables from collections of scientific papers. Notable benchmarks include:
- SciReviewGen, ARXIV2TABLE, MSLR2022, ScholarQABench, and Manalyzer:Evaluation metrics include ROUGE, human preference ratings, semantic coverage, factual consistency, and hallucination rates.
- Findings:Some large language models and transformer-based models achieve promising results, but challenges remain in factuality, citation accuracy, and domain adaptation. Human evaluation is often used to supplement automated metrics, reflecting the complexity of assessing synthesis quality.
Benchmarks for Structured Literature Analysis Tasks
Structured analysis tasks include risk of bias assessment, data element extraction, compliance and traceability, and meta-evaluation. Key benchmarks:
- RoBBR, EvidenceBench, ECACT, SciArena-Eval, and internal datasets:These benchmarks use composite or task-specific metrics such as ECACT score, macro-F1, accuracy, and Cohen’s kappa.
- Findings:Automation is advancing, but human oversight remains essential, especially in high-stakes or regulatory contexts.
Evaluation Methodologies Across Benchmarks
Benchmark Category
Primary Metrics
Evaluation Method
Performance Measurement Focus
Retrieval/Screening
Recall, precision, Mean Average Precision, normalized Discounted Cumulative Gain, Work Saved over Sampling, Average Precision
Automated scoring, comparison to human or baseline models
Relevance, workload reduction, ranking effectiveness
Summarization/Synthesis
ROUGE, human ratings, semantic coverage, hallucination rate
Automated metrics, human evaluation
Informativeness, factuality, coherence, citation accuracy
Structured Analysis
Macro-F1, accuracy, Cohen’s kappa, composite scores (ECACT)
Automated scoring, statistical analysis, human validation
Bias, compliance, extraction accuracy, traceability
Meta-evaluation
Elo ratings, accuracy, agreement
Human preference voting, model-based evaluator assessment
Alignment with human judgment, evaluator reliability
toof
Pageof
Summary of Evaluation Approaches:
- Primary metrics:Accuracy was the only metric used in more than one category (2 out of 4 categories). All other metrics were unique to a single category, including recall, precision, Mean Average Precision, normalized Discounted Cumulative Gain, Work Saved over Sampling, Average Precision, ROUGE, human ratings, semantic coverage, hallucination rate, macro-F1, Cohen’s kappa, composite scores (ECACT), Elo ratings, and agreement.
- Evaluation methods:Automated evaluation methods were used in 3 out of 4 categories. Human-based evaluation methods were used in all 4 categories. Comparison-based methods, statistical analysis, and model-based evaluator assessment were each used in 1 out of 4 categories.
- Coverage:We did not find any categories without at least one automated and one human-based evaluation method. No information was missing for any category.
References
Download BIBDownload RISDownload TXT
Report
Status
Gather sources
1000 sources found
Details
Screen sources
40 sources included
Details
Extract data
160 data points extracted
Details
Generate report
Save PDF
BIBLaTeX, ZoteroRISZotero, MendeleyTXTAPA BibliographyPDFPDFDOCXMicrosoft Word
Chat
Got some follow-up questions?
Sign up or sign in to chat with this report.
Back
Measuring Risk of Bias in Biomedical Reports: The RoBBR Benchmark
Jianyou Wang, Weili Cao, Longtian Bao, Youze Zheng, Gil Pasternak, Kaicheng Wang, Xiaoyue Wang, R. Paturi, Leon Bergen
arXiv.org·
2024·
0 citations
SourceDOI
Plain textPDFFigures
Searching for PDF
Unable to find PDF from source
Back
Benchmark Type
Risk of Bias Assessment Benchmark
Benchmark Dataset Characteristics
- Total number of items: 63 Cochrane meta-analyses and 532 papers - Types of items: Cochrane meta-analyses and papers referenced in those meta-analyses - Domain or field of research: Biomedical research, specifically intervention reviews - Data collection method: Obtained from BioC database and converted to XML using GROBID - Source of data items: Cochrane meta-analyses and papers from BioC database - Additional details: Includes 2000 expert-generated bias annotations and is human-validated
Evaluation Metrics
- Task 1: macro-F1, accuracy, weighted accuracy - Task 2: Bias Recall @ 3, Bias Recall @ Optimal - Task 3: accuracy - Task 4: accuracy, weighted accuracy
Benchmark Models and Comparisons
- Models tested: Claude3-Opus, GPT-4o, Gemini-1.5-Pro - Performance relative to the benchmark: These models fall significantly short of expert-level performance. - Comparative analysis: Claude3-Opus and GPT-4o perform best in different tasks; GPT-4o excels in sentence retrieval, while Claude3-Opus is strong in support judgment selection.
Systems that answer questions by reviewing the scientific literature are becoming increasingly feasible. To draw reliable conclusions, these systems should take into account the quality of available evidence, placing more weight on studies that use a valid methodology. We present a benchmark for measuring the methodological strength of biomedical papers, drawing on the risk-of-bias framework used for systematic reviews. The four benchmark tasks, drawn from more than 500 papers, cover the analysis of research study methodology, followed by evaluation of risk of bias in these studies. The benchmark contains 2000 expert-generated bias annotations, and a human-validated pipeline for fine-grained alignment with research paper content. We evaluate a range of large language models on the benchmark, and find that these models fall significantly short of expert-level performance. By providing a standardized tool for measuring judgments of study quality, the benchmark can help to guide systems that perform large-scale aggregation of scientific data. The dataset is available at https://github.com/RoBBR-Benchmark/RoBBR.
1Introduction
Systems that automatically answer questions by reviewing the scientific literature are becoming increasingly feasible. These systems offer the potential to provide scientists with on-demand access to knowledge that is synthesized from across the literature. This can help scientists understand what is already known about topics outside of their research focus, and help clinicians and practitioners keep up to date with best practices.
When assessing what is known about a field, not all studies should be weighed equally I et al. [2023] . Studies with stronger methodologies contribute more to a body of evidence than those with weaker methodologies. By weighing studies appropriately, systems can increase the reliability of their summaries and recommendations Turner et al. [2009].
In biomedical research, there is a large body of work which investigates the factors that decrease the validity of a study's methods, and best practices for addressing these issues I et al. [2023] , Welton et al. [2009]. Reporting bias occurs when there are systematic differences in how outcomes are reported or disclosed between the groups that are compared (e.g., the results of the treatment group are more frequently or favorably published than those in the placebo group). This bias is often best addressed by adhering to strict reporting standards and protocols, such as registering the study in advance and committing to publish all results Kotz and West [2022]. Other biases, such as detection bias, attrition bias, and selection biases, can frequently be addressed using careful study design and implementation Berger [2005], JPT et al. [2023 JPT et al. [ ], I et al. [2023]].
Figure 1 : The figure illustrates the components of a risk of bias assessment. Given a bias category and a research study, a meta-reviewer identifies factors that make the study high or low risk for that bias. The meta-reviewer summarizes their findings in a support judgment, and then make a final decision about the risk level. This paper presents the RoBBR benchmark for assessing the methodological strength of biomedical studies. The benchmark follows the risk of bias framework developed by Cochrane Jpt et al. [2023] , JPT et al. [2023] , Higgins et al. [2011] , Practice and of Care [EPOC], Sterne et al. [2014], a global independent network that conducts systematic reviews of healthcare interventions. Cochrane's risk of bias framework is widely recognized as a standard for evaluating the quality and reliability of research studies. It serves as an important resource for national health agencies in various countries, including the United States Viswanathan et al. [2012], the United Kingdom Alderson and Tan [2011], and Australia NHMRC [2019] . Notably, Cochrane's systematic reviews are used to formulate national and international guidelines on healthcare practices Bunn et al. [2015]. Furthermore, empirical comparisons between Cochrane and non-Cochrane systematic reviews consistently demonstrate higher methodological quality in those conducted by Cochrane, highlighting its critical role in advancing evidence-based healthcare Goldkuhle et al. [2018], Page et al. [2018], Matthias et al. [2020], Tsoi et al. [2020].
2Background
The RoBBR benchmark aims to provide an expert-validated tool for assessing whether models can make risk of bias judgments for biomedical papers. The benchmark uses a risk-of-bias framework developed in the context of systematic reviews, which is the gold standard for assessing biomedical studies. We provide more background on systematic reviews and the risk of bias framework below.
Systematic reviews and meta-analyses are tools used in evidence-based medicine, which synthesize evidence from multiple studies to answer important clinical questions. They help researchers and healthcare professionals make informed decisions based on the best available evidence Ahn and Kang [2018], TJ et al. [2023], JJ et al. [2023].
A systematic review is a comprehensive search of the literature to identify all relevant studies on a particular topic TJ et al. [2023 TJ et al. [ ], C et al. [2023]] . The review process follows a pre-specified protocol to minimize bias and ensure transparency. The protocol outlines the research question, search strategy, study selection criteria, data extraction methods, and plans for quality assessment and synthesis of result Jpt et al. [2023] , Page et al. [2021].
A meta-analysis is a type of systematic review that uses statistical methods to combine the results from multiple studies Ahn and Kang [2018]. By pooling the data across studies, a meta-analysis can provide a more precise estimate of the treatment effect JJ et al. [2023], Guevara et al. [2004] and explore potential sources of heterogeneity Higgins and Thompson [2002].
The research question for a systematic review is often structured using the PICO framework, which stands for Participant, Intervention, Comparison, and Outcome Aslam and Emmanuel [2010]. The PICO elements help define the key components of the research question and guide the study selection process J et al. [2023[ ], JE et al. [2023a]]. For example, a systematic review on the effectiveness of a new hypertension drug might have the following PICO: Participant: Adults with primary hypertension; Intervention: New antihypertensive drug; Comparison: Placebo or standard care; Outcome: Change in systolic and diastolic blood pressure.
By clearly defining the PICO elements, the review authors can develop a focused search strategy and establish explicit criteria for selecting studies to include in the review J et al. [2023[ ], JE et al. [2023b]].
Inclusion criteria typically specify the characteristics of the study population, the interventions and comparisons of interest, the outcomes to be measured, and the study designs to be included JE et al.
[2023b], Page et al. [2021], Tawfik et al. [2019] (e.g., randomized controlled trials). Exclusion criteria describe reasons why a study must be excluded from an analysis, such as using the wrong study design C et al. [2023] , Page et al. [2021], Tawfik et al. [2019].
Risk of bias assessment involves evaluating each included study to determine the likelihood that its results may be biased JPT et al. [2023] , Practice and of Care [EPOC], Sterne et al. [2014]. The Cochrane tool for assessing risk of bias Higgins et al. [2011] is widely used and assesses several types of bias, including random sequence generation, allocation concealment, blinding of participants and outcome assessors, incomplete outcome data, selective reporting, and other sources of bias. During the assessment, two review authors independently evaluate the risk of bias using the Cochrane tool and resolve disagreements through discussion. They record their rationale for each bias assessment. These rationales are referred to as "support judgments."
For example, a support judgment of Reporting Bias could be "The study protocol was registered (ClinicalTrials.gov NCT01863706) but not all of the outcomes projected by methodological descriptions were reported as results in the study report (cases of diarrhoea, nausea and vomiting were not completely reported). Moreover, the study publication reports outcomes (hypotension, nausea, transfusion) not listed in the registered protocol." A decision regarding the risk of reporting bias is then made on the basis of the support judgment. See Figure 1 for an illustration.
Studies with a high risk of bias may overestimate or underestimate the true treatment effect, leading to inaccurate conclusions in the systematic review. In this paper, we consider eight categories of biases: selection bias, attrition bias, performance bias, detection bias, reporting bias, analysis bias, special bias (specific to individual meta-analyses), and other bias (i.e. biases not covered by the other categories). We benchmark model performance on assessing papers for each of these bias categories.
3Benchmark Development
The RoBBR benchmark is derived from 63 Cochrane meta-analyses and 532 papers referenced in those meta-analyses. All of the meta-analyses and papers have a Creative Commons license or are public domain. The benchmark is designed to evaluate a model's ability to judge the risk of bias in biomedical reports.
The data were collected in April 2024. We included all meta-analyses from Cochrane that are in the BioC database, are classified as intervention reviews, have a CC-BY-NC license, and contribute at least one data point for each task. We obtained the XML format of the papers included in these meta-analyses either through the BioC API or by manually downloading and converting PDFs to XML using GROBID. We then extract task data from the XML of these papers.
The benchmark is divided into four main tasks (See in Appendix A.3 for detailed statistics):
• Study Inclusion/Exclusion (Task 1): Determine whether a study's objectives and methodology fit a set of requirements. • Bias Retrieval (Task 2): Retrieve sentences from the paper that support a judgment about the study's risk of bias. • Support Judgment Selection (Task 3): From a list of options, select the judgment that best supports a determination of the study's risk of bias. • Risk Level Determination (Task 4): Assess the study's risk of specific biases.
These four tasks mirror the procedure followed by meta-reviewers. Initially, reviewers decide if a study should be included or excluded based on predefined criteria, which reflect the goals of the review (Task 1). Included studies will be analyzed for indications of the risk of bias (Task 2). Reviewers will then summarize the evidence collected from the study, and analyze its implications (Task 3). Finally, they assess the risk level of specific biases for each identified bias (Task 4).
By structuring the dataset around these tasks, we provide a comprehensive framework for simulating the nuanced process of risk of bias assessment in meta-analysis.
3.1Train/Test Split
We randomly select 23 meta-analyses and their 228 corresponding papers in the dev set. We put the other 40 meta-analyses and their corresponding 304 papers in the test set.
Figure 2 : Study Inclusion/Exclusion. In Task 1, the model sees a search protocol and objective, and is asked to determine whether a study's methodology fits those criteria.
3.2Task 1: Study Inclusion/Exclusion
For each potentially relevant study to be included in a meta-analysis, the search protocol information and meta-analysis objective are provided as input, and the task is to decide if the study should be included in the meta-analysis. The objective is included in order to provide similar context to what is normally available for a meta-reviewer and make it easier to interpret the criteria in the search protocol. An example is provided in Figure 2 .
3.3Task 2: Risk of Bias Sentence Retrieval
When judging the risk of bias for a specific study, the review authors provide support judgments which justify their risk of bias rating. These support judgments are grounded in specific aspects of the study, which the support judgment describes. Task 2 aims to test a model's ability to retrieve the correct source for the support judgment from a paper's text. The support judgments are intentionally omitted in the Bias Retrieval task. This exclusion simulates the systematic review process by challenging the model to independently generate these justifications. Including them would come close to providing the answer, undermining the purpose of the task. See Figure 3 for an illustration. Some review authors directly quote sentences from the paper in their support judgment, allowing for straightforward extraction and matching of these sentences with the paper's text. However, most reviewers prefer to paraphrase the sentences or engage in deeper analysis, synthesizing information from multiple parts of the paper.
To match these judgments with sentences from the paper, we use a three part annotation pipeline.
First, a support judgment often contains multiple pieces of information. We decompose the support judgment into distinct, non-overlapping pieces, each focusing on a specific aspect of the original support judgment.
Second, we exclude any part of the support judgment containing a negation or non-specific commentary, because these parts of the judgment cannot be matched to any sentence. The remaining parts of the support judgments generally contain specific claims about the study.
Third, these aspects are mapped to specific sentences within the paper, allowing us to pinpoint the exact sources of information that underlie each aspect of the judgment. A single aspect could potentially be supported by multiple sentences from the paper, and similarly a single sentence can support multiple aspects.
Each of these steps, including the aspect decomposition, uses GPT-4 (gpt-4-0125) OpenAI [2023] . The prompts can be found in the Appendix C.
We manually inspect the 2221 support judgment aspects generated in the second stage, to ensure that all aspects are related to specific information from the paper.
In the next section, we evaluate annotation quality by comparing to human judgments.
Figure 3 : Risk of Bias: Sentence Retrieval. In Task 2, the goal is to retrieve sentences from the paper (the Candidate Pool) which support a risk of bias judgment. Performance is evaluated against a Cochrane review support judgment, which has been split into aspects.
3.3.1Evaluating LLM Annotations
To match each aspect from a support judgment with specific sentences in the paper, we need to evaluate each sentence (on average, 200 sentences) against each aspect. For a support judgment with 5 aspects, this results in approximately 1,000 decisions. Given the scale of our entire dataset, which includes over 500 (bias, study) pairs, this translates to more than 1 million decisions -an impractical task for human annotators to perform manually.
We hypothesize that determining whether a sentence matches an aspect is straightforward enough for GPT-4 to achieve human-level performance with a specialized prompt. Following the protocol introduced by Wang et al. [2023], we created a development set of 30 papers. A single support judgment aspect was chosen for each paper, and every sentence in the paper was annotated for that aspect, totaling 8789 decisions. We optimized the instruction prompt for GPT-4 on this development set. Details about the prompt and experimental procedure are provided in the Appendix C.
Once the prompt was finalized, four annotators were tasked with annotating 50 randomly sampled (aspect, full paper) pairs (all annotators saw the same pairs). Every sentence in each paper was annotated, requiring 13575 annotations in total. The annotators were divided into two teams, with each team consisting of two annotators. Each annotator performed the annotation tasks individually given a pre-defined annotation guideline. Annotators from the same team then collaborated to resolve differences and eliminate mistakes. Refer to the Appendix B for the full annotation guidelines.
Each team produced a set of annotation results. GPT-4 annotated the same (aspect, paper) pairs. We calculated Binary Accuracy, F1-binary, Spearman Correlation, and Kappa Coefficient between the two human teams and between each human team and GPT-4. Table 1 shows that the human teams matched each other and GPT-4 more than 99% of the time. The other agreement metrics have values in the low-to-mid 70's, indicating substantial agreement among annotators.3 Bootstrapped hypothesis tests do not demonstrate any statistical difference between GPT-4 and the human annotators. The previous task evaluates a model's ability to extract information from biomedical papers to in order to justify assessments of bias. However, this approach alone is insufficient for a risk-of-bias assessment. The evaluation must also include the model's ability to reason about the retrieved information. For example, a paper might report the loss to follow-up yet fail to provide reasons for missing data, which raises the possibility of attrition bias. Similarly, while a paper may state that it implemented double-blinding, expert reviewers could consider such blinding infeasible due to the nature of the trial, potentially introducing performance bias.
To address this, we propose a multiple-choice task where the model selects the correct support judgment. This task includes three synthetically generated hard options, three options derived from other papers' support judgments for the same bias category, and the Cochrane reviewer's actual judgment. We prompt GPT-4 to generate the three synthetic options by imitating support judgments from other papers concerning the same type of bias. These options are tailored to be paper-specific while maintaining the underlying reasoning. See Appendix D for details on construction of these synthetic judgments.
In order to ensure that all incorrect options were actually incorrect, we performed filtering in two stages. GPT-4 was used to perform preliminary filtering (the prompt is shown in Figure 13 of Appendix D). In the second stage, we manually reviewed all of the remaining options, and removed any that were not incorrect.
Figure 4 shows an example of a multiple-choice question with one correct answer and three synthetically generated answers. Both options C and D refer to the same information from the paper, in this case the operational challenges affecting the trial's methodology, but describe different reasoning given this information. This demonstrates that retrieving the correct information from the paper is not sufficient for solving this task. This assessment requires synthesizing various elements identified in the previous tasks, including the accurate retrieval and interpretation of relevant sentences and the ability to critically evaluate the sufficiency and reliability of the evidence presented regarding each bias. We use chain-of-thought as prompting strategy to stabilize the model performance. We optimize our prompts on a separate development set. See Appendix E for details on prompt optimization.
4.2Evaluation metrics
For Task 1, study inclusion/exclusion, we report macro-F1, accuracy, and weighted accuracy.
For Task 2, Risk of Bias Sentence Retrieval, we report aspect-level recall metrics, which measure the proportion of the bias aspects that are covered by the retrieved sentences. Specifically, Bias Recall (BR) @ 3 measures the proportion aspects covered when three sentences are retrieved. BR @ Optimal measures recall when the "optimal" number of sentences are retrieved. The optimal number of sentences is defined as the fewest which are sufficient to cover all bias aspects.
For Task 3, Risk of Bias Support Judgment Selection, we report accuracy.
For Task 4, Risk Level Determination, we report accuracy and weighted accuracy.
4.3Categorization
For Tasks 2, 3, and 4, we consider 8 categories of biases: selection bias, attrition bias, performance bias, detection bias, reporting bias, analysis bias, special bias (the biases that are meta-analysis specific), and other bias (i.e. biases that do not fall into any of the other categories). In order to reduce evaluation costs, we randomly sampled 50 task instances for some categories. It is important to note that systematic reviews vary in which biases they report. When the authors judge that a bias is not important for the objective of a review, they may decide not to evaluate a specific bias. As a result, not all categories have the same number of examples, and some categories have fewer than 50 examples. See Table 10 for detailed statistics.
5.1Performance on the Four Tasks
In Task 1 (Study Inclusion/Exclusion), shown in Table 2 , Claude3-Opus and GPT-4o perform best in terms of F1 score. Claude3-Opus has higher accuracy and GPT-4o higher weighted accuracy.
In Task 2 (Sentence Retrieval Evaluation Results) 3, GPT-4o has stronger results than both Gemini-1.5 and Claude3-Opus for almost all bias types. However, in Task 3 (Support Judgment Selection Evaluation Results) 4, Claude3-Opus has the strongest results.
Tasks 2 and 3 are critical for evaluating the different capabilities of the LLMs. Task 2 is a longcontext task, requiring retrieval of a non-redundant set of sentences from a 200-sentence paper. Task 3 requires complex reasoning to evaluate the retrieved sentences and identify the implications for risk of bias. Our results demonstrate that GPT-4o performs better in long-context retrieval tasks, while Claude3-Opus exhibits stronger reasoning abilities. Task 4 (Risk Level Determination Evaluation Results) 5 provides an overall assessment of each model's ability to determine the risk level, integrating all information from the previous tasks. The three models show comparable performance in this task.
5.2Performance across Bias Types
The three models exhibit significant variations in performance across different biases due to the varying challenges associated with each bias type. All models have stronger performance at assessing selection bias, as biomedical papers typically detail their randomization processes explicitly.
In contrast, assessing reporting bias requires a comprehensive understanding of the entire paper to determine whether all outcomes specified in the study protocol are reported. Evaluating attrition bias involves extracting data on participant dropout during the trial and assessing how this dropout impacts the study's statistical estimates. The most challenging bias type to assess is "other bias," where the model must grasp all potential biases, such as those stemming from funding sources or bias due to lack of power. Successfully identifying them requires a thorough understanding of all possible biases a study might have, thereby necessitating the treatment of each bias type as a distinct task.
6 Related Work Evaluation of Zero-shot LLM Extraction Kartchner et al. [2023] show that ChatGPT performed inconsistently on various meta-analysis extraction tasks. For instance, while it correctly extracted cancer type roughly 90% of the time, treatment type accuracy fell below 25%. This inconsistency is a considerable motive to more comprehensively understand LLM reliability on various meta-analysis extraction tasks.
Limitation Tasks in Scientific Documents Early efforts in assessing papers' limitations focused on performing summary extraction and evaluating the result using human evaluation Cohan et al. [2018Cohan et al. [ , 2022]], Liu and Shah [2023].
For understanding scientific documents, identifying and retrieving a paper's limitation is essential to understand the utility and viability of claims and evidences presented in the paper. Faizullah et al.
[2024] evaluate fine-tuned embedding models and LLMs' ability to generate limitation using the extracted limitation from the original paper. Marshall et al. [2016] use direct quotes in risk of bias judgment to obtain annotations for review authors' judgments, where exact matching texts are labeled as relevant to a risk of bias. Suster et al. [2023] present a quality assessment task to assess a biomedical paper using the Grading of Recommendation, Assessment, Development, and Evaluation with justifications.
7Conclusion
We have presented RoBBR, a benchmark for assessing the ability to judge risk of bias in biomedical papers. The benchmark is based on gold-standard risk of bias assessments from Cochrane, and uses a human-validated annotation pipeline for scalably constructing task examples. The four benchmark tasks evaluate different skills required for these assessments, including retrieval and reasoning.
Performance of state-of-the-art language models on the benchmark tasks reveals that current models fall significantly short of expert-level performance. By providing a standardized tool for measuring risk of bias judgments, the RoBBR benchmark can help guide the development of AI systems that aim to automatically assess study quality and synthesize scientific evidence. As systems that review the scientific literature are developed, it is important that they can reliably judge the strength of evidence in order to draw sound conclusions. ML systems which perform well on this benchmark could potentially reduce the time required for risk of bias assessments in systematic reviews, which currently averages nearly six hours per study Crocker et al. [2023]. We hope the RoBBR benchmark will spur further research into developing trustworthy systems for this task.
8Limitations
The RoBBR benchmark has several limitations. The source of all bias judgments in the benchmark is Cochrane meta-analyses. While these are considered very high quality meta-analyses among biomedical researchers, the bias judgments that they contain are likely to have occasional errors. In addition, there can be disagreements in judgment among different meta-analysis authors.
The Cochrane risk of bias framework does not necessarily identify all types of bias in research studies.
The framework uses a predefined set of bias categories, which may not be appropriate for all studies.
The benchmark only includes interventional studies, and therefore excludes observational studies. Assessment of risk of bias in observational studies requires a different procedure than for interventional studies. Our benchmark does not measure performance for observational studies.
Only a limited number of prompting strategies were evaluated for the LLMs. It is likely that improved prompting and task presentation would improve performance on the tasks.
Appendix: The RoBBR dataset is made available under CC-BY-NC. A copy of the full license can be found at https://github.com/RoBBR-Benchmark/RoBBR/blob/main/LICENSE.md.
The code used in this paper is released under the MIT License. The MIT License is a permissive open-source license that allows for the free use, modification, and distribution of the code, as long as the original license is included with any derivative work. A copy of the full license can be found at https://github.com/RoBBR-Benchmark/RoBBR/blob/main/LICENSE.md.
A.2 Dataset Hosting, Accessibility and Maintenance
The RoBBR dataset with its meta-data is released and can be accessed freely at (https://github.com/RoBBR-Benchmark/RoBBR/blob/main/LICENSE.md). We commit to regularly maintain the dataset and codebase by incorporating user feedback. We will potentially introduce more features as part of future work in the next version of RoBBR. We confirm that the current version of RoBBR will always remain accessible at the same link.
A.3 Dataset Statistics
See table 6 7 8 9 for detailed test set statistics of the four tasks. objective: The meta-analysis objective.
-search_protocol: search protocol information of the meta-analysis.
-full_paper: The full paper content.
label: One of [included, excluded], showing whether the paper is included or excluded in the meta-analysis. • task2_ROBSR_test.json and task2_ROBSR_dev.json: Test and dev set of task 2 Risk of Bias Sentence Retrieval -paper_doi: The DOI of the paper.
bias: The bias to be considered.
-PICO: PICO of a study in the paper, including Methods, Participants, Intervention, Outcome, and Notes. objective: The meta-analysis objective.
-paper_as_candidate_pool: A tuple of text elements from the paper. Each text element is a sentence, a section title, a table, or a figure caption. aspects: A dictionary that maps aspect id to bias aspect.
-aspect2sentence_indices: a mapping (i.e. dictionary) between aspect id and all sentence indices that independently are source of information for that aspect, as annotated by our pipeline. -sentence_index2aspects: a mapping (i.e. dictionary) between sentence index and all aspect ids that this sentence is the source of information of. -bias_retrieval_at_optimal_evaluation: This is a dictionary that contains the necessary information for evaluating the model's performance on the task Bias Retrieval @Optimal. * optimal: A positive integer, which is the smallest number of sentences needed to cover the largest number of aspects. * one_selection_of_sentences: a list of sentence indices. The list size is the optimal number. The list of sentences cover the largest number of aspects. Note, there are potentially other lists of sentences that has the same size and also cover the largest number of aspects. * covered_aspects: the list of aspects that are covered. In this case, the list of aspects covered is list of all aspects. -bias_retrieval_at_3_evaluation: This is a dictionary that contains the necessary information for evaluating your model's performance on the task Bias Retrieval @3. * one_selection_of_sentences: a list of 3 sentence indices. The list of sentences cover the largest number of aspects that can be covered under the restriction of 3 sentences. Note, there are potentially other lists of sentences that has size 3 and cover the same number of aspects.
\* covered_aspects: the list of aspects that are covered. In this case, this list of aspects may not be all the aspects. Since in the paper, we calculate aspect recall by diving the number of aspects covered by model's retrieved sentences against the total number of aspects, for Bias Retrieval @3, the maximum possible performance is not 100%.
• task3_SJS_test.json and task3_SJS_dev.json: Test and dev set of task 3 Support Judgment Selection
-paper_doi: The DOI of the paper.
bias: The bias to be considered.
-PICO: PICO of a study in the paper, including Methods, Participants, Intervention, Outcome, and Notes. objective: The meta-analysis objective.
-full_paper: The full paper content.
options: The seven options for the multiple choice.
label: The index of the correct option.
• task4_RLD_test.json and task4_RLD_dev.json: Test and dev set of task 4 Risk Level Determination
-paper_doi: The DOI of the paper.
bias: The bias to be considered.
-PICO: PICO of a study in the paper, including Methods, Participants, Intervention, Outcome, and Notes. objective: The meta-analysis objective.
-full_paper: The full paper content.
label: One of [low,high,unclear] , representing the risk level of the bias.
B Annotation Guideline
Below, we show the annotation guideline for aspect mapping of 50 randomly sampled (aspect, full paper) pair. The four annotators form two teams of two persons, all seeing the same annotation guideline.
You have a total of 50 annotation task packets. Each task packet is a docx. file that contains the following information.
• An Aspect (one piece of important information/detail).
• The support judgment and the bias.
• The doi of the paper.
• An indexed list of text elements of the paper (a text element could be a sentence, tables, a figure caption, etc.)
You have to pledge the following conditions are met during annotation for each task packet.
• For words you are not familiar with and believe are important for comprehension, conduct the search to understand its meaning. • For every text element in the list, you have to look at it and read it at least once. • You cannot talk to the other annotator team about anything related to your task, including progress and insights. • You should independently do the task first, and then consult with the other person in your team after completing the task. • You have to take a mandatory 5 minute break after every 1 hour of performing annotation.
• You cannot exceed 8 hours of annotation per day.
Below is the recommended procedure for annotating each packet.
• Read and understand the aspect and the support judgment first. You need to understand the context of the aspect, i.e., the role of the aspect in the support judgment given the bias. You also need to understand why this aspect is important for judging the bias. You can consult with LLM to understand this aspect.
• Decide what details are important in the aspect. Geo, temporal, and numerical data are all important details. • For each text element, you need to decide if a significant amount of important details can be implied from the text element.
- learning examples did not improve agreement rates and tended to cause overfitting. Instead, we found that embedding the same instruction at both the beginning and the end of the prompt effectively helped the model maintain focus on the task of identifying text elements that significantly cover the details specified in an aspect.
We also implemented chain-of-thought reasoning, prompting GPT4 to articulate its thought process following a specific keyword. This approach not only enhanced the quality of the model's reasoning but also stabilized its performance and reduced common-sense errors.
To address the issue of entity forgetting in long-context tasks (a typical paper might contain around 5000 tokens), we employed a sliding window technique. Each window, containing 10 text elements with a 5-element overlap, allowed GPT4 to process and evaluate each text element within a manageable context size. The hyperparameters, window length and overlap size, were optimized using the development set. This overlapping approach ensures that each text element is evaluated twice, significantly reducing the likelihood of false negatives.
Figure 9 illustrates the final prompt template used to match text elements (such as sentences, tables, figure captions, etc.) with the aspects. Each template inputs one aspect, the 10 text elements within a sliding window, and the contextual background (specifically, the summary of the IARC paper from which the aspect was derived).
This methodology not only ensures high fidelity in aspect-text alignment but also leverages the model's capabilities to provide consistent and accurate annotations across extensive text bodies.
Figure 9 : Prompt for generating synthetic options.
C.4 A Motivating Example
Here we provide an example of how we build the bias retrieval task.
Bias
Selective reporting (reporting bias)
Support Judgment For the Bias
The trial authors' original plan was to use Indirect Fluorescent Antibody Test (IFAT) at 6 and 12 months, but due to operational problems, data on IFAT results were not considered valid for the analysis, and serology was not used as a marker of infection in the trial. Problems with serology were poor sensitivity and reproducibility. The authors decided not to use IFAT results in the trial and relied on conversion of the Montenegro skin test (MST) at 18 months of follow-up as the only outcome measure, since no clinical cases of Visceral Leishmaniasis (VL) were detected among the studied population.
Decomposition of Support Judgment into Aspects
• Aspect 1: The trial authors' original plan was to use IFAT test at 6 and 12 months • Aspect 2: Due to operational problems, data on IFAT results were not considered valid for the analysis • Aspect 3: Serology was not used as a marker of infection in the trial • Aspect 4: Problems with serology were poor sensitivity and reproducibility.
• Aspect 5: The authors decided not to use Indirect Fluorescent Antibody Test (IFAT) results in the trial • Aspect 6: The authors relied on conversion of the Montenegro skin test (MST) at 18 months of follow-up as the only outcome measure • Aspect 7: No clinical cases of VL were detected among the studied population
Aspect Filtering
Using prompt 8, Aspect 4 is filtered since it is a commentary of the reviewer.
Mapping Aspects to Sentences in the paper
Utilizing the procedure described in section C.3, we map the remaining 6 aspects to all sentences in the paper Werneck et al. [2014].
We only show sentences from the paper that are matched with at least one aspect.
• Sentence 4: The main outcome is the incidence of infection assessed by the conversion of the Montenegro skin test (MST) after 18 months of follow-up in residents aged ≥1 year with no previous history of visceral leishmaniasis (VL). (Mapped with aspect 6) • Sentence 66: The original plan was to repeat the IFAT test at 6 and 12 months, but due to operational problems, data on IFAT results were not considered valid for the analysis, and serology was not used as a marker of infection in the study. (Mapped with aspect 2, 3, 5, 7) • Sentence 73: In any case, we decided not to use IFAT results in this study and relied on conversion of the MST at 18 months of follow-up as the only outcome measure, since no clinical cases of VL were detected among the studied population. (Mapped with aspect 1, 3, 5, 6) Bias Recall @ Optimal One of our evaluation metric, Bias Recall @ Optimal, measures the the optimal number of sentences to cover all aspects. In this example, we only need two sentences, sentence 66 and sentence 73, to cover all aspects.
D Task 3 Synthetic Option Generation Procedure and Prompt
We first generate three detailed synthetic options that are specifically crafted to imitate the support judgments from other papers concerning the same type of bias, ensuring that they are tailored to be relevant to the specific paper in question while maintaining the foundational reasoning. See Figure 10 for the prompt template. Following this, we condense these detailed options into shorter versions that preserve the original meaning, using prompt 11. To prevent heuristic algorithms from solving the task easily, we randomly select either the long or short version of each synthetic option to include in the multiple choice questions.
For selecting the three options derived from other papers' support judgments within the same bias category, we use prompt 12.
Finally, once the six incorrect options are constructed, we conduct a manual review with the help of prompt 13 on all data points. This review ensures that the options are incorrect and not false negatives.
E Experiment Details
Due to budget limit, we only evaluate the LLMs on a subset of our test set. We provide this subset in our GitHub.
E.1 Prompt Optimization
All evaluation prompts are optimized using disjoint development set for each task. Each prompt includes specialized instructions designed to elicit chain-of-thought reasoning, thereby enhancing the models' reasoning abilities. These instructions are repeated twice: once at the beginning and once at the end of the prompt, ensuring that the models retain the instructions even after processing the entire paper. Empirical evidence shows that few-shot in-context learning does not improve model performance. Consequently, evaluation prompts do not incorporate in-context learning. It is important to note that prompt optimization is not tailored to any specific large language model (LLM); instead, the aggregated performance across all models is used to guide the optimization of evaluation prompts. See Figure 15 for prompt to evaluate Bias Retrieval (Task 2). We use an additional multi-turn prompt 16 when the model outputs more than required.
C.2 Aspect Filtering Prompt
See Figure 7 for prompt to filter negations.
See Figure 8 for prompt to filter non-specific commentaries.
C.3 Aspect-to-Sentence Mapping Prompt and Optimization
To enhance the agreement between human annotators and the GPT4-0125-turbo model, we developed and tested various prompt engineering strategies on a development set consisting of 30 unique (aspect, paper) pairs. It is important to note that no paper or aspect from the development set overlapped with the 50 tasks in the hypothesis testing set. GPT4-0125-preview was chosen due to its robust instruction-following capabilities and our familiarity with its performance across different settings. Our experimentation revealed that using in-context E.2.3 Prompt to Evaluate Support Judgment Selection (Task 3) See Figure 17 for prompt to evaluate Support Judgment Selection (Task 3).
E.3 Results with Standard Error
We provide bootstrapped standard errors for the experimental results 11,12,13,14. Bias Recall@Optimal Claude3-Opus 39.0 ± 2.4 55.4 ± 4.0 22.0 ± 4.7 57.3 ± 6.39 46.4 ± 6.5 11.0 ± 4.9 47.9 ± 12.3 13.9 ± 8.3 4.4 ± 2.9 Gemini-1.5-Pro 41.1 ± 2.5 57.2 ± 4.0 28.7 ± 5.2
49.1 ± 6.4 48.8 ± 6.3 23.3 ± 6.4 54.2 ± 11.4 13.9
44.8 ± 2.5 60.3 ± 4.0 28.6 ± 5.0 64.6 ± 5.5 48.7 ± 6.4 32.9 ± 7.3 28.7 ± 9.7 28.0 ± 14.1 2.3 ± 2.2
Bias Recall@3 Claude3-Opus 52.3 ± 2.5 67.9 ± 3.8 40.2 ± 5.8 63.1 ± 5.9 55.9 ± 6.6 25.5 ± 7.1 55.9 ± 12.3 64.6 ± 16.9 19.7 ± 7.8 Gemini-1.5-Pro 56.1 ± 2.4 69.7 ± 3.8 44. 48.0 ± 6.9 46.5 ± 6.9 44.2 ± 7.0 50.0 ± 17.9 0.0 ± 0.0 Gemini-1.5-Pro 49.3 ± 3.0 57.1 ± 5.3 59.9 ± 6.5 46.4 ± 7.0 42.8 ± 6.8 40.2 ± 7.0 49.3 ± 17.7 16.6 ± 15.5 GPT-4o 45.8 ± 2.9 52.3 ± 5.4 63.1 ± 6.2 42.5 ± 6.7 37.7 ± 6.5 28.0 ± 6.2 74.5 ± 15.7 0.0 ± 0.0
F Full Example
We provide the full version of the example presented in the main paper. The example is shown in its entirety here, whereas in the main paper it was shortened to save space.
F.1 Search Protocol Gonzalez et al. [2015] Types of Studies Randomized controlled trials (RCTs)
Types of Participants
People living in leishmaniasis endemic regions.
Types of Interventions
Any intervention that aims to reduce leishmaniasis incidence through vector or reservoir control.
Types of Outcome Measures
People developing CL or VL infections. Estimates of the vector density measured by an appropriate technique (adult sandfly density estimated by counts of vectors either landing on exposed body parts of humans acting as baits or collected resting inside buildings, for example, on walls). Number of participants with positive immunological or biochemical tests that detect contact with the parasite (for example, leishmanin skin test conversion rates or lymphocyte proliferation rates, or both). Adverse effects on people. Adherence to control measures; for example, the extent to which specified intervention components were delivered as prescribed. Measures of environmental impact (assessment of the possible impact -positive or negative -that the interventions may have on the natural environment) or sustainability (assessment of the ability to change biological and human processes, functions, biodiversity and productivity), or both.
F.2 Objective of the Meta-Analysis Gonzalez et al. [2015]
To assess the effects of vector and reservoir control interventions for cutaneous and for visceral leishmaniasis.
F.3 Full Paper
See Werneck et al. for full paper content. Higgins and Green [2011] Bias Name selective reporting (reporting bias)
Bias Definition
Reporting bias due to selective outcome reporting.
Methods
Trial design: Cluster-RCT.
Failed to load PDF:
StripeM-Inner
Paper sources
Abstract screening pilot
Abstract screening results
Extraction pilot
Extraction results
Research report
Modify setup