Evaluating Elicit’s Systematic Literature Review Capabilities - Elicit

Evaluating Elicit’s Systematic Literature Review Capabilities

May 6, 2026

Last year, Elicit launched an end-to-end systematic review workflow as part of our mission to radically increase good reasoning.

We've added new features along the way and also spent considerable time upgrading the models behind the system — how it searches, how it applies screening criteria, how it extracts data, and how it reasons through each decision. Today we're publishing how we evaluated whether that work translated into the rigor systematic reviews demand.

Data collection

In order to rigorously evaluate Elicit’s performance across all phases, we knew we had to collect a large set of gold standard systematic reviews. We built a ground truth dataset from open-access Cochrane reviews — Cochrane reviews are widely considered the gold standard for evidence synthesis in health, medicine, and beyond. We sampled 1,000 reviews, balanced by round-robin sampling across 12 MeSH areas (Hematologic, Neurology, Digestive, Immunologic, Infectious Disease, Musculoskeletal, Eye, Endocrine/Metabolic, Respiratory, Neoplasms, Cardiovascular, and Kidney), yielding roughly 83–84 reviews per area. After dropping duplicates, we were left with 994 unique reviews covering 38,493 study records.

A study record is the entry for the study in the review’s characteristics of included studies or characteristics of excluded studies section, usually named with a short label like “Smith 2018.” It is not the same thing as a paper. A single study record can point to one paper or to several papers, e.g. the main trial report, a conference abstract, and a follow-up analysis. We resolved each study record to a single representative paper where we could do so confidently and dropped those studies where we could not.

We first matched each study record to the corresponding entry in the references section. This worked for almost every listed study: 99.9% had at least one entry in the references section, and 87.7% could be matched to one unambiguous paper. However, only 58.4% of listed studies had a DOI present in the references section, so DOI-based search scoring necessarily covers a subset of the full dataset. We use that subset because DOI matching gives us a conservative, high-confidence way to decide whether Elicit found the same paper.

Matching step Share of listed studies
Had at least one bibliography entry 99.9%
Matched to one unambiguous paper 87.7%
Had a DOI 58.4%
Matched to one unambiguous paper with a DOI 56.7%
Stage Reviews
Unique reviews in dataset 994
With at least one parsed study record 938
With at least one resolvable canonical DOI 888

Our evaluations of search, abstract screening, and extraction each apply their own filters on top of this shared dataset of 888 studies / papers. Full-text screening uses a separately constructed Cochrane dataset, built with the same methodology, because that evaluation required full-text annotations and fetchable PDFs.

Search

Using only the review title as the query, Elicit finds 95.0% of included studies

When we evaluate Elicit Search, we measure: given a systematic review question, can Elicit find all the papers relevant to that review? Elicit allows users to search over 138 million papers.

What we are benchmarking here is our semantic search. In semantic search, a query is passed to an embedding model, which encodes it into a vector representation. That vector is then matched against Elicit's corpus to find the most relevant results. We don’t evaluate keyword search as it is fully deterministic and the output depends on the keywords the user inputs.

We use the review title as the Elicit query and nothing else. This is a conservative way to evaluate Elicit. In practice, a user running a systematic review on Elicit would provide substantially more context than a title — they can ask an in-depth research question, add information about the protocol, and run multiple search strategies mixing keyword and semantic approaches.

What does "relevant" mean here? We consider all papers that the review lists as both included studies and excluded studies as these are studies that are either included in the final review, or in the judgement of the review’s authors would have been retrieved by the search or otherwise warranted full-text assessment.

Results

The graph at the beginning of this section shows the mean percentage of included papers retrieved per review. This graph shows the percentage of all papers (included and excluded) retrieved:

Additionally, 79.8% of reviews had 100% of their included studies found by our search, and an additional 11.6% of them had over 80% of their included studies found by their search. For about 2% of reviews we were unable to find more than 50% of papers.

Review recall threshold % of reviews at or above threshold
100% 79.8%
90% 85.7%
80% 91.4%
70% 93.7%
60% 96.5%
50% 98.2%
40% 98.5%
30% 98.5%
20% 99.5%
10% 99.5%
0% 100.0%

Abstract Screening

Elicit achieved 96.9% sensitivity on abstract screening with 92.5% specificity

After search retrieves candidate papers, the next phase is abstract screening: given a review's eligibility criteria and a candidate paper's title and abstract, does Elicit correctly advance this paper to full-text review?

Starting from the shared dataset, we kept only reviews whose search strategies reported PubMed and MEDLINE search queries. We used a language model to translate each review's search syntax into PubMed syntax (if necessary). Then we ran each search in PubMed ESearch. We filtered out from the results all papers published after the review was published.

Then, for all of the reviews where a PubMed search result existed, we only kept those where the search results count was within 50% to 200% of the original PRISMA search results count. This was to avoid outliers where the PubMed search would be unrepresentative of the SLR’s actual search. From this subset, we further excluded reviews with fewer than five papers included or excluded. To create the negative set (papers that should be screened out), we randomly sampled 50 papers retrieved from the PubMed search strategy that weren’t in either the included or excluded tables in the review. This left us with 108 reviews.

On manual inspection, the excluded studies table was not a consistent proxy for abstract screening positives. Many papers in the excluded section violated the abstract screening criteria, and so we restricted the positive set to final-included studies only. We include examples in Appendix 7.

After filtering out papers whose abstracts were unavailable, 931 positives and 5,162 negatives made the final dataset. Our model makes a yes, no, or maybe decision on each screening criterion. If the paper is a "yes" or "maybe" on all criteria then it is screened in; if there are any "no"s it is screened out. We screen in maybes because it is much worse to miss a paper that should be included than it is to erroneously screen in a paper that will need to be screened out later at full-text screening.

Metric Value
Recall / sensitivity 96.89%
Specificity 92.54%
Precision 70.09%
Accuracy 93.21%

Elicit achieved 96.89% recall, 92.54% specificity, and 93.21% accuracy. In a study of human screening for pharmacological and public health reviews, Gartlehner et al. 2020 found that single-reviewer abstract screening achieved 86.6% sensitivity and dual-reviewer screening achieved 97.5%. Our dataset and evaluation approach are different from Gartlehner et al. 2020, and so we should approach comparisons with caution. Nevertheless, it is interesting to observe that our sensitivity exceeds their single-reviewer performance and approaches dual-reviewer screening. Specificity is harder to compare directly. Gartlehner et al.'s negatives are expert judgments from a systematic review. With that in mind, our specificity of 92.5% exceeds their 68.7% dual reviewer specificity.

Full-Text Screening

99.5% paper-level recall with 94.8% per-criterion accuracy

Full-text screening is the final gate before a paper enters the review. Abstract screening reviews titles and abstracts, full-text screening reads the entire paper and applies more detailed versions of the review's eligibility criteria.

Our full-text screening evaluation uses a separately constructed dataset built with the same round-robin sampling methodology across the same 12 MeSH areas, albeit with a smaller sample size of 100 reviews.

We then used reviews where the excluded-studies table appeared to represent papers rejected after full-text review, rather than papers rejected during title/abstract screening or papers listed for some other reason. After this check, the constructed label dataset contained 4,491 study rows across 92 reviews. To run the evaluation, we kept papers where we could resolve a study DOI, retrieve the paper full text, and parse the PDF. This produced 1,358 criterion-level screening examples across 400 papers and 76 reviews. We then dropped papers whose parsed source had no abstract text, leaving the final eval dataset: 1,271 criterion-level screening judgements across 377 papers and 74 reviews. The evaluation set is therefore much smaller than we would like, but it reflects a genuine constraint of working with full-text data.

While running the evaluation across multiple frontier and open-source models, we found 138 (10.9% of the dataset) screening judgements that all models consistently judged the opposite of the judgement in the Cochrane review. We manually examined roughly 30 of these and found they fell into two categories: cases where the review appeared to apply different screening criteria than the stated ones, and cases where we believe the reviewer was looking at a different paper than what the model saw as there are multiple papers that correspond to a study and we may have picked up on a different version. We excluded these from the final dataset.

Much as for abstract screening, we screen in papers judged as a “yes” or “maybe” on all criteria, and screen out studies judged as a “no” on at least one criterion.

At the paper level, sensitivity and specificity were:

Metric Value
Recall / sensitivity 99.5%
Specificity 70.1%
Precision 81.2%
Accuracy 86.7%

At the per-criterion level, our model evaluated 1,133 criterion-paper pairs with an overall accuracy of 94.8%:

Extraction

95.6% correct on Methods, Participants, and Interventions

The final stage of a systematic review is extraction: pulling structured information from each included paper. Cochrane reviewers do this manually, filling in "Characteristics of included studies" tables with fields like Methods, Participants, and Interventions for every paper in the review.

We used our 994 SLR corpus to create the evaluation dataset for extraction tasks. We first filtered for reviews that have a “Characteristics of included studies” table, leaving us with 873 reviews, and then filtered for reviews whose body had a section titled “Criteria for considering studies” leaving us with 430 reviews. We then kept reviews that weren’t diagnostic test accuracy reviews (which evaluate medical tests rather than interventions, and use a different data format), dropping us down to 340 reviews with 5,308 studies in the aggregate. Overall, these studies had 18,924 extraction answers. Then we filtered for studies which had a canonical DOI (as explained above) and were left with only 3,241 studies in 320 reviews. We then attempted to fetch PDFs from open-access sources, which succeeded for 198 studies across 98 reviews and 769 extraction answers.

When we manually looked at questions we realized that many questions were too generic when compared to the gold standards. A generic question like "who were the participants?" is not a faithful test of extraction quality. Each review defines its own data extraction schema: one might care about disease severity and sample size, another about age distribution and comorbidities. To reconstruct these review-specific questions, we asked a language model to look at the answers and infer what the review's extraction question was based on the answers.

On this benchmark, Elicit gets 95.6% of extraction tasks correct, as shown in the graph at the start of this section.

Each model answer was graded by a separate language model that saw the question, the Cochrane gold answer, the model answer, and the paper text.

We randomly selected 25 of the answers that were graded correct to review by hand. We agreed with the grade for all but one of the answers (which was an ambiguous case). We also reviewed by hand all 17 answers that were graded wrong, and agreed with the grade for 14/17.

Within the 42 total answers that we reviewed, there were 7 where it seemed like the data extraction in the Cochrane review may have been wrong (though it’s hard to tell without talking to review and study authors). There were another 5 where it seemed impossible to answer correctly based solely on the paper provided to the model, and the model would need additional or alternative papers about the study in question.

Conclusion

We found that Elicit achieved greater than 95% recall or accuracy at each stage of systematic review —

We believe that Elicit is likely to perform well on a wide variety of systematic reviews.

What we learned about systematic reviews along the way

We (re-)learned that systematic reviews are complex and diverse, and that it isn’t possible to precisely reproduce a review just based on the information in the review itself. Cochrane reviews are known as the gold standard for evidence synthesis in health and beyond, and even Cochrane reviews contain heterogeneous excluded-studies tables, occasional screening and extraction errors, study records that resolve ambiguously across multiple papers, and screening and extraction methodology that is not explained in full detail.

Limitations