# Evaluating Elicit’s Paper Search

\
\
Pradyumna Prasad\
\
Member of Evals Staff

###### Jul 17, 2026

###### 10 min read

On BioASQ, a widely used biomedical search benchmark, Elicit found more of the papers experts used to answer their questions than any of the five other search systems we tested. Across questions, Elicit found 60.3% of each question’s gold-standard papers on average in the first 50 results, compared with 47.4% for the next-best system. Elicit also had the highest recall at every result count tested, from 10 to 200.

## Motivation

We recently [launched](/content/blog/the-elicit-api-and-mcp-powering-autonomous-research-engines/index.html) the Elicit API and MCP, bringing Elicit’s building blocks to you. Our ambition is to be the most comprehensive scientific search engine. When called by an agent, Elicit needs to find and return the best evidence so that the agent can investigate the mechanism behind a drug target or a compound’s safety signal. So, we set out to measure our search performance using two sources: BioASQ, a widely used benchmark for biomedical question answering and search, and a sample of anonymized queries from Elicit users.

Elicit achieved the highest recall on BioASQ, finding more of the relevant papers than any other search system we tested. On a separate sample of 399 anonymized Elicit searches, papers from the updated search were also preferred over those from the version it replaced.

## What we evaluated

BioASQ is an established benchmark for biomedical question answering and search. Its dataset and construction were described in a [peer-reviewed paper in _Scientific Data_,](https://www.nature.com/articles/s41597-023-02068-4) a Nature Portfolio journal. A 500-question version of BioASQ is also included in BEIR, a widely used benchmark suite for comparing information retrieval systems. We use Phase A of Task B where experts write questions which reflect their information needs. For each question, the expert identifies a set of papers sufficient to support an answer, which becomes our gold standard retrieval set.

We used BioASQ because the questions are drawn from the real world needs of practicing biomedical experts who also identify papers they consider relevant and sufficient to answer each question. We used BioASQ to compare Elicit’s semantic search API with other commercial and academic search APIs.

We also wanted to ensure that improvements to Elicit’s search engine held up across the broader range of questions our users ask. Alongside BioASQ, we evaluated 399 anonymized searches: 300 from Elicit’s public API and 99 from the Find Papers workflow.

## How we evaluated

### The BioASQ benchmark

We combined all questions from 12 years of BioASQ’s official Golden Enriched test sets, which pair each question with papers biomedical experts considered sufficient to answer it. To recreate the literature available when each question was evaluated, we excluded papers published in later years when making API calls to search engines. Together, the test set contained 5,486 questions, with an average of 10.2 papers in the gold standard per question and a median of 6. The section “Benchmark construction and deduplication” in the appendix has more details.

One BioASQ question asks, “Are astronauts in higher risk [sic] for developing cancer?”

The gold standard for this question contains two papers:

- [Chancellor et al. (2018), _Limitations in predicting the space radiation health risk for exploration astronauts_ (PMID 29644336)](https://pubmed.ncbi.nlm.nih.gov/29644336/)
- [Elgart et al. (2018), _Radiation Exposure and Mortality from Cardiovascular Disease and Cancer in Early NASA Astronauts_ (PMID 29855508)](https://pubmed.ncbi.nlm.nih.gov/29855508/)

### How we queried each system

For semantic search providers, we submitted the original BioASQ question. For keyword search providers, we used Claude Opus 4.8 to generate a separate query tailored to each provider’s syntax. For example, the astronaut question above became `astronauts cancer risk space radiation` for Google Scholar. When Claude returned a safety refusal, GPT 5.6 Sol generated the query using the same prompt. We also used GPT 5.6 Sol to make minimal repairs to queries containing unsupported wildcard syntax. The appendix gives the full counts and describes an additional formatting fix for Semantic Scholar.

We asked each provider for up to 200 results. Consensus returned at most 20, and OpenAlex’s semantic search at most 50.

### How we scored the results

We wanted to measure whether each search engine returned the papers BioASQ experts identified as sufficient to answer each question. We used recall, which measures the percentage of those papers found by the search engine. We matched results to BioASQ’s gold standard using PMIDs where available, then DOIs, and finally titles. The appendix gives the full breakdown by search system.

To see how performance changed as we considered more results, we measured recall at several cutoffs. At a cutoff of 20, for example, only papers among the first 20 results in the API’s ranking count. We calculated recall for each question, then averaged across questions so each question contributed equally to the final score.

We chose recall rather than precision or mean average precision because BioASQ’s gold standard is not exhaustive. Experts identified enough papers to answer each question, not every relevant paper. A search engine may return a useful paper that BioASQ’s experts did not include. Precision and mean average precision would count that paper as irrelevant, penalizing the search engine because the label is missing rather than because the result is poor. Recall does not directly count unlabeled papers as errors. It measures whether the search engine found the known gold-standard papers.

Each provider’s average includes only requests recorded as successful. Elicit and Consensus completed all 5,486 requests, while other providers failed on some questions. We excluded those failures rather than assigning them zero recall and report the number included for each provider. As a result, the averages are not always based on exactly the same questions.

## Results

Elicit’s search API has the highest recall of any search API we tested. Below is a table showing the recall of each search API at increasing cutoffs (10, 20, 50, 100, and 200 results). We compute the recall of every question and report the average of averages (macro-recall).

| System | @10 | @20 | @50 | @100 | @200 | n |
| --- | --- | --- | --- | --- | --- | --- |
| **Elicit** | **0.390** | **0.483** | **0.603** | **0.670** | **0.708** | 5,486 |
| Consensus | 0.369 | 0.434 | * * * | — | — | 5,486 |
| Semantic Scholar | 0.254 | 0.311 | 0.397 | 0.465 | 0.535 | 5,485 |
| OpenAlex (keyword) | 0.286 | 0.364 | 0.474 | 0.557 | 0.630 | 5,477 |
| OpenAlex (semantic) | 0.183 | 0.258 | 0.299 | — | — | 4,891 |
| Google Scholar | 0.287 | 0.333 | 0.406 | 0.458 | 0.502 | 5,451 |

Elicit achieved the highest recall of all six search systems we tested at every result depth, from 10 to 200 results. Higher recall means that, for the same biomedical question and number of search results, an agent using Elicit would retrieve more of BioASQ’s gold-standard papers on average. The agent could then build its answer from a more complete evidence base. Elicit’s lead persisted at every cutoff. The largest gap was at 50 results, where Elicit achieved 60.3% average recall, compared with 47.4% for the next-best workflow, OpenAlex keyword search. That is a 13 percentage-point advantage, or 27% higher recall.

## Beyond BioASQ

The BioASQ results above are for an updated version of Elicit search. We wanted to make sure that its stronger performance on BioASQ did not come at the expense of relevance on the searches people actually run in Elicit.

We compared the updated search with the version it replaced on 399 searches from Elicit users. Before evaluating them, we removed all personally identifiable information. The sample included 300 searches from Elicit’s public API and 99 from the Find Papers workflow.

The searches covered a broader range of fields and formats than BioASQ. API searches included short topic descriptions, such as an overview of CAR-T therapy or clean-energy industrial policy and supply chains. Find Papers searches were more often complete research questions, including how PFAS behaves during plasma water treatment.

For each search, we selected six pairs of papers from the two versions’ top 10 results, with one paper from each version in every pair. GPT 5.5 judged which paper was more relevant without knowing which version returned it or where it ranked. We wrote the judging instructions and manually reviewed a sample of its judgments as a quality check.

The updated search received a 54.1% preference score, where 50% indicates no overall preference between the versions and ties count as half. The 95% confidence interval was 51.7% to 56.4%. The result was above 50% for both sources: 53.2% for public API searches and 56.9% for Find Papers searches.

The updated search performed better not only on BioASQ, but also across this separate sample of real Elicit searches.

## Limitations

BioASQ’s sufficient papers are not an exhaustive list of every relevant paper. Recall therefore measures how often each system retrieved the papers BioASQ’s experts identified as sufficient, rather than every paper that could help answer the question. BioASQ is also biomedical and draws its papers from PubMed, so this evaluation does not establish that the same ordering holds across every scientific field.

The comparison is a snapshot of each API when we ran it. Other providers failed on some questions, and we calculated their averages only over successful requests rather than assigning those failures zero recall. This may make their reported averages higher than if failures counted as misses. Our evaluation on Elicit searches used GPT 5.5 judgments of sampled paper pairs rather than judgments from researchers or a measure of downstream task performance. A public, cross-domain benchmark would make it easier to test scientific search across more fields and research tasks.

## Conclusion

Across 5,486 BioASQ questions, Elicit achieved the highest recall of all six systems at every result depth tested. The updated search was also preferred over the version it replaced on a separate sample of 399 real Elicit searches. For researchers and agents, higher recall puts more of the evidence needed to answer a question within the same number of search results.

## Reproduce this evaluation

The [evaluation repository](https://github.com/elicit/elicit-paper-search-eval) contains the code and data needed to reproduce the Elicit results, including BioASQ inputs, keyword-generation prompts and queries, configuration, Elicit outputs, and data attribution. It also includes fetchers for running the evaluation against the other search APIs. We do not include competitor result payloads.