Elicit: Comparative Analysis of Academic Databases

Comparative Analysis of Academic Databases

What is the comparative coverage and retrieval performance of multidisciplinary academic databases (OpenAlex, Semantic Scholar, Dimensions) versus traditional subscription databases (Web of Science, Scopus, EMBASE) across different academic disciplines?

Multidisciplinary databases such as OpenAlex demonstrate higher retrieval rates and better coverage of open access and social sciences content compared to traditional databases, whereas subscription services like Scopus and Web of Science excel in metadata accuracy and citation tracking for STEM disciplines.

Abstract

Multidisciplinary academic databases tend to retrieve a broader range of records than traditional subscription services. Several studies report that OpenAlex, for example, returns up to 100% more records than Scopus and 27.9–29.3% more than Web of Science in specific comparisons. Open sources consistently capture more open access, non‐English, and social sciences and humanities content, as highlighted by studies that observed better coverage in the Global South and for emerging fields. In contrast, traditional databases such as Web of Science, Scopus, and EMBASE consistently deliver higher metadata quality, more accurate document type classification, and more reliable citation tracking, particularly in STEM and health-related literature.

Methods

We analyzed 23 sources from an initial pool of 500, using 7 screening criteria. Each paper was reviewed for 8 key aspects that mattered most to the research question.

Records from Elicit search

n = 0
Papers screened using: Database Comparison, Performance Metrics, Study Design, Disciplinary Scope, Database Focus, Empirical Comparison, Publication Type
n = 500
Papers screened out
n = 477
Papers included for extraction
n = 23

Paper search

We performed a semantic search across over 138 million academic papers from the Elicit search engine, which includes all of Semantic Scholar and OpenAlex.

We ran this query: “What is the comparative coverage and retrieval performance of multidisciplinary academic databases (OpenAlex, Semantic Scholar, Dimensions) versus traditional subscription databases (Web of Science, Scopus, EMBASE) across different academic disciplines? Focus on papers after 2024 in your analysis.”

We retrieved 500 papers most relevant to the query for screening.

Screening

We screened in sources based on their abstracts that met these criteria:

Database Comparison: Does this study directly compare at least one multidisciplinary academic database (OpenAlex, Semantic Scholar, Dimensions) with at least one traditional subscription database (Web of Science, Scopus, EMBASE)?
Performance Metrics: Does this study measure coverage metrics (e.g., number of indexed publications, disciplinary breadth, citation coverage) or retrieval performance metrics (e.g., recall, precision, search effectiveness)?
Study Design: Is this an empirical study (comparative analysis, bibliometric study, database evaluation study, systematic review, or meta-analysis) that provides quantitative evidence about database performance?
Disciplinary Scope: Does this study analyze database performance across multiple academic disciplines or provide discipline-specific comparisons?
Database Focus: Does this study include the specified multidisciplinary databases (OpenAlex, Semantic Scholar, Dimensions) or traditional databases (Web of Science, Scopus, EMBASE) rather than focusing solely on specialized or discipline-specific databases?
Empirical Comparison: Does this study provide empirical performance comparisons rather than only describing database features, interfaces, or search functionalities?
Publication Type: Is this a research article with original empirical data rather than an opinion piece, editorial, or commentary article?

We considered all screening questions together and made a holistic judgement about whether to screen in each paper.

Data extraction

We asked a large language model to extract each data column below from each paper. We gave the model the extraction instructions shown below for each column.

Database Comparison: Identify which databases were compared in the study, specifying:
- Multidisciplinary databases tested (OpenAlex, Semantic Scholar, Dimensions)
- Traditional subscription databases tested (Web of Science, Scopus, EMBASE, PubMed)
- Any other databases included in the comparison
- Whether the comparison was direct head-to-head or part of a larger evaluation
Study Methodology: Extract the methodological approach used to assess database performance including:
- Type of comparison (coverage analysis, search performance, metadata quality, etc.)
- Sample selection method and size (e.g., random DOI sample, specific query results)
- Search strategies or identification methods used
- Metrics employed (percentage coverage, recall, precision, overlap coefficients)
- Data collection period and search dates
- Quality assessment criteria if applicable
Disciplines Examined: Document the academic disciplines or subject areas analyzed:
- Specific fields mentioned (e.g., chemistry, immunology, social sciences, humanities)
- Disciplinary classification system used (if any)
- Whether analysis was multidisciplinary or focused on specific domains
- Any discipline-specific findings or patterns noted
Time Period: Extract the temporal scope of publications analyzed:
- Publication year range of papers included in the analysis
- Whether focus was on recent publications (2024+) as specified in research question
- Any temporal trends or changes in database performance noted
- Cut-off dates for data collection
Coverage Performance: Extract all quantitative and qualitative findings about database coverage including:
- Specific coverage percentages or numbers for each database
- Relative performance comparisons (e.g., ‘OpenAlex includes 27.9% more records than Web of Science’)
- Coverage gaps or missing content identified
- Document type coverage differences (articles, reviews, books, etc.)
- Geographic or linguistic coverage variations
Retrieval Performance: Document findings about search and retrieval capabilities including:
- Search accuracy metrics (precision, recall, F-scores)
- Metadata quality assessments (author disambiguation, affiliation matching)
- Classification accuracy (document types, subject categories)
- False positive/negative rates
- Retrieval completeness for systematic searches
Disciplinary Differences: Extract findings about how database performance varies across disciplines:
- Which databases performed better in specific fields
- Disciplinary bias or gaps identified (e.g., better coverage of STEM vs humanities)
- Field-specific coverage patterns
- Subject classification accuracy differences
- Discipline-related limitations noted by authors
Contextual Factors: Identify factors that may affect the interpretation or generalizability of findings:
- Geographic scope and limitations
- Language bias or multilingual coverage issues
- Open access vs subscription publication patterns
- Institutional or publisher-specific coverage patterns
- Data quality issues or limitations acknowledged
- Temporal limitations or currency of database updates

Results

Characteristics of Included Studies

Study	Database(s) Studied	Performance Metrics Evaluated	Sample Size	Key Focus Area	Full text retrieved
Thelwall and Jiang, 2025	OpenAlex, Scopus	Coverage, citation counts, document type/coverage, subject classification, citation indicators	28.6 million articles, 97,816 Research Excellence Framework (REF) 2021 articles	Comparative coverage and citation indicator performance	Yes
Delgado-Quirós and Ortega, 2025	Dimensions, OpenAlex, Semantic Scholar, Microsoft Academic, Scilit, The Lens	Coverage, metadata quality, author/affiliation/venue detection	>115,000 Digital Object Identifiers (DOIs)	Research entity information and coverage	No
Fuente-Gutiérrez and Kippes, 2025	OpenAlex, Web of Science, Scopus	Coverage, disciplinary representation, growth rates	47,354 articles	Co-authorship and coverage in Argentina-Spain collaboration	Yes
Mokhnacheva, 2025	OpenAlex, The Lens, Web of Science Core Collection	Coverage, representation of Russian publications	No mention found	Russian publication coverage in global context	No
Rajit et al., 2025	OpenAlex, Semantic Scholar, Embase, PubMed	Coverage, overlap, risk of bias	2023 polycystic ovary syndrome (PCOS) guideline articles	Automated evidence retrieval for living guidelines	No
Mongeon et al., 2025	OpenAlex, Web of Science	Metadata quality, document type accuracy	7.6 million Web of Science records, 6.6 million matched in OpenAlex	Document type discrepancies	Yes
Simard et al., 2025	OpenAlex, Web of Science, Scopus, Directory of Open Access Journals (DOAJ)	Coverage, Gini coefficients, discipline/country/language	No mention found	Open access (OA) journal coverage (gold/diamond)	Yes
Li et al., 2025	OpenAlex, Web of Science	OA book coverage, growth rates, discipline	255,810 (OpenAlex), 8,713 (Web of Science)	OA academic book trends	Yes
Mezquita et al., 2025	OpenAlex, Scopus, Web of Science	Coverage, citation counts	No mention found	Nursing literature coverage	No
Jahn, 2025	OpenAlex (hoaddata), Scopus, Web of Science, Crossref, Journal Citation Tracker (JCT)	Coverage, metadata quality, OA indicators	>13,000 hybrid journals	Transformative agreement impact on hybrid OA	Yes

Coverage Performance

Study	Database	Coverage Rate (%)	Overlap with Other Databases (%)	Disciplinary Variations
Thelwall and Jiang, 2025	OpenAlex	>100% of Scopus (2x articles)	No mention found	Better for Global South, more books, more OA/data journals
Delgado-Quirós and Ortega, 2025	OpenAlex	89.9% authors with one publication	No mention found	Problems with books/book chapters
Fuente-Gutiérrez and Kippes, 2025	OpenAlex	27.9% more than Web of Science, 29.3% more than Scopus	No mention found	Better in social sciences and humanities, less bias to Spanish
Mokhnacheva, 2025	OpenAlex	No mention found	Correlates with Web of Science Core Collection	Better for Russian publications than The Lens
Rajit et al., 2025	OpenAlex	98.6%	90.5% (all 4 databases)	Health sciences (PCOS)
Mongeon et al., 2025	OpenAlex	No mention found	No mention found	Broader in humanities, non-English, Global South
Simard et al., 2025	OpenAlex	Most DOAJ journals	16% (diamond, all 3 databases)	Best for social sciences and humanities, multilingual journals
Li et al., 2025	OpenAlex	9.46% OA books	No mention found	Better in emerging fields, OA books
Mezquita et al., 2025	OpenAlex	Nearly all nursing journals	No mention found	More non-Western sources
Jahn, 2025	OpenAlex	No mention found	No mention found	Limited corresponding author data

Metadata Quality and Retrieval Accuracy

OpenAlex:
- Metadata quality is variable, with issues in author disambiguation, affiliation matching, and document type classification.
- One large-scale study found that 97% of document type misclassification errors occurred in OpenAlex.
- Author and affiliation metadata are generally available for most records, but missing for a notable minority, especially in non-English and book content.
- Retrieval completeness for systematic reviews is high (up to 98%), but precision and recall can vary depending on the search strategy and discipline.
- Automated citation searching using OpenAlex yields high precision but lower recall compared to traditional methods.
Semantic Scholar and Dimensions:
- Show strong performance in author disambiguation and thematic classification.
- Coverage of books and book chapters remains problematic.
Traditional Databases (Web of Science, Scopus):
- Maintain higher metadata quality and more accurate document type classification.
- More limited in coverage, especially for open access, non-English, and non-article content.

Citation Tracking Performance

OpenAlex:
- Provides more citations than Scopus and Web of Science in several studies.
- Missing citations and incomplete reference lists are still observed, especially for recent publications and non-English content.
- Cross-referencing multiple databases improves citation tracking completeness.
Traditional Databases (Web of Science, Scopus):
- Fewer missing citations, but more limited coverage can result in missed citations for non-mainstream or open access publications.
Automated Citation Searching:
- Using OpenAlex and Semantic Scholar, automated citation searching outperforms traditional methods in precision and F1 score (harmonic mean of precision and recall), but underperforms in recall, particularly for grey literature and clinical trial reports.