Elicit: Benchmarks for Scientific Literature Review Performance (public)
Benchmarks for Scientific Literature Review Performance (public)
Research reportView only
Create alertChat
September 12, 2025
Which benchmarks measure performance on scientific literature review tasks?
Benchmarks for scientific literature review tasks include datasets like LitSearch, SciReviewGen, SWIFT‐Review, and RoBBR that measure performance across retrieval, generation, screening, extraction, and meta-evaluation areas.
Abstract
Benchmarks for scientific literature review tasks address five primary areas: retrieval, generation, screening, extraction, and meta-evaluation. Seventeen studies target retrieval using datasets such as LitSearch, CLEF TAR, CSMeD, SIGIR2017-PICO-Collection, FASS‐BSLR, RELISH, and ResearchArena. These studies report performance using metrics such as recall, precision, Mean Average Precision, normalized Discounted Cumulative Gain, and Work Saved over Sampling. Eleven studies focus on generation tasks—including review, table, or abstract writing—with benchmarks like SciReviewGen, ARXIV2TABLE, MSLR2022, ScholarQABench, and Manalyzer; performance is measured by ROUGE scores, hallucination rates, semantic coverage, and human ratings. Nine studies on screening tasks (using tools such as SWIFT‐Review and LGAR) report measures such as precision at high recall levels and WSS@95, while five studies on extraction or structured analysis (using RoBBR, EvidenceBench, and ECACT) employ metrics such as macro‐F1, accuracy, and Cohen’s kappa. One study adopts a meta‐evaluation benchmark using Elo ratings and inter‐annotator agreement. Overall, papers combine automated retrieval and overlap metrics with human evaluation methods to capture the multifaceted nature of performance in scientific literature review tasks.
Methods
We analyzed 40 sources from an initial pool of 1000, using 5 screening criteria. Each paper was reviewed for 4 key aspects that mattered most to the research question. More on methods
Papers identified with Elicit search
n = 1000
Papers screened using: Benchmark Focus, Computational Literature Review Systems, Beyond Methods Description Only, Literature Review Specific Application, Computational Component
n = 1000
Papers screened out
n = 960
Papers included for extraction
n = 40
Press enter or space to select a node.You can then use the arrow keys to move the node around. Press delete to remove it and escape to cancel.
Press enter or space to select an edge. You can then press delete to remove it or escape to cancel.
Paper search
Using your research question “Which benchmarks measure performance on scientific literature review tasks?”, we searched across over 126 million academic papers from the Semantic Scholar corpus. We retrieved the 1000 papers most relevant to the query.
Screening
We screened in sources based on their abstracts that met these criteria:
- Benchmark Focus: Does this study describe, develop, evaluate, present datasets for, or validate benchmarks/evaluation frameworks specifically designed to measure performance on scientific literature review tasks?
- Computational Literature Review Systems: Does this study focus on computational systems that perform literature search, screening, extraction, synthesis, or other core literature review functions, OR does it synthesize knowledge about literature review benchmarks through systematic/narrative reviews?
- Beyond Methods Description Only: Does this study go beyond only describing literature review methods to actually present, evaluate, or analyze benchmarks or evaluation metrics?
- Literature Review Specific Application: Are the benchmarks or evaluation approaches specifically applied to literature review tasks (rather than focusing solely on general information retrieval or text mining without literature review application)?
- Computational Component: Does this study include computational components rather than focusing exclusively on manual literature review processes?
We considered all screening questions together and made a holistic judgement about whether to screen in each paper.
Data extraction
We asked a large language model to extract each data column below from each paper. We gave the model the extraction instructions shown below for each column.
- Benchmark Type:
Identify and describe the specific type of benchmark created in the study. Look for explicit statements about the benchmark’s purpose, focus, and unique characteristics.
Extraction guidelines:
- Locate the benchmark description in the introduction or methods section
- Capture the specific scientific literature review task the benchmark addresses
- If multiple benchmark types are described, list all of them
- Be precise about the benchmark’s scope (e.g., retrieval, generation, search)
Examples might include:
- Hierarchical catalogue generation benchmark
- Literature search retrieval benchmark
- Information distillation benchmark
If no clear benchmark is described, write “Not applicable” or “No benchmark created”
- Benchmark Dataset Characteristics:
Extract detailed information about the dataset used to create or evaluate the benchmark.
Look for:
- Total number of items in the dataset
- Types of items (e.g., research papers, citations, queries)
- Domain or field of research covered
- Data collection method
- Source of data items
Extraction guidelines:
- Prioritize quantitative details (e.g., “7.6k literature review catalogues and 389k reference papers”)
- Note the specific research domains if mentioned
- If multiple datasets are used, list all with their characteristics
- If dataset details are incomplete, note “Insufficient information”
Format examples:
“7,600 literature review catalogues from scientific papers”
“597 literature search queries across ML and NLP domains”
Evaluation Metrics:
Identify and describe the specific metrics used to assess benchmark performance.
Extraction guidelines:
- Locate metrics in results, methods, or discussion sections
- Capture both quantitative metrics and qualitative assessment approaches
- Note the specific performance dimensions being measured
- If multiple metrics are used, list all of them
Look for metrics such as:
- Recall@5
- Semantic similarity scores
- Informativeness ratings
- Comparative performance against baseline models
Format examples:
“24.8% difference in absolute recall@5”
“Two evaluation metrics for informativeness and structural similarity”
Benchmark Models and Comparisons:
Document the models or systems tested against the benchmark.
Extraction guidelines:
- List all models/systems evaluated
- Note their performance relative to the benchmark
- Capture any comparative analysis between different approaches
- Include both state-of-the-art and baseline models
Look for:
- Specific model names (e.g., BART, ChatGPT)
- Performance rankings
- Comparative performance percentages
Format examples:
- “BART and ChatGPT tested, with performance variations noted”
- “Commercial search engines performed 32 points below best dense retriever”
Results
Characteristics of Included Studies
Study
Study Focus
Benchmark/Dataset Name
Literature Review Task Type
Evaluation Approach
Full text retrieved
Zhu et al., 2023
Hierarchical catalogue generation for literature reviews
HiCaD
Hierarchical catalogue generation
CEDS (semantic/structural similarity), CQE (informativeness)
Yes
Ajith et al., 2024
Literature search retrieval
LitSearch
Literature search/retrieval
Recall at 5, recall at 20, normalized Discounted Cumulative Gain at 10 (nDCG@10)
Yes
Kanoulas et al., 2019
Systematic review retrieval and ranking
CLEF 2019 e-Health Technology-Assisted Review (TAR)
Retrieval, ranking for systematic reviews
No mention found in abstract
No
Kasanishi et al., 2023
Automatic literature review generation
SciReviewGen
Literature review generation (summarization)
Recall-Oriented Understudy for Gisting Evaluation (ROUGE), human evaluation (relevance, coherence, informativeness, factuality)
Yes
Howard et al., 2016
Automated citation screening
SWIFT-Review datasets
Citation screening
Work Saved over Sampling at 95% (WSS@95%), precision at 95% recall
Yes
Wang et al., 2025a
Literature review table generation
ARXIV2TABLE
Table generation for literature reviews
Recall, precision, F1 for schema/cell/pairwise overlap
Yes
Kusa et al., 2023a
Outcome-based evaluation of systematic review automation
CLEF TAR 2019
Systematic review automation (retrieval)
Mean Average Precision (MAP), Recall at k%, WSS, Area Under the Curve (AUC), outcome difference
Yes
Kusa et al., 2023b
Automated citation screening
CSMeD, CSMeD-FT
Citation/full-text screening
True Negative Rate at 95% (TNR@95%), normalized Precision at 95% (nP@95%), nDCG@10, MAP, macro-precision/recall/F1
Yes
Scells et al., 2017
Retrieval for systematic reviews
SIGIR2017-PICO-Collection
Retrieval, screening prioritization
Precision-recall, F-beta, WSS, Average Precision (AP), nDCG, MAP
Yes
Budau and Ensan, 2024
Automated study search for biomedical systematic literature reviews
FASS-BSLR
Study search (retrieval, Boolean query generation)
Precision, Recall, NDCG, MAP, Recall at 1000
No
toof
Pageof
View 30 more rows
Note:All abbreviations and acronyms are defined at first use in the table or in this note. “No mention found” indicates that we did not find mention of the information in the available abstract or full text.
Distribution of Literature Review Automation Tasks
- Retrieval, search, or ranking tasks:Addressed in 17 studies. These focus on identifying and prioritizing relevant literature for systematic reviews or evidence synthesis.
- Generation tasks:Addressed in 11 studies. These include review, table, summary, reference, or abstract generation.
- Screening tasks:Addressed in 9 studies. These include citation, abstract, or full-text screening.
- Extraction tasks:Addressed in 5 studies. These include data, evidence, risk of bias, or meta-analysis.
- Meta-evaluation or meta-analysis:Addressed in 1 study.
- Other tasks:Addressed in 10 studies. These include query generation, compliance, information distillation, academic survey, trial design, systematic review-style reasoning, or multiple systematic literature review tasks.
Distribution of Evaluation Approaches
- Information retrieval metrics:Used in 27 studies. Metrics include precision, recall, F1, Mean Average Precision (MAP), normalized Discounted Cumulative Gain (nDCG), Mean Reciprocal Rank (MRR), Average Precision (AP), and others.
- Human evaluation:Used in 6 studies. Includes inter-annotator agreement, Cohen’s kappa, or explicit human ratings.
- Overlap, coverage, or consistency metrics:Used in 6 studies. Includes schema/cell overlap, coverage, factual consistency, or hallucination rate.
- Specialized or composite metrics:Used in 7 studies. Includes Elo ratings, CEDS, CQE, ECACT, or outcome difference.
- No mention found of evaluation approach:In 6 studies.
Some studies addressed multiple task types or used multiple evaluation approaches. We did not find mention of evaluation approach for 6 studies, and task type information was sometimes ambiguous due to overlapping or unclear descriptions.
Thematic Analysis
Benchmarks for Literature Retrieval and Screening Tasks
Benchmarks in this category focus on identifying, retrieving, and prioritizing relevant literature for systematic reviews or evidence synthesis. Key benchmarks include:
- LitSearch, CLEF Technology-Assisted Review (TAR) (2017–2019), CSMeD, SIGIR2017-PICO-Collection, FASS-BSLR, RELISH, and ResearchArena:These benchmarks use large-scale datasets such as PubMed, MEDLINE, and S2ORC, and evaluate models using recall, precision, Mean Average Precision, normalized Discounted Cumulative Gain, Work Saved over Sampling, and related metrics.
- Model types:Both classical information retrieval models (such as BM25 and term frequency-inverse document frequency) and modern transformer-based or large language model approaches are tested.
- Findings:Some studies (e.g., Kang and Xiong, 2024) report that keyword-based retrieval still outperforms large language model-based methods in certain settings, while others (e.g., Skarlinski et al., 2024) demonstrate large language models matching or exceeding human performance in specific retrieval and synthesis tasks.
Benchmarks for Literature Summarization and Synthesis Tasks
This theme covers benchmarks designed to evaluate the ability of models to generate summaries, reviews, or structured tables from collections of scientific papers. Notable benchmarks include:
- SciReviewGen, ARXIV2TABLE, MSLR2022, ScholarQABench, and Manalyzer:Evaluation metrics include ROUGE, human preference ratings, semantic coverage, factual consistency, and hallucination rates.
- Findings:Some large language models and transformer-based models achieve promising results, but challenges remain in factuality, citation accuracy, and domain adaptation. Human evaluation is often used to supplement automated metrics, reflecting the complexity of assessing synthesis quality.
Benchmarks for Structured Literature Analysis Tasks
Structured analysis tasks include risk of bias assessment, data element extraction, compliance and traceability, and meta-evaluation. Key benchmarks:
- RoBBR, EvidenceBench, ECACT, SciArena-Eval, and internal datasets:These benchmarks use composite or task-specific metrics such as ECACT score, macro-F1, accuracy, and Cohen’s kappa.
- Findings:Automation is advancing, but human oversight remains essential, especially in high-stakes or regulatory contexts.
Evaluation Methodologies Across Benchmarks
Benchmark Category
Primary Metrics
Evaluation Method
Performance Measurement Focus
Retrieval/Screening
Recall, precision, Mean Average Precision, normalized Discounted Cumulative Gain, Work Saved over Sampling, Average Precision
Automated scoring, comparison to human or baseline models
Relevance, workload reduction, ranking effectiveness
Summarization/Synthesis
ROUGE, human ratings, semantic coverage, hallucination rate
Automated metrics, human evaluation
Informativeness, factuality, coherence, citation accuracy
Structured Analysis
Macro-F1, accuracy, Cohen’s kappa, composite scores (ECACT)
Automated scoring, statistical analysis, human validation
Bias, compliance, extraction accuracy, traceability
Meta-evaluation
Elo ratings, accuracy, agreement
Human preference voting, model-based evaluator assessment
Alignment with human judgment, evaluator reliability
toof
Pageof
Summary of Evaluation Approaches:
- Primary metrics:Accuracy was the only metric used in more than one category (2 out of 4 categories). All other metrics were unique to a single category, including recall, precision, Mean Average Precision, normalized Discounted Cumulative Gain, Work Saved over Sampling, Average Precision, ROUGE, human ratings, semantic coverage, hallucination rate, macro-F1, Cohen’s kappa, composite scores (ECACT), Elo ratings, and agreement.
- Evaluation methods:Automated evaluation methods were used in 3 out of 4 categories. Human-based evaluation methods were used in all 4 categories. Comparison-based methods, statistical analysis, and model-based evaluator assessment were each used in 1 out of 4 categories.
- Coverage:We did not find any categories without at least one automated and one human-based evaluation method. No information was missing for any category.
References
Download BIBDownload RISDownload TXT
Report
Status
Gather sources
1000 sources found
Details
Screen sources
40 sources included
Details
Extract data
160 data points extracted
Details
Generate report
Save PDF
BIBLaTeX, ZoteroRISZotero, MendeleyTXTAPA BibliographyPDFPDFDOCXMicrosoft Word
Chat
Got some follow-up questions?
Sign up or sign in to chat with this report.
Back
LitSearch: A Retrieval Benchmark for Scientific Literature Search
Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, Tianyu Gao
Conference on Empirical Methods in Natural Language Processing·
2024·
18 citations
SourceDOI
Plain textPDFFigures
Searching for PDF
Unable to find PDF from source
Back
Benchmark Type
Literature search retrieval benchmark
Benchmark Dataset Characteristics
597 literature search queries across ML and NLP domains; 64,183 papers (59,383 ACL Anthology papers and 4,807 ICLR papers) from S2ORC
Evaluation Metrics
- Recall@5 - Recall@20 - nDCG@10 (for comparison with other benchmarks)
Benchmark Models and Comparisons
- State-of-the-art models tested: GTR, Instructor, E5, GritLM - Baseline model: BM25 - Performance relative to benchmark: GritLM outperforms BM25 by 24.8% in recall@5; LLM-based reranking improves GritLM by 4.4% - Comparative analysis: Commercial search engines like Google Search perform 32 points below the best dense retriever
Literature search questions, such as “where can I find research on the evaluation of consistency in generated summaries?” pose significant challenges for modern search engines and retrieval systems. These questions often require a deep understanding of research concepts and the ability to reason over entire articles. In this work, we introduce LitSearch, a retrieval benchmark comprising 597 realistic literature search queries about recent ML and NLP papers. LitSearch is constructed using a combination of (1) questions generated by GPT-4 based on paragraphs containing inline citations from research papers and (2) questions about recently published papers, manually written by their authors. All LitSearch questions were manually examined or edited by experts to ensure high quality. We extensively benchmark state-of-the-art retrieval models and also evaluate two LLM-based reranking pipelines. We find a significant performance gap between BM25 and state-of-the-art dense retrievers, with a 24.8% difference in absolute recall@5. The LLM-based reranking strategies further improve the best-performing dense retriever by 4.4%. Additionally, commercial search engines and research tools like Google Search perform poorly on LitSearch, lagging behind the best dense retriever by 32 points. Taken together, these results show that LitSearch is an informative new testbed for retrieval systems while catering to a real-world use case.
1Introduction
Finding literature via a specific search query-for example, collecting related work, checking if a method has been proposed before, or recalling a previously seen paper-is a critical task for researchers. Developing systems that recommend citations pertinent to such inquiries holds the potential to enhance researchers' productivity and expedite scientific discovery (Färber and Jatowt, 2020). However, this task is inherently challenging as it often requires deep domain expertise and reasoning through lengthy papers.
Prior to this study, the task of citation recommendation was often formalized by using inline citation mentions from existing papers as queries, and the cited papers as targets (He et al., 2010;Gu et al., 2022). For instance, given the citation mention "RoBERTa and T5 are based on recent advances in masked language modeling [citation]," the text surrounding the citation mention is used as a retrieval query, and the cited paper is the target literature. However, directly using inline citations often leads to queries that are noisy, overly broad (e.g., "Large Language Models [citation]"), or highly context-dependent (e.g., "We follow the hy-
Word overlap filtering
Manual filtering Specificity Quality
Paper corpus
Source paper title: RoBERTa: ... : ... : Unlike , ... we do not train with a reduced sequence length for the first 90% of updates.
Source paper abstract Citation mention Devlin et al., 2019 Sample an inline citation mention Question Target paper
: Is there a pre-trained LM that is trained with reduced sequence length for the first 90% steps? : (BERT: Pre-training ...) Devlin et al., 2019 Figure 2 : The pipeline for generating inline-citation questions. We first sample a citation mention and prompt GPT-4 to generate a question. Next, we filter questions based on word overlap with the target paper title and perform manual inspections to annotate their specificity and quality (see rubrics in Table 1 ).
perparameters of [citation]").
In this work, we propose a new literature retrieval benchmark called LitSearch. As illustrated in Figure 1 , a literature search question seeks papers that meet specific criteria, closely reflecting actual research workflows. LitSearch consists of two subsets: (1) For inline-citation questions, we sample citation mentions from a collection of scientific papers and use GPT-4 (OpenAI, 2023) to rewrite them into literature search questions (Figure 2 ). We retain questions with a low word overlap with the title of the target papers, and perform manual examination to ensure high quality. (2) For author-written questions, we invited authors of ACL 2023 and ICLR 2024 papers to write literature search questions for their own papers. This subset is also manually examined and filtered to remove any inaccurate or easy questions. LitSearch contains 597 questions in total, each paired with one or more scientific papers as the ground truth.
LitSearch has several unique characteristics: (1)
To the best of our knowledge, LitSearch is the first dataset featuring realistic literature search questions, providing a new testbed for citation recommendation and retrieval systems. (2) LitSearch is challenging, requiring deep understanding and reasoning over entire articles. The average document length (6,041/134 words for full texts/titles and abstracts) is significantly longer than that of most existing retrieval benchmarks (e.g., 56 for (Nguyen et al., 2017)). (3) LitSearch is of high quality, with all questions manually examined by the authors.
We conduct extensive experiments on both stateof-the-art retrieval models and reranking with large language models (LLMs). On LitSearch, the best dense retrieval model, GritLM (Muennighoff et al., 2024), achieves an average recall@5 of 74.8%, outperforming BM25 (Robertson et al., 2009) by 24.8%. The recall@5 of GritLM is further improved by 4.4% with GPT-4o reranking. On the other hand, commercial search engines and re-search tools like Google Search perform poorly on this task, only achieving an average recall@5 of 42.8% at most. Furthermore, compared to existing retrieval datasets from BEIR (Thakur et al., 2021) and MTEB (Muennighoff et al., 2022), Lit-Search effectively reflects the performance differences among various embedding models, making it an informative testbed for evaluating state-of-theart retrieval systems.
2LitSearch
Our benchmark LitSearch consists of (a) a large corpus of scientific papers P and (b) pairs of literature search questions and one or more target papers from P. Our desiderata are scientific questions that researchers may use while conducting literature surveys. We use two different strategies to collect such questions: (1) we construct questions using the surrounding context from inline citations in published papers (Section 2.1), and (2) we invited the authors of recent conference publications to manually write questions about their own papers (Section 2.2). For both subsets, we ensure high question quality via manual inspection and filtering conducted by the authors of this work (Section 2.3).
2.1Inline-citation Questions
We define the following concepts for the ease of description: (a) An inline citation mention is a paragraph from the main text of a paper that mentions another paper. For example, this paragraph from the RoBERTa paper (Liu et al., 2019), "... Unlike Devlin et al., (2019), ... we do not train with a reduced sequence length for the first 90% of updates ..." mentions the BERT paper (Devlin et al., 2019). (b) The source paper is the paper the inline citation mention is sampled from. (c) A target paper is a paper that is cited by the inline citation mention.
Figure 2 provides an overview of our data collection methodology for inline-citation questions. We utilize the Semantic Scholar Open Research Corpus (S2ORC; Lo et al., 2020) , a large corpus of academic papers obtained from publishers, archives, and the Internet. We randomly sample inline citation mentions from the S2ORC 2 and prompt GPT-4 to rewrite these citation mentions into literature search questions. These questions are filtered to remove those with a high word overlap with the title of the target papers, and are further manually examined to ensure high quality.
Sampling inline citation mentions. We limit the target papers to be only from the ACL Anthology, for the purpose of aligning with the expertise of the manual annotators, i.e., authors of this work. However, we do not limit where the source papers come from. Depending on the source papers, we call these questions ACL sourced or non-ACL sourced.
Prompting GPT-4 to generate questions. Given a sampled citation mention, we prompt GPT-4 (OpenAI, 2023) to generate a literature search question. In the prompt, we provide (1) the sampled paragraph (the inline citation mention) from the source paper and (2) the titles of the cited papers, and instruct GPT-4 to generate a literature search question based on the paragraph that would be answered by one or more of the papers cited in the paragraph. We use in-context learning (Brown 2 S2ORC provides detailed citation information for most inline citations (including the position of the citation in the paper text and the unique identifier of the target paper), enabling us to easily sample inline citation mentions and match them to S2ORC papers. We used the 2024-03-26 version. et al., 2020) and include two demonstrations. The prompt we use can be found in Table 12 .
Word-overlap filtering. We notice that inlinecitation questions generated in the last step can have very high word overlap with the target paper titles, which makes their retrieval trivial even for BM25 and suggests that the questions may not be of interest to researchers. We calculate the word overlap as the percentage of words in the generated question that are also included in the target paper titles. We filter out ACL sourced questions that have an overlap score higher than 0.3 and non-ACL sourced questions that have an overlap score higher than 0.1. In this step, we filter out 5% of the ACL sourced questions and 80% of the non-ACL sourced questions.
2.2Author-written Questions
Besides generating questions using existing inline citation mentions, we also collect questions directly from human annotators. As writing literature search questions requires deep understanding of the research field and the target paper, we invite researchers to write search queries that are answered by their own published papers. One additional benefit of this setup is that the correctness of the questions is better guaranteed.
We invited authors of ACL 2023 and ICLR 2024 papers to write one literature search question for each of their papers. We chose the two venues as they were among the latest natural language and machine learning conferences at the time of the data collection, hence the papers represent the latest research development and are unlikely to have already been included in the pre-training data of LLMs and retrievers used in our evaluations. We sent out invitations to 623 ACL 2023 authors and 404 ICLR 2024 authors, and received 175 questions from ACL 2023 authors and 117 questions from ICLR 2024 authors.
2.3Manual Filtering to Ensure High Quality
Finally, the authors of this work manually examine every question from both the inline-citation and author-written subsets and annotate these for specificity and quality (guidelines in Table 1 ). Questions that are too general (there are more than 20 papers from the corpus can fit the question) are assigned a quality score of 0 and are excluded. We include only questions with a quality score of 1 or 2 in the final dataset. We also rewrite questions if they have minor mistakes and can be fixed easily. Each question is assigned to one author for annotation.
As the examples in Table 1 show, questions of both specificity types can be realistic and valuable, but they exhibit distinct traits. We use the specificity scores to distinguish broad and specific questions in the evaluation.
For the inline-citation subset, we manually examined 382 ACL sourced questions and 450 non-ACL sourced questions. 26% (98 instances) of the ACL questions and 56% (253 instances) of the non-ACL questions are kept. For the author-written subset, since all questions are written by experts, we avoid rewriting them as much as possible. In the end, we kept 89% (155) questions from ACL 2023 authors and 78% (91) questions from ICLR 2024 authors.
2.4Dataset Statistics
Our final dataset contains 597 questions, with 351 in the inline-citation subset and 246 in the authorwritten subset. Dataset statistics, including the number of questions, the average question length, and the average word overlap between the question and the target papers (titles and abstracts), are presented in Table 2 . We find that author-written questions are shorter and have a higher word overlap rate with the target papers (0.43 vs. 0.33 for inline-citation questions). This is expected: when writing questions for their own papers, authors tend to re-use terminology from their papers and focus on the main findings which are usually included in the abstracts or titles. In contrast, inline-citation questions can be anchored to any span of the reference documents, irrespective of the main findings or the main focuses of the target papers.
2.5The Retrieval Corpus
The LitSearch retrieval corpus P consists of ACL Anthology and ICLR papers extracted from S2ORC (see Appendix C for details). We do not use the full S2ORC corpus for efficiency reasons. In total, this yields 64, 183 papers (59, 383 ACL Anthology papers and 4, 807 ICLR papers) 3 . The average number of words for the documents in P is 134 / 6,041 (titles and abstracts / full texts).
3.1Experimental Setup
We compare the performance of different retrieval systems (enumerated below) on our LitSearch benchmark. Due to the limited context sizes of existing embedding models, we only use the paper titles and abstracts to embed the papers in our retrieval corpus P by default.
For all systems we compare, we report the recall@K for both the broad and specific subsets of LitSearch. We report results for K = 5, 20 for the specific subset and K = 20 for the broad subset; these values (5 and 20) correspond to the guidelines followed by the authors while determining the specificity of a given question (see Table 1 ).
3.2Baselines
We benchmark both retrieval models and LLMbased rerankers in this work.
Retriever models. We evaluate using the classic BM25 algorithm (Robertson et al., 2009), as well as several state-of-the-art dense retrieval (embedding) models, including GTR (Ni et al., 2022), Instructor (Su et al., 2023), E5 (Wang et al., 2022), and GritLM (Muennighoff et al., 2024). 4 More details are provided in Appendix D.
LLM-based reranking. In addition to vanilla retrieval, we also use strong LLMs (GPT-4o5 in our case) to rerank the top retrieved results from the above retrievers. We use two strategies:
Vanilla reranking. We include the top-n retrieved papers (titles and abstracts) in the context
Inline-citation
Author-written Avg.
Avg. Broad Specific Broad Specific Broad Specific R@20 R@5 R@20 R@20 R@5 R@20 R@20 R@5 BM25 Here we only use the titles and abstracts of papers for retrieval and reranking. We report recall@20 (R@20) for broad questions and recall@5 and @20 (R@5, R@20) for specific questions. "Broad" and "specific" correspond to the annotations during our manual filtering stage (defined in Table 1 ). and prompt GPT-4o to rerank these based on the question (see our prompt in Table 13 ). This is similar to prior works (Sun et al., 2023b; Ma et al., 2023). We use n = 100, resulting in an average context length of 13,844 words.
One-hop reranking. Inspired by Tang et al. (2023), we leverage the fact that for some questions, there may exist lexically similar inline citation mentions in the retrieval corpus. Due to our data collection pipeline, this is particularly true for the ACL sourced inline citation questions. We posit that the retrieval models will be able to retrieve these source papers (that cite the target papers) based on the questions.
We extract the top m retrieved papers and construct a new candidate list by adding papers cited by each of these seed retrieved papers. We concatenate papers in the following order, skipping duplicates: [rank-1 paper p 1 , papers cited by p 1 , rank-2 paper p 2 , papers cited by p 2 , ..., p m , papers cited by p m ]. To avoid very long contexts, we truncate this list after the first n papers and use the same prompt for GPT-4o based reranking as above.
In our experiments, we use m = 50 and n = 200 resulting in average length of 27,544 words.
3.3Results
We outline the performance of the above systems on LitSearch in Table 3 . First, we observe that all instruction-finetuned embedding models, e.g. Instructor, E5, and GritLM, substantially outperform BM25 on our benchmark. In fact, they also perform better than the GTR model. Overall, we found that GritLM-7B achieves the best performance (70.8 recall@20 on broad questions and 74.8 recall@5 on specific questions), leaving a large gap compared to other baselines.
Impact of reranking. We also report the performance improvement brought by the reranking methods on the weakest (BM25) and strongest (GritLM) retrievers in Table 3 . We observe that both vanilla and one-hop reranking improve over the base retrieval performance. For example, on the specific subset of inline questions, the vanilla GPT-4o reranking improves the recall@5 of BM25 and GritLM by 21.5% and 5.5% respectively. Interestingly, the improvements from one-hop reranking are generally lower than vanilla reranking across all subsets, when using GritLM as the base retriever; this shows that our benchmark cannot be easily "gamed" by mimicking the data collection pipeline or exploiting similar citation mentions to the question from other papers.
Impact of question specificity. Table 3 and Figure 3 show that retrieval systems generally report higher recall performance on the specific subset. This is expected: there exists a smaller number of "competing" papers, i.e. those that also satisfy the search question, for the specific subset in the retrieval corpus. Note that our human annotation tagged questions with approximately 5 relevant papers as specific and 20 relevant papers as broad (see Table 1 for details). We keep both subsets in our dataset as this stratified reporting presents a more nuanced view of retriever capabilities.
Inline-citation vs. author-written questions.
We observe very different performance trends for the two subsets (Table 3 and Figure 3 ). In particular, inline-citation questions are harder than authorwritten questions for all retriever systems, on both broad and specific questions.
We attribute this difference to the higher semantic or lexical overlap of author-written questions with the paper titles and abstracts (see Table 2 for statistics). This reflects expected tendency of paper authors to formulate the questions around the main contributions from the abstracts and re-use terminologies. Such annotator biases have been widely discussed in prior data collection efforts as well, particularly when humans write content from scratch (Gururangan et al., 2018).
Inline Author
Broad Spec Broad Spec R@20 R@5 R@20 R@5 BM25 Table 5 : Retrieval results of using only titles and abstracts vs. using titles, abstracts, and full text (w/ full).
We do not observe consistent improvements from including the full text for existing retrieval models.
ity subsets. Recall that we manually annotated the quality of all questions (Table 1 ). We observe that questions with a quality score of 2, i.e. determined to be more realistic and difficult by manual annotators, are consistently more challenging for retrieval models. This demonstrates the high annotation quality of our manual inspection step. The presence of these different quality questions in our dataset leads to higher diversity and better coverage over the varied information seeking needs of users.
4Analysis
4.1 Does Including More Paper Content Improve Retrieval Performance?
In the previous section, we only used the titles and abstracts (on average 134 words) to encode the papers in the retrieval corpus. Here, we evaluate whether encoding more paper content can improve retrieval performance. For all retriever models compared, we create embeddings using the full paper text (on average 6,041 words) up to their allowed context lengths. 6 We compare this setting against our default setting (only titles and abstracts).
Our results are outlined in Table 5 . Surprisingly, we find that the addition of more paper text does not improve performance on LitSearch consistently. In fact, we only observe substantial improvement on the author-written broad questions for BM25 and some embedding models. In other cases, more text more often hinders instead of im- Results show that the performance improvement from one-hop reranking over BM25 is subtantially higher for ACL sourced questions.
proving performance. Note that the maximum context length of the tested models is 2,048 (GritLM) and the average length of their training data is even shorter-for example, the commonly used MS-MARCO (Nguyen et al., 2017) and NaturalQuestions (Lee et al., 2019) have an average document length of 56 and 79. This is significantly shorter than the full text of papers from our retrieval corpus averaging 6,041 words in length, potentially leading to the unsatisfying performance when using full texts with embedding models.
4.2Does the Source of Inline Citation
Questions Matter?
Next, we study how the different sources of inlinecitation questions affect the model performance. Table 6 outlines the performance of retrieval models on ACL sourced vs. non-ACL sourced inlinecitation questions. Our results show that the two different sets report similar trends and model rankings for different retrieval models, particularly on the specific subset of questions. Interestingly, we find that the performance improvement from onehop reranking is very different for the ACL and non-ACL questions.
For BM25, we observe that one-hop reranking is significantly better than the vanilla reranking on ACL sourced questions (+11.4% recall@5 on specific); but the gap is much smaller on non-ACL sourced questions (+1.2% recall@5 on specific). We posit that this is because BM25 can better exploit the data annotation pipeline on the ACL sourced questions. It can likely first identify Table 7 : Recall@5 for commercial search engines on a random subset of 80 specific questions. Search engines generally report poor performance. Note that the comparison is not apples-to-apples as search engines use a much larger retrieval corpus.
the source ACL paper where the citation mention comes from, and then find the target paper via onehop reranking. Including the non-ACL questions to LitSearch prevents systems from exploiting such "shortcuts" as non-ACL source papers are not part of the retrieval corpus.
For GritLM, we do not observe similarly large performance gains when using one-hop reranking. We hypothesize that this is because when using GritLM, the initial top retrieval results already include the target papers and the one-hop strategy does not bring further improvement.
4.3Performance of Search Engines
In practice, researchers use search engines like Google Search, Google Scholar, or Elicit7 to search for relevant papers for their scientific queries. We conduct a human study to understand how these search engines perform on LitSearch: We randomly sample 80 questions (all specific; 40 inline-citation and 40 author-written) from our dataset. We manually input8 these questions into the above search engines and report recall@5. 9 We note that this is not an apples-to-apples comparison against the retrieval models in earlier sections due to the discrepancy in the retrieval corpus.
Table 7 outlines the results of our human study. It shows that all three search engines deliver similarly low recalls on inline-citation questions. On the author-written questions, Google Search performs much better than the other two. Although not directly comparable, this performance is generally worse than the embedding models, demonstrating the potential of these strong dense retrieval models for citation recommendation applications.
4.4Comparing Other Retrieval Benchmarks
We compare model performance on LitSearch to several popular retrieval benchmarks included in BEIR (Thakur et al., 2021) and MTEB (Muennighoff et al., 2022)-namely MS-MARCO (Nguyen et al., 2017), SCIDOCS (Cohan et al., 2020), and NQ (Lee et al., 2019). 10 We also compare to ArXiv (Gu et al., 2022), a previous citation recommendation benchmark directly using inline citations as queries. Table 8 shows that LitSearch generally agrees with existing retrieval benchmarks. However, LitSearch can differentiate retriever models better: for example, the gap between GritLM and E5 on LitSearch (specific) is 15 points (nDCG@10), while they perform almost the same on MSMARCO. LitSearch provides an informative testbed that can effectively reflect the recent (and future) advancement in embedding models.
5Related Work
Citation recommendation. The community has proposed a number of citation recommendation datasets (Färber and Jatowt, 2020), including global citation recommendation datasets (directly using a paper as the query and papers it cites as target papers; Cohan et al., 2020;Bhagavatula et al., 2018), and local citation recommendation datasets (using inline citation mentions as queries; He et al., 2010;Medić and Snajder, 2020;Jeong et al., 2020;Gu et al., 2022). There are also language models and retrieval models specifically trained for scientific document understanding and retrieval tasks, such as SciBERT (Beltagy et al., 2019) and SPECTER (Cohan et al., 2020). Compared to existing citation recommendation datasets, LitSearch is comprised of manually annotated, natural language literature search questions, providing a more realistic and challenging evaluation for citation recommendation systems.
Retrieval benchmarks. There have been numerous datasets evaluating retrieval systems from Wikipedia (Kwiatkowski et al., 2019;Lee et al., 2019), web queries (Nguyen et al., 2017), biomedical questions (Voorhees and Tice, 2000), and more. Recently, there have been several benchmarks combining multiple datasets and evaluating retrieval or embedding models across different domains and different use cases, such as KILT (Petroni et al., 2021), BEIR (Thakur et al., 2021), and MTEB (Muennighoff et al., 2022). LitSearch offers a unique perspective by exploring the novel literature search question type, effectively complementing the existing benchmarks. Contemporary with our work, CiteME (Press et al., 2024) introduces a benchmark for identifying references based on claims made in a paper's inline text (as opposed to research questions in our case). Their focus differs from ours in that it is aimed more at LLM-based agents rather than retrieval systems.
Retrieval systems. Traditional retrieval systems rely on bag-of-word algorithms such as TF-IDF and BM25. Dense retrieval (embedding) models have gained more popularity due to their abilities to do semantic search without relying on exact keyword matches (Pennington et al., 2014; Reimers and Gurevych, 2019) . State-of-the-art dense models are mostly adopted by fine-tuning pre-trained language models (Devlin et al., 2019;Touvron et al., 2023) with a contrastive learning objective on either supervised or unsupervised data (Karpukhin et al., 2020;Gao et al., 2021;Izacard et al., 2022;Ni et al., 2022;Khattab and Zaharia, 2020). Recent development introduces "instructions" when encoding queries and documents, which significantly improves the versatility of embeddings across tasks (Su et al., 2023;Wang et al., 2022;Wu et al., 2022;Lee et al., 2024;BehnamGhader et al., 2024).
6Conclusion
In this paper, we propose LitSearch, a new retrieval benchmark comprising 597 manually-curated literature search questions. LitSearch includes an inline-citation question set and an author-written question set, both undergoing manual inspection from the authors of LitSearch. We conduct extensive experiments with BM25, state-of-the-art embedding models, and LLM reranking. Our experiments demonstrate the superior performance of state-of-the-art instruction-finetuned embedding models, with additional improvement via GPT-4obased reranking. We also verify that commercial search engines like Google struggle with LitSearch questions. The comparison with existing retrieval benchmarks shows that LitSearch better differentiates the performance of retrieval systems.
Limitations
Even though we manually examined the dataset, there still exist questions that are either slightly out of distribution compared to what researchers would ask, or too easy due to high overlap with the target papers. The author-written questions are easier than we expected, as writing challenging literature search questions is non-trivial even for experienced researchers. Even though we experimented with several state-of-the-art systems, it was not an exhausted evaluation and we left out more sophisticated retrieval or reranking systems. This research primarily focuses on only English questions and research papers.
Ethics Statement
The research artifact of this paper, LitSearch, is manually inspected and has been ensured to have no unsafe or inappropriate content. However, the process to generate the dataset may introduce certain biases: for example, the inline-citation questions contain more target papers that have high citations due to the sampling; the author-written questions only cover ACL 2023 and ICLR 2024 papers.
A Annotator Acknowledgments
We would like to thank Marah I Abdin, Jaewoo Ahn, Kabir Ahuja, Xi Ai, Satoshi Akasaki, Anastasios N Angelopoulos, Jinheon Baek, Eslam Mohamed Bakr, Pablo Barceló, Claudio Battiloro, Jonas Belouadi, Abhik Bhattacharjee, Valeriia Bolotova, Pengshan Cai, Nitay Calderon, Qingqing Cao, Defu Cao, Souradip Chakraborty, Jun Shern Chan, Sachin Chanchani, Yulong Chen, Yiming Chen, Xinyuan Chen, Nuo Chen, Hanjie Chen, Xiudi Chen, Zeming Chen, An-Chieh Cheng, Xize Cheng, Cheng-Han Chiang, Josef Dai, David Dale, Yue Deng, Yifan Deng, Shizhe Diao, Bosheng Ding, Xuan Long Do, Yilun Du, Yupei Du, Salijona Dyrmishi, Dante Everaert, Zhenghan Fang, Bahare Fatemi, Jiazhan Feng, Shangbin Feng, Patrick Fernandes, Javier Ferrando, Christopher Fifty, Sarah E Finch, Matthew Finlayson, Lea Frermann, Mikhail Galkin, Songyang Gao, Ziteng Gao, Silin Gao, Sara Ghazanfari, Nathan Godey, Navita Goyal, Xinran Gu, Yuxian Gu, Yu Gu, Anchun Gui, Jiacheng Guo, Ashim Gupta, Paul Hagemann, Tianxing He, Zhengfu He, Juncai He, Leonhard Hennig, Konstantin Hess, Jennifer Hu, Xiaoyang Hu, Zhilei Hu, Weidong Huang, Yichong Huang, Ayyoob Imani, Qi Jia, Yifan Jiang, Hanwen Jiang, Yiding Jiang, Yang Jin, Youngjin Jin, Zhijing Jin, Emmeran Johnson, Josef Jon, David Jurgens, Ehsan Kamalloo, Junmo Kang, Jian Kang, Mikhail Khodak, Hyunjae Kim, Soroush Abbasi Koohpayegani, Suhas Kotha, Jeongyeol Kwon, Sunjae Kwon, Philippe Laban, Zhibin Lan, Nayoung Lee, Deokjae Lee, Celine Lee, Heejun Lee, Jie Lei, Wenhao Li, Yafu Li, Yufei Li, Yanzeng Li, Yanzhou Li, Ziqiang Li, Zhaoyi Li, Ziheng Li, Xiaonan Li, Yinghao Li, Yu Li, Chengrui Li, Yingjie Li, Yunlong Liang, Baohao Liao, Kezhou Lin, Licong Lin, Enrico Liscio, Xiangyan Liu, Chenzhengyi Liu, Yixin Liu, Xingbin Liu, Haolin Liu, Xiao Liu, Yajiao Liu, Meng Liu, Tianyang Liu, Wei Liu, Qingyu Lu, Pan Lu, Junyu Lu, Zhengyi Luo, Yang Luo, Ang Lv, Junhyung Lyle, Jiajun Ma, Kaixin Ma, Ziqiao Ma, Mounica Maddela, Chaitanya Malaviya, Zhiyu Mei, Ethan Mendes, Fatemehsadat Mireshghallah, Niloofar Mireshghallah, Mircea Mironenco, Takeru Miyato, Fengran Mo, Xinyi Mou, Niklas Muennighoff, Cheolwon Na, Piotr Nawrot, Mang Ning, Longshen Ou, Siqi Ouyang, Lorenzo Pacchiardi, Ziqi Pang, Sara Papi, Letitia Parcalabescu, Tanmay Parekh, Aleksandar Petrov, Lucía Pitarch, Moritz Plenz, Manish Prajapat, Joan Puigcerver, Valentina Pyatkin, Shuofei Qiao, Yujia Qin, Chengwei Qin, Sigal Raab, Hossein A Rahmani, Siyu Ren, Yubing Ren, Ruiyang Ren, Yangjun Ruan, Michael J Ryan, Shoumik Saha, Vageesh Saxena, Michael Saxon, Alexander Scarlatos, Agam Shah, Erfan Shayegani, Behzad Shayegh, Xiangqing Shen, Sheng Shen, Ruizhe Shi, Zhengliang Shi, Kensen Shi, Ziyi Shou, Prasann Singhal, Jasivan Alex Sivakumar, Junru Song, Chunjin Song, Nikita Srivatsan, Michal Štefánik, Hao Sun, Mingjie Sun, Weiwei Sun, Zhiqing Sun, Xiaohang Tang, Liyan Tang, Eshaan Tanwar, Jiayan Teng, Davide Testa, Changyao Tian, Yufei Tian, Eric Todd, Benjamin Towle, Austin Tripp, Yi Tu, Rheeya Uppaal, Lazar Valkov, Neeraj Varshney, Artem Vazhentsev, Yiming Wang, Qifan Wang, Zhaoyang Wang, Lirui Wang, Zhicheng Wang, Weiqi Wang, Jiaan Wang, Boshi Wang, Haiming Wang, Huimin Wang, Yun-Cheng Wang, Runzhe Wang, Yu Wang, Yidong Wang, Licheng Wen, Te-Lin Wu, Yu-Yu Wu, Qianhui Wu, Dongming Wu, Tong Wu, Zijun Wu, Mengzhou Xia, Jian Xie, Yiming Xie, Weiwen Xu, Yi Xu, Xilie Xu, Derek Xu, Shohei Yamasaki, Hao Yan, Chenghao Yang, Xianjun Yang, Sen Yang, Bingsheng Yao, Qinyuan Ye, Fan Yin, Haneul Yoo, Kiyoon Yoo, Xinyan Velocity Yu, Jianfei Yu, Qiying Yu, Mo Yu, Zichun Yu, Yue Yu, Youliang Yuan, Zihao Yue, Xiang Yue, Yuanwen Yue, Daoguang Zan, Zhiyuan Zeng, Guangtao Zeng, Yuheng Zha, Runzhe Zhan, Jiaxu Zhang, Zhexin Zhang, Chen Zhang, Xinlu Zhang, Yabo Zhang, Renrui Zhang, Kechi Zhang, Ruoyu Zhang, Feng Zhang, Siyan Zhao, Junhao Zheng, Wenjie Zheng, Ming Zhong, Yan Zhou, Pei Zhou, Yangqiaoyu Zhou, Aojun Zhou, Xuekai Zhu, Luyao Zhu, Yanqiao Zhu, Dele Zhu, Andrew Zhu, Wenjie Zhuo and Caleb Ziems for contributing author-written questions about their ACL 2023 and/or ICLR 2024 papers.
B Annotation Details
We provide instructions regarding manually inspecting questions in Table 1 . We sent out emails and Google Forms to recruit ACL 2023 and ICLR 2024 authors for author-written questions, and the templates can be found in Table 10 and Table 11 respectively.
C Retrieval Corpus
The LitSearch retrieval corpus P consists of ACL Anthology and ICLR papers extracted from S2ORC.
Here we describe how we identify those papers in S2ORC: We isolate ACL anthology papers from S2ORC by identifying entries whose metadata includes an ACL Anthology ID. We identify ICLR papers utilizing a combination of the venue-based queries to Semantic Scholar's Academic Graph API and by title-matching using titles of accepted papers scraped from the official ICLR website.
D Retriever details
We list the full HuggingFace checkpoint paths corresponding to the dense retrievers we use in Table 9 . We use the following instructions for the instruction-finetuned embedding models: "Represent the research question for retrieving relevant research paper abstracts:" for encoding queries; "Represent the title and abstract of the research paper for retrieval:" for encoding papers when using Instructor-XL for retrieval using paper titles and abstracts; when performing retrieval using paper titles and abstacts with GritLM-7B, we use the instruction "Given a research query, retrieve the title and abstract of the relevant research paper".
Retriever
HuggingFace Checkpoint GTR-T5-large sentence-transformers/gtr-t5-large
Table 9 : HuggingFace checkpoints we use for each dense retriever.
E Prompts and Additional Statistics
Table 12 shows the prompt we use for generating inline-citation questions via GPT-4. Table 13 shows the reranking prompt for GPT-4o. We are trying to create a new challenging retrieval benchmark where the task is to retrieve relevant research papers given a research query. For example, the research query "Which paper first found that large language models can do in-context learning?" should retrieve the GPT-3 paper. We could use your help for creating such a query that should be answered by your own paper! Could you provide one high-quality, literature-search-type query about your paper that you think would be challenging even for state-of-the-art retrieval systems? Please make sure to follow these guidelines when writing your query. Your query must be
(1) Realistic: It should be plausible that a researcher working in a related field may ask this exact question. Do not ask a question that puts too many unrealistic constraints (such as "What work that uses the NaturalQuestions dataset trains for 15 epochs and uses a learning rate of 3e-5?").
(2) Specific: Your query should be answered by/correspond to one particular (representative) paper or a small number of papers. Do not ask an overly broad question like "What are some multimodal models?" Instead, you can ask "Which multimodal model was the first to use interleaved image and text data?"
(3) Challenging: Our goal is to collect a set of queries that are extremely challenging even for SOTA retrievers today. Make sure not to submit a query that can easily be answered via keyword matching or a Google search like "Which paper proposed masked language modeling?" (BERT). Instead, you can use the following formats to make the question more challenging: (a) Asking a detailed question that the abstract does not cover: "Which text pre-training paper first used the data mixture of wikipedia and bookscorpus?" (BERT) (b) Rephrasing (reducing word overlap to the paper): "Is there such a machine learning dataset, where for some questions, there is no correct answer and model should abstain?" (SQuAD v2) (c) Asking about the main technique innovation: "Is there any paper that combines distillation and structured pruning for language models?" (CoFi pruning)
Thank you for your contribution! We will make sure that:
-Your name will be credited in our paper unless you choose otherwise.
-We will make the dataset available for open-source development.
Table 11 : Instructions provided in the Google Forms sent to ICLR 2024 and ACL 2023 authors for collecting author-written questions.
I will provide you with a excerpt from an scientific article that cites various papers in the positions {cite_001}, {cite_002}, etc. I would like you to write *general* questions about the academic literature using this paragraph, which will be answered with the corresponding citations.
Here are the constraints you must follow:
(1) The questions should make sense and be interesting without reading the paragraph. The questions should be general, and you should only use the paragraph to find the ground-truth answer.
(2) Try to ask questions which cover as many citations as possible and are as detailed as possible.
Pack as much information into the question as you can.
(3) If the context is not clear, skip the citation. Make sure that every question is of the highest quality, as these questions will be used for important work concerning information extraction from the scientific literature.
(4) The answer should ONLY contain the citation key.
TEXT: We argue there are two underlying motivations for the ad text generation task, especially for product descriptions. Application-wise, the utility is to improve the seller experience for e-commerce services when registering a new product. The generated descriptions can reduce the need for manual data entry, and potentially improve sales due to better descriptions (in terms of attractiveness, structure, and persuasiveness). Research-wise, ad text generation is an under-studied task, and arguably a good proxy for persuasive text generation {cite_016}{cite_017}{cite_018}{cite_019}.
PAPER TITLES:
-cite_016: Is this post persuasive? ranking argumentative comments in online forum -cite_017: On the role of discourse relations in persuasive texts -cite_018: Measuring online debaters' persuasive skill from text over time -cite_019: Analyzing the Persuasive Effect of Style in News Editorial Argumentation -cite_020: Persuaide! an adaptive persuasive text generation system for fashion domain -cite_021: A statistical framework for product description generation -cite_022: SILVER: Generating persuasive Chinese product pitch QUESTION: I want to read some papers that try to study and quantify how persuasive text can be. I'm coming at this from an applications-perspective, as I'm interested in using these insights for product development. The prompt used for generating questions from inline citations using GPT-4.
(system) You are RankGPT, an intelligent assistant that can rank papers based on their relevancy to a research query. Only respond with the ranking results, do not say any words or explain.
Table 13 : The prompt used for reranking retrieved documents using GPT-4o (adapted from Sun et al., 2023a) .
Acknowledgements
AcknowledgementsWe want to acknowledge Dan Friedman, Howard Yen, Jiayi Geng, Lucy He, and other members of the Princeton NLP group for their useful feedback and discussion.We also acknowledge all the ACL 2023 and ICLR 2024 authors that contributed questions to LitSearch (listed in Appendix A).Tianyu Gao is supported by an IBM PhD Fellowship.This work is gratefully supported by an NSF CAREER award (IIS-2239290), and Microsoft Azure credits through the "Accelerate Foundation Models Academic Research" Initiative.
Failed to load PDF:
StripeM-Inner
Paper sources
Abstract screening pilot
Abstract screening results
Extraction pilot
Extraction results
Research report
Modify setup