Elicit: Benchmarks for Scientific Literature Review Performance (public)
Benchmarks for Scientific Literature Review Performance (public)
Research reportView only
Create alertChat
September 12, 2025
Which benchmarks measure performance on scientific literature review tasks?
Benchmarks for scientific literature review tasks include datasets like LitSearch, SciReviewGen, SWIFT‐Review, and RoBBR that measure performance across retrieval, generation, screening, extraction, and meta-evaluation areas.
Abstract
Benchmarks for scientific literature review tasks address five primary areas: retrieval, generation, screening, extraction, and meta-evaluation. Seventeen studies target retrieval using datasets such as LitSearch, CLEF TAR, CSMeD, SIGIR2017-PICO-Collection, FASS‐BSLR, RELISH, and ResearchArena. These studies report performance using metrics such as recall, precision, Mean Average Precision, normalized Discounted Cumulative Gain, and Work Saved over Sampling. Eleven studies focus on generation tasks—including review, table, or abstract writing—with benchmarks like SciReviewGen, ARXIV2TABLE, MSLR2022, ScholarQABench, and Manalyzer; performance is measured by ROUGE scores, hallucination rates, semantic coverage, and human ratings. Nine studies on screening tasks (using tools such as SWIFT‐Review and LGAR) report measures such as precision at high recall levels and WSS@95, while five studies on extraction or structured analysis (using RoBBR, EvidenceBench, and ECACT) employ metrics such as macro‐F1, accuracy, and Cohen’s kappa. One study adopts a meta‐evaluation benchmark using Elo ratings and inter‐annotator agreement. Overall, papers combine automated retrieval and overlap metrics with human evaluation methods to capture the multifaceted nature of performance in scientific literature review tasks.
Methods
We analyzed 40 sources from an initial pool of 1000, using 5 screening criteria. Each paper was reviewed for 4 key aspects that mattered most to the research question. More on methods
Papers identified with Elicit search
n = 1000
Papers screened using: Benchmark Focus, Computational Literature Review Systems, Beyond Methods Description Only, Literature Review Specific Application, Computational Component
n = 1000
Papers screened out
n = 960
Papers included for extraction
n = 40
Press enter or space to select a node.You can then use the arrow keys to move the node around. Press delete to remove it and escape to cancel.
Press enter or space to select an edge. You can then press delete to remove it or escape to cancel.
Paper search
Using your research question “Which benchmarks measure performance on scientific literature review tasks?”, we searched across over 126 million academic papers from the Semantic Scholar corpus. We retrieved the 1000 papers most relevant to the query.
Screening
We screened in sources based on their abstracts that met these criteria:
- Benchmark Focus: Does this study describe, develop, evaluate, present datasets for, or validate benchmarks/evaluation frameworks specifically designed to measure performance on scientific literature review tasks?
- Computational Literature Review Systems: Does this study focus on computational systems that perform literature search, screening, extraction, synthesis, or other core literature review functions, OR does it synthesize knowledge about literature review benchmarks through systematic/narrative reviews?
- Beyond Methods Description Only: Does this study go beyond only describing literature review methods to actually present, evaluate, or analyze benchmarks or evaluation metrics?
- Literature Review Specific Application: Are the benchmarks or evaluation approaches specifically applied to literature review tasks (rather than focusing solely on general information retrieval or text mining without literature review application)?
- Computational Component: Does this study include computational components rather than focusing exclusively on manual literature review processes?
We considered all screening questions together and made a holistic judgement about whether to screen in each paper.
Data extraction
We asked a large language model to extract each data column below from each paper. We gave the model the extraction instructions shown below for each column.
- Benchmark Type:
Identify and describe the specific type of benchmark created in the study. Look for explicit statements about the benchmark’s purpose, focus, and unique characteristics.
Extraction guidelines:
- Locate the benchmark description in the introduction or methods section
- Capture the specific scientific literature review task the benchmark addresses
- If multiple benchmark types are described, list all of them
- Be precise about the benchmark’s scope (e.g., retrieval, generation, search)
Examples might include:
- Hierarchical catalogue generation benchmark
- Literature search retrieval benchmark
- Information distillation benchmark
If no clear benchmark is described, write “Not applicable” or “No benchmark created”
- Benchmark Dataset Characteristics:
Extract detailed information about the dataset used to create or evaluate the benchmark.
Look for:
- Total number of items in the dataset
- Types of items (e.g., research papers, citations, queries)
- Domain or field of research covered
- Data collection method
- Source of data items
Extraction guidelines:
- Prioritize quantitative details (e.g., “7.6k literature review catalogues and 389k reference papers”)
- Note the specific research domains if mentioned
- If multiple datasets are used, list all with their characteristics
- If dataset details are incomplete, note “Insufficient information”
Format examples:
“7,600 literature review catalogues from scientific papers”
“597 literature search queries across ML and NLP domains”
Evaluation Metrics:
Identify and describe the specific metrics used to assess benchmark performance.
Extraction guidelines:
- Locate metrics in results, methods, or discussion sections
- Capture both quantitative metrics and qualitative assessment approaches
- Note the specific performance dimensions being measured
- If multiple metrics are used, list all of them
Look for metrics such as:
- Recall@5
- Semantic similarity scores
- Informativeness ratings
- Comparative performance against baseline models
Format examples:
“24.8% difference in absolute recall@5”
“Two evaluation metrics for informativeness and structural similarity”
Benchmark Models and Comparisons:
Document the models or systems tested against the benchmark.
Extraction guidelines:
- List all models/systems evaluated
- Note their performance relative to the benchmark
- Capture any comparative analysis between different approaches
- Include both state-of-the-art and baseline models
Look for:
- Specific model names (e.g., BART, ChatGPT)
- Performance rankings
- Comparative performance percentages
Format examples:
- “BART and ChatGPT tested, with performance variations noted”
- “Commercial search engines performed 32 points below best dense retriever”
Results
Characteristics of Included Studies
Study
Study Focus
Benchmark/Dataset Name
Literature Review Task Type
Evaluation Approach
Full text retrieved
Zhu et al., 2023
Hierarchical catalogue generation for literature reviews
HiCaD
Hierarchical catalogue generation
CEDS (semantic/structural similarity), CQE (informativeness)
Yes
Ajith et al., 2024
Literature search retrieval
LitSearch
Literature search/retrieval
Recall at 5, recall at 20, normalized Discounted Cumulative Gain at 10 (nDCG@10)
Yes
Kanoulas et al., 2019
Systematic review retrieval and ranking
CLEF 2019 e-Health Technology-Assisted Review (TAR)
Retrieval, ranking for systematic reviews
No mention found in abstract
No
Kasanishi et al., 2023
Automatic literature review generation
SciReviewGen
Literature review generation (summarization)
Recall-Oriented Understudy for Gisting Evaluation (ROUGE), human evaluation (relevance, coherence, informativeness, factuality)
Yes
Howard et al., 2016
Automated citation screening
SWIFT-Review datasets
Citation screening
Work Saved over Sampling at 95% (WSS@95%), precision at 95% recall
Yes
Wang et al., 2025a
Literature review table generation
ARXIV2TABLE
Table generation for literature reviews
Recall, precision, F1 for schema/cell/pairwise overlap
Yes
Kusa et al., 2023a
Outcome-based evaluation of systematic review automation
CLEF TAR 2019
Systematic review automation (retrieval)
Mean Average Precision (MAP), Recall at k%, WSS, Area Under the Curve (AUC), outcome difference
Yes
Kusa et al., 2023b
Automated citation screening
CSMeD, CSMeD-FT
Citation/full-text screening
True Negative Rate at 95% (TNR@95%), normalized Precision at 95% (nP@95%), nDCG@10, MAP, macro-precision/recall/F1
Yes
Scells et al., 2017
Retrieval for systematic reviews
SIGIR2017-PICO-Collection
Retrieval, screening prioritization
Precision-recall, F-beta, WSS, Average Precision (AP), nDCG, MAP
Yes
Budau and Ensan, 2024
Automated study search for biomedical systematic literature reviews
FASS-BSLR
Study search (retrieval, Boolean query generation)
Precision, Recall, NDCG, MAP, Recall at 1000
No
toof
Pageof
View 30 more rows
Note:All abbreviations and acronyms are defined at first use in the table or in this note. “No mention found” indicates that we did not find mention of the information in the available abstract or full text.
Distribution of Literature Review Automation Tasks
- Retrieval, search, or ranking tasks:Addressed in 17 studies. These focus on identifying and prioritizing relevant literature for systematic reviews or evidence synthesis.
- Generation tasks:Addressed in 11 studies. These include review, table, summary, reference, or abstract generation.
- Screening tasks:Addressed in 9 studies. These include citation, abstract, or full-text screening.
- Extraction tasks:Addressed in 5 studies. These include data, evidence, risk of bias, or meta-analysis.
- Meta-evaluation or meta-analysis:Addressed in 1 study.
- Other tasks:Addressed in 10 studies. These include query generation, compliance, information distillation, academic survey, trial design, systematic review-style reasoning, or multiple systematic literature review tasks.
Distribution of Evaluation Approaches
- Information retrieval metrics:Used in 27 studies. Metrics include precision, recall, F1, Mean Average Precision (MAP), normalized Discounted Cumulative Gain (nDCG), Mean Reciprocal Rank (MRR), Average Precision (AP), and others.
- Human evaluation:Used in 6 studies. Includes inter-annotator agreement, Cohen’s kappa, or explicit human ratings.
- Overlap, coverage, or consistency metrics:Used in 6 studies. Includes schema/cell overlap, coverage, factual consistency, or hallucination rate.
- Specialized or composite metrics:Used in 7 studies. Includes Elo ratings, CEDS, CQE, ECACT, or outcome difference.
- No mention found of evaluation approach:In 6 studies.
Some studies addressed multiple task types or used multiple evaluation approaches. We did not find mention of evaluation approach for 6 studies, and task type information was sometimes ambiguous due to overlapping or unclear descriptions.
Thematic Analysis
Benchmarks for Literature Retrieval and Screening Tasks
Benchmarks in this category focus on identifying, retrieving, and prioritizing relevant literature for systematic reviews or evidence synthesis. Key benchmarks include:
- LitSearch, CLEF Technology-Assisted Review (TAR) (2017–2019), CSMeD, SIGIR2017-PICO-Collection, FASS-BSLR, RELISH, and ResearchArena:These benchmarks use large-scale datasets such as PubMed, MEDLINE, and S2ORC, and evaluate models using recall, precision, Mean Average Precision, normalized Discounted Cumulative Gain, Work Saved over Sampling, and related metrics.
- Model types:Both classical information retrieval models (such as BM25 and term frequency-inverse document frequency) and modern transformer-based or large language model approaches are tested.
- Findings:Some studies (e.g., Kang and Xiong, 2024) report that keyword-based retrieval still outperforms large language model-based methods in certain settings, while others (e.g., Skarlinski et al., 2024) demonstrate large language models matching or exceeding human performance in specific retrieval and synthesis tasks.
Benchmarks for Literature Summarization and Synthesis Tasks
This theme covers benchmarks designed to evaluate the ability of models to generate summaries, reviews, or structured tables from collections of scientific papers. Notable benchmarks include:
- SciReviewGen, ARXIV2TABLE, MSLR2022, ScholarQABench, and Manalyzer:Evaluation metrics include ROUGE, human preference ratings, semantic coverage, factual consistency, and hallucination rates.
- Findings:Some large language models and transformer-based models achieve promising results, but challenges remain in factuality, citation accuracy, and domain adaptation. Human evaluation is often used to supplement automated metrics, reflecting the complexity of assessing synthesis quality.
Benchmarks for Structured Literature Analysis Tasks
Structured analysis tasks include risk of bias assessment, data element extraction, compliance and traceability, and meta-evaluation. Key benchmarks:
- RoBBR, EvidenceBench, ECACT, SciArena-Eval, and internal datasets:These benchmarks use composite or task-specific metrics such as ECACT score, macro-F1, accuracy, and Cohen’s kappa.
- Findings:Automation is advancing, but human oversight remains essential, especially in high-stakes or regulatory contexts.
Evaluation Methodologies Across Benchmarks
Benchmark Category
Primary Metrics
Evaluation Method
Performance Measurement Focus
Retrieval/Screening
Recall, precision, Mean Average Precision, normalized Discounted Cumulative Gain, Work Saved over Sampling, Average Precision
Automated scoring, comparison to human or baseline models
Relevance, workload reduction, ranking effectiveness
Summarization/Synthesis
ROUGE, human ratings, semantic coverage, hallucination rate
Automated metrics, human evaluation
Informativeness, factuality, coherence, citation accuracy
Structured Analysis
Macro-F1, accuracy, Cohen’s kappa, composite scores (ECACT)
Automated scoring, statistical analysis, human validation
Bias, compliance, extraction accuracy, traceability
Meta-evaluation
Elo ratings, accuracy, agreement
Human preference voting, model-based evaluator assessment
Alignment with human judgment, evaluator reliability
toof
Pageof
Summary of Evaluation Approaches:
- Primary metrics:Accuracy was the only metric used in more than one category (2 out of 4 categories). All other metrics were unique to a single category, including recall, precision, Mean Average Precision, normalized Discounted Cumulative Gain, Work Saved over Sampling, Average Precision, ROUGE, human ratings, semantic coverage, hallucination rate, macro-F1, Cohen’s kappa, composite scores (ECACT), Elo ratings, and agreement.
- Evaluation methods:Automated evaluation methods were used in 3 out of 4 categories. Human-based evaluation methods were used in all 4 categories. Comparison-based methods, statistical analysis, and model-based evaluator assessment were each used in 1 out of 4 categories.
- Coverage:We did not find any categories without at least one automated and one human-based evaluation method. No information was missing for any category.
References
Download BIBDownload RISDownload TXT
Report
Status
Gather sources
1000 sources found
Details
Screen sources
40 sources included
Details
Extract data
160 data points extracted
Details
Generate report
Save PDF
BIBLaTeX, ZoteroRISZotero, MendeleyTXTAPA BibliographyPDFPDFDOCXMicrosoft Word
Chat
Got some follow-up questions?
Sign up or sign in to chat with this report.
Back
Can LLMs Generate Tabular Summaries of Science Papers? Rethinking the Evaluation Protocol
Weiqi Wang, Jiefu Ou, Yangqiu Song, Benjamin Van Durme, Daniel Khashabi
arXiv.org·
2025·
2 citations
SourceDOI
Plain textPDFFigures
Searching for PDF
Unable to find PDF from source
Back
Benchmark Type
Literature Review Table Generation Benchmark (ARXIV2TABLE)
Benchmark Dataset Characteristics
1,957 tables and 7,158 papers from scientific research, particularly in computer science, built upon the ARX-IVDIGESTABLES dataset
Evaluation Metrics
- Recall for paper selection - Precision (P), recall (R), and F1 scores (F1) for schema content overlap, single-cell value overlap, and pairwise value comparisons - Ratio of 'yes' answers for schema, individual values, and pairwise relationships
Benchmark Models and Comparisons
- Models tested: GPT-4o, GPT-4o-mini, DeepSeek-V3, LLAMA-3.3, Mistral-Large - Performance relative to the benchmark: All models struggle with distinguishing relevant papers from distractors. - Comparative analysis: The proposed method improves performance across all aspects and models. Larger models lead to better performance.
Literature review tables are essential for summarizing and comparing collections of scientific papers. We explore the task of generating tables that best fulfill a user's informational needs given a collection of scientific papers. Building on recent work (Newman et al., 2024), we extend prior approaches to address real-world complexities through a combination of LLM-based methods and human annotations. Our contributions focus on three key challenges encountered in real-world use: (i) User prompts are often under-specified; (ii) Retrieved candidate papers frequently contain irrelevant content; and (iii) Task evaluation should move beyond shallow text similarity techniques and instead assess the utility of inferred tables for information-seeking tasks (e.g., comparing papers). To support reproducible evaluation, we introduce ARXIV2TABLE, a more realistic and challenging benchmark for this task, along with a novel approach to improve literature review table generation in real-world scenarios. Our extensive experiments on this benchmark show that both open-weight and proprietary LLMs struggle with the task, highlighting its difficulty and the need for further advancements. Our dataset and code are available at https://github.com/JHU-CLSP/arXiv2Table.
1Introduction
Literature review tables play a crucial role in scientific research by organizing and summarizing large amounts of information from selected papers into a concise and comparable format (Russell et al., 1993). At the core of these tables are the schema and values that define their structure, where schema refers to the categories or aspects used to summarize different papers and values correspond to the specific information extracted from each paper. A well-defined schema allows each work to be represented as a row of values, enabling structured and transparent comparisons across different studies.
(T2) Paper Selection (T3) Table Induction: Schema + Value Generation
Generate a table that analyzes various image-guided fashion retrieval methods, highlighting the evolution of network architectures, the datasets they were evaluated on, the evaluation criteria used, and the specific loss functions applied. The table should provide a detailed comparison of these aspects to facilitate understanding of differences across the listed papers.
(T1) Candidate paper retrieval IR User intent prompt inferred table
Figure 1 : Overview of our proposed task: Given a user's demand, a simulated information retrieval (IR) engine first retrieves semantically relevant papers. Then, a language model further filters them and induces the table's corresponding schema and values to satisfy the user's demand. The grayed region indicates the scope covered by our method and benchmark (ARXIV2TABLE).
With recent advancements in large language models (LLMs; OpenAI, 2025b; DeepSeek-AI et al., 2025) , several studies (Newman et al., 2024;Dagdelen et al., 2024; Sun et al., 2024) have explored generating literature review tables by prompting LLMs with a set of pre-selected papers and the table's caption. While these efforts represent meaningful progress, we argue that the existing task definition and evaluation protocols are somewhat unrealistic, thus hindering the practical applicability of generation methods.
First, existing pipelines assume that all provided papers are relevant and should be included in the table. However, in real-world scenarios, distractor papers-those that are irrelevant or contain limited useful information-are common (OpenAI, 2025a) . Models should be able to identify and filter out such papers before table construction. Additionally, current pipelines use the ground-truth table's descriptive caption as the objective for generation. These captions often lack sufficient context, making it difficult for LLMs to infer an appropriate schema, or they may inadvertently reveal the schema and values, leading to biased evaluations.
In this paper, we introduce our task, as illustrated in Figure 1 , which improves upon previous task definitions through two key adaptations. First, our pilot study shows that LLMs struggle to retrieve relevant papers from large corpora. To benchmark this, we introduce distractor papers by selecting them based on semantic similarity to papers in the ground-truth table. LLMs must first determine which papers should be included before generating the table. Second, we replace table captions with abstract user demands that describe the goal of curating the table, making the task more aligned with real-world scenarios. We build upon the ARX-IVDIGESTABLES (Newman et al., 2024) dataset and construct a sibling benchmark through human annotation to verify the selected distractors, comprising 1,957 tables and 7,158 papers.
Meanwhile, current evaluation methods rely on static semantic embeddings to estimate schema overlap between generated and ground-truth tables and require human annotations to assess the quality of unseen schemas and values. However, semantic embeddings struggle to capture nuanced, context-specific variations due to their reliance on pre-trained representations, while human annotation is costly and time-consuming. Moreover, the most effective table generation approaches define schemas primarily based on paper abstracts. This method risks missing important aspects present in the full text, leading to loosely defined schemas with inconsistent granularity.
To address these issues, we propose an annotation-free evaluation framework that instructs an LLM to synthesize QA pairs based on the ground-truth table and assess the generated table by answering these questions. These QA pairs evaluate table content overlap across three dimensions: schema-level, single-cell, and pairwise-cell comparisons. Additionally, we introduce a novel table generation method that batches input papers, iteratively refining paper selection and schema definition by revisiting each paper multiple times. Extensive experiments using five LLMs demonstrate that they struggle with both selecting relevant papers and generating high-quality tables, while our method significantly improves performance on both fronts. Expert validation further confirms the reliability of our QA-synthetic evaluations.
In summary, our contributions are threefold: (1) We introduce an improved task definition for literature review tabular generation, benchmarking it in a more realistic scenario by incorporating dis-tractor papers and replacing table captions with abstract user demands; (2) We propose an annotationfree evaluation framework that leverages LLMgenerated QA pairs to assess schema-level, singlecell, and pairwise-cell content overlap, addressing the limitations of static semantic embeddings and human evaluation; and (3) We develop a novel iterative batch-based table generation method that processes input papers in batches, refining schema definition and paper selection iteratively.
To the best of our knowledge, we are the first to introduce a task that simulates real-world use cases of scientific tabular generation by incorporating user demands and distractor papers, providing a more robust assessment of LLMs in this domain.
2Related Works
Scientific literature tabular generation Prior works primarily attempt to generate scientific tables through two stages: schema induction and value extraction. For schema induction, early methods like entity-based table generation (Zhang and Balog, 2018) focused on structured input, while recent work has explored schema induction from user queries (Wang et al., 2024) and comparative aspect extraction (Hashimoto et al., 2017). For value extraction, various approaches such as documentgrounded question-answering (Kwiatkowski et al., 2019;Dasigi et al., 2021;Lee et al., 2023), aspectbased summarization (Ahuja et al., 2022), and document summarization (DeYoung et al., 2021;Lu et al., 2020) have been proposed to extract relevant information. Beyond these methods, several datasets have been introduced to support scientific table-related tasks, such as TableBank (Li et al., 2020 ), SciGen (Moosavi et al., 2021) , and Sc-iTabQA (Lu et al., 2023). Recently, Newman et al. (2024) proposed streamlining schema and value generation with LLMs sequentially and curated a large-scale benchmark for evaluation. However, all these methods assume a clean and fully relevant set of papers and rely on predefined captions or abstract-based schemas, which risk missing key details. In contrast, we argue for an evaluation approach where candidate papers include tangentially relevant or distracting papers, aligning more closely with real-world literature review workflows. (Li et al., 2023;Wu et al., 2022) or as a question-answering problem (Sundar et al., 2024;Tang et al., 2023). Similar to these works, our framework is capable of better handling both structured and distractive input for real-world literature review and knowledge synthesis.
3Task Definition
We first define a pipeline consisting of three subtasks that extend prior definitions and better capture the real-world usage of literature review tabular generation. For all the following tasks, we are given a user demand prompt p, which specifies the intended purpose of creating the table. (T1) Candidate Paper Retrieval: We begin with a given universe of papers (e.g., the content of Google Scholar or arXiv) from which relevant papers need to be identified. Given a large collection, the goal is to use a search engine (IR) to retrieve a subset of candidate papers C := {d i } M i=1 of size M , which may include distractor papers-i.e., papers that resemble the user demand prompt but do not fully satisfy the requirement. (T2) Paper Selection: Given C, the second subtask is to select the relevant subset of size m (m < M ): R := {d i } m i=1 ⊆ C, which best aligns with the user demand p. T2 differs from T1 in scale. Due to the large scale of T1, IR engines must optimize for recall, ensuring that as many relevant papers as possible are retrieved. However, T2 operates at a smaller scale, where precision is the priority, as it focuses on filtering out distractors and selecting only the most relevant papers. (T3) Table Induction: Given the selected papers R, the objective is to generate a table with m rows and N columns, where N ≥ 2 (i.e., no singlecolumn tables). Each row r i ∈ {r 1 , r 2 , . . . , r m } corresponds to a unique input document d i ∈ R, and each column c j ∈ {c 1 , c 2 , . . . , c N } represents a unique aspect of the documents. We refer to these N columns as the schema of the table and the N × m cells as the values of the table. The value of each cell is derived from its respective document according to the aspect defined by the corresponding column.
4ARXIV2TABLE Construction
We then construct ARXIV2TABLE based on the ARXIVDIGESTABLES dataset which consists of literature tables (extracted from computer science papers) and their corresponding captions. We filter out tables that are structurally incomplete or lack full text for all referenced papers. As a result, we are left with 1,957 tables (with captions) which have rows referring to 7,158 papers. Our construction involves three pillars: user demand inference ( §4.1), a simulated paper retrieval ( §4.2) and evaluation through utilization ( §4.3).
4.1Constructing User Demand Prompts
The first step is to collect user demands p that explicitly describe the desired table (can be understood without the table content) and do not reveal the table's schema or specific values.
Table captions are not appropriate prompts
While the input dataset contains one caption per table, collected from arXiv papers, these captions are meant to complement tables rather than fully describe them. As a result, they are generally concise. For example, a table caption might read: "Performance comparison of different approaches," which is too vague to understand without seeing the table. Consequently, using table captions as prompts may not yield a well-defined task. A more contextually self-contained rewritten user demand might instead be: "Draft a table that compares different knowledge editing methods, focusing on their performance on QA datasets."
Our prompt construction To address this issue, we propose rewriting the captions of literature review tables into abstract yet descriptive user intentions using LLMs. We guide GPT-4o with a prompt (see §A) that first explains the task to the LLM, specifying that the user demand should be sufficiently contextualized to clearly state the table's purpose while avoiding the inclusion or direct description of column names or specific values. GPT-4o is then expected to infer the user demand for the given table and its caption. Here, LLM is used solely for rewriting existing table captions into user demand prompts and for generating QA pairs grounded in ground-truth tables. These reformulations are strictly tied to observed data and do not require external factual knowledge, minimizing risks of contamination or model-specific bias. For simplicity, we collect only one user demand per table . More examples are provided in Appendix C.
Table captions vs. constructed user demand prompts
To verify that our collected user demands align with our objective, we visualize: (1) the distribution of the number of tokens in the original and modified user demands, and (2) the ratio of captions and user demands of different lengths that have token overlap with the schema or values. From Figure 2 , we observe that our modified user demands are generally longer than the original captions, providing a more detailed description of the table's goal. Furthermore, as shown in Table 1 , user demands exhibit a significantly lower overlap ratio with the schema and table values, resulting in fewer overlapping tokens. This ensures a fairer subsequent evaluation.
1XPEHURI7RNHQV )UHTXHQF\RI7DEOHV
&DSWLRQ/HQJWK 8VHU'HPDQG/HQJWK
4.2Paper Retrieval Simulation
The unreliability of paper retrieval Next, we approach the first subtask, candidate paper retrieval, by conducting a pilot study to assess whether LMs can reliably retrieve relevant papers from a large corpus. For each table, we employ a Sentence-BERT (Reimers and Gurevych, 2019) encoder as a retrieval engine, selecting papers from the entire corpus based on the highest similarity between the table's user demand and each paper's title and abstract. We vary the number of retrieved papers between 2 and 100 and plot the precision and recall of retrieval against the ground-truth papers in the original table (Figure 3 ). We observe consistently low precision and recall across different retrieval sizes, highlighting the challenge of retrieving relevant papers from a noisy corpus. This demonstrates that the first subtask is non-trivial and may introduce noise into subtask T2. However, various information retrieval engines, such as Google Scholar and Semantic Scholar, can replace LMs in this subtask. Thus, we decide to simulate T1 by manually adding noisy distractor papers into C to construct R, ensuring a noisy input for T2. This allows us to focus on evaluating LLMs' capabilities in the T2 and T3 subtasks. each table to simulate a potentially noisy document pool before constructing the table. Ideally, distractor candidates should be semantically related to the table but exhibit key differences that fail to meet the user demand. To select such candidates, we adopt a retrieve-then-annotate approach. First, we use a SentenceBERT encoder F to obtain embeddings for (1) the user demand F (p) and (2) all papers in the corpus {F (d i ) | d i ∈ C}. Each paper's embedding is computed by encoding the concatenation of its title and abstract. We then rank all papers d i / ∈ R based on the average of two cosine similarities: (1) the similarity between the candidate and the user demand, and (2) the average similarity between the candidate and each referenced paper:
Similarity-based paper retrieval
Higher values of s(d i ) indicate stronger semantic relevance, and we select the top 10 ranked papers for each table as its distractor candidates.
Candidates verification via human annotation
After selecting these candidates, we conduct human annotations to verify whether they should indeed be excluded from the table. Given that annotating these tables requires expert knowledge in computer science, we recruit seven postgraduate students with research experience in the field as annotators. To ensure they are well-prepared for the task, the annotators undergo rigorous training, including pilot annotation exams. Their task is to make a binary decision on whether a given distractor paper-based on its title, abstract, user demand, the ground-truth table, and the titles and abstracts of all referenced papers-should be included in the table. Each table contains annotations for 10 papers, with each distractor paper initially assigned to two randomly selected annotators. If both annotators agree on the label, it is finalized. Otherwise, two additional annotators review the paper until a consensus is reached. In the first round, the inter-annotator agreement (IAA) is 94% based on pairwise agreement, and the Fleiss' Kappa (Fleiss, 1971) score is 0.73, indicating a substantial level of agreement (Landis and Koch, 1977). Finally, for each table, we randomly select a number of distractor papers between [m, 10] and merge them with R to form C.
4.3Evaluation via LLM-based Utilization
After constructing the benchmark, we propose evaluating the quality of generated tables from a utilization perspective to address the challenge of aligning schemas and values despite potential differences in phrasing. This is achieved by synthesizing QA pairs based on the ground-truth table and using the generated table to answer them, or vice versa. The flexibility of this QA synthesis allows us to evaluate multiple dimensions of the table while ensuring a structured and scalable assessment. An overview with running examples is shown in Figure 4 .
Dimensions of evaluating a table with QAs
We introduce three key aspects for evaluating a table in terms of its usability: (1) Schema: whether a specific column is included in the generated schema, (2) Unary Value: whether a particular cell from the ground-truth table appears in the generated table, and (3) Pairwise Value: whether relationships between two cells remain consistent in the generated table.
Recall evaluation We guide GPT-4o in generating these binary QA pairs based on the groundtruth table. For the first two aspects, we generate QA pairs for all columns and cells, whereas for the third aspect, we randomly sample 10 pairs of cells per table and synthesize them into QA pairs. We then prompt GPT-4o to answer these questions based on the generated table, providing yes/no responses. If the answer cannot be found, the model is instructed to respond with "no," and vice versa for "yes." The ratio of "yes" answers indicates how well the generated table preserves the schema, in-dividual values, and pairwise relationships. This represents the recall of the ground-truth
The ratio of "correct" indicates the recall.
5Tabular Generation Methodologies
We explore a range of methods to evaluate on our proposed task, starting from several baselines inspired by prior work ( §5.1) and then our proposed approach ( §5.2).
5.1Baseline Methods
We first introduce three methods for generating literature review tables to evaluate their performance on our task and use them as baselines for our proposed method. For easy reference, these methods are termed numerically. First, Baseline 1 generates the table in a one-step process. It takes all available papers R and the user demand p as input, and the model is asked to select all relevant papers and output a table with a welldefined schema and filled values in a single round of conversation. However, this method struggles with extremely long prompts that exceed the LLMs' context window when generating large tables.
To address this issue, Baseline 2 processes papers individually. For each document, the model decides whether it should be included based on the user demand. If included, the model generates a table for that document. After processing all documents, the final table is created by merging the schemas of all individual tables using exact string matching and copying the corresponding values. While this approach reduces the input prompt length, it results in highly sparse tables due to inconsistent schema across papers and the potential omission of relevant information when individual papers lack sufficient context to define comprehensive table aspects.
To overcome both issues, Newman et al. (Newman et al., 2024) introduces a two-stage process. In the first stage, the model selects papers relevant to the user demand based on their titles and abstracts, then generates a corresponding schema. In the second stage, the model loops through the selected papers and fills in the respective rows based on the full text of each document. A minor drawback of this method is that the schema is generated solely from titles and abstracts, which may overlook details present only in the full text. Note that this method is the strongest recent baseline for scientific tabular generation while other text-to-table methods (Deng et al., 2024b) are not directly applicable due to different assumptions.
5.2Iterative Batch-based Tabular Generation
Then, we introduce our proposed method for generating literature review tables. Our approach consists of three steps: (A) key information extraction, (B) paper batching, and (C) paper selection and schema refinement, where the latter two steps can be iterated multiple times.
(A) Key Information Extraction Processing multiple papers simultaneously using their full text often results in excessively long prompts that exceed the LLMs' context window. To address this, we first shorten each paper by instructing the LLM to extract key information from the full text that is relevant to the user's requirements. Notably, we do not rely solely on the abstract, as important details often appear in the full text but are omitted from the abstract. For each paper, we provide the LLM with its title, abstract, and full text, along with the user's request, and ask it to generate a concise paragraph that preserves all potentially relevant details. These summary paragraphs serve as condensed representations of the papers for subsequent processing.
(B) Paper Batching Next, we divide all key information paragraphs into smaller batches. Processing too many papers at once negatively affects the model's performance (as demonstrated by the comparison of Baseline 1 in Table 2 ), whereas batching facilitates more efficient comparisons within each batch. For simplicity, we set a batch size of 4 and randomly partition R into |R| 4 batches.
(C) Paper Selection and Schema Refinement
We initialize an empty schema and table, then sequentially process each batch with the LLM by providing it with the user's request and summaries of batched papers. The LLM is instructed to (1) decide whether each paper should be included or removed based on its key information and (2) refine the schema based on the current batch of papers. Schema refinement involves adding or removing specific columns or modifying existing values to align with different formats. For new papers that are not deemed suitable for inclusion yet are not in the current table, we also prompt the LLM to insert a new row according to the refined schema. This ensures that the table remains dynamically structured, continuously adapting to new information while maintaining consistency across batches.
Afterward, we iterate steps B and C for k iterations. Here k is a hyper-parameter and we set k = 5 in our experiments. The rationale is that multiple iterations allow the schema and table contents to progressively improve, ensuring better alignment with user demands. In each iteration, the batches are newly randomized so that each paper is compared with different subsets, enabling more robust decision-making and reducing bias from specific batch compositions. This iterative refinement also mitigates errors from earlier batches by revisiting and adjusting prior decisions based on newly processed information. After completing all iterations, we individually prompt the LLM to revisit the full text of the selected papers to verify the values, thereby completing the tabular generation process.
6 Experiments and Analyses
6.1Experiment Setup
To demonstrate the generalizability of our method and evaluations, we conduct experiments using two proprietary and three open-source LLMs as backbone model representatives: GPT-4o (OpenAI, 2024b), GPT-4o-mini (OpenAI, 2024a), DeepSeek-V3 (685B; DeepSeek-AI et al., 2024), LLAMA-3.3 (70B; Dubey et al., 2024) , and Mistral-Large (123B; Mistral-AI, 2024). We apply all baseline methods and our proposed method to each model and use our evaluation framework to assess the quality of the generated tables based on our benchmark, focusing on four aspects: paper selection (Paper), schema content overlap (Schema), singlecell value overlap (Unary Value), and comparisons across cells (Pairwise Value). For paper selection, we use recall as the metric to measure the number of ground-truth papers successfully selected 1 . For the latter three tasks, we report precision (P), recall (R), and F1 scores (F1), as explained in §4.3.
6.2Main Evaluation Results
We report the main evaluation results in Table 2 and summarize our key findings as follows. A vi-1 Recall is emphasized instead of precision since missing relevant papers is more critical than including slightly noisy ones in literature review tasks.
sual comparison of model-wise performance across methods is also provided in Figure 5 .
(1) All methods and models struggle to distinguish relevant papers from distractors. For example, even with their best-performing methods, LLAMA-3.3 and GPT-4o achieve only 65.4% and 71.3% recall on average, respectively. This indicates that a significant number of distractor papers are still being included in the generated tables. Additionally, we observe that processing papers individually or using only abstracts for inclusion decisions yields better performance than concatenating full texts. This suggests that excessively long prompts may weaken LLMs' ability to make accurate inclusion decisions for each paper.
(2) Aligning generated schemas with the groundtruth table remains challenging. Among the baselines, the second method consistently achieves higher recall (e.g., 69.3% with LLAMA-3.3), primarily because it generates a larger number of columns, leading to more overlaps with the groundtruth schema. However, other methods exhibit significantly lower recall, indicating that LLMs still struggle to generate meaningful columns that align well with the ground-truth structure.
(3) While unary values are well preserved, pairwise comparisons suffer substantial losses. Most methods, especially our proposed approach, extract unary values with relatively high F1 scores. However, extracting and maintaining pairwise rela- tionships remains challenging. For instance, using LLAMA-3.3, our method achieves a unary F1 score of 51.1 but drops to 41.0 for pairwise values. This trend is consistent across different models, suggesting that while individual entries are correctly identified, capturing the relationships between them remains difficult. The significant gap highlights the challenge of preserving complex relational comparisons within the generated tables.
(4) Our proposed method improves performance across all aspects and models. Across all backbone models and evaluation criteria, our method consistently outperforms the baselines. For example, it achieves the highest recall and F1 scores for both unary and pairwise metrics, regardless of model size. This demonstrates that our approach not only enhances overall performance but also provides a more robust solution for handling distractor paper selection and precise table generation.
(5) Larger models lead to better performance.
For the three open-source LLMs, we observe a clear trend that increasing the model size improves performance across all aspects when using the same method. For instance, with our approach, scaling from 70B to 123B parameters leads to consistent improvements in most aspects and metrics, reinforcing the importance of stronger generative capabilities in addressing this task.
6.3Ablation Study on Iteration Number
We further study the impact of the number of iterations, k, in our proposed method to illustrate the importance of refining the schema and table contents over multiple iterations using different batches of papers. As described in §5.2, we perform one round of paper selection and schema refinement five times to achieve optimal performance.
In this section, we analyze this process by studying the model's performance across previous rounds. We select GPT-4o as the backbone model and visualize changes in the F1 scores for schema, unary value, pairwise comparison overlap, and their average, by applying the same evaluation protocol to the generated tables across iterations ranging from 1 (the first cycle) to 5.
The results are plotted in Figure 6 . We observe that during the first four iterations, performance steadily improves across all aspects, demonstrating the effectiveness of iteratively refining paper selection and table schema through multiple iterations and comparisons between different subsets of papers. At the fifth iteration, however, the improvement slows down, and in some cases, performance even decreases. One possible reason is that the table starts overfitting by including additional values that do not appear in the ground-truth table, reducing precision and leading to lower F1 scores. Considering the overall performance, k = 5 is supported as the optimal number of iterations.
,WHUDWLRQV 3HUIRUPDQFH 6FKHPD 8QDU\9DOXH
3DLUZLVH9DOXH $YHUDJH
Figure 6 : Ablation study on the number of iterations for our iterative batch-based table generation method.
6.4Validation of Utilization-Based Evaluation
To verify the reliability of synthesizing QA pairs using LLMs for evaluating tabular data, we conduct two complementary expert assessments. First, we invited the authors (as domain experts) to manually inspect a random sample of 200 QA pairs-spanning schema-level, unary value, and pairwise value comparisons. Annotators were asked to assess (1) whether each QA pair is firmly grounded in the source table, and (2) whether the LLM's answer is correct based on the generated target table. As shown in Table 3 , the expert acceptance rates exceed 98% in all categories, confirming the quality of the synthesized QA pairs. Second, we conducted an additional human study to assess whether our LLM-based evaluation aligns with human judgment across different generation methods. For each method, we sampled 300 QA pairs, answered them using both LLMs and human annotators, and measured the agreement rate. As shown in Table 4 , LLM and human "yes" response rates are highly consistent, with over 97% agreement across all methods. These results reinforce the robustness of our evaluation framework, demonstrating that LLM-synthesized QA pairs provide a scalable and trustworthy proxy for human judgment in assessing semantically diverse tabular outputs. Specifically, these results indicate that the high agreement is not driven by an inherent bias of LLMs toward their own generated QA pairs. Table 4 : Comparison between GPT-4o and human annotators on 300 QA pairs per method. We report the proportion of "yes" answers by each and their overall agreement.
6.5Computational Cost Comparisons
To assess the efficiency and scalability of our iterative batch-based method, we report computational statistics in Table 5 . Each method was run using the same LLaMA-3.3 model backend. We measure three aspects: (1) generation success rate, defined as the proportion of prompts yielding complete tables within the context window, (2) average token usage per table, and (3) average runtime per table.
Our method achieves a 100% success rate, outperforming the baselines that occasionally fail due to context limitations or prompt instability. While our runtime is moderately longer than Baseline 1 and Baseline 2, it remains comparable to Newman et al. and stays well within acceptable latency for practical usage. Furthermore, token usage remains controlled, confirming that our iterative approach does not incur excessive computational cost despite its multi-step structure. These results demonstrate that our method offers a favorable trade-off between performance and efficiency.
7Conclusion
In this work, we introduce an improved literature review table generation task that incorporates distractor papers and replaces table captions with abstract user demands to better align with real-world scenarios, and curated an associated benchmark. Additionally, we propose an annotation-free evaluation framework using LLM-synthesized QA pairs and a novel method to enhance table generation.
Our experiments show that current LLMs and existing methods struggle with our task, while our approach significantly improves performance. We envision that our work paves the way for more automated and scalable literature review table generation, ultimately facilitating the efficient synthesis of scientific knowledge in large-scale applications.
Limitations
A minor limitation is that our work uses ARXIVDI-GESTABLES as the source of literature review tables for subsequent data reconstruction. However, Newman et al. (2024) have included their pipeline for scalably extracting literature review tables from scientific papers, thus resolving the data reliance gap. Beyond the computer science domain, our formulation and methodology are readily applicable to other scientific fields such as medicine, physics, and social sciences, where structured comparisons across publications are equally valuable. Moreover, the core task-generating structured tables from noisy, unstructured input with under-specified intent-extends naturally to real-world applications like news fact aggregation, personalized knowledge card generation, and structured database population from web or legal documents. Another limitation of our work is its reliance on GPT-4o, a proprietary LLM, for benchmark curation and subsequent evaluation, which may introduce several issues. First, it raises concerns about data contamination (Deng et al., 2024a; Dong et al., 2024), as the model may generate user demands (during benchmark curation) and synthesis evaluation questions (when evaluating a generated table against the ground truth) that are similar to its training data, potentially leading to inflated performance in table generation. A data provenance check (Longpre et al., 2024) can be further implemented to address this issue. Second, the benchmark and evaluation process may inherit the internal knowledge or semantic distribution biases of GPT-4o, which could skew the evaluation of other LLMs and reduce the generalizability of our findings. Lastly, a minor issue is scalability, as curating larger datasets using a proprietary model can be resource-intensive and may limit accessibility when extending our framework to other literature or domains. Future work can explore the use of open-source LLMs to replicate the entire process for convenient adaptation to other tabular datasets.
our dataset is safe and will not yield any negative or harmful impact.
Our human annotations are conducted by recruiting five graduate-level students who have sufficient experience in data collection for training large language models. They are proficient in English, primarily from Asia, and are paid above the minimum wage in their local jurisdictions. They receive thorough training on the task and are reminded to have a clear understanding of the task instructions before proceeding to annotation. The high level of inter-agreement also confirms the quality of our annotation. The expert annotators have agreed to participate as their contribution to the paper without receiving any compensation.
Appendices A Implementation Details
In this section, we provide additional implementation details about our benchmark curation and evaluation pipeline, including the prompt we used and the models we accessed.
A.1 Prompts Used
We first introduce the prompt used to construct the ARXIV2TABLE benchmark, as explained in Section 4. The main step involves prompting LLM is to collect user demands that describe the purpose of creating the table while remaining contextually self-contained and not revealing the actual schema or values of the table. We use the following prompt to instruct GPT-4o in generating these user demands. The distribution of number of papers per table in ARXIV2TABLE is shown in Figure 7 .
A.2 Evaluation Implementations
We access all open-source LLMs via the Hugging Face library (Wolf et al., 2020). The models used are meta-llama/Llama-3.3-70B-Instruct, mistralai/Mistral-Large-Instruct-2411, and deepseek-ai/DeepSeek-V3.
For GPT models, we access them via the official OpenAI Batch API3 . The models used are gpt-4o-mini-2024-07-18 and gpt-4o-2024-08-06.
Note that the DeepSeek model family has a context window limit of 64K tokens, whereas the others have a limit of 128K tokens. The generation temperature is set to 0.5 for all experiments. All experiments are repeated twice and the average performance is reported.
B Annotation Details
To ensure the high quality of our human annotations, we implement strict quality control measures. First, we select only postgraduate students with research experience in computer science to ensure they are familiar with relevant topics. All selected annotators undergo qualification rounds, and we invite only those who demonstrate satisfactory performance to serve as our main annotators.
For each task, we provide workers with comprehensive task explanations in layman's terms to enhance their understanding. Additionally, we offer detailed definitions and multiple examples for each choice to help annotators make informed decisions. Each entry requires the worker to provide a binary vote on whether the paper should be excluded or not. Our annotation interface is shown in Figure 8 .
To ensure comprehension, we require annotators to confirm that they have thoroughly read the instructions by ticking a checkbox before starting the annotation task. We also manually monitor the performance of annotators throughout the annotation process and provide feedback based on common errors. Spammers or underperforming workers are disqualified. As described in Section 4.2, the interannotator agreement supports the quality of our collected annotations. Table 8 illustrates schema, unary value, and pairwise value questions designed to assess the quality of generated tables, ensuring alignment with ground-truth information. The results reveal that this QA-based evaluation effectively quantifies schema retention, individual value accuracy, and consistency in relationships, providing a structured approach for benchmarking table generation models.
C Case Studies
In addition, we present two pairs of ground-truth and generated tables as examples for a case study on table generation, as shown in Table 9 . From these tables, we observe that the generation process is capable of incorporating many useful columns, thereby enriching the available information. For instance, in the first example, the generated table introduces new columns such as Number of Images, Number of Subjects, and Avg. Images per Subject, which add valuable quantitative insights beyond the original ground truth table. However, it is also evident that some columns present in the ground truth, like the Evaluation Metric, are not fully covered in the generated version. In the second example, the user demand for detailed descriptions has led to a generated table with numerous specific columns, including ID, Method Used, Performance Metric, and Results Achieved. Although these additional details enhance the descriptive quality of the table, they also suggest a potential issue: the need for further polishing and refinement of the user demand to balance detail with clarity.
Ethics Statement
The ARXIVDIGESTABLES (Newman et al., 2024) dataset used in our work is shared under the Open Data Commons License, which grants us access to it and allows us to improve and redistribute it for research purposes. Regarding language models, we access all open-source LMs via the Hugging Face Hub (Wolf et al., 2020) and proprietary GPT models through their official API 2 . The number of these models, if available, is marked in Table 2 . All associated licenses for these models permit user access for research purposes, and we commit to following all terms of use.
When prompting GPT-4o to generate user demands and synthetic QA questions, we explicitly state in the prompt that the LLM should not generate any content that contains personal privacy violations, promotes violence, racial discrimination, hate speech, sexual, or self-harm contents. We also manually inspect a random sample of 100 data entries generated by GPT-4o for offensive content, and none are detected. Therefore, we believe that
Original Table Caption
User Demand
Comparison of Trajectory and Path Planing Approach
Generate a table that compares different trajectory and path planning approaches, focusing on their collision avoidance techniques, benefits, limitations, and applicable scenarios. The table should include detailed columns to capture these aspects for each method mentioned in the relevant papers.
Publications with deep-learning focused sampling methods. We cluster the papers based on the space the sample through and how the samples are evaluated. Some approaches further consider an optional refinement stage.
Create a table that categorizes publications focused on deep-learning-based sampling methods for grasp detection, organizing them by the space in which samples are generated, the evaluation criteria used, and whether a refinement stage is included. The table should provide a comprehensive yet concise overview of the methodological variations and enhancements across different papers.
Categorization of textual explanation methods.
Create a table that categorizes the methods used for providing textual explanations in visual question answering systems, focusing on the types of texts generated and the reasoning processes employed. The table should use succinct columns to differentiate between these methodological aspects for each paper.
Metadata of the three benchmarks that we focus on. XSumSota is a combined benchmark of cite:1400aac and cite:d420ef8 for summaries generated by the state-of-the-art summarization models.
Create a table that details the metadata for three summarization benchmarks, focusing on the composition of annotators, the dataset sizes for validation and testing, and the distribution of positive and negative evaluations. The table should provide a comprehensive comparison across these aspects for each benchmark.
Review of open access ground-based forest datasets
Create a table that reviews various open-access forest datasets, focusing on the publication and data recording years, types of data collected, and their applicability to specific forestryrelated tasks. The table should offer a concise summary of each dataset's attributes, including the number of classification categories and geographical location.
Comparison of existing consistencytype models.
Create a table that compares different models focusing on their purpose, the trajectory they follow, the main objects they equate, and their methodological approach. The table should provide detailed insights into how each model addresses consistency issues, drawing from specified papers.
Failed to load PDF:
StripeM-Inner
Paper sources
Abstract screening pilot
Abstract screening results
Extraction pilot
Extraction results
Research report
Modify setup