PaperQA2: language agents achieving superhuman synthesis of scientific knowledge

Workflow & human oversight

A literature agent coordinates multi-step retrieval and synthesis over scientific papers. Humans define questions, provide corpus access and must check citations and conclusions.

Workflow level is not an evidence score; higher is not better evidence.

Problem

Language models hallucinate, and it is unclear whether they are accurate and reliable enough for scientific literature work. The authors ask how an agent optimized for factuality compares against subject-matter experts given full access to the internet, search tools and time.

Approach

The authors build a rigorous human-AI comparison methodology and a hard literature-research benchmark (LitQA2) that guided PaperQA2's design. PaperQA2 is then applied to contradiction detection across a random subset of biology papers.

Outcomes (as reported by the originating team)

The authors report PaperQA2 matches or exceeds expert performance on three literature research tasks, produces cited summaries more accurate than existing Wikipedia articles, and identifies 2.34 +/- 1.99 contradictions per paper in a random subset of biology papers, 70% of which were validated by human experts (arXiv 2409.13740, 2024).

Limitations

Preprint: not peer-reviewed at source-check time. The human-expert comparisons and contradiction validation were run by the authors themselves. Contradiction detection is demonstrated on biology papers; generality to other domains is untested here.

Reproducibility

The paper and benchmark are public on arXiv; code is published at the linked repository (link verified live 2026-10-15). Running the agent requires model API access and literature-database licensing in some cases.

Adoption requirements

Model API access and a literature corpus (e.g. PDF collections or licensed databases); willingness to spot-check citations, since the core claim is improved factuality on cited summaries, not infallibility.