RobotReviewer: assisting risk-of-bias assessment in clinical trial reports

Workflow & human oversight

Bounded task assistance: the model predicts risk-of-bias domains and selects supporting sentences from trial reports. Human systematic reviewers check the evidence and make final assessments; it does not conduct an entire review.

Workflow level is not an evidence score; higher is not better evidence.

Problem

Systematic reviewers must assess trial bias and locate supporting passages, a time-consuming component of research synthesis.

Approach

The authors trained a multi-task model using 12,808 trial PDFs algorithmically annotated from Cochrane reviews; 20 blinded experienced reviewers assessed supporting-text relevance.

Outcomes (as reported by the originating team)

The authors report overall risk-of-bias judgment accuracy of 71.0% for the model versus 78.3% for published-review judgments. Supporting-text relevance was comparable in their evaluation; the apparent advantage was not statistically significant. Potential workload reduction is not a measured productivity gain here.

Limitations

Model judgments were less accurate than published-review judgments. The evaluation concerns clinical trial bias assessment, not general research correctness; expert review remains necessary.

Reproducibility

The primary paper describes training and evaluation, including comparison data from Cochrane reviews. Reproduction depends on access to the relevant corpus and modeling setup; not rerun by this catalogue.

Adoption requirements

Clinical trial PDFs, bias-assessment expertise, access to an appropriate implementation and human checks on predictions and evidence passages.