Scholarly peer review is a cornerstone of scientific advancement, but the system is under strain due to increasing manuscript submissions and the labor-intensive nature of the process. Recent advancements in large language models (LLMs) have led to their integration into peer review, with promising results such as substantial overlaps between LLM- and human-generated reviews. However, the unchecked adoption of LLMs poses significant risks to the integrity of the peer review system. In this study, we comprehensively analyze the vulnerabilities of LLM-generated reviews by focusing on manipulation and inherent flaws. Our experiments show that injecting covert deliberate content into manuscripts allows authors to explicitly manipulate LLM reviews, leading to inflated ratings and reduced alignment with human reviews. In a simulation, we find that manipulating 5% of the reviews could potentially cause 12% of the papers to lose their position in the top 30% rankings. Implicit manipulation, where authors strategically highlight minor limitations in their papers, further demonstrates LLMs’ susceptibility compared to human reviewers, with a 4.5x higher consistency with disclosed limitations. Additionally, LLMs exhibit inherent flaws, such as potentially assigning higher ratings to incomplete papers compared to full papers and favoring well-known authors in single-blind review process. These findings highlight the risks of over-reliance on LLMs in peer review, underscoring that we are not yet ready for widespread adoption and emphasizing the need for robust safeguards.
Explicit manipulation is the act of injecting deliberate content into manuscripts to influence LLM-generated reviews, where authors embeds manipulative review content into the manuscript PDF using extremely small white font, rendering it nearly invisible against the background.
Explicit manipulation could increases the paper rating to 8 (clear accept).
Explicit manipulation could cause systematic impact to the review system.
We find that LLM reviewers are more likely to reiterate the limitations disclosed by the authors than humans. This raises the risk of implicit manipulation where authors could strategically highlight minor limitations in the manuscript to influence LLM-generated reviews.
(Above) The figure illustrates a case where the LLM reviewer generates a review nearly identical to the manuscript’s limitations section.
(Right) The figure compares human reviewers with the LLM reviewer, highlighting that LLMs are more likely to reiterate the limitations disclosed by the authors than human reviewers.
LLM reviewers could generate similar reviews for papers with full content and incomplete content. We find that LLM reviewers could even generate plausible reviews for an empty paper.
LLM reviewers have bias, including favoring well-known authors in single-blind review processes and length preference.
LLM reviewer has length preference.
LLM reviewer has bias against authorship.
@article{ye2024we,
title={Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review},
author={Ye, Rui and Pang, Xianghe and Chai, Jingyi and Chen, Jiaao and Yin, Zhenfei and Xiang, Zhen and Dong, Xiaowen and Shao, Jing and Chen, Siheng},
journal={arXiv preprint arXiv:2412.01708},
year={2024}
}