A Missing Piece for Trustworthy AI Reviewers: From Benchmarking Rhetorical Robustness to SciCore Review

AI reviewers show "false robustness" to wording changes: The study found that some AI reviewers appeared insensitive to how manuscripts were rewritten, but this was misleading—they actually collapsed their scores across all papers rather than maintaining meaningful distinctions between good and poor research. This means low sensitivity to rewording doesn't guarantee fair evaluation.

Current AI review practices may reward writing style over scientific quality: The research demonstrates that AI reviewers can assign significantly different scores to identical scientific content presented with different rhetoric, potentially incentivizing authors to optimize their writing rather than improve their actual research—a misalignment with the goals of peer review.

Standard prompting techniques don't reliably solve the problem: Even when researchers tried to improve AI reviewers by instructing them to focus on content rather than presentation, this "content-focused prompting protocol" did not consistently enhance robustness across different AI models, suggesting the issue runs deeper than simple instruction design.

Human reviewers and AI reviewers prioritize different things: The study found that human alignment (how well AI reviewers match human judgments) and rhetorical robustness (resistance to wording changes) rank reviewers differently, meaning an AI reviewer that mimics human behavior may still be vulnerable to manipulation through rhetorical optimization.

A dual-analysis approach shows promise for more trustworthy AI review: The proposed SciCore system, which combines full-manuscript assessment with analysis of extracted scientific content, achieved better balance between stability and discrimination while maintaining competitive alignment with human reviewers, offering a potential path toward more robust AI-assisted peer review.