Zero-shot reasoning for simulating scholarly peer-review
By: Khalid M. Saqr
Potential Business Impact:
Checks science papers for AI cheating.
The scholarly publishing ecosystem faces a dual crisis of unmanageable submission volumes and unregulated AI, creating an urgent need for new governance models to safeguard scientific integrity. The traditional human-only peer review regime lacks a scalable, objective benchmark, making editorial processes opaque and difficult to audit. Here we investigate a deterministic simulation framework that provides the first stable, evidence-based standard for evaluating AI-generated peer review reports. Analyzing 352 peer-review simulation reports, we identify consistent system state indicators that demonstrate its reliability. First, the system is able to simulate calibrated editorial judgment, with 'Revise' decisions consistently forming the majority outcome (>50%) across all disciplines, while 'Reject' rates dynamically adapt to field-specific norms, rising to 45% in Health Sciences. Second, it maintains unwavering procedural integrity, enforcing a stable 29% evidence-anchoring compliance rate that remains invariant across diverse review tasks and scientific domains. These findings demonstrate a system that is predictably rule-bound, mitigating the stochasticity of generative AI. For the scientific community, this provides a transparent tool to ensure fairness; for publishing strategists, it offers a scalable instrument for auditing workflows, managing integrity risks, and implementing evidence-based governance. The framework repositions AI as an essential component of institutional accountability, providing the critical infrastructure to maintain trust in scholarly communication.
Similar Papers
The AI Imperative: Scaling High-Quality Peer Review in Machine Learning
Artificial Intelligence
AI helps scientists check research faster.
Tenure Under Pressure: Simulating the Disruptive Effects of AI on Academic Publishing
Computers and Society
AI floods journals, making it harder to publish.
ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review
Artificial Intelligence
AI helps scientists check research papers faster.