Six AI Reviewers Assessed 137 Papers. Some Approved Almost Everything

The hardest task for a journal reviewer is often saying no to a manuscript that sounds convincing. The paper may be clear, relevant and ambitious, yet its central conclusion may not follow from the evidence. A language model trained to be helpful and agreeable can produce a sophisticated-looking review without making that difficult judgment. Research presented at the 2026 CHI conference tested six OpenAI models as manuscript reviewers and found a striking problem: several were willing to accept virtually everything in the experiment.[1]

Nitin Verma and Asheley R. Landrum evaluated GPT-4.1, GPT-4o, o1, o3, o3-mini and GPT-5 against 137 manuscripts from the PeerRead dataset, using the corresponding human-generated reviews as a comparison. The authors report that o3 and GPT-5 accepted submissions at rates close to the human benchmark, around 67%. The other tested models approved more than 98% of submissions. A system that rarely rejects can appear exceptionally encouraging while providing little help in distinguishing strong evidence from weak evidence.[1]

Acceptance rate is only the beginning of the audit

The crucial distinction is between how often a model says yes and whether its decisions agree with the comparison judgments. A classifier that approves every paper could post a seemingly high rate of agreement in a collection containing mostly accepted work. It would still fail nearly every important rejection decision. The study reports that all six tested models performed very poorly on accuracy, precision and recall measures in the benchmark, including those whose overall acceptance share resembled the human rate.[1]

This is why an institution should not begin by asking whether AI-written reviews sound professional. It should ask whether the system discriminates reliably among substantively different papers, catches methodological flaws and handles disagreements across fields. The cost of false acceptance may be publication of misleading work; false rejection may suppress unconventional ideas. Both errors matter, and their relative importance differs between a journal, a funding programme and an informal pre-submission assistant.[1]

The experiment also has a clear boundary: the human decisions in a dataset are a benchmark, not infallible ground truth about the intrinsic value of a paper. Human peer review has well-known inconsistencies, and an AI judgment that disagrees with it is not automatically wrong. But an automated decision-maker that cannot reproduce even that reference standard needs much stronger justification before anyone makes it a gatekeeper.[1]

What the researchers were actually probing

The authors went beyond acceptance counts. They used a values questionnaire based on Schwartz’s framework to characterize the models’ apparent value orientations. The paper reports tendencies toward self-transcendence and openness to change, and less emphasis on conservation and self-enhancement. Those are results from applying a questionnaire designed for human values to AI outputs. They should not be read as a psychological diagnosis of a model or proof that its internal motivations resemble those of a human reviewer.[1]

Still, the exercise points toward a design problem. A generally supportive assistant may be useful for helping an author clarify an argument or improve the structure of a manuscript. That same conversational tendency can be harmful when the role changes to identifying a fatal flaw, recommending rejection or asking whether a result merits public trust. Systems that are optimized to cooperate with the person asking the question may need different objectives and checks when they are asked to scrutinize that person’s work.[1]

There is also a distinction between a generative review and an editorial decision. A model can summarize a manuscript, list statistical assumptions, identify references worth checking and ask reproducible questions without being asked to vote accept or reject. Some of those narrower functions may offer substantial value to overstretched reviewers. The study’s negative results on gatekeeping do not establish that all use of AI in reviewing is futile.[1]

A safer design separates triage from authority

An editorial workflow could use AI to flag missing data statements, inconsistent sample sizes or unaddressed limitations, while reserving interpretive judgment for qualified reviewers. Before deployment, the system should be tested on papers with known defects and on examples from underrepresented disciplines. Its false-acceptance and false-rejection rates should be reported separately, with change logs showing what prompts, models and evaluation conditions were used.[1]

The study was published in April 2026, so this is not a new product announcement. Its continuing relevance is that the pressure to automate scientific review is growing just as frontier systems become increasingly fluent. The underlying decision problem has not disappeared. A review that sounds rigorous can still be an instrument for approving almost everything. Until performance on difficult negative judgments is demonstrated, the model’s most defensible role is a tool for reviewers, not the reviewer whose signature settles the case.[1]

Report an error or suggest a correction