Can AI Reviewers Recognize Better Papers After Revision?

Introduction

Revision is where authors add experiments, clarify the problem, reorganize the argument, and respond to criticism. A useful AI reviewer should recognize these changes and reward a manuscript when its scientific quality improves. A new large-scale study suggests that this connection is far less reliable than expected.

What the study examined

The researchers used PaperTrajectory-2026, a dataset covering 6,959 paper trajectories from five AI conferences. It records successive manuscript versions, automatically annotates changes along 14 dimensions, and provides separate judgments of overall quality. The team then asked eight AI reviewers to assess manuscript versions independently, testing whether the systems could detect improvement across a trajectory.

The largest improvements tended to concern empirical evidence and framing. In practice, this means that authors were more likely to strengthen experimental support, sharpen the research setup, or improve how the contribution was presented. Later versions, by contrast, were more likely to receive incremental rather than fundamental changes.

Key findings

• Most revisions labeled as major improvements produced no score change from AI reviewers.
• Agreement among reviewers on whether a revision raised or lowered the score was weak.
• Some revision dimensions were associated with higher scores for particular reviewers, but the patterns did not generalize consistently and offered limited predictive separation.
• A short negative user opinion could erase or reverse a score advantage that a revised paper received under neutral evaluation.

Why it matters

The results do not show that AI reviewers are incapable of seeing improvement. They show that their response is inconsistent. For authors, adding evidence or improving the paper’s framing does not guarantee that an AI score will move in the same direction. For system designers, a final score alone is therefore an inadequate measure of whether a reviewer has understood a revision.

The study also highlights the role of user framing. If a brief negative description can override an improvement that the system previously recognized, the reviewer is responding to more than the scientific content. That creates a challenge for reliable and fair AI-assisted peer review.

Future evaluations should compare versions directly, measure sensitivity to substantive evidence changes, and test whether judgments remain stable when the underlying science is unchanged. The goal is not simply a higher score for a revised paper, but a review process that tracks meaningful progress and resists rhetorical manipulation.

Source: Hugging Face Daily Papers