Back to articles

Are Peer Reviews Becoming More Alike in the LLM Era?

3 min read

Peer review is meant to provide more than several votes on whether a paper is sound. Independent readers can uncover different weaknesses, giving authors and editors a broader map of what needs attention. If reviewers repeatedly identify the same problems, the number of reviews may overstate how much distinct feedback a paper has received. The paper “Twice-Told Critiques” examines this possibility across conference and journal reviews in the LLM era.

The study reports that critique overlap among reviews from the same stage at ICLR increased from 12.0% in 2026 to 21.3% in 2027. Year-on-year increases were also reported at NeurIPS, ICML, and the F1000 and PLOS journal sources, although the supplied summary does not give their precise magnitudes. A related measure points in the same direction: a third ICLR review added an average of 5.46 new problems beyond the first two in 2027, compared with 6.12 in 2026. This indicates fewer additional issues in the third review on average; it does not, by itself, explain why.

The author also applied MELD, a detector for LLM-generated text, to the reviews. For a given paper, two reviews flagged by MELD had more overlapping critiques than two unflagged reviews. The gap was 3.0 percentage points at ICLR in 2024–2025, 4.5 points there in 2026–2027, and 5.4 points across five journal sources. These findings connect detector flags with greater similarity between reviews. They do not establish that a flagged review was written by an LLM, nor that LLM use caused the overlap.

Key points

  • Review critiques overlap more than before across several conference and journal sources; the reported ICLR increase is especially clear.
  • The third ICLR review contributed fewer new problems on average in 2027 than in 2026.
  • MELD-flagged reviews were more similar to one another, but a detector label is not definitive evidence of authorship or tool use.

The broader concern is whether multiple reviewers still provide multiple perspectives. If the critiques are increasingly redundant, authors may receive less independent diagnostic value than the review count suggests. Editors could therefore consider the distinctiveness of feedback alongside the number of reviews when assessing how much scrutiny a paper has received. The study also shifts attention beyond surface style: the quality of a review process depends partly on whether different readers identify different, specific, useful problems.

The available summary does not describe the overlap calculation in detail, the full sample composition, or other factors that could contribute to the trend. It would therefore be premature to attribute the rise directly to LLMs. MELD’s output should likewise be treated as a detector result, not confirmation. The careful conclusion is narrower: critique diversity may be declining, and reviews flagged as LLM-generated are associated with greater overlap. Establishing what drives these patterns will require further evidence.

Source: Hugging Face Daily Papers

Comments

Checking sign-in status...

Loading comments...

Related articles