PremiseCheck Turns Peer-Review Fact-Checking into Verifiable Premises

Peer review faces a quantity problem, but it also faces a reliability problem. A review may overstate an experiment, omit an important qualification, or combine several unstated assumptions into a single sentence. Such mistakes can appear in comments written by humans as well as in comments generated by AI, potentially distorting author responses, editorial decisions, and readers’ understanding of a paper.

A benchmark for factual review errors

The paper introduces ReviewFact, a benchmark containing 9,267 comments associated with 217 papers in natural language processing, machine learning, and the natural sciences. The comments are labeled for factual errors under a shared definition. Importantly, the benchmark evaluates not only detection quality, but also the cost and latency required to produce a check. That makes it more relevant to conference and journal workflows, where thousands of comments may need to be screened.

PremiseCheck addresses a weakness in two common strategies. A frontier LLM can inspect a whole comment effectively, but it may need to read the entire paper for every check. A cheaper decision model can determine whether a statement is supported by a text, yet it performs poorly when a comment contains several implicit claims. PremiseCheck divides the work:

• An LLM rewrites the comment into explicit, testable premises without reading the paper.
• A smaller decision model checks each premise against the paper.
• The weakest premise determines the comment-level score, and a flag identifies the premise that appears unsupported.

The division is significant. The LLM is used for linguistic normalization and claim extraction, while the less expensive model handles repeated evidence matching. This avoids asking a large model to perform the entire process for every comment and gives the system a more interpretable failure signal than a single holistic label.

Results and limitations

When comments are evaluated individually, PremiseCheck achieves AUROC comparable to GPT-5.5 across the three fields at approximately half the cost. In the NLP setting, a decomposer without reasoning obtains an AUROC of 63.1 versus 63.2 for the reference system, while costing about one quarter as much and taking less than half the latency. At the review level, combining batched PremiseCheck scores with GPT-5.5 scores reaches 71.5 AUROC at 28% of the reference cost.

The numbers should not be read as evidence that factual verification is solved. The NLP task remains difficult, with AUROC around 63 for both systems. The machine-learning and science subsets are smaller and have wide intervals. The benchmark also excludes judgment errors: whether a criticism is fair, an experiment is important, or a contribution is novel is different from whether a comment accurately describes the paper.

Why it matters

PremiseCheck offers a practical cost-sensitive design for review support. Editors or program committees could use it to surface comments that deserve closer inspection, allowing experts to focus on high-risk cases. Its premise-level output may also make errors easier to investigate than a black-box score.

Still, the final decision must return to the paper and domain expertise. Missing context, ambiguous wording, or claims supported across multiple sections can confuse both the decomposer and the decision model. The most realistic role for PremiseCheck is therefore triage: a scalable first pass that improves where humans spend their attention, not an automated replacement for peer reviewers.

Source: Hugging Face Daily Papers