When AI Text Is Humanized, Do the Facts Survive?
Introduction
AI-text humanizers are designed to make machine-written prose sound more natural and less detectable. Their public evaluations usually emphasize detector evasion and embedding similarity. A new audit asks a more practical question: after rewriting, can readers still trust the details that can be checked?
The study tracks numbers, dates, person names, organization names, and quoted spans. A rule-based checker was validated by two AI annotators, achieving precision between 0.84 and 1.00 and recall between 0.93 and 0.95. The authors explicitly note that there was no human annotation, so the validation should not be treated as a complete substitute for expert review.
Key findings
- In the October 2026 cycle of a public humanizer leaderboard, all 11 commercial tools changed or dropped at least one tracked unit in 35% to 65% of documents that contained such a unit.
- Quoted spans were the most frequently changed category. Even when quotations were excluded, 10 of the 11 tools altered at least one number or name in 20% or more of the relevant documents. The September cycle showed a similar pattern, ranging from 19% to 50%.
- An open 12B humanizer preserved at least 97% of numbers and names in fact-dense U.S. federal agency text, but rewrote 67% to 78% of quoted spans despite being prompted to keep them unchanged.
- Retention and detector evasion were not clearly connected. Across tools, the Spearman correlation was -0.46, with a 95% confidence interval from -0.65 to 0.08. The four released files of the open model also failed to show the preregistered monotone trade-off.
- On matched human, AI-styled, and humanized federal texts, the general-purpose decision model Jev separated human originals from humanized rewrites with an AUROC of 0.864. Yet at a 0.5 threshold, it labeled 24% of human originals as AI-generated.
Why it matters
The results challenge the idea that a successful rewrite is simply one that fools a detector. A changed quotation can become an invented paraphrase, while a small change to a date, number, or proper name can alter the meaning of a report. These risks matter in journalism, research, public-sector communication, and business writing, where factual details carry more weight than stylistic smoothness.
The detector results add another warning. On AI-styled rewrites produced by 2026 models, two perplexity-based detectors scored below chance, with AUROCs of 0.37 and 0.39. Removing line breaks did not fix the problem. Jev performed better in this setting, but its false-positive rate shows why detector output should not be treated as conclusive evidence about authorship.
The practical implication is straightforward: leaderboards should report factual retention alongside evasion and similarity. Users should independently verify every number, date, name, and quotation after humanization. The researchers released the checker and per-item results, offering a useful starting point for more accountable text-generation evaluations.
Source: Hugging Face Daily Papers
Comments
Checking sign-in status...
Loading comments...