Beyond Compilation: Why Lean Success Does Not Guarantee Faithful Formalization

Introduction

For natural-language-to-Lean systems, compilation is often treated as the clearest sign of success. Yet Lean checks syntax, typing, and the requirements of the formal environment—not whether the generated declaration captures the user’s intended theorem. A statement can compile after dropping a condition, changing a quantifier, or misreading a mathematical term, while silently expressing a different claim.

The paper Beyond Compilation studies this problem without assuming that a canonical Lean target exists. Its central question is how to evaluate faithful statement formalization when there is no single reference implementation to compare against.

Key findings

• Compilation is only a necessary condition. The proposed criterion first requires a declaration to compile in Lean. It then requires agreement from two LLM judges drawn from different model families on whether the formal statement matches the natural-language input.
• The criterion tracks human judgments reasonably well. In an independent random audit, it agreed with the human majority in 91.5% of cases. The researchers report that it rarely accepts an unfaithful output, although it rejects some translations that humans consider faithful.
• The measured gap is substantial. Across eight systems and 227 university-level statements, the difference between compilation and accepted outputs ranged from 1.3 to 29.5 percentage points. A tool-augmented GPT-5.2 agent compiled 87.2% of the items, but only 57.7% were accepted as faithful.
• Many failures are genuine translation errors. Human review of the agent’s rejected cases attributed 23.8 of its 29.5 percentage points to real translation mistakes. A separate two-stage review of seven one-shot systems found that, for six systems, 78–100% of their gaps were translation errors.
• Judges can also be the limiting factor. GPT-5.2 accepted 83% of the outputs from both GPT-6 Astra and Claude Opus 5.5, compared with 21% for one-shot GPT-5.2. Review of the newer models’ rejected cases classified only 4 of 54 as translation errors, suggesting that much of their measured gap came from judge errors rather than faulty formalization.

Why it matters

The results challenge a convenient but incomplete definition of progress. Compilation remains valuable: it filters out structural and type errors. It cannot, however, establish semantic equivalence with the source statement. A system might improve its compilation rate by weakening a proposition, omitting a hypothesis, or altering the relationship between variables, without becoming more faithful.

Future benchmarks should therefore report compilation, semantic acceptance, human audits, and evaluator error separately. The study also highlights a less obvious risk: as generation models improve, an outdated evaluator may become the bottleneck. Reliable formalization will require not only stronger code generation, but also better semantic auditing of what the generated theorem actually says.

Source: Hugging Face Daily Papers