PerfReasoning Tests Whether LLMs Really Understand Hardware Performance

Introduction

Performance modeling is a foundation of hardware design and software optimization. A useful model must connect several factors at once: the computation being performed, how data is reused, where intermediate results are stored, how a workload is mapped onto hardware, and how much data moves between storage levels. PerfReasoning asks whether large language models can reason about these relationships reliably—and whether they can turn that reasoning into usable analytical models.

Key findings

• Two capabilities are evaluated separately. The benchmark asks models to answer performance questions, such as comparing mappings and estimating off-chip traffic or buffer requirements. It also asks them to generate analytical performance-model code, a more demanding task that exposes consistency and implementation errors.
• Reasoning answers are much easier than model construction. The strongest closed-source systems score above 90% on reasoning-oriented questions, while the best open-weight model reaches 82.4%. Once the task shifts to constructing a model, however, every configuration other than GPT-5.6 Sol averages below a 15% pass rate. Results also vary substantially across runs.
• Task-specific RL helps. Training for the target task raises the mapping-reasoning accuracy of a 4B model by 15.7 percentage points. This suggests that focused optimization can matter more than general-purpose fluency for structured hardware analysis.
• Unsupervised self-revision is unreliable. Asking a model to revise its own answer over several rounds, without external feedback, does not consistently improve performance. Detecting a wording problem is not the same as validating the assumptions, equations, and edge cases in a performance model.

Why it matters

The benchmark highlights a gap that is easy to miss in ordinary LLM evaluations: a model may sound knowledgeable about computer architecture while still failing to construct a model that runs correctly or remains valid across different mappings. Hardware performance analysis depends on consistency between computation, reuse, storage capacity, and movement cost. A plausible narrative is therefore only an initial signal, not engineering evidence.

For chip design and compiler optimization, current models may already be useful for exploring design choices, summarizing bottlenecks, and proposing candidate analyses. However, automated generation of dependable models will require stronger execution-based testing, formal or numerical checks, and feedback from external evaluators. The reported reinforcement-learning result points toward task-specific training as a promising direction, while feedback-free prompt iteration appears insufficient as a substitute for verification.

The authors plan to release PerfReasoning publicly. A shared benchmark could make progress easier to reproduce and clarify whether future systems are improving at genuine performance modeling rather than simply producing more convincing explanations.

Source: arXiv