Turning Scientific Rules into Tools for LLM Coding Agents

Introduction

Scientific coding agents must do more than translate a prompt into code. They may need to interpret equations, boundary conditions, numerical constraints, and output requirements, then determine whether a repaired program actually satisfies them. In a conventional workflow, these rules remain in natural language and the model must reason about them while inspecting or testing its own changes. The paper Rules to Tools explores a different design: implement public scientific requirements as callable executable checks that an agent can use during repair.

Key findings

• The experiment isolates the role of executable feedback. In matched SciCode repair groups, both sides receive the same written checks, starting programs, model, and budget. The tool group additionally receives a callable implementation of the requirements.
• Tools improve some aggregate scores, but the evidence is not universal. Across two task-ID cohorts, the text group completes 26 of 30 repairs, compared with 29 of 30 for the tool group. In an eight-ID cohort, the corresponding results are 13/16 and 15/16. However, a task-cluster bootstrap 95% interval for the difference ranges from -12.5 to 43.75 percentage points, reflecting substantial uncertainty at this sample size.
• Task-level variation matters. Three task IDs favor tools, one favors text, and eleven tie. In a larger cohort using shared definitions, both groups score 13/24. On five development-exposed tasks with alternate starting programs, the text and tool groups score 3/10 and 7/10 respectively. These comparisons indicate that the benefit depends heavily on the structure of the task and the available check.
• Checks are not automatically ground truth. Initial checks flag task 17, while they report no violation for tasks 77 and 11. Task 37 favors the text condition and has no initially reported violation. Such cases highlight the importance of check coverage, correctness, and diagnostic quality.
• Lower model output does not mean lower total resource use. In a matched PDE comparison, detailed text reaches 23/24 and checks reach 24/24, while reported model output is 31.2% lower for the check condition. Yet public CPU use rises in both task-ID cohorts, and output savings vary by cohort. A separate source-through-Python setup also reaches 15/16, matching the dedicated command aggregate.

Why it matters

The paper reframes verification for scientific agents as an interface-design problem: how should a written requirement enter the agent’s workflow? A reliable executable check can provide concrete feedback about whether a program violates a boundary condition or another scientific constraint. That may reduce the amount of explanation and repeated code generation needed from the model, particularly in numerical programming tasks where purely textual verification is difficult.

The results also argue against treating tool use as a universal upgrade. Checks must be authored, maintained, and independently validated. A narrow implementation can miss an important failure, while an incomplete diagnostic can mislead the agent. Practical systems may therefore need a combination of natural-language specifications, executable checks, and separate tests. Evaluation should track not only repair success, but also model output, CPU consumption, and the cost of building the checks.

Rules to Tools offers evidence for moving scientific agents from merely reading rules to actively calling them. Its more cautious conclusion is equally important: executable feedback can help on selected repair tasks, but its value depends on task characteristics, check quality, and a complete accounting of resources.

Source: Hugging Face Daily Papers