Lean4AgenticRL Uses Formalized Multi-Level Rewards to Train Stronger Agents

Introduction

For an LLM agent working on software engineering or terminal tasks, success is not determined by the final answer alone. The agent must interpret the task, select suitable tools, react to outputs, and keep a long sequence of actions on track. Yet much of the current agentic reinforcement learning literature relies on high-level outcome rewards. Tool calls and intermediate workflow decisions receive relatively little direct supervision, leaving the model with a sparse and delayed training signal.

Lean4AgenticRL, presented in an arXiv paper, explores a formal-methods-based alternative. The framework uses Lean4 as a common formalization schema for multi-level process rewards and is used to train two models, LAR-9B and LAR-27B, for software engineering and general terminal-use scenarios.

Key points

• Rewards cover several levels. Instead of asking only whether the final task was completed, the framework evaluates lower-level tool-call behavior and mid-level workflow stages as well as the overall outcome.
• Workflows are tailored to individual problems. Lean4AgenticRL first constructs a problem-specific workflow. The stated goal is to balance the model’s ability to act independently with enough control to make execution verifiable.
• Formalization supports verification. The workflow is represented formally, and a general harness is used to verify agent behavior and produce reward annotations within the same framework.
• Training uses normalized GRPO. The multi-level signals are incorporated into a workflow-stage-normalized version of GRPO, intended to account for differences among stages during optimization.
• Reported benchmark gains. LAR-27B scores 77.3% on TerminalBench-2.1. The paper also reports that the trained models improve over their base models by 9.89% on average and exceed the second-best comparable-size model known to the authors by 5.07%.

Why it matters

The central idea is to make agent training more diagnostic. A failed task can result from a bad tool choice, an incomplete stage transition, or an error that appears only at the end. By assigning rewards at multiple levels, Lean4AgenticRL attempts to expose these distinctions during training instead of treating every unsuccessful trajectory as a single negative outcome.

The work also points to a broader role for formal methods in agent development. Lean4 is commonly associated with formal reasoning and verification; here, it is used to structure supervision for open-ended tool-using behavior. If this approach scales, it could make long-horizon agent training easier to inspect and potentially more controllable.

The available material does not establish how well the approach transfers across workflow designs, task families, or larger models. It also leaves open practical questions about the cost of constructing problem-specific workflows, the overhead of verification, and whether stronger process constraints could reduce useful exploration. Those issues will be important for evaluating the method beyond the reported benchmark results.

Overall, Lean4AgenticRL offers a concrete connection between formal verification, process rewards, and agentic RL, targeting one of the field’s persistent problems: sparse feedback in long tool-use trajectories.

Source: arXiv