OR-License Tests LLM Agents on Real-World Operations Research

Introduction

Many operations research benchmarks begin with a clean problem statement: the variables, constraints, objective, and input data are already organized. Real consulting work is different. Clients provide fragmented documents, spreadsheets, and business requirements; important constraints may remain unstated; and the final result must often follow a particular delivery format. OR-License is designed to measure this complete path from raw business material to a usable optimization solution.

What the benchmark changes

• It uses industry-derived tasks. OR-License includes 31 instances across 10 industrial scenarios. They originate from paid operations research engagements with a combined value of $3.68 million. After de-identification, the benchmark retains the original requirements, input data, deliverable formats, and information boundaries.
• It evaluates the whole workflow. An agent must understand the client’s problem, obtain missing information, formulate and solve the optimization task, and produce the requested deliverable. The task is not reduced to solving a pre-formulated mathematical exercise.
• It separates business facts from OR expertise. ORSim is a controlled client simulator that reveals relevant business facts without handing the agent an expert formulation. This makes it possible to study how an agent asks questions, clarifies assumptions, and advances solution development and validation.
• It makes feasibility decisive. Hidden checks examine whether a delivery satisfies the required constraints and expectations, then compare objective quality with the original expert solution accepted by the client. Deployment-Adjusted Quality, or DAQ, averages expert-relative quality while assigning zero to infeasible deliveries.
• The reliability gap remains. Across 11 frontier LLMs and coding-agent harnesses, several systems achieved higher objective quality than the expert reference when considering only feasible outputs. However, none matched the expert on DAQ. The strongest primary Claude Code configuration achieved 88.39% strict feasibility and a DAQ of 0.87, compared with an expert reference of 1.

Why it matters

The benchmark highlights a distinction that conventional optimization tests often miss: a better objective value is not enough. A solution that violates a business constraint, omits a required output, or fails the client’s delivery expectations may be unusable in practice. By treating infeasible deliveries as zero-quality outcomes, OR-License measures operational reliability rather than isolated solver performance.

It also frames information gathering as part of solution construction. Agent questions do more than resolve ambiguity; later interactions can shape the model, support validation, and expose hidden assumptions. Enterprise-grade OR agents will therefore need more than code generation and solver access. They will need constraint tracking, structured validation, robust delivery checks, and mechanisms for recovering from failed attempts. OR-License, together with its released code and dataset, offers a more realistic starting point for measuring those capabilities.

Source: Hugging Face Daily Papers