LIBERO-Industry Brings Factory Constraints to Robot Evaluation

Introduction

A robot that can grasp an object in a benchmark scene is not necessarily ready for a factory. High-mix, low-volume manufacturing requires systems to cope with changing products, moving material flows, short execution windows, and precise placement requirements. These conditions turn a seemingly simple manipulation task into a tightly timed process in which small perception or control errors can accumulate quickly.

Turning production conditions into a benchmark

Researchers from the IAIE Physical AI Group at Tongji University introduced LIBERO-Industry to address this gap. Built on the widely used LIBERO robot-learning benchmark, the new suite translates common manufacturing constraints into controlled task settings and measurable success criteria. Its goal is not simply to add more scenarios, but to test whether generalist policies can remain reliable when the operating conditions change.

The benchmark includes three task families:

• Static-to-Dynamic (S2D) loading, where an object is placed onto a moving target or material flow;
• Dynamic-to-Static (D2S) unloading, where an object is retrieved from motion and placed at a fixed location;
• Dynamic-to-Dynamic (D2D) transfer, where the robot moves material between two dynamic processes.

Across these suites, 216 tasks vary object geometry, conveyor speed, the duration of the available interaction window, and the size of the geometric acceptance region. This structure allows researchers to separate different sources of difficulty and then study their combined effect. It also makes industrial readiness less dependent on a single overall success number.

What the experiments show

The authors evaluated five generalist robot policies, including vision-language-action models and world-action models. In simulation, performance declined progressively as industrial constraints were added and combined. The result highlights a familiar but important distinction: success on a static, clean manipulation task does not demonstrate robustness to timing pressure, moving targets, product variation, or narrow placement tolerances.

The study also compared training conditions under a shared fine-tuning and evaluation protocol. In real-world experiments, the LIBERO-Industry co-training condition achieved higher observed target-transfer completion than the original-LIBERO condition. This does not establish that one model is universally superior. Instead, it suggests that exposing policies to industrially relevant variation during training can improve their behavior on the corresponding transfer task.

Why it matters

LIBERO-Industry provides a more concrete way to discuss industrial readiness in embodied AI. Researchers can examine how a policy responds to faster conveyors, shorter action windows, changing geometries, or smaller acceptance regions, rather than treating factory deployment as an all-or-nothing claim. The benchmark may also help compare different policy families under a shared set of process constraints.

At the same time, benchmark performance should not be confused with readiness for large-scale deployment. Real factories add safety requirements, hardware variation, fault recovery, wear, and long-horizon reliability. LIBERO-Industry is better understood as a bridge between general-purpose manipulation research and industrial validation: a controlled way to measure the constraints that conventional benchmarks often leave out.

Source: Hugging Face Daily Papers