RecreationWorld Tests Whether Computer-Use Agents Can Rebuild Software

Why this matters

Computer-use agents have developed along two largely separate tracks. One focuses on operating graphical interfaces through clicks, typing, and visual perception. The other relies on code, terminals, and development tools to create or modify software. Actual digital work rarely keeps these capabilities separate: an agent may inspect an application, implement a feature, launch the result, and then return to the interface to verify what was rendered.

RecreationWorld is designed around that combined workflow. Instead of receiving a fixed sequence of actions, an agent is given a running reference application and must discover how it behaves. It then builds an implementation that matches the reference as closely as possible, using execution and visual checks to find defects. The central challenge is therefore not simply code generation, but the coordination of exploration, implementation, and verification.

What the framework provides

• Five platform environments. The framework covers Ubuntu, macOS, Windows, Android, and the Web, with a unified harness that combines native GUI control and coding tools.
• A reference-grounded oracle. The running reference supports hidden behavioral tests and execution-based rewards. This makes the target more demanding than producing code that merely appears plausible.
• Scalable trajectory generation. The researchers use high-quality open-source applications to produce training trajectories and study whether the resulting skills transfer beyond the recreation setting.
• Programmatic and visual evaluation. RecreationBench contains 250 diverse tasks across platforms and domains. Its assertions examine action-conditioned outcomes at multiple interaction depths, and were validated on the reference and reviewed by people before being frozen for automatic scoring.

What the results reveal

GPT-6 Astra leads the reported evaluation with an overall score of 58.1%, but it passes all programmatic tests on only 2.8% of tasks. The gap is important: a visually convincing interface does not necessarily reproduce the underlying application behavior. Agents are more reliable at copying static structure than at matching interactions or computed outputs. Their generated applications are also generally smaller and more monolithic than the references.

Training on the framework’s trajectories improves performance on five out-of-distribution coding and hybrid computer-use benchmarks. Models also verify their rendered outputs more often, suggesting that verification behavior can transfer beyond the original recreation tasks. The finding is encouraging, although it does not remove the difficulties of reproducing complex software behavior.

Broader implications

RecreationWorld moves evaluation away from isolated clicks or code completion toward a complete loop: observe, implement, run, and verify. It highlights why screenshots and single-action success are insufficient measures of agent reliability. A capable system must also preserve state changes, support multi-step interaction, and produce the right computed results.

By releasing the environments, benchmark, and test suites, the project offers a common basis for comparing agents across operating systems and application types. Its most important lesson may be that the next stage of computer-use research will depend less on making agents act more fluently and more on proving that their actions reproduce the intended behavior.

Source: Hugging Face Daily Papers