SoftwareGym Scales Computer-Use Agent Training with Synthetic Apps

Introduction

Computer-use agents need more than visual recognition. To complete practical tasks, they must interpret unfamiliar interfaces, plan several actions, recover from mistakes, and verify that the requested result has actually been produced. Training these capabilities requires interactive software environments at scale, but building and integrating native applications by hand is expensive.

SoftwareGym, introduced by a Shanghai Jiao Tong University team, addresses this bottleneck with an automated environment-synthesis pipeline. Rather than relying only on screenshots, scripted web pages, or fixed mock interfaces, it creates runnable GUI applications with executable functionality. The applications are then tested through real interaction and repaired when their behavior does not satisfy the task requirements.

How the pipeline works

The approach has three central stages:

• Diverse design generation: SoftwareGym produces applications with varied layouts, interface structures, and interaction patterns. This is intended to reduce dependence on a single application style.
• Runnable realization: The generated designs are implemented as software environments that agents can operate, rather than as static prototypes.
• GUI-based verification: Tasks are executed through the interface itself. Failures can trigger iterative repair, making functional validation part of the generation process.

Using this pipeline, the researchers synthesized 182 applications spanning 12 categories. Their mean GUI task pass rate was 91.5%, suggesting that the environments were generally usable for collecting interaction data. The project also introduced SoftwareGym-Bench, a 120-task suite covering the same categories.

Results and significance

The team collected trajectories from the synthesized applications and used them for computer-use agent training. On the strictly out-of-domain OSWorld benchmark, the resulting model improved Pass@3 from 41.39% to 56.97%. It achieved 40.0% on SoftwareGym-Bench. Because OSWorld is outside the synthesized training environments, the improvement indicates that the collected trajectories may teach transferable interaction patterns rather than only memorized details of particular applications.

The broader contribution is a change in the economics of environment creation. Manual integration can restrict coverage and encourage agents to overfit to a small set of familiar applications. Browser-hosted tasks are easier to deploy, but they may not represent the depth or workflow complexity of full software interfaces. SoftwareGym attempts to combine the scalability of synthesis with the realism of executable GUI interaction.

The results should still be interpreted carefully. A high task pass rate during environment validation does not mean that generated applications fully match production software. Likewise, a 40.0% score on SoftwareGym-Bench shows that the agents still struggle with these interfaces. More testing across operating systems, application families, and longer workflows will be needed to determine how far synthetic environments can support real-world generalization.

Even with these limitations, SoftwareGym offers a practical direction for CUA research. If application generation, testing, and repair can be made reliable, synthetic software may become a scalable source of supervision for agents that must operate computers rather than merely describe them.

Source: Hugging Face Daily Papers