CUA-SWE Brings Computer-Use Agents Into the Software Engineering Loop

Introduction

Software development is not limited to producing a patch. In practice, developers repeatedly launch an application, interact with it, inspect what appears on screen, infer what went wrong, and decide what to change next. Coding agents are commonly evaluated on repository-level edits and terminal work, while computer-use agents are often studied as interface operators. CUA-SWE explores what happens when both capabilities are required within the same engineering task.

What the benchmark tests

• A combined workflow. An agent must modify source code or configuration, execute commands, interact with running software, and inspect visual feedback rather than solving each capability in isolation.
• Information hidden in the interface. Some task specifications or operational details are available only through the application’s visual interface. The agent therefore has to gather information by using the software, not merely by reading repository files.
• Diagnosis across representations. When an interaction fails, the agent must connect the observed behavior with the responsible implementation, make a repair, and use the application again to check the result.
• Executable correctness criteria. Every task includes deterministic, task-specific tests. These tests assess whether the requested behavior is implemented and whether specified existing behavior remains intact.
• Coverage across four domains. The benchmark studies how agents behave across four software engineering areas and under different levels of information access.

Why it matters

The central contribution of CUA-SWE is a broader definition of software-engineering competence for agents. A screenshot is not merely a final presentation artifact: it can contain a requirement, reveal a runtime failure, or provide evidence that a change worked. An effective agent must move between source code, the command line, and the graphical application while maintaining a coherent model of the task.

This setup also gives evaluation researchers a way to connect interaction behavior with software correctness. Instead of judging success solely from a textual explanation or a sequence of clicks, they can use executable tests to determine whether the resulting system satisfies the requirement. The supplied material does not report model rankings, task counts, or numerical performance results, so CUA-SWE should be read primarily as a benchmark and research framework rather than proof that one agent family is superior.

The direction is increasingly relevant as applications depend on rich interfaces, runtime state, and configuration flows. CUA-SWE offers a practical testbed for agents that do more than generate code: they must run the software, see its behavior, act on that evidence, and produce a verifiable repair.

Source: Hugging Face Daily Papers