OpenGCUA Opens the Full Stack for Generalist Computer-Use Agents

Introduction

Computer-use agents are moving beyond the familiar pattern of reading a screenshot and clicking a button. Alibaba’s MAI-UI team has released the first version of OpenGCUA, or Open Generalist Computer Use, with the goal of opening the full stack behind this class of systems. The project is not limited to releasing a model: it exposes the agent harness, prompts, evaluation traces, real-device tooling, and an early cross-device SDK.

From GUI control to autonomous orchestration

OpenGCUA defines a generalist computer-use agent as a system in which the model decides how to observe its environment, which actions to take, and how to organize the task. A screenshot is only one possible source of information. The agent can also read the accessibility tree, call application or system APIs, execute shell commands, and return to visual interaction when structured information is insufficient.

This changes the main design question. Instead of forcing every task into a screenshot-click loop, the system gives the model several ways to inspect and manipulate the computer. A task may be completed through structured state, programmatic interfaces, visual feedback, or a combination of all three.

Code execution as the control layer

The project combines GUI operations, accessibility access, API calls, and shell commands inside executable programs. Loops, conditional branches, and intermediate outputs can determine what happens next. For example, the model may request a new screenshot only when it cannot infer the current state from structured data. It can also use retrieved text, files, or system status to revise its plan.

OpenGCUA also introduces what it calls “harness-to-model decision transfer.” Some decisions that would traditionally be hard-coded in the harness—such as when to observe or how to interpret task state—are handed to the model. The harness supplies the mechanisms, while the model chooses when to invoke them. This may reduce procedural overhead, but it places greater demands on planning, tool selection, and recovery from errors.

Results and what is being released

On OSWorld-v2, the project reports improvements for all eight evaluated models, including GPT-6 Astra and Claude Opus 5. In the full configuration, the average partial score increased by 19.6 points, while reported steps per task fell by an average of 53.0%. The result suggests that improving the execution framework can matter as much as changing the underlying model. At the same time, these figures come from the project’s specified prompts, harness, and evaluation setup, so broader replication remains important.

The release includes implementation code, prompts, and evaluation trajectories from all eight models. It also provides a macOS real-device harness and a preview cross-device SDK. The SDK is designed to connect computers, browsers, and phones through one programming interface, allowing a single program to coordinate several devices in one task. An interactive lab demonstrates code execution, real-device operation, and cross-device workflows.

Why it matters

OpenGCUA shifts attention from model capability alone to the joint design of models, tools, execution environments, and benchmarks. Real devices add practical complications that are easy to miss in simulators, including permissions, changing state, and hardware-specific behavior.

Important questions remain. How can code execution become more efficient? How should an open tool environment handle safety and permissions? What benchmarks can measure reliable performance on real devices? And how should browser use be integrated with broader computer use? OpenGCUA offers an open starting point, but its lasting impact will depend on reproducibility, community contributions, and stronger safety mechanisms.

Source: Hugging Face Daily Papers