OS-Helix Teaches Computer Agents to Coordinate GUI Actions and Code

Introduction

Real desktop work rarely stays inside a single interaction mode. A user may inspect a document through a graphical application, use a script to rename or transform a large collection of files, and then return to the interface to confirm the result. A computer-using agent therefore needs more than visual grounding or program synthesis in isolation. It must decide which mode is appropriate at each step and connect the two modes into one reliable workflow.

OS-Helix, proposed by a team from the University of Science and Technology of China, is designed around that problem. The system is an end-to-end multimodal agent that treats GUI actions and code execution as parts of one policy. The motivation comes from preliminary experiments in which adding a code tool to a GUI agent made performance worse. Tool availability alone does not teach a model how to route a task, estimate the consequences of an action, or switch modalities at the right time.

What the system changes

• A shared action space. Mouse and keyboard interactions are trained alongside code actions instead of being bolted on through prompt-level tool composition. The policy can therefore learn modality selection as part of task execution.
• Executable application-artifact environments. The researchers pair desktop applications with authentic file artifacts to instantiate runnable Ubuntu states. This is intended to produce environments that resemble actual desktop work rather than abstract instructions detached from files and applications.
• Decoupled task generation and evaluation. Instructions and evaluators are generated separately. The evaluator checks the final environment state, allowing a task to be judged consistently whether the agent solved it through the GUI, through code, or through a combination of both.
• A two-stage training recipe. OS-Helix is first cold-started with supervised fine-tuning on filtered hybrid trajectories. It is then improved through online reinforcement learning in live desktop environments.
• Step-balanced GSPO. A GUI action may be represented by only a few tokens, while a code action can contain a long program. Aggregating importance weights over raw assistant tokens would allow long code sequences to dominate optimization. The proposed method aggregates at the action-step level, seeking a better balance across heterogeneous actions.

Results and implications

On OSWorld-Verified, OS-Helix-27B and OS-Helix-35B-A3B report relative gains of 35.0% and 33.7%, respectively, over their matched Qwen3.5 hybrid baselines. After reinforcement learning, OS-Helix-27B-RL reaches a 66.44% success rate, a reported relative improvement of 42.9%. The project also reports 63.87% on WindowsAgentArena without Windows-specific training data, suggesting that the learned behavior may transfer beyond the Ubuntu environments used in training.

The broader contribution is a shift in how desktop agents are framed. The central problem is not merely whether a model has access to a browser, a terminal, or a mouse controller. It is whether the model can learn a policy for coordinating these interfaces under a shared objective. Authentic artifacts make the state of the environment more concrete, while final-state evaluation avoids penalizing valid solutions simply because they use a different modality.

The results should still be read within the limits of the reported benchmarks. High performance on curated desktop tasks does not by itself establish robustness on arbitrary operating systems, long-running workflows, unexpected failures, or safety-sensitive operations. Further evidence will be needed on recovery behavior, permission boundaries, and reproducibility. If the planned release of training and evaluation code is completed, OS-Helix could nevertheless become a useful reference point for studying hybrid computer-use agents.

Source: Hugging Face Daily Papers