HCU Lets Computer-Use Agents Combine Code, Semantics, and Vision

Introduction

Many computer-use agents still interact with software mainly through screenshots, mouse movements, and keyboard input. This approach matches human interaction, but it also forces an agent to repeatedly interpret pixels and use indirect actions even when the computer exposes more structured capabilities. The paper Hybrid Computer Use proposes HCU, a unified interface that lets an agent choose among programmatic, semantic, and visual interaction.

What HCU changes

• Three channels in one interface: The programmatic channel supports code and command-line tools. The semantic channel exposes the accessibility, or AX, structure of an application. The visual channel preserves screenshots and GUI actions for cases where visual context is necessary.
• Model-driven selection: HCU does not place a separate planner or router above the agent. The model chooses code, AX, or GUI at each step and can switch channels during a task.
• Lower interaction overhead: AX controls can be addressed through short references, with up to 64 actions issued in a single call. Instead of repeatedly returning the full interface state, the tools report AX diffs. They also check the live target before sending input, helping guard against actions based on stale state.
• Practical deployment options: The project is provided as an MCP server, SDK, and CLI. According to the supplied material, it works with agent harnesses across Linux, macOS, and Windows.

Reported evaluation

On OSWorld 2.0, HCU recorded the highest Partial and Binary scores among the compared systems: 72.37% Partial with GPT-6 Astra and 35.19% Binary with Claude Opus 5. On OSWorld 1.0, replacing CUA Driver with HCU improved all four evaluated models by an average of 6.48 points, while reducing the number of steps by 17% and cost by 30%. The authors also report higher scores at roughly half the cost of CUA Driver across Windows, macOS, and Linux.

Why it matters

The central contribution is not simply adding more tools. It challenges the assumption that visual interaction should be the default interface for computer-use agents. A structured accessibility tree can be more direct than interpreting a screenshot, while code may be more efficient for deterministic file or system operations. GUI interaction remains useful when visual layout, rendering, or an unavailable semantic description is important.

The reported trajectories show that different models distribute work across the three channels differently, yet all evaluated models improve with HCU. This suggests that a common interface can support multiple operating styles rather than forcing every model through the same action pattern.

There are still open questions. The model must reliably decide when to use code, AX, or GUI, and benchmark gains may not transfer uniformly to every professional workflow. Even so, HCU offers a concrete direction for building agents that treat the computer as both a visual environment and a structured, programmable system.

Source: Hugging Face Daily Papers