StateAct puts program state before pixels for computer-use agents

Introduction

Most work on computer-use agents has focused on better perception: read the screenshot more accurately, locate the right button, and click with fewer mistakes. StateAct, a new paper highlighted on Hugging Face Daily Papers, takes a different path. Its central claim is simple but consequential: a screenshot is only a lossy projection of what matters. Files, application backends, DOM trees and saved artifacts are the actual task state; pixels are merely one rendering of that state.

Key points

• State first, pixels second: StateAct makes program state the main interface. The primary agent uses code to inspect and modify files, DOM structures and other underlying data, instead of relying mainly on screenshots. A dedicated GUI subagent is still available, but only for subgoals that truly require visual interaction.
• Multi-agent decomposition for long tasks: Long-horizon computer tasks can stretch across hundreds of steps. StateAct keeps the main agent focused by handing subgoals to fresh subagents, reducing context clutter from local operations. In the reported setup, only 28 of 108 tasks required the GUI subagent, and GUI use accounted for just 1.1% of main-agent steps.
• Verification is state-grounded too: The system adds an independent finish gate that checks whether the final output is missing, unsaved, or written to the wrong path. These structural failures are hard to catch reliably from pixels alone but are more directly observable through state access.
• Higher scores at lower cost: On OSWorld 2.0, StateAct raises Claude Opus 4.8 from 20.6% to 26.9% binary success and from 54.8% to 61.6% partial success. Compared with a screenshot-only setup using the same model, the reported cost per task is about nine times lower. A code-only variant without the GUI subagent reaches 45.9% partial success, below the screenshot baseline’s 54.8%, showing that the method is not about abandoning visual interaction entirely.

Why it matters

The broader message is that computer use should not be reduced to screen understanding. Many failures happen because an agent loses track of the task, saves the wrong file, overlooks a hidden state change, or fails to verify the final artifact. StateAct’s “state-grounding” approach shifts the bottleneck from perception toward reasoning, planning and reliable state management.

The results also suggest that the harness may be model-agnostic. The source reports gains not only with Claude Opus 4.8 across several benchmarks, but also with an internal 31B SFR-CUA model. That does not mean GUI agents are obsolete; real environments still require visual judgment and interaction. But it does imply that the next gains in computer-use agents may come less from looking harder at screenshots and more from acting on the program state underneath them.

Source: Hugging Face Daily Papers