GMA Puts Mobile Agents Through More Realistic, Complex Workflows

Mobile interfaces are becoming an important test environment for multimodal agents. Yet completing a realistic phone task involves much more than locating a button. A user request may span several screens, require information to be remembered across steps, and depend on the agent maintaining an accurate view of what has already happened. These demands expose weaknesses that are easy to miss in short, isolated interaction tasks.

The GMA benchmark, short for General Mobile Assistants, is designed to make those weaknesses more visible. The authors position it as a complement to benchmarks such as AndroidWorld and MobileWorld. Those benchmarks established useful foundations for testing mobile agents, but the authors argue that existing application coverage and task design do not fully represent the diversity and complexity of everyday mobile use.

GMA introduces seven applications built from open-source projects. The applications cover areas including lifestyle sharing and travel planning, giving agents a broader range of interfaces and user objectives to handle. The benchmark contains 300 tasks organized into four difficulty tiers. At the simpler end are atomic actions; at the harder end are workflows that require multiple connected operations and sustained attention to the user’s requirements.

Key points

• Broader task coverage: The benchmark moves beyond isolated taps and text entries to test whether an agent can interpret a goal and carry it through across a mobile application.
• Complexity sharply changes outcomes: An evaluation of eight frontier models found that performance declines substantially as tasks become more complicated. Interface fluency alone does not guarantee dependable task completion.
• The harness is part of the system: Controlled ablations examined choices such as retaining context and tracking state explicitly while keeping the environment, model setting, and task taxonomy shared. Appropriate harness designs improved performance, particularly on demanding workflows, but their benefits varied across foundation models.

The findings suggest that mobile-agent reliability is not solely a question of the underlying model. The surrounding execution system also determines how information is preserved, how the current state is represented, whether intermediate results are checked, and how the agent responds when a step fails. In practice, a capable model can still lose track of a workflow if the harness does not provide an effective way to maintain that structure.

This changes how mobile assistants should be evaluated. Instead of asking only whether a model can identify the right visual target, researchers need to ask whether the model and its execution framework can jointly satisfy an open-ended requirement. Long workflows are where planning, memory, perception, and action become inseparable.

For developers, GMA offers a way to locate failure points in extended interactions and compare different approaches to context and state management. For researchers, it provides a testbed with wider application coverage and a more demanding task taxonomy. The benchmark does not imply that more tasks alone will solve reliability. Its central message is more specific: before mobile assistants can be trusted with ordinary real-world requests, they must become substantially more consistent at carrying complex workflows through to completion.

Source: arXiv