iPrefBench Tests Whether Mobile Agents Know When to Infer or Ask

Introduction

A mobile agent that can tap buttons, fill forms, and move between apps is not necessarily acting on behalf of its user. Everyday requests are often incomplete. A person may omit which contact to message, which preference to apply, or which of several acceptable options should be selected. A useful assistant must recover some of these details from the user’s history, while recognizing when guessing would be unjustified and asking for clarification instead.

iPrefBench is designed to measure this distinction. The benchmark places personal mobile-use agents in an executable iOS environment with an interactive simulated user. Its central question is not simply whether an agent can follow a workflow, but whether it can decide what to infer and what to ask before completing that workflow.

Key points

• A coherent user history anchors personalization. The benchmark provides a consistent persona and 3,051 past activity events. This makes it possible to test whether an agent uses user-specific evidence rather than relying only on generic assumptions.
• Tasks span a broad mobile setting. iPrefBench contains 130 underspecified tasks across 26 apps. Every omitted user-dependent requirement is deliberately classified as either inferable from history or ask-only.
• Execution and interaction are evaluated together. The benchmark checks both whether the agent reaches the correct personalized outcome and whether it seeks user input when the available context is insufficient.
• High action scores can hide weak personalization. In an evaluation of seven frontier agents, the best action completion score was 91.6%, but the best full task success score was only 47.7%. An agent may therefore perform the interface steps correctly while making the wrong decision for a particular user.
• Ask-only information is especially difficult. The evaluated agents often failed to request enough necessary information, revealing a persistent weakness in deciding when autonomous action should stop.

Why it matters

The benchmark shifts attention from GUI competence to the broader question of agency. A personal assistant that asks about every detail becomes cumbersome, while one that silently fills in unknowns can produce incorrect or unwanted results. Reliable assistance requires a calibrated boundary between memory-based inference and explicit confirmation, along with a way to ask only the questions that materially affect the outcome.

This makes iPrefBench useful not only as a leaderboard but also as a diagnostic framework. A failure can originate in retrieving relevant history, interpreting a preference, planning the task, or timing a clarification. Separating these behaviors helps researchers understand why an agent failed instead of treating every error as a generic execution problem.

For mobile agents to mature from automation tools into genuine personal delegates, they will need progress across all three dimensions: dependable interaction with apps, effective use of user history, and concise clarification dialogue. iPrefBench offers a focused setting for comparing those capabilities and for studying the trade-off between autonomy and user control.

Source: Hugging Face Daily Papers