EgoProceVQA Tests Whether Egocentric Video Models Understand Procedures

Introduction

Egocentric video has become a natural interface for AI systems that may one day run on wearable devices. Such systems do not only need to recognize objects or actions; they must understand how a task unfolds over time. The arXiv paper EgoProceVQA: A Novel Egocentric Procedural Understanding Task with Self-Skill-Exploration Agent addresses this gap by proposing a new video question answering task centered on procedural understanding.

The authors argue that many daily activities are inherently procedural. Cooking, assembling, cleaning, or repairing all involve ordered steps, key actions, and dependencies between what has already happened and what should happen next. Yet many existing egocentric video benchmarks pay limited attention to whether multimodal large language models can reason over these key steps.

Key points

• A new task: EgoProceVQA. The benchmark evaluates egocentric procedural reasoning through six types of key-step-centric questions under the widely used VQA paradigm.
• A generation platform: EgoProceGen. The authors build a data generation platform designed to efficiently create question-answer data for different procedural question types.
• Benchmark scale and scope. The resulting benchmark contains 3,600 questions, covering four common procedural scenarios and 31 everyday procedural tasks.
• A focus on process, not just perception. Instead of only asking what appears in a video, the task probes whether models can understand step order, key actions, and task-relevant procedural context.
• A new agent framework: EgoProceAgent. The paper proposes a self-skill-exploration agent with a generic procedural tool library and a standardized sub-skill library shared across tools and models.

Why it matters

The distinction between visual recognition and procedural understanding is important. A model may correctly identify a knife, a cup, or a hand movement, but still fail to infer which step of a task is being performed or whether a necessary action has been skipped. For AI assistants deployed on wearable devices, this capability is central: the assistant must track progress, interpret intent, and answer questions grounded in the user’s ongoing activity.

EgoProceVQA therefore reframes egocentric video evaluation around task structure. This is especially relevant for future personal assistants, embodied AI systems, and robotics-adjacent applications, where the goal is not merely to caption a video but to provide context-aware procedural support.

The proposed EgoProceAgent also reflects a broader trend in AI systems: moving from single-shot model responses toward agentic reasoning with tools and composable skills. By exploring how to select and combine sub-skills without ground-truth supervision, the framework attempts to discover effective strategies for different procedural questions. According to the paper, this allows it to achieve state-of-the-art performance among open-source models on multiple tasks.

The work should still be read as an early research benchmark rather than a complete solution for real-world wearable assistants. Its dataset covers a defined set of scenarios and tasks, and broader deployment would require more diverse, noisy, and safety-sensitive real-world data. Still, EgoProceVQA highlights a crucial evaluation gap: seeing a procedure is not the same as understanding it.

Source: arXiv