Back to articles

Detecting Procedural Mistakes as They Happen in Video

3 min read

Introduction

For a video assistant to help with a procedure, spotting an error after the task is over may be too late. The more useful challenge is to recognize a deviation while the person can still recover. That requires more than recognizing an action: a system must connect what it sees to the task instructions, determine which step is being attempted, locate the action in time, and judge how it was performed. This study investigates that problem in egocentric procedural video, where the camera follows the person performing the task.

What the study tests

The researchers compare two ways of producing these judgments. A modular pipeline separates step identification, temporal localization, and execution verification into successive components. A joint model predicts the outputs together in a single step. Both operate online, relying on the video observed so far, the instructions, and prior predictions rather than waiting for the full recording to be available.

For each completed attempt, the system identifies the relevant instruction step, locates its time span, and assigns an execution category: correct, modified, slip, correction, or addition. Omitted steps are handled differently: they can only be established after the video ends. The distinction matters because an online assistant cannot confidently label an unperformed step as omitted while the person may still perform it.

The reported results show a nuanced trade-off:

  • Step-segmentation F1@0.5 is similar for the modular and joint approaches: 93.6% and 92.1%, respectively.
  • For binary mistake detection, the joint model scores 84.1% F1@0.5, versus 80.7% for the modular pipeline.
  • The largest category-level gap is on slips, where the joint model reaches 56.4% compared with 37.5% for the modular design.

The component analysis suggests that errors can propagate from one stage to the next in the modular approach, weakening the final mistake judgment even when step segmentation is strong. The study also repartitions EgoPER to include erroneous videos in every split, creating a basis for supervised training and evaluation that includes mistakes throughout the data partitions. On test videos containing mistakes, both proposed designs outperform released AEM and ESTANet checkpoints under the authors’ evaluation protocol. That comparison should be read as protocol-specific, not as a universal ranking of the systems.

Why it matters—and what remains open

The contribution is a framing of mistake detection as an instruction-conditioned, streaming task rather than an offline review problem. The results suggest that predicting step identity, timing, and execution quality together can reduce the impact of intermediate errors. The particularly large difference for slips is relevant because subtle execution deviations may be harder to catch than clearly incorrect or missing actions.

Still, these results do not establish that such a system is ready to guide real-world procedures. The reported evidence concerns benchmark performance; the provided material does not establish deployment latency, robustness across different tasks, or whether interventions improve user outcomes. Nor can every error be flagged in time: omissions are determined only after the recording ends. Further work would need to assess reliability in varied settings and the consequences of both false alarms and missed mistakes.

Hugging Face Daily Papers

Comments

Checking sign-in status...

Loading comments...

Related articles