HazardAuditor Brings Execution-Grounded Safety Supervision to Computer-Use Agents

Introduction

Safety evaluation becomes substantially harder when an AI system can do more than generate text. A computer-use agent may browse websites, invoke terminal commands, access files, or communicate with external services. In such settings, the relevant risk may not be visible in the initial prompt or the final answer. It can emerge from the sequence of actions taken during execution.

HazardAuditor is designed around this shift. Instead of treating safety as a static input-output classification problem, the framework uses the agent’s execution trace as the primary source of supervision. The goal is to make safety guards better suited to systems that act in an environment, not merely respond in a chat window.

Key points

• Execution is part of the safety object. Existing guard models commonly focus on prompts and responses. HazardAuditor extends the view to tool calls, commands, environment interactions, and other runtime events that may determine whether an episode is safe.
• A common representation across frameworks. The infrastructure runs Claude Code, Codex, Hermes, and OpenClaw in controlled environments. Their interactions are then normalized into a canonical event representation, allowing supervision and comparison across otherwise different agent implementations.
• A training objective designed for generative guards. The paper identifies a mismatch in token-level post-training: long rationales can contribute more heavily to gradient updates than the final safety verdict. Guard Policy Optimization, or GuardPO, converts deterministic safety outcomes into sequence-level advantages and normalizes the rationale and verdict regions.
• Reported gains across heterogeneous settings. According to the paper, HazardAuditor improves accuracy by as much as 16.5 percentage points over the strongest prior guard across multiple benchmarks and computer-use systems. The authors also state that code, models, and evaluation artifacts will be made available through the project page.

Why it matters

The broader contribution is a change in what should be monitored. For an agent with access to tools, checking what it says is only one layer of safety. Evaluators also need to track what it does, in what order, and how those actions affect the environment. A canonical event format offers a shared observation layer for agents built with different orchestration frameworks, which could make cross-system safety training more practical.

GuardPO addresses a separate but important issue in guard-model training. A lengthy explanation is not necessarily a better safety decision. If token-level objectives allow rationale length to dominate optimization, a model may produce elaborate reasoning without improving its final classification. By emphasizing sequence-level outcomes, GuardPO attempts to align the learning target with the decision that operators ultimately need.

The reported improvement should still be read within the paper’s evaluation setup. The supplied material does not include a benchmark-by-benchmark breakdown, so the maximum 16.5-point gain should not be treated as a universal result for every agent or production environment. Future evaluation will need to test whether execution-grounded supervision remains reliable across more tools, attack patterns, and real-world workflows.

Source: Hugging Face Daily Papers