APort Vault Draws a Harder Authorization Boundary for AI Agents

Introduction

Once an AI agent can call a payment tool, safety is no longer just a question of whether the model refuses a suspicious prompt. The more important question is whether the final action is checked against the recipient and the authorization policy before money moves. APort Vault, presented on Hugging Face Daily Papers, is designed to measure that boundary.

Key findings

• A large, attack-based evaluation. The benchmark replays 4,371 attacks written by humans during a public capture-the-flag event. It covers 14 models from eight labs, five policy configurations, and two replay tracks: one with a deterministic pre-action check implementing the Open Agent Passport specification, and one without it. In total, 225,964 evaluations were completed.
• More than one metric matters. The study records separate events for each evaluation, including whether the agent made a request, whether a payment was executed, whether policy denied a call, and whether the recipient was authorized. This avoids turning different failure modes into a single headline rate.
• Configuration changes request behavior. For model-only evaluations, request rates at Levels 1 through 4 were 10.9%, 3.0%, 0.1%, and 79.4%. These levels do not form a perfectly isolated experiment, because each attack belongs to one configuration and the attack cohorts therefore vary with policy.
• Level 4 produced broad model agreement. Across 1,293 Level 4 prompts evaluated by every model, request rates ranged from 71.2% to 84.3%. A total of 809 prompts triggered a request from all 14 models, and each resulted in a successful payment to the recipient allowed at that level.
• The sharpest difference appeared at the authorization boundary. Across Levels 2 to 4, model-only runs produced 140 transfers to recipients not permitted by the passport, out of 76,842 evaluations. With the OAP layer, the corresponding count was zero across 69,297 evaluations. In matched model, prompt, and track triples, the comparison was 105 to zero.

Why it matters

The result does not come from refusing every payment. Behind the layer, 25,370 payments were executed. Of 25,640 transfer calls evaluated by the policy, 187 were denied, including 148 involving a forbidden recipient. This suggests a useful design pattern: keep the agent capable of completing permitted transactions, while moving the final authorization check into a deterministic control layer rather than relying solely on model judgment.

The finding should still be read as a benchmark result, not as a proof of universal security. The attacks, policy levels, and replay conditions define the tested scope, and the attack cohort is coupled to configuration. The reported zero spans 790 source sessions, yielding a per-session upper bound of 0.38%. APort Vault’s broader contribution is methodological: payment-agent evaluations should separately measure intent, tool execution, policy decisions, and recipient authorization. Only then is it possible to see whether a system is refusing too much, allowing too much, or enforcing the wrong boundary.

Source: Hugging Face Daily Papers