Security Agents Need Cost-Aware Benchmarks, Not Just Success Rates

Introduction

AI security agents are increasingly being tested on tasks that resemble real security work: finding vulnerabilities, developing exploits, completing CTF challenges, and assisting analysts with investigations. Yet many evaluations still focus on a simple question: did the agent solve the task when given a generous inference budget?

The paper “Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents” argues that this view is incomplete. In operational security, every model reasoning step, tool invocation, telemetry query, and enrichment request has a cost. A model that solves more tasks only by spending far more may not be the best fit for production environments.

Key points

• The benchmark lens shifts from success to efficiency. Traditional security-agent evaluations often emphasize peak offensive capability, especially in vulnerability discovery, penetration testing, exploit development, and CTF completion. The authors argue that these results are useful but do not capture whether an agent is economically practical.
• The study covers both offensive and defensive work. The evaluation uses offensive Cybench challenges and defensive Splunk BOTS v1 investigation tasks. This matters because attacking a challenge and investigating a SOC incident require different behaviors, tools, and decision patterns.
• Models are compared at fixed cost levels. Rather than only reporting the best result a system can achieve, the paper compares models under cost constraints. It also breaks performance down by inference spend and tool spend, making it easier to see whether success comes from more reasoning, better tool use, or both.
• Red-team and blue-team tasks scale differently. Offensive CTF performance improves with additional test-time compute. The paper also notes that scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive in this setting.
• SOC investigation is not just a reasoning-budget problem. Defensive tasks do not scale in the same way. Success depends more on disciplined tool use, effective telemetry navigation, and selective enrichment than on simply giving the model more reasoning budget.

Why it matters

This work highlights a practical gap between leaderboard-style evaluation and security operations. A security team does not only need an agent that can eventually reach the right answer; it needs one that can do so within a budget, without excessive queries, unnecessary enrichments, or uncontrolled tool use.

The findings also suggest that open-weight models may be competitive in certain offensive settings when test-time compute is scaled carefully. But the defensive side appears more demanding in a different way: better SOC agents may require stronger tool policies, query planning, telemetry awareness, and mechanisms for deciding when additional information is worth the cost.

The broader message is that security-agent benchmarks should measure operational fit, not just task completion. Cost-aware and SOC-native evaluations can help teams understand which agents are useful today and where current systems still fall short.

Source: Hugging Face Daily Papers