CyberArena Puts AI Agents in Live Attack-and-Defense Combat

Introduction

An AI agent that can compromise a fixed target may be good at solving a security task, but that does not necessarily mean it can operate in a real cyber conflict. Most existing benchmarks isolate the agent from an adaptive opponent and evaluate whether it can break into a static service. CyberArena takes a different approach: it treats security work as a continuing contest in which attackers search for weaknesses while defenders monitor, patch and respond.

A benchmark built as a contest

CyberArena places two agents in an isolated arena under the same rules. Each agent must attack the opponent’s service and capture a flag verified by the environment, while also defending its own service and trying to retain its flag. The match proceeds through multiple rounds, and the services evolve as both sides act. Success therefore depends on more than finding an exploit once. Agents must decide when to scan, exploit, patch or react, often while the opponent is changing the situation.

The reported evaluation covers 40 challenges: 14 adapted from classic CTF tasks and 26 based on real-world vulnerabilities in open-source software. Twelve frontier language-model agents played 466 ranked matches. GPT-5.6 Sol ranked first with a score of 79%, followed by Claude Opus 5 at 74% and GLM-5.3 at 69%. The same leading models also performed best defensively, retaining their flags in 83% and 81% of rounds, compared with 43–61% for the remaining agents.

Live defense changes the result

The clearest finding is that a static benchmark can make different systems look equivalent. Three frontier models each solved the same 30 challenges when the service had no active defender. Once a live defender was introduced, their capture rates fell to 35–52% by the third round, and the models began to separate. The result highlights a central property of operational cybersecurity: the challenge is not simply discovering a vulnerability, but continuing to act effectively after an adversary has noticed and changed the environment.

To keep the benchmark from quickly becoming saturated, the project includes an agentic pipeline that generates, reviews and validates new environments. It also proposes a dynamical model that converts match trajectories into unified, multidimensional scores covering offensive, defensive and domain-specific capabilities. The engine, challenges and match replays are available as open-source resources.

Why it matters

CyberArena shifts the question from “Can the agent break in?” to “Can it maintain useful behavior through an adaptive attack-and-defense loop?” That makes speed, precision, resilience and the balance between offense and defense visible in the same evaluation. Developers may gain more actionable evidence than a single pass/fail result, including whether a model is strong at exploitation but weak at patching or sustained response.

The available material mainly describes the benchmark and aggregate findings. It does not yet establish which tools or strategies produced individual outcomes, or how strongly the scores predict performance in operational environments. Public replays, broader environments and comparisons with other benchmarks will be important for interpreting the results.

Source: Hugging Face Daily Papers