CyberFactory Turns Real-World Vulnerabilities into Verifiable Agent Training Tasks

Introduction

Cybersecurity models need more than large parameter counts or static vulnerability descriptions. They must inspect code, operate tools, interact with target environments, interpret failures, and revise their plans. CyberFactory presents an open-source recipe for producing training data that captures this workflow. The system reconstructs executable tasks from public vulnerability artifacts, synthesizes tool-using trajectories, and keeps only outputs that pass task-specific checks. The authors use the resulting data to train OpenAegis.

Building tasks from public artifacts

CyberFactory covers proof-of-concept generation, vulnerability patching, and cybersecurity question answering. Its sources include ARVO, OSS-Fuzz, and CVEs collected from the wild. For each relevant case, the pipeline reconstructs vulnerable and patched program states, writes a task description, and withholds privileged verification signals before trajectory generation.

PoC generation is treated as a behavioral problem rather than a text-generation problem. A candidate must trigger the target vulnerability in the pre-patch build and fail to trigger it in the post-patch build. This differential oracle creates a machine-checkable loop: propose a solution, execute it, inspect the evidence, and refine the approach. Patch-generation examples are grounded in vulnerability-fix records, while CyberQA examples are tied to execution outcomes, code structure, or authoritative reports.

Teaching a workflow and internalizing it

During synthesis, the teacher model receives a reusable vulnerability-analysis skill. It encourages the model to inspect the target and build constraints, apply domain-guided exploration, validate evidence, and revise its method when validation fails. The skill does not contain the instance-specific answer; the teacher still has to solve each case through interaction with the environment.

The intended transfer is especially important: OpenAegis does not receive the skill prompt at inference time. Instead, supervised fine-tuning is used to move the workflow into the model’s parameters. In the reported behavior analysis, domain-prior-guided exploration appears in 85.2% of OpenAegis trajectories, compared with 0.6% for Qwen3.5. Exploration and validation calls also become more frequent after training.

Results and caveats

With the same scaffold and a one-hour CyberGym budget, OpenAegis records 58.1% Pass@1, compared with 29.6% for Qwen3.5, 43.3% for GLM 5.2, and 51.7% for Kimi K2.7. The reported gain over the Qwen3.5 base model is 28.5 percentage points.

The explicit-skill comparison requires careful reading. GLM 5.2 with the skill uses 15 minutes per attempt and five repetitions, while the no-skill setting uses one 60-minute attempt. It therefore indicates improved synthesis throughput and a change in behavior, not a compute-matched performance comparison.

Why it matters

CyberFactory’s main contribution is an end-to-end data recipe. Public vulnerability records become executable instances; execution-based filters turn them into grounded supervision; and fine-tuning transfers tool use, validation, and revision into a model. This is a stronger foundation than rewarding answers that merely sound technically plausible.

The approach still depends on accurate environment reconstruction, well-designed verifiers, and adequate test coverage. CyberGym results also represent a specific benchmark configuration rather than a complete measure of real-world security performance. Even so, CyberFactory offers a concrete open-source framework for studying scalable cybersecurity agents.

Source: Hugging Face Daily Papers