AI-Generated Code Can Run. Can It Be Trusted?

Introduction

AI coding agents can now generate functions, write files, and assemble source trees from natural-language instructions. Yet working software is not necessarily secure software. The Last Secure Code Benchmark examines a practical question: if an agent receives only a functional specification and no explicit security guidance, will the resulting code withstand hidden vulnerability tests?

What the benchmark covers

The researchers built 450 tasks from 150 real-world vulnerabilities. The tasks span Java, C, C++, JavaScript, Python, PHP, and Go, and cover 24 CWE categories. This includes 21 weaknesses from the 2025 CWE Top 25 and eight weaknesses from the Top 10 KEV list.

The evaluation reflects three common vibe-coding workflows: completing a function, writing a file, and building a repository source tree from scratch. The prompts describe what the software should do, but do not tell the agent to implement particular security controls. Hidden functional and security tests then evaluate the output independently.

Key findings

• Functional pass rates across six frontier agents ranged from 73.6% to 91.3%.
• Passing the functional tests did not translate into secure implementations: the combined pass rate was only 16.4% to 25.1%.
• The strongest reported system, Claude Fable 5, reached a 25.1% functional-and-secure pass rate.
• Across all vibe-coding settings, the best model passed only 12% of the vulnerability scenarios.

Why it matters

The central lesson is not that AI coding tools are unusable. It is that functional success is an incomplete quality signal. Agents can often reproduce the expected business behavior while missing unsafe input handling, authorization boundaries, dangerous API usage, or edge cases that attackers may exploit. When the prompt omits security requirements, the model may optimize for a fast, plausible implementation rather than secure defaults.

For engineering teams, generated code should move through multiple gates: functional tests, static analysis, dependency and secret scanning, fuzzing, and review by developers who understand the threat model. High-risk components such as authentication, payment flows, file processing, and network-facing services deserve especially strict controls.

The benchmark also points to a gap in how coding agents are evaluated. Compilation and unit-test scores alone cannot describe production readiness. Future evaluations should measure security under ordinary prompts, multi-file changes, long-horizon repository work, and realistic vulnerability distributions. Until then, vibe coding should be treated as assisted implementation—not as a replacement for secure software engineering.

Source: Hugging Face Daily Papers