Last Secure Code Benchmark: Is the Code You Vibe Secure?
cs.CR, cs.AI
Submitted: 2026-10-10
Updated: 2026-10-10
Comments: 10 pages
License: http://creativecommons.org/licenses/by/4.0/
The gist: AI coding agents now write much of the code that ships, yet whether that code is secure remains unclear.
Terminology
Abstract
AI coding agents now write much of the code that ships, yet whether that code is secure remains unclear. We introduce the Last Secure Code Benchmark, a comprehensive evaluation of secure code generation by coding agents. Built from 150 real vulnerabilities, its 450 tasks span seven widely used languages (Java, C, C++, JavaScript, Python, PHP, and Go), 24 CWEs including 21 of the 2025 CWE Top 25 and 8 of the Top 10 KEV weaknesses, more than any benchmark we compare with, and three vibe-coding scenarios: filling in a function, writing a file, and building a repository's source tree from scratch. Every task asks only for functionality, through a specification that never mentions security, and hidden functional and security tests grade the result. Across six frontier agents, 73.6% to 91.3% of tasks pass functionally, yet even the strongest, Claude Fable 5, produces code that is both functional and secure for only 25.1% of tasks, and the others for 16.4% to 24.7%. The best model only passes 12% of the vulnerability scenarios across all vibe coding setting, posing strong concern on the security of the code written by vibe coding.
Sources
- Program Synthesis with Large Language Models
- Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models
- Evaluating Large Language Models Trained on Code
- SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories
- NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents
- GLM-5: from Vibe Coding to Agentic Engineering
- Instruction Tuning for Secure Code Generation
- Measuring Coding Challenge Competence With APPS
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- PurpCode: Reasoning for Safer Code Generation
- CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability
- Training Software Engineering Agents and Verifiers with SWE-Gym
- Vibe coding: programming through conversation with artificial intelligence
- Kimi K3: Open Frontier Intelligence
- BaxBench: Can LLMs Generate Correct and Secure Backends?
- ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?
- Secure Code Generation via Online Reinforcement Learning with Vulnerability Reward Model
- SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution
- ProSec: Fortifying Code LLMs with Proactive Security Alignment
- Autoregressive, Yet Revisable: In Decoding Revision for Secure Code Generation
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills
- Token Composition: A Graph Based on EVM Logs