SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities
cs.CR, cs.AI
Submitted: 2026-09-27
Updated: 2026-09-27
License: http://creativecommons.org/licenses/by/4.0/
The gist: Assessing cybersecurity vulnerability awareness in coding agents requires evaluations that reveal capability gaps and remain informative as models evolve.
Terminology
Abstract
Assessing cybersecurity vulnerability awareness in coding agents requires evaluations that reveal capability gaps and remain informative as models evolve. Static benchmarks offer fixed coverage and difficulty, while scarce vulnerable repositories and costly expert authoring limit their renewal at scale. We introduce SecProbe, a framework for adaptive evaluation that combines Item Response Theory (IRT) with on-demand synthesis of repository-scale vulnerability-repair tasks. From observed performance, SecProbe estimates agent ability and identifies where additional evidence is most informative, selecting existing tasks or synthesizing new ones accordingly. As one use case, we construct 353 tasks spanning six programming languages and 151 CWE types and evaluate nine frontier models with two agent harnesses. Success rates peak at 28.33%, highlighting substantial gaps in vulnerability recognition and repair. Compared with random and one-shot baselines, SecProbe achieves comparable agent ability estimates while requiring agents to solve up to 29.5% fewer tasks. These results support adaptive evaluation as an efficient and discriminative approach to assessing cybersecurity vulnerability awareness.
Sources
- RAS-Eval: A Comprehensive Benchmark for Security Evaluation of LLM Agents in Real-World Environments
- Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models
- A Rosetta Stone for AI Benchmarks
- MiniMax Sparse Attention
- UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
- SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents
- How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench
- The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark
- Kimi K3: Open Frontier Intelligence
- SWE-Mirror: Scaling Issue-Resolving Datasets by Mirroring Issues Across Repositories
- Simple synthetic data reduces sycophancy in large language models
- How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench
- Oasis: One Image is All You Need for Multimodal Instruction Data Synthesis
- SWE-Explore: Benchmarking How Coding Agents Explore Repositories
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Estimating the Social Cost of Corporate Data Breaches
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills