Benchmarking and Exploring the Capabilities of LLMs for Attack Investigations
cs.CR, cs.CL
Submitted: 2026-06-09
Updated: 2026-09-26
Code: https://github.com/FiveDirections/OpTC-data
License: http://creativecommons.org/licenses/by/4.0/
The gist: This paper presents AuditBench, a new benchmark dataset for evaluating the capabilities of LLMs at investigating security-related system audit logs.
Terminology
Abstract
This paper presents AuditBench, a new benchmark dataset for evaluating the capabilities of LLMs at investigating security-related system audit logs. We design and use this benchmark to explore the performance of LLMs on four log-investigation tasks that incident response teams commonly perform, ranging from triaging alerts generated by detectors to identifying persistence mechanisms on compromised systems. AuditBench consists of system audit logs collected from Linux and Windows machines, and spans over 50 different security investigation scenarios, including both malicious and benign activity. Using our benchmark, we evaluate and analyze the performance of five frontier LLMs at analyzing audit logs for attack investigations. Our analysis illuminates how LLM performance and error profiles vary according to different design choices, such as differences in model size, data representation, prompt construction, and specific investigation tasks. Additionally, we characterize the quality of the explanations produced by LLMs and the types of errors that models make across our benchmark. Collectively, our work provides a foundation for assessing the capabilities of LLMs for investigating security logs, novel insights for practitioners using LLMs in security operations, and important directions for future research.
Sources
- LLM-based event log analysis techniques: A survey
- LogLLM: Log-based Anomaly Detection Using Large Language Models
- Does Prompt Formatting Have Any Impact on LLM Performance?
- SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
- RedChronos: A Large Language Model-Based Log Analysis System for Insider Threat Detection in Enterprises
- Just Ask One More Time! Self-Agreement Improves Reasoning of Language Models in (Almost) All Scenarios
- Audit-LLM: Multi-Agent Collaboration for Log-based Insider Threat Detection
- BaxBench: Can LLMs Generate Correct and Secure Backends?
- CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale
- ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation
- BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Estimating the Social Cost of Corporate Data Breaches
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills