Closing the Sim-to-Real Gap: An Evaluation Framework for Autonomous Cyber Defense Configuration of Commercial EDR

arXiv:2606.08168 · cs.CR, cs.AI · Submitted 2026-06-06 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "Closing the Sim-to-Real Gap".

Elias: Here's the gist of the paper. The gist: This paper presents an evaluation framework for autonomous defense agents hardening commercial EDRs by instantiating it in a Game of Active Directory lab with Horizon3.ai’s NodeZero as the autonomous pentester and Microsoft Defender XDR as the EDR, reporting three lessons learned that highlight a sim-to-real gap for enterprise defense and motivate evaluation methodology for benchmarking autonomous defense agents in environments with black-box, autonomous tools.

Nadia: The need for an evaluation framework stems from several critical gaps in existing literature regarding commercial EDRs and cyber defense configurations In this section, we motivate the need for an evaluation framework that integrates commercial EDR by examining gaps in MITRE ATT&CK technique coverage, differences in vendor-claimed and observed EDR behavior, false positives and lack of per-policy attribution, and black-box, autonomous features Attack Coverage Gap: Recent research has demonstrated that commercial EDRs exhibit coverage gaps of the MITRE ATT&CK techniques

13: and multi-step, chained attacks EDR systems do not natively reason about kill-chain causality and fragment multi-step attacks into single-step alerts

14: ,

15: In 2024, Virkud et al.

16: found that EDR detection rule sets from Carbon Black, Splunk, and Elastic only cover about 48% to 55% of MITRE ATT&CK techniques and that coverage of an ATT&CK technique does not guarantee coverage of the same real-world attacks Expected vs.

Nadia: Observed Behavior Gap: Vendor-claimed, expected EDR effectiveness diverges substantially from observed performance under real-world attacks Karantzas and Patsakis

17: empirically assessed eleven commercial EDR products against four Advanced Persistent Threat (APT) scenarios and found that most products neither block nor log the majority of attacks

17: They also echo the need for holistic event analysis for effective defense and identify assessing true-positive, false-negative, false-positive results produced by EDRs and the response time of EDRs during real-world scenarios as a critical gap in the literature

17: False Positives and Per-Policy Attribution Gap: In a four-year longitudinal study of real-world network alert logs from Security Operations Centers (SOCs), Yang et al.

Title and authors: Nadia: So this paper is called "Closing the Sim-to-Real Gap: An Evaluation Framework for Autonomous Cyber Defense Configuration of Commercial EDR". It sounds like they’re trying to figure out how to test defenses that are now running on their own, not just following simple rules.

Elias: Yeah, that's the core idea. They’re looking at these commercial endpoint detection and response products where the AI components are starting to operate alongside or even instead of what a human operator sets up.

Nadia: It means we're moving from tuning a static tool to dealing with something black-box that can make vendor-specific decisions on its own.

Elias: Exactly, they’re testing those autonomous defense agents by putting them in a Game of Active Directory lab with NodeZero as the attacker and Microsoft Defender XDR as the EDR.

Nadia: So, what's the big deal about that setup? It sounds like a very specific test bed for seeing how these agents actually perform under pressure.

Elias: Well, the whole point is to report on three lessons learned that neither simulation nor open-source EDR evaluation can surface. They want to see what happens when you combine an autonomous attacker with a commercial EDR in this way.

Priya: I’m interested in those lessons, because right now we don't really know how much we can trust the results from these automated configurations when they are operating like that.

Nadia: It sounds like they’re setting up a system to see if these autonomous agents can actually harden the EDR effectively in a way that matters in the real world.

Elias: That's right, the framework itself is about integrating commercial EDR by looking at coverage gaps, observed versus claimed behavior, and those tricky false positives.

The paper's summary: Nadia: Okay, so the summary of this paper explains that there are several big gaps in how we currently evaluate commercial EDRs. They point out issues like not enough MITRE ATT andCK technique coverage for what these tools actually detect, and they don't account for how multi-step attacks get fragmented into single alerts.

Elias: And then there’s the gap between what vendors claim their EDR can do and what it actually does when you run real attacks. They also highlight the problem of false positives overwhelming analysts, which is a huge issue in SOCs today.

Priya: That aspect about false positives is really important for me because if an agent makes a change that just floods the system with noise, then it’s not helping defense at all.

Nadia: Right. And they also point out that EDR policy optimization is this incredibly complex, combinatorially intractable problem when you have so many settings available in something like Microsoft Defender for Endpoint.

Elias: That leads into the final gap they address, which is these black-box autonomous features embedded in things like Microsoft Defender XDR. You can have automation toggles that are all one thing, and you don't really get visibility into what’s driving the response decisions.

Nadia: So they’re basically saying we need a way to evaluate these autonomous systems, not just their individual components, but how they work together in this feedback loop.

Elias: Precisely. The paper proposes a closed-loop evaluation framework where an agent proposes a policy change, you run it through a commercial pentest like NodeZero, and then the results feed back to the agent for another round of decisions.

The paper's improvements: Nadia: So they outline some improvements they suggest for this whole process. One big thing is decoupling experiment orchestration from the policy-decision algorithms using a pluggable proposer interface.

Elias: That way, you can test different AI architectures or models against the same testing harness without rewriting the whole framework every time, which makes benchmarking across different agentic designs much cleaner.

Priya: And they also suggest incorporating a deterministic gating mechanism before deployment to check transition cost budgets and game value.

Nadia: That means they want to make sure that when an agent proposes a policy change, it actually fits within defined operational budgets and doesn't just open the door for more attacker gain.

Elias: It also points toward being able to provide per-policy attribution, separating what the defense agent does from what the autonomous EDR does.

Priya: That would be useful because it lets us quantify exactly how much of the observed block came from our policy tuning versus just the vendor's built-in autonomous features.

Nadia: And they also suggest accounting for variable EDR behavior during evaluation windows by treating that evolving layer as a measurement axis, meaning you have to run ablation studies on things like automation levels.

Conclusion: Elias: So to wrap up the "Closing the Sim-to-Real Gap: An Evaluation Framework for Autonomous Cyber Defense Configuration of Commercial EDR" paper, it’s about establishing a structured way to evaluate these autonomous defense agents by running them in a controlled loop against a real attacker.

Nadia: They found that this closed-loop process, using NodeZero and Microsoft Defender XDR in the GOAD lab, gives us concrete lessons about what we don't know about commercial EDRs right now.

Priya: From my side, it sounds like the most important thing is moving toward having per-policy attribution so we can actually measure if our tuning efforts are paying off in terms of risk reduction.

Elias: And they emphasize that this framework helps us see how these black-box autonomous features interact with operator choices and whether the agent can actually make good decisions under those constraints.

Nadia: So, it’s a way to test hardening without risking actual production systems by simulating the real-world impact of tuning an automated defense tool.

Priya: I think this is a solid step toward making sure that when we deploy these autonomous agents, we have a method to verify their behavior and not just trust the vendor's marketing materials.

Kerri Prinos, Lilianne Brush

Horizon3.ai

cs.CR, cs.AI

Submitted: 2026-06-06

Updated: 2026-10-21

Comments: 17 pages including references and appendix

Code: https://github.com/Orange-Cyberdefense/GOAD

Project page: https://www.microsoft.com/en-us/security/business/siem-and-xdr/microsoft-defender-xdr

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 90/100

The gist: The gist: This paper presents an evaluation framework for autonomous defense agents hardening commercial EDRs by instantiating it in a Game of Active Directory lab with Horizon3.ai’s NodeZero as

Key concepts

Sim-to-Real Gap: This refers to the difficulty in accurately predicting how a defense system will perform in a live, real-world network versus its performance during testing. The paper investigates this gap by using a controlled lab environment to simulate real attacks against commercial EDRs.
Closed-Loop Evaluation Framework: A systematic process where an autonomous agent proposes policy changes, which are tested via penetration testing, and the results are fed back to the agent for refinement. This iterative cycle allows the system to learn optimal defense configurations over time.
Autonomous Defense Agents: These are AI systems designed to automatically tune security settings. They operate by proposing configuration changes based on observed attack data and testing those changes in a controlled environment before deploying them, aiming to improve security without constant manual intervention.
Black-Box Autonomous Features: These are advanced detection and response capabilities embedded within commercial EDRs that operate autonomously. The paper focuses on the challenge of controlling these features because their internal decision-making processes are often hidden from the user, making it hard to understand exactly how they work.

Terminology

Benchmarking — 2× in this paper · explained in 4 episodes
Evaluation Framework — 2× in this paper · explained in 3 episodes
Sim-to-Real Gap — 2× in this paper · explained in 7 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Large Language Model (LLM) — 1× in this paper · explained in 3 episodes

Summary

The gist: This paper presents an evaluation framework for autonomous defense agents hardening commercial EDRs by instantiating it in a Game of Active Directory lab with Horizon3.ai’s NodeZero as the autonomous pentester and Microsoft Defender XDR as the EDR, reporting three lessons learned that highlight a sim-to-real gap for enterprise defense and motivate evaluation methodology for benchmarking autonomous defense agents in environments with black-box, autonomous tools.

Background

The need for an evaluation framework stems from several critical gaps in existing literature regarding commercial EDRs and cyber defense configurations In this section, we motivate the need for an evaluation framework that integrates commercial EDR by examining gaps in MITRE ATT&CK technique coverage, differences in vendor-claimed and observed EDR behavior, false positives and lack of per-policy attribution, and black-box, autonomous features

** Attack Coverage Gap: Recent research has demonstrated that commercial EDRs exhibit coverage gaps of the MITRE ATT&CK techniques [13] and multi-step, chained attacks EDR systems do not natively reason about kill-chain causality and fragment multi-step attacks into single-step alerts [14], [15] In 2024, Virkud et al. [16] found that EDR detection rule sets from Carbon Black, Splunk, and Elastic only cover about 48% to 55% of MITRE ATT&CK techniques and that coverage of an ATT&CK technique does not guarantee coverage of the same real-world attacks **

** Expected vs. Observed Behavior Gap: Vendor-claimed, expected EDR effectiveness diverges substantially from observed performance under real-world attacks Karantzas and Patsakis [17] empirically assessed eleven commercial EDR products against four Advanced Persistent Threat (APT) scenarios and found that most products neither block nor log the majority of attacks [17] They also echo the need for holistic event analysis for effective defense and identify assessing true-positive, false-negative, false-positive results produced by EDRs and the response time of EDRs during real-world scenarios as a critical gap in the literature [17] **

** False Positives and Per-Policy Attribution Gap: In a four-year longitudinal study of real-world network alert logs from Security Operations Centers (SOCs), Yang et al. [19] found that analysts are overwhelmed with 24k to 134k alerts per day, but only 0.01% of the alerts are associated with true attacks In the workforce, more than 25% of analysts’ time is spent handling false positives [20], contributing to alert fatigue and burnout [20], [21] The EDR Telemetry Project [18] provides insights into technique to policy mapping, but not a per-policy attribution analysis of false positive rates **

** Vast Configuration Space and No One-SizeFits-All Solution: EDR policy optimization is a combinatorially intractable problem Consider Microsoft Defender for Endpoint (MDE), which has more than 100 baseline policy settings available each with multiple options [22] This 30-day audit-then-enforce cadence functions as an effective maintenance-window budget: an enterprise can responsibly introduce only a limited number of policy updates per cycle without risking line-of-business disruption [24] **

** Black Box Autonomous Features: Today’s leading commercial EDR and XDR products, such as Microsoft Defender XDR, CrowdStrike Falcon, and SentinelOne Singularity, embed autonomous detection and response capabilities that operate alongside or – in some products – in place of operator-configured rule-based policies The granularity of operator control over these autonomous layers varies by vendor For example, Microsoft Defender XDR’s Automated Investigation and Response (AIR) exposes coarse controls automation-level toggles (vendor-default is full automation [25]) and per-account exclusion lists, with no per-decision visibility into the classifier that drives ContainedUser**

Approach

The paper presents a closed-loop evaluation framework for autonomous tuning of commercial EDR This framework deploys policy configurations selected by a pluggable proposer (e.g., defense LLM-based agent), measures the effect of each change by running a commercial penetration test (pentest), and feeds the resulting telemetry back to the proposer for the next round of decisions The experiment controller executes four sequential stages: Preflight, Setup, Loop, and Teardown

** Experiment Controller Overview: The orchestration controller connects to five external components as shown in Fig. 1: the GOAD lab [11], an eight-host Active Directory environment on Azure with Microsoft Defender XDR; Horizon3.ai’s commercial pentesting platform NodeZero fills the role of autonomous attacker; Redshift, which stores NodeZero’s attack telemetry (events, paths, weaknesses, credentials); and Microsoft Defender XDR, which produces detection and response telemetry via the Advanced Hunting API; and a proposer wrapper for the autonomous defense agent under test that receives the telemetry and returns policies to deploy **

** Environment: In this initial implementation of the evaluation framework, we use the GOAD (Game of Active Directory) lab as the test environment We extend the original GOAD lab to include eight hosts: seven Windows hosts and one Linux endpoint (Fig. 2) The two-forest cross-trust forces NodeZero to perform crossdomain credential theft and trust traversal rather than trivially reaching every host from any compromised host; the three member servers (one per domain) provide non-DC pivot points so that lateral-movement edges into the attack graph are sensitive to policies that apply only to nonDC Windows hosts; and the single workstation (ws01) is the typical initial-access target and the only host where browser-**

** Attacker: We select NodeZero, Horizon3.ai’s commercial autonomous pentester, which chains multi-step attacks to mimic real attacker behavior [12] NodeZero is plugged into the framework via its H3 CLI connector **

** Commercial EDR/XDR: Microsoft Defender XDR was selected because it is the market-share leader among commercial EDR products and because its Advanced Hunting query surface preserves per-event data at a granularity sufficient for raw-telemetry correlation **

Closed-Loop Evaluation Stages

The experiment controller executes four stages (Fig. 3) for experiment runs: Preflight, Setup, Loop, and Teardown

Preflight: This stage catches the most common failure modes before the experiment starts, running seven automated checks in parallel across host reachability, MDE onboarding state, credentials, and infrastructure health
Setup: Setup establishes a uniform baseline configuration across all hosts, runs a pentest to measure the starting state, and builds an initial attack graph from the resulting offensive and defensive telemetry that the proposer will reason over in subsequent rounds
Loop: The experiment controller executes the closedloop policy-tuning protocol in the loop stage Each iteration proceeds through eight stages until the stop criterion is met This includes Propose, Reconfig, Wait, Provision, Export, Enrich, Reveal, and Compare
Teardown and Cleanup: After every experiment, the teardown stage resets each host to its baseline configuration and clears accumulated cloud-side state The host reset removes all deployed policies (ASR rules, WDAC, firewall changes, etc.) so the lab returns to its pre-experiment state

Sample Benchmarking Use Case

The paper runs four paired experiments where each pair shares a single baseline pentest; both proposers then run their closedloop trials from that common starting point <ref:2606.

Improvements for AI systems

Improved evaluation framework for autonomous defense agents by decoupling experiment orchestration from policy-decision algorithms through a pluggable proposer interface to benchmark performance across different agentic architectures and models.
The improved AI system can perform closed-loop experiments with integrated commercial EDR, executing four sequential stages: Preflight, Setup, Loop, and Teardown to measure the effect of policy changes by running a commercial pentest against the environment.
The system can incorporate a deterministic gating mechanism that checks transition cost budget constraints and game value before deployment to ensure proposed policy sets fit within defined operational budgets and do not increase attacker gain.
The improved system can provide per-policy attribution to separate defense agent actions from autonomous EDR actions, allowing researchers to quantify the contribution of operator-deployed rules versus vendor-autonomous components in observed blocks.
The system can account for variable EDR behavior during evaluation windows by treating the vendor-autonomous layer's evolving behavior as a measurement axis, requiring experiments and ablation studies to vary dynamic environment knobs like AIR automation levels.

Abstract

Leading commercial endpoint detection and response (EDR) products have shifted from operator-configured rule sets to multi-component systems where autonomous AI components operate alongside, and increasingly in place of, operator-deployed policies. Autonomous defense agents using commercial EDR as their hardening tool are no longer tuning a passive tool, but a black-box autonomous system capable of making vendor-specific decisions. We present the first evaluation framework for autonomous defense agents hardening commercial EDR. We instantiate it in a Game of Active Directory (GOAD) lab with Horizon3's NodeZero as the autonomous pentester and Microsoft Defender XDR as the primary EDR under test. We extend the same framework to CrowdStrike Falcon and SentinelOne Singularity to test whether our Microsoft Defender XDR findings hold across vendors. We run a sample benchmark of defense agents with two large language model (LLM) backbones (Claude Sonnet 4.6 and Cisco Foundation-Sec-8B). We report three lessons learned that neither simulation nor open source EDR evaluation can surface: (i) commercial EDR telemetry is engineered for Security Operations Center (SOC) analyst workflows rather than for measuring policy effectiveness; (ii) per-policy attribution is important for separating the defense agent's actions from the EDR's autonomous actions; and (iii) EDR behavior varies between runs, largely because containment varies and accumulates. Together, these findings highlight a sim-to-real gap for enterprise defense and motivate evaluation methodology for benchmarking autonomous defense agents in environments with black-box, autonomous tools.