STRAL: What Adaptation Gains, Loses, and Costs in SOC Alert Screening

arXiv:2605.22324 · cs.CR · Submitted 2026-05-21 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "STRAL: What Adaptation Gains, Loses, and Costs in SOC Alert Screening".

Elias: The gist:

Nadia: First, who's behind it and why it matters.

Title and authors: Nadia: So we're looking at this paper called "STRAL: What Adaptation Gains, Loses, and Costs in SOC Alert Screening." It’s about making active learning smarter for security operations centers by dealing with those low-prevalence alert streams where you get a ton of false alerts even if your accuracy is decent.

Elias: Yeah, it focuses on how adaptation affects the actual cost to the analyst rather than just some aggregate score. It seems they're building something that wraps an existing frozen classifier with smarter ways to decide when and what to query.

Priya: I’m curious about what "cost" actually means here in terms of real work for someone sitting at a desk, not just some abstract metric they calculate. Does it translate into fewer clicks or less time spent?

Nadia: Exactly, Priya. They are trying to control the false positive burden directly using bounded analyst feedback and some specific query selection rules to keep the workload manageable.

Elias: The core idea seems to be combining three things: a frozen screener, an ADWIN trigger that watches for score shifts in the predicted probabilities, and a hybrid rule for choosing what alerts to ask an analyst about.

Priya: That combination sounds interesting because it acknowledges that just having a model isn't enough; you need a mechanism to know when the data is starting to change or when the model needs tuning.

Nadia: Right. They use this ADWIN trigger over the predicted probability stream, which isn't waiting for ground truth errors, but looking at how the scores themselves are behaving over time.

Elias: That unsupervised score-shift triggering allows them to adapt when alert distributions evolve because attacker tactics shift, even without a fancy tool that knows exactly what concept drift is happening.

Priya: But then there’s the query selection part, and they use a hybrid acquisition rule that splits the query batch between samples near the current operating threshold and high-score samples. What does that actually do for the analyst?

Nadia: It means they aren't just asking about what’s definitely wrong or what looks highly suspicious; they are balancing exploring uncertainty right around their current decision boundary with picking some high-scoring alerts to investigate.

Title and authors: Elias: That hybrid rule is key because the paper shows that this acquisition strategy contributes more than just the timing of when you query, which is a point they call a matched-trigger ablation.

Priya: So, if we look at their results on benchmarks like AIT-ADS and BOTSv1, what’s the main tangible benefit they report for someone managing these alert streams?

Nadia: On AIT-ADS, PACT hits the lowest benign-normalized false positive burden among all the adaptive methods they tested. On BOTSv1, it cuts that burden by twenty-one percent relative to a frozen baseline <ref:2605.22324#pg1,relative to a frozen baseline>.

Elias: That twenty-one percent reduction on BOTSv1 is pretty significant when you're dealing with those low-prevalence streams where the false alert volume explodes because of the base rate <ref:2605.22324#pg1>.

Priya: But there are trade-offs, right? The paper mentions that achieving that lowest FP burden comes at a cost to positive-window recall. How big is that compromise?

Nadia: They say it costs roughly ten percentage points of positive-window recall when they achieve that lowest FP burden on both datasets. That means you might miss more actual attacks for every reduction in false alerts.

Elias: It seems they are mapping out a Pareto frontier here, showing that you can’t optimize everything at once; you have to choose your operating point based on what matters most for your specific SOC scenario.

Priya: I see the point about the trade-off, but what about how this impacts real-world deployment? Does this mean we should just blindly deploy PACT everywhere?

Nadia: The paper is pretty clear that it’s not a universal solution. It's most attractive when you have constrained labeling capacity and a high volume of false alerts where FP/1M benign is the main problem <ref:2605.22324#pg1>.

Elias: They also point out that the threshold-only baseline can cut the FP burden by ninety-four percent on AIT-ADS, but it drastically raises missed positives from forty-one to three hundred nineteen which shows you what happens when you only optimize for one metric.

Priya: It’s important they flag that this entire evaluation is done in an offline simulator, so we don't see the latency or throughput issues when running this on live production telemetry. That’s a limitation they state plainly.

Title and authors: Nadia: Right, and another thing they mention is that the ADWIN trigger here is over the predicted probability stream, not some validated concept-drift oracle, which means it relies on heuristics rather than guaranteed drift detection.

Elias: They also acknowledge that simply reporting aggregate metrics doesn't tell the whole story because you need to know what labels are actually used to update a deployed screener. That gap in reporting is something they highlight.

Priya: So, if someone is looking at their alert management system right now, what does this paper suggest about how they should think about balancing accuracy against analyst fatigue?

Nadia: It suggests thinking of it as choosing an operating point on a frontier where you weigh the burden of false alerts against the risk of missing an actual attack window.

Elias: And that choosing the right adaptive ingredients, like combining that ADWIN trigger with this hybrid query rule, gives you a better operational choice than just relying on one simple threshold setting.

Priya: For me, it means we need to look at the data not just for its accuracy scores, but for the actual composition of the labels being requested to see if that aligns with what an analyst would actually find useful.

Nadia: So, to wrap up on this "STRAL: What Adaptation Gains, Loses, and Costs in SOC Alert Screening," it’s a framework for managing alert fatigue in those tricky low-prevalence environments by smartly combining drift detection with smarter query selection.

Elias: It shows that the way you choose to adapt—using ADWIN over the score stream and splitting your query budget—is what drives the actual gains and losses in terms of analyst workload.

Priya: Ultimately, it’s a sophisticated way to handle alert screening where you have limited resources, but we still need to be mindful that these results come from a simulated environment under specific conditions.

Nadia: That’s what they found with PACT—a way to make adaptation more cost-aware for the analyst. That’s all we have time for this one paper.

The paper's summary: Nadia: So, we're looking at this paper called "STRAL: What Adaptation Gains, Loses, and Costs in SOC Alert Screening." It basically frames adaptive learning not just on how accurate a model is, but on what the actual cost is to an analyst working through alerts.

Elias: Right. The core idea here is that when you're dealing with those low-prevalence alert streams—where most of your data is normal traffic and only a tiny fraction are real threats—just chasing the highest score doesn't work well because you get way too many false alarms.

Priya: So what they’re proposing is a controller that manages this workload, trying to balance catching real attacks with not drowning the analyst in noise, using some kind of feedback loop.

Nadia: Exactly. They wrap an existing frozen classifier with an ADWIN trigger over the predicted probabilities and a hybrid query rule. That means they’re looking for when the alert distribution itself starts shifting because of attacker tactics, not just when a single ground truth label shows something is wrong.

Elias: The authors are focusing on that score shift detection, using the condition where the mean of one window minus the mean of another window is greater than some small epsilon value to flag a change point in the probability stream.

Priya: And they’re testing this on these public benchmarks, AIT-ADS and BOTSv1, looking at things like how many false positives you get per million benign events. They’re trying to find the settings that minimize that FP burden while still keeping recall from missing actual attacks acceptable.

Nadia: The results show they found a way to get the lowest benign-normalized false positive burden compared to every other adaptive strategy they tested on those datasets. On BOTSv1, for example, they saw a reduction in false positives from frozen models down to around twenty-one thousand per million events.

Elias: That reduction is pretty solid when you think about the scale of the alert streams they're dealing with—that’s twenty-one percent lower noise. But they also show that this lowest FP point isn't perfect across the board.

Priya: They explicitly mention trade-offs, showing that getting that low false positive burden means sacrificing some of the recall you get for catching actual threats, like missing about ten percentage points on average for both datasets.

Nadia: It highlights a real operational dilemma in security—you can't just optimize one metric in isolation; you have to choose an operating point on a frontier based on what’s most important right now.

Elias: The paper also shows that the hybrid acquisition rule—splitting the query budget between samples near the threshold and high-score samples—is better than just using one selection method alone, because it gives you both uncertainty exploration and some positive evidence.

Priya: But I still see that this whole thing is tested in an offline simulator, so we have to remember that this doesn't tell us how fast the system would run on live production telemetry or what the latency would be when it’s actually under attack.

Nadia: True. And they also point out that the ADWIN trigger here is based on predicted probabilities, not some validated drift detector based on actual labeled errors, which means it’s a heuristic way to guess when things are changing.

Elias: So for someone just listening in, the main point is that adaptation needs to be cost-aware. It’s about finding the balance between keeping analyst fatigue low and making sure you aren't blind to a real attack because you were too focused on reducing false alerts.

Priya: It changes how we think about active learning—it’s not just about getting the most accurate model; it’s about designing a system that understands the cost of being wrong in terms of analyst time.

The paper's improvements: Nadia: So, we're looking at what the authors suggest they should improve about PACT, which is their main contribution to this work on SOC alert screening adaptation. They’re not just showing one good setup; they're suggesting how to make it even more robust under different conditions.

Elias: Right. They point out that the current ADWIN trigger, which monitors the score stream, could be replaced by something that reacts directly to labeled error signals instead of relying on just the predicted probability stream drift.

Priya: So they’re suggesting a switch from an unsupervised score shift to a more supervised drift detector based on actual ground truth errors—that means if we get a few confirmed false positives or true positives, that information should immediately influence the model's adaptation strategy.

Nadia: Exactly. They argue that using labeled error signals gives you more reliable evidence of concept drift because it’s grounded in what the analyst actually labels as wrong, not just a mathematical fluctuation in the prediction scores.

Elias: That makes sense from a cryptographer's view; it grounds your parameter selection in verifiable data points rather than just statistical noise over time. The authors suggest this would give you more reliable updates to that frozen core.

Priya: From a measurement standpoint, if the trigger is based on error signals, we get better control over when and how often the model needs to be re-engaged with new labels, which helps manage those analyst labeling costs much more precisely.

Nadia: And they also look at improving that warm-start booster update. Instead of just adding a few trees, they suggest using a more controlled process—a bounded number of new trees and an event-based cooldown period—to stop the model from getting destabilized by too many single-sample updates.

Elias: That’s practical engineering advice, I like it. It addresses the issue where updating the model too quickly can introduce instability, which is a big problem when you are working with streaming data and limited retraining budgets.

Priya: So, in short, they want to move towards a system where adaptation isn't just reactive to score changes, but actively guided by confirmed analyst feedback and controlled model updates.

Nadia: It’s about making the adaptive layer more intelligent by giving it better inputs on what actually matters—the labeled data—and controlling the learning process itself.

Elias: This shifts the focus from just building a detection mechanism to building a whole adaptive framework that manages risk and computational load simultaneously.

Priya: If this direction is taken, I think we could start looking at how we can design feedback mechanisms for active learning systems that are inherently aware of analyst workload constraints from the very beginning.

Conclusion: Tom: So we're wrapping up this talk on "STRAL: What Adaptation Gains, Loses, and Costs in SOC Alert Screening." It boils down to how you choose your adaptive strategy for alert screening based on whether you prioritize minimizing false alerts or maximizing recall under real analyst constraints.

Nadia: Basically, the paper shows that there isn't one single best setting; it’s a choice of operating point on a trade-off curve where you decide what cost—false positives versus missed attacks—you’re willing to pay.

Elias: And they showed that combining an ADWIN trigger with a hybrid query rule is the most effective way to navigate that frontier without just picking one simple metric to optimize.

Priya: I think what this means for us is that when we design these screening systems, we can’t just chase the highest F1 score; we have to quantify the analyst's actual time investment in terms of those false positives.

Nadia: Exactly. The numbers on BOTSv1 showing that twenty-one percent reduction in benign false positives is concrete evidence that this approach is actually useful when you’re dealing with those low-prevalence streams they study.

Elias: It proves that the mechanism matters more than the raw score of the model itself, especially when dealing with concept drift in security data where attack patterns evolve quickly.

Priya: I just want to emphasize that we have to keep remembering these are simulations, because translating those offline results into a live production environment involves a whole lot more uncertainty regarding latency and real-time throughput.

Nadia: True. And the authors did flag that the ADWIN trigger uses predicted probabilities, which is a heuristic guess about when things are changing, rather than being based on some perfect ground truth signal.

Elias: It’s a practical limitation; it means the system relies on its internal model of probability stability to know when to shift gears.

Priya: So, moving forward, I think the next big step is figuring out how we can build those feedback loops so they are truly representative of what an analyst experiences in a high-pressure environment.

Nadia: Right. We need to focus on designing systems that are cost-aware from the start, not just accuracy-driven later on.

Elias: It’s a useful framework for anyone working with streaming models and uncertainty, like the one presented in this paper, "STRAL: What Adaptation Gains, Loses, and Costs in SOC Alert Screening."

Priya: I think understanding these trade-offs between FP burden and recall is something we need to integrate into our core measurement strategies for any AI system deployed in a critical setting.

National Institute of Information and Communications Technology · Kobe University

cs.CR

Submitted: 2026-05-21

Updated: 2026-10-18

Comments: 11 pages, 4 figures, 8 tables. v2: PACT renamed STRAL to avoid a naming collision. Substantially revised model selection, threshold calibration, and empirical evaluation, including AMiner, Wazuh/AIT-ADS, NAD-CD, Guard, detection turnover, acquisition comparisons, classifier transfer, and sensitivity analyses. Earlier BOTSv1 and focal-loss results remain in v1

Code: https://github.com/splunk/botsv1

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The gist: PACT introduces a Paretoaware controller for triggered active learning that wraps an already-deployed frozen XGBoost-Focal screener with an adaptive windowing score-shift trigger and a

Key concepts

Frozen XGBoost Screener: This is a pre-trained machine learning model (XGBoost) that has been trained offline and kept fixed during real-time operation. It acts as the initial filter for incoming security alerts, classifying them into potential threats based on established patterns learned from historical data. It provides a stable, fast baseline for screening.
ADWIN Score-Shift Trigger: This is an unsupervised statistical method used to monitor the stream of predicted probabilities from the model. It looks for significant changes in the mean probability distribution over time, signaling that the underlying alert patterns or concept have shifted. When a shift is detected, it triggers a change in how alerts are screened.
Hybrid Acquisition Rule: This strategy decides which new alerts to query for analyst labeling. It splits the query budget: half of the queries target samples near the current decision threshold (uncertainty), while the other half targets samples with very high predicted positive scores. This combination ensures both boundary exploration and exploitation of highly suspicious events.
Warm-Start Booster Update: Instead of retraining the entire model, this technique updates it incrementally by adding a limited number of new trees to the existing ensemble. This process is stabilized by requiring a minimum batch size (32 labels) and an event-based cooldown period, ensuring that model improvements are gradual and do not destabilize the screening performance.

Terminology

Ablation — 1× in this paper · explained in 1 episode
Active Learning — 1× in this paper · explained in 6 episodes
Adaptation — 1× in this paper · explained in 1 episode
Benchmark — 1× in this paper · explained in 2 episodes
F1 Score — 1× in this paper · explained in 4 episodes
XGBoost — 1× in this paper · explained in 2 episodes

Summary

The gist: PACT introduces a Paretoaware controller for triggered active learning that wraps an already-deployed frozen XGBoost-Focal screener with an adaptive windowing score-shift trigger and a hybrid acquisition rule to reduce false positive burden in low-prevalence SOC alert streams.

Problem Setting and Motivation

Security operations centers face persistent alert fatigue where even low false-positive rates generate substantial investigation load, while aggregate F1 scores obscure analyst burden<ref:2605.22324#pg2>. IDS-derived alert streams are low-prevalence, meaning that even a 0.1% false-positive rate (FPR) over a one-million-event ingestion day produces a thousand false alerts, enough to materially affect analyst workload<ref:2605.22324#pg2>. The paper addresses the asymmetry where prevalence and drift motivate adaptive alert screening that uses bounded analyst feedback to control FP burden<ref:2605.22324#pg2>.

The PACT Architecture

PACT is a streaming alert-screening controller that operates in four stages: frozen-core scoring of the incoming alert stream, ADWINbased score-shift triggering, hybrid query selection with bounded analyst feedback, and warm-start booster update via hot-swap<ref:2605.22324#pg2>. The Pareto-aware label describes the evaluation, not an internal optimizer<ref:2605.22324#pg2>. The architecture involves a frozen, imbalance-aware classification core trained offline, which is then wrapped by a streaming loop that includes an ADWIN trigger and a hybrid query rule<ref:2605.22324#pg2>.

Key Components of the Controller

The PACT controller combines three existing ingredients in a downstream SOC screening setting: a controller around an already-deployed frozen screener, an unsupervised score-shift trigger over the predicted-probability stream, and an analyst-efficient hybrid query rule<ref:2605.22324#pg2>. The score-shift triggering uses ADWIN to flag candidate change points during stream operation by monitoring the predicted probability stream<ref:2605.22324#pg4>. The hybrid acquisition rule splits the query batch, with half going to samples closest to the operating threshold ranked by their threshold-relative score distance and the other half going to high-score acquisition<ref:2605.22324#pg5>.

Experimental Evaluation and Results

The paper evaluates PACT on two public low-prevalence benchmarks, AIT-ADS and BOTSv1, focusing on operational metrics like benign-normalized false-positive (FP) burden, positive-window recall, missed positives, and realized query rate<ref:2605.22324#pg2>. PACT attains the lowest benign-normalized FP burden among the adaptive strategies on both datasets<ref:2605.22324#pg2>. On BOTSv1, PACT reduces FP/1M benign from 27,023 (frozen) to 21,271, a roughly 21% reduction<ref:2605.22324#pg9>.

Trade-offs and Operating Points

PACT is not the absolute lowest FP operating point on these streams; it is a less destructive operating point on the FP, recall, and query Pareto frontier than frozen, periodic, ADWIN-random, or threshold-only FP minimization<ref:2605.22324#pg9>. On AIT-ADS, PACT attains 72.24% average positive-window recall versus 82.37% for periodic and 82.40% for ADWIN-random. The matched-trigger ablation shows that acquisition contributes beyond timing alone, with the hybrid rule being the lowest FP policy on both datasets.

Operational Implications

The results suggest that no single-score selection captures the full operating-point picture: FP/1M benign captures investigation load, positive-window recall captures attack-window safety, and realized query rate captures analyst labeling cost. PACT is the most attractive of the tested strategies when labeling capacity and false-alert volume are constrained, but it is not universally preferable. The threshold-only baseline cuts FP/1M benign by 94% on AIT-ADS but raises missed positives from 41 to 319 and lowers positive-window recall from 82.37% to 69.72%.

Limitations

The headline claims apply to AIT-ADS/BOTSv1-like low-prevalence alert streams, with stream prevalences under 1.5% and attack patterns concentrated mid-stream, rather than to SOC streams in general. ADWIN here is a score-shift trigger over the predicted-probability stream, not a labeled drift detector. The three-seed audit is a limited sweep, not evidence of broad statistical robustness. The streaming evaluation runs in an offline simulator, so per-event scoring latency and warm-start update throughput on production telemetry are not characterized here.

Conclusion

PACT recasts adaptive alert screening on low-prevalence streams as an operating-point choice. Two ingredients drive the result on the streams we study: applying ADWIN to the score stream rather than to labeled error, and splitting the query budget between threshold-relative uncertainty and high-score sampling. A matched-trigger ablation controls trigger timing and shows that the acquisition rule contributes beyond timing alone. The threshold-only baseline makes the limits of single-metric optimization explicit on these streams: it reaches a lower FP burden than every adaptive setting tested but collapses BOTSv1 recall by 55 percentage points. Among the strategies tested, an unsupervised score-shift trigger paired with positive-blind querying produced the least attractive operating point.

Data Availability

BOTSv1 and AIT-ADS are public benchmark datasets available from their original providers. The artifact package will include split-generation scripts, the preprocessing and leakage-mitigation pipeline, the retained-feature manifest in machine-readable form, all run configuration files, raw perseed metric traces, and the figure and table generation scripts used to produce the reported results.

Ethical Considerations

This study uses previously released benchmark datasets and an offline streaming simulator. It does not involve human subjects, live attacks, or intervention on production systems. Reported trade-offs concern alert-screening behavior under simulated deployment and are not used to automate real incident-response decisions. The per-dataset feature manifests are released so that operators can verify, before any deployment, that retained features are observable predecision in their environment.

References

[1] S. Bhatt, P. K. Manadhata, and L. Zomlot, “The operational role of security information and event management systems,” IEEE Security & Privacy, vol. 12, no. 5, pp. 35–41, 2014<ref:2605.22324#pg14>.

[7] H.-J. Liao, C.-H. R. Lin, Y.-C. Lin, and K.-Y. Tung, “Intrusion detection system: A comprehensive review,” Journal of Network and Computer Applications, vol. 36, no. 1, pp. 16–24, 2013<ref:2605.22324#pg8>.

[9] T. Ban, T. Takahashi, S. Ndichu, and D. Inoue, “Breaking alert fatigue: AI-assisted SIEM framework for effective incident response,” Applied Sciences, vol. 13, no. 11, p. 6610, 2023<ref:2605.22324#pg10>.

[14] G. Andresini, F. Pendlebury, F. Pierazzi, C. Loglisci, A. Appice, and L. Cavallaro, “INSOMNIA: Towards concept-drift robustness in network intrusion detection,” in Proceedings of the 14th ACM Workshop on Artificial Intelligence and Security (AISec). ACM, 2021, pp.

Improvements for AI systems

Bold Header: PACT Controller for Triggered Active Learning Implementation

PACT is a Paretoaware controller for triggered active learning that wraps an already-deployed frozen XGBoost-Focal screener with an adaptive windowing score-shift trigger and a hybrid acquisition rule. This allows the system to adapt to evolving alert distributions by selecting queries based on threshold-relative uncertainty with high-score sampling, aiming for the lowest benign-normalized false-positive (FP) burden among the adaptive methods tested.

Bold Header: Automated Alert Screening with Bounded Query Budget

The improved system can perform downstream alert screening by maintaining a bounded analyst workload. PACT's operation yields roughly one analyst review per 9,000 events on average on a large stream (BOTSv1), demonstrating how it can achieve a lower FP burden than frozen, periodic, ADWIN-random, or threshold-only FP minimization.

Bold Header: Adaptive Query Strategy Based on Uncertainty and Score

The system utilizes a hybrid acquisition rule that splits the query budget between samples closest to the operating threshold (threshold-relative uncertainty) and highest predicted positive-class probabilities (high-score sampling). This hybrid rule is shown to be superior, as it avoids the failure mode of pure uncertainty sampling by supplying positive evidence, while preserving boundary exploration.

Bold Header: Dynamic Concept Drift Detection via Score Shift

The system employs an unsupervised ADWIN trigger over the predicted probability stream rather than delayed ground-truth error, using the condition where "mean(W1) − mean(W2) > ϵ to flag a heuristic evidence of scoredistribution shift. This enables adaptation when alert distributions also evolve as attacker tactics change," even without a validated concept-drift oracle.

Bold Header: Robust Model Updates via Warm-Start Booster

Instead of retraining from scratch, the system uses warm-start booster update by appending a bounded number of new trees to the existing ensemble. This update mechanism is stabilized by two safeguards: waiting for a minimum update batch of Bmin = 32 labels and an event-based cooldown period, preventing destabilizing single-sample updates.