Label Unification for Cross-Dataset Generalization in Cybersecurity NER

summary

The gist

The gist The field of cybersecurity NER lacks standardized labels, making it challenging to combine datasets.

In short

The study tried to combine four cybersecurity datasets by unifying their labels into a single set. They evaluated this unified approach using BiLSTM models across different datasets to see if it improved generalization. The results showed that simple label unification introduced errors, and alternative architectures like multi-head and graph-based transfer models did not provide consistent performance gains over single-dataset baselines.

Key concepts

Label Unification: This process involves standardizing the different naming conventions for entities across four separate cybersecurity datasets into one consistent set of labels. The goal is to make the datasets compatible so that a model trained on one can potentially be tested on another, increasing data resource usability.
Cross-Dataset Evaluation: This method tests how well a model trained on one dataset (e.g., APTNER) performs when applied to an entirely different dataset (e.g., CYNER). The researchers used a 4x4 matrix of span-F1 scores to systematically compare performance across all possible pairings, assessing the model's ability to generalize.
Coarse-Grained Labeling: Instead of using highly specific labels, the researchers grouped many unique entity types into broader categories like 'Organization' or 'Malware'. This simplification was done to create a manageable and unified label set that could be applied consistently across all four original datasets.
Model Architectures (Multi-head/Graph-based Transfer): These are alternative ways to handle the limitations of simple label merging. A multi-head model allows the system to consider multiple aspects of the data simultaneously, while a graph-based transfer model uses relationships between data points to help move knowledge from one dataset to another.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Retrieval-Augmented Generation (RAG) — explained in 21 episodes
Sample Complexity — explained in 20 episodes
Flow Matching — explained in 19 episodes
Quantum Fisher Information (QFI) — explained in 19 episodes
Federated Learning — explained in 17 episodes

This episode discusses

The paper

Label Unification for Cross-Dataset Generalization in Cybersecurity NER · Read on arXiv

Johan Hausted Schmidt, Maciej Jalocha, William Michelseen

ITU

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Label Unification for Cross-Dataset Generalization in Cybersecurity NER".

Jane: Here's the gist of the paper. The gist The field of cybersecurity NER lacks standardized labels, making it challenging to combine datasets.

Tom: Label Unification and Cross-Dataset Evaluation The study investigates label unification across four cybersecurity datasets to increase data resource usability.

Jane: They performed a coarse-grained label unification and conducted pairwise cross-dataset evaluations using BiLSTM models.

Tom: The main contributions include a systematic analysis of cross-dataset performance for recent cybersecurity datasets enabled by a unification of labels.

Paper summary: Tom: So, to recap what we just heard, this paper by Hausted Schmidt and Jalocha looks at combining four cybersecurity datasets—APTNER, CYNER, DRNTI, and ATTACKER—by unifying their labels.

Jane: The core thesis is that while unifying the labels helps you combine the data resources for cybersecurity NER studies, it introduces errors because of annotation differences and label mismatches.

Tom: They performed a coarse-grained label unification and then conducted pairwise cross-dataset evaluations using BiLSTM models to see how well the unified data generalizes across different sets.

Lu: They used four distinct datasets: APTNER with two hundred sixty thousand one hundred thirty-four tokens and twenty-one entity types, CYNER with one hundred six thousand nine hundred ninety-one tokens and five entity types, DRNTI with one hundred seventy-five thousand two hundred twenty tokens and thirteen entity types, and ATTACKER with seventy-eight thousand nine hundred eighty-seven tokens and eighteen entity types <ref:2507.13870#pg2,CYNER with 106,991 tokens and 5>.

Meng: So you're training on one dataset and testing on all the others to check for cross-dataset generalization across that four by four matrix? That sounds like a lot of computational work.

Tom: It is, but they did it using a BiLSTM model setup, and they contrasted the diagonal entry, which is the direct training set performance, with one set of off-diagonal predictions drawn from an adjacent entry for each evaluation dataset.

Jane: And what did that cross-dataset evaluation show in terms of performance metrics like span F1 score?

Tom: Table one shows that APTNER and DNRTI seem to do significantly better when trained on one and evaluated on the other, which they attribute to similar scraped sources or annotation tools, as well as similarity in some authors and the same annotation tool used <ref:2507.13870#pg1>.

Lu: They also found that when training on CYNER or ATTACKER, the model performs quite poorly because those datasets contain a smaller amount of training data relative to DNRTI and APTNER.

Meng: So it’s not just about the labels; it’s about the underlying data volume too, which is something we have to consider when deploying these things at scale.

Jane: And what about the actual errors they found? Did unifying labels introduce specific types of mistakes that we need to watch out for?

Tom: Yes, there are definition discrepancies for tokens, like "sample" being mapped differently or label definition discrepancies, such as mapping "threat actor" to "organisation."

Lu: They also noted annotation discrepancies in span lengths across the datasets, which further complicates things when you try to combine them.

Conclusion: Tom: The paper, "Label Unification for Cross-Dataset Generalization in Cybersecurity NER," by Hausted Schmidt and Jalocha is focused on showing that simply unifying labels isn't a magic fix for combining cybersecurity datasets.

Jane: They conclude that while label unification enabled combined training sets, it unfortunately introduced errors due to annotation differences, label mismatches, and an increased bias toward predicting non-entities.

Tom: The authors tested alternative architectures like a multihead model with weight sharing and a graph-based transfer model to try and overcome these unification limitations.

Lu: The results show that the multihead model with shared embedding weights provides only marginal improvements over unified training, and the graph-based transfer model built on BERT-base-NER shows no significant performance gains compared to BERTbase-NER1.

Meng: So, for anyone looking to deploy a solution right now, they can't rely on just merging labels and hoping for better generalization across different security reports.

Jane: The implication is that future work should explore more targeted domain adaptation strategies instead of broad label unification.

Tom: That’s the main message: you need something more specific than just merging everything together to get reliable results in this domain.

More episodes

← Home