Label Unification for Cross-Dataset Generalization in Cybersecurity NER

arXiv:2507.13870 · cs.CL · Submitted 2025-07-18 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Label Unification for Cross-Dataset Generalization in Cybersecurity NER".

Jane: Here's the gist of the paper. The gist The field of cybersecurity NER lacks standardized labels, making it challenging to combine datasets.

Tom: Label Unification and Cross-Dataset Evaluation The study investigates label unification across four cybersecurity datasets to increase data resource usability.

Jane: They performed a coarse-grained label unification and conducted pairwise cross-dataset evaluations using BiLSTM models.

Tom: The main contributions include a systematic analysis of cross-dataset performance for recent cybersecurity datasets enabled by a unification of labels.

Paper summary: Tom: So, to recap what we just heard, this paper by Hausted Schmidt and Jalocha looks at combining four cybersecurity datasets—APTNER, CYNER, DRNTI, and ATTACKER—by unifying their labels.

Jane: The core thesis is that while unifying the labels helps you combine the data resources for cybersecurity NER studies, it introduces errors because of annotation differences and label mismatches.

Tom: They performed a coarse-grained label unification and then conducted pairwise cross-dataset evaluations using BiLSTM models to see how well the unified data generalizes across different sets.

Lu: They used four distinct datasets: APTNER with two hundred sixty thousand one hundred thirty-four tokens and twenty-one entity types, CYNER with one hundred six thousand nine hundred ninety-one tokens and five entity types, DRNTI with one hundred seventy-five thousand two hundred twenty tokens and thirteen entity types, and ATTACKER with seventy-eight thousand nine hundred eighty-seven tokens and eighteen entity types <ref:2507.13870#pg2,CYNER with 106,991 tokens and 5>.

Meng: So you're training on one dataset and testing on all the others to check for cross-dataset generalization across that four by four matrix? That sounds like a lot of computational work.

Tom: It is, but they did it using a BiLSTM model setup, and they contrasted the diagonal entry, which is the direct training set performance, with one set of off-diagonal predictions drawn from an adjacent entry for each evaluation dataset.

Jane: And what did that cross-dataset evaluation show in terms of performance metrics like span F1 score?

Tom: Table one shows that APTNER and DNRTI seem to do significantly better when trained on one and evaluated on the other, which they attribute to similar scraped sources or annotation tools, as well as similarity in some authors and the same annotation tool used <ref:2507.13870#pg1>.

Lu: They also found that when training on CYNER or ATTACKER, the model performs quite poorly because those datasets contain a smaller amount of training data relative to DNRTI and APTNER.

Meng: So it’s not just about the labels; it’s about the underlying data volume too, which is something we have to consider when deploying these things at scale.

Jane: And what about the actual errors they found? Did unifying labels introduce specific types of mistakes that we need to watch out for?

Tom: Yes, there are definition discrepancies for tokens, like "sample" being mapped differently or label definition discrepancies, such as mapping "threat actor" to "organisation."

Lu: They also noted annotation discrepancies in span lengths across the datasets, which further complicates things when you try to combine them.

Conclusion: Tom: The paper, "Label Unification for Cross-Dataset Generalization in Cybersecurity NER," by Hausted Schmidt and Jalocha is focused on showing that simply unifying labels isn't a magic fix for combining cybersecurity datasets.

Jane: They conclude that while label unification enabled combined training sets, it unfortunately introduced errors due to annotation differences, label mismatches, and an increased bias toward predicting non-entities.

Tom: The authors tested alternative architectures like a multihead model with weight sharing and a graph-based transfer model to try and overcome these unification limitations.

Lu: The results show that the multihead model with shared embedding weights provides only marginal improvements over unified training, and the graph-based transfer model built on BERT-base-NER shows no significant performance gains compared to BERTbase-NER1.

Meng: So, for anyone looking to deploy a solution right now, they can't rely on just merging labels and hoping for better generalization across different security reports.

Jane: The implication is that future work should explore more targeted domain adaptation strategies instead of broad label unification.

Tom: That’s the main message: you need something more specific than just merging everything together to get reliable results in this domain.

Johan Hausted Schmidt, Maciej Jalocha, William Michelseen

ITU

cs.CL

Submitted: 2025-07-18

Updated: 2026-10-20

Comments: 7 pages, 15 tables, 1 equation

Code: https://github.com/PLtier/NLP-Cyber-NER

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 69/100

The gist: The gist The field of cybersecurity NER lacks standardized labels, making it challenging to combine datasets.

Key concepts

Label Unification: This process involves standardizing the different naming conventions for entities across four separate cybersecurity datasets into one consistent set of labels. The goal is to make the datasets compatible so that a model trained on one can potentially be tested on another, increasing data resource usability.
Cross-Dataset Evaluation: This method tests how well a model trained on one dataset (e.g., APTNER) performs when applied to an entirely different dataset (e.g., CYNER). The researchers used a 4x4 matrix of span-F1 scores to systematically compare performance across all possible pairings, assessing the model's ability to generalize.
Coarse-Grained Labeling: Instead of using highly specific labels, the researchers grouped many unique entity types into broader categories like 'Organization' or 'Malware'. This simplification was done to create a manageable and unified label set that could be applied consistently across all four original datasets.
Model Architectures (Multi-head/Graph-based Transfer): These are alternative ways to handle the limitations of simple label merging. A multi-head model allows the system to consider multiple aspects of the data simultaneously, while a graph-based transfer model uses relationships between data points to help move knowledge from one dataset to another.

Terminology

Generalization — 1× in this paper · explained in 13 episodes

Summary

The gist The field of cybersecurity NER lacks standardized labels, making it challenging to combine datasets.

Label Unification and Cross-Dataset Evaluation

The study investigates label unification across four cybersecurity datasets to increase data resource usability. They performed a coarse-grained label unification and conducted pairwise cross-dataset evaluations using BiLSTM models. The main contributions include a systematic analysis of cross-dataset performance for recent cybersecurity datasets enabled by a unification of labels. This analysis helps understand the differences between the currently available datasets and shows the viability of such a label unification in practice.

Methodology and Data Preparation

The project uses four different NER cyber-security datasets: APTNER, CYNER, DRNTI, and ATTACKER. To make the datasets compatible with NER models, they were converted to the standard CoNLL 2003 format using BIO2 labelling. A label unification was performed by standardizing label sets based on those presented in the CYNER paper, choosing coarse-grained labels like Organization, System, Vulnerability, and Malware. Labels that could not be mapped were discarded and relabelled as O (non-entity). Duplicate sentences across train, development, and test sets were removed for each train-test pairing to avoid data leakage.

Model Architectures and Evaluation Setup

A BiLSTM model setup was used for the evaluation. Cross-dataset evaluations were performed by training the model on each training dataset and testing it on all other development datasets to assess cross-dataset generalization. This results in a 4x4 matrix where each entry is the span-F1 score. The analysis contrasted the diagonal entry with one set of cross-dataset predictions drawn from a rowadjacent, off-diagonal entry for each evaluation dataset. Jensen-Shannon divergence (JS-div) was used to measure similarity between training sets.

Analysis of Performance and Limitations

Table 1 shows that no row-adjacent entry outperforms the relevant diagonal entry in span-F1 score, recall, or precision. APTNER and DNRTI seem to do significantly better when trained on one and evaluated on the other, likely due to similar scraped sources or annotation tools. Errors related to label unification include definition discrepancies for tokens (e.g., sample being mapped differently), definition discrepancies for labels (e.g., threat actor mapped to organisation), and annotation discrepancies in span lengths. The analysis also found that reference models using original label sets had a 5-15% lower ratio of FNs on relevant tokens compared to models using unified labels.

Proposed Model Adjustments

To address unification limitations, alternative architectures were proposed. A multihead model with weight sharing provided only marginal improvements over unified training. The graph-based transfer model built on BERT-base-NER showed no significant performance gains compared to BERTbase-NER1. The analysis also noted that the noise introduced by the combined dataset hindered performance on CYNER and APTNER when using a model trained on all four unified datasets. The multihead model with shared embedding weights showed improvements in span-F1 for two datasets (DNRTI & ATTACKER) compared to the reference model. Overall, the results suggest that simple label merging is not sufficient for robust generalization.

Conclusion

The investigation concludes that while unification enabled combined training datasets, it introduced errors due to annotation differences, label mismatches, and increased bias toward predicting nonentities. Multi-head models showed slight gains by preserving original label sets, while LST-NER offered no clear advantage over BERT-base-NER. Future work should explore more targeted domain adaptation strategies.

References

Md Tanvirul Alam, Dipkamal Bhusal, Youngja Park, and Nidhi Rastogi. 2022. Cyner: A python library for cybersecurity named entity recognition. arXiv preprint arXiv:2204.05754.

Pritam Deka, Sampath Rajapaksha, Ruby Rani, Amirah Almutairi, and Erisa Karafili. 2024. Attacker: towards enhancing cyber-attack attribution with a named entity recognition dataset. In International Conference on Web Information Systems Engineering.

Pavlos Evangelatos, Christos Iliou, Thanassis Mavropoulos, Konstantinos Apostolou, Theodora Tsikrika, Stefanos Vrochidis, and Ioannis Kompatsiaris. 2021. Named entity recognition in cyber threat intelligence using transformer-based models. In 2021 IEEE International Conference on Cyber Security and Resilience (CSR).

IEEE.

Chen Gao, Xuan Zhang, Mengting Han, and Hui Liu. 2021a. A review on cyber security named entity recognition. Frontiers of Information Technology & Electronic Engineering.

Chen Gao, Xuan Zhang, and Hui Liu. 2021b. Data and knowledge-driven named entity recognition for cyber security. Cybersecurity.

Casey Hanks, Michael Maiden, Priyanka Ranade, Tim Finin, Anupam Joshi, and 1 others. 2022. Recognizing and extracting cybersecurity entities from text. In Workshop on Machine Learning for Cybersecurity, International Conference on Machine Learning.

Abhinav Ramesh Kashyap, Devamanyu Hazarika, MinYen Kan, and Roger Zimmermann. 2020. Domain divergences: A survey and empirical analysis. arXiv preprint arXiv:2010.12198.

Peipei Liu, Hong Li, Zuoguang Wang, Jie Liu, Yimo Ren, and Hongsong Zhu. 2022. Multi-features based semantic augmentation networks for named entity recognition in threat intelligence. In 2022 26th International Conference on Pattern Recognition (ICPR).

Jinghui Lu, Maeve Henchion, and Brian Mac Namee. 2021. Diverging divergences: Examining variants of jensen shannon divergence for corpus comparison tasks.

Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. arXiv preprint arXiv:1603.01354.

Md Abu Marjan and Toshiyuki Amagasa. 2024. Domain-adaptive entity recognition: unveiling the potential of cser in cybersecurity and beyond. International Journal of Machine Learning and Cybernetics, pages 1–19.

Inoussa Mouiche and Sherif Saad. 2024. Ti-nermerger: Semi-automated framework for integrating ner datasets in cybersecurity. International Conference on Security and Cryptography.

Stefano Silvestri, Giuseppe Felice Russo, Giuseppe Tricomi, and Mario Ciampi. 2022. A dataset for the fine-tuning of llm for the ner task in the cyber security domain.

Smita Srivastava, Biswajit Paul, and Deepa Gupta. 2023. Study of word embeddings for enhanced cyber security named entity recognition. Procedia Computer Science, 218:449–460.

Xuren Wang, Songheng He, Zihan Xiong, Xinxin Wei, Zhengwei Jiang, Sihan Chen, and Jun Jiang. 2022. Aptner: A specific dataset for ner missions in cyber threat intelligence field. In 2022 IEEE 25th international conference on computer supported cooperative work in design (CSCWD), pages 1233–1238.

Xuren Wang, Xinpei Liu, Shengqin Ao, Ning Li, Zhengwei Jiang, Zongyi Xu, Zihan Xiong, Mengbo Xiong, and Xiaoqing Zhang. 2020. Dnrti: A large-scale dataset for named entity recognition in threat intelligence. In 2020 IEEE 19th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), pages 1842–1848.

Junhao Zheng, Haibin Chen, and Qianli Ma. 2024. Cross-domain named entity recognition via graph matching. arXiv preprint arXiv:2408.00981.

Improvements for AI systems

Based on Definition discrepancies for tokens, develop a context-aware token disambiguation module that uses distributional shifts in word and POS tags (as noted in Table 6) to predict if a token like sample should be mapped to "O" or a specific entity type based on the training dataset's vocabulary.
Implement a graph-based transfer mechanism using BERT-base-NER, as suggested by the Graph Matching approach, which allows for the transfer of knowledge from the resourceful dataset to the more sparse dataset without requiring label correspondence. This system can enable cross-dataset entity recognition even when label sets do not match.
Deploy a multihead model architecture where each head predicts labels from a specific dataset's set while sharing weights in the LSTM cell and embedding matrix, aiming for marginal improvements over unified training by leveraging shared context representations.
Develop an automated label refinement tool that incorporates expert feedback to address label definition discrepancies, potentially mitigating issues like the mapping of threat actor to the broader category organisation when dealing with niche labels.