Rethinking Generalization in Knowledge Graph Embedding: An Empirical Study and Sparse Peak Regularization

arXiv:2505.23442 · cs.LG, cs.CL · Submitted 2025-05-29 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Rethinking Generalization in Knowledge Graph Embedding".

Jane: Here's the gist of the paper. The gist The proposed Sparse Peak Regularization (SPR) method rethinks regularization for Knowledge Graph Completion (KGC) by selectively penalizing important components in embeddings to alleviate overfitting and enable models to surpass their original performance upper bounds Necessity of Regularization Extensive empirical studies on various KGC models reveal that many existing KGC models have overfitting problems, evidenced by a large discrepancy between their high training performance and low validation performance Moreover, without regularization, the MR index gradually increases, which further illustrates worsening stability in model predictions and poor generalization Adding regularization significantly improves the generalization ability of the model on different data sets and breaking the upper bound of the model’s performance Proposed SPR Regularization Method The paper introduces a novel sparse-regularization method called SPR, which uses the idea of sparsity to selectively discard some unimportant items and selectively retain the operation items to achieve the purpose of regularization The core idea is to “sparse” the elements that are input to the regularization term so that only important components (i.e., components with large squared magnitudes) are penalized, while many almost negligible components (which may just be noise) are discarded The sparsification is achieved by sorting the (non-negative) squared terms and “masking” those whose cumulative sum is lower than δ and “discarding” them (i.e., zeroing them) The SPR regularization term of the complete triple is defined as SPRVh i, R r j, V t k = V h i 2 sparse + V t k 2 sparse + V h i ⊙ R r j sparse + V t k ⊙ R r j sparse Empirical Performance and Comparison Comparative experiments on multiple datasets and multiple models show that the SPR regularization method is better than other regularization methods and can enable the KGC model to further break through the performance margin For example, in the ComplEx model in the FB15K-237 dataset, the MRR of the basic ComplEx is 0.35, while ComplEx-SPR increases it to 0.371, achieving a significant improvement In all datasets and compared to regularized baseline methods such as F2, N3, DURA, and ER, SPR always maintains the best or near-best performance on most models and indicators Theoretical Advantages of SPR The theoretical analysis shows that for linear or bilinear KGC scorers, the empirical Rademacher complexity is bounded by a constant multiple of the expected l2-norm of the effective embeddings Furthermore, SPR acts as an adaptive sparsity-inducing regularizer by masking smaller components, which reduces the effective dimensionality of the embedding space This selective retention of dominant components mirrors feature selection strategies and enhances the signal-to-noise ratio by eliminating noise Impact on Model Optimization The optimization dynamics show that SPR’s gradient is zero for masked components, effectively freezing those dimensions during updates, which reduces the parameter space Additionally, SPR’s deterministic nature yields zero conditional variance in the gradient, unlike Dropout’s stochastic noise This stability enhances gradient update reliability and potentially accelerates convergence Generalization Across Model Types The SPR method is versatile enough to enhance performance consistently across various types of KGC models, including GNN-based, translation-based, tensor decomposition-based KGC models, and Temporal Knowledge Graph Completion (TKGC) models The experimental results for the CompGCN variants show that introducing dropout alone leads to substantial improvements across the board, and SPR goes a step further by penalizing specific entity–relation interactions The experimental results clearly show that regularization is also very important in the task of TKGC In conclusion, SPR’s deterministic and selective regularization offers superior control over model complexity, gradient stability, and optimization efficiency This method mitigates overfitting by selectively penalizing the most important components in the entity and relation embeddings while discarding less influential and potentially noisy elements This selective approach leads to better generalization by focusing on informative features while discarding noise This method is simple and universal as possible in the KGC model and achieves better results The paper hopes to be as universal as possible in the KGC model and as simple and universal as possible How Datasets Composition Influences Regularization Performance The overall trend of training set size and regularization improvement reveals that when observing the data of YAGO3-10 (the largest training set), no matter which model (CP, ComplEx, GIE, RESCAL), the percentage of Hits@1 improvement is generally the lowest Conversely, the data of WN18RR (the smallest training set) generally has the highest or close to the highest percentage of Hits@1 improvement This suggests that large-scale datasets with more training sets themselves

Paper summary: Tom: Okay, so to recap where we are—we’ve talked about how existing KGC models often overfit and that SPR is a new method they propose. The paper, "Rethinking Generalization in Knowledge Graph Embedding: An Empirical Study and Sparse Peak Regularization," claims that careful design of regularization helps models break through their original performance upper bounds.

Jane: What’s the core message they are trying to get across about why this matters for the field? It seems like they found that overfitting is common, even across translation-based, tensor decomposition-based, and GNN-based KGE models.

Lu: The main claim is that appropriate regularization doesn't just reduce variance; it enables these models to surpass their original performance limits. They show this through extensive empirical studies on multiple KG datasets.

Meng: So, what’s the practical reason for that? If a model can perform better than its theoretical limit, what does that mean for us when we actually try to use it in production?

Tom: It means we get models that are more robust and generalize better to new data, which is exactly what you want when you deploy something. They’re not just making the training loss go down; they're improving how the model performs on unseen validation data.

Jane: So, it’s about getting models that are not just technically accurate on the training set, but actually useful and stable when applied to real-world knowledge graphs.

Lu: And they introduce Sparse Peak Regularization as a novel way to achieve this by selectively penalizing dominant embedding components based on their squared magnitudes.

Tom: They claim SPR is model-agnostic because it works across various architectures, which is a big deal since we have so many different KGE setups.

Jane: It’s interesting that they found that when you compare the performance of a basic ComplEx model to one trained with SPR on the FB15K-two hundred thirty-seven dataset, the MRR goes from zero point three five up to zero point three seven one. That jump is what they highlight as significant improvement.

Meng: A small gain like that is good, but if this method helps us consistently break those upper bounds across all these model types you mentioned, that’s something we can work with for scalable systems.

Lu: The theoretical backing they provide shows that SPR acts like an adaptive sparsity-inducing regularizer by reducing the effective dimensionality of the embedding space.

Tom: So, in short, the paper is arguing that selective regularization is a powerful tool because it focuses on informative features while discarding noise. That’s what matters for generalization here.

Jane: It really shifts the focus from just applying a standard penalty to designing a smarter penalty that targets the most important parts of the embedding space.

Conclusion: Tom: So, wrapping up this discussion on "Rethinking Generalization in Knowledge Graph Embedding: An Empirical Study and Sparse Peak Regularization," the authors are essentially saying that we need to rethink how we apply regularization methods in knowledge graph completion.

Jane: They’ve done a lot of empirical work across several model families—translation-based, tensor decomposition, GNN models—and they found that overfitting is a common problem everywhere.

Lu: The paper highlights that their proposed Sparse Peak Regularization method offers superior control over model complexity and gradient stability compared to methods like N3 or DURA.

Meng: From an engineering standpoint, the implication is that we can use this method to get better results without having to tune dozens of different hyperparameters for every single model architecture.

Tom: It’s about having a method that works well across the board, which makes it much more practical for shipping production models.

Jane: The implication is that selective regularization is an effective and broadly applicable strategy for improving the generalization of these models without sacrificing predictive power on test sets.

Lu: They've demonstrated that this approach can be simple and universal as possible in the context of KGC, focusing on the most important embedding components.

Tom: So, if we look at the overall picture, this paper suggests that controlling model complexity through selective regularization is a very effective way to improve how well these models learn from data.

Jane: It’s a call for more systematic attention in KGE research where we should be designing regularizers that specifically target the high-energy parts of the embedding space.

Linyu Li, Zhi Jin, Yuanpeng He, Dongming Jin, Haoran Duan, Zhengwei Tao, Xuan Zhang, Jiandong Li

School of Computer Science, Peking University

cs.LG, cs.CL

Submitted: 2025-05-29

Updated: 2026-10-20

Code: https://github.com/MIRALab-USTC/KGE-DURA

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: The gist The proposed Sparse Peak Regularization (SPR) method rethinks regularization for Knowledge Graph Completion (KGC) by selectively penalizing important components in embeddings to alleviate

Key concepts

Knowledge Graph Embedding (KGE): KGE maps entities and relations in knowledge graphs into continuous vector spaces. This allows machine learning models to learn patterns from graph data, enabling tasks like link prediction or question answering by representing complex relationships numerically.
Overfitting: Overfitting occurs when a model learns the training data too well, including its noise, causing it to perform excellently on training examples but poorly on new, unseen data. This is often seen in KGE models where they capture spurious patterns instead of generalizable knowledge.
Sparse Peak Regularization (SPR): SPR is a novel regularization technique that selectively penalizes only the dominant components in the embedding vectors—those with high squared magnitudes. It discards low-energy, noisy components, effectively reducing the model's complexity while retaining crucial information for improved generalization.

Terminology

Regularization — 7× in this paper · explained in 3 episodes
Knowledge Graph — 3× in this paper · explained in 8 episodes
Core — 1× in this paper · explained in 2 episodes
Generalization — 1× in this paper · explained in 13 episodes
Knowledge Graph Completion (KGC) — 1× in this paper · explained in 2 episodes
Knowledge graphs — 1× in this paper · explained in 2 episodes
Sparsity — 1× in this paper · explained in 5 episodes

Summary

The gist The proposed Sparse Peak Regularization (SPR) method rethinks regularization for Knowledge Graph Completion (KGC) by selectively penalizing important components in embeddings to alleviate overfitting and enable models to surpass their original performance upper bounds

Necessity of Regularization

Extensive empirical studies on various KGC models reveal that many existing KGC models have overfitting problems, evidenced by a large discrepancy between their high training performance and low validation performance Moreover, without regularization, the MR index gradually increases, which further illustrates worsening stability in model predictions and poor generalization Adding regularization significantly improves the generalization ability of the model on different data sets and breaking the upper bound of the model’s performance

Proposed SPR Regularization Method

The paper introduces a novel sparse-regularization method called SPR, which uses the idea of sparsity to selectively discard some unimportant items and selectively retain the operation items to achieve the purpose of regularization The core idea is to “sparse” the elements that are input to the regularization term so that only important components (i.e., components with large squared magnitudes) are penalized, while many almost negligible components (which may just be noise) are discarded The sparsification is achieved by sorting the (non-negative) squared terms and “masking” those whose cumulative sum is lower than δ and “discarding” them (i.e., zeroing them) The SPR regularization term of the complete triple is defined as SPRVh i, R r j, V t k = V h i 2 sparse + V t k 2 sparse + V h i ⊙ R r j sparse + V t k ⊙ R r j sparse

Empirical Performance and Comparison

Comparative experiments on multiple datasets and multiple models show that the SPR regularization method is better than other regularization methods and can enable the KGC model to further break through the performance margin For example, in the ComplEx model in the FB15K-237 dataset, the MRR of the basic ComplEx is 0.35, while ComplEx-SPR increases it to 0.371, achieving a significant improvement In all datasets and compared to regularized baseline methods such as F2, N3, DURA, and ER, SPR always maintains the best or near-best performance on most models and indicators

Theoretical Advantages of SPR

The theoretical analysis shows that for linear or bilinear KGC scorers, the empirical Rademacher complexity is bounded by a constant multiple of the expected l2-norm of the effective embeddings Furthermore, SPR acts as an adaptive sparsity-inducing regularizer by masking smaller components, which reduces the effective dimensionality of the embedding space This selective retention of dominant components mirrors feature selection strategies and enhances the signal-to-noise ratio by eliminating noise

Impact on Model Optimization

The optimization dynamics show that SPR’s gradient is zero for masked components, effectively freezing those dimensions during updates, which reduces the parameter space Additionally, SPR’s deterministic nature yields zero conditional variance in the gradient, unlike Dropout’s stochastic noise This stability enhances gradient update reliability and potentially accelerates convergence

Generalization Across Model Types

The SPR method is versatile enough to enhance performance consistently across various types of KGC models, including GNN-based, translation-based, tensor decomposition-based KGC models, and Temporal Knowledge Graph Completion (TKGC) models The experimental results for the CompGCN variants show that introducing dropout alone leads to substantial improvements across the board, and SPR goes a step further by penalizing specific entity–relation interactions The experimental results clearly show that regularization is also very important in the task of TKGC

Conclusion

In conclusion, SPR’s deterministic and selective regularization offers superior control over model complexity, gradient stability, and optimization efficiency This method mitigates overfitting by selectively penalizing the most important components in the entity and relation embeddings while discarding less influential and potentially noisy elements This selective approach leads to better generalization by focusing on informative features while discarding noise This method is simple and universal as possible in the KGC model and achieves better results The paper hopes to be as universal as possible in the KGC model and as simple and universal as possible

How Datasets Composition Influences Regularization Performance

The overall trend of training set size and regularization improvement reveals that when observing the data of YAGO3-10 (the largest training set), no matter which model (CP, ComplEx, GIE, RESCAL), the percentage of Hits@1 improvement is generally the lowest Conversely, the data of WN18RR (the smallest training set) generally has the highest or close to the highest percentage of Hits@1 improvement This suggests that large-scale datasets with more training sets themselves

Improvements for AI systems

Develop a more robust Knowledge Graph Completion model by integrating Sparse Peak Regularization (SPR) to mitigate overfitting on unseen data. This will enable models to break through the upper bound of their original performance by selectively penalizing only the most important components in the entity and relation embeddings.
Enhance the generalization capability across diverse KGC architectures, such as Graph Neural Network based models, by applying SPR regularization. This allows GNN-based models to surpass their conventional performance ceiling and achieve better results than those trained without regularization.
Improve prediction stability by employing SPR to stabilize gradients during training in KGC models. Since SPR has zero conditional variance in the gradient, this stability enhances gradient update reliability, potentially accelerating convergence compared to methods like Dropout which introduce stochastic noise.
Optimize embedding quality for entities by using SPR to selectively retain informative features and discard noise. This selective retention focuses the model on informative features while discarding less influential elements, resulting in significantly better clustering in visualization experiments.
Increase the model's ability to handle high-dimensional embeddings effectively by using SPR as an adaptive sparsity inducer. By masking smaller components, SPR reduces the effective dimensionality of the embedding space, which is advantageous in high-dimensional settings where many dimensions might represent noise rather than signal.