Self-Aware Knowledge Probing: Evaluating Language Models' Relational Knowledge through Confidence Calibration

arXiv:2601.18901 · cs.CL · Submitted 2026-01-26 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Self-Aware Knowledge Probing".

Tom: The gist The proposed framework assesses model reliability by probing relational knowledge through three confidence estimation modalities: intrinsic confidence, structural consistency, and semantic grounding.

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So, we've talked about the basics of this paper, "Self-Aware Knowledge Probing: Evaluating Language Models' Relational Knowledge through Confidence Calibration." We’ve established that existing methods often ignore how reliable a model’s confidence scores actually are >

Jane: That’s right. This paper proposes a novel calibration probing framework for relational knowledge that covers three modalities of model confidence: intrinsic confidence, structural consistency, and semantic grounding >

Lu: The main thrust is to move beyond just measuring attainment—which is whether the model can predict the correct token—and focus on the model’s awareness of its own knowledge boundaries >

Meng: They use a unified scoring rule called BEAR which shifts things from open-ended generation to a closed-set ranking by estimating the full log-likelihood for each statement created with a template >

Lalam: The results show that most models, especially those pre-trained with the masking objective, are overconfident when measured by these different confidence estimates >

Tom: They specifically point out that "CMargin" consistently performs better than "CBase" because CBase is markedly overconfident >

Jane: And they found a reliability gap where MLMs remain fundamentally less aware of their own knowledge boundaries than those from causal language models, despite similar factual accuracy >

Lu: Also, they noticed a scale-dependency for semantic grounding: smaller models seem "semantically blind" to expressions of doubt, but larger models start aligning their internal confidence with linguistic cues >

Meng: That’s a practical concern. If smaller models don't register uncertainty well, it limits how much we can trust them when they are unsure about something >

Lalam: They also looked at adding epistemic markers like "certainly" or "possibly," and those markers mostly hurt calibration because LMs are consistently overconfident >

Tom: So, the key thing here is that getting well-calibrated confidences for relational knowledge is really hard, which hints that LMs might not have a deeper understanding of their own knowledge boundaries >

Jane: Exactly. The paper shows that regardless of the pre-training objective or the model family you pick, confidences tend to be overconfident, and the MLM scoring generally makes this overconfidence behavior worse >

Lu: It seems like they’re pushing for a more self-aware way to probe knowledge where the AI can assess its own uncertainty >

Meng: From an engineering standpoint, if we can get better at predicting these calibrated scores, it helps us build systems that are more transparent about when they might be wrong >

Lalam: It gives us a way to assess the quality of the relational knowledge internalized during pre-training, not just whether a single answer is correct >

Conclusion: Tom: So wrapping up this discussion on "Self-Aware Knowledge Probing: Evaluating Language Models' Relational Knowledge through Confidence Calibration." The authors are looking at how much relational knowledge language models have learned during pre-training and how reliable their confidence scores are >

Jane: They propose a framework that looks at intrinsic confidence, structural consistency, and semantic grounding to get a more nuanced view of model reliability >

Lu: The main implication is that current methods for measuring knowledge probes focus too much on attainment, like just predicting the correct token, instead of assessing the model’s awareness of its own knowledge boundaries >

Meng: This suggests we need better ways to evaluate AI systems that can tell us when they are unsure about a fact or a relationship >

Lalam: The paper shows that even with different training objectives, like masking versus causal language models, there's a consistent tendency for overconfidence in the scores >

Tom: And it points toward the idea that we need self-aware knowledge probing to truly understand how these models know what they know and where their limits are >

Christopher Kissling, Elena Merdjanovska, Alan Akbik

Humboldt-Universität zu Berlin

cs.CL

Submitted: 2026-01-26

Updated: 2026-10-08

Code: https://github.com/diamir707/knowledge-calibration

License: http://creativecommons.org/licenses/by-sa/4.0/

Importance score: 83/100

The gist: The gist The proposed framework assesses model reliability by probing relational knowledge through three confidence estimation modalities: intrinsic confidence, structural consistency, and semantic

Key concepts

Knowledge Probing: This is a method used to check how much relational knowledge a language model has learned during its training. It acts like a diagnostic tool, quantifying the specific facts or relationships the model has internalized. The goal is to measure the depth and reliability of this acquired knowledge.
Confidence Calibration: Calibration refers to how accurately a model's confidence scores reflect its true level of certainty regarding an answer. If a model is well-calibrated, when it says it is 80% sure, it should be correct 80% of the time. Poor calibration means the model is often overconfident or underconfident.
Intrinsic Confidence: This measure comes from the raw mathematical outputs of a language model's prediction process, specifically its logprobabilities. It includes metrics like Base-Confidence and Margin-Confidence, which look at how spread out the probabilities are. These scores are used to estimate a model's internal certainty before considering external factors.
Structural Consistency: This modality evaluates confidence by checking if the model's answer remains consistent even when the question or statement is slightly rephrased. It tests whether the model understands the underlying relationship rather than just memorizing a specific phrasing, making it a robust measure of relational knowledge.

Terminology

Calibration — 3× in this paper · explained in 7 episodes
Accuracy — 1× in this paper · explained in 1 episode
Consistency — 1× in this paper · explained in 3 episodes
Grounding — 1× in this paper · explained in 3 episodes

Summary

The gist The proposed framework assesses model reliability by probing relational knowledge through three confidence estimation modalities: intrinsic confidence, structural consistency, and semantic grounding.

Knowledge Probing Framework

Knowledge probes serve as the primary diagnostic tool for quantifying the relational knowledge internalized by language models during pre-training. This framework proposes a novel calibration probing framework for relational knowledge that covers three modalities of model confidence: (1) intrinsic confidence, (2) structural consistency and (3) semantic grounding. This multi-faceted approach allows for a granular analysis of how model architecture and scale influence the reliability of factual retrieval.

Confidence Estimation Modalities

The paper decomposes model confidence into three distinct modalities.

Intrinsic Confidence is derived from raw logprobabilities. This includes estimates like Base-Confidence and Margin-Confidence, where CBase is defined as the maximum value after softmax normalization and CMargin is the difference between the largest and second-largest values returned by the softmax. Aggregation strategies include plurality voting and minimum confidence aggregation.
Semantic Grounding is evaluated through the model’s response to explicit linguistic markers of uncertainty.

Experimental Setup and Findings

The evaluation uses a closed-set probing setup called BEAR which provides a unified scoring rule by shifting the task from open-ended generation to a closed-set ranking. The framework tests whether an LM distinguishes the true fact from distractors by estimating the full log-likelihood of each statement created with a template. Furthermore, CMargin consistently performs well compared to CBase, as CBase is markedly overconfident. The six largest models—gemma-7b, gemma-2b, opt-6.7b, opt-1.3b, gpt2-large and gpt2-xl—have the lowest ACEs and Brier Scores.

Model Comparison and Reliability Gap

The analysis reveals a critical reliability gap where MLMs—despite competitive factual accuracy—remain fundamentally less aware of their own knowledge than CLMs. Moreover, the results suggest a clear scale-dependency for semantic grounding: while smaller models are semantically blind to expressions of doubt, larger models begin to align their internal confidence with linguistic cues.

Impact of Linguistic Confidence Expressions

The study examined the impact of adding epistemic markers like certainly (strengthener) and possibly (weakener) on accuracy and calibration. Injected markers mostly hurt calibration, as LMs are consistently overconfident. However, verbalized strengthening injections consistently increase ACE across all models.

Conclusion

The paper proposes self-aware knowledge probing which assesses the reliability of language models’ relational knowledge through calibration. The findings indicate that obtaining well-calibrated confidences for relational knowledge is highly challenging, strongly indicating that LMs lack a deeper understanding of their own knowledge boundaries. Regardless of the pre-training objective and model, confidences are consistently overconfident, and the MLM scoring generally worsens this behavior. Structural consistency estimates effectively address this issue but reveal a high sensitivity to formulation changes.

Limitations

A limitation noted is that calibration error values are unlikely to transfer directly to more practical settings because the probing is specific to closed-answer-set scenarios. Furthermore, the predefined answer options limit the evaluation to output uncertainty.

Acknowledgments

Elena Merdjanovska and Alan Akbik are supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC 2002/1 “Science of Intelligence” – project number 390523135. Alan Akbik is supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Emmy Noether grant “Eidetic Representations of Natural Language” (project number 448414230)

References

The references listed include Brier 1950. Petroni et al. 2019 Kalo and Fichtel 2022. Wiland et al. 2024. Nixon et al.

Improvements for AI systems

Model reliability assessment through three modalities: intrinsic confidence, structural consistency and semantic grounding. This allows for a granular analysis of how model architecture and scale influence the reliability of factual retrieval.
Quantifying overconfidence: The paper demonstrates that models are overconfident, suggesting that reliance on raw log-probabilities is insufficient. Utilizing the Margin-Confidence estimate, which is defined as the difference between the largest and second-largest softmax values, can help mitigate this by showing how CMargin consistently performs well in terms of ACE and Brier Score across all LMs.
Architecture comparison for training objectives: The framework allows for a head-to-head comparison between causal language models (CLMs) and masked language models (MLMs), revealing that MLMs—despite competitive factual accuracy—remain fundamentally less aware of their own knowledge than CLMs.
Scale-dependency analysis: The study identifies a clear scale-dependency for semantic grounding: while smaller models are 'semantically blind' to expressions of doubt, larger models begin to align their internal confidence with linguistic cues.
Verbalized uncertainty integration: By injecting explicit epistemic markers like certainly or possibly, the system can be tuned to express its limits, as seen when Verbal markers correctly shift calibration curves for large models.

Abstract

Knowledge probing quantifies how much relational knowledge a language model (LM) has acquired during pre-training. Existing knowledge probes evaluate model capabilities through metrics like prediction accuracy and precision. Such evaluations fail to account for the model's reliability, reflected in the calibration of its confidence scores. In this paper, we establish an empirical framework to evaluate the calibration of relational knowledge in a closed answer-set setting. We cover three modalities of model confidence: (1) intrinsic confidence, (2) paraphrase consistency and (3) linguistic confidence. Our extensive analysis of 21 causal and 8 masked language models reveals that most models, but especially those pre-trained with the masking objective, are overconfident. The best-calibrated scores come from confidence estimates that account for inconsistencies due to statement rephrasing. Moreover, calibration improves when increasing the number of answer options.