Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking
cs.CL
Submitted: 2026-05-11
Updated: 2026-09-19
Code: https://github.com/ledengary/BICR
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large vision-language models (LVLMs) suffer from visual ungroundedness: they can produce a fluent, confident, and even correct response driven entirely by language priors, with the image contributing
Terminology
Abstract
Large vision-language models (LVLMs) suffer from visual ungroundedness: they can produce a fluent, confident, and even correct response driven entirely by language priors, with the image contributing nothing to the prediction. Existing confidence estimation methods cannot detect this, as they observe model behavior under normal inference with no mechanism to determine whether a prediction was shaped by the image or by text alone. We introduce BICR (Blind-Image Contrastive Ranking), a model-agnostic confidence estimation framework. The LVLM stays entirely frozen; instead, we extract its hidden states twice per sample, once with the real image-question pair and once with the image blacked out while the question is held fixed. A small, separate probe is then trained on the real-image hidden state, regularized by a ranking loss that penalizes higher confidence on the blacked-out view, teaching it to treat visual grounding as a signal of reliability. At inference only the single real-image pass is used, so BICR adds zero cost beyond a standard single-view probe. Evaluated across five modern LVLMs and seven baselines on a benchmark covering visual question answering, object hallucination detection, medical imaging, and financial document understanding, BICR achieves the best cross-LVLM average on both calibration and discrimination simultaneously, with statistically significant discrimination gains robust to cluster-aware analysis, at 4-18x fewer parameters than the strongest probing baseline.
Sources
- Instinct vs. Reflection: Unifying Token and Verbalized Confidence in Multimodal Large Models
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- The Internal State of an LLM Knows When It's Lying
- Same Answer, Different Representations: Hidden instability in VLMs
- Object-Level Verbalized Confidence Calibration in Vision-Language Models via Semantic Perturbation
- VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoning
- VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimation
- Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens
- Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding
- Confidence Calibration in Vision-Language-Action Models
- Qwen3 Technical Report
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- Gemma 3 Technical Report
- DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding
- Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification
- Finetuning Language Models to Emit Linguistic Expressions of Uncertainty
- Uncertainty Estimation in Autoregressive Structured Prediction
- Trained on Tokens, Calibrated on Concepts: The Emergence of Semantic Calibration in LLMs
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering