Diagnosing Visual Ignorance in Vision-Language Models
cs.CV, cs.LG
Submitted: 2026-06-05
Updated: 2026-09-27
Code: https://github.com/huggingface/accelerate
License: http://creativecommons.org/licenses/by/4.0/
The gist: Vision-language models (VLMs) achieve high accuracy on many visual question-answering benchmarks, yet it remains unclear whether this accuracy reflects reliable use of the visual details a question
Terminology
Abstract
Vision-language models (VLMs) achieve high accuracy on many visual question-answering benchmarks, yet it remains unclear whether this accuracy reflects reliable use of the visual details a question depends on. We examine this tension through the lens of visual ignorance: cases where a model can represent task-relevant visual evidence without using it to determine the answer. Across three VLMs and twelve benchmarks, we track the answer-relevant information decodable at each decoder layer and measure when it becomes coupled to the final output. The visual evidence needed for the correct answer can already be read out from intermediate layers, yet it has little effect on the output until the final layers of the decoder. Causal residual interchange confirms that shifting the answer requires the late states, not the earlier ones. We then evaluate all twelve benchmarks under progressive Gaussian blur and find that substantial fractions of predictions survive the entire degradation sequence while continuing to receive benchmark credit. These results distinguish the availability of visual information from its influence on prediction, and show that standard accuracy can coexist with limited sensitivity to the tested visual detail. They motivate training and evaluation that make answer-relevant visual distinctions necessary to answer correctly.
Sources
- MIRAGE: The Illusion of Visual Understanding
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- Eliciting Latent Predictions from Transformers with the Tuned Lens
- Sparse Autoencoders Find Highly Interpretable Features in Language Models
- Hidden in plain sight: VLMs overlook their visual representations
- BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
- SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
- k-Sparse Autoencoders
- RePOPE: Impact of Annotation Errors on the POPE Benchmark
- Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Vision Language Models are Biased
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models