SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders
cs.CV, cs.CL
Submitted: 2026-10-08
Updated: 2026-10-08
Comments: Accepted to EMNLP 2026 Findings
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent large vision-language models (VLMs) pair a visual encoder with a large language model (LLM) and perform well on diverse image-text tasks, yet their reliability is often limited by decoder
Terminology
Abstract
Recent large vision-language models (VLMs) pair a visual encoder with a large language model (LLM) and perform well on diverse image-text tasks, yet their reliability is often limited by decoder attention pathologies that suppress visual evidence and exacerbate hallucinations. In this paper, we revisit visual attention sinks and uncover a structured, layer-dependent behavior: across prompts, early and late decoder layers exhibit prompt-invariant attention collapse onto the same few image regions, which we term PIS (Prompt-Invariant Sinks), whereas mid layers become prompt-conditioned and drive vision-language alignment. This split suggests that treating sinks as a uniform effect is incomplete. Building on this insight, we propose SAGE (Sink-Aware Guided Emphasis), a lightweight intervention that steers decoder attention away from PIS and toward query-dependent regions of interest (ROIs) using token-aligned ROI masks derived from standard vision backbones such as CLIP, ViT, and DINOv3. Evaluated on diverse vision-encoder + decoder-only LLM VLM families, SAGE improves visual grounding, reduces hallucinations, and yields consistent gains across public downstream vision-language benchmarks, including fine-grained visual discrimination settings where localized evidence is crucial, when instantiated with backbone-derived ROI masks.
Sources
- GPT-4 Technical Report
- Qwen Technical Report
- Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations
- PaLI-X: On Scaling up a Multilingual Vision and Language Model
- PaLI: A Jointly-Scaled Multilingual Language-Image Model
- A Survey of Multimodal Hallucination Evaluation and Detection
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models?
- Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models
- Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models
- Evaluating Object Hallucination in Large Vision-Language Models
- A Survey on Hallucination in Large Vision-Language Models
- Large Language Models: A Survey
- DINOv3
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Efficient Large Language Models: A Survey
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- Efficient Streaming Language Models with Attention Sinks
- Qwen3 Technical Report
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models