SHOVIR: A Benchmark for Evaluating Vision Shortcut Learning in Radiology Report Generation

summary

The gist

The gist: SHOVIR introduces a benchmark for evaluating vision shortcut learning in Radiology Report Generation by extending existing chest X-ray datasets with per-box CheXpert labels and defining

In short

SHOVIR creates a benchmark to test if AI models generating radiology reports truly rely on visible image evidence or learn shortcuts from spurious correlations. By occluding specific visual regions, researchers can distinguish between models that use actual pathology and those that rely on learned patterns, revealing a critical risk in current vision-language approaches.

Key concepts

Vision Shortcut Learning: This refers to when a model achieves good results not by understanding the actual visual evidence of a disease, but by exploiting superficial statistical correlations or patterns present in the training data. It means the model is 'cheating' by relying on learned shortcuts instead of genuine visual grounding.
Image-Level Occlusion: This experiment tests how much a model depends on the entire image content. By replacing an entire image with synthetic noise, researchers measure if a model's performance drops significantly, indicating it relies heavily on the complete visual scene rather than just localized features.
Disease-Level Occlusion: This method isolates whether a model anchors its diagnosis to a specific anatomical region or relies on surrounding context. By selectively hiding boxes corresponding to different pathologies (e.g., occluding one disease box while leaving another intact), researchers can see if the prediction remains stable or degrades, revealing reliance on co-occurring findings.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Retrieval-Augmented Generation (RAG) — explained in 21 episodes
Sample Complexity — explained in 20 episodes
Flow Matching — explained in 19 episodes
Quantum Fisher Information (QFI) — explained in 19 episodes
Federated Learning — explained in 17 episodes

This episode discusses

The paper

SHOVIR: A Benchmark for Evaluating Vision Shortcut Learning in Radiology Report Generation · Read on arXiv

Filippo Ruffini, Marco Salme, Rosa Sicilia, Valerio Guarrasi, Paolo Soda

Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Universita Campus Bio-Medico di Roma · Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umea University · UniCamillus-Saint Camillus International University of Health Sciences

Current evaluation protocols for Vision-Language Models (VLMs) in Radiology Report Generation (RRG) rely on report-level metrics that measure lexical overlap or aggregate clinical correctness. However, such metrics do not test whether individual diagnostic statements stem from the actual pathological evidence visible in the image. This allows models to achieve competitive scores by exploiting learned priors or spurious correlations, a failure mode we refer to as vision shortcut. We introduce SHOVIR, a benchmark for evaluating vision shortcut behavior in RRG. SHOVIR extends two spatially annotated chest X-ray datasets, MIMIC-CXR and PadChest-GR, with per-box CheXpert labels, and defines image-level and disease-level occlusion experiments that contrast baseline performance on clean images against localized, region-specific perturbations. Comparing predictions across these conditions isolates two failure modes at the disease-class level: direct shortcuts, where a finding persists after its visual evidence is removed, and contextual shortcuts, where detection degrades once co-occurring pathologies are occluded despite the target region remaining intact. Benchmarking eight state-of-the-art VLMs, we find that shortcut behavior varies substantially across architectures and datasets. Models achieving the highest baseline report quality do not necessarily rank highest in spatial grounding, revealing that clinically fluent generation can coexist with shallow reliance on visual evidence. These findings expose a blind spot in current RRG evaluation and motivate region-aware assessment protocols.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "SHOVIR: A Benchmark for Evaluating Vision Shortcut Learning in Radiology Report Generation".

Jane: Here's the gist of the paper. The gist: SHOVIR introduces a benchmark for evaluating vision shortcut learning in Radiology Report Generation by extending existing chest X-ray datasets with per-box CheXpert labels and defining image-level and disease-level occlusion experiments to contrast baseline performance against localized, region-specific perturbations.

Tom: Current evaluation protocols for Vision-Language Models (VLMs) in Radiology Report Generation (RRG) rely on report-level metrics that measure lexical overlap or aggregate clinical correctness, but these metrics do not test whether individual diagnostic statements stem from the actual pathological evidence visible in the image.

Jane: Such metrics allow models to achieve competitive scores by exploiting learned priors or spurious correlations, a failure mode referred to as vision shortcut.

Tom: The paper aims to rigorously determine whether current VLMs for RRG derive diagnostic statements from the clinically informative image evidence, or whether their apparent success is partially sustained by vision shortcut behaviors learned from spurious training correlations.

Title and authors: Tom: Let's talk about how SHOVIR actually works, based on what the paper says. They extend existing chest X-ray datasets, MIMIC-CXR and PadChest-GR, by adding per-box CheXpert labels >

Jane: That means they take those X-rays and give them a lot more detail than just a general report. They add bounding boxes that link every finding in the report to its specific anatomical region >

Lu: And they don't stop there. They introduce image-level and disease-level occlusion experiments, which is how they test this idea of vision shortcut behavior >

Meng: So, they are systematically removing visual evidence from the images in a controlled way to see what happens to the model's output >

Lalam: The experiment design is pretty specific. They look at two types of shortcuts: direct shortcuts, where a finding persists even after its visual evidence is removed, and contextual shortcuts, where the prediction degrades when co-occurring pathologies are occluded despite the target region remaining intact >

The paper's summary: Tom: The core of SHOVIR is this four-stage benchmark. First, they derive these structured annotation sets with per-box CheXpert labels on MIMIC-CXR and PadChest-GR >

Jane: Then they construct those image-level and disease-level occlusion experiments to progressively remove the visual evidence in a way that isolates different types of shortcuts >

Lu: The third stage is applying this entire framework to eight different vision-language models using a unified protocol across language branch and vision branch components >

Meng: And finally, they evaluate the generated reports using NLP metrics for linguistic quality, clinical metrics for correctness, and shortcut metrics that specifically measure the effect of visual evidence removal at both image and disease-class levels >

Lalam: The evaluation protocol uses three groups of metrics: NLP metrics like BLEU-one and ROUGE-L which check surface overlap, clinical F1 scores like F1-CheXbert, and shortcut metrics that quantify how much the correctness drops when visual evidence is selectively removed > <ref:2606.30201#pg1>

The paper's improvements: Tom: The authors suggest a few key improvements to this benchmark to make it more rigorous. One big thing they propose is developing region-aware assessment protocols for radiology report generation by incorporating spatial grounding checks directly into the evaluation metrics >

Jane: That’s important because it lets them distinguish between direct shortcuts and contextual shortcuts, which is critical for ensuring diagnostic statements stem from actual pathological evidence rather than just learned priors >

Lu: They also suggest implementing a rigorous noise-based evaluation framework that uses distribution-preserving noise to quantify visual reliance by measuring the visual-reliance drop, which allows researchers to assess if the performance gap with clean images is due to loss of genuine visual content or exploitation of low-level statistical cues >

Meng: I'm interested in that noise thing. If we use distribution-preserving noise, it should give us a better idea of whether the model is relying on actual visual content or just some low-level statistical cues in the training data >

Lalam: And they suggest designing targeted occlusion experiments for disease-level analysis using Random Occlusion, Object Class Occlusion, and Different Object Class Occlusion to determine if predictions are grounded in the target anatomical region or distributed across contextual cues >

Conclusion: Tom: To wrap up SHOVIR, the main point is that all models exhibit substantial shortcut behavior. High baseline report quality doesn't guarantee faithful spatial grounding >

Jane: It means models that produce clinically fluent reports can simultaneously rely on dataset priors and co-occurrence patterns rather than on the underlying visual evidence >

Lu: The paper shows that this shortcut reliance isn't intrinsic to individual pathologies, but it emerges from the interaction between disease prevalence, co-occurrence structure, and report style in the training distribution >

Meng: So what this means for us deploying models is that we can't just trust high baseline scores. We have to test how they react when you deliberately remove visual evidence >

Lalam: SHOVIR reveals that all models exhibit substantial shortcut behavior, a critical clinical concern. High baseline report quality does not imply faithful spatial grounding >

More episodes

← Home