SHOVIR: A Benchmark for Evaluating Vision Shortcut Learning in Radiology Report Generation

The gist: SHOVIR introduces a benchmark for evaluating vision shortcut learning in Radiology Report Generation by extending existing chest X-ray datasets with per-box CheXpert labels and defining image-level and disease-level occlusion experiments to contrast baseline performance against localized, region-specific perturbations.

Introduction and Motivation

Current evaluation protocols for Vision-Language Models (VLMs) in Radiology Report Generation (RRG) rely on report-level metrics that measure lexical overlap or aggregate clinical correctness, but these metrics do not test whether individual diagnostic statements stem from the actual pathological evidence visible in the image. Such metrics allow models to achieve competitive scores by exploiting learned priors or spurious correlations, a failure mode referred to as vision shortcut. The paper aims to rigorously determine whether current VLMs for RRG derive diagnostic statements from the clinically informative image evidence, or whether their apparent success is partially sustained by vision shortcut behaviors learned from spurious training correlations.

SHOVIR Benchmark Design

SHOVIR extends two spatially annotated chest X-ray datasets, MIMICCXR and PadChest-GR, with per-box CheXpert labels, and defines image-level and disease-level occlusion experiments that contrast baseline performance on clean images against localized, region-specific perturbations. The benchmark leverages two chest X-ray datasets annotated with three layers: radiology reports, spatially grounded bounding boxes linking each finding to its anatomical region, and per-box pathology labels.

The experiments are designed to isolate two distinct behaviors at the disease-class level: direct shortcuts, where a finding persists after its visual evidence is removed, and contextual shortcuts, where detection degrades once co-occurring pathologies are occluded despite the target region remaining intact.

The three stages of SHOVIR are: (i) derivation of two structured annotation sets that extend existing bounding boxes in MIMIC-CXR and PadChest-GR with per-box CheXpert labels; (ii) construction of image-level and disease-level occlusion experiments that progressively remove visual evidence from the images; (iii) inference framework applied to eight VLMs under a unified protocol; and (iv) evaluation of generated reports via NLP, clinical, and shortcut metrics that measure the effect of visual evidence removal at both image and disease-class level.

Experimental Conditions

The experiments are designed based on the spatial scale of intervention: Image-Level and Disease-Level.

Image-Level experiments include the BASELINE setting for original, unperturbed images and the FULL NOISE IMAGE setting, where the entire image is replaced with synthetic noise to quantify dependence on genuine visual content.

Disease-Level experiments introduce three region-level occlusion conditions for a target class c: Random Occlusion (RO), Object Class Occlusion (OCO), and Different Object Class Occlusion (DOCO).

The RO condition serves as a control, where noise is inserted into bounding boxes placed at random locations overlapping with neither Bc nor Bc¯.

The OCO condition targets direct visual evidence for class c by occluding all disease boxes Bc while leaving the rest of the image intact.

The DOCO condition applies the complementary intervention by occluding all co-disease boxes Bc¯ while preserving Bc, removing surrounding pathological context without altering the target region.

Evaluation Metrics

The evaluation protocol comprises three complementary groups of metrics: NLP metrics that assess linguistic quality, clinical metrics that assess diagnostic correctness, and shortcut metrics that quantify performance changes under controlled interventions.

NLP Metrics include BLEU-1, BLEU-4 [37], and ROUGE-L [28] which capture surface-level n-gram and subsequence overlap but do not assess clinical correctness.

Clinical Metrics employ three metrics: F1-CheXbert [41] extracts pathology labels from generated and reference reports across the 14 CheXpert classes, F1-RadGraph [20] evaluates finer-grained clinical content by comparing entity-relation graphs extracted from both reports, and GREEN [36] complements these with a semantically informed assessment via a fine-tuned LLM.

Shortcut Metrics quantify shortcut behavior by measuring how diagnostic correctness changes when visual evidence is selectively removed.

At the image level, the visual-reliance drop is defined as F1(·) = F1(Baseline) - F1(FullNoise). A larger ∆FN indicates stronger visual grounding.

At the Disease-Level, the corrected drops for OCO and DOCO are defined as:

ΔOCO = µ-F1(RO) - µ-F1(OCO).

ΔDOCO = µ-F1(RO) - µ-F1(DOCO).

Evaluation Results Summary

All models exhibit substantial shortcut behavior, and high baseline report quality does not imply faithful spatial grounding. Models that produce clinically fluent reports can simultaneously rely on dataset priors and co-occurrence patterns rather than on the underlying visual evidence.

In the image-level analysis, CheXagent-2, MedGemma, and MAIRA-2 display the largest ∆FN (≥ 0.45 on MIMICCXR; ≥ 0.36 on PadChest-GR), with F114 under FULL NOISE IMAGE dropping to 0.03-0.11 and GREEN scores collapsing to near zero. Conversely, LLaVA-Rad is the least sensitive to noise, retaining the highest F114 under full corruption (0.320 on MIMIC-CXR; 0.175 on PadChest-GR) and yielding the smallest ∆FN on both datasets.

In disease-level occlusion, OCO directly tests whether predictions are grounded in the target anatomical region by occluding all disease boxes Bc. Most models maintain stable µ-F1 up to approximately p=40%, then degrade sharply at higher perturbation ratios, confirming that models do anchor predictions to the correct pathology regions.

Under DOCO, µ-F1 drops markedly even though the target pathology region remains fully visible. This degradation indicates that predictions partly depend on contextual findings rather than solely on the target anatomy, pointing to a plausible shortcut behavior.

Overall, high baseline report quality does not guarantee faithful spatial grounding. Shortcut reliance is not intrinsic to individual pathologies but emerges from the interaction between co-occurrence structure and report style in the training distribution.

Conclusion

SHOVIR reveals that all models exhibit substantial shortcut behavior, a critical clinical concern. High baseline report quality does not imply faithful spatial grounding. Models that produce clinically fluent reports can simultaneously rely on dataset priors and co-occurrence patterns rather than on the underlying visual evidence. Shortcut reliance is not intrinsic to individual pathologies but emerges from the interaction between disease prevalence, co-occurrence structure, and report style in the training distribution. This benchmark opens several directions for future work.

The project page is available at https://github.com/arco-group/ShoViR-Bench.

References

[1] Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi.

1. Develop region-aware assessment protocols for Radiology Report Generation (RRG) by incorporating spatial grounding checks into evaluation metrics. This allows for distinguishing between direct shortcuts and contextual shortcuts, as identified in Section 3.2, which are critical for ensuring diagnostic statements stem from actual pathological evidence rather than learned priors.

2. Implement a rigorous noise-based evaluation framework that uses distribution-preserving noise to quantify visual reliance, as described in Section 3.2 (Image-Level experiments). This system can measure the visual-reliance drop (Equation 2), allowing researchers to assess whether a model's performance gap with clean images is due to loss of genuine visual content or exploitation of low-level statistical cues.

3. Design targeted occlusion experiments for disease-level analysis, specifically using Random Occlusion (RO), Object Class Occlusion (OCO), and Different Object Class Occlusion (DOCO) as outlined in Section 3.2. This enables the system to determine if predictions are grounded in the target anatomical region or distributed across contextual cues, by measuring the effect of removing specific disease boxes versus co-occurring pathology boxes.

4. Fine-tune model training paradigms to encourage localized evidence use, based on findings that MAIRA-2's success is linked to its design (Section 4.3). This involves modifying loss functions or supervision signals to incentivize fine-grained spatial representations rather than relying solely on report-level optimization which could offer weaker incentives for fine-grained spatial representations (Section 4.3).

5. Create a dataset curation pipeline that unifies freetext reports and anatomically localized bounding boxes with per-box CheXpert labels, following the methodology in Section 3.1. This standardized resource will serve as the foundation for any future region-level visual assessment in RRG, ensuring consistent evaluation across different datasets like MIMIC-CXR and PadChest-GR.

Current evaluation protocols for Vision-Language Models (VLMs) in Radiology Report Generation (RRG) rely on report-level metrics that measure lexical overlap or aggregate clinical correctness. However, such metrics do not test whether individual diagnostic statements stem from the actual pathological evidence visible in the image. This allows models to achieve competitive scores by exploiting learned priors or spurious correlations, a failure mode we refer to as vision shortcut. We introduce SHOVIR, a benchmark for evaluating vision shortcut behavior in RRG. SHOVIR extends two spatially annotated chest X-ray datasets, MIMIC-CXR and PadChest-GR, with per-box CheXpert labels, and defines image-level and disease-level occlusion experiments that contrast baseline performance on clean images against localized, region-specific perturbations. Comparing predictions across these conditions isolates two failure modes at the disease-class level: direct shortcuts, where a finding persists after its visual evidence is removed, and contextual shortcuts, where detection degrades once co-occurring pathologies are occluded despite the target region remaining intact. Benchmarking eight state-of-the-art VLMs, we find that shortcut behavior varies substantially across architectures and datasets. Models achieving the highest baseline report quality do not necessarily rank highest in spatial grounding, revealing that clinically fluent generation can coexist with shallow reliance on visual evidence. These findings expose a blind spot in current RRG evaluation and motivate region-aware assessment protocols.