Structure is Supervision: Multiview Masked Autoencoders for Radiology

arXiv:2511.22294 · cs.CV, cs.LG · Submitted 2025-11-27 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Structure is Supervision".

Jane: Here's the gist of the paper. The gist The authors introduce MVMAE, a self-supervised framework that leverages the natural multi-view organization of radiology studies to learn view-invariant and disease-relevant representations.

Tom: How it works MVMAE is a study-centric pretraining framework that couples masked reconstruction with cross-view alignment to learn view-invariant representations without relying on external supervision MVMAE treats the different image projections acquired within the same study as natural positive pairs and jointly optimizes (i) a masked image modeling objective per view to capture the complementary information from multiple X-ray views acquired during a patient examination; and (ii) a study-level alignment objective to regularize embeddings across projections of the same medical study This converts clinical redundancy into self-supervision The framework adopts a unified FLU (Frontal–Lateral–Unknown) policy that includes all studies containing at least one valid projection, indexing each available view as v ∈ V(i), with V(i) = the set of view types present in study i, n(i) ≥ 1 being the number of views per study (i) The model architecture involves a Vision Transformer (ViT) encoder that processes unmasked input tokens to produce latent representations z(i)v=fθ(x(i)vvis), which are then passed to a lightweight decoder gϕ for reconstruction To make the encoder aware of the projection type without assigning a separate head per view instance, they introduce a learnable modality embedding em ∈ R D for each projection category m ∈ 3, such that all occurrences of a frontal view share the same embedding efrontal This design allows the model to represent projection-specific acquisition bias while keeping a shared encoder across all views How it works MVMAE-V2T extends MVMAE by incorporating radiology reports as an auxiliary supervision signal during pretraining, introducing MVMAE-Vision to Text (MVMAEV2T) Reports are used only during pretraining as an auxiliary signal in addition to image-based objectives; at test time, the model remains a vision-only encoder This is achieved by applying a captioning-style objective inspired by the CapPa framework, where for each view x(i)v of a study i and its study report r(i) = n r(i), we extend MVMAE with a transformer decoder hψ that causally predicts each report token r(i)k conditioned on all previous report tokens r(i)<k, as well as the encodings of the unmasked visual embeddings z(i)v via cross-attention The joint optimization is defined by L(i)V 2T = L(i)Rec + β · L(i)Align + γ · L(i)CE This ensures that complementary information from multiple projections contributes coherently to the text generation pretext, reinforcing the study-level structure inherent in clinical data How it works The evaluation strategy is clinically realistic, adopting a study-level organization and a unified policy that retains single and multi-view exams, incorporates uncertain views, and aggregates multiple projections within a study The experiments investigate four complementary questions: (i) How effective are the representations learned by a pretrained encoder when the model is fully finetuned on a downstream classification task?

Jane: (ii) How robust are MVMAE representations under limited supervision?

Tom: (iii) Does leveraging the structure at pretraining help with model calibration?

Title and authors: Tom: We started by looking at the paper's title, "Structure is Supervision: Multiview Masked Autoencoders for Radiology," and who wrote it. It immediately tells you that the main idea isn't just about seeing an image, but about how the structure of the data itself should be used for learning.

Jane: That’s a pretty big claim. What does "Structure is Supervision" mean in plain terms? Is it suggesting that we don't need external labels as much if the images are organized in a certain way?

Lu: It means they are using the inherent organization of medical data—the fact that multiple views belong to one patient study—as a source of supervision. They aren't just feeding random images into a standard masked autoencoder setup.

Meng: So, what’s the mechanism there? How does that structure actually translate into learning better representations for the AI?

Tom: The mechanism involves combining masked image reconstruction with this cross-view alignment loss, which they call converting clinical redundancy across projections into a powerful self-supervisory signal. They are making that redundancy work for them.

Jane: That sounds like they’re taking something that usually just wastes data—having multiple images of one patient—and turning it into a training advantage.

Lu: Precisely, and they are showing that this structure encodes strong geometric and semantic coherence across views that radiologists routinely exploit for interpretation, which is what makes the representations useful.

Meng: So the implication is that if you can identify those structural relationships in your data, you don't have to rely solely on massive external datasets to teach the model how to see context.

Tom: They are showing that this specific structure of medical imaging provides a clear path forward for learning robust representations without needing a huge amount of expert labels upfront.

Jane: So, they’re arguing that the way clinical data is naturally organized offers a strong internal signal for self-supervision.

Lu: Right. And they are leveraging those known relationships to learn features that are more meaningful and transferable across different institutions, which is what they aim for when they talk about view-invariant representations.

Meng: That sounds promising if you can apply that concept to other structured data types where you have inherent organization, not just raw unstructured noise.

Tom: If we can formalize that concept—using structure as supervision—it could change how we design pretraining strategies for almost any domain involving multi-modal or multi-view data.

Jane: So the title really points to a way of thinking about representation learning that goes beyond just pattern matching on pixels.

Lu: It shifts the focus from just finding patterns in pixels to understanding the underlying geometry and coherence of the data organization itself.

Meng: That’s ambitious, but if it works, it means we can build models that are inherently more robust to variations in how a patient is scanned or imaged.

Tom: The paper is laying out a clear methodology for using structure as supervision in medical imaging, and that’s the focus here.

Jane: So we're going to see how they put this structural insight into practice across the study.

The paper's summary: Tom: Now let’s look at the actual summary of "Structure is Supervision: Multiview Masked Autoencoders for Radiology." It outlines exactly what MVMAE and MVMAE-V2T are doing.

Jane: So, the core idea is that they introduce MVMAE, which combines masked image reconstruction with cross-view alignment to learn view-invariant and disease-relevant representations using the natural multi-view organization of radiology studies.

Lu: They treat different image projections acquired within the same study as natural positive pairs and jointly optimize two things: first, a masked image modeling objective per view to capture complementary information from multiple X-ray views acquired during a patient examination; and second, a study-level alignment objective to regularize embeddings across projections of the same medical study.

Meng: So they are essentially forcing the model to learn that the different views aren't independent entities, but parts of a single coherent whole.

Tom: That’s right. They are treating clinical redundancy across projections as a powerful self-supervisory signal for learning these view-invariant representations without needing external supervision.

Jane: And they then extend it with MVMAE-V2T, which introduces radiology reports as an auxiliary text supervision signal during pretraining, while keeping the test time purely vision-based.

Lu: For that extension, they use a vision-to-text objective where each study's report is causally predicted token by token based on previous tokens and the visual embeddings through cross-attention.

Meng: So they are using those reports to guide the visual encoder toward disease-relevant semantics during pretraining, which is smart because it leverages that existing knowledge.

Tom: The overall optimization looks like L(i)V2T equals L(i)Rec plus beta times L(i)Align plus gamma times L(i)CE, ensuring complementary information from multiple projections contributes coherently to the text generation pretext.

Jane: It’s a unified framework that handles both the image structure and the textual description simultaneously during pretraining, reinforcing that study-level structure.

Lu: They also introduce a specific design where they use learnable modality embeddings for each projection category to represent projection-specific acquisition bias while keeping a shared encoder across all views.

Meng: So they’re handling the variability of how an X-rays are taken by giving the model specific tokens for frontal versus lateral views without needing separate encoders.

Tom: It sounds like they’ve built a comprehensive system that accounts for both the image projections and the text context in a structured way.

Jane: So, to recap, it's about using structural alignment and text guidance as complementary forces to learn robust representations from medical images.

The paper's improvements: Tom: Let’s talk about what they propose as improvements in "Structure is Supervision: Multiview Masked Autoencoders for Radiology." They aren't just presenting a single idea, but a couple of enhancements to the original MVMAE concept.

Jane: So, the main improvement is that they show that combining masked image reconstruction with cross-view alignment is more effective than doing either objective alone.

Lu: That combination allows them to learn view-invariant representations by exploiting the natural multi-view organization, effectively converting clinical redundancy across projections into a powerful self-supervisory signal.

Meng: So, it’s not just one loss; it’s the synergy between reconstruction and alignment that's doing the heavy lifting for learning those coherent features.

Tom: That synergy captures geometric and physiological coherence across views that radiologists routinely exploit for interpretation, which is what makes the representations useful in a clinical context.

Jane: And they also improved things by introducing MVMAE-V2T, which incorporates radiology reports as an auxiliary text signal during pretraining to enhance semantic grounding.

Lu: That extension leads to clear gains under limited labeled data without requiring any text at inference time, because the model learns from structured, domain-specific supervision to guide the visual encoder toward disease-relevant semantics.

Meng: So this means you can get better results when you're working with scarce labels, which is a major practical advantage for deployment.

Tom: And they also showed that this leads to superior label efficiency when finetuned on scarce data regimes because MVMAE consistently outperforms all baselines in those settings under linear probing.

Jane: They also improved calibration by showing that cross-view regularization encourages more reliable confidence estimates alongside improved discrimination, which is a key point for reliability.

Lu: That means the model’s confidence scores are better because the process of aligning views forces it to make more reliable decisions during inference.

Meng: So we're looking at gains in both representation quality and how much data you need later on, which is what matters when you’re trying to ship a model.

Tom: So the key improvements boil down to using these structural relationships for self-supervision and adding text guidance for better semantics in those pretraining phases.

Conclusion: Jane: So we’ve covered a lot about "Structure is Supervision: Multiview Masked Autoencoders for Radiology," wrapping up our discussion on the paper. To summarize, the main point is that using the inherent structure of medical data as a source of supervision works really well for learning robust representations.

Lu: The paper introduced MVMAE and MVMAE-V2T as ways to combine image reconstruction with cross-view alignment and text guidance to create these high-quality, view-invariant features.

Meng: It gives us a practical framework that accounts for projection bias while also provides a way to handle limited labels in a deployment setting.

Tom: The results show MVMAE achieves the highest performance overall, and MVMAE-V2T achieves the best Top-five classification performance across those datasets.

Jane: And they also proved that these structural pretraining methods can be used to build foundation models that are more robust because they generalize well across institutions.

Lu: The principles extend naturally to other structured modalities like temporal follow-ups or multi-sequence imaging, suggesting this framework is broadly applicable beyond just radiology.

Meng: It means we can start building foundation models grounded in structural and semantic coherence instead of needing massive annotation scales across every modality just to get good results.

Tom: The whole point is that the paper establishes a scalable way to learn representations by leveraging the inherent organization of clinical data rather than relying on annotation scale alone.

Jane: It’s a solid piece of work that shows how combining structural alignment and text guidance can lead to better, more reliable AI systems for medical imaging.

Lu: The framework is a scalable approach for representation learning in this domain.

Sonia Laguna, Andrea Agostini, Alain Ryser, Samuel Ruiperez-Campillo, Irene Cannistraci, Moritz Vandenhirtz

Department of Computer Science, ETH Zurich

cs.CV, cs.LG

Submitted: 2025-11-27

Updated: 2026-10-21

Journal ref: Transactions on Machine Learning Research (TMLR) 2026

Code: https://github.com/agostini335/MVMAE

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The gist The authors introduce MVMAE, a self-supervised framework that leverages the natural multi-view organization of radiology studies to learn view-invariant and disease-relevant representations.

Key concepts

MVMAE
A self-supervised framework that treats different X-ray projections from the same patient study as natural positive pairs. It learns view-invariant representations by simultaneously reconstructing masked images for each view and aligning their embeddings across all views to capture complementary information.
Cross-view Alignment
An objective used in MVMAE to regularize the model's embeddings across different projections of the same medical study. This forces the model to learn representations that are consistent regardless of which specific X-ray view is presented, effectively exploiting clinical redundancy for self-supervision.
MVMAE-V2T
An extension that integrates radiology reports as an auxiliary text signal during pretraining. This helps guide the visual encoder toward disease-relevant semantics, leading to enhanced performance and superior label efficiency when fine-tuned on scarce labeled data.

Terminology

Summary

The gist The authors introduce MVMAE, a self-supervised framework that leverages the natural multi-view organization of radiology studies to learn view-invariant and disease-relevant representations.

How it works

MVMAE is a study-centric pretraining framework that couples masked reconstruction with cross-view alignment to learn view-invariant representations without relying on external supervision MVMAE treats the different image projections acquired within the same study as natural positive pairs and jointly optimizes (i) a masked image modeling objective per view to capture the complementary information from multiple X-ray views acquired during a patient examination; and (ii) a study-level alignment objective to regularize embeddings across projections of the same medical study This converts clinical redundancy into self-supervision The framework adopts a unified FLU (Frontal–Lateral–Unknown) policy that includes all studies containing at least one valid projection, indexing each available view as v ∈ V(i), with V(i) = the set of view types present in study i, n(i) ≥ 1 being the number of views per study (i)

The model architecture involves a Vision Transformer (ViT) encoder that processes unmasked input tokens to produce latent representations z(i)v=fθ(x(i)vvis), which are then passed to a lightweight decoder gϕ for reconstruction To make the encoder aware of the projection type without assigning a separate head per view instance, they introduce a learnable modality embedding em ∈ R D for each projection category m ∈ 3, such that all occurrences of a frontal view share the same embedding efrontal This design allows the model to represent projection-specific acquisition bias while keeping a shared encoder across all views

How it works

MVMAE-V2T extends MVMAE by incorporating radiology reports as an auxiliary supervision signal during pretraining, introducing MVMAE-Vision to Text (MVMAEV2T) Reports are used only during pretraining as an auxiliary signal in addition to image-based objectives; at test time, the model remains a vision-only encoder This is achieved by applying a captioning-style objective inspired by the CapPa framework, where for each view x(i)v of a study i and its study report r(i) = n r(i), we extend MVMAE with a transformer decoder hψ that causally predicts each report token r(i)k conditioned on all previous report tokens r(i)<k, as well as the encodings of the unmasked visual embeddings z(i)v via cross-attention The joint optimization is defined by L(i)V 2T = L(i)Rec + β · L(i)Align + γ · L(i)CE This ensures that complementary information from multiple projections contributes coherently to the text generation pretext, reinforcing the study-level structure inherent in clinical data

How it works

The evaluation strategy is clinically realistic, adopting a study-level organization and a unified policy that retains single and multi-view exams, incorporates uncertain views, and aggregates multiple projections within a study The experiments investigate four complementary questions: (i) How effective are the representations learned by a pretrained encoder when the model is fully finetuned on a downstream classification task? (ii) How robust are MVMAE representations under limited supervision? (iii) Does leveraging the structure at pretraining help with model calibration? and (iv) How does performance vary with the number of views per study

How it works

The evaluation involves benchmarking MVMAE and MVMAE-V2T against a spectrum of pretrained encoders, including Supervised, Independent, and Vision–Language models like BiomedCLIP and CheXagent The performance is assessed across three large public datasets: MIMIC-CXR, CheXpert Plus, and PadChest The evaluation policy for downstream classification uses a late-fusion ensemble that averages prediction scores from unimodal single-view models for both the proposed MVMAE and baselines

How it works

The results show that MVMAE achieves the highest performance overall, with strong generalization across institutions and outperforming both supervised and vision–language baselines Specifically, Table 1 reports that MVMAE-V2T achieves the highest Top-5 classification performance across datasets Furthermore, in label efficiency experiments, MVMAE consistently outperforms all baselines under linear probing, demonstrating more linearly separable representations and superior label efficiency

How it works

MVMAE also demonstrates superior label efficiency, achieving high performance with significantly fewer annotations, a crucial property in clinical domains where expert labeling is costly and often incomplete The results indicate that MVMAE yields label-efficient representations, while MVMAE-V2T provides additional gains when finetuning under scarce label conditions The framework generalizes beyond medicine to any multimodal domain where structured relationships between views or modalities can serve as self-supervision for learning robust and transferable representations

How it works

MVMAE yields better-calibrated predictions among pretraining baselines, suggesting that cross-view regularization encourages more reliable confidence estimates alongside improved discrimination The results highlight the importance of aligning pretraining objectives and evaluation protocols with the real-world organization of clinical data Overall, these results establish a scalable framework for representation learning in medical imaging that leverages the inherent organization of clinical data Additionally, MVMAE-V2T exhibits advantages in label-scarce regimes, which are highly representative of clinical practice The same principles naturally extend to other structured modalities, such as temporal follow-ups or multi-sequence imaging, paving the way toward clinically grounded foundation models built on structural and semantic coherence rather than annotation scale

How it works

The authors introduced MVMAE, the first multi-view masked autoencoder for radiology that adapts established masked reconstruction and cross-view alignment paradigms to the structure of medical data They further proposed MVMAE-V2T, which integrates radiology reports as an auxiliary text-based learning signal Together, MVMAE and MVMAE-V2T demonstrate that structural and textual supervision can act as complementary forces: structural alignment provides robustness and transferability, and text guidance enriches semantic understanding The same principles naturally extend to other structured modalities, such as temporal follow-ups or multi-sequence imaging The paper concludes by establishing a scalable framework for representation learning in medical imaging that leverages the inherent organization of clinical data

How it works

The authors introduced MVMAE, the first multi-view masked autoencoder that leverages the study-level structure of clinical imaging data to learn robust and transferable representations without relying on textual inputs during pretraining By combining masked reconstruction with cross-view alignment, MVMAE effectively exploits the inherent multiview nature of radiology studies, yielding consistent improvements in performance and label efficiency across MIMIC-CXR, CheXpert, and PadChest Building upon this foundation, further proposed MVMAE-V2T, which integrates radiology reports as an auxiliary text-based learning signal This extension enhances semantic grounding and boosts performance in low-label regimes after full finetuning., while preserving fully vision-based inference Together, MVMAE and MVMAE-V2T demonstrate that structural and textual supervision can act as complementary forces: structural alignment provides robustness and transferability, and text guidance enriches semantic understanding Overall, these results establish a scalable framework for representation learning in medical imaging that leverages the inherent organization of clinical data The same principles naturally extend to other structured modalities, such as temporal follow-ups or multi-sequence imaging More broadly, the framework generalizes beyond medicine to any multimodal domain where structured relationships between views or modalities can serve as self-supervision for learning robust and transferable representations

How it works

The authors introduced MVMAE, the first multi-view masked autoencoder that leverages the study-level structure of clinical imaging data to learn robust and transferable representations without relying on textual inputs during pretraining By combining masked reconstruction with cross-view alignment, MVMAE effectively exploits the inherent multiview nature of radiology studies, yielding consistent improvements in performance and label efficiency across MIMIC-CXR, CheXpert, and PadChest Building upon this foundation, further proposed MVMAE-V2T, which integrates radiology reports as an auxiliary text-based learning signal This extension enhances semantic grounding and boosts performance in low-label regimes after full finetuning.

Improvements for AI systems

  1. Multi-view Masked Autoencoder (MVMAE) can learn view-invariant representations by combining masked image reconstruction with a study-level alignment objective, effectively converting clinical redundancy across projections into a powerful self-supervisory signal. This allows the model to capture geometric (anatomical and physiological) and semantic coherence across views that radiologists routinely exploit for interpretation.

  2. MVMAE-V2T can enhance semantic grounding by incorporating radiology reports as an auxiliary textbased learning signal during pretraining, which leads to clear gains under limited labeled data, without requiring text at inference. This extension allows the model to learn from structured, domain-specific supervision to guide the visual encoder toward disease-relevant semantics.

  3. The system can achieve superior label efficiency when finetuned on scarce data regimes because MVMAE consistently outperforms baselines in this setting, demonstrating more linearly separable representations and superior label efficiency compared to methods that do not use labels during pretraining.

  4. The model provides better-calibrated predictions than independent baselines because cross-view regularization encourages more reliable confidence estimates alongside improved discrimination, resulting in a lower Brier score among pretraining baselines under linear probing conditions on MIMIC-CXR and PadChest.

  5. Performance gains are maximized when leveraging study structure, as MVMAE with two views consistently achieves the highest performance across datasets, suggesting that multiview pretraining provides transferable benefits even when only one view is available.

Abstract

Building robust medical machine learning systems requires pretraining strategies that exploit the intrinsic structure present in clinical data. We introduce Multiview Masked Autoencoder (MVMAE), a self-supervised framework that leverages the natural multi-view organization of radiology studies to learn view-invariant and disease-relevant representations. MVMAE combines masked image reconstruction with cross-view alignment, transforming clinical redundancy across projections into a powerful self-supervisory signal. We further extend this approach with MVMAE-V2T, which incorporates radiology reports as an auxiliary text-based learning signal to enhance semantic grounding while preserving fully vision-based inference. Evaluated on a downstream disease classification task on three large-scale public datasets, MIMIC-CXR, CheXpert, and PadChest, MVMAE consistently outperforms supervised and vision-language baselines. Furthermore, MVMAE-V2T provides additional gains, particularly in low-label regimes where structured textual supervision is most beneficial. Together, these results establish the importance of structural and textual supervision as complementary paths toward scalable, clinically grounded medical foundation models.

Sources

Related papers