MMATR-X: High-Resolution Optical-SAR Joint Pretraining via Conditioned Alignment and Degraded-Target Prediction

arXiv:2604.16952 · cs.CV · Submitted 2026-04-18 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "MMATR-X: High-Resolution Optical-SAR Joint Pretraining via Conditioned Alignment and Degraded-Target Prediction".

Jane: Here's the gist of the paper. The gist The proposed method, CoDe-MAE, tackles representation degradation in high-resolution optical-SAR pretraining by pioneering a better synergy with less alignment philosophy through Conditioned Contrastive Learning and Cross-Modal Degraded Reconstruction.

Tom: The Challenge of Heterogeneity and Paradox High-resolution joint optical and synthetic aperture radar (SAR) pretraining faces the Heterogeneity-Resolution Paradox, where finer spatial scales drastically amplify the physical divergence between complex radar geometries and non-homologous optical textures <ref:2604.16952#pg2> The core motivation of joint pretraining is to achieve modality synergy that mutually enhances single-source representations, imbuing optical features with weather-robust structural geometries, and SAR features with fine-grained perceptual clarity <ref:2604.16952#pg4> This challenge is amplified because HR SAR magnifies the modality gap between optical and radar details <ref:2604.16952#pg3> CoDe-MAE Framework Components CoDe-MAE introduces a paradigm of better synergy with less alignment by employing three synergistic mechanisms <ref:2604.16952#pg4> 1) Optical-anchored Knowledge Distillation (OKD) implicitly regularizes SAR’s speckle noise by mapping it into a pure semantic manifold <ref:2604.16952#pg3> 2) Conditioned Contrastive Learning (CCL) utilizes cross-attention as a gradient buffer to align shared consensus while safely preserving divergent physical signatures <ref:2604.16952#pg4> 3) Cross-Modal Degraded Reconstruction (CDR) deliberately strips non-homologous spectral pseudo-features, truncating the inherently ill-posed mapping to capture true structural invariants <ref:2604.16952#pg3> Mechanism Analysis of Synergy The framework is designed to avoid the pitfalls of rigid alignment paradigms, which either force feature suppression or contamination <ref:2604.16952#pg4> Conditioned Contrastive Learning (CCL) shifts the alignment target from unconstrained raw embeddings to strictly conditioned embeddings via a dynamic buffering mechanism that acts as an information bottleneck <ref:2604.16952#pg4> Concurrently, Cross-Modal Degraded Reconstruction (CDR) breaks the illusion of completeness-driven recovery by formulating crossmodal reconstruction as a target-degraded optimization, which physically strips non-homologous pseudofeatures and prevents feature contamination <ref:2604.16952#pg3> Empirical Validation and Performance Extensive analyses validate the theoretical claims, demonstrating that CoDe-MAE successfully prevents representation degradation by preserving the highest effective feature rank <ref:2604.16952#pg4> Pretrained on 1M samples, CoDe-MAE establishes new state-of-the-art (SOTA) performance across diverse single and bi-modal downstream tasks, substantially outperforming foundation models scaled on vastly larger datasets <ref:2604.16952#pg4> The method's synergistic effect is confirmed by the fact that conditioning the contrastive space and degrading reconstruction targets are highly synergistic for decoupling shared semantics from distinct physical priors <ref:2604.16952#pg4> Key Contributions The main contributions include: 1) Mechanism Analysis of Representation Degradation, formally identifying the Heterogeneity-Resolution Paradox specifically within the context of HR optical-SAR pretraining <ref:2604.16952#pg4> 2) A Novel Pretraining Philosophy and Framework, proposing CoDeMAE as a generalized framework grounded in a pioneering better synergy with less alignment philosophy <ref:2604.16952#pg4> 3) Remarkable Data Efficiency and SOTA Performance, establishing new state-of-the-art (SOTA) performance across diverse single and dual-modal downstream tasks <ref:2604.16952#pg4> The paper concludes that by synergizing CCL and CDR, CoDe-MAE safely extracts shared structural consensus while truncating epistemic uncertainty, elegantly preventing representation degradation and preserving modality-unique priors <ref:2604.16952#pg4> The framework provides a rigorously grounded reference for tackling modality gaps in broader multi-modal visual representation learning.

Jane: Page 1 --- PREPRINT 1 Better with Less: Tackling Heterogeneous Multi-Modal Image Joint Pretraining via Conditioned and Degraded Masked Autoencoder Bowen Peng, Yongxiang Liu, Jie Zhou, Xiaodong Chen, Tianpeng Liu, Xiaogang Yu, Li Liu Abstract—Learning robust representations across extremely heterogeneous modalities remains a fundamental challenge in multimodal vision.

Tom: As a critical and profound instantiation of this challenge, high-resolution (HR) joint optical and synthetic aperture radar (SAR) pretraining seeks modality synergy to mutually enhance single-source representations; its potential is severely hindered by the Heterogeneity-Resolution Paradox: finer spatial scales drastically amplify the physical divergence between complex radar geometries and non-homologous optical textures.

Paper summary: Tom: We're talking about the "MMATR-X: High-Resolution Optical-SAR Joint Pretraining via Conditioned Alignment and Degraded-Target Prediction" paper. The thesis is that joint pretraining optical and SAR data is super hard because the fine spatial scales drastically amplify the physical difference between radar geometries and optical textures <ref:2604.16952#pg2>.

Jane: So, what’s the core problem they are trying to solve when they look at this paper?

Tom: The problem is that if you try rigid alignment, it either crushes one modality or lets noise from one mess up the other because the physical mechanisms are so different <ref:2604.16952#pg1>.

Lu: They propose a way out called CoDe-MAE, which is grounded in a better synergy with less alignment philosophy <ref:2604.16952#pg3>.

Meng: So they are trying to find a middle ground where you get the shared structural consensus without contamination from noise or mismatched physical priors? That’s what I need to understand for deployment <ref:2604.16952#pg3>.

Jane: Exactly. It suggests that instead of trying to make every single pixel look identical across modalities, you align on the important structural elements and let the unique physical details remain intact where they matter <ref:2604.16952#pg3>.

Tom: They introduce three parts: Optical-anchored Knowledge Distillation to regularize SAR noise, Conditioned Contrastive Learning to align consensus safely, and Cross-Modal Degraded Reconstruction to strip away the messy stuff that doesn't match physically <ref:2604.16952#pg2>.

Lu: They are using Optical-anchored Knowledge Distillation as an implicit speckle regularization tool to map noisy SAR inputs into a pure semantic manifold <ref:2604.16952#pg2>.

Meng: I’m curious about the degradation part. How do they stop the reconstruction from just guessing and getting weird artifacts?

Jane: They use Cross-Modal Degraded Reconstruction to deliberately strip non-homologous spectral pseudo-features, which truncates that ill-posed mapping to capture true structural invariants <ref:2604.16952#pg2>.

Tom: From my side of things, it’s the mechanics I like. They trained this thing on one million samples from a real optical-SAR corpus called OSPretrain-1M—and it actually showed new state-of-the-art performance across diverse single and dual tasks <ref:2604.16952#pg3>.

Meng: A million samples is still big, but showing that you don't need foundation models scaled on way larger datasets to get this level of performance makes sense for deployment. What about the cost of training this?

Lu: The paper shows that conditioning the contrastive space and degrading reconstruction targets actually work synergistically, which means they are decoupling shared semantics from those distinct physical priors very cleanly <ref:2604.16952#pg2>.

Jane: So, for someone just listening who wants to know why this matters in the real world, it means we can start building better tools for remote sensing and Earth observation that actually understand both the visual texture and the underlying radar structure simultaneously <ref:2604.16952#pg3>.

Conclusion: Tom: So, to recap, this paper is about taking high-resolution optical and SAR data and training them together using CoDeMAE, which uses Conditioned Contrastive Learning and Cross-Modal Degraded Reconstruction. This is what the title "MMATR-X: High-Resolution Optical-SAR Joint Pretraining via Conditioned Alignment and Degraded-Target Prediction" actually means.

Jane: It seems like the title itself is pretty descriptive—"High-Resolution Optical-SAR Joint Pretraining via Conditioned Alignment and Degraded Target Prediction." I wonder what that actually means for someone just listening who isn't a computer scientist.

Lu: Basically, they’re tackling the problem of getting optical and SAR features to work together when they are on the same fine spatial scale, which is really hard because the physics are so different <ref:2604.16952#pg3>.

Meng: So it’s not just putting them in one big box; it's a specific way of aligning them that tries to respect their differences instead of forcing them into the same mold <ref:2604.16952#pg3>.

Tom: Exactly. The core idea is this "better synergy with less alignment" thing, which they call CoDeMAE. It moves away from the old idea that you just need a rigid alignment relation for everything to work <ref:2604.16952#pg1>.

Jane: That sounds like it’s trying to find a middle ground, right? Because if you force them too much, one modality just gets crushed by the other's noise or its own specific physical quirks <ref:2604.16952#pg3>.

Lu: Yes. They propose three main parts: Optical-anchored Knowledge Distillation to regularize SAR noise, Conditioned Contrastive Learning to align consensus safely, and Cross-Modal Degraded Reconstruction to strip away the messy stuff that doesn't match physically <ref:2604.16952#pg2>.

Tom: From my side of things, it’s the mechanics I like. They trained this thing on one million samples from a real optical-SAR corpus—OSPretrain-1M—and it actually showed new state-of-the-art performance across various single and dual tasks <ref:2604.16952#pg3>.

Tom: It moves us closer to a future where AI doesn't just see an image or a radar picture; it understands the physical scene through both lenses at high resolution.

Lu: Yeah, it provides a solid framework for how we can handle those modality gaps in general visual representation learning, which is the bigger picture here <ref:2604.16952#pg3>.

Jane: So, they’ve shown that by using this careful balancing act between conditioning and degradation, we can safely extract what both sensors have to offer without getting bogged down in the noise.

Bowen Peng, Yongxiang Liu, Jie Zhou, Xiaodong Chen, Tianpeng Liu, Xiaogang Yu, Li Liu

National University of Defense Technology (NUDT) · Beijing Institute of Remote Sensing Information

cs.CV

Submitted: 2026-04-18

Updated: 2026-10-18

Code: https://github.com/scenarri/MMATR-X

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: The gist The proposed method, CoDe-MAE, tackles representation degradation in high-resolution optical-SAR pretraining by pioneering a better synergy with less alignment philosophy through Conditioned

Key concepts

Heterogeneity-Resolution Paradox: This paradox describes the problem where increasing image resolution makes the physical differences between optical images and SAR images much more pronounced. Finer details in high-resolution data highlight how different sensors capture vastly different physical properties, making it extremely difficult to find common features across modalities.
Conditioned Contrastive Learning (CCL): CCL uses cross-attention as a dynamic buffer to align representations. Instead of forcing all features into one space, it aligns them based on specific conditions while preserving unique physical details. This allows the model to find shared semantic understanding without erasing the distinct physical signatures of each sensor.
Cross-Modal Degraded Reconstruction (CDR): CDR addresses feature contamination by deliberately degrading reconstruction targets. It forces the model to reconstruct only true structural invariants by stripping away spectral pseudo-features that don't match across modalities. This prevents features from being corrupted by noise or non-homologous details.

Terminology

Contrastive Learning — 1× in this paper · explained in 7 episodes
Distillation — 1× in this paper · explained in 2 episodes
Downstream Tasks — 1× in this paper · explained in 2 episodes
Epistemic Uncertainty — 1× in this paper · explained in 12 episodes
Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
Knowledge Distillation — 1× in this paper · explained in 15 episodes

Summary

The gist The proposed method, CoDe-MAE, tackles representation degradation in high-resolution optical-SAR pretraining by pioneering a better synergy with less alignment philosophy through Conditioned Contrastive Learning and Cross-Modal Degraded Reconstruction.

The Challenge of Heterogeneity and Paradox

High-resolution joint optical and synthetic aperture radar (SAR) pretraining faces the Heterogeneity-Resolution Paradox, where finer spatial scales drastically amplify the physical divergence between complex radar geometries and non-homologous optical textures <ref:2604.16952#pg2> The core motivation of joint pretraining is to achieve modality synergy that mutually enhances single-source representations, imbuing optical features with weather-robust structural geometries, and SAR features with fine-grained perceptual clarity <ref:2604.16952#pg4> This challenge is amplified because HR SAR magnifies the modality gap between optical and radar details <ref:2604.16952#pg3>

CoDe-MAE Framework Components

CoDe-MAE introduces a paradigm of better synergy with less alignment by employing three synergistic mechanisms <ref:2604.16952#pg4>

Optical-anchored Knowledge Distillation (OKD) implicitly regularizes SAR’s speckle noise by mapping it into a pure semantic manifold <ref:2604.16952#pg3>
Conditioned Contrastive Learning (CCL) utilizes cross-attention as a gradient buffer to align shared consensus while safely preserving divergent physical signatures <ref:2604.16952#pg4>
Cross-Modal Degraded Reconstruction (CDR) deliberately strips non-homologous spectral pseudo-features, truncating the inherently ill-posed mapping to capture true structural invariants <ref:2604.16952#pg3>

Mechanism Analysis of Synergy

The framework is designed to avoid the pitfalls of rigid alignment paradigms, which either force feature suppression or contamination <ref:2604.16952#pg4> Conditioned Contrastive Learning (CCL) shifts the alignment target from unconstrained raw embeddings to strictly conditioned embeddings via a dynamic buffering mechanism that acts as an information bottleneck <ref:2604.16952#pg4> Concurrently, Cross-Modal Degraded Reconstruction (CDR) breaks the illusion of completeness-driven recovery by formulating crossmodal reconstruction as a target-degraded optimization, which physically strips non-homologous pseudofeatures and prevents feature contamination <ref:2604.16952#pg3>

Empirical Validation and Performance

Extensive analyses validate the theoretical claims, demonstrating that CoDe-MAE successfully prevents representation degradation by preserving the highest effective feature rank <ref:2604.16952#pg4> Pretrained on 1M samples, CoDe-MAE establishes new state-of-the-art (SOTA) performance across diverse single and bi-modal downstream tasks, substantially outperforming foundation models scaled on vastly larger datasets <ref:2604.16952#pg4> The method's synergistic effect is confirmed by the fact that conditioning the contrastive space and degrading reconstruction targets are highly synergistic for decoupling shared semantics from distinct physical priors <ref:2604.16952#pg4>

Key Contributions

The main contributions include:

Mechanism Analysis of Representation Degradation, formally identifying the Heterogeneity-Resolution Paradox specifically within the context of HR optical-SAR pretraining <ref:2604.16952#pg4>
A Novel Pretraining Philosophy and Framework, proposing CoDeMAE as a generalized framework grounded in a pioneering better synergy with less alignment philosophy <ref:2604.16952#pg4>
Remarkable Data Efficiency and SOTA Performance, establishing new state-of-the-art (SOTA) performance across diverse single and dual-modal downstream tasks <ref:2604.16952#pg4>

The paper concludes that by synergizing CCL and CDR, CoDe-MAE safely extracts shared structural consensus while truncating epistemic uncertainty, elegantly preventing representation degradation and preserving modality-unique priors <ref:2604.16952#pg4> The framework provides a rigorously grounded reference for tackling modality gaps in broader multi-modal visual representation learning.

--- Page 1 ---

PREPRINT 1

Better with Less: Tackling Heterogeneous Multi-Modal Image Joint Pretraining via Conditioned and Degraded Masked Autoencoder Bowen Peng, Yongxiang Liu, Jie Zhou, Xiaodong Chen, Tianpeng Liu, Xiaogang Yu, Li Liu Abstract—Learning robust representations across extremely heterogeneous modalities remains a fundamental challenge in multimodal vision. As a critical and profound instantiation of this challenge, high-resolution (HR) joint optical and synthetic aperture radar (SAR) pretraining seeks modality synergy to mutually enhance single-source representations; its potential is severely hindered by the Heterogeneity-Resolution Paradox: finer spatial scales drastically amplify the physical divergence between complex radar geometries and non-homologous optical textures. Consequently, migrating medium-resolution-oriented rigid alignment paradigms to HR scenarios triggers either severe feature suppression to force equivalence, or feature contamination driven by extreme epistemic uncertainty. Both extremes inevitably culminate in profound representation degradation and negative transfer. To overcome this bottleneck, we propose CoDe-MAE, pioneering a better synergy with less alignment philosophy. First, Optical-anchored Knowledge Distillation (OKD) implicitly regularizes SAR’s speckle noise by mapping it into a pure semantic manifold. Building on this, Conditioned Contrastive Learning (CCL) utilizes a gradient buffering mechanism to align shared consensus while safely preserving divergent physical signatures. Concurrently, Cross-Modal Degraded Reconstruction (CDR) deliberately strips non-homologous spectral pseudo-features, truncating the inherently ill-posed mapping to capture true structural invariants. Extensive analyses validate our theoretical claims. Pretrained on 1M samples, CoDe-MAE demonstrates remarkable data efficiency, successfully preventing representation degradation and establishing new state-of-the-art performance across diverse single- and bi-modal downstream tasks, substantially outperforming foundation models scaled on vastly larger datasets. Code and weights: https://github.com/scenarri/CoDeMAE. Index Terms—Self-supervised learning, remote sensing, multimodality learning, masked autoencoders I. INTRODUCTION In multi-modal self-supervised learning (SSL) [1]–[3], balancing the capture of modality-shared invariance with the preservation of modality-unique priors for complementary synergy poses a fundamental dilemma. This bottleneck is drastically amplified when confronting inherently non-isomorphic sensors. Serving as a critical and severe instantiation of such extreme physical divergence, joint optical and synthetic aperture radar Bowen Peng, Yongxiang Liu, Jie Zhou, Xiaodong Chen, Tianpeng Liu, and Li Liu are with the College of Electronic Science and Technology, National University of Defense Technology (NUDT), Changsha 410073, China. Email: pbow16@nudt.edu.com, lyx bible@sina.com, zhoujie @nudt.edu.cn, cxd@nudt.edu.com, everliutianpeng@sina.cn, liuli nudt@nudt.edu.cn Xiaogang Yu is with the Beijing Institute of Remote Sensing Information, Beijing 100192, China Email: 13366351343@189.cn This work was supported by the National Natural Science Foundation of China (NSFC) under Grant 62376283 and 62531026, the Innovation Research Foundation of National University of Defense Technology (JS2023-03), and the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (JYB2025XDXM110). Corresponding authors: Yongxiang Liu and Li Liu. Fig. 1. The Heterogeneity-Resolution Paradox. Optical and SAR sensors observe the Earth through fundamentally distinct physical mechanisms. This inherent heterogeneity, quantified here by the Structural Similarity (SSIM) between image pairs, drastically amplifies at finer spatial scales <ref:2604.16952#pg2> The equivalent resolution starts from the original 0.5m ground sample distance (GSD) provided by MSAW dataset [7], with coarser scales generated via Gaussian Pyramid downsampling following Scale-MAE [8] For SAR imagery, these scales represent the resampled pixel spacing consistent with the optical grids. radar (SAR) pretraining for Earth observation represents a vital frontier [4]–[6]. It seeks to learn superior representations that empower both single- and dual-modal downstream tasks. However, unlocking this potential is severely hindered by profound heterogeneity (Fig. 1), characterized by two coupled challenges: 1) Physical Divergence vs. Modality Synergy: Optical sensors capture rich spectral textures and colors but are highly susceptible to cloud interference. Conversely, SAR records dielectric structures, offering all-weather visibility, yet suffers from inherent speckle noise and lacks fine-grained semantic clarity. The core motivation of joint pretraining is to achieve modality synergy that mutually enhances single-source representations, imbuing optical features with weather-robust structural geometries, and SAR features with fine-grained perceptual clarity. However, their fundamentally distinct physical mechanisms create a profound semantic gap that makes explicit cross-modal interaction intrinsically difficult <ref:2604.16952#pg2> 2) The Heterogeneity-Resolution Paradox: Transitioning to high-resolution (HR) pretraining is strictly necessary for fine-grained spatial understanding and object-level recognition. Yet, finer spatial scales drastically amplify the aforementioned modality gap <ref:2604.

Improvements for AI systems

Embedded Knowledge Distillation via OKD for Speckle Regularization: The system can map SAR’s complex multiplicative noise into a pure semantic manifold by leveraging optical data as an Optical-anchored Knowledge Distillation (OKD) baseline, implicitly regularizing the SAR input without forcing destructive explicit alignment.
Dynamic Gradient Buffering via Conditioned Contrastive Learning: The system can selectively align shared consensus while safely preserving divergent physical signatures by using cross-attention as a gradient buffer to ensure that the alignment mechanism does not trigger severe feature suppression.
Truncated Uncertainty via Cross-Modal Degraded Reconstruction: The system can avoid completeness-driven recovery and prevent feature contamination by predicting targets based on a loss function like LCDR, which explicitly strips non-homologous spectral pseudo-features to align only on structural invariants.