FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities

arXiv:2610.21779 · cs.CV · Submitted 2026-10-15 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities".

Jane: The gist The proposed framework FuseAdapt separates modality-private attention adaptation from cross-modal exchange through FFN low-rank residuals and complements this design with target-conditioned missing-modality latent prediction to improve both full…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, looking at "FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities," the authors are proposing this parameter-efficient framework that fuses information within the adaptation space of a frozen vision foundation model.

Jane: What they're saying is that this approach explores whether the foundation model's existing priors can actually serve as an effective shared backbone for multi-modal segmentation.

Lu: Essentially, they move away from adding standalone fusion networks and instead use lightweight adaptation branches for each modality and adaptation-space fusion by keeping the query and value low-rank adaptations private to each modality.

Meng: And the whole mechanism relies on routing only FFN low-rank residual activations across modalities inside that frozen backbone to get cross-modal exchange happening.

Tom: And they handle missing data with Predictive Modality Modeling, which is a direct representation-level objective for cross-modal redundancy during training, where they mask a target modality and predict its embedding from the visible ones.

Jane: The main implication here is that this method can significantly improve segmentation performance even when some sensors are unavailable at test time.

Lu: It suggests that representation learning from large foundation models can be used to build a backbone capable of handling the complexities of multi-modal input robustly.

Tom: So, in simple terms, they separated how each modality adapts privately from how they exchange information through those low-rank residuals and PMM complements this design with target-conditioned missing-modality latent prediction.

Jane: That’s a concise way to put it. The paper shows that this approach yields performance improvements across three benchmarks, MCubeS, DELIVER, and MUSES for both full and missing modality segmentation with a small trainable budget.

Conclusion: Tom: So, FuseAdapt is this parameter-efficient way to fuse multiple sensors when some of them are missing at test time. It’s been about a few weeks since this paper came out, and I want to talk about what it actually does.

Jane: It’s really smart because instead of building a huge fusion network, they keep the main vision model frozen and only train small "adaptation branches" for each sensor input. The core idea is that the foundation model already understands vision really well, so we just fine-tune it slightly to handle different inputs.

Lu: Exactly. They are doing this "adaptation-space fusion," which means they keep the query and value projections low-rank adaptation private to each modality, and then they only route FFN low-rank residual activations across modalities inside that frozen backbone. It’s keeping the heavy lifting in the frozen model while learning how to mix them intelligently.

Meng: From a deployment standpoint, that sounds promising for efficiency because we aren't adding a massive new module to run constantly. But what about the missing sensor part? How does it actually handle when one of those inputs is gone during real-time operation?

Jane: That’s where Predictive Modality Modeling, or PMM, comes in. During training, they mask one target modality and make the model predict that modality's embedding using only the ones that are visible. It’s a way to build redundancy directly into the representation learning process.

Tom: So it’s a direct way to learn cross-modal relationships even when data is missing, which is much better than relying on a separate fusion module that might just break if one stream goes dark. The results they show—up to plus three point zero one mIoU under full modalities and twelve point five six mIoU when modalities are missing—they are pretty strong.

Jane: That improvement in the missing modality case is really what caught my attention. It shows that this approach doesn't just work on perfect data; it actually gets better when things go wrong, which is crucial for real-world applications like autonomous driving or robotics.

Lu: The ablation studies confirm that routing those adaptation exchanges across the FFNs consistently improves performance, so it’s not just random mixing; the model learns *how* to mix based on what's available. It confirms the idea that different modalities have distinct roles in feature refinement.

Meng: I'm still watching how much train time this takes versus a standard multi-branch encoder setup. If the adaptation paths are too complex, it could slow down our inference pipeline at scale. What is the actual parameter count increase we’re looking at for these adaptation branches?

Jane: They keep the trainable budget small, which is the key word here. They show that you get significant gains in segmentation performance without needing a massive amount of new parameters to train or store. It’s very focused on being practical for existing vision foundation models.

Tom: So, looking at the title again—FuseAdapt—it really captures the essence: it’s parameter-efficient fusion that specifically tackles missing modalities. It moves away from brute-force fusion and instead uses smart adaptation within a frozen backbone, and PMM handles the missing data gracefully during inference.

Jane: Right. What this paper means for us is that we don't always need a completely new, massive architecture just to handle multi-sensor input; we can adapt existing powerful models in a much more targeted way to make them robust against real-world sensor failures. (Sound of gentle radio static swelling slightly)

Tom: This leads us perfectly into how this works technically, so let's see exactly how they built those adaptation branches next.

Xin Zhang, Robby T. Tan

National University of Singapore

cs.CV

Submitted: 2026-10-15

Updated: 2026-10-15

Comments: 25 pages, 6 figures, 19 tables. Accepted to NeurIPS 2026

Project page: https://devinxzhang.github.io/FuseAdapt-Page

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 85/100

The gist: The gist The proposed framework FuseAdapt separates modality-private attention adaptation from cross-modal exchange through FFN low-rank residuals and complements this design with target-conditioned

Key concepts

Adaptation Space Fusion
This method performs fusion inside the adaptation space of a frozen vision foundation model (VFM). Instead of adding new fusion layers, it learns modality-specific adjustments (like low-rank updates to Q/V projections) and routes only FFN low-rank residual activations across modalities. This keeps the core VFM fixed while learning how to combine different inputs effectively.
Predictive Modality Modeling (PMM)
PMM is a training technique designed for missing modalities. During training, it masks a target modality and forces the model to predict its embedding using only the visible modalities. This prediction is supervised by a 'teacher' target computed when all modalities are present, creating a redundancy that helps the model handle incomplete sensor data during testing.
Low-Rank Adaptation (LoRA) in Fusion
The framework keeps Query (Q) and Value (V) low-rank adaptation private to each modality. This means only small, trainable updates are applied to these projections for each input stream. Cross-modal interaction is then achieved by routing only the FFN low-rank residual activations across modalities, ensuring that unavailable sources are masked before they affect the final output.
Missing Modality Segmentation (MISS)
This refers to testing performance when one or more sensor modalities are absent during inference. FuseAdapt is specifically designed to improve segmentation accuracy in this 'missing modality' scenario, achieving significant gains over prior methods by leveraging the learned cross-modal interactions and the PMM predictive modeling objective.

Terminology

Summary

The gist The proposed framework FuseAdapt separates modality-private attention adaptation from cross-modal exchange through FFN low-rank residuals and complements this design with target-conditioned missing-modality latent prediction to improve both full and missing modality segmentation performance

How it works

FuseAdapt performs fusion within the adaptation space of a frozen vision foundation model (VFM) by assigning each modality a lightweight adaptation branch and performing adaptation-space fusion by keeping Q/V lowrank adaptation private to each modality and routing only FFN low-rank residual activations across modalities inside the frozen backbone The pretrained VFM remains fixed while modality-specific corrections and explicit cross-modal interaction are learned through the adaptation paths

  1. Self-Attention Adaptation involves adding trainable low-rank updates only to the query and value projections for each modality, keeping the key and output projections frozen and shared

  2. Cross-modal exchange is introduced in the FFN adaptation path where a receiver-conditioned router mixes only FFN low-rank residual activations across modalities, ensuring that unavailable sources are masked before normalization

Predictive Modality Modeling (PMM)

To handle missing modalities, FuseAdapt introduces Predictive Modality Modeling (PMM), which is a direct representation-level objective for cross-modal redundancy During training, PMM masks a target modality and predicts its embedding from the visible ones, supervised by a teacher target computed under full-modality input The PMM loss is defined as Lpmm = 1 − (1/N) Σ X i D(zˆ(i)t, sg[y(i)t]) where D(·, ·) denotes cosine similarity

Experimental Setup and Results

FuseAdapt was evaluated across three multi-modal segmentation benchmarks: MCubeS, DELIVER, and MUSES under both modality-complete semantic segmentation (MCSS) and modality-incomplete semantic segmentation (MISS) protocols FuseAdapt consistently improves both modality-complete and modality-incomplete segmentation with a small trainable budget, yielding up to +3.01 mIoU under full modalities and +12.56 mIoU under missing modalities over prior state-of-the-art

(MCSS Results)

On MCubeS, FuseAdapt obtains 57.62 mIoU with all four modalities (RADN) and 56.30 mean mIoU, compared with 53.29 for StitchFusion and 52.43 for MAGIC

(MISS Results)

On MCubeS MISS, FuseAdapt achieves 52.89 mean mIoU, outperforming prior state-of-the-art PMM exceeds output KD by 3.05 mean mIoU and fused-feature KD by 2.32

Ablation and Controls

Ablation studies confirm that FuseAdapt performs best when both modality-specific adaptation and routed adaptation exchange are included, as Full FuseAdapt performs best in both tables The routing fusion in FFNs consistently improves performance, demonstrating that learned routing effectively captures the distinct roles that different modalities play in multi-modal feature refinement

Efficiency and Robustness

FuseAdapt's three-input latency is close to CMNeXt and below StitchFusion in this profile, while its unique deployed parameter count is smaller than both The method maintains strong performance across all backbones when tested with EVA-CLIP-B and NeCo-B on MCubeS The shift experiment shows that the Query prediction has the smaller drop in all 12 seed–target comparisons, indicating sensitivity of latent prediction to these translations

The paper concludes that FuseAdapt separates modality-private attention adaptation from cross-modal exchange through FFN low-rank residuals and PMM complements this design with target-conditioned missing-modality latent prediction The main segmentation comparisons are reported from single runs, without a statistical-significance claim The inference cost still grows with the number of sensor streams, and deployment in robotics or driving requires application-specific testing, failure detection [Chen et al., 2025b], uncertainty estimation [Chen et al., 2026], and validated fallback mechanisms

The paper is supported by the Ministry of Education, Singapore, under its MOE AcRF Tier 3 Grant (MOE-MOET32022-0001) The references include works on multi-modal semantic segmentation [Huang et al., 2022], parameter-efficient adaptation [Lester et al., 2021], and predictive modeling [Assran et al., 2023].

--- Page 1 ---

FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities

for Semantic Segmentation with Missing Modalities Xin Zhang National University of Singapore x.zhang@u.nus.edu Robby T. Tan National University of Singapore robby.tan@nus.edu.sg Abstract Multi-modal semantic segmentation benefits from complementary sensors, but this benefit is fragile when some modalities are degraded or missing at test time Many existing methods rely on dedicated fusion modules or multi-branch cross-modal encoders, and the fusion often becomes unreliable when modalities are absent, leading to substantial performance drops We propose FuseAdapt, a parameterefficient framework that performs fusion within the adaptation space of a frozen vision foundation model (VFM), exploring whether VFM priors can serve as an effective shared backbone for multi-modal segmentation Instead of attaching a standalone fusion network, FuseAdapt assigns each modality a lightweight adaptation branch and performs adaptation-space fusion by keeping Q/V lowrank adaptation private to each modality and routing only FFN low-rank residual activations across modalities inside the frozen backbone The pretrained VFM remains fixed while modality-specific corrections and explicit cross-modal interaction are learned through the adaptation paths To handle missing modalities, we further introduce Predictive Modality Modeling (PMM), a direct representationlevel objective for cross-modal redundancy During training, PMM masks a target modality and predicts its embedding from the visible ones, supervised by a teacher target computed under full-modality input PMM is used only during training and adds no inference-time overhead Across three multi-modal segmentation benchmarks (MCubeS, DELIVER, MUSES), FuseAdapt consistently improves both modality-complete and modality-incomplete segmentation with a small trainable budget, yielding up to +3.01 mIoU under full modalities and +12.56 mIoU under missing modalities over prior state-of-the-art Project page: https://devinxzhang.github.io/FuseAdapt-Page/. 1 Introduction Scene understanding often benefits from combining multiple sensing modalities [Huang et al., 2022, Jia et al., 2023, Park et al., 2017, Zhang et al., 2018] RGB [Wei et al., 2024, Zhang and Tan, 2025, Zhang et al., 2018], depth [Cao et al., 2021, Song et al., 20:36], LiDAR [Yan et al., 20:36], and event streams [Xia et al., 20:36] provide complementary cues about appearance, geometry, and motion In practice, however, this multi-sensor benefit is fragile: a modality may be noisy [Yang et al., 2026a,b], degraded, spatially imperfect, or unavailable at inference time Such cases are common in autonomous driving [Caesar et al., 20:36] and embodied settings [Wang et al., 20:36], where illumination, weather [Du et al., 2026, Teng et al., 20:5], texture, sensor failure, or calibration conditions [Yuan et al., 20:5] can affect sensors unequally A practical multi-modal segmentation model should therefore work in two settings: modality-complete semantic segmentation (MCSS), where all modalities are available and should be fused effectively, Preprints. arXiv:2610.21779v1 [cs.CV] 15 Oct 2026

--- Page 2 ---

Figure 1: Comparison of multi-modal fusion strategies. (a) Separate encoders with mid-level fusion can replicate backbone computation across modalities (b) Shared-encoder or late-fusion designs reduce this cost, but can make fusion depend strongly on the dominant or cleanest modality and modality-incomplete semantic segmentation (MISS), where only a subset is available and the model should remain reliable. Existing multi-modal segmentation methods often trade off modality-specific modeling, compactness, and robustness Separate-encoder designs fuse intermediate features from different modalities [Broedermann et al., 2023, Zhang et al.

Improvements for AI systems

  1. Bold Header: Separation of Private Attention Adaptation from Cross-Modal Exchange

FuseAdapt separates modality-private Q/V adaptation from receiver-conditioned crossmodal exchange restricted to FFN low-rank residual activations, which allows each modality to maintain sensor-specific correction while enabling controlled information flow across modalities.

  1. Bold Header: Robust Missing Modality Prediction via Predictive Modality Modeling (PMM)

PMM train[s] the model to predict a masked modality representation from visible modalities, supervised by a teacher target computed with full-modality context, allowing the system to maintain reliability when sensors are unavailable by learning cross-modal redundancy.

  1. Bold Header: Adaptive Cross-Modal Feature Refinement via Routing Fusion

The framework implements cross-modal exchange implemented by routing FFN low-rank residual activations across modalities, where routing is normalized over the available sensors, enabling the model to dynamically select and combine modality-dependent features based on context.

  1. Bold Header: Enhanced Robustness Under Spatial Shifts

The query predictor in PMM shows superior performance in spatial shifts, as it exhibits a 32.2 times smaller drop compared to per-token regression when shifting visible sensors relative to the clean full-context EMA target, demonstrating resilience to misalignment.

  1. Bold Header: Parameter-Efficient Adaptation and Scalability

FuseAdapt utilizes lowrank adaptation private Q/V and keeps the VFM frozen, resulting in a small trainable budget where the relative overhead on parameters, memory, and computation remains very small (e.g., 1.12% of full model parameters for DINOv2-S).

Abstract

Multi-modal semantic segmentation benefits from complementary sensors, but this benefit is fragile when some modalities are degraded or missing at test time. Many existing methods rely on dedicated fusion modules or multi-branch cross-modal encoders, and the fusion often becomes unreliable when modalities are absent, leading to substantial performance drops. We propose FuseAdapt, a parameter-efficient framework that performs fusion within the adaptation space of a frozen vision foundation model (VFM), exploring whether VFM priors can serve as an effective shared backbone for multi-modal segmentation. Instead of attaching a standalone fusion network, FuseAdapt assigns each modality a lightweight adaptation branch and performs adaptation-space fusion by keeping Q/V low-rank adaptation private to each modality and routing only FFN low-rank residual activations across modalities inside the frozen backbone. The pretrained VFM remains fixed while modality-specific corrections and explicit cross-modal interaction are learned through the adaptation paths. To handle missing modalities, we further introduce Predictive Modality Modeling (PMM), a direct representation-level objective for cross-modal redundancy. During training, PMM masks a target modality and predicts its embedding from the visible ones, supervised by a teacher target computed under full-modality input. PMM is used only during training and adds no inference-time overhead. Across three multi-modal segmentation benchmarks (MCubeS, DELIVER, MUSES), FuseAdapt consistently improves both modality-complete and modality-incomplete segmentation with a small trainable budget, yielding up to +3.01 mIoU under full modalities and +12.56 mIoU under missing modalities over prior state-of-the-art. Project page: https://devinxzhang.github.io/FuseAdapt-Page/.

Sources

Related papers