VesperMUSE: Multi-Modal Fusion Vision Transformer for Image Semantic Segmentation Under Diverse Light Conditions

Comprehensive Research Summary of VesperMUSE: A Unified Multi-Modal Framework for Robust Low-Light Semantic Segmentation

VesperMUSE is a novel, unified multi-modal framework specifically engineered to achieve robust image semantic segmentation across a wide spectrum of illumination conditions, addressing the severe performance degradation typically observed in low-light environments due to noise, artifacts, and color distortion. The core philosophy of VesperMUSE is to tackle the light deficiency problem through a multi-layered strategy: synthetic adaptation, thermal-guided enhancement, and sophisticated multi-modal fusion.

Architectural Overview and Core Components

The framework operates across four distinct stages:

1. Low-Light Synthetic Adaptation: This initial stage employs a novel physics-informed pipeline designed to generate realistic degradations that accurately model natural exposure variations, characteristic noise profiles, and color distortions inherent in low-light scenes.

2. Low-Light Image Enhancement (LLIE): The adapted low-light maps are then processed using thermal data within a supervised approach to predict the original RGB image map. This module is architecturally defined by a U-shaped encoder-decoder structure incorporating a Thermal-aware Transformer Block, which features an RGB-Thermal Cross-Attention mechanism coupled with a Feed Forward Network. The enhancement quality is optimized via a weighted hybrid loss function: LLIE = alpha 1L 1 + alpha 2L perc + alpha 3L SSIM + alpha 4L sem.

3. Surface Normal Map Generation: To enrich the input modalities, surface normal maps are derived from a monocular depth estimation model (specifically DepthAnything V2) applied to the main RGB modality, followed by gradient extraction.

4. Fusion-based Semantic Segmentation: The final stage involves a powerful Fusion Vision Transformer that processes features from the primary RGB stream alongside auxiliary inputs (thermal and surface normals). This module utilizes a highly scalable, modality-agnostic self-attention mechanism capable of extracting both global context and fine local features from an arbitrary number of auxiliary inputs. The encoding phase strategically employs asymmetric backbone selection—using ResNet50 for the RGB stream and ResNet18 for auxiliary streams—to balance representational power with computational efficiency.

The key mathematical formulations underpinning the process include:

• Enhanced RGB Map: RGB = VesperMUSELLIE(IRGB, I Thermal), where RGB in R H times W times 3.

• Final Output Feature: RGB = conv1 times 1(F final), where RGB in R H times W times 3.

Key Technical Innovations and Modality Handling

The central innovation lies in the Fusion Self-Attention Module, which is designed to be truly modality-agnostic. This mechanism allows the network to dynamically treat one input (e.g., RGB) as primary while treating all others (thermal, normals) as auxiliary inputs, effectively supporting an arbitrary number of modalities seamlessly. This approach leverages cross-attention to resolve ambiguities and fuse complementary information effectively.

Furthermore, VesperMUSE incorporates critical geometric priors by explicitly modeling depth via DepthAnything V2, which justifies the increased computational load by providing essential structural context for segmentation tasks.

Empirical Performance and Contributions

Extensive evaluations confirm that VesperMUSE delivers state-of-the-art (SOTA) performance in challenging scenarios:

• CART Dataset: The framework establishes a new benchmark on the CART dataset, achieving an mIoU of 0.7672, significantly outperforming previous leading methods such as CM-SSM [73] (0.7460) by 2.1%.

• PST900 Dataset: On the PST900 dataset, VesperMUSE achieves a competitive mIoU of 0.8559, setting new accuracy benchmarks for task-critical objects and showing a 1.4% improvement over LASNet [65].

• LLIE Efficiency: The LLIE module demonstrates significant performance gains, achieving a 29.5% PSNR increase over LYT-Net [47] and a 2.3% increase over Retinexformer [50] when tested on synthetically adapted CART data.

Limitations and Future Directions

Despite its success, the research identifies specific areas for future refinement:

1. Synthetic Adaptation Robustness: The synthetic low-light adaptation algorithm exhibits strong degradation under certain conditions, suggesting a need for further tuning of this pipeline.

2. LLIE Output Colour Imbalance: A noticeable limitation is the potential color imbalance in the output of the LLIE module, which has been addressed by proposing solutions such as integrating a dedicated color loss term to decouple intensity from color information.

Future Development Trajectory:

The subsequent development of VesperMUSE is strategically aimed at simultaneously enhancing performance and reducing computational complexity. This involves exploring:

• Backbone Optimization: Adopting more efficient backbone architectures that may surpass the current ResNet-based choices.

• Modality-Specific Backbones: Investigating the use of dedicated backbones for thermal and structural inputs, as these are expected to yield superior results compared to the current convolutional adaptation strategy.

• Representation Alternatives: Exploring alternative image representations, such as operating in different color spaces or explicitly decomposing inputs into low- and high-frequency components within the LLIE module, which promises a more principled treatment of light irradiation.

Broader Context and Significance

Beyond pure accuracy metrics, the paper emphasizes the societal implications of robust perception systems. The framework is positioned to establish a foundation for reliable semantic segmentation in critical environments—such as safety-sensitive missions or survivor identification—where system failure has profound consequences. Therefore, advancing DL approaches in this domain is not merely about incremental accuracy gains but about fostering trust, robustness, and accountability in systems operating under uncertainty. The inherent high computational requirements are mitigated by demonstrating a competitive performance-to-complexity trade-off through careful architectural choices.

1. Real-world physics-informed synthetic low-light adaptation pipeline for data generation adds robustness by computing a cumulative illumination distribution using the Y channel from multiple datasets (LOL [32], LOL-v2 real [33], etc.) and matching the illumination map to this distribution, ensuring the generated images faithfully reproduce the deficiencies of natural low-light environments.

2. Thermal-guided Low-Light Image Enhancement (LLIE) module optimizes downstream segmentation by integrating thermal representations, as demonstrated by the finding that VesperMUSE LLIE demonstrates highly competitive performance, attaining 27.47 dB/0.870 on PST900 LL and 27.74 dB/0.858 on CART LL.

3. Modality-agnostic Fusion Vision Transformer enables scalable scene understanding by supporting an arbitrary number of inputs, as the architecture allows it to selectively attend to relevant cross-modal cues without requiring any architectural modifications when different sensor configurations are used.

4. Comprehensive evaluation across multiple modalities provides a more informative segmentation framework by utilizing auxiliary modalities like thermal and surface normals, which contribute an average of ≈ 3% performance deficit when the surface normal is not used and a average improvement of ≈ 5% for both modalities on the PST900 dataset.

5. Enhanced semantic segmentation performance on challenging outdoor environments by achieving a new state-of-the-art mIoU of 0.7672 on the CART dataset [25], specifically showing superior semantic preservation of dynamic obstacles, surpassing recent baselines like MCNet-T [83] on the highly challenging Person class (0.3426 vs. 0.3150).