Source-Learned Reliance for Selective Test-Time Adaptation of Multimodal Time Series

The goal is to synthesize these excerpts into a comprehensive, high-fidelity summary that captures the core methodology, key contributions, and empirical validation of the paper.

Here is the detailed synthesis:

Comprehensive Research Summary: CARAT (Conditional Modality Composition and Reliance-Guided Attenuation for Efficient Test-Time Adaptation)

Overview and Core Proposal:

The paper introduces CARAT, a novel multimodal test-time adaptation (TTA) framework specifically designed for efficiency. The central innovation of CARAT lies in its ability to intelligently combine two critical pieces of information: the current input abnormality detected at test time, and a source-learned modality reliance profile derived during the source training phase. This allows CARAT to guide the adaptation process—either by selectively omitting (dropping) suspect modalities or attenuating (downweighting) them—without resorting to computationally expensive online candidate-subset evaluations.

Methodology and Key Contributions:

CARAT achieves this efficiency by amortizing reliance estimation through source training. During this phase, the backbone is trained not only to perform its primary task but also as a byproduct, to predict outputs even when certain input streams are missing. This process yields a measure of how much the backbone relies on each individual modality.

The key technical contributions are twofold:

1. Reliance Estimation via Asymmetric Curriculum: The framework employs an asymmetric curriculum for missingness robustness and reliance estimation. It modulates modality-dropout probabilities using windowed input-projection gradient norms. This mechanism prepares the backbone to handle missing inputs and derives a frozen, backbone-specific reliance proxy that serves as the guidance signal for adaptation.

2. Guidance Mechanism: At test time, a lightweight detector flags suspect streams based on features like FITS reconstruction-error statistics and log-spectral-distance (LSD) features relative to clean source statistics computed during training. The learned reliance profile dictates the intervention:

• If unflagged modalities retain sufficient reliance mass (beta i at least nu), the sample is routed to modality-composition adaptation (omission—replacing the suspect stream with a missingness symbol).

• Otherwise, it is routed to latent-attenuation adaptation (downweighting representations via learnable scalars).

This routing decision is governed by a deployment-time lever, nu, which allows researchers to trade off efficiency against conservatism: smaller nu favors aggressive omission (dropping), while larger nu favors attenuation, preserving partially informative streams at a higher computational cost.

Empirical Validation and Performance:

The performance of CARAT has been rigorously validated across diverse experimental settings, demonstrating superior results compared to existing baselines.

• Superior Performance: CARAT achieves the highest overall macro-F1 and best mean rank (2.42) across 12 equally weighted dataset–backbone settings, significantly exceeding EATA (the strongest baseline) by 1.58 F1 points.

• Computational Efficiency: Crucially, CARAT is highly efficient. It utilizes 9.49% fewer GFLOPs and updates 47.82% fewer parameters than EATA across five profiled configurations, confirming its promise for deployment-time adaptation where computational constraints are significant.

• Robustness Across Conditions: The framework shows nuanced performance depending on the corruption type: it exhibits its lowest average rank under waveform anomalies and its highest under frequency attenuation.

• Dataset Specificity: Performance varies by dataset; CARAT achieves its lowest average rank on DSADS, DaliaHAR, and PAMAP2, whereas EATA performs best on WESAD.

Structural Insights and Future Directions:

The research provides valuable structural insights into multimodal TTA: it suggests that the separation of sensor abnormality detection from backbone reliance estimation is key to achieving competitive performance across different data characteristics (e.g., IMU-dominated vs. heterogeneous datasets). Furthermore, empirical checks confirm that unconditional dropping of suspect streams is insufficient; the reliance-guided choice between omission and attenuation is necessary for optimal results.

Future research directions are clearly outlined, focusing on refining the frozen prior (slow-timescale refresh), developing uncertainty-aware adaptation strategies, improving suspect-set selection, and validating the framework under more complex natural failure scenarios.

**In summary, CARAT represents a significant advancement in efficient multimodal TTA by intelligently leveraging source training to create a modality reliance prior. This prior is then dynamically combined with runtime corruption detection to make informed decisions on whether to omit or attenuate suspect streams, resulting in state-of-the-art performance metrics coupled with substantial computational savings compared to established baselines.

1. Reliance Estimation at Training Time for Omission Guidance: The CARAT mechanism allows for reliance estimated at training time, corruption detected at test time, which guides adaptation by using a frozen, backbone-specific reliance proxy. This proxy is derived from asymmetric modality-dropout curriculum using windowed input-projection gradient norms to produce a profile that determines the intervention: either replacing suspect streams with the missingness symbol or attenuating their representations.

2. Adaptive Omission/Attenuation Routing Based on Source Reliance: The system implements a decision rule where the share of the reliance profile assigned to unflagged modalities determines whether to replace the entire suspect set with the trained missingness symbol or attenuate its encoder outputs. This replaces unrestricted retain/remove choices admit 2 K subsets with a single, guided choice based on learned prior knowledge, as summarized by the formula: The share of the training-time reliance budget assigned to the unflagged modalities of sample i is βi = P j /∈Si ρ j Ml=1 ρl ∈ [0, 1].

3. Model-Specific Reliance Profiling for Heterogeneous Data: The architecture allows for backbones like MAESTRO, ShaSpec, and Flex-MoE to learn distinct reliance profiles across different sensing regimes. For instance, the paper notes that on heterogeneous wearable datasets such as WESAD and DaliaHAR, reliance distributions are highly non-uniform and strongly backbone-dependent, enabling the system to tailor its omission strategy based on whether it is running on a heterogeneous dataset or an IMU-dominated homogeneous dataset.

4. Efficiency Gains via Parameter Reduction: CARAT achieves significant computational savings by utilizing the source training information, as it uses 9.49% fewer GFLOPs and updates 47.82% fewer parameters than EATA, positioning it as a practical default to wearable TTA with modest computational requirements.

5. Detector-Agnostic Corruption Detection: The suspect modality detection is decoupled from the adaptation logic, as the framework uses a lightweight one-class detector that computes a corruption score using two complementary feature families (FITS reconstruction-error statistics and log-spectral-distance features), ensuring that CARAT remains agnostic to the specific detector choice.

Multimodal wearable systems must remain reliable when sensor streams become noisy or unavailable. Existing multimodal test-time adaptation (TTA) methods often assess reliability online, but cross-modal agreement can be misleading when sensors measure different physical processes, and evaluating alternative modality configurations adds inference cost. We propose CARAT, which decouples model reliance from runtime corruption detection to guide omission or attenuation, amortizing reliance estimation through source training. An asymmetric modality-dropout curriculum prepares a missingness-resilient backbone for omission and derives a frozen, backbone-specific reliance proxy from windowed input-projection gradient norms. At deployment, a lightweight one-class detector flags suspect streams, and the proxy guides a joint choice between replacing the suspect set with the backbone's trained missingness symbol and attenuating its representations before fusion, without candidate-subset evaluation. Across four wearable datasets, five corruption types, three backbones, and eight TTA baselines, CARAT achieves the highest overall macro-F1 and best mean rank (2.42), exceeding EATA, the strongest baseline, by 1.58 F1 points across 12 equally weighted dataset-backbone settings. Across five profiled configurations, CARAT uses 9.49% fewer GFLOPs and updates 47.82% fewer parameters than EATA. A pattern also emerges across sensing regimes: multimodal TTA methods such as PTA are competitive on IMU-dominated homogeneous datasets, whereas unimodal TTA methods like TENT and EATA match or exceed it on heterogeneous datasets. These results position CARAT as a practical default to wearable TTA, offering competitive robustness with modest computational requirements and benefits that vary across backbones and dataset regimes.