When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependence

arXiv:2606.26473 · cs.LG, eess.AS · Submitted 2026-06-25 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependence".

Jane: Here's the gist of the paper. The gist The authors propose a leakage-safe diagnostic to test whether quality signals influence model decisions during inference by permuting reliability scores while keeping other factors fixed, finding that native quality estimates do not measurably change fused predictions across tested multimodal systems.

Tom: Decision-level identifiability diagnostic For each cross-validation fold, the diagnostic asks a targeted question: after a multimodal model has been trained, do its predictions change when only the match between quality estimates and test instances is broken Separating evidence, availability, and quality.

Jane: For modality m and instance i, we represent the input using three components: evidence Em,i, availability Mm,i ∈

0, 1: , and quality signals Qm,i <ref:2606.26473#pg4>.

Tom: This separation distinguishes an outage <ref:2606.26473#pg4> where a modality is absent (Mm,i = 0), from a degradation where the modality is present but corrupted (Mm,i = 1 with unreliable Em,i).

Title and authors: Tom: So we’ve looked at the title, "When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependence," and it’s really setting up a test to see if those reliability scores are driving decisions or just correlating with performance.

Jane: It’s asking that question directly, Tom. It suggests that we need a way to test whether reliability information is used during inference or if it's just sitting there doing some background math without affecting the final output.

Lu: The authors are proposing a simple diagnostic to test this hypothesis by permuting the reliability scores across test examples while keeping other factors fixed, which is a really tight control experiment.

Meng: I’m curious about the mechanics of how they're isolating that quality effect from just having missing data, because in real deployment, missing data is always a factor.

Lu: That shuffling is what they call an alignment-breaking control, where Clean-Q is the observed quality, but Broken-Q is when you shuffle those qualities while keeping evidence E and availability M fixed.

Jane: It’s a test of whether the fusion rule is sensitive to the specific alignment between quality and evidence at inference time.

Lu: The paper then moves on to describing this aligned instance table where each row stores the evidence E, availability masks M, quality values Q, and the label for each task instance or segment.

Meng: That’s a lot of data preparation just to set up this precise control experiment before they even get to running the diagnostic.

Lalam: Yeah, and in StressID they structure it by using physiology instance IDs because that modality provides the most consistent coverage across tasks.

Tom: It shows how much setup is needed just to make sure you’re testing exactly what you want to test on a granular level.

Jane: And that level of detail is what allows us to see if quality signals are actually doing something specific for each piece of data rather than just being a general summary.

The paper's summary: Tom: Moving into the summary, the paper lays out this leakage-safe diagnostic as a way to test if reliability information is used during inference by permuting those scores while keeping other factors fixed.

Jane: Essentially, it’s showing that we can isolate quality effects from missingness by creating an alignment-breaking control where you shuffle qualities across held-out test rows where the modality is present, but you keep evidence E and availability M fixed.

Lu: The goal of this specific control is to preserve the marginal distribution of quality values and the availability pattern, but break that instance-wise link between a quality estimate and its corresponding evidence.

Meng: So we are trying to isolate whether the model depends on how well that specific quality estimate lines up with its actual evidence at inference time.

Lalam: Exactly. And they restrict evaluation to fully observed instances, DFULL, where availability is constant, so any difference between Clean-Q and Broken-Q reflects dependence on quality–evidence alignment rather than missing modalities.

Tom: And the key metric they use is the permutation gap Delta perm, which is calculated as S(Clean-Q) minus Ek

S(Broken-Q (k)): , and a positive value indicates that the fused prediction relied on quality in an instance-specific way.

Jane: If that delta perm is near zero, it means shuffling quality doesn't measurably change the decision rule, suggesting the model isn't using it in a way that matters.

Lu: They show empirical findings on StressID and CMU-MOSEI datasets where shuffling native quality signals produces near-zero performance changes, even though an oracle that selects the best modality per example shows that better routing is possible.

Meng: That’s the core tension: a theoretical possibility versus what we actually see when we test frozen fusion rules.

Lalam: On CMU-MOSEI, they found the Clean–Broken gap is small and nonsignificant even with oracle headroom available.

Tom: So the summary boils down to this: quality awareness in architecture doesn't imply quality dependence in decisions; reliability must predict which expert is correct for the current instance, not merely summarize generic signal cleanliness.

Jane: That’s a very important clarification for anyone building multimodal systems—the paper on When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependence really highlights this boundary.

Lu: It’s about where the line is drawn between summary and actionable information.

The paper's improvements: Tom: Now let's talk about the suggested improvements, which are focused on making the diagnostic more robust and practical for real-world use.

Jane: The first improvement they suggest is to evaluation of reliability signals during inference in multimodal systems allows for more robust decision-level fusion rules that only rely on quality when it "reliably predicts unimodal correctness," preventing models from being misled by spurious correlations in quality estimates.

Lu: That means we want the system to be conditioned so that it only uses the quality information when it reliably points to which modality is correct for that specific instance, rather than just using it broadly.

Meng: If we can do that, then our fusion rules only use the quality when it actually predicts unimodal correctness and not just some spurious correlation.

Lalam: This helps ensure that if the model is relying on quality, it’s because the quality signal is reliably predicting which modality is correct for that specific instance.

Tom: The second improvement is developing a new fusion architecture, specifically a conditioning aware mixture of experts model that learns a linear softmax router that maps availability and quality features to expert weights based on an input vector of Mm and Qm.

Jane: That sounds like we can actually implement something where the system dynamically adjusts its contribution of each modality based on whether it is available and how reliable its evidence is.

Lu: That architecture lets the system learn a linear softmax router that takes that vector

Mm,i, Qm,i: as input to determine which expert to weigh most heavily.

Meng: I could see how using a linear router based on those features would be much cleaner than having some complex learned non-linear mapping for routing.

Lalam: The paper suggests this conditioning aware mixture of experts model is the way to go for implementing that dynamic adjustment you’re talking about.

Tom: They also introduced the oracle headroom metric, which is a measure of performance gain by comparing perfect per-instance expert selection against quality-aware routing, quantifying how much potential there is.

Jane: That headroom metric allows researchers to quantify exactly how much performance gain we could realistically get from using quality signals that correctly identify the reliable modality.

Lu: These improvements focus on building systems where reliability estimates guide the fusion process dynamically instead of just being passively available features.

Conclusion: Tom: So wrapping up, the paper "When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependence" shows that quality awareness in architecture doesn't imply quality dependence in decisions; it must be tied to reliability predicting expert correctness for the current example.

Jane: It’s a very clear boundary, and it tells us that the bottleneck isn't necessarily fusion capacity but how reliability aligns with expert correctness, not just summarizing generic signal cleanliness.

Lu: The core finding is that shuffling native reliability estimates barely changed fused decisions across StressID and a near-fully observed CMU-MOSEI boundary case.

Meng: So for us shipping models at scale, the implication is that we need to focus on making sure our routing mechanism is actually using the quality information for decision-making, not just feeding it into a weight calculation.

Lalam: I'm confident that this diagnostic helps us know whether we can rely on quality signals to predict which input stream to trust for a specific example.

Tom: It’s about finding that alignment between reliability and correctness, not just having the fusion capacity to combine different modalities effectively.

Jane: We’ve seen how this diagnostic lets us test if those scores are actually influencing the decisions or if they're just being used as a useful summary of signal cleanliness.

Lu: And for future work, they flag that the structural diagnostics require labels and are evaluation tools rather than deployment mechanisms, which means we need real labels to run these tests before we can use this diagnostic in production.

Meng: That makes sense. We can’t just deploy a diagnostic that needs perfect ground truth labels because it needs to know exactly what the paper says it does.

Lalam: So they also point out that fully observed evaluation isolates quality effects but doesn't test those interactions with modality outages, which means we need more data that covers both missing modalities and variation in quality.

Tom: So the diagnostic on When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependence is a powerful tool for testing if reliability scores are influencing decisions or just correlating with performance.

Jane: It’s a very concrete way to ask that question and it gives us a lot to think about as we look at how we build multimodal AI systems in the next few years.

Jaden Moon, Arvind Pillai, Andrew Campbell

Dartmouth College

cs.LG, eess.AS

Submitted: 2026-06-25

Updated: 2026-06-25

Comments: Accepted to INTERSPEECH 2026. 5 pages, 1 figure, 5 tables

Journal ref: Proc. Interspeech 2026, pp. 3842-3846

DOI: 10.21437/Interspeech.2026-2989

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: The gist The authors propose a leakage-safe diagnostic to test whether quality signals influence model decisions during inference by permuting reliability scores while keeping other factors fixed,

Key concepts

Decision-level identifiability diagnostic: This diagnostic asks if a model's prediction changes when only the link between quality estimates and test instances is broken. It separates whether performance depends on missing modalities (outage) or corrupted evidence (degradation), isolating the effect of quality signals.
Alignment-breaking control (Broken-Q): This involves shuffling observed quality values across test rows for a specific modality while keeping evidence and availability fixed. This breaks the link between a quality estimate and its corresponding evidence, allowing researchers to isolate whether decisions rely on this alignment or just the presence of data.
Permutation gap (∆perm): This metric compares the performance of clean quality data against shuffled quality data. A positive gap suggests that if a model uses quality in an instance-specific way, it will perform better when quality aligns with evidence, indicating that quality awareness matters for decision-level fusion.

Terminology

Multimodal Fusion — 1× in this paper · explained in 4 episodes

Summary

The gist The authors propose a leakage-safe diagnostic to test whether quality signals influence model decisions during inference by permuting reliability scores while keeping other factors fixed, finding that native quality estimates do not measurably change fused predictions across tested multimodal systems.

Decision-level identifiability diagnostic

For each cross-validation fold, the diagnostic asks a targeted question: after a multimodal model has been trained, do its predictions change when only the match between quality estimates and test instances is broken Separating evidence, availability, and quality. For modality m and instance i, we represent the input using three components: evidence Em,i, availability Mm,i ∈ [0, 1], and quality signals Qm,i <ref:2606.26473#pg4>. This separation distinguishes an outage <ref:2606.26473#pg4> where a modality is absent (Mm,i = 0), from a degradation where the modality is present but corrupted (Mm,i = 1 with unreliable Em,i). The resulting multimodal decision rule is p(yi E m,i, Mm,i, Qm,i m∈M) <ref:2606.26473#pg4>.

Alignment-breaking control

Let Clean-Q denote the observed quality values used at inference. Broken-Q is formed by shuffling Qm,i across held-out test rows where modality m is present, while keeping evidence E and availability M fixed. This preserves the marginal distribution of quality values and the availability pattern, but breaks the instance-wise link between a quality estimate and its corresponding evidence. To isolate quality effects from missingness, evaluation is restricted to fully observed instances. On this set, availability is constant, so any Clean-Q versus Broken-Q difference reflects dependence on quality–evidence alignment rather than missing modalities.

Identifiability statistic and interpretation

Let S(·) denote the evaluation metric, taken to be Balanced Accuracy (equivalent to Unweighted Average Recall in the binary case). For each permutation k, let Q(k) denote a Broken-Q sample obtained by shuffling Qm,i within for each modality m, while keeping E and M fixed. The permutation gap is ∆perm = S(Clean-Q) − Ek[S(Broken-Q (k))]. If the trained fusion rule uses quality in an instance-specific way, Clean-Q should outperform its shuffled controls and ∆perm should be positive. A near-zero gap is most informative when quality-aware routing had room to matter.

Empirical findings on datasets

The diagnostic is evaluated on two complementary multimodal datasets, StressID and CMU-MOSEI. On StressID, shuffling native quality signals produces near-zero performance changes, even though an oracle that selects the best modality per example shows that better routing is possible. In contrast, positive controls produce large performance changes when quality is constructed to track corruption or unimodal correctness. On CMU-MOSEI, the Clean–Broken gap is small and nonsignificant despite oracle headroom.

Structural preconditions and conclusion

The null hypothesis is not that quality is useless in general; it is that this trained fusion rule is invariant to quality–instance alignment at inference time. A positive permutation gap indicates that quality influences fused decisions under fixed experts and fusion parameters, while a near-zero gap indicates that shuffling quality does not measurably change the decision rule. Thus, quality awareness in architecture does not imply quality dependence in decisions; reliability must predict which expert is correct for the current instance, not merely summarize generic signal cleanliness. Quality-aware fusion helps only when reliability estimates identify which input stream to trust for the current example.

Limitations and future work

The structural diagnostics require labels and are evaluation tools, not deployment mechanisms. Fully observed evaluation isolates quality effects but not quality–outage interactions, and positive controls are StressID-only. An ideal follow-up dataset would combine natural modality dropout, broader within-modality quality variation, and modality-specific positive controls, enabling the same diagnostic to test both outage and degradation.

Conclusion

Quality-aware multimodal fusion helps only when reliability estimates identify which input stream to trust for the current example. Across StressID and a near-fully observed CMU-MOSEI boundary case, shuffling native reliability estimates barely changed fused decisions, even though selecting the best modality for each example showed that better routing was possible. StressID positive controls changed decisions when reliability tracked corruption or unimodal correctness, showing that the same frozen fusion rules can use reliability when it is aligned with expert correctness. Thus, reliability–correctness alignment, not fusion capacity, is the bottleneck.

How it works

The diagnostic involves freezing the model and quality values while shuffling them across test examples to create Broken-Q.
The evaluation is restricted to fully observed instances (DFULL) where availability is constant, so any difference reflects dependence on quality–evidence alignment.
A one-sided permutation value pperm is computed using the unbiased estimator of Phipson and Smyth [21].
The null hypothesis tests whether the trained fusion rule is invariant to quality–instance alignment at inference time.

Fusion methods

The diagnostic tests decision-level fusion so that it checks routing among fixed unimodal experts rather than changes in representation learning. The quality-aware version of late fusion weights each expert by its normalized quality value, wm,i = P Qm,i m′ Qm′, i, pClean-Q(yi = 1) = X m∈M wm,i pm,i. The conditioning-aware mixture of experts learns a linear softmax router that maps availability and quality features to expert weights.

Datasets

The diagnostic is evaluated on StressID, which contains heterogeneous modality availability, and CMU-MOSEI as a near-fully observed boundary case. StressID contains synchronized facial video, speech, and physiological signals from 65 participants performing 11 tasks. CMU-MOSEI contains 23,453 annotated opinion segments from 3,228 videos and 1,000 speakers across 250 topics.

Key metrics

The authors report oracle headroom (Headroom = S(Oracle) − S(Clean-Q)) and competitiveness (∆i = p(1),i(yi) − p(2),i(yi)), and quality–correctness alignment (ρm = corr Qm,i, 1(y m,i = yi)).

References

[1] T. Baltrusaitis, C. Ahuja, and L.-P. Morency, “Multimodal ma- machine learning: A survey and taxonomy,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 2, pp. 423–443, 2019.

[17] H. Chaptoukaev, V. Strizhkova, M. Panariello, B. Dalpaos, A. Reka, V. Manera, S.

Improvements for AI systems

Evaluation of reliability signals during inference in multimodal systems allows for more robust decision-level fusion rules that only rely on quality when it reliably predicts unimodal correctness. This prevents models from being misled by spurious correlations in quality estimates, leading to higher performance on unseen data.
Implementation of a leakage-safe diagnostic involving permuting reliability scores while keeping evidence and modality availability fixed ensures that any observed performance degradation is due to the decision rule explicitly depending on the quality signal, rather than simply reflecting pre-existing dataset correlations.
Development of a new fusion architecture where expert weights are conditioned on both modality availability and estimated quality, such as a conditioning aware mixture of experts model that learns a linear softmax router based on an input vector [ Mm,i, Qm,i ]. This system can dynamically adjust the contribution of each modality based on whether it is available and how reliable its evidence is.
Creation of an oracle headroom metric allows researchers to quantify oracle headroom, which is the difference between performance with a perfect per-instance expert selection and performance with quality-aware routing [Headroom = S(Oracle) − S(Clean-Q)]. This metric precisely measures how much performance gain is possible by using quality signals that correctly identify the reliable modality.
Training multimodal fusion models only on observed training rows, followed by a test where quality values are shuffled across held-out test rows while evidence and availability remain fixed (Broken-Q), will reveal if the trained fusion rule uses quality in an instance-specific way [pperm = (1 + P k1[sk ≥ s0])/(K + 1)]. This directly tests whether the model's internal decision process is sensitive to the alignment between quality estimates and actual evidence at inference time.