FusionAudit: Pathway-Conditioned Robustness Auditing for Native Multimodal Models

The gist The diagnostic-driven attacks substantially raise the estimated residual risk, showing that PaRMA provides a falsifiable, evidence-driven auditing methodology for uncovering vulnerabilities missed by standard benchmarks

How it works

PaRMA is an evidence-based auditing framework that defines pathway-conditioned risk over two primary imagelevel interventions: pixel perturbation and rendered-text insertion It uses pathway-specific diagnostics to expose architecture-specific blind spots, moving beyond arbitrary attack enumeration This framework instantiates FusionAudit, which introduces two diagnostic-motivated attacks: MANTIS and Nat-Evid The audit’s principal scalar is the residual pathway-complete risk, yielding an empirical lower bound on pathway-complete risk

Native Multimodal VLMs: Taxonomy and Threat Scope

The paper introduces a graded nativeness taxonomy (L1–L3) to organize audit targets across fusion designs

** L1 (Alignment-era): A frozen or lightly adapted vision encoder connects to a language model via a projection module, where modalities remain in structurally separate spaces; the bottleneck is the only junction This contrasts with L3 (Deep/early fusion), where visual information participates throughout the language computation, including DeepStack and Early Fusion models The audit formalizes a VLM as a function mapping a visual input xv and a text prompt xt to an answer, fixing the text prompt and intervening exclusively on xv The visual attack surface is decomposed into two primary pathways: perceptual corruption (πpix), which involves pixel-space perturbations, and rendered-text insertion (πtxt), which involves typographic overlays The audit is strictly conditioned on the model’s clean-correct set—the subset of examples the model answers correctly before any intervention The visual pathways are defined as πpix where Bpix is the perturbation domain and ϕpix(xv; δ) is clip(xv + δ), and πtxt where Btxt is the space of candidate rendered strings of length at most L, and ϕtxt(xv; s, p) is render(xv; s, p) The empirical pathway-complete risk RbFA is the union failure rate across all attack suites Ak for each pathway k PaRMA departs from naive multi-attack unions by strengthening the risk via pathway-resolved diagnostics The strengthened estimator Rb+FA serves as the audit’s principal, falsifiable deliverable The diagnostic-driven augmentations are MANTIS, a margin-aware pixel attack concentrating gradients at the first-divergence token t⋆, and Nat-Evid, a host-congruent semantic intervention that naturalizes text payloads to bypass visual rejection The gap Rb+FA − Rb 0FA measures the additional risk surfaced by our diagnostics The level-mean risk increases by +10.6% in L1, +16.9% in L2, and +25.1% in L3 when moving from Rb0 to Rb+ This demonstrates that standard end-to-end loss can spread gradient pressure across the sequence, leaving failures that are surfaced by concentrating gradient at the answer-discriminative token In deeper-fusion architectures, most flips survive restoration at every depth, consistent with visual features re-entering computation across many layers The diagnostic signature for the see-but-not-deceived signature is high task-prompt OAR (> 1) and positive RUG (Read–Use Gap) This suggests that native VLMs develop a fine-grained sensitivity to visual congruence, making direct typographic attacks incomplete and leaving additional semantic failures unmeasured due to visual rejection The diagnostic-driven risk gap persists across the chain, consistent with the view that MANTIS and Nat-Evid expose vulnerabilities associated with the native multimodal shift 5.3 Pixel Pathway Deep Dive**

The pixel pathway deep dive explores stability and ablation of MANTIS Figure 2 suggests that MANTIS’s gains are not only an artifact of hyperparameter tuning, as the min-max variance is small relative to the absolute ASR, with a median standard deviation of just 1.3% Optimization budget analysis shows that ASR largely saturates by 30–40 steps, indicating that the 40-step setting is a reasonable audit budget Stability ablation confirms that MANTIS is not heavily dependent on initialization, as the min-max variance across three seeds is small relative to the absolute ASR The relative vulnerability ordering across fusion levels (L1 > L2 > L3) is preserved in tested settings, suggesting that the trend exposed by MANTIS is not specific to the chosen threat model capacity

Text Pathway Deep Dive

On the rendered-text pathway, Typo-Evid is strongest in L1, driven by one alignment-era model, whereas Nat-Evid is stronger than Typo-Evid by 13.6% in L2 and 15.0% in L3 This pattern suggests that direct overlays are not a uniform proxy for rendered-text risk in native models, as the payload may be processed but not necessarily adopted as the task answer The behavioral diagnostic RUG shows Pread increases from L2 to L3 while Puse decreases, widening RUG from +52.7% to +69.

1. The improved system can perform pathway-complete robustness auditing using FusionAudit to estimate the residual pathway-complete risk, yielding an empirical lower bound on pathway-complete risk by combining diagnostic findings from baseline attacks with MANTIS and Nat-Evid augmentations.

2. The system can identify the see-but-not-deceived signature where native VLMs attend heavily to typographic overlays but visually reject them, allowing developers to design interventions like Nat-Evid, which uses host-congruent semantic insertion to bypass this visual rejection.

3. The system can provide a more stable ranking of model robustness by moving beyond legacy single-attack proxies, as shown in the result: Fixed-model paired bootstraps put the corresponding 95% CIs at [9.2, 12.0], [16.1, 17.8], and [24.2, 26.0] percentage points, contrasting with unstable rankings from attacks like End-PGD for native models.

4. The system can pinpoint the exact location of adversarial decision-making via MANTIS, which concentrates gradient pressure exclusively at t⋆, the first token where the two continuations diverge, providing a more precise understanding of where model failure occurs compared to end-to-end attacks that spread gradient pressure across all output tokens.

5. The system can better understand architectural vulnerability by measuring layer restoration performance, noting that Recovery falls with fusion level: 92.2% [87.6, 96.1] in L1, 62.6% [55.5, 69.6] in L2, and 25.6% [20.7, 30.7] in L3, indicating where visual features are most resilient or susceptible to attack based on the native fusion architecture (L1–L3).

6. The system can distinguish between different types of text pathway risks by comparing Typo-Evid and Nat-Evid, as Typo-Evid is strongest in L1, driven by one alignment-era model, whereas Nat-Evid is stronger than Typo-Evid by 13.6% in L2 and 15.0% in L3, suggesting that direct overlays are not a uniform proxy for rendered-text risk.