Trust Before Fusion: QIMG-7 and Source-Aware Resolution for Polluted Multimodal RAG

arXiv:2607.10798 · cs.CL · Submitted 2026-07-12 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Trust Before Fusion".

Jane: Here's the gist of the paper. The gist The introduction introduces QIMG-7, a controlled benchmark for multimodal retrieval pollution in multi-sentence factual QA, and proposes Source-Aware Trust Resolution (SATR), a training-free approach that compares candidate answers to select among them or fall back based on source reliability.

Tom: Benchmark Construction QIMG-7 is constructed as a controlled stress-test benchmark for multi-sentence factual QA, spanning four datasets, seven image-attack families covering metadata, pixel, and style perturbations, and 16 evaluation regimes (Table 1).

Jane: The benchmark starts from four prompt-only factual QA datasets: LongFact (Wei et al., 2024), Biography (Min et al., 2023), AlpacaFact (Lin et al., 2024), and FAVA (Mishra et al., 2024) The final QIMG-7 benchmark uses query-based image retrieval, selecting up to five candidate images per question Each evaluated question appears under 16 regimes: clean/polluted text crossed with clean image or one of seven image-pollution families For each query and regime, three isolated candidate answers are generated: Parametric (answer with no retrieved evidence), Text-only (answer using only retrieved text evidence), and Full-MM (answer using both retrieved text and image evidence) The Answer-Consensus baseline computes pairwise string/token similarity among the three candidate answers to select the most central answer Source-Aware Trust Resolution (SATR) SATR is a training-free family that reasons over isolated Parametric, Text-only, and Full-MM candidates plus structured reliability fields to select, compose, or fall back The Field-Selector variant achieves the best balanced score of 0.816, improving over Full-MM by 11.7 points and over the Cascaded Router by 2.7 points Algorithm 1 SATR Field-Selector decision requires the LLM to produce reliability fields and candidate scores, followed by a deterministic screen applying conservative fallback rules Evaluation and Results The primary metric is an LLM-as-judge support score against trusted clean evidence, where the judge is blinded to method identity, route choice, regime label, and attack family Field-Selector achieves the best balanced score (0.816) and the highest polluted-text score (0.751), improving over the Cascaded Router on both clean and polluted regimes The results show that naive multimodal fusion is unsafe by default, as FullMM performs worse than Parametric answering under polluted text Overall, QIMG-7 and SATR suggest that robust RAG under multimodal retrieval pollution should shift from unconditional fusion to source-aware trust resolution Field-Selector is the strongest main method in the main gpt-4o-mini stack, substantially reducing the clean-to-polluted gap The results support selective trust rather than unconditional fusion Limitations The benchmark scope is designed as a controlled stress-test benchmark for multimodal retrieval pollution in multi-sentence factual QA, where polluted text is the dominant failure channel The cross-model results show that trust routing depends on the gate model’s ability to assess evidence reliability SATR should therefore be viewed as a model-agnostic framework rather than a guarantee for every generator/gate stack Future work includes learned reliability models, image-forensics-aware routing, and multilingual, cross-domain robustness testing beyond factual QA Ethical Considerations The project studies how multimodal retrieval can be polluted with misleading text and images, and the goal is defensive to understand failure modes and improve robustness in retrieval-grounded multisentence factual QA All polluted items are explicitly labeled as synthetic manipulated benchmark artifacts in the released metadata and are not presented as factual evidence The benchmark is intended for robustness evaluation, retrieval-pollution analysis, and defensive multimodal RAG research, not deceptive-content generation or deployment References Akari Asai et al.

Title and authors: Jane: So, we’ve talked about the setup and the specific method, now let's go over the actual summary they provide for "Trust Before Fusion: QIMG-seven and Source-Aware Resolution for Polluted Multimodal RAG <ref:2607.10798#pg1,Trust Before Fusion: QIMG-7 and Source-Aware Resolution for Polluted Multimodal>." What exactly is this paper summarizing?

Lu: They’re summarizing four main contributions. First, they introduce QIMG-seven which is this controlled benchmark for testing multimodal retrieval pollution in multi-sentence factual QA <ref:2607.10798#pg1>.

Tom: And then the second contribution is the empirical finding: they show that naive multimodal fusion is unsafe by default because under polluted text, Full-MM support drops from zero point nine zero eight with clean text to zero point four nine zero with polluted text, often making Parametric fallback safer than retrieval itself <ref:2607.10798#pg1,Full-MM support drops from 0.908 with clean text to 0>.

Jane: That’s a big drop in performance, isn't it? So the third contribution is the method: they introduce Source-Aware Trust Resolution as a training-free family that reasons over isolated Parametric, Text-only, and Full-MM candidates plus structured reliability fields to select or compose or fall back.

Meng: That sounds like they’re giving the model explicit tools—those reliability fields—to make these tough choices instead of just guessing which answer is best.

Lu: And the final contribution is the mechanistic analysis, which digs into how different attacks and families affect this process, showing that prompt-based trust resolution effectiveness depends on the resolver's ability to assess reliability.

Tom: So it’s basically saying, we need a way for retrieval augmented generation systems to decide when to use multimodal retrieval versus just sticking with text or parametric knowledge based on some kind of source trust metric.

Jane: That’s the core takeaway: the paper is summarizing that unconditional fusion is brittle under pollution and that a selective approach using source trust resolution offers a better outcome.

Lu: And they are doing this by creating this controlled environment with QIMG-seven to rigorously test those specific failure modes across different datasets and attack families <ref:2607.10798#pg1>.

Meng: It sounds like they’re not just pointing out a problem, but providing the exact framework for how to build defenses against these kinds of adversarial manipulations in multimodal RAG.

Tom: Right, so it’s moving beyond just saying "be careful" to showing exactly how a specific mechanism—SATR—improves the score and handles those pollution scenarios better than prior methods.

Jane: It shows that even for complex tasks like multi-sentence factual QA, you can get measurable improvements by focusing on the quality of the retrieval decision rather than just the quantity of evidence combined.

Lu: That’s because they are using claim-level evaluation techniques, like FActScore and VeriScore, to get a granular look at factual precision in long form text generation.

Meng: So they’re tying it all together: a strong benchmark, a clear failure mode demonstration, and a specific method that addresses that failure mode with structured decision-making.

Tom: Exactly. It’s about making the system more robust by introducing explicit reasoning over source reliability before deciding on the final output.

Jane: It gives us a concrete direction for how we should be designing these multimodal systems going forward, focusing on trust resolution rather than just simple evidence aggregation.

The paper's summary: Tom: Okay, so we know the setup and the summary. Now let's get into what they actually improved in terms of results and mechanics in "Trust Before Fusion: QIMG-seven and Source-Aware Resolution for Polluted Multimodal RAG <ref:2607.10798#pg1,Trust Before Fusion: QIMG-7 and Source-Aware Resolution for Polluted Multimodal>."

Lu: They introduced SATR, Source-Aware Trust Resolution, which is this training-free family that reasons over the isolated Parametric, Text-only, and Full-MM candidates plus structured reliability fields to select or compose or fall back.

Jane: And they show that a specific variant of this called FieldSelector gets a balanced score of zero point eight one six, and it improves over the Full-MM approach by about twelve points and over the Cascaded Router by about three points, which is pretty significant for performance gains.

Meng: So that twelve point improvement over Full-MM is substantial enough to make it a strong candidate for actual deployment testing. That's what I care about—a method that actually performs well when it matters.

Tom: They also found that FieldSelector achieves the best balanced score of zero point eight one six and the highest polluted-text score of zero point seven five one, which is an improvement over the Cascaded Router on both clean and polluted regimes.

Jane: That high polluted-text score is interesting because it means it’s not just good at handling clean data, but it handles the real mess quite well too.

Lu: The mechanistic analysis shows that a field ablation was really important, which helped them pinpoint exactly how different attack families impact this selection process.

Meng: That helps us understand where the model is weak so we can focus our efforts on fixing those specific decision-making gaps rather than just trying to make the whole model better generally.

Tom: And they also mentioned that explicit text reliability modeling was the dominant driver of these gains in a text-first setting, which suggests that when text evidence is marked as suspicious, that’s when you need to be most careful.

Jane: So, the improvement isn't just adding another layer; it’s specifically about how the system weighs textual and visual reliability against each other during the selection process.

Lu: They are moving away from this unconditional fusion idea toward a framework where we explicitly reason over those sources before making a final choice.

Meng: If we can design a way to reliably feed that structured reliability information into our system, then this approach seems like it solves the problem of brittle multimodal systems.

Tom: So the main improvement is moving from an automatic, naive decision process to one that reasons over source trust fields before selecting or falling back on an answer.

Jane: It’s a really concrete proposal for making RAG systems more dependable in environments where the retrieved data might be intentionally corrupted by attacks.

The paper's improvements: Tom: Alright, we’ve covered the benchmark setup, the summary of what they found, and how their specific method improved performance in "Trust Before Fusion: QIMG-seven and Source-Aware Resolution for Polluted Multimodal RAG <ref:2607.10798#pg1,Trust Before Fusion: QIMG-7 and Source-Aware Resolution for Polluted Multimodal>." Let’s wrap things up.

Jane: So to summarize the implications for us listening right now, we’ve seen that when retrieval can return misleading text or images, just fusing them together isn't safe; it actually makes the system worse under pollution.

Lu: The big shift suggested by this paper is moving away from unconditional fusion toward a source-aware trust resolution approach where the AI explicitly reasons about source reliability before making a final choice on which candidate answer to use.

Meng: From an engineering standpoint, this means we should prioritize building reliable mechanisms for generating those structured reliability fields, because that seems to be the critical part for this method to actually deliver tangible gains in robustness.

Tom: They showed that Field-Selector is a strong main method in the main gpt-4o-mini stack, and it substantially reduces the gap between clean and polluted performance <ref:2607.10798#pg1,in the main gpt-4o-mini stack>.

Jane: It supports selective trust rather than just unconditional fusion, which is a much safer way to handle multimodal retrieval when you know there’s a risk of data corruption in your inputs.

Lu: This work provides a framework for building RAG systems that are more aware of their own input sources and can adapt their behavior accordingly when those sources are untrustworthy.

Meng: I think we should keep an eye on these methods, especially how they handle specific attacks like semantic edits, because that’s where the real-world risk is hiding.

Tom: The paper "Trust Before Fusion: QIMG-seven and Source-Aware Resolution for Polluted Multimodal RAG" gives us a clear path toward more robust multimodal systems by emphasizing source trust resolution over naive fusion <ref:2607.10798#pg1,Trust Before Fusion: QIMG-7 and Source-Aware Resolution for Polluted Multimodal>.

Conclusion: Tom: So we've been looking at "Trust Before Fusion: QIMG-seven and Source-Aware Resolution for Polluted Multimodal RAG." Essentially, they're showing that just smashing retrieved text and images together isn't safe when there’s pollution, especially with polluted text.

Jane: Right, so the core of it is this controlled benchmark called QIMG-seven where they test how retrieval systems handle those noisy inputs across different image attacks and text regimes.

Lu: And they introduce Source-Aware Trust Resolution, or SATR, which is a training-free method that lets the AI reason over different candidate answers—parametric, text-only, and full multimodal—using structured reliability fields.

Meng: So it’s giving the model explicit tools to check if the text evidence or image evidence is even trustworthy before deciding how to combine them.

Tom: And they found that this Field-Selector variant actually performed better than a few other methods, getting a balanced score of zero point eight one six and beating the Full-MM approach by about twelve points.

Jane: That's a big number, Tom; it means this selective approach to trust resolution isn't just theoretical stuff anymore; it actually translates into measurable performance gains on those polluted datasets.

Lu: And the mechanism they used relied heavily on that explicit text reliability modeling because when the text evidence is marked as suspicious or weak, that’s exactly when the system needs to fall back to a safer answer.

Meng: From a deployment standpoint, if we can get an AI model to reliably produce those structured reliability scores, it helps us control what gets passed forward at scale.

Tom: The big result here is that this method supports selective trust rather than just trying to force an unconditional fusion, which is the naive approach we’ve seen fail under pollution.

Jane: It really highlights that for multimodal RAG to be dependable, it needs to shift from blindly combining everything to making a thoughtful decision about what sources you can actually trust.

Lu: The limitation they mentioned is that this framework depends on the gate model's ability to assess evidence reliability; so it’s not a magic fix for every single generator or gate stack out there.

Meng: Exactly, so we aren't looking for one perfect solution, but we need to design systems that can handle these trust routing decisions in their own way.

Tom: So overall, "Trust Before Fusion: QIMG-seven and Source-Aware Resolution for Polluted Multimodal RAG" suggests that robustness comes from better source awareness during the retrieval decision itself.

Jane: It’s a clear signal that future work needs to focus on building those reliable input evaluation systems for multimodal models.

Lu: And that leads us perfectly into the next topic, because we should also look at how other researchers are trying to build these trust routing mechanisms using learned vision routers in their own papers.

Saadeldine Eletter Owais Aijaz, Preslav Nakov

Mohamed bin Zayed University of Artificial Intelligence

cs.CL

Submitted: 2026-07-12

Updated: 2026-10-18

Comments: 23 pages, 6 figures, 23 tables. Preprint under review

Code: https://github.com/SaadElDine/Trust_Before_Fusion

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: The gist The introduction introduces QIMG-7, a controlled benchmark for multimodal retrieval pollution in multi-sentence factual QA, and proposes Source-Aware Trust Resolution (SATR), a training-free

Key concepts

QIMG-7: A controlled stress test benchmark for factual question answering using multimodal retrieval. It combines four factual datasets with seven types of image attacks (like metadata or pixel changes) and sixteen different evaluation scenarios to rigorously test how well RAG systems handle polluted text and images.
Parametric Candidate: An answer generated without any retrieved evidence. This serves as a baseline to measure the system's ability to generate an answer independently of the retrieved multimodal data. It represents the most 'unsupported' version of a potential response.
Source-Aware Trust Resolution (SATR): A training-free strategy that reasons over different answer candidates and structured reliability fields. Instead of blindly fusing all evidence, SATR uses these fields to decide which candidate to select or if it is safe enough to fall back on, prioritizing reliable information.
Field-Selector: A specific variant of SATR that achieves the best performance by requiring the LLM to output reliability scores for text and image sources. This allows the system to make a balanced decision between different answer types based on how trustworthy each piece of retrieved evidence is.

Terminology

RAG — 2× in this paper · explained in 3 episodes
Ablation — 1× in this paper · explained in 1 episode
Benchmark — 1× in this paper · explained in 2 episodes
Generator — 1× in this paper · explained in 2 episodes
Multimodal Fusion — 1× in this paper · explained in 4 episodes
Retrieval-Augmented Generation (RAG) — 1× in this paper · explained in 21 episodes

Summary

The gist The introduction introduces QIMG-7, a controlled benchmark for multimodal retrieval pollution in multi-sentence factual QA, and proposes Source-Aware Trust Resolution (SATR), a training-free approach that compares candidate answers to select among them or fall back based on source reliability.

Benchmark Construction

QIMG-7 is constructed as a controlled stress-test benchmark for multi-sentence factual QA, spanning four datasets, seven image-attack families covering metadata, pixel, and style perturbations, and 16 evaluation regimes (Table 1). The benchmark starts from four prompt-only factual QA datasets: LongFact (Wei et al., 2024), Biography (Min et al., 2023), AlpacaFact (Lin et al., 2024), and FAVA (Mishra et al., 2024) The final QIMG-7 benchmark uses query-based image retrieval, selecting up to five candidate images per question Each evaluated question appears under 16 regimes: clean/polluted text crossed with clean image or one of seven image-pollution families

Methodology and Candidates

For each query and regime, three isolated candidate answers are generated: Parametric (answer with no retrieved evidence), Text-only (answer using only retrieved text evidence), and Full-MM (answer using both retrieved text and image evidence) The Answer-Consensus baseline computes pairwise string/token similarity among the three candidate answers to select the most central answer

Source-Aware Trust Resolution (SATR)

SATR is a training-free family that reasons over isolated Parametric, Text-only, and Full-MM candidates plus structured reliability fields to select, compose, or fall back The Field-Selector variant achieves the best balanced score of 0.816, improving over Full-MM by 11.7 points and over the Cascaded Router by 2.7 points Algorithm 1 SATR Field-Selector decision requires the LLM to produce reliability fields and candidate scores, followed by a deterministic screen applying conservative fallback rules

Evaluation and Results

The primary metric is an LLM-as-judge support score against trusted clean evidence, where the judge is blinded to method identity, route choice, regime label, and attack family Field-Selector achieves the best balanced score (0.816) and the highest polluted-text score (0.751), improving over the Cascaded Router on both clean and polluted regimes The results show that naive multimodal fusion is unsafe by default, as FullMM performs worse than Parametric answering under polluted text

Conclusion

Overall, QIMG-7 and SATR suggest that robust RAG under multimodal retrieval pollution should shift from unconditional fusion to source-aware trust resolution Field-Selector is the strongest main method in the main gpt-4o-mini stack, substantially reducing the clean-to-polluted gap The results support selective trust rather than unconditional fusion

Limitations

The benchmark scope is designed as a controlled stress-test benchmark for multimodal retrieval pollution in multi-sentence factual QA, where polluted text is the dominant failure channel The cross-model results show that trust routing depends on the gate model’s ability to assess evidence reliability SATR should therefore be viewed as a model-agnostic framework rather than a guarantee for every generator/gate stack Future work includes learned reliability models, image-forensics-aware routing, and multilingual, cross-domain robustness testing beyond factual QA

Ethical Considerations

The project studies how multimodal retrieval can be polluted with misleading text and images, and the goal is defensive to understand failure modes and improve robustness in retrieval-grounded multisentence factual QA All polluted items are explicitly labeled as synthetic manipulated benchmark artifacts in the released metadata and are not presented as factual evidence The benchmark is intended for robustness evaluation, retrieval-pollution analysis, and defensive multimodal RAG research, not deceptive-content generation or deployment

References

Akari Asai et al. 2024. Self-RAG: Learning to retrieve, generate, and critique through self-reflection Shuai Bai et al. 2025. Qwen2.5-VL technical report Yann Dubois et al. 2023. AlpacaFarm: A simulation framework for methods that learn from human feedback Saadeldine Eletter et al. 2026. MIRAGE: Defending long-form RAG against misinformation pollution Hyeonjeong Ha et al. 2025. MM-PoisonRAG: Disrupting multimodal RAG with local and global poisoning attacks Wenbo Hu et al. 2025. MRAGBench: Vision-centric evaluation for retrievalaugmented multimodal models Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering Patrick Lewis et al. 2020. Retrieval-augmented generation for knowledgeintensive NLP tasks Yifan Li et al. 2023. Evaluating object hallucination in large vision-language models Sheng-Chieh Lin et al. 2024. FLAME: Factuality-aware alignment for large language models Yinuo Liu et al. 2025. Poisoned-MRAG: Knowledge poisoning attacks to multimodal retrieval augmented generation Meta AI. 2024. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models OpenAI. 2024. GPT-4o mini: Advancing costefficient intelligence OpenAI. 2025. Introducing GPT-4.1 in the API Yikang Pan et al. 2023. On the risk of misinformation pollution with large language models Yixiao Song et al. 2024. VeriScore: Evaluating the factuality of verifiable claims in long-form text generation Zhiqing Sun et al. 2024. Aligning large multimodal models with factually augmented RLHF Jerry Wei et al. 2024. Long-form factuality in large language models Chong Xiang et al. 2024. Certifiably robust RAG against retrieval corruption Shi-Qi Yan et al. 2024. Corrective retrieval augmented generation Sewon Min et al. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation Abhika Mishra et al. 2024. Fine-grained hallucination detection and editing for language models Matin Mortaheb et al. 2025. RAG-check: Evaluating multimodal retrieval augmented generation performance Open AI. 2024. GPT-4o mini: Advancing costefficient intelligence Open AI. 2025. Introducing GPT-4.1 in the API Meta AI. 2024. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models Accessed: 2026-05-26. OpenAI. 205 GPT-4o mini: Advancing costefficient intelligence Accessed: 2026-05-26. OpenAI. 20GPT-4.1 in the API Accessed: 20GPT-4o mini: Advancing costefficient intelligence Accessed: 20OpenAI. 20GPT-4.1 in the API Accessed: 26-05-26. OpenAI. 20GPT-4o mini: Advancing costefficient intelligence Accessed: 26-05-26. OpenAI. 20GPT-4o mini: Advancing costefficient intelligence Accessed: 26-05-26. OpenAI. 20GPT-4o mini: Advancing costefficient intelligence Accessed: 26-05-26. OpenAI. 20GPT-4o mini: Advancing costefficient intelligence Accessed: 26-05-26. OpenAI. 20GPT-4o mini: Advancing costefficient intelligence Accessed: 26-05-26. OpenAI.

Improvements for AI systems

Implement a source-aware trust resolution family, such as Field-Selector, to decide between Parametric, Text-only, and Full-MM candidate answers based on structured reliability fields like text reliability and image reliability. This allows the system to select among candidate answers or fall back based on source reliability, moving beyond unconditional fusion.
Design a decision policy that utilizes a Cascaded Router structure where a binary self-check gate first assesses if retrieved evidence is trustworthy enough to use before proceeding to a triage gate for full multimodal or text-only selection. This improves robustness by filtering globally unreliable retrieval before inspecting internal disagreements among candidate answers.
Develop an atomic factuality audit pipeline that decomposes generated long-form answers into atomic factual claims and judges each claim against trusted clean evidence for the same question. This provides a claim-level metric to corroborate answer-level support scores, making the system more sensitive to specific injected falsehoods like years or locations.
Incorporate explicit text reliability modeling into the source selection process, as ablations show that explicit text-reliability modeling is the dominant driver of these gains in a text-first setting. This ensures that when text evidence is marked conflicting, suspicious, or weak, the system correctly falls back to a safer answer rather than relying on it.
Create mechanisms for detecting and mitigating specific image pollution families, such as caption flip or semantic entity rewrite, by using specialized VLM prompts or local editing models. This allows the system to distinguish between metadata-only attacks and pixel-level edits, improving performance against visually altered evidence.
Use a learned vision router based on features like CLIP or OCR for future development to replace prompt-based self-checks, as the paper suggests that Stronger learned routing is left for future work when moving beyond current setup limitations.