Beyond Generation: Using Multimodal LLMs as Detectors of LLM-Generated Deepfake Facial Images

arXiv:2610.21760 · cs.CV, cs.ET · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Beyond Generation: Using Multimodal LLMs as Detectors of LLM-Generated Deepfake Facial Images".

Jane: The gist The results imply that MLLMs can detect structural artefacts without relying on watermarks, but performance varies substantially across model generations.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper called "Beyond Generation: Using Multimodal LLMs as Detectors of LLM-Generated Deepfake Facial Images." It’s about whether these multimodal models can actually tell if an image is fake without needing some sort of digital watermark.

Jane: That sounds really practical, Tom. Basically, the big question here is if we can use these large language models to do a forensic analysis—to spot synthetic media—by looking for physical inconsistencies instead of just checking for embedded watermarks.

Lu: The authors are using six different Multimodal Large Language Models: GPT-4o, GPT-five GPT-five point two, Gemini two point five, Gemini three and Seed one point six to test this idea across a dataset of four hundred images #pg1GPT-4oGPT-5GPT-5 point 2Gemini 2 point 5Gemini 3Seed one point six <ref:2610.21760#pg1,GPT-4o, GPT-5, GPT-5.2, Gemini 2.5, Gemini 3>

Meng: A dataset of four hundred images—that’s the baseline for this study—with fifty being real photographs and three hundred fifty being generated by seven leading image models like Gemini three GPT-Image-one and Seedream four point zero #pg1 <ref:2610.21760#pg1>

Lalam: The core focus is isolating text-to-image synthesis, meaning they are specifically looking at images made entirely from text prompts, rather than just face swaps or identity manipulation techniques #pg2

Tom: Right, and the structure of their test involves three main stages: single-image classification, pairwise comparison against other models, and a focused experiment specifically on how the models detect things themselves.

Jane: That’s interesting because they aren't just looking at one model in isolation; they are testing how well these systems work when they compare different types of AI detectors against each other.

Lu: The methodology involves building that custom four hundred-image corpus and then setting up those three experimental tiers to address specific hypotheses about detection performance and reliability #pg3

Meng: And a key part of the setup is this "Model Justification Audit," where they manually check the natural-language reasons the AI gives for its decisions to make sure it’s actually reasoning or just making guesses about what it sees #pg3

The paper's summary: Tom: So, to sum up "Beyond Generation: Using Multimodal LLMs as Detectors of LLM-Generated Deepfake Facial Images," these researchers are showing that Multimodal Large Language Models can spot fake images without needing some hidden digital watermark.

Jane: It’s like they’re proving that the AI can look at the structure of an image, the physical details, and figure out it’s a forgery, instead of just checking for a specific sticker on it.

Lu: The core finding is that structural reasoning proves more reliable for forensic detection than relying on pixel-level data or any kind of metadata #pg3

Meng: I mean, so it’s not about the specific AI model you pick; it's about the underlying ability of the Multimodal Large Language Model to recognize physical inconsistencies in an image #pg3

Tom: Exactly. The study looked at six different models, and they found that GPT-five point two performed really well in a lot of tests, hitting nearly ninety-eight percent accuracy in one comparison #pg3 <ref:2610.21760#pg1>

Jane: And what’s interesting is how they handled the watermarks—they even found that removing those invisible SynthID signals actually hurt Gemini's detection accuracy by ten percent #pg3

Lu: The paper also pointed out that no hallucinations were recorded across any of the models in their manual audit, which suggests these models are doing genuine forensic reasoning rather than just guessing patterns #pg3

Meng: That’s a big deal for deployment because if they’re doing real reasoning, you can actually trust the justification they give you when they flag something #pg3

The paper's improvements: Tom: So, moving past the results, what are these authors suggesting we should do next with this research on detecting generated images #pg3

Jane: They’re basically saying we need to stop focusing so much on just one type of detection method and start focusing on the structural analysis aspect.

Lu: Yeah, they want to prioritize those structural artifacts over things like pixel-level noise or metadata because that’s where the reliable detection comes from #pg3

Meng: It means the next generation of detectors needs to be built around recognizing physical inconsistencies rather than just looking for a specific digital pattern #pg3

Tom: And they are calling out the masking phenomenon, which is when invisible watermarks actually hide those structural signals that these AI detectors need to work #pg3

Jane: They want the forensic community to reevaluate how we use invisible watermarking systems, not just for resisting attacks but for how they affect the structural signals #pg3

Lu: And they’re pointing out that because performance varies so much between different model generations, we can't just pick one detector and assume it will work everywhere #pg3

Meng: So, for the engineers shipping these models, the implication is that we need to stress-test these detectors against lower-quality imagery to make sure their reasoning holds up when the picture isn't perfect #pg3

Tom: I agree. They also suggested using degraded reference images in future experiments, like with "Ablated Anchors," to see if that performance drop is because of bad image quality or if the model can actually do the pairwise detection #pg3

Jane: And they want more human-baseline comparisons, so we get an idea of how these Multimodal Large Language Models’ justifications map against what actual human forensic experts see #pg3

Lu: They also mentioned that future audits should involve multiple auditors reviewing the AI's justifications and then averaging them to get a final verdict #pg3

Conclusion: Tom: So, to wrap up "Beyond Generation: Using Multimodal LLMs as Detectors of LLM-Generated Deepfake Facial Images," these authors are showing that structural reasoning is a much more reliable way to spot fake images than relying on watermarks.

Jane: It’s about prioritizing those physical inconsistencies in the image over digital signatures, which is a big deal for how we think about deepfake detection.

Lu: The implication is that we need to design detectors that focus on those structural artifacts instead of just pixel-level or metadata-based approaches #pg3

Meng: For us in production, it means the next generation of detection systems has to be built around recognizing physical inconsistencies rather than just looking for a specific digital pattern #pg3

Lalam: I'm confident that structural reasoning is the way forward because no hallucinations were recorded across any of the six models in their manual audit #pg3

Tom: And they also pointed out that watermarking systems themselves might be creating a masking phenomenon, which is when invisible watermarks actually hide those structural signals that these AI detectors depend on #pg3

Jane: It seems like we have to rethink how we use invisible watermarking, not just for resisting attacks but for how they affect the structural signals #pg3

Lu: And because performance varies so much between model generations, we can't just pick one detector and assume it will work everywhere #pg3

Meng: So, the next step for us in production is to stress-test these new detectors against lower-quality imagery to make sure their reasoning holds up when the picture isn't perfect #pg3

Tom: And they suggested using degraded reference images in future experiments, like with "Ablated Anchors," to see if that performance drop is because of bad image quality or if the model can actually do the pairwise detection #pg3

Jane: They also want more human-baseline comparisons, so we get an idea of how these Multimodal Large Language Models’ justifications map against what actual human forensic experts see #pg3

Lu: And they also mentioned that future audits should involve multiple auditors reviewing the AI's justifications and then averaging them to get a final verdict #pg3

Olivia Salamon, Virginia N. L. Franqueira, Shujun Li

Institute of Cyber Security for Society (iCSS) and School of Computing, University of Kent, London and South East University Group

cs.CV, cs.ET

Submitted: 2026-10-15

Updated: 2026-10-16

Code: https://github.com/OliviaSalamon/dissertationcorpus

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 75/100

The gist: The gist The results imply that MLLMs can detect structural artefacts without relying on watermarks, but performance varies substantially across model generations.

Key concepts

Structural Artefacts: These are physical inconsistencies within an image that indicate it was generated by AI. Examples include unnatural object geometry, inconsistent textures, or blurred backgrounds. MLLMs are trained to recognize these visual flaws as signs of synthetic media.
Watermark Ablation: This experiment involved removing visible and invisible watermarks from AI-generated images to see how it affected detection accuracy. The study found that while watermarks help, detection still reached 100% when all watermarks were removed, meaning structural analysis is the primary method.
Manual Justification Audit: This process involved having human auditors review the MLLM's reasoning for every classification decision. The results showed no hallucinations across models, suggesting that MLLMs are performing genuine forensic reasoning rather than just guessing patterns.
Masking Phenomenon: This refers to a situation where invisible watermarks suppress or hide the structural signals that MLLM detectors rely on. The study found that some watermarking systems, like SynthID, act as a subtle inhibitor of these necessary structural cues.

Summary

The gist The results imply that MLLMs can detect structural artefacts without relying on watermarks, but performance varies substantially across model generations.

Background and Aim

This study investigates whether Multimodal LLMs (MLLMs) can be used as a form of forensic analysis by identifying synthetic media Background/Aim As image generation models become increasingly capable of producing highly realistic content, traditional forensic techniques are struggling to keep up Specifically, the study aims to determine whether these models detect fake images by recognising physical inconsistencies, or whether they rely on digital artefacts such as embedded watermarks

Methods and Dataset Construction

The researchers benchmarked six MLLMs—GPT-4o, GPT-5, GPT-5.2, Gemini 3, Gemini 2.5, and Seed 1.6—using a dataset of 400 images Fifty images were authentic photographs while the remaining 350 were generated using seven leading image models: Gemini 3, Gemini 2.5, Flux.2 Pro, GPT-Image-1, GPT-Image-1.5, Seedream 4.0, and Seedream 4.5 The evaluation was structured in three stages: single-image classification, pairwise comparison, and a focused “self-detection” experiment on Gemini The dataset construction involved building a custom 400-image corpus, 50 authentic photographs from the Flickr-Faces-HQ dataset in conjunction with 350 images from seven top-ranked generators

Experimental Framework

The methodology developed a three-tiered framework to address four hypotheses

Experiment 1 (Single-Image Classification) provided the baseline detection data required to evaluate H1 (Self- and Cross-Model Detection) and H4 (Justification Reliability)
Experiment 2 (Pairwise Image Analysis) was designed to test H2 (Pairwise Image Advantage) by comparing accuracy against the Experiment 1 baseline
Experiment 3 (Gemini Watermark Ablation) directly addressed H3 (Effects of Watermarks) by isolating the contribution of visible and invisible watermark signals to detection accuracy

Key Findings on Detection Performance

Several MLLMs achieved a high detection accuracy, with GPT-5.2 reaching a peak of 97.9% in Experiment 2, the pairwise experiment, against specific generators and an overall mean pairwise accuracy of 82.5% by identifying structural or physical inconsistencies within images However, results from the self-detection experiment indicate that the presence of invisible SynthID watermarks leads to a 10% reduction in Gemini’s detection accuracy compared to the clean control group GPT-5.2 was the leading detector in every part of this study, achieving 78.8% accuracy in single-image analysis and 93.8% in the qualitative audit

Justification Reliability and Watermark Impact

The Manual Justification Audit revealed that no hallucinations occurred in any of the models and that every error recorded was simply a misclassification rather than a correct guess GPT-5.2 led the qualitative evaluation with 93.8% citing structural artefacts correctly such as unusual object geometry, unnatural image texture and blurred image backgrounds Experiment 3 showed that visible watermarks provide a significant detection boost (C2: p = 0.0075, r = 0.378), and SynthID functions as a strong independent signal in isolation (C5: p = 0.0020, r = 0.437) However, when all watermarks are removed entirely from the AI-generated images, detection still remains at 100% through native artefact analysis

Hypothesis Verdicts

H1 (Self- and Cross-Model Detection) was partially supported because GPT-5.2 and Gemini 2.5 showed modest same-ecosystem advantages, but generator image quality proved a far stronger predictor of detection difficulty than ecosystem alignment H4 (Justification Reliability) was partially rejected as no hallucinations were recorded across any model, suggesting genuine forensic reasoning rather than shortcut-driven pattern matching H3 (Effects of Watermarks) was partially rejected because detection still reached 100% through native artefact analysis even when all watermarks were removed H2 (Contextual Advantage) showed that the GPT-series demonstrated consistent descriptive gains, though these did not reach statistical significance, while Gemini 2.5 showed a statistically significant decline

Conclusion and Implications

Structural reasoning proves more reliable for forensic detection than pixel-level or metadata-based approaches The masking phenomenon identified within this study demonstrates that the forensic community should reevaluate how invisible watermarking systems are used, not only for their durability against adversarial attacks but also for their impact on the structural signals that many MLLM detectors depend upon While MLLMs should not yet replace human forensic experts, they do provide auditable natural-language justifications for each classification decision, which differentiates them from traditional CNN detectors Future work should focus on stress-testing MLLM detectors against lower-quality imagery and expanding the ablation corpus to confirm if these findings have direct implications for forensic watermark design

Limitations

The study relies exclusively on high-quality, studio style portraits from the FFHQ and state-of-the-art generative sources It remains unclear whether the “semantic reasoning” proficiency of detectors such as GPT-5.2 would maintain this performance under true forensic settings The absence of hallucinations across the six models within the Manual Justification Audit directly challenges the predictions of Sun et al. [37] and Li et al. [22]

Future Work

A dedicated “Pairwise Reasoning Audit” should use “Ablated Anchors” (degraded reference images) to reveal whether the performance drop within the tier 2 experiment results are a result of the image quality, or the models ability to perform pairwise detection Future research should incorporate a human-baseline comparison to provide an insight into how the MLLMs and their justifications map against human forensic experts Additionally, future audits should employ multiple auditors to review the MLLMs justifications and then get the average for the final verdict

References

[1] Artificial Analysis. 2025. Text to Image Leaderboard

[3] Nuria Alina Chandra, Ryan Murtfeldt, Lin Qiu, Arnab Karmakar, Hannah Lee, Emmanuel Tanumihardja, Kevin Farhat, Ben Caffee, Sejin Paik, Changyeon Lee, Jongwook Choi, Aerin Kim and Oren Etzioni. 2025. Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark

[5] Sumanth Dathathri, Abigail See, Sumedh Ghaisas, Po-Sen Huang, Rob McAdam, Johannes Welbl, Vandana Bachani, Alex Kaskasoli and Robert Stanforth. 2024. Scalable watermarking for identifying large language model outputs

[9] Apurva Gandhi and Shomik Jain. 2020. Adversarial Perturbations Fool Deepfake Detectors

[35] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.

Improvements for AI systems

Deepfake detection systems should be designed to prioritize structural artifact analysis over reliance on watermarking, as structural reasoning proves more reliable for forensic detection than pixel-level or metadata-based approaches. This allows for a more robust defense against Trace Removal Attack (TRA) described by Liu et al. [25] which targets embedded signals.
MLLM detectors should be implemented with mechanisms to mitigate the comparison noise effect observed in Gemini models, where the introduction of a real image appears to interfere with Gemini’s classification process rather than assist detection. This could involve developing specialized architectures that maintain consistent reasoning when presented with dual inputs.
Detection pipelines should incorporate an automated qualitative audit for natural-language justifications that actively checks for hallucinations, as the paper noted that no hallucinations were recorded across any model, suggesting these models engage in genuine forensic reasoning rather than shortcut-driven pattern matching. This ensures that high accuracy scores are based on genuine forensic evaluations by the MLLM rather than a statistical guess.
Watermarking systems should be redesigned to avoid creating a masking phenomenon where invisible watermarks suppress necessary structural signals, as demonstrated by the finding that SynthID is a subtle inhibitor of the native structural artefacts that underpin successful detection. This suggests future watermarking designs must account for their impact on MLLM forensic capabilities.
Model selection for deepfake detection should be based on generation quality metrics rather than ecosystem alignment, as generator image quality matters more than whether the image is a product of the same ecosystem generator. This moves the focus from H1's finding that ecosystem alignment was not a statistically significant predictor of baseline accuracy to a more practical metric.

Abstract

As image generation models become increasingly capable of producing highly realistic content, traditional forensic techniques are struggling to keep up. This paper investigates whether MLLMs can instead be used as a form of forensic analysis by identifying synthetic media. Specifically, the study aims to determine whether these models detect fake images by recognising physical inconsistencies, or whether they rely on digital artefacts such as embedded watermarks.Six MLLMs: GPT-4o, GPT-5, GPT-5.2, Gemini 2.5, Gemini 3, and Seed 1.6, are benchmarked using a dataset of 400 images. Fifty images are authentic photographs, while the remaining 350 are generated using seven leading image models: Gemini 3, Gemini 2.5, Flux.2 Pro, GPT-Image-1, GPT-Image-1.5, Seedream 4.0, and Seedream 4.5. The evaluation is structured in three stages: single-image classification, pairwise comparison, and a focused "self-detection" experiment on Gemini. Several MLLMs achieved a high detection accuracy, with GPT-5.2 reaching a peak of 97.9% in the pairwise experiment, against specific generators and an overall mean pairwise accuracy of 82.5% by identifying structural or physical inconsistencies within images. However, results from the self-detection experiment indicate that the presence of invisible SynthID watermarks leads to a 10% reduction in Gemini's detection accuracy compared to the clean control group. One possible explanation is that SynthID embedding reduces some of the visible artefacts the models rely on for detection.These results imply that MLLMs can detect structural artefacts without relying on watermarks, but performance varies substantially across model generations. The findings from comparison C5 also suggest that SynthID embedding may suppress some of the structural artefacts used during detection.