LaP-Forensics: Latent-Pixel Consistency Guided Multimodal Reasoning for Deepfake Detection

arXiv:2607.25962 · cs.CV · Submitted 2026-07-28 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "LaP-Forensics: Latent-Pixel Consistency Guided Multimodal Reasoning for Deepfake Detection".

Jane: Here's the gist of the paper. The gist The LaP-Forensics framework presents a multimodal approach that integrates RGB semantics with reconstruction-based forensic evidence to detect deepfakes, perform reasoning, and localize artifacts.

Jane: LaP-Forensics is a dual-stream framework that combines an RGB semantic stream and a DDIM reconstruction-residual stream for image detection, reasoning, and artifact localization The localization/reasoning model and the standalone detector evaluated in Section 4.4 use the same fixed DDIM residual construction but separate task heads and training objectives Dual-Stream Visual Representation Semantic Feature Extraction involves processing the input image x using a shared-weight frozen CLIP-ViT-L/14 encoder fCLIP to extract patch-level features Z(s) = phiproj(fCLIP (x)) To capture scene understanding and object-level semantics, we process the input image x using a shared-weight frozen CLIP encoder fCLIP to extract patch-level features Z(s) = phiproj(fCLIP (x)) Latent-Pixel Consistency Stream involves performing DDIM inversion with a fixed T = 50 schedule on a frozen Stable Diffusion v1.5 backbone to provide a fixed reconstruction reference The resulting residual map R is computed as the element-wise compatibility residual R = x − xˆ Regions with lower residuals are more readily reconstructed by that reference, whereas higher residuals indicate greater mismatch Feature Fusion and Image-Level Classification involves processing the projected patch tokens from both streams, X =

Z(s), Z(c), tprompt: , so the MLLM can attend to both semantic context and the reconstruction signal The standalone detector concatenates only the two

CLS: features c(s) and c(c) before its MLP head Two-Stage Training and Alignment 1.

Tom: Supervised Fine-Tuning (SFT) on curated forensic reasoning data is performed, where LoRA adapters are trained on the LLaMA-2-7B backbone together with the semantic projector phiproj, consistency projector psiproj, and text-to-prompt adapter ftp The comprehensive SFT loss LSFT integrates cross-entropy for text generation (LCE) with Dice (LDice) and Binary Cross-Entropy (LBCE) losses for precise mask supervision 2.

Paper summary: Tom: So we’ve been looking at LaP-Forensics, which is about using reconstruction evidence to help detect deepfakes and figure out where the fake parts are.

Jane: It’s a multimodal framework that mixes what an image looks like with how well it reconstructs when you try to build it back up.

Tom: Yeah, the authors are Wang et al., from places like Hong Kong Polytechnic University and Oxford. They're using a frozen Stable Diffusion model for that reconstruction part.

Jane: That frozen model acts as this fixed reference point against which they measure how compatible different parts of the input image are.

Tom: Right, they create this residual map where low numbers mean the area reconstructs easily with that reference, and high numbers mean it’s a mismatch.

Jane: And then they feed both the original RGB data and this residual map into a reasoning model to predict what's fake and where exactly the artifacts are.

Tom: The training is pretty smart, they use supervised fine-tuning followed by Group Relative Policy Optimization, or GRPO. It rewards good mask overlap but also how well their output structure matches what’s expected.

Jane: So, even though the text part isn't perfectly true in a deep sense, it’s trained to point to where the evidence—the residual map—is strong.

Tom: The key finding they highlight is that this reconstruction residual stream is super important; removing it causes a big performance drop in their mask accuracy, from seventy-two point one nine down to sixty-one point eight three <ref:2607.25962#pg3>.

Jane: That shows that just looking at the surface isn't enough; you need this internal signal about how the image was generated.

Tom: And they found that combining all their reward terms actually gave them the best results, hitting seventy-two point one nine mIoU and sixty-three point six two F1 score <ref:2607.25962#pg3>.

Jane: So what does this mean for us? It means we can build detectors that look deeper than just pixel differences by using the generative model's own internal process as a forensic clue.

Tom: Exactly, it shifts the focus from surface appearance to underlying latent consistency.

Jane: And they’re still looking at things like how reliable the textual explanations are when you do post-processing, which is an open area for future work.

Conclusion: Tom: So we’re wrapping up on LaP-Forensics, which is about using reconstruction evidence to help detect deepfakes and figure out where the fake parts are.

Jane: It’s a multimodal framework that mixes what an image looks like with how well it reconstructs when you try to build it back up.

Tom: The paper is called "LaP-Forensics: Latent-Pixel Consistency Guided Multimodal Reasoning for Deepfake Detection."

Jane: And the authors are Wang et al., from places like Hong Kong Polytechnic University and Oxford.

Tom: They used a frozen Stable Diffusion model for that reconstruction part, which acts as this fixed reference point.

Jane: That means they get a residual map that shows how compatible different parts of the input image are with that reference.

Tom: Low residuals mean it reconstructs easily, and high residuals mean it’s a mismatch.

Jane: And then they feed both the original RGB data and this residual map into a reasoning model to predict an artifact mask and some text analysis.

Tom: The training involved supervised fine-tuning followed by GRPO, which rewards good mask overlap plus how well the output structure matches what’s expected.

Jane: So, what this means is that surface appearance alone isn't enough anymore because generative models can make things look really convincing with few visual artifacts.

Tom: We need these kinds of deeper checks by looking at how well the image fits back into its generative model’s latent space.

Jane: The authors are trying to show that treating the residual not as a direct manipulation probability but as an auxiliary forensic piece of evidence unlocks better detection performance.

Tom: What this suggests for deployment is that you need a robust pipeline that can handle these two different streams and weigh them appropriately during inference for real-world use cases.

Jane: For listeners who only listen, the main thing to grasp is that if you want deepfake detection beyond just spotting obvious flaws, you need to look at how well the image fits back into its latent space.

Tom: That’s exactly it; it shifts the focus from surface appearance to underlying latent consistency.

Jane: So, while this framework shows strong detection and localization capabilities on benchmarks like UniversalFakeDetect and SynthScars, they still have some limitations with how reliable those textual explanations are under post-processing.

Hong Kong Polytechnic University · University College London · Tsinghua University · Sun Yat-sen University · National University of Singapore

cs.CV

Submitted: 2026-07-28

Updated: 2026-10-20

Comments: Accepted at ACM Multimedia 2026 (ACM MM 2026)

Code: https://github.com/QwenLM/Qwen3-VL

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist The LaP-Forensics framework presents a multimodal approach that integrates RGB semantics with reconstruction-based forensic evidence to detect deepfakes, perform reasoning, and localize

Key concepts

RGB Semantics: This refers to the standard visual understanding of an image, capturing what is visible in terms of color, texture, and objects. The framework uses a frozen CLIP encoder to extract these features from the input image so the AI understands the scene's general content and object relationships.
DDIM Inversion-Reconstruction: This process uses a fixed Stable Diffusion model to mathematically reverse the generation process (inversion) and then try to recreate it (reconstruction). The difference between these two versions creates a 'residual map,' which highlights areas where the image doesn't match the original generation.
Where-What-Why Model: This is a structured reasoning model that takes both the visual features (RGB and residual map) and a text prompt as input. It is designed to predict three things: 'Where' an artifact is, 'What' the artifact is, and 'Why' it might be present, leading to textual analysis.
Group Relative Policy Optimization (GRPO): This is a specialized training method used after initial fine-tuning. It optimizes the model by rewarding it not just for correctly identifying artifact masks, but also for producing structured outputs that align with the evidence and textual constraints, improving both localization and reasoning quality.

Terminology

Consistency — 2× in this paper · explained in 3 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
GRPO — 1× in this paper · explained in 6 episodes
Generative Model — 1× in this paper · explained in 2 episodes
Generative Models — 1× in this paper · explained in 3 episodes
Generator — 1× in this paper · explained in 2 episodes
Group Relative Policy Optimization (GRPO) — 1× in this paper · explained in 9 episodes
Multimodal Reasoning — 1× in this paper · explained in 5 episodes
Verifier — 1× in this paper · explained in 2 episodes

Summary

The gist The LaP-Forensics framework presents a multimodal approach that integrates RGB semantics with reconstruction-based forensic evidence to detect deepfakes, perform reasoning, and localize artifacts.

How it works

LaP-Forensics is a dual-stream framework that combines an RGB semantic stream and a DDIM reconstruction-residual stream for image detection, reasoning, and artifact localization The localization/reasoning model and the standalone detector evaluated in Section 4.4 use the same fixed DDIM residual construction but separate task heads and training objectives

Dual-Stream Visual Representation

** Semantic Feature Extraction involves processing the input image x using a shared-weight frozen CLIP-ViT-L/14 encoder fCLIP to extract patch-level features Z(s) = phiproj(fCLIP (x)) To capture scene understanding and object-level semantics, we process the input image x using a shared-weight frozen CLIP encoder fCLIP to extract patch-level features Z(s) = phiproj(fCLIP (x)) **

** Latent-Pixel Consistency Stream involves performing DDIM inversion with a fixed T = 50 schedule on a frozen Stable Diffusion v1.5 backbone to provide a fixed reconstruction reference The resulting residual map R is computed as the element-wise compatibility residual R = x − xˆ Regions with lower residuals are more readily reconstructed by that reference, whereas higher residuals indicate greater mismatch **

** Feature Fusion and Image-Level Classification involves processing the projected patch tokens from both streams, X = [Z(s), Z(c), tprompt], so the MLLM can attend to both semantic context and the reconstruction signal The standalone detector concatenates only the two [CLS] features c(s) and c(c) before its MLP head **

Two-Stage Training and Alignment

Supervised Fine-Tuning (SFT) on curated forensic reasoning data is performed, where LoRA adapters are trained on the LLaMA-2-7B backbone together with the semantic projector phiproj, consistency projector psiproj, and text-to-prompt adapter ftp The comprehensive SFT loss LSFT integrates cross-entropy for text generation (LCE) with Dice (LDice) and Binary Cross-Entropy (LBCE) losses for precise mask supervision
Outcome-Driven Alignment (Stage II) uses Group Relative Policy Optimization (GRPO) to combine mask accuracy with observable text-structure constraints The reward is Rtotal = alpha maskRmask + alpha formatRformat, with alpha mask = 0.7 and alpha format = 0.6 This stage jointly improves artifact localization and structured evidence referencing, but does not guarantee the semantic faithfulness of the generated explanations

Key Findings and Ablation Studies

** The reconstruction residual is the most critical component, as removing it causes the largest performance drop in mIoU from 72.19 to 61.83 The multi-step DDIM residual clearly outperforms both RGB-only input and the single-step VAE residual, with T = 50 providing the best balance among the tested inversion horizons **

** The reward ablation shows that combining all reward terms achieves the best overall results of 72.19 mIoU, 63.62 F1, and 97.5% CoT-FormatRate Adding the format reward substantially improves performance to 70.84 mIoU and 61.15 F1, together with a CoT-Format-Rate of 94.6% **

** Counterfactual interventions confirm that the predicted masks materially depend on the consistency map, as removing the consistency signal reduces mIoU@0.5 from 0.721 to 0.604 The residual map is neither a calibrated manipulationprobability map nor an indicator of the source generator **

In conclusion, LaP-Forensics integrates RGB semantics with DDIM-derived reconstruction residuals and a structured Where–What–Why protocol to provide explainable deepfake detection and artifact localization The framework demonstrates strong cross-generator detection performance and competitive artifact localization under the evaluated protocols. The best result in each group is highlighted in bold for the ablation studies.

--- Page 1 ---

LaP-Forensics: Latent-Pixel Consistency Guided Multimodal Reasoning for Deepfake Detection Can Wang The Hong Kong Polytechnic University Hong Kong, China Yuhao Wang University College London London, United Kingdom Yushe Cao Tsinghua University Beijing, China Canran Xiao Sun Yat-sen University Guangzhou, China Fei Shen∗ National University of Singapore Singapore, Singapore Abstract Recent generative models can produce images with few obvious visual artifacts, weakening detectors and explanations that rely only on surface appearance We present LaP-Forensics, a multimodal framework that augments RGB semantics with reconstructionbased forensic evidence A frozen Stable Diffusion DDIM inversion–reconstruction model provides a fixed reconstruction reference, and its residual map measures local compatibility with that reference Independent projectors encode the RGB image and residual map before a structured Where–What–Why model predicts a textual analysis and an artifact mask Supervised fine-tuning is followed by Group Relative Policy Optimization (GRPO), whose reward combines mask overlap with output-structure and evidencereference terms Experiments show cross-generator detection on UniversalFakeDetect and competitive artifact localization on the official SynthScars benchmark Controlled cue-construction, inversion-horizon, component, reward-term, and counterfactual analyses support the utility of the residual stream under the evaluated settings, while free-form textual faithfulness and reliability under post-processing remain open limitations

--- Page 2 ---

Wang et al., Can Wang, Yuhao Wang, Yushe Cao, Canran Xiao, and Fei Shen LaP-Forensics (Latent-Pixel Forensics) complements RGB semantics with an explicit reconstruction-based compatibility signal for explainable image forensics Specifically, a frozen Stable Diffusion v1.5 model performs deterministic DDIM inversion and reconstruction, providing a fixed reference against which the input image is compared The resulting residual map characterizes local reconstruction compatibility: regions with lower residuals are more readily reconstructed by the reference model, whereas higher residuals indicate greater mismatch Rather than treating this residual as a direct manipulation probability or source-attribution cue, LaP-Forensics encodes it jointly with RGB content as auxiliary forensic evidence Building upon the Where–What–Why reasoning paradigm [56], the model jointly produces structured forensic analyses and pixel-level artifact masks To further align its outputs with the supplied evidence, we introduce a GRPO-based objective that rewards localization quality, structural compliance, and explicit references to the residual map

--- Page 3 ---

LaP-Forensics: Latent-Pixel Consistency Guided Multimodal Reasoning for Deepfake Detection Wang et al., Can Wang, Yuhao Wang, Yushe Cao, Canran Xiao, and Fei Shen LaP-Forensics complements RGB semantics with an explicit reconstruction-based compatibility signal for explainable image forensics Specifically, a frozen Stable Diffusion v1.

Improvements for AI systems

Bold header: Latent-Pixel Consistency Guidance for Artifact Localization

The improved system can perform cross-generator detection on UniversalFakeDetect and competitive artifact localization on the official SynthScars benchmark by using a DDIM-derived compatibility residual computed against a fixed reconstruction reference to measure local compatibility with that reference.

Bold header: Structured, Evidence-Referenced Reasoning

The system can generate structured forensic analyses by employing a Where–What–Why model and an objective that optimizes the model to refer to the consistency map through a reward term described as evidence-reference regularization rather than a guarantee of explanation faithfulness.

Bold header: Joint Optimization of Localization and Structure

The improved MLLM-based agent can be trained using Group Relative Policy Optimization (GRPO) whose reward combines mask overlap with output-structure and evidencereference terms, leading to better alignment between the predicted artifacts and the required forensic structure.

Bold header: Robustness to Ambiguous Textual Claims

The system is designed such that it can handle scenarios where free-form textual faithfulness and reliability under post-processing remain open limitations by using a reward function focused on structural and evidence-reference regularization rather than a faithfulness reward.

Bold header: Enhanced Spatial Cue Sensitivity

By utilizing the DDIM residual, the system can be tuned to better identify localized mismatches, as the ablation studies show that the multi-step inversion–reconstruction process yields a more informative spatial compatibility pattern than a single-step reconstruction and that T = 50 provides the best balance between residual sensitivity and localization reliability.

Abstract

Recent generative models can produce images with few obvious visual artifacts, weakening detectors and explanations that rely only on surface appearance. We present LaP-Forensics, a multimodal framework that augments RGB semantics with reconstruction-based forensic evidence. A frozen Stable Diffusion DDIM inversion-reconstruction model provides a fixed reconstruction reference, and its residual map measures local compatibility with that reference. Independent projectors encode the RGB image and residual map before a structured Where-What-Why model predicts a textual analysis and an artifact mask. Supervised fine-tuning is followed by Group Relative Policy Optimization (GRPO), whose reward combines mask overlap with output-structure and evidence-reference terms. These text-side terms encourage the model to refer to the consistency map but do not constitute a verifier of free-form textual truth. A separate image-level head fuses RGB and DDIM-residual class features. Experiments show cross-generator detection on UniversalFakeDetect and competitive artifact localization on the official SynthScars benchmark. Controlled cue-construction, inversion-horizon, component, reward-term, and counterfactual analyses support the utility of the residual stream under the evaluated settings, while free-form textual faithfulness and reliability under post-processing remain open limitations.