Adaptive Forensic Feature Refinement via Intrinsic Importance Perception

The gist: I2P proposes an SID framework centered on intrinsic importance perception that adaptively identifies critical layer representations and constrains task-driven parameter updates to improve cross-distribution generalization while preserving pretrained structures.

Motivation and Analysis

Pretrained visual foundation models (VFM) have the potential to improve the generalization capabilities of synthetic image detection (SID) by learning general-purpose visual representations from large-scale datasets <ref:2604.16879#pg4>. Existing VFM-based SID transfer strategies are often coarse-grained, either directly using final layer representations or simply fusing multi layer features, lacking explicit modeling of the optimal representational hierarchy for transferable forgery cues <ref:2604.16879#pg2>. The paper hypothesizes that the effective discriminative information relied upon by SID is more concentrated in the intermediate layer representations of VFM <ref:2604.16879#pg4>. This is supported by an analysis using the Information Bottleneck (IB) principle, which posits that an ideal representation should preserve information relevant to the target while maximally compressing task-irrelevant details <ref:2604.16879#pg4>. The analysis of layer-wise representations using Gram matrix entropy and effective rank shows that both metrics reach their peaks around the intermediate layers, suggesting they form a critical intermediate layer suitable for SID readout <ref:2604.16879#pg4>.

Methodology: Critical Layer Identification (CLI)

To address the question of where forgery cues should be extracted from, I2P introduces Critical Layer Identification (CLI) to adaptively locate the critical layer representation that is most discriminative for SID <ref:2604.16879#pg6>. CLI characterizes the relative contributions of representations from different layers to the SID task by introducing a learnable layer scoring function, where each candidate representation fl is assigned a score α (fl) <ref:2604.16879#pg6>. This score is calculated as α (fl) = W2 (tanh(W1 fl) + b1) + b2, and these scores are then normalized via softmax to obtain a layer-wise weight distribution πl <ref:2604.16879#pg6>. An aggregated representation ˆf is constructed based on this distribution, and the prediction logits z are obtained by feeding it into the classification head g(·) <ref:2604.16879#pg6>. The critical layer l∗ is then selected as arg max l πl <ref:2604.16879#pg6>. This process allows for the selection of the single layer with the greatest discriminative value, avoiding information dilution from indiscriminate use of multiple layers <ref:2604.16879#pg6>.

Methodology: Controlled Knowledge Injection (CKI)

To address how to inject task knowledge while minimizing disturbance to the pretrained structure, I2P proposes Controlled Knowledge Injection (CKI) <ref:2604.16879#pg5>. CKI constrains parameter updates driven by downstream supervisory signals within a low-sensitivity parameter subspace <ref:2604.16879#pg4>. The local sensitivity of different parameter directions is evaluated using a practical approximation based on second-order information to estimate parameter importance <ref:2604.16879#pg6>. Specifically, for a linear mapping layer W ∈ R m×d, the local curvature is approximated by H ≈ 1/n X⊤X + λI <ref:2604.16879#pg6>. The importance Sⱼ of parameter wiⱼ is defined as Sⱼ = w2 i j [H−1]j j <ref:2604.16879#pg6>. After obtaining these estimates, the lowest η% of parameters in terms of importance are selected as the updatable subset <ref:2604.16879#pg6>. The parameter update process is then constrained by θ ← θ −γ (M ⊙ ∇θ L), where M is a binary mask matrix indicating which parameters are allowed to be updated <ref:2604.16879#pg6>.

Experimental Results and Contributions

The systematic experiments conducted on multiple cross-domain benchmarks verify the effectiveness of I2P <ref:2604.16879#pg4>. In Setting 1, where models are trained on SDv1.4 and evaluated on eight generators, I2P achieves an average ACC of 98.22% and an average AP of 99.72%, attaining the best overall performance <ref:2604.16879#pg6>. In Setting 3, which covers more challenging cross-distribution benchmarks like Chameleon and COSPY, I2P achieves the best performance on both, demonstrating stronger generalization to real distribution shifts and emerging generators <ref:2604.16879#pg6>. Ablation studies confirm that introducing either CLI or CKI alone consistently improves performance, indicating that the two components are complementary <ref:2604.16879#pg6>. Furthermore, testing across different VFM backbones shows that I2P consistently yields performance gains across architectures and model scales, suggesting its strategy is transferable <ref:2604.16879#pg6>.

Conclusion

I2P unifies the selection of critical representational hierarchy and controlled task adaptation into a single framework, thereby enhancing the discriminative capability of SID while more robustly preserving the cross-distribution transfer capability of VFM <ref:2604.16879#pg3>. The framework successfully addresses the tension between identifying suitable forgery cues and constraining disturbance to the pretrained structure through adaptive identification and controlled injection <ref:2604.16879#pg6>. Although I2P shows strong generalization ability in cross-distribution SID, it still has several limitations, including a lack of rigorous theoretical justification and limited investigation into its applicability to finer-grained forensic scenarios <ref:2604.16879#pg7>. The overall findings validate that selective adaptation based on intrinsic importance perception is a robust strategy for VFM in SID <ref:2604.16879#pg6>.

--- Page 1 ---

Adaptive Forensic Feature Refinement via Intrinsic Importance Perception

Perception Jiazhen Yang Zhejiang University Hangzhou, China jiazhenyang@zju.edu.cn Junjun Zheng Alibaba Inc Hangzhou, China fangcheng.zjj@alibaba-inc.com Kejia Chen Zhejiang University Hangzhou, China kejiachen@zju.edu.cn Xiangheng Kong Alibaba Inc Hangzhou, China yongheng.kxh@alibaba-inc.com Jie Lei Zhejiang University of Technology Hangzhou, China jasonlei@zjut.edu.cn Zunlei Feng Zhejiang University Hangzhou, China zunleifeng@zju.edu.cn Bingde Hu Zhejiang University Hangzhou, China tonyhu@zju.edu.cn Yang Gao Zhejiang University Hangzhou, China roygao@zju.edu.cn

Abstract With the rapid development of generative models and multimodal content editing technologies, the key challenge faced by synthetic image detection (SID) lies in cross-distribution generalization to unknown generation sources <ref:2604.16879#pg2>. In recent years, visual foundation models (VFM), which acquire rich visual priors through large scale imagetext alignment pretraining, have become a promising technical route for improving the generalization ability of SID <ref:2604.16879#pg2>. However, existing VFM-based methods remain relatively coarse-grained in their adaptation strategies <ref:2604.16879#pg2>. They typically either directly use the final layer representations of VFM or simply fuse multi layer features, lacking explicit modeling of the optimal representational hierarchy for transferable forgery cues <ref:2604.16879#pg2>. Meanwhile, although directly fine-tuning VFM can enhance task adaptation, it may also damage the cross-modal pretrained structure that supports open-set generalization <ref:2604.16879#pg2>. To address this task specific tension, we reformulate VFM adaptation for SID as a joint optimization problem: it is necessary both to identify the critical representational layer that is more suitable for carrying forgery discriminative information and to constrain the disturbance caused by task knowledge injection to the pretrained structure <ref:2604.16879#pg2>. Based on this, we propose I2P, an SID framework centered on intrinsic importance perception <ref:2604.16879#pg2>. I2P first adaptively identifies the critical layer representations that are most discriminative for SID, and then constrains task-driven parameter updates within a low sensitivity parameter subspace, thereby improving task specificity while preserving the transferable structure of pretrained representations as much as possible <ref:2604.16879#pg2>. We conduct systematic experiments and ablation analyses on multiple cross-domain benchmarks, and the results show that I2P achieves stable and competitive generalization performance on cross-distribution synthetic image detection tasks <ref:2604.16879#pg2>.

1. Bold Header: Adaptive Critical Layer Identification (CLI) for Feature Extraction

This improvement allows a VFM-based SID system to adaptively identify the critical layer that is most discriminative for SID from the hierarchical representations of a frozen VFM, thereby determining a more appropriate representational readout location. This ensures that subsequent detection relies only on the layer where forgery cues are most robust, avoiding information dilution that may be caused by indiscriminate use of multiple layers.

2. Bold Header: Controlled Knowledge Injection (CKI) for Parameter Stability

This mechanism constrains parameter updates to preserve the pretraining structure by restricting changes to a low-sensitivity subspace, as described by the approximation: L (thetapre + Δtheta) ≈ L (thetapre) + 1/2 Δtheta⊤H Δtheta. This prevents representational drift and ensures adaptation is achieved while minimizing disturbance to the pretrained representational structure as much as possible.

3. Bold Header: Unified Framework for Cross-Distribution Generalization

The I2P framework unifies layer selection and parameter control, meaning it can enhance the discriminative capability of SID while more robustly preserving the cross-distribution transfer capability of VFM. This results in a system that achieves stable and competitive generalization performance on cross-distribution synthetic image detection tasks across unseen generative models.

4. Bold Header: Robustness Against Generator Distribution Shifts

By focusing on intrinsic importance rather than relying solely on pixel-level artifacts, the improved system exhibits better stability against distribution shifts. The paper notes that methods exhibit more pronounced performance degradation on generators that differ substantially from the training distribution, whereas I2P shows more stable consistency across different test generators.

5. Bold Header: Versatility Across VFM Architectures

The proposed strategy demonstrates generality, as the paper shows that I2P consistently yields performance gains across architectures and model scales, indicating it is a transferable SID adaptation strategy rather than being tied to a specific pretrained model or fixed layer hierarchy.

With the rapid development of generative models and multimodal content editing technologies, the key challenge in synthetic image detection (SID) lies in cross-distribution generalization to unknown generation sources. Visual foundation models (VFM), which acquire rich visual priors through large scale image-text alignment pretraining, have become a promising approach to improving SID generalization. However, existing VFM-based methods remain coarse-grained in their adaptation strategies. They typically either directly use the final layer representations of VFM or simply fuse multi layer features, lacking explicit modeling of the optimal representational hierarchy for transferable forgery cues. Although fine-tuning VFM can enhance task adaptation, it may also damage the cross-modal pretrained structure that supports open-set generalization. To address this task specific tension, we reformulate VFM adaptation for SID as a joint optimization problem: it is necessary both to identify the critical representational layer that is more suitable for carrying forgery discriminative information and to constrain the disturbance caused by task knowledge injection to the pretrained structure. Based on this, we propose I2P, an SID framework centered on intrinsic importance perception. I2P first adaptively identifies the critical layer representations that are most discriminative for SID, and then constrains task-driven parameter updates within a low sensitivity parameter subspace, thereby improving task specificity while preserving the transferable structure of pretrained representations as much as possible. We conduct systematic experiments and ablation analyses on multiple cross-domain benchmarks, and the results show that I2P achieves stable and competitive generalization performance.