SIGMA: Semantic-Difference Instruction-Grounding Mask Annotator for Text-Driven Image Manipulation Localization

The gist: SIGMA proposes a two-branch framework that fuses semantic-feature differencing with instruction grounding via bidirectional cross-modal refinement to automatically generate pixel-level supervision for image manipulation localization from public editing corpora.

How it works

SIGMA redefines scalable IML annotation as semantic change localization, demonstrating that pixel differencing and instruction grounding are complementary but individually insufficient for diffusion-based edits The framework is powered by four key components: a semantic difference branch, an instruction-grounding branch, a bidirectional cross-modal refinement module, and a mask decoder <ref:2605.27924#pg2>.

The Semantic-Difference Branch utilizes a frozen DINOv2-Base encoder to extract multi-level features at three designated layers L = l1, l2, and l3 <ref:2605.27924#pg5>. This branch computes difference representations by calculating absolute difference d l abs and cosine similarity d l cos for normalized features, fusing them into a shared embedding d l <ref:2605.27924#pg6>. These raw difference features are then refined using Transformer-based Difference Refinement Blocks (DRBs) to produce a dense per-patch visual evidence representation E(0)v <ref:2605.27924#pg7>. This branch answers the question: where does the semantic content of the image pair differ? <ref:2605.27924#pg8>.

The Instruction-Grounding Branch supplies a structurally independent stream of evidence distilled from the editing instruction T, which is first parsed into a tuple (c o, c e, op) using a Semantic Transformation Parser (STP) based on Qwen2.5-0.5B <ref:2605.27924#pg9>. This branch then uses LangSAM to derive concept-conditioned attention maps Ao and Ae <ref:2605.27924#pg10>. The Action-Conditioned Spatial Fusion (ACSF) module encodes the spatial relationship between Ao and Ae using an action-conditioned rule Mintent(p) based on the operation type op <ref:2605.27924#pg11>. This branch answers where the change was intended, but it is not designated as authoritative <ref:2605.27924#pg16>.

Bidirectional Cross-Modal Refinement

The Bidirectional Cross-Modal Refinement (BCMR) module fuses the two branches by stacking cross-attention blocks with an asymmetric order <ref:2605.27924#pg12>. Specifically, the semantic-difference branch first attends to the current prior to amplify edit-relevant signals, while the instruction branch attends to this freshly updated state to re-center the prior on regions with strongest visual support <ref:2605.27924#pg10>. This ordering enforces an asymmetry where semantic differencing carries the primary signal, while instruction conditioning continuously self-refines <ref:2605.27924#pg13>. After K iterations, the two refined representations are fused into a shared consensus space Z = Proj E(K)v E(K)i <ref:2605.27924#pg11>.

Noise-Calibrated Progressive Domain Adaptation

To bridge supervision scarcity and the diffusion-domain shift, SIGMA employs a two-stage training strategy <ref:2605.27924#pg14>. Stage I pre-trains on inpainting pairs with exact masks via a loss Lseg = BCE(f(I o inp, Ie inp), Minp) + Dice(f(I o inp, Ie inp), Minp) to yield a strong prior <ref:2605.27924#pg14>. Stage II adapts SIGMA to unlabeled text-driven pairs from public corpora using three self-supervised signals: (i) Noise calibration via Lcalib = BCE(f(I o, ˆI o), 0) where ˆI o is a VAE-roundtrip noise calibrated image <ref:2605.27924#pg14>. (ii) EMA selftraining via Lpl = C(M̂) · BCE(f(I o, Ie), sg(M̂)) to generate pseudo-labels <ref:2605.27924#pg14>. (iii) Edit-noise disentanglement via Ldisent = 1/P P i∈omegap Pi max(0,⟨d(i) edit, d(i) noise⟩) to suppress diffusion-induced false positives <ref:2605.27924#pg14>. The full objective is LStage II = 10Lseg + 0.1Lcalib + 0.5Lpl + 0.5Ldisent <ref:2605.27924#pg14>.

Results and Impact

SIGMA outperforms existing automatic mask generators by +12.20% F1 on five benchmarks and boosts six diverse detectors by +18.34% F1 in cross-dataset generalization when applied to public editing corpora <ref:2605.27924#pg4>. The framework establishes a new state of the art with 89.83% average F1 and 83.52% IoU after Stage II, surpassing the strongest baseline by +12.20% on average <ref:2605.27924#pg8>. Crucially, gains appear most notable on the toughest benchmarks, such as DEAL-300K, where SIGMA achieves a striking +55.53% F1 over DDPM-CD <ref:2605.27924#pg8>. Furthermore, fine-tuning six representative IML models on SIGMA-annotated data yields mean F1 improvements from +4.58% (PSCC-Net) to +30.79% (MVSSNet) <ref:2605.27924#pg9>. This confirms that SIGMA serves as a model-agnostic supervisory resource for the IML community <ref:2605.27924#pg3>.

Limitations

The instruction-grounding branch is intrinsically vulnerable to the precision limits of off-the-shelf parsers and grounders, as illustrated by a failure mode where it over-extracts context, causing contamination of the intent map <ref:2605.27924#pg16>. Therefore, the central design decision mandates that semantic differencing and instruction grounding must serve as mutually verifiable sources rather than either being trusted alone <ref:2605.27924#pg16>. The framework's gains are most pronounced on diffusion-edit benchmarks, suggesting that while it addresses the supervision scarcity and domain shift, it does not inherently solve the problem of pixel-level ground truth acquisition without this specific mechanism <ref:2605.27924#pg8>.

SIGMA aims to strengthen detection of text-driven forgeries and thereby support media verification <ref:2605.27924#pg11>. The same automated mask-generation capability could in principle refine adversarial editing pipelines; however, SIGMA improves only the detector side rather than the editor, and we believe the net benefit to media integrity is positive <ref:2605.27924#pg11>. This approach converts four large-scale datasets into a million-scale IML training set that boosts six diverse detectors by +18.34% F1 in cross-dataset generalization, establishing SIGMA-annotated data as a model-agnostic supervisory resource for the IML community <ref:2605.27924#pg3>. The paper presents SIGMA, a framework that turns publicly available image-editing datasets into pixelaccurate supervision for image manipulation localization (IML) <ref:2605.27924#pg11>. Building on the observation that pixel differencing and instruction-only grounding are complementary but individually insufficient under diffusion-based editing, SIGMA performs semantic-feature differencing in a frozen foundation backbone and fuses it with a parsed-instruction prior via bidirectional cross-modal refinement, while a two-stage training strategy bridges supervision scarcity and the diffusion-domain shift <ref:2605.27924#pg11>. The paper presents SIGMA, a framework that turns publicly available image-editing datasets into pixelaccurate supervision for image manipulation localization (IML) <ref:2605.27924#pg11>. Building on the observation that pixel differencing and instruction-only grounding are complementary but individually insufficient under diffusion-based editing, SIGMA performs semantic-feature differencing in a frozen foundation backbone and fuses it with a parsed-instruction prior via bidirectional cross-modal refinement, while a two-stage training strategy bridges supervision scarcity and the diffusion-domain shift <ref:2605.27924#pg11>. The paper presents SIGMA, a framework that turns publicly available image-editing datasets into pixelaccurate supervision for image manipulation localization (IML) <ref:2605.27924#pg11>. Building on the observation that pixel differencing and instruction-only grounding are complementary but individually insufficient under diffusion-based editing, SIGMA performs semantic-feature differencing in a frozen foundation backbone and fuses it with a parsed-instruction prior via bidirectional cross-modal refinement, while a two-stage training strategy bridges supervision scarcity and the diffusion-domain shift <ref:2605.27924#pg11>. The paper presents SIGMA, a framework that turns publicly available image-editing datasets into pixelaccurate supervision for image manipulation localization (IML) <ref:2605.27924#pg11>. Building on the observation that pixel differencing and instruction-only grounding are complementary but individually insufficient under diffusion-based editing, SIGMA performs semantic-feature differencing in a frozen foundation backbone and fuses it with a parsed-instruction prior via bidirectional cross-modal refinement, while a two-stage training strategy bridges supervision scarcity and the diffusion-domain shift <ref:2605.27924#pg11>. The paper presents SIGMA, a framework that turns publicly available image-editing datasets into pixelaccurate supervision for image manipulation localization (IML) <ref:2605.27924#pg11>. Building on the observation that pixel differencing and instruction-only grounding are complementary but individually insufficient under diffusion-based editing, SIGMA performs semantic-feature differencing in a frozen foundation backbone and fuses it with a parsed-instruction prior via bidirectional cross-modal refinement, while a two-stage training strategy bridges supervision scarcity and the diffusion-domain shift <ref:2605.27924#pg11>.

1. Bold semantic-feature differencing via frozen DINOv2: The system can perform semantic-feature differencing in a vision foundation backbone to identify where the content actually differs, providing a robust signal that overcomes limitations of pixel-level differencing under diffusion noise.

2. Bidirectional cross-modal refinement for instruction injection: The system can inject an instruction-derived spatial prior into this visual stream via bidirectional crossmodal refinement, amplifying the difference signal at intended-edit regions when the editor faithfully realizes user intent.

3. Two-stage training strategy for domain shift: The system can bridge supervision scarcity and diffusion-domain shift by Stage I pre-training on inpainting masks and Stage II adaptation using VAE-roundtrip zero-edit calibration, exponential moving average (EMA) self-training, and an edit-noise feature disentanglement loss.

4. Million-scale dataset construction: The system can construct a million-scale IML training set from public editing corpora at no regeneration or manual annotation cost, providing a model-agnostic supervisory resource for IML across diverse editing operations.

5. Semantic transformation parsing of instructions: The system can extract the core edit intent by using a Semantic Transformation Parser (STP) to derive structured tuples like (original concept, edited concept, action type) from text instructions.

6. Action-conditioned spatial fusion for prior guidance: The system can utilize Action-Conditioned Spatial Fusion (ACSF) to encode structurally different spatial relationships between concepts, ensuring the predicted mask reflects the specific operation requested in the instruction, such as A e(p) op = add.

7. Noise-calibrated progressive domain adaptation: The system can suppress diffusion artifacts by employing edit-noise feature disentanglement loss, effectively distinguishing between true edits and diffusion-induced false positives without needing target-domain annotation.

Text-driven image editing has advanced rapidly, but reliably localizing these manipulations requires image manipulation localization (IML) models trained on large pixel-annotated datasets, and there is still no low-cost way to obtain such training data at scale. We observe that these data already exist in disguise: public editing datasets contain millions of structurally identical(original, edited) pairs to IML training samples, lacking only pixel-level masks. Recovering these masks automatically is non-trivial: pixel differencing is overwhelmed by diffusion-induced perturbations across all pixels, and instruction-only grounding localizes only what the prompt describes, missing unintended editor side-effects. We propose SIGMA (S emantic-difference I nstruction- G rounding M ask A nnotator), which performs semantic-feature differencing in a vision foundation backbone and injects an instruction-derived spatial prior into this visual stream via bidirectional cross-modal refinement, amplifying the difference signal at intended-edit regions when the editor faithfully realizes user intent. SIGMA is trained in two complementary stages: Stage I supervises on inpainting masks and Stage II closes the diffusion-domain shift via VAE-roundtrip noise calibration, EMA self-training, and an edit-noise disentanglement loss. SIGMA outperforms existing automatic mask generators on five benchmarks (+12.20% F1,+11.16% IoU). When applied to public editing corpora, it produces a about 1.1M IML training set that improves four diverse detectors by+20.98% F1 across five datasets, turning previously unused editing data into a model-agnostic supervisory resource for IML. For reproducibility, our project page is available at.