Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation

arXiv:2605.13122 · cs.CV · Submitted 2026-05-13 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation".

Jane: Here's the gist of the paper. The gist The work proposes a training-free framework that repurposes pretrained image editing models for zero-shot referring image segmentation by exploiting their intermediate representations, revealing strong foreground-background separability at the earliest denoising timestep.

Jane: The approach reformulates referring image segmentation as an instruction-based image editing problem, showing that generative models inherently support discriminative visual grounding without requiring task-specific supervision.

Paper summary: Tom: So, wrapping up the "Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation" paper, the authors are essentially showing us that instruction-based image editing models have an inherent capacity for visual semantic grounding <ref:2605.13122#pg1>.

Jane: They found this by looking at how these models process instructions to modify regions, and they showed that strong foreground-background separability appears in the internal representations right at the first denoising timestep <ref:2605.13122#pg1>.

Lu: So what this means is that we don't need pixel-level annotations for referring image segmentation anymore; we can use these models to do it in a zero-shot manner by repurposing their existing internal workings <ref:2605.13122#pg3>.

Meng: It’s about using what the model already knows, which is smart, but we still have to be careful about the quality of that initial training data when we deploy this at scale <ref:2605.13122#pg3>.

Tom: The framework solves this by breaking localization down into attention-based spatial priors and feature-based semantic discrimination, and they showed it works with a speedup over RefAM <ref:2605.13122#pg3>.

Jane: The title of the paper, "Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation," really captures the core idea of how language understanding leads to visual localization without needing specific segmentation training <ref:2605.13122#pg1>.

Conclusion: Tom: So, we've been looking at this paper called "Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation." Basically, they’re showing that these models used for editing images can do referring image segmentation without any specific training data.

Jane: That sounds like a big deal. So, what does it mean when they say the models inherently perform language-conditioned visual semantic grounding? What are we actually talking about there?

Lu: They’re saying that when you ask an AI to edit a picture based on text, it already understands *where* the object is and *what* it is, even before you tell it to change anything. It’s built into the way they process things.

Meng: But we ship these models and they don't automatically know how to segment pixels just because they can edit a picture. What’s the actual mechanism here that lets them do that for segmentation?

Lalam: The core idea is that strong foreground-background separability shows up in the internal representations of these models at the very first denoising timestep. That’s when it starts, before any visible change happens to the image.

Tom: That’s a concrete number—the first denoising timestep. It's not some abstract concept; it's a point in the generation process where they already have this structure ready for use. So how does that translate into segmentation masks?

Jane: The paper says they decompose the localization into two parts: one part to figure out *where* to focus, and another part to determine *what* to segment based on features. That sounds like a really practical way to tackle the problem.

Lu: Exactly. They use attention-based spatial priors for the 'where,' and then feature-based semantic discrimination for the 'what.' It’s not one single magical step; it’s combining these two ideas.

Meng: From a production standpoint, that sounds efficient because they only need to look at those intermediate representations from just one denoising step to get the mask. That cuts down on the processing time significantly compared to methods that need full image synthesis first.

Tom: Right, so we're looking at this training-free framework repurposing existing models. It suggests that object localization in referring image segmentation can arise naturally from the semantic understanding models already have for editing instructions.

Jane: It implies we might not need massive, task-specific datasets just to teach an AI how to find objects in a picture when given a text description. That’s a huge shift for how we approach zero-shot tasks.

Lalam: I'm pretty sure the evidence is strong here. Their experiments on RefCOCO, RefCOCO+, and RefCOCog show this method consistently beats existing zero-shot baselines across different metrics like oIoU and mIoU.

Tom: So, to sum up the title—"Early Semantic Grounding"—it’s about unlocking the visual knowledge already present in generative models through careful decomposition. What we have here is a clear path forward for applying these capabilities to many image understanding tasks without starting from scratch.

Jane: It definitely opens up some new avenues for how we can design AI systems that respond intelligently to natural language instructions. Now, with this foundation established, the real question is how reliable this works when you move it out of the controlled experiment and into a messy, real-world environment.

Jingxuan He, Xiyu Wang, Yunke Wang, Mengyu Zheng, *Chang Xu

The University of Sydney

cs.CV

Submitted: 2026-05-13

Updated: 2026-10-19

Comments: Accepted to NeurIPS 2026

Code: https://github.com/black-forest-labs/flux

Project page: https://jessie459.github.io/EarlySemanticGrounding

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: The gist The work proposes a training-free framework that repurposes pretrained image editing models for zero-shot referring image segmentation by exploiting their intermediate representations,

Key concepts

Instruction-based Image Editing (IIE): Models that modify specific image regions based on natural language instructions. This capability suggests the models are already performing language-conditioned visual semantic grounding, meaning they understand what parts of an image correspond to certain concepts described in text.
Foreground-Background Separability: A measure of how clearly the model distinguishes between objects (foreground) and their surroundings (background) within its internal representations. The paper shows this separability is strong as early as the first denoising step, even before any visible image changes occur during generation.
Attention-based Spatial Priors (ARP): A component of the proposed framework that estimates the coarse location of an object. It uses cross-attention to look at how different prompt tokens relate to image parts, creating a map that suggests where the target region is located spatially.
Feature-based Semantic Discrimination (FSL): The second component that determines what exactly should be segmented. This module creates prototypes for foreground and background features and uses cosine similarity to classify individual pixels based on how well they match these semantic prototypes.

Terminology

Grounding — 3× in this paper · explained in 3 episodes
Segmentation Masks — 1× in this paper · explained in 2 episodes
Training-Free Framework — 1× in this paper · explained in 2 episodes

Summary

The gist The work proposes a training-free framework that repurposes pretrained image editing models for zero-shot referring image segmentation by exploiting their intermediate representations, revealing strong foreground-background separability at the earliest denoising timestep.

How it works

The approach reformulates referring image segmentation as an instruction-based image editing problem, showing that generative models inherently support discriminative visual grounding without requiring task-specific supervision.
Strong foreground-background separability emerges in the internal representations of these models at the earliest denoising timestep, well before any visible image transformation occurs.
The framework decomposes localization into two complementary components: attentionbased spatial priors that estimate where to focus, and feature-based semantic discrimination that determines what to segment.

Feature Separability Analysis

The paper analyzes the foreground-background separability of MMSA features extracted from clean image tokens to define a separability score J(t,l). This metric measures the ratio between inter-class distance and intra-class variance, where larger values indicate stronger feature discriminability. The analysis on FLUX.2-klein and Step1X-Edit shows that both models exhibit strong feature separability as early as the first denoising timestep, indicating that semantic grounding is already established at early timesteps.

Attention-based Region Proposal (ARP)

The ARP module derives a coarse spatial prior from cross-attention to estimate the target region. This is achieved by extracting the image-to-prompt submatrix and aggregating it across prompt tokens and DiT blocks to form the Coarse Attention Map (CAM). To improve spatial coherence, an affinity-based propagation mechanism is introduced that leverages self-attention to propagate cross-attention responses across semantically related regions, resulting in the Refined Attention Map (RAM).

Feature-based Semantic Localization (FSL)

The FSL module exploits the strong semantic separability of latent features by constructing foreground and background prototypes, Pfg and Pbg, via masked average pooling over the normalized features. A binary mask M(t) is obtained by thresholding the refined attention map, and pixel-wise classification is performed using cosine similarity between each prototype and the upsampled features to obtain the final segmentation mask S.

Experimental Results and Efficiency

Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg demonstrate that the method consistently outperforms existing zero-shot baselines, with Step1X-Edit achieving the best results across dataset splits under both oIoU and mIoU metrics. The framework is training-free, operating by extracting internal representations from a single timestep and leveraging them for localization. Furthermore, the proposed method reduces inference time to approximately one second, achieving a speedup of more than 30× over RefAM.

Ablation Studies and Conclusions

Ablation studies confirm that CAMs alone yield limited performance, while introducing RAMs leads to substantial improvements across all datasets. Incorporating FSL further improves oIoU and mIoU, highlighting the importance of leveraging semantically separable latent features for precise foreground-background discrimination. The work concludes that object localization in RIS can arise naturally from the semantic grounding capability in generative models. This suggests that referring image segmentation can be effectively addressed by decomposing localization into attention-based spatial priors and feature-based semantic discrimination.

--- Page 9 ---

[1] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923

[2] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18392–18402, 2023

[3] Huanqia Cai, Sihan Cao, Ruoyi Du, Peng Gao, Steven Hoi, Zhaohui Hou, Shijie Huang, Dengyang Jiang, Xin Jin, Liangchen Li, et al. Z-image: An efficient image generation foundation model with single-stream diffusion transformer. arXiv preprint arXiv:2511.22699

[4] Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. Sam 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719

[5] Xiaofu Chen, Yaxin Luo, Gen Luo, Jiayi Ji, Henghui Ding, and Yiyi Zhou. Dvin: Dynamic visual routing network for weakly supervised referring expression comprehension. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14347–14357, 2025

[6] Silin Cheng, Yang Liu, Xinwei He, Sebastien Ourselin, Lei Tan, and Gen Luo. Weakmcn: Multi-task collaborative network for weakly supervised referring expression comprehension and segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 9175–9185, 2025

[7] Yong Xien Chng, Henry Zheng, Yizeng Han, Xuchong Qiu, and Gao Huang. Mask grounding for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26573–26583, 2024

[8] Ming Dai, Wenxuan Cheng, Jiang-jiang Liu, Sen Yang, Wenxiao Cai, Yanpeng Sun, and Wankou Yang. Deris: Decoupling perception and cognition for enhanced referring image segmentation through loopback synergy. arXiv preprint arXiv:2507.01738

[9] Qiyuan Dai and Sibei Yang. Curriculum point prompting for weakly-supervised referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13711–13722, 2024

[10] Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics, 7(2): 179–188, 1936

[15] Anna Kukleva, Enis Simsar, Alessio Tonioni, Muhammad Ferjad Naeem, Federico Tombari, Jan Eric Lenssen, and Bernt Schiele. Refam: Attention magnets for zero-shot referral segmentation. arXiv preprint arXiv:2509.22650

[16] Black Forest Labs. Flux. https://github.com/black-forest-labs/flux, 2024

[17] Black Forest Labs. FLUX.2: Frontier Visual Intelligence. https://bfl.ai/blog/flux-2, 2025

[18] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014

[19] Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, Chunrui Han, et al. Step1x-edit: A practical framework for general image editing. arXiv preprint arXiv:2504.17761

[20] Sihan Liu, Yiwei Ma, Xiaoqing Zhang, Haowei Wang, Jiayi Ji, Xiaoshuai Sun, and Rongrong Ji. Rotated multi-scale interaction network for referring remote sensing image segmentation.

Improvements for AI systems

The improved system can perform zero-shot referring image segmentation (RIS) by repurposing pretrained instruction-based image editing models for pixel-level localization, specifically by exploiting strong foreground-background separability emerges in the internal representations of these models at the earliest denoising timestep. This allows the system to produce accurate and robust localization without requiring full image synthesis.
The system will implement a training-free framework that decomposes localization into two complementary components: attentionbased spatial priors that estimate where to focus and feature-based semantic discrimination that determines what to segment. This decoupling ensures the model achieves high performance by leveraging both spatial guidance and feature separability, as demonstrated by the results in Table 2.
The system can achieve superior performance over existing zero-shot baselines on benchmarks like RefCOCO, RefCOCO+, and RefCOCog, surpassing methods that rely on costly pixel-level annotations. This is achieved by utilizing the proposed mechanism where pixel-wise classification is performed via cosine similarity between the upsampled features and constructed foreground/background prototypes.
The system will be optimized for inference efficiency, achieving a speedup of more than 30× over RefAM by operating using only a single denoising step. This drastically reduces inference time compared to prior methods that require multiple denoising steps or complex linguistic preprocessing.
The framework can adapt its localization strategy based on the analysis of latent representations, such as utilizing features from deep layers (approximately two-thirds of the model depth) for feature extraction in the FSL module, as this configuration yields peak segmentation performance across all benchmarks.