Structure-Semantic Co-optimized Latent Diffusion Model for Fast Visual Anagram Synthesis

arXiv:2606.16241 · cs.CV · Submitted 2026-06-15 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Structure-Semantic Co-optimized Latent Diffusion Model for Fast Visual Anagram Synthesis".

Jane: Here's the gist of the paper. The gist The S2CO-Anagram method proposes a structure-semantic co-optimization framework to generate high-quality visual anagram images with superior visual harmony and semantic faithfulness while achieving substantially faster inference speed How it works The overall architecture of S2CO-Anagram is built upon a pretrained Latent Diffusion Model (LDM) and inherits the multi-view parallel denoising paradigm.

Tom: The model is conditioned on text embeddings (y1, y2) from the two prompts and a null-text embedding y∅ from an empty prompt for classifier-free guidance (CFG).

Jane: At each denoising step t, predicted noises ϵ(1)t and ϵ(2)t are computed using the guidance strength ω in CFG.

Tom: These predicted noises are then integrated via the attention-guided noise fusion module to yield the fused noise ϵˆt that incorporates semantic concepts of both P1 and P2.

Title and authors: Tom: So we're looking at this paper titled Structure-Semantic Co-optimized Latent Diffusion Model for Fast Visual Anagram Synthesis by Xiang Gao and Yunpeng Jia. I was reading the abstract, and it sounds like they’re tackling a few problems that other work has run into when trying to make these visual anagrams.

Jane: Yeah, it seems like their main goal is generating high-quality visual anagrams, but they are focused on making them fast and better looking than what we see right now. It mentions that existing methods often have issues with computational inefficiency and weak semantic fidelity, so this paper aims to fix those things by using a structure-semantic co-optimization framework.

Lu: I see the title suggests they are balancing two things: structure, which is the spatial arrangement of things like flipping or rotating an image, and semantics, which is making sure the meaning from the text prompts matches across those different views. It’s about optimizing both simultaneously in a latent diffusion model setting.

Meng: From my side, I'm interested in the "fast" part. If we can do this with minimal computational cost while maintaining quality, that opens up possibilities for deployment where speed matters more than just generating something perfect every single time.

Lalam: It sounds like they are proposing a new way to guide the diffusion process by optimizing how structure and meaning interact throughout the denoising steps, which is what the S2CO framework is all about.

The paper's summary: Tom: Okay, so they dive into how this works. It builds on a pretrained Latent Diffusion Model, but instead of just doing one thing at a time, they use a multi-view parallel denoising paradigm. They condition the model on two text embeddings from different prompts and an empty prompt embedding for classifier-free guidance.

Jane: That’s where it gets interesting. At every denoising step, they predict noise for both views using some guidance strength omega, and then they fuse those predicted noises using an attention-guided module to get a fused noise prediction. This fused noise incorporates the semantic concepts from both prompts.

Lu: They introduce three specific innovations to handle the problems of structural divergence and semantic dilution that usually plague multi-view setups. First, they have this null-text structure alignment optimization, which uses a View-Cycle Low-Frequency Consistency loss to penalize structural deviation in the DCT domain across views.

Meng: Penalizing deviation at each step sounds like it adds complexity to the training or inference pipeline. What does that actually do for the output? Does it just keep things looking geometrically correct?

Lalam: It’s designed to suppress structural divergence along the synchronous denoising trajectory, which they say promotes semantic faithfulness and visual harmony from a frequency-domain perspective. That means it’s trying to keep the shapes aligned even as the model refines them.

Tom: And then they have the second point: semantic enhancement optimization. This tunes the input latent at each step to maximize cross-attention activations associated with key concepts of both prompts, which is supposed to enhance conceptual fidelity and discernibility of the results.

Jane: So it’s not just about making sure it looks right spatially; they are actively trying to make sure the concepts in the image match what you asked for in both texts.

Lu: Exactly. And finally, they have attention-guided noise fusion, which uses those extracted attention maps to guide how the noise is blended spatially, aiming to preserve semantically salient features of both views more faithfully than just simple noise averaging.

The paper's improvements: Tom: So if I’m tracking the mechanics here, they set up a few concrete things. They are using a few-step LDM built on SDXL-Turbo, and they set the total number of denoising steps to T=four with a guidance strength omega=one point five. The inner loop step K is set to five as the optimal trade-off between model performance and computational efficiency for this approach.

Meng: Four steps is pretty low for high quality, isn't it? What’s the actual comparison they used? They claim they outperform related approaches qualitatively and quantitatively in visual anagram generation while achieving substantially faster inference time compared to Diffusion Illusions or LookingGlass.

Lalam: The evaluation on CIFAR10 two-views showed that S2CO-Anagram outperforms related approaches in all measured dimensions, including text fidelity metrics like Amin, C, and Aavg, and the visual aesthetic metric AS.

Lu: What’s really interesting from the ablation studies is that they confirmed Null-Text Structure Alignment—NTSA—plays the most crucial role in promoting semantic fidelity of the generated anagram images. That suggests structural alignment is a major bottleneck for getting good results here.

Jane: And structurally, they showed that NTSA consistently narrows the structure distance between the generated two images in the original view, which means it’s actually doing its job at keeping them spatially consistent.

Tom: So structurally and semantically, they argue that NTSA is key, but they also have semantic enhancement and attention-guided noise fusion as necessary components to get that high quality. It’s a three-part system working together.

Conclusion: Jane: To wrap things up on the Structure-Semantic Co-optimized Latent Diffusion Model for Fast Visual Anagram Synthesis, the main implication is that you can generate high-quality visual anagrams efficiently without needing those very high computational costs associated with other methods. It provides a way to achieve superior visual harmony and semantic faithfulness.

Tom: Yeah, it moves the needle on making these kinds of artistic tools more accessible for people who want to create them without having massive GPU resources or waiting forever for an inference to finish. It’s about efficiency meeting quality here.

Lu: From a research standpoint, this work demonstrates that integrating structure alignment directly into the guidance mechanism using null-text embeddings can be a very effective way to manage the inherent challenges of multi-view synthesis. It gives us a concrete mechanism for controlling divergence across views.

Meng: For deployment, if this method is truly faster than Diffusion Illusions and it maintains high fidelity, that’s what matters because we need models that can scale up without needing massive infrastructure for every single generation task.

Lalam: I think the overall message of the paper is that by focusing on these three co-optimization techniques—structure alignment, semantic enhancement, and attention-guided noise fusion—you can overcome the structural divergence and semantic dilution issues in multi-view denoising effectively. It’s a practical framework for better image creation.

Jane: So, to summarize the Structure-Semantic Co-optimized Latent Diffusion Model for Fast Visual Anagram Synthesis, they give us a lightweight model that generates superior anagrams with clear semantic discrimination and high computational efficiency. It’s a useful tool for artists and designers.

Xiang Gao, Yunpeng Jia

Beijing University of Posts and Telecommunications

cs.CV

Submitted: 2026-06-15

Updated: 2026-10-17

Code: https://github.com/christophschuhmann/improved-aesthetic-predictor

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: The gist The S2CO-Anagram method proposes a structure-semantic co-optimization framework to generate high-quality visual anagram images with superior visual harmony and semantic faithfulness while

Key concepts

Null-text structure alignment (NTSA)
This optimization uses a specific loss function to penalize structural differences between the two generated views during the denoising process. It focuses on aligning the spatial structure of clean latents in the DCT domain, which helps suppress structural divergence and prevents artifacts from appearing as images are synthesized.
Semantic enhancement optimization
This technique tunes the input latent at every step to maximize how strongly it activates vision-text cross-attention maps related to the key concepts in both prompts. By focusing on these concept-relevant regions, it ensures that the semantic distinctiveness and expressiveness of the final image are maximized.
Attention-guided noise fusion
Instead of just averaging noise from different views, this module uses attention maps derived from both views to guide how the noises are blended. This spatially adaptive fusion preserves semantically important features from both original prompts more accurately than standard noise blending techniques.

Terminology

Summary

The gist The S2CO-Anagram method proposes a structure-semantic co-optimization framework to generate high-quality visual anagram images with superior visual harmony and semantic faithfulness while achieving substantially faster inference speed

How it works

The overall architecture of S2CO-Anagram is built upon a pretrained Latent Diffusion Model (LDM) and inherits the multi-view parallel denoising paradigm. The model is conditioned on text embeddings (y1, y2) from the two prompts and a null-text embedding y∅ from an empty prompt for classifier-free guidance (CFG). At each denoising step t, predicted noises ϵ(1)t and ϵ(2)t are computed using the guidance strength ω in CFG. These predicted noises are then integrated via the attention-guided noise fusion module to yield the fused noise ϵˆt that incorporates semantic concepts of both P1 and P2. The next-step denoising latent zt−1 is computed using a modified DDPM update formula involving the fused noise prediction ϵˆt.

Key Innovations in S2CO Framework

The S2CO framework comprises three key innovations to counteract structural divergence and semantic dilution.

  1. Null-text structure alignment optimization is proposed to penalize structural deviation at each denoising step by optimizing the null-text embedding in classifier-free guidance (CFG) with our proposed View-Cycle Low-Frequency Consistency (VLC) Loss. This loss aligns the spatial structure of the predicted clean latents across views in the DCT domain, suppressing structural divergence along the synchronous denoising trajectory and mitigating semantic vagueness and structural artifacts.

  2. Semantic enhancement optimization is proposed to explicitly enhance semantic distinctiveness by optimizing the input latent at each denoising step to maximally activate vision-text cross-attention maps associated with key concepts of P1 and P2. This approach enhances semantic expressiveness by maximizing pixel activation of M˜(1) and M˜(2) w.r.t. the input latent zt, reinforcing semantic expressiveness only in the most concept-relevant regions.

  3. Attention-guided noise fusion is proposed to leverage the concept saliency spatial indication provided by extracted attention maps M˜(1) and M˜(2) for spatially adaptive noise fusion. This scheme promotes semantic saliency and distinctiveness by leveraging cross-attention maps to guide spatially adaptive noise fusion, thereby preserving semantically salient features of both views more faithfully than simple noise averaging.

Model Components and Hyperparameters

S2CO-Anagram utilizes a few-step LDM built upon SDXL-Turbo, an adversarially distilled LDM enabling high-resolution and few-step T2I synthesis. The total number of denoising steps is set to T=4 and the CFG guidance strength is ω=1.5. The inner loop step K in the S2CO framework is set to K=5 as the optimal trade-off between model performance and computational efficiency. The percentile parameter p in semantic enhancement optimization is set to p=60, which was found to yield the best aesthetic quality of the generated anagram images.

Performance and Evaluation

Quantitative evaluations on CIFAR10 2-views showed that S2CO-Anagram outperforms related approaches in all measured dimensions, including text fidelity metrics (Amin, C, Aavg) and visual aesthetic metric AS. The method delivers superior image resolution with significantly lower inference time compared to other methods like Diffusion Illusions [5] and LookingGlass [7]. Ablation studies confirmed that Null-Text Structure Alignment (NTSA) plays the most crucial role in promoting semantic fidelity of the generated anagram images. Furthermore, the quantitative verification of structural divergence showed that NTSA consistently narrows the Structure Distance between the generated two images in the original view. The S2CO-Anagram framework demonstrates that high-quality visual anagram synthesis is attainable using latent-based T2I diffusion models, even distilled few-step ones.

Conclusion and Impact

S2CO-Anagram successfully addresses the structural divergence and semantic dilution issues inherent in multi-view parallel denoising by integrating null-text structure alignment, semantic enhancement, and attention-guided noise fusion. The framework provides a democratizing tool for artists, designers, and content creators by generating high-quality visual anagrams efficiently. This technology has potential applications in educational tools illustrating visual perception principles and therapeutic applications such as visual rehabilitation exercises. Despite its success, limitations include poor performance on realistic styles and difficulty with complex transformations like jigsaw rearrangement. The proposed method ultimately generates the most visually appealing anagram images with the clearest semantic discrimination and the highest computational efficiency.

--- Page 1 ---

Structure-Semantic Co-optimized Latent Diffusion Model for Fast Visual Anagram Synthesis

Xiang Gao, Yunpeng Jia School of Digital Media and Design Arts Beijing University of Posts and Telecommunications

--- Page 2 ---

Diagonal flipping Vertical flipping 90° rotation 180° rotation “oil painting of a girl” “oil painting of an elderly man” “watercolor painting of a boy” “watercolor painting of a granny” “watercolor painting of Paris” “watercolor painting of a boy” “watercolor painting of a park” “watercolor painting of Einstein” “painting of a boat” “painting of a truck” “painting of a street corner“,“watercolor paintingof city street”, etc. Figure 1: Our method enables fast synthesis

--- Page 3 ---

DM DM DM ViewV-Cieywc-lCeyLcloewLo-Fwr-eFqreuqeunecnycCy oCnosnissitsetnenccyyLoss Semantic Enhancement Loss Attention -Guided Noise Fusion Figure 2: Method overview. Our method synchronously denoises two latent views linked by a specified transformation followed by fusing their noise estimations at each denoising step. Grounding on this process, a novel optimization framework comprising null-text structure alignment, semantic enhancement, and attention-guided noise fusion is proposed to enhance the generated results

--- Page 4 ---

DM DM DM DM DM DDM DDM DDM DDM ViewV-Cieywc-lCeyLcloewLo-Fwr-eFqreuqeunecnycCy Loss Semantic Enhancement Loss Attention -Guided Noise Fusion Figure 2: Method overview. Our method synchronously denoises two latent views linked by a specified transformation followed by fusing their noise estimations at each denoising step. Grounding on this process, a novel optimization framework comprising null-text structure alignment, semantic enhancement, and attention-guided noise fusion is proposed to enhance the generated results

--- Page 5 ---

Figure 2: Method overview. Our method synchronously denoises two latent views linked by a specified transformation followed by fusing their noise estimations at each denoising step. Grounding on this process, a novel optimization framework comprising null-text structure alignment, semantic enhancement, and attention-guided noise fusion is proposed to enhance the generated results

--- Page 6 ---

“ oil painting of LeBron James ” “ oil painting of Kobe Bryant” “painting of a soldier” “painting of a robot” “a charcoal drawing of a penguin” “a charcoal drawing of a goat” “painting of a rabbit” etc. Figure 3: Visual anagram synthesis results

--- Page 7 ---

Diffusion Illusions [5] exhibits relatively weak semantic fidelity, where key concepts from text prompts are not sufficiently and clearly expressed, which we attribute to the instability, high-variance gradient estimates, and mode collapse inherent in the score distillation optimization. SyncTweedies [44] generates results that are plagued by non-trivial noise and artifacts. Visual Anagrams [30] and Anagram-MTL [76] produce results with limited semantic clarity, where key concepts from text prompts are not sufficiently discernible. LookingGlass [7] synthesizes higher-resolution anagram images by leveraging latent-based model, though its dependence on image-space warping operations introduces a lack of smoothness and naturalness into the synthesized results.

Improvements for AI systems

  1. Bold header: Null-Text Structure Alignment (NTSA) optimization for structural fidelity. This component penalizes structural deviation at each denoising step by optimizing the null-text embedding in classifier-free guidance (CFG) with our proposed View-Cycle Low-Frequency Consistency (VLC) Loss, which is designed to suppress structural divergence along the synchronous denoising trajectory.

  2. Bold header: Semantic Enhancement Optimization for conceptual expressiveness. This optimization tunes the input latent to maximize cross-attention activations associated to key concepts of P1 and P2, which helps in enhancing semantic distinctiveness by elevating cross-attention activations associated with key concepts.

  3. Bold header: Attention-Guided Noise Fusion for spatially adaptive noise blending. By leveraging attention maps, this scheme is formulated as: ϵˆt = ϵ(1)t · M˜ (1)/(M˜ (1) + g−1(M˜ (2))) + g−1(ϵ(2)t) · g−1(M˜ (2))/(M˜ (1) + g−1(M˜ (2)), which is proposed to preserve semantically salient features of both views more faithfully than simple noise averaging.

  4. Bold header: S2CO-Anagram generation capability. The improved system can generate higher-resolution anagram images with noticeably superior visual harmony and semantic faithfulness than related SOTA approaches, all while achieving substantially faster inference speed.

Sources

Related papers