WAFFLE: Finetuning Multi-Modal Models for Automated Front-End Development

arXiv:2410.18362 · cs.SE, cs.CL, cs.CV · Submitted 2024-10-24 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "WAFFLE: Finetuning Multi-Modal Models for Automated Front-End Development".

Jane: Here's the gist of the paper. The gist: WAFFLE introduces a fine-tuning pipeline for UI-to-HTML code generation that utilizes structure-aware attention and contrastive learning to improve multi-modal models' understanding of HTML structure and visual differences.

Tom: WAFFLE's Core Strategy WAFFLE is a new fine-tuning strategy that uses a structure-aware attention mechanism to improve LLMs’ understanding of HTML’s structure and a contrastive fine-tuning approach to align LLMs’ understanding of UI images and HTML code Abstract, WAFFLE, a new fine-tuning strategy that uses a structure-aware attention mechanism to improve LLMs’ understanding of HTML’s structure and a contrastive fine-tuning approach to align LLMs’ understanding of UI images and HTML code.

Jane: Key Components of WAFFLE The approach combines two main techniques to tackle the challenges in UI-to-HTML code generation WAFFLE, a new fine-tuning strategy that uses a structure-aware attention mechanism to improve LLMs’ understanding of HTML’s structure and a contrastive fine-tuning approach to align LLMs’ understanding of UI images and HTML code.

Paper summary: Jane: So wrapping up this discussion on WAFFLE: Finetuning Multi-Modal Models for Automated Front-End Development, the authors are proposing that this combination of structure-aware attention and contrastive learning is a solid way to make AI better at turning UI designs into functional webpages.

Lu: The title points out that they are focusing on these multi-modal models specifically, meaning the research isn't just about making code generation better in isolation, but about integrating visual understanding with textual generation.

Meng: From a deployment view, it suggests that if we want robust front-end AI tools, we should probably look for fine-tuning methods that explicitly bake in structural knowledge like this instead of relying on pure prompt engineering or standard fine-tuning.

Lalam: The paper shows that WAFFLE outperforms standard fine-tuning on two backbone MLLMs, specifically showing improvements of up to nine point zero zero pp higher HTML match, zero point zero nine eight two higher CW-SSIM, thirty-two point nine nine higher CLIP, and twenty-seven point one two pp for LLEM on our new benchmark WebSight-Test and an existing benchmark Design2Code.

Tom: So the takeaway here is that this strategy provides measurable improvements across several metrics when you apply it to models that can handle both images and text inputs for front-end tasks.

Jane: It really highlights how much the way we fine-tune these models matters, and WAFFLE seems like a promising direction for researchers looking to build more capable tools that can handle complex visual interfaces reliably.

Conclusion: Tom: So we’re wrapping up on WAFFLE, this paper from VLMWebSight and Moondream2, which is about fine-tuning multi-modal models for front-end development.

Jane: It’s about taking those models that can see images and understand text, and making them way better at turning a design into actual working HTML code.

Tom: Exactly. The authors are showing how adding structure awareness and contrastive learning makes a big difference in how the model understands the layout of HTML versus what it sees visually.

Jane: And they report some solid numbers, showing improvements on benchmarks like WebSight-Test and Design2Code, with things like up to nine percent higher HTML match.

Lu: The core idea is using structure-aware attention to teach the AI the rules of how an HTML tree actually works—like parent styles flowing down to children.

Meng: But it’s not just about structure; they also use contrastive learning to make sure the model notices those tiny visual differences in UI images that might cause code errors.

Lalam: I’m seeing that contrastive learning is key for alignment, specifically helping the model focus on visual details when generating text.

Tom: So, what does this actually mean when we think about deploying these kinds of tools?

Jane: It means we can start trusting AI to handle more complex UI tasks because the models are learning the structure correctly and paying attention to those subtle visual cues.

Lu: The point is that this approach isn't just a trick; it’s a fine-tuning strategy that makes the models more robust when they have to generate code based on a visual prompt.

Meng: From an engineering standpoint, if this works across different models, it suggests we can use this framework to improve any existing multi-modal model for front-end generation without having to start from scratch.

Tom: It’s about making the AI more reliable in production environments where those small structural mistakes can cause big problems.

Jane: So, the paper shows how these two techniques—structure awareness and contrastive learning—work together to give the AI a much better grasp of both what it sees and what it needs to write.

Tom: It’s definitely a solid step forward in getting this kind of specific task done reliably by AI models.

Jane: Now, we’ve seen the results on those benchmarks, but we should probably look closer at how these kinds of fine-tuning methods hold up when you try to apply them to completely different types of visual inputs.

Shanchao Liang, Nan Jiang, Shangshu Qian, Lin Tan

Purdue University

cs.SE, cs.CL, cs.CV

Submitted: 2024-10-24

Updated: 2026-10-20

Code: https://github.com/lt-asset/Waffle

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 86/100

The gist: The gist: WAFFLE introduces a fine-tuning pipeline for UI-to-HTML code generation that utilizes structure-aware attention and contrastive learning to improve multi-modal models' understanding of HTML

Key concepts

Structure-Aware Attention: This mechanism teaches the model the rules of HTML structure by allowing tokens to focus on three specific types of previous code segments: parent elements, preceding sibling elements, and self-attention. This helps the model understand how different parts of the HTML code relate to each other structurally.
Contrastive Learning: This technique is used to teach models to focus on subtle visual differences between UI images and corresponding HTML text. It forces the model to learn which visual details are important for generating correct, distinct code, preventing it from generating identical code for visually different inputs.
HTML Match: This metric measures how closely the generated HTML matches a ground truth design. WAFFLE improved this score by up to 9.00 percentage points on new benchmarks, indicating that the model is much better at producing structurally correct code compared to standard methods.

Terminology

Fine-Tuning — 3× in this paper · explained in 5 episodes
Benchmark — 2× in this paper · explained in 2 episodes
Hierarchical Structure — 2× in this paper · explained in 3 episodes
Attention Mechanism — 1× in this paper · explained in 3 episodes
Language Models (LLMs) — 1× in this paper · explained in 2 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes

Summary

The gist: WAFFLE introduces a fine-tuning pipeline for UI-to-HTML code generation that utilizes structure-aware attention and contrastive learning to improve multi-modal models' understanding of HTML structure and visual differences.

WAFFLE's Core Strategy

WAFFLE is a new fine-tuning strategy that uses a structure-aware attention mechanism to improve LLMs’ understanding of HTML’s structure and a contrastive fine-tuning approach to align LLMs’ understanding of UI images and HTML code Abstract, WAFFLE, a new fine-tuning strategy that uses a structure-aware attention mechanism to improve LLMs’ understanding of HTML’s structure and a contrastive fine-tuning approach to align LLMs’ understanding of UI images and HTML code.

Key Components of WAFFLE

The approach combines two main techniques to tackle the challenges in UI-to-HTML code generation WAFFLE, a new fine-tuning strategy that uses a structure-aware attention mechanism to improve LLMs’ understanding of HTML’s structure and a contrastive fine-tuning approach to align LLMs’ understanding of UI images and HTML code.

Structure-Aware Attention: This mechanism is designed to teach models the domain knowledge of HTML structures Regarding the first challenge, there are three basic structural aspects of HTML code. Firstly, all the styles of the parent element are directly passed to the children unless specifically overridden. Secondly, the layout of the siblings affects each other. Thirdly, nodes are not affected by the subtrees of their siblings Regarding the first challenge, there are three basic structural aspects of HTML code. Firstly, all the styles of the parent element are directly passed to the children unless specifically overridden. Secondly, the layout of the siblings affects each other. Thirdly, nodes are not affected by the subtrees of their siblings.
Contrastive Learning: This approach is used to teach models to focus on important visual differences in UI images and text understanding of HTML code For the second challenge, minor variations in layout structure can still impact the content of the generated code. However, MLLMs often fail to capture these subtle differences and generate identical code for visually distinct inputs. To enable MLLMs to recognize subtle differences in UI images due to minor changes in the code, we adopt contrastive learning (Zhai et al., 2023; Radford et al., 2021; Gao et al., 2021) to the current task to teach MLLMs to focus on important visual differences.

Experimental Setup and Results

WAFFLE was implemented on two backbone MLLMs, VLMWebSight and Moondream2, both fine-tuned using DoRA On top of the fine-tuned MLLMs from the first step, we apply the structure-aware attention and contrastive learning approach. The model’s weights are updated using the AdamW (Loshchilov and Hutter, 2019) optimizer, with the learning rate set to 3e−5 On top of the fine-tuned MLLMs from the first step, we apply the structure-aware attention and contrastive learning approach. The model’s weights are updated using the AdamW (Loshchilov and Hutter, 2019) optimizer, with the learning rate set to 3e−5.

WAFFLE improved backbone MLLMs by achieving up to 9.00 pp higher HTML Match, 0.0982 higher CW-SSIM, 32.99 higher CLIP, and 27.12 pp higher LLEM on our new benchmark WebSight-Test and an existing benchmark Design2Code Models fine-tuned with WAFFLE show up to 9.00 pp (absolute percentage point) higher HTML match, 0.0982 higher CW-SSIM, 32.99 higher CLIP, and 27.12 pp higher LLEM on our new benchmark WebSight-Test and an existing benchmark Design2Code.

Impact of Components

Ablation studies confirmed that both key components contribute to better cross-modality understanding and more robust code generation WAFFLE is a generalizable fine-tuning pipeline that can benefit any pre-trained MLLMs. Contrastive learning and structure-aware attention significantly improve performance over standard fine-tuning Summary: Contrastive learning and structureaware attention significantly improve performance over standard fine-tuning.

The integration of structure-aware attention brings stability to model generation, making models more robust against intermediate mistakes, and reducing the performance drop by 23.31%, from 27.55% to 4.34% WAFFLE-attn, the averaged CW-SSIM across the 20 samples drops by 0.2205 (from 0.8002 to 0.5797) if the model makes intermediate mistakes Summary: Integrating structure-aware attention brings stability to model generation, making models more robust against intermediate mistakes, and reducing the performance drop by 23.31%, from 27.55% to 4.34%.

WAFFLE is model-independent and can be applied to improve any MLLMs for UI-to-HTML code generation We highlight that WAFFLE as a fine-tuning approach, is model-independent and can be applied to improve any MLLMs for UI-to-HTML code generation.

Limitations and Future Directions

One limitation of WAFFLE is that it has only been implemented on two models: VLM-WebSight and Moondream2 One limitation of WAFFLE is that it has only been implemented on two models: VLM-WebSight and Moondream2. Another limitation is that the metrics we use do not fully capture human evaluation Another limitation is that the metrics we use do not fully capture human evaluation. HTML-Match overlooks CSS styling, and metrics like CW-SSIM, CLIP, and LLEM are similarity-based, which can lead to unreliable scores Another limitation is that the metrics we use do not fully capture human evaluation. HTML-Match overlooks CSS styling, and metrics like CW-SSIM, CLIP, and LLEM are similarity-based, which can lead to unreliable scores.

Conclusion

WAFFLE introduces a structure-aware attention mechanism to capture HTML structure and employs contrastive learning to align visual and textual understanding WAFFLE introduces a structure-aware attention mechanism to capture HTML structure and employs contrastive learning to align visual and textual understanding, aiding MLLMs in distinguishing subtle webpage differences. WAFFLE outperforms standard finetuning on two backbone MLLMs, with improvements of up to 9 pp in HTML Match, 0.0982 in CWSSIM, 32.99 in CLIP, and 27.12 pp for LLEM WAFFLE outperforms standard finetuning on two backbone MLLMs, with improvements of up to 9.00 pp higher HTML-Match, 0.0982 higher CW-SSIM, 32.99 higher CLIP, and 27.12 pp for LLEM on our new benchmark WebSight-Test and an existing benchmark Design2Code. WAFFLE is model-independent and can enhance any MLLMs for UI-to-HTML code generation WAFFLE is model-independent and can enhance any MLLMs for UI-to-HTML code generation.

How it works

The structure-aware attention captures the structure information of the HTML code by explicitly allowing tokens to focus on three types of previous code segments that are the most relevant WAFFLE implements a novel attention mask that provides each element with a pruned view of all previous tokens, including parent-attention, sibling-attention, and self-attention.

The structure-aware attention has three parts:

Parent-Attention: This is from each element’s tokens to its parent element’s tokens Parent-Attention. The parent-attention is from each element’s tokens to its parent element’s tokens.
Sibling-Attention: This is from each element’s tokens to its preceding sibling elements’ tokens Sibling-Attention.

Improvements for AI systems

Structure-Aware Attention for HTML Code Generation: Implement a novel attention mask that provides each element with a pruned view of all previous tokens, including parent-attention, sibling-attention, and self-attention to teach models the structure knowledge of HTML code. This allows the model to focus on parts of the code that have the most influence on the resulting UI design.
Contrastive Learning for Visual and Textual Alignment: Apply contrastive learning to boost MLLMs’ understanding of the details in the rendered UI images and the HTML code, specifically by maximizing similarity between image and code embeddings, which aims to teach MLLMs to focus on important visual differences.
Improved Robustness Against Intermediate Errors: Integrate structure-aware attention to enhance model stability against generation errors, as demonstrated by how WAFFLE reduces the drop in CW-SSIM when the model makes intermediate mistakes from 27.55% to 4.34%.
Enhanced Cross-Modality Alignment: Use contrastive learning objectives, defined by the loss function Lcl, to maximize similarity between code and image embeddings, which is shown to enhance MLLM visualtextual alignment, making image and code embeddings more cohesive.
Domain-Specific Code Generation Capability: The improved AI system can generate functional HTML/CSS for UI designs that are significantly better than standard fine-tuning, achieving improvements up to 9.00 pp higher HTML Match and 27.12 pp higher LLEM on benchmarks like WebSight-Test and Design2Code.

Abstract

Web development involves turning UI designs into functional webpages, which can be difficult for both beginners and experienced developers due to the complexity of HTML's hierarchical structures and styles. While Large Language Models (LLMs) have shown promise in generating source code, two major challenges persist in UI-to-HTML code generation: (1) effectively representing HTML's hierarchical structure for LLMs, and (2) bridging the gap between the visual nature of UI designs and the text-based format of HTML code. To tackle these challenges, we introduce Waffle, a new fine-tuning strategy that uses a structure-aware attention mechanism to improve LLMs' understanding of HTML's structure and a contrastive fine-tuning approach to align LLMs' understanding of UI images and HTML code. Models fine-tuned with Waffle show up to 9.00 pp (percentage point) higher HTML match, 0.0982 higher CW-SSIM, 32.99 higher CLIP, and 27.12 pp higher LLEM on our new benchmark WebSight-Test and an existing benchmark Design2Code, outperforming current fine-tuning methods.