Loomis Painter: Learning Painting Processes by Reversing Image-to-Video Generation

The gist: The authors propose a unified framework for multi-media painting process generation that employs cross-medium style augmentation and a reverse-painting training strategy to ensure consistent texture evolution and human-aligned generation.

Introduction

Sketching and painting are creative processes that are hard to master, but existing instructional videos lack interactivity and personalization because they are inherently passive While recent generative models have advanced artistic image synthesis, they struggle to generalize across media and often show temporal or structural inconsistencies, hindering faithful reproduction of human creative workflows. The authors propose a unified framework for multi-media painting process generation with a semantics-driven style control mechanism that embeds multiple media into a diffusion model’s conditional space and uses cross-medium style augmentation. This enables consistent texture evolution and process transfer across styles. A reverse-painting training strategy further ensures smooth, human-aligned generation.

Methodology

The method combines two processoriented components: cross-media conditioning, which infuses medium-aware semantics to guide strokes and textures, and reverse-painting learning, which aligns temporal supervision with human intuition for a progressive buildup from structure to detail.

1. Video Diffusion: The framework is based on a pretrained Video Diffusion model [31] consisting of a video-VAE to encode a given video from pixel space into latent space. This latent is used to train a Diffusion Transformer (DiT) [19] with the Flow Matching objective. The loss function is defined as L = Ex0,x1,ctext,t∥u(xt, ctext, t; θ) − vt∥2 2, where ctext is the text embedding sequence and u(xt, ctext, t; θ) is the predicted velocity. Image-to-video (I2V) approaches extend an input image into a sparse video tensor VI by placing I in the first frame and padding the remaining T −1 frames with zeros.

2. Art Media Aware Painting Process: Medium-aware semantic embedding constructs a combined semantic prompt p = [m; s] where m is the artistic medium and s is the scene description, which is transformed into a semantic embedding ctext = Etext(p) ∈ R d, (2). This embedding drives both stylistic and procedural characteristics, enabling medium-appropriate workflow synthesis. Cross-media training involves applying a style transformation to obtain I∗, preserving objects and semantics while removing the identity of the original art medium. The standard imageto-video (I2V) loss is defined as LI2V = Ex0,x1,I∗,ctext,t∥u(xt, I∗, ctext, t; θ) − vt∥2 2, where I∗ is the transformed counterpart.

3. Reverse-Painting Learning Strategy: Instead of predicting the next stroke forward in time, the model learns to gradually reveal the painting in reverse order. This reversal introduces a monotonic “detail removal” process, where high-frequency textures gradually fade and color regions simplify into coarse structural blocks.

Dataset Curation and Evaluation

A large-scale dataset of real drawing and painting processes spanning multiple artistic media is constructed, including Acrylic, Oil, Pencil, and Loomis Portraits. The curation pipeline involves temporal trimming using GroundingDINO [16] to isolate the core painting process, canvas localization via GroundingDINO or intensity gradient splitting for split-screen tutorials, frame sampling and occlusion removal using InSPyReNet [10] or BiRefNet [39], and post-processing to remove logos and text with GroundingDINO [16].

The Perceptual Distance Profile (PDP) is a novel metric designed to compare the sequence of a generated video against its ground truth counterpart by measuring the perceptual distance of every frame to the final, completed painting. The PDP score is computed as the L2 distance between two curves, Cgen and Cgt, after interpolating raw distance profiles onto a common time axis.

Experiments and Results

The method was evaluated against baselines including Inverse Painting [2], ProcessPainter [25], and PaintsUndo [29]. Quantitative comparison using LPIPS, CLIP, and Dinov2 metrics shows that the authors' method achieves strong performance on these metrics The Perceptual Distance Profile (PDP) evaluation reveals that their method achieves the best performance across all metrics. Qualitative results show that their method closely mirrors the ground truth by beginning with a structural sketch, gradually layering in color, and incrementally adding detail to reach the final painting. For example, in Figure 8, their method closely mirrors the ground truth.

Conclusion

The trained model demonstrates a deep understanding of the painting process by sketching outlines, deploying hatching techniques in pencil sketches, using layering techniques, and showing a natural understanding of objects and their depth in the scene. The authors have shown that a video diffusion model can successfully recreate the painting process of a given image.

Limitation & Future Work

Limitations include an occlusion detector that cannot detect hand shadows, leading to dark artifacts in pencil painting generations, and the base model struggling to paint portraits since it has never seen them during training. Future work suggests understanding the painting process also requires indicating which colors were selected, how they were mixed, which tools (pencils or brushes) were used, and how to apply them on the canvas.

References

[1] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127

[2] Bowei Chen, Yifan Wang, Brian Curless, Ira Kemelmacher Shlizerman and Steven M Seitz. Inverse painting: Reconstructing the painting process. In SIGGRAPH Asia 2024 Conference Papers, pages 1–11

[3] Nicola Dall’Asen, Willi Menapace, Elia Peruzzo, Enver Sangineto, Yiming Wang, and Elisa Ricci. Collaborative neural painting. CVIU, page 104298

[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020

[5] D.C Dowson and B.V Landau. The frechet distance between multivariate normal distributions. Journal of Multivariate Analysis, 12(3):450–455, 1982

[6] Patrick Esser, Sumith Kulal, Andreas Blattmann et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024

[7] Yoav HaCohen, Nisan Chiprut, Benny Brazowski et al. Ltx-video: Realtime video latent diffusion. arXiv preprint arXiv:2501.00103, 2024

[8] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020

[9] Edward J Hu, Yelong Shen, Phillip Wallis et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3

[10] Taehun Kim, Kunhee Kim, Joonyeong Lee et al. Revisiting image pyramid structure for high resolution salient object detection. In ACCV, pages 108–124, 2022

[11] Black Forest Labs. Flux. urlhttps://github.com/black-forest-labs/flux, 2024

[12] Bin Lin, Yunyang Ge, Xinhua Cheng et al. Open-sora plan: Open-source large video generation model. arXiv preprint arXiv:2412.00131, 2024

[13] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu et al. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

[14] Haotian Liu, Chunyuan Li, Yuheng Li et al. Improved baselines with visual instruction tuning, 2023

[15] Songhua Liu, Tianwei Lin et al. Paint transformer: Feed forward neural painting with stroke prediction. In ICCV, pages 6598–6607, 2021

[16] Shilong Liu, Zhaoyang Zeng et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection.

1. Data Curation Pipeline Improvement: The occlusion detector in Section A.1 Our occlusion detector in the data collection pipeline cannot detect hand shadows, leading to dark artifacts in the training data should be replaced or augmented with a method capable of detecting hand shadows to improve training data quality for pencil painting generations.

2. Cross-Media Style Transfer Enhancement: Implement a more robust style transformation mechanism for cross-media training, specifically addressing the limitation noted that ControlNet follows exactly the outlines of the reference paintings but looses information such as color due to the underlying control mechanism. This would allow the model to better capture medium-specific color and texture evolution.

3. Reverse Painting Strategy Refinement: Modify Section 3.3's reverse-painting learning strategy by incorporating explicit modeling of material properties, such as how different media learn how to uncover the painting from one generated frame to the next, ensuring that the learned progression is physically plausible for the specified medium.

4. Perceptual Distance Profile Application: Utilize the Perceptual Distance Profile (PDP) metric to quantitatively model and guide procedural learning; specifically, use it to measure the creative sequence, i.e., composition, color blocking, and detail refinement, thereby providing a principled way to characterize the trajectory of perceptual changes across frames.

5. Fine-Grained Control Integration: Integrate the semantic conditioning mechanism described in Section 3.2.1 more deeply into the diffusion process to ensure that medium semantics directly influence the temporal evolution of latent feature ht at each timestep t, enabling finer control over procedural patterns like color layering behavior in oil painting or progressive hatching patterns in pencil sketching.

Step-by-step painting tutorials are used for learning artistic techniques, but existing video resources lack interactivity and personalization. While recent generative models have advanced artistic image synthesis, they still struggle to generate temporally and structurally consistent videos. Video models excel in consistent frame generation, yet fine-tuning an image-to-video model on forward painting sequences creates a long-range dependency on the first-frame reference, disrupting its pretrained attention pattern and hindering reconstruction. We address this issue by training on reversed sequences that better preserve the reference-attention behavior. Building on this, we present Loomis Painter, a framework for reconstructing coherent painting processes. We further introduce a media-transfer training strategy that decouples input appearance from the target process, enabling generation across different artistic media. Trained on a curated dataset of real-world painting processes, our method outperforms existing baselines across LPIPS, DINO, and CLIP metrics and is consistently preferred in expert and non-expert user studies. Code available at https://github.com/Markus-Pobitzer/wlp.